DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control

Authors: jeff  zichen
  cui
hengkai  pan
aadhithya  iyer
siddhant  haldar
lerrel  pinto
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 22:21:43

Abstract: Imitation learning has proven to be a powerful tool for training complex visuo-motor policies. However, current methods often require hundreds to thousands of expert demonstrations to handle high-dimensional visual observations. A key reason for this poor data efficiency is that visual representations are predominantly either pretrained on out-of-domain data or trained directly through a behavior cloning objective. In this work, we present DynaMo, a new in-domain, self-supervised method for learning visual representations. Given a set of expert demonstrations, we jointly learn a latent inverse dynamics model and a forward dynamics model over a sequence of image embeddings, predicting the next frame in latent space, without augmentations, contrastive sampling, or access to ground truth actions. Importantly, DynaMo does not require any out-of-domain data such as Internet datasets or cross-embodied datasets. On a suite of six simulated and real environments, we show that representations learned with DynaMo significantly improve downstream imitation learning performance over prior self-supervised learning objectives, and pretrained representations. Gains from using DynaMo hold across policy classes such as Behavior Transformer, Diffusion Policy, MLP, and nearest neighbors. Finally, we ablate over key components of DynaMo and measure its impact on downstream policy performance. Robot videos are best viewed at https://dynamo-ssl.github.io.

Main paper: 
 introduction: 
 Learning visuo-motor policies from human demonstrations is an exciting approach for training difficult control tasks in the real world [1] [2] [3] [4] [5] . However, a key challenge in such a learning paradigm is to efficiently learn a policy with fewer expert demonstrations. To address this, prior works have focused on learning better visual representations, often by pretraining on large Internet-scale video datasets [6] [7] [8] [9] [10] [11] . However, as shown in Dasari et al. [12] , these out-of-domain representations may not transfer to downstream tasks with very different embodiments and viewpoints from the pretraining dataset.An alternative to using Internet-pretrained models is to train the visual representations 'in-domain' on the demonstration data collected to solve the task [13, 4] . However, in-domain datasets are much smaller than Internet-scale data. This has resulted in the use of domain-specific augmentations [13] to induce representational invariances with self-supervision or to collect larger amounts of demonstrations [2, 14] . The reliance of existing methods on large datasets might suggest that in-domain self-supervised pretraining is ineffective for visuo-motor control, and we might be better with simply training end-to-end. In this work, we argue the contrary -in-domain self-supervision can be effective with a better training objective that extracts more information from small datasets. Prevalent approaches for using self-supervision in downstream control often make a bag-of-frames assumption, using contrastive methods [15, 16] or masked autoencoding [11, 8] on individual frames for self-supervision. Most of these approaches ignore a rich supervision signal: action-based causality. Future observations are dependent on past observations, and unobserved latent actions. Can we obtain a good visual representation for control by simply learning the dynamics? In fact, this idea is well-established in neuroscience: animals are thought to possess internal models of the motor apparatus and the environment that facilitate motor control and planning [17] [18] [19] [20] [21] [22] [23] [24] .In this work, we present Dynamics Pretraining for Visuo-Motor Control (DynaMo), a new selfsupervised method for pretraining visual representations for visuomotor control from limited indomain data. DynaMo jointly learns the encoder with inverse and forward dynamics models, without access to ground truth actions [25, 26] .To demonstrate the effectiveness of DynaMo, we evaluate our representation on four simulation suites -Franka Kitchen [27] , Block Pushing [28] , Push-T [3] , and LIBERO [29] , as well as eight robotic manipulation tasks on two real-world environments. Our main findings are summarized below:1. DynaMo exhibits an overall 39% improvement in downstream policy performance over prior state-of-the-art pretrained and self-supervised representations, especially on the harder closed-loop control tasks in Block Pushing and Push-T (Table 1 ), and on real robot experiments (Table 2 ).2. DynaMo is compatible with various policy classes, can be used to fine-tune pretrained weights, and works in the low-data regime with limited demonstrations on a real-world Allegro hand (Tables 4, 5 , and 2 respectively).3. Through an ablation analysis, we study the impact of each component in DynaMo on downstream policy performance ( §4.6).All of our datasets, and training and evaluation code will be made publicly available. Videos of our trained policies can be seen here: https://dynamo-ssl.github.io.  
visual pretraining for policy learning: 
 Our goal is to pretrain the visual encoder f θ using a dataset of sequential raw visual observations D = {o t } t to support downstream policy learning. During pretraining, we do not assume access to the ground-truth actions {a t } t .Prior work has shown that pretraining encoders on large out-of-domain datasets can improve downstream policy performance [6] [7] [8] [9] [10] [11] . However, such pretraining may not transfer well to tasks with different robot embodiments [12] .Alternatively, we can directly pretrain the encoder in-domain using self-supervised methods. One approach is contrastive learning with data augmentation priors, randomly augmenting an image twice and pushing their embeddings closer. Another approach is denoising methods, predicting the original image from a noise-degraded sample (e.g. by masking [11, 8, 30] ). A third approach is contrastive learning with temporal proximity as supervision, pushing temporally close frames to have similar embeddings [31, 32] . 
dynamo: 
 Limitations of prior self-supervised techniques: Prior self-supervised techniques can learn to fixate on visually salient features and ignore fine-grained features important for control. We illustrate this limitation using the Block Pushing environment from Florence et al. [28] . In this task, the goal is to push a block into a target square. While the robot arm occupies much of the raw pixel space, the blocks are central to the task despite being smaller in the visual field. Figure 2 visualizes a random frame from the demonstration data and its 20 nearest neighbors in the embedding space learned by several self-supervised techniques.We observe that prior self-supervised methods (details in §4.2) focus on the visually dominant robot, matching the whole robot arm extremely accurately. However, they fail to capture the block positions, which are important to the task despite being much less salient visually.Can we learn a visual encoder that extracts task-specific features better? We know that the demonstrations are sequential: each observation is dependent on the previous observation, and an action (unobserved in this setting). Prior self-supervised methods ignore this sequential structure. Contrastive augmentations [16, 33] and autoencoding objectives [30, 8, 11] assume that the demonstration video is a bag of frames, discarding temporal information altogether. Temporal contrast [32, 31] uses temporal proximity but discards the sequential information in the observations: the contrastive objectives are usually symmetric in time, disregarding past/future order.Instead of a contrastive or denoising objective, we propose a dynamics prediction objective that explicitly exploits the sequential structure of demonstration observations.The key insight of our method is that we can learn a good visual representation for control by modeling the dynamics on demonstration observations, without requiring augmentations, contrastive sampling, or access to the ground truth actions. Given a sequence of raw visual observations (o 1 , . . . , o T ), we jointly train the encoder f θ : o t → s t , a latent inverse dynamics model q(z t:t+h-1 |s t:t+h ), and a forward dynamics model p(ŝ t+1:t+h |s t:t+h-1 , z t:t+h-1 ). We model the actions as unobserved latents, and train all models end-to-end with a consistency loss on the forward dynamics prediction. For our experiments, we use a ResNet18 [34] encoder, and causally masked transformer encoders [35] for the inverse and forward dynamics models. The architecture is illustrated in Figure 3 .First, we sample an observation sequence o t:t+h of length h and compute its representation s t:t+h = f θ (o t:t+h ). For convenience, we will write s t:t+h as s :h , and s t+1:t+h as s 1:h below. At any given step, the distribution of possible actions can be multimodal [5] . Therefore, the forward dynamics transition p(s 1:h |s :h-1 ) can also have multiple modes. To address this, we first model the inverse dynamics q(z :h-1 |s :h ), where z t is the latent transition between frames. We assume z t to be welldetermined and unimodal given consecutive frames {s t , s t+1 }. We have z ∈ R m , s ∈ R d , m ≪ d such that the latent cannot trivially memorize the next frame embedding. Finally, we concatenate (s t , z t ) and predict the one-step forward dynamics p(ŝ 1:h |s :h-1 , z :h-1 ).We compute a dynamics loss L dyn (ŝ, s * ) on the one-step forward predictions ŝt+1:t+h , where s * t+1:t+h are the target next-frame embeddings; and a covariance regularization loss L cov from Bardes et al. [36] on a minibatch of observation embeddings S:EQUATIONFor environments with multiple views, we compute a loss over each view separately and take the mean. We choose λ = 0.04 following Bardes et al. [36] for the total loss L. We find that covariance regularization slightly improves downstream task performance.Naively, this objective admits a constant embedding solution. To prevent representation collapse, for L dyn (ŝ, s * ), we follow SimSiam [37] and set the target embedding s * t := sg(s t ), where sg is the stop gradient operator. Alternatively, our objective is also compatible with a target from a momentum encoder fθ [33, 16] , s * t := st = fθ(o t ), where θ is an exponential moving average of θ. We train all three models end-to-end with the objective in Eq. 1, and use the encoder for downstream control tasks. 
experiments: 
 We evaluate our dynamics-pretrained visual representation on a suite of simulated and real benchmarks. We compare DynaMo representations with pretrained representations for vision and control, as well as other self-supervised learning methods. Our experiments are designed to answer the following questions: (a) Does DynaMo improve downstream policy performance? (b) Do representations trained with DynaMo work on real robotic tasks? (c) Is DynaMo compatible with different policy classes? (d) Can pretrained weights be fine-tuned in domain with DynaMo? (e) How important is each component in DynaMo?We evaluate DynaMo on four simulated benchmarks and two real robot environments (depicted in Figure 4 ). We provide a brief description below with more details included in Appendix A.(a) Franka Kitchen [27] : The Franka Kitchen environment consists of seven simulated kitchen appliance manipulation tasks with a 9-dimensional action space Franka arm and gripper. The dataset has 566 demonstration trajectories, each completing three or four tasks. The observation space is RGB images of size (224, 224) from a fixed viewpoint. We evaluate for 100 rollouts and report the mean number of completed tasks (maximum 4).(b) Block Pushing [28] : The simulated Block Pushing environment has two blocks, two target areas, and a robot pusher with 2-dimensional action space (end-effector translation). Both the blocks and targets are colored red and green. The task is to push the blocks into either same-or opposite-colored targets. The dataset has 1 000 demonstration trajectories. The observation is RGB images of size (224, 224) from two fixed viewpoints. We evaluate for 100 rollouts and report the mean number of blocks in targets (maximum 2).(c) Push-T [3]: The environment consists of a pusher with 2-dimensional action space, a T-shaped rigid block, and a target area in green. The task is to push the block to cover the target area. The dataset has 206 demonstration trajectories. The observation space is a top-down view of the environment, rendered as RGB images of size (224, 224). We evaluate for 100 rollouts and report the final coverage of the target area (maximum 1).(d) LIBERO Goal [29] : The environment consists of 10 manipulation tasks with a 7dimensional action space simulated Franka arm and gripper. The dataset has 500 demonstration trajectories in total, 50 per task goal. The observation space is RGB images of size (224, 224) from a fixed external camera, and a wrist-mounted camera. We evaluate a goal-conditioned policy for 100 rollouts in total, 10 per task goal, and report the average success rate (maximum 1).(e) Allegro Manipulation: A real-world environment with an Allegro Hand attached to a Franka arm. We evaluate on three tasks: picking up a sponge (6 demonstrations), picking up a teabag (7 demonstrations), and opening a microwave (6 demonstrations). The observation space is RGB images of size (224, 224) from a fixed external camera. The action space is 23-dimensional, consisting of the Franka pose (7) , and Allegro hand joint positions ( 16).(f) xArm Kitchen: A real-world multi-task kitchen environment with an xArm robot arm and gripper. The environment consists of five manipulation tasks. The dataset includes 65 demonstrations across five tasks. The observation space is RGB images of size (128, 128) from three fixed external cameras, and an egocentric camera attached to the gripper. The action space is 7-dimensional with the robot end effector pose and the gripper state.We evaluate each representation by training an imitation policy head on the frozen embeddings, and reporting the downstream task performance on the simulated environments. We use Vector-Quantized Behavior Transformer (VQ-BeT) [1] for the policy head. For xArm Kitchen, we use a goal-conditioned BAKU [38] with a VQ-BeT action head. MAE-style baselines (VC-1, MVP, MAE) use a ViT-B backbone. All other baselines and DynaMo use a ResNet18 backbone.For environments with multiple views, we concatenate the embeddings from all views for the downstream policy. Further training details are in Appendix B. Table 1 provides comparisons of DynaMo pretrained representations with other self-supervised learning methods, and pretrained weights for vision and robotic manipulation:• Random, ImageNet, R3M: ResNet18 with random, ImageNet-1K, and R3M [9] weights.• VC-1: Pretrained weights from Majumdar et al. [11] .• MVP: Pretrained weights from Xiao et al. [8] .• BYOL: BYOL [16] pretraining on demonstration data.• BYOL-T: BYOL + temporal contrast [32] . Adjacent frames o t , o t+1 are sampled as positive pairs, in addition to augmentations.• MoCo-v3: MoCo [33] pretraining on demonstration data.• RPT: RPT [39] trained on observation tokens.• TCN: Time-contrastive network [31] pretraining on demonstrations. MV: multi-view objective; SV: single view objective.• MAE: Masked autoencoder [30] pretraining on demonstrations.• DynaMo: DynaMo pretraining on demonstrations.The best pretrained representation is underlined and the best self-supervised representation is bolded. We find that our method matches prior state-of-the-art visual representations on Franka Kitchen, and outperforms all other visual representations on Block Pushing, Push-T, and LIBERO Goal. We evaluate the representations pre-trained with DynaMo on two real-world robot environments: the Allegro Manipulation environment, and the multi-task xArm Kitchen environment. For the Allegro environment, we use a k-nearest neighbors policy [40] and initialize with ImageNet-1K features for all pretraining methods, as the dataset is relatively small with around 1 000 frames per task. In the xArm Kitchen environment, we use the BAKU [38] architecture for goal-conditioned rollouts across five tasks. For our real-robot evaluations, we compare DynaMo against the strongest performing baselines from our simulated experiments (see Table 1 ). The results are reported in Table 2 . We observe that DynaMo outperforms the best baseline by 43% on the single-task Allegro hand and by 20% on the multi-task xArm Kitchen environment. Additionally, as shown in Table 3 , DynaMo exceeds the performance of pretrained representations by 50% on the Allegro hand. These results demonstrate that DynaMo is capable of learning effective robot representations in both single-task and multi-task settings.On the Push-T environment [3] , we compare all pretrained representations across four policy classes: VQ-BeT [1] , Diffusion Policy [3] , MLP (with action chunking [2] ), and k-nearest neighbors with locally weighted regression [40] . We present the results in Table 4 . We find that DynaMo representa- tions improve downstream policy performance across policy classes compared to prior state-of-the-art representations. We also note that our representation works on the robot hand in §4.3 with a nearest neighbor policy.4.5 Can pretrained weights be fine-tuned in domain with DynaMo? We fine-tune an ImageNet-1K-pretrained ResNet18 with DynaMo for each simulated environment, and evaluate with downstream policy performance on the frozen representation as described in §4.2. The results are shown in Table 5 . We find that Dy-naMo is compatible with ImageNet initialization, and can be used to fine-tune out-of-domain pretrained weights to further improve in-domain task performance. We also note that our method works in the low-data regime with ImageNet initialization on the real Allegro hand in Table 2 .In Table 6 , we ablate each component in DynaMo and measure its impact on downstream policy performance on our simulated benchmarks.We replace the one-step forward prediction target s * 1:h with the same-step target s * :h-1 . To prevent the model from trivially predicting s * t given s t , we replace the forward dynamics input (s :h-1 , z :h-1 ) with only z :h-1 . The ablated objective is essentially a variant of autoencoding s t . We observe that removing forward dynamics prediction degrades performance across environments. Inverse dynamics to a transition latent: As described in §3.1, the forward dynamics loss assumes that the transition is unimodal and requires an inferred transition latent. We observed that removing the latent from the forward dynamics input results in a significant performance drop.Bottleneck on the transition latent dimension: For the transition latent z and the observation embedding s, we find that having dim z ≪ dim s stabilizes training. Here we set dim z := dim s, and find that our model can still learn a reasonable representation in some environments, but training can destabilize, leading to a high variance in downstream performance.Covariance regularization: We find that covariance regularization from Bardes et al. [36] improves performance across environments. Training still converges without it, but the downstream performance is slightly worse.We observe that removing techniques like momentum encoder [33, 16] and stop gradient [37] leads to representation collapse [41, 16, 36] .Observation context: The dynamics objective requires at least 2 frames of observation context. For Franka Kitchen, we find that a context of 2 frames works best. For the other environments, a longer observation context (5 frames) improves downstream policy performance. Details of hyperparameters used for DynaMo visual pretraining can be found in Appendix B.1.In Table 7 , we compare with two variants of DynaMo where we assume access to ground truth action labels during visual encoder training.Only inverse dynamics to ground truth actions: as proposed in Brandfonbrener et al. [26] , we train the visual encoder by learning an inverse dynamics model to ground truth actions, with covariance regularization, and without forward dynamics.Full model + inverse dynamics to ground truth actions: we train the full DynaMo model plus an MLP head to predict the ground truth actions given the transition latents inferred by the inverse dynamics model.We observe that in both cases, having access to ground truth actions during visual pretraining does not seem to improve downstream policy performance. We hypothesize that this is because the downstream policy already has access to the same actions for imitation learning. 
related works: 
 This work builds on a large body of research on self-supervised visual representations, learning from human demonstrations, neuroscientific basis for learning dynamics for control, predictive models for decision making, learning from videos for control, and visual pretraining for control.Self-supervised visual representations: Self-supervised visual representations have been widely studied since the inception of deep learning. There are several common approaches to self-supervised visual representation learning. One approach is to recover the ground truth from noise-degraded samples using techniques like denoising autoencoders [42, 43] and masked modeling [44, 45, 30] . Another approach is contrastive learning, which leverages data augmentation priors [41, 16, 33, 36, 37] or temporal proximity [31, 46] to produce contrastive sample pairs. A third self-supervised method is generative modeling [47] [48] [49] , which learns to sequentially generate the ground truth data. More recently, self-supervision in the latent space rather than the raw pixel space has proven effective, as seen in methods that predict representations in latent space [50, 51] .Learning from demonstrations: Learning from human demonstrations is a well-established idea in robotics [52] [53] [54] [55] . With the advances in deep learning, recent works such as [3, 2, 5, 4, 1, 56] show that imitation learning from human demonstrations has become a viable approach for training robotic policies in simulated and real-world settings.Neural basis for learning dynamics: It is widely believed that animals possess internal dynamics models that facilitate motor control. These models learn representations that are predictive of sensory inputs for decision making and motor control [57] [58] [59] [60] . Early works such as [17] [18] [19] [20] propose that there exists an internal model of the motor apparatus in the cerebellum for motor control and planning. [21, 22] propose that the central nervous system uses forward models that predict motor command outcomes and model the environment. Learning forward and inverse dynamics models also helps with generalization to diverse task conditions [23, 24] .Predictive models for decision making: Predictive model learning for decision making is wellestablished in machine learning. Learning generative models that can predict sequential inputs has achieved success across many domains, such as natural language processing [61] , reinforcement learning [62] [63] [64] , and representation learning [46, 65] . Incorporating the prediction of future states as an intrinsic reward has also been shown to improve reinforcement learning performance [66] [67] [68] . Moreover, recent work demonstrates that world models trained to predict environment dynamics can enable planning in complex tasks and environments [69] [70] [71] [72] [73] .Learning from video for control: Videos provide rich spatiotemporal information that can be leveraged for self-supervised representation learning [74] [75] [76] [77] [78] [79] . These methods have been extended to decision-making through effective downstream policy learning [7] [8] [9] [10] [11] 6] . Further, recent work also enables learning robotic policies directly from in-domain human demonstration videos by incorporating some additional priors [80] [81] [82] [83] [84] , as well as learning behavioral priors from actionless demonstration data [85] [86] [87] .Visual representation for control: Visual representation learning for control has been an active area of research. Prior work has shown that data augmentation improves the robustness of learned representations and policy performance in reinforcement learning domains [88, 89] . Additionally, pretraining visual representations on large out-of-domain datasets before fine-tuning for control tasks has been shown to outperform training policies from scratch [10, 12, 9, 11, 90, 8, 91] . More recent work has shown that in-domain self-supervised pretraining improves policy performance [92] [93] [94] [95] and enables non-parametric downstream policies [40] . 
discussion and limitations: 
 In this work, we have presented DynaMo, a self-supervised algorithm for robot representation learning that leverages the sequential nature of demonstration data. DynaMo incorporates predictive dynamics modeling to learn visual features that capture the sequential structure of demonstration observations. During pretraining, DynaMo jointly optimizes the visual encoder with dynamics models to extract task-specific features. These learned representations can then be used for downstream control tasks, leading to more efficient policy learning compared to prior approaches. We believe that training DynaMo on larger unlabeled datasets could potentially improve generalization. Additionally, while promising for control tasks, more research is needed to evaluate DynaMo's effectiveness on robotic manipulation outside of lab settings.The Franka Kitchen environment introdued by Gupta et al. [27] consists of a Franka arm with a 9-dimensional action space. This environment includes seven tasks and a dataset of 566 humancollected demonstrations. While the original environment is state-based, we created an image-based variant by rendering the states to 224 × 224 RGB images.In the Block Pushing environment introduced by Florence et al. [28] , the objective is for the robot to push two colored blocks (red and green) into two target squares (also red and green). The training dataset consists of 1 000 trajectories, evenly distributed among the four possible combinations of block target and push order. These trajectories were collected by a scripted expert controller.In the Push-T environment introduced by Chi et al. [3] , the goal is to push a T-shaped block to a designated target position on a table. The dataset for this environment contains 206 demonstrations collected by human operators. The action space in this environment is a two-dimensional end-effector position control. Similar to the Franka Kitchen environment, we have created an image-based variant by rendering demonstrations to 224 × 224 RGB images.In the LIBERO Goal environment introduced by Liu et al. [29] , there are 10 manipulation tasks, each with 50 teleoperated demonstrations for goal-conditioned policy benchmarking. The environment has a 7-dimensional action space and an observation space of 224 × 224 RGB images from two cameras (fixed external view, and wrist-mounted egocentric view).The environment consists of an Allegro hand attached to a Franka arm, and a fixed camera for image observations. The observation space is 224 × 224 RGB images. The action space is 23-dimensional, consisting of Cartesian position and orientation of the Franka robot arm (7 DoF), and 16 joint positions of the Allegro Robot Hand. The demonstrations are collected at 50Hz for Franka, and 60Hz for the Allegro hand. The learned policies are rolled out at 4Hz.We evaluate on three contact-rich dexterous manipulation tasks that require precise multi-finger control and arm movement, described in detail below.Sponge picking: This task requires the hand to reach to the position of the sponge, grasp the sponge, and lift the sponge from the table. We collect 6 demonstrations via OpenTeach [96] for the task, starting from different positions, with 543 frames in total. The task is considered successful if the robot hand can grasp the sponge from the table within 120 seconds.Teabag picking: This task is similar to the previous task, but more difficult with a smaller task object. We collect 7 demonstrations via OpenTeach with 1 034 frames in total. In this task, the robot needs reach the teabag, grasp the teabag with two fingers, then pick it up. The task is considered successful if the robot hand can grasp the teabag from the table within 240 seconds.Microwave opening: This task requires the hand to reach the microwave door handle, grasp the handle, and pull down the door. We collect 6 demonstrations via OpenTeach with 735 frames in total.The task is considered successful if the robot hand can open the door within 240 seconds.This is a real-world multi-task kitchen environment comprising a Ufactory xArm 7 robot with an xArm Gripper. The policies are trained on RGB images of size 128 × 128 obtained from four different camera views, including an egocentric camera attached to the robot gripper. The action space comprises the robot end effector pose and the gripper state. We collect a total of 65 demonstrations across 5 tasks, depicted in Figure 5 . The demonstrations were collected using OpenTeach [96] at 30Hz. The learned policies are deployed at 10Hz. Figure 5 shows real-world task rollouts for the multitask policy learned for all 5 tasks.Put ketchup bottle inside fridge: Pick up the bottle of tomato ketchup and put it inside the fridge.Fetch tea bottle from fridge door: Take the bottle of green tea out from the door of the fridge.Fetch water bottle from fridge: Take the bottle of vitamin water out of the fridge.Put yogurt bottle in fridge door: Pick up the bottle of yogurt and place it in the door of the fridge.Fetch yogurt bottle from fridge door: Take the bottle of yogurt out from the door of the fridge. We present the DynaMo hyperparameters below. In this work, we present DynaMo, a new self-supervised learning method for pretraining in-domain visual representations for downstream policy learning. Our work takes an important step towards improves data efficiency by explicitly modeling the dynamics of the demonstration observations, improving upon prior state-of-the-art self-supervised methods, especially in the low-data regime, which can be valuable for robotics and visuomotor policy learning research with limited in-domain data.Answer: [Yes] Justification: we will release code and data on the website for reproduction. Guidelines: Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: this work poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: we credit all authors in our paper and respect license and terms of use in our codebase.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
 
 Appendix:  


Decision: DynaMo is a self-supervised learning approach for robotics that jointly learns inverse and forward dynamics models using in-domain data. This approach avoids suppressing important data dimensions for visuomotor control. The model assumes unobserved action representations, which is novel. The paper could benefit from a better discussion of prior art and context of this technical advance 
Empirical The paper presents its application in both simulated and real-world robot experiments. 
While the approach shows promise, the review highlights concerns about the experimental setup, including simplified real-world scenarios and potentially unfair comparisons in simulations. Furthermore other reviewers requested well-tuned baselines and exploring full fine-tuning setups, as well as additional experiments with more complex scenarios and varied backbones to better demonstrate DynaMo's effectiveness and provide a fairer evaluation.

However, the rebuttal did a good job of providing clarifications with more than one reviewers updating the scores leaning more positively.  All reviewers lean accept after rebuttal phase, and the Meta-reviewer concurs on the overall agreement. 
The authors are advised to review the final comments, and update the manuscript accordingly.. In addition to including clarifications in the main paper, the authors are also advised to describe the possible limitations of this work in more detail.

Reviews: review: paper_summary: paper_summary: This paper presents DynaMo, using in-domain data for self-supervision. It jointly learns a latent inverse dynamics model and a forward dynamics model over a sequence of image embeddings. The
strength_weakness: strength_weakness: Simplified Real-World Setup:
The real-robot experiments appear overly simplistic. Objects seem to be fixed in place, indicated by the red marker on the table, suggesting a lack of randomization in object placement. This setup makes the task easier for conventional imitation learning methods like diffusion policy and act, potentially allowing them to achieve a 100% success rate.

Suggestion: Introduce spatial randomization to the scene. Conduct additional experiments under these conditions to demonstrate Dynamo's superiority in more complex and varied scenarios.

2) Unfair Comparisons in Simulation: 
In Table 1, Dynamo is compared with several baselines that use different backbones, which makes the comparison potentially unfair.
Impact: The difference in backbones could skew the performance results, making it difficult to accurately assess Dynamo's relative performance. 

Suggestion: Include more experiments of Dynamo with various backbones such as ViT and ResNet-50. Compare these results against the baselines to provide a fairer and more comprehensive evaluation.

3) The motivation for using SSL in this context is unclear. Typically, SSL is advantageous due to its ability to learn from massive datasets without human labels. However, in the field of robotics, in-domain data are often scarce. This could make the application of SSL less persuasive and potentially less effective.
questions: questions: Currently, I believe this paper has significant flaws in its experimental design, both in simulation and real-robot settings. As such, my initial score is 4, with the real-robot experiments being a notable strength. However, the existing experiments do not sufficiently support the claims made in the paper. If the authors can provide additional experiments based on my suggestions above, and if the results substantiate their claims, I would be willing to raise my rating.
limitations: limitations: No.
paper_summary: paper_summary: This paper presents a self-supervised learning method for robot learning that learns representations by using data from demonstrations. The objective is based on learning latent actions from inverse dynamics, and learning forward dynamics model that uses such latent actions as inputs. Several techniques are utilized to prevent the model from finding trivial solutions and thus collapsing. Experiments are conducted in both real-world and simulation environments.
strength_weakness: strength_weakness: - As other self-supervised learning models are trained on non-standard robotic datasets, it is not clear whether they are trained well with good hyperparameters -- for instance without collapses -- is there a way to ensure that baseline methods are well-tuned?
- I understand that the main focus of this paper is to introduce a self-supervised learning method and compare its performance to other baselines. But what would the performance look like if you consider the full fine-tuning setup that uses gradients from behavior cloning for updating the encoder? Can we squeeze more information and maybe performance boost from fully fine-tuning the encoder? How would all the methods perform in this setup? This could further strengthen the claims of this paper that we should focus on extracting more information from demonstrations.
- One important missing baseline is [1] that pre-trains (optionally causal) transformer with masked modelling objective. Even though it uses a pre-trained visual encoder, using features from the causal transformer can be still a baseline Moreover, it's a bit awkward that MAE trained on demonstrations is missing from the baseline even though MVP is selected as a pre-trained representation baseline. Including MAE, maybe optionally its multi-view variant [2], can make results be more convincing.

[1] Radosavovic, Ilija, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. "Robot learning with sensorimotor pre-training." In Conference on Robot Learning, pp. 683-693. PMLR, 2023.

[2] Seo, Younggyo, Junsu Kim, Stephen James, Kimin Lee, Jinwoo Shin, and Pieter Abbeel. "Multi-view masked world models for visual robotic manipulation." In International Conference on Machine Learning, pp. 30613-30632. PMLR, 2023.
questions: questions: See Weaknesses
limitations: limitations: N/A
paper_summary: paper_summary: This paper presents a self-supervised model, DynaMo, for pretraining visual encoders adopted for visuo-motor control. The targeted downstream task is imitation learning for robotic manipulation. Instead of using an out-of-domain dataset for pretraining and then transferring to a new domain using alternative techniques, the authors propose exploiting sequences of observations from in-domain demonstrations to pretrain three different models, a visual encoder, a forward and an inverse model. Once this is done a policy can be learned with observations encoded using the pre-trained visual encoder.
strength_weakness: strength_weakness: The paper is written as if there were no research in the area before the deep learning boom. Only one citation out of 70 citations is older than 10 years. The paper suggests that training exclusively on in-domain data is new, even if this used to be the way it was typically done before the arrival of data-hungry deep-learning-based models, models that forced people to a greater extent to rely on offline training on out-of-domain data with data augmentation, contrastive learning, etc. 

The idea to train pairs of inverse and forward models online has existed in psychology and robotics for at least 25 years, such as in the works of Wolpert et al [1]. Using similar models, imitation learning has been a common theme over the years, with [2] being just an example. Without this connection back to earlier research, this paper gives the impression of trying to reinvent the wheel, and it becomes unclear what the contributions really are. 

Even if the experiments suggest that DynaMo can be beneficial also in real-world settings, the presented experiments are too few to be conclusive. The real world is way more diverse with more than just a small selected set of objects that can be manipulated. However, this weakness is pointed out in the conclusions, which makes it less problematic.  

[1] Wolpert and Kawato, “Multiple paired forward and inverse models for motor control”, Neural Networks, 11, 1998.

[2] Demiris and Hayes, “Imitation as a dual-route process featuring predictive and learning components: a biologically plausible computational model”, in Imitation in Animals and Artifacts, MIT Press, 2002.
questions: questions: * Are the visual encoder, inverse and forward models only trained on the demonstrations from the respective datasets? Even if this is only for pretraining, the demonstrations, at least the real-world ones, are very few compared to the complexity of the tasks learned. Why not exploit all possible sequences available on the same embodiment, even for tasks that will eventually not be of interest? 
* How restrictive is the assumption that forward models are unimodal? Has this become a weakness during the experiments?
* Since both the inverse and forward models seem to be ignored after pretraining, what is the motivation for a separation between the two? Why not train a network to predict the next encoded observation from earlier ones, essentially with the inverse and forward models merged into one? Why is the latent action representation needed at all?
limitations: limitations: Yes, some limitations related the real-world experiments and the unimodality of models are brought up in the conclusions.
paper_summary: paper_summary: This paper presents a way of pre-training vision encoder for robot control. Specifically, instead of using vanilla contrastive or masked autoencoder approaches, this method creates two models: 1) an inverse dynamics model that estimates the transition latent (actions) and 2) a forward dynamics model that takes in the current encoded visual latent and the transition latent and predicts the next latent observation. The results suggest that the method improves upon existing visual pre-training methods for robotics.
strength_weakness: strength_weakness: 1. Prior visual pre-training for robotics operates under the premise that we have an image or video dataset, where we pre-train on these datasets and then finetune for a particular task. However, this method performs pre-training on the task-specific dataset, which is better aligned with the downstream tasks. Instead of having pre-training and fine-tuning using the same dataset and solving the same task, the objective of visual pretraining (exemplified by MAE, MoCo, etc.) is that if we train on a mass amount of data, we can finetune to a specific task (i.e. ImageNet pre-training then COCO segmentation finetuning). 
2. A few prior works [1,2] have tried to model forward and inverse dynamics concurrently. [1] also uses forward and inverse dynamics to train a visual encoder. The key difference between these works is that here action is modeled as a latent variable. Why ground truth action values are not used in pre-training (especially when pre-training and fine-tuning happen on the same task) is not justified in the manuscript. It would be quite convincing if pre-training is done on natural videos, or large-scale robot datasets where action spaces cannot be standardized, and then shows improved finetuning performance. 

[1] Agrawal, Pulkit, Ashvin V. Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. "Learning to poke by poking: Experiential learning of intuitive physics." Advances in neural information processing systems 29 (2016).

[2] Fragkiadaki, Katerina, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. "Learning visual predictive models of physics for playing billiards." arXiv preprint arXiv:1511.07404 (2015).
questions: questions: The reviewer wants to ask for two sets of experiments to address weakness 1:
1. How do masked pre-training methods compare to DynaMo when they are trained on the same data? I.e. train two networks analogously with the method provided in VC-1 and MVP on the task-specific dataset, and evaluate their task performance. This experiment would demonstrate that even for specific tasks, pre-training with DynaMo outperforms existing visual pre-training methods on task-specific datasets. 
2. How does DynaMo generalize to unseen tasks (in the sense that it can generalize to tasks outside of the dynamics seen in training)? I.e. pre-train DynaMo on (1) Put Yogurt (2) Get yogurt (3) Get Tea and evaluate on (1) Put ketchup (2) Get Water.
limitations: limitations: Limitation section is present.

Comments: title: Author response
Response to rebuttal
comment: This reviewer wants to thank the authors for an informative rebuttal and is looking forward to reading the paper with the promised changes included. Despite an unfortunate lack of references back to earlier research, there clearly is a connection that ought to be highlighted, since it partially explains why SSL makes sense at all for an embodied agent constantly adapting to an ever-changing environment.
Thank you for the response. With the added experiments, I decided to raise the score.
Thank you for the response. I have read other reviews and the response, and currently decided to maintain the score.

The reason for not increasing the score is that I agree with the other reviewers that the submission can be improved by (i) improving the experimental design especially with architecture and (ii) making the reasoning of not using actions when they are available more clearer. 

But I'd also like to note that I disagree with other reviewers in that SSL should be designed for large-scale datasets as representation learning on scare demonstration data can be very useful for robotics, which is why I'm not decreasing the score.
Thank you for your reply. We are confident that our rebuttals are by the rules (“should not contain any links to external pages”). To facilitate the discussion, we have updated the existing paper website for transparency and in good faith. If you would like to disregard that anyway, we invite you to look at **Figure 1 in the uploaded PDF** as an alternative visualization for your point above, as well as **the additional kitchen experiment in global comment (5)** with starting position variations explicitly addressing this concern.

As for your second point, we believe there is a misunderstanding. Allow us to elaborate below.

**Whether ViT performs worse than small ResNets depends on the experimental conditions**: We would like to kindly note that in the low-data regime, larger models ≠ better performance: the original ViT paper [1] has observed that ViTs require larger datasets (“14M-300M images”) to reach performance parity, whereas for robotics and decision-making, most datasets have 10k-100k frames. It is not surprising that ViTs underperform ResNets at this scale. The main focus of this paper is an SSL method that works in the low-data regime for control tasks, rather than a pretrain-then-fine-tune setup for visual encoders in general.

**Current experiments are insufficient and not convincing**: Could you give us specific setups in existing published work that you think we should follow? We are happy to take a look.

**The experimental setup should be clearly elaborated and compared**: To enable us to address your concerns effectively, would you clarify which parts of the experimental setup are unclear? For all our main results, we follow the same evaluation procedure: pretrain an encoder from random initialization, then train a downstream policy on the frozen embeddings and use environment rollout metrics to evaluate the encoder performance. The encoder is not fine-tuned during policy training. This setup is used for the ViT training as well.

[1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Thank you for your reply.

1. I appreciate the explanation and checked the new video uploaded by the author. However, I am unsure if updating the webpage violates the rebuttal rules. Therefore, even though this video addresses my concern, my evaluation will not take it into account.

2. Whether ViT performs worse than small ResNets depends on the experimental conditions (e.g., freeze, LoRA, or full parameter fine-tuning) and hyper-parameter settings (e.g., same or lower initial learning rate compared to the pretrain stage). The experimental setup should be clearly elaborated and compared. Current experiments are insufficient and not convincing. 

Therefore, I have decided to keep my initial rating after the rebuttal.