DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control
Authors: Zichen Jeff Cui
Hengkai Pan
Aadhithya Iyer
Siddhant Haldar
Lerrel Pinto
Venue: NeurIPS 2024 poster (2024)
Keywords: Robot learning
representation learning
self-supervised learning
Primary area: robotics
TL;DR: DynaMo, a new self-supervised method for pretraining visual encoders for downstream visuomotor control by explicitly modeling dynamics in the demonstration observations.

Abstract: Imitation learning has proven to be a powerful tool for training complex visuo-motor policies. However, current methods often require hundreds to thousands of expert demonstrations to handle high-dimensional visual observations. A key reason for this poor data efficiency is that visual representations are predominantly either pretrained on out-of-domain data or trained directly through a behavior cloning objective. In this work, we present DynaMo, a new in-domain, self-supervised method for learning visual representations. Given a set of expert demonstrations, we jointly learn a latent inverse dynamics model and a forward dynamics model over a sequence of image embeddings, predicting the next frame in latent space, without augmentations, contrastive sampling, or access to ground truth actions. Importantly, DynaMo does not require any out-of-domain data such as Internet datasets or cross-embodied datasets. On a suite of six simulated and real environments, we show that representations learned with DynaMo significantly improve downstream imitation learning performance over prior self-supervised learning objectives, and pretrained representations. Gains from using DynaMo hold across policy classes such as Behavior Transformer, Diffusion Policy, MLP, and nearest neighbors. Finally, we ablate over key components of DynaMo and measure its impact on downstream policy performance. Robot videos are best viewed at https://dynamo-ssl.github.io.

DynaMo: In-Domain Dynamies Pretraining for Visuo-Motor Control
Zichen Jeff Cui"
Hengkai Pan
Aadhithya Iyer
Siddhant Haldar
Lerrel Pinto
New York University
Abstract
Imitation learning has proven to be powerful tool for training complex visuo- motor policies. However, current methods often require hundreds to thousands of expert demonstrations handle high-dimensional visual observations. key reason for this poor data efficiency is that visual representations are predominantly either pretrained on out-of-domain data or trained directly through behavior cloning objective. In this work, we present DynaMo, new in-domain, self- supervised method for learning visual representations. Given a set of expert demonstrations, we jointly learn a latent inverse dynamics model and forward dynamics model over a sequence image embeddings, predicting the next frame in latent space, without augmentations, contrastive sampling, access ground truth actions. Importantly, DynaMo does not require any out-of-domain data such Internet datasets cross-embodied datasets. On suite of six simulated and real environments, we show that representations learned with DynaMo significantly improve downstream imitation learning performance over prior self-supervised learning objectives, and pretrained representations. Gains from using DynaMo hold across policy classes such Behavior Transformer, Diffusion Policy, MIP, and nearest neighbors. Finally, we ablate over key components of DynaMo and measure its impact on downstream policy performance. Robot videos are best viewed at https: haaps://dynng /dynamo-- ssl. wet.ggittuub... io.
1 Introduction
Learning visuo-motor policies from human demonstrations is exciting approach for training difficult control tasks in the real world [1-5]. However, a key challenge in such learning paradigm to efficiently learn a policy with fewer expert demonstrations. To address this, prior works have focused on learning better visual representations, often by pretraining on large Internet-scale video datasets [6-11]. 1]. However, shown in Dasari al. [12], these out-of-domain representations may transfer to downstream tasks with very different embodiments and viewpoints from the pretraining dataset.
An alternative to using Internet-pretrained models is to train the visual representations 'in-domain' on the demonstration data collected to solve the task [13, 4]. However, in-domain datasets much smaller than Internet-scale data. This has resulted the use of domain-specific augmentations [13] to induce representational invariances with self- supervvpevn strations [2, 14]. The reliance of existing methods on large datasets might suggest that in-domain self superviedd pretraining is ineffective for visuo-motor control, and might be better with simply training end-to-end. In this work, we argue the contrary in-domain self-supervision can be effective with better training objective that extracts more information from small datasets.
collect larger amounts of demon-
"Corresponding author. Email: jeff. cui mniinyu.edd edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Observations
Embedd ings
Env Environe ronments
$
Encoder
Inverse
Policy
sg
0t1
DynaMo
5/4
Oseb
Representation learning
(b) Policy pretrained representations
the battle of the
Put yogurt bottle fridge door
Put the ketchup battle inside the Fridge
Lift the teabag off the table
Real-world rollouts
Figure (a) We present DynaMo, new self-supervised method for learning visual representations for visuomotor control. DynaMo exploits the causal structure in demonstrations by jointly learning the encoder with inverse and forward dynamics models. DynaMo requires no augmentations, contrastive sampling, or access to ground truth actions. This enables downstream policy learning using limited in-domain data across simulated and real-world rohotics tasks. For each environment, pretrain the visual representation in-domain with DynaMo learn a policy on the pretrained embeddings. (b) provide real- world rollouts policies learned with DynaMo representation on our multi-task xArm Kitchen and Allegro Manipulation environments.
Prevalent approaches for using self-supervision in downstream control often make bag-of-framee assumption, using contrastive methods [15, 16] masked autoencoding [11, 8] on individual frames for self-supervision. Most of these approaches ignore a rich supervision signal: action- causality. dependent past observations, and unobserved latent actions. Can
Future observations obtain a good visual representation for control by simply learning the dynamics? In fact, this idea is well-established in neuroscience: animals are thought possess internal models of the motor apparatus and the environment that facilitate motor control and planning [17-24].
self- supervised method for pretraining visual representations for visuomotor control from limited in- domain data. DynaMo jointly learns the encoder with inverse and forward dynamics models, without access to ground truth actions [25, 26].
In this work, present Dynamics Pretraining for Visuo-Motor Control (DynaMo),
To demonstrate the effectiveness of DynaMo, we evaluate our representation on four simulation suites Franka Kitchen [27], Block Pushing [28], Push-T [3], and LIBERO [29], as well eight robotic manipulation tasks on two weal-world environments. Our main findings are summarized below:
1. DynaMo exhibits an overall 39% improvement in downstream policy performance over prior state-of-the oft------t--tttt pretrained and self-supervised representations, especially on the harder closed-loop control tasks in Block Pushing and Push-T (Table 1), and real robot experiments (Table 2).
DynaMo is compatible with various policy classes, can be used fine-tune pretrained weights, and works in the low-data regime with limited demonstrations on real-world Allegro hand (Tables 5, and 2 respectively).
Through an ablation analysis, we study the impact of each component in DynaMo on downstream policy performance ($4.6).

---

All of our datasets, and training and evaluation code will be made publicly available. Videos of our trained policies can be seen here: https: http::/ddynaa http://dyymmoaaaa ssl. wwt.esssss........ee.eeee..eeaae github. io.
2 Background
2.1 Visual imitation learning
Our work follows the general framework for visual imitation learning. Given demonstration data D {(o,, Jt. where o, are raw visual observations and a₂ are the corresponding ground-truth actions, first employ visual encoder fo or -> to map the raw visual inputs to lower- dimensional embeddings 8t- We then learn policy π (acess) to predict the appropriate actions. For rollouts, we model the environment
Markov Decision Process (MDP), where each subsequent observation 0++11 depends on the previous observation-action pair (0r, az). We assume the action- conditioned transition distribution p(0 (r, de) to be unimodal for manipulation tasks.
2.2 Visual pretraining for policy learning
Our goal is to pretrain the visual encoder fe using a dataset of sequential raw visual observations D = {or] to support downstream policy learning. During pretraining, we do not assume access to the ground-truth actions (acht-
Prior work has shown that pretraining encoders on large out-of-domain datasets can improve down- stream policy performance [6-11]. However, such pretraining may not transfer well to tasks with different robot embodiments [12].
Alternatively, we can directly pretrain the encoder in-domain using self-supervised methods. One approach is contrastive learning with data augmentation priors, randomly augmenting an image twice and pushing their embeddings closer. Another approach is denoising methods, predicting the original image from a noise-degraded sample (e.g. by masking [11,8 8, 30)). third approach is contrastive learning with temporal proximity as supervision, pushing temporally close frames to have similar embeddings [31, 32].
3 DynaMo
Limitations of prior self-supervised techniques: Prior self-supervised techniques can learn to fixate on visually salient features and ignore fine-grained features important for control. We illustrate this limitation using the Block Pushing environment from Florence et al. [28]. In this task, the goal is to push block into target square. While the robot arm occupies much of the raw pixel space, the blocks are central to the task despite being smaller in the visual field. Figure 2 visualizes a random frame from the demonstration data and its 20 nearest neighbors in the embedding space learned by several self-supervised techniques.
We observe that prior self-supe supervieee methods (details in $4.2) focus on the visually dominant robot, matching the whole robot arm extremely accurately. However, they fail to capture the block positions, which are important to the task despite being much less salient visually.
Can learn visual encoder that extracts task-specific features better? We know that the demon- strations are sequential: each observation is dependent on the previous observation, and an action (unobserved in this setting). Prior self-supervised methods ignore this sequential structure. Con- trastive augmentations [16, 33] and autoencoding objectives [30, 8, 11] assume that the demonstration video bag of frames, discarding temporal information altogether. Temporal contrast [32, 31] uses temporal proximity but discards the sequential information in the observations: the contrastive objectives are usually symmetric time, disregarding past/future order.
Instead of contrastive or denoising objective, we propose a dynamics prediction objective that explicitly exploits the sequential structure of demonstration observations.
The key insight of our method is that we can learn good visual repre- sentation for control by modeling the dynamics on demonstration observations, without requiring augmentations, contrastive sampling, access to the ground truth actions. Given sequence of raw
Overview of DynaMo:

---

bloce
block
Neuraa neighters
Figure 2: Embedding nearest neighhor matches for DynaMo, BYOL, MoCo, and TCN the Block Pushing environment. (Top) The nearest neighhor matches visualized in pixel space. (Bottom) Matches visualized in top-down view. We see that DynaMo representation captures task-relevant features (end effector, block, and target locations in this case), whereas prior work fixates on the large robot arm.
concat
CORCaT
soncat
concat
Osservation eabeddings ings
observations
A
Appa
Transition Latents
Encoder (nndiviidal
dynanics (causally
Observation exbedcings
Ayei
Inverse dynanics (causality
predictions
Target enbeddiing
Transition latents
Teel
Apok-1
dynamics prediction
Figure 3: Architecture of DynaMo. DynaMo jointly learns image encoder, an inverse dynamics model, and forward dynamics model with forward dynamics prediction loss.
visual observations (o.....,.7) we jointly train the encoder fo - st, a latent inverse dynamics model g ( and forward dynamics model p( 81+1 [8 21:+h-1). We model the actions as unobserved latents, and train all models end-to-end with a consistency loss on the forward dynamics prediction. For our experiments, we use a ResNet18 [34] encoder, and causally masked transformer encoders [35] for the inverse and forward dynamics models. The architecture is illustrated Figure
3.1 Dynamics as visual self-supervised learning objective
First, sample observation sequence 0tr4h of length h and compute its representation St.thh fo(o:tt)). ). For convenience, we will write St.th as 8.A. and 814144h as s1sh below. At any given step, the distribution of possible actions can be multimodal [5]. Therefore, the forward dynamics transition ([s1:n|s:h-)) can also have multiple modes. To address this, we first model the inverse dynamics q(2.h11 ). where is the latent transition between frames. We assume to be well- determined and unimodal given consecutive frames (st, St+1), ). We have E Rm Rd & d such that the latent cannot trivially memorize the next frame embedding. Finally, we concatenate (St, 21) and predict the one-step forward dynamics p($1.h - 2.)

---

We compute a dynamics loss ( s3) on the one-step forward predictions $1+1:2+h, where s1+1++ are the target next-frame embeddings; and a covariance regularization loss Loo from Bardes et al. [36] on minibatch of observation embeddings S:
(S,, 87)
Ldyn (St,, 56 +{)== =1 | 12 || ll2
Leov (5) 1 1 [Cov (ov(
(I)
C= Ldyu + XLeov
For environments with multiple views, we compute a loss over each view separately and take the mean. We choose À = 0.04 following Bardes al. [36] for the total loss C. We find that covariance regularization slightly improves downstream task performance.
Naively, this objective admits constant embedding solution. To prevent representation collapse, for (a s*) we follow SimSiam [37] and set the target embedding 8 == sg();), where sg is the stop gradient operator. Alternatively,
objective is also compatible with target from momentum encoder fa [33, 16]. of == & = fā(o)), where is an exponential moving average of
We train all three models end-to-end with the objective in Eq. 1, and use the encoder for downstream control tasks.
Experiments
We evaluate Synammcs-preerrineed visual representation suite of simulated and real bench- marks. We compare DynaMo representations with pretrained representations for vision and control, as well other self-supervised learning methods. Our experiments designed to answer the following questions: (a) Does DynaMo improve downstream policy performance? (b) Do representations trained with DynaMo work real robotic tasks? (c) Is DynaMo compatible with different policy classes? (d) Can pretrained weights be fine-tuned in domain with DynaMo? (e) How important is each component DynaMo?
4.1 Environments and datasets
We evaluate DynaMo on four simulated benchmarks and two real robot environments (depicted in Figure 4). We provide brief description below with more details included in Appendix A.
(a) Franka Kitchen [27]: The Franka Kitchen environment consists of seven simulated kitchen appliance manipulation tasks with a 9-dimensional action space Franka arm and gripper. The dataset has 566 demonstration trajectories, each completing three four tasks. The observation space is RGB images of size (224, 224) from a fixed viewpoint. We evaluate for 100 rollouts and report the mean number of completed tasks (maximum 4).
(b) Block Pushing [28]: The simulated Block Pushing environment has two blocks, two target areas, and robot pusher with 2-dimensional action space (end-effector translation). Both the blocks and targets are colored red and green. The task is to push the blocks into either same- or opposite-colore targets. The dataset has 1 0000 demonstration trajectories. The observation is RGB images of size (224, 224) from two fixed viewpoints. We evaluate for 100 rollouts and report the mean number blocks in targets (maximum 2).
(c) Push-T [3]: The environment consists a pusher with 2-dimensional action space, T-shaped rigid block, and target area in green. The task is to push the block to cover the target area. The dataset has 206 demonstration trajectories. The observation space is top-down view of the environment, rendered as RGB images of size (224, 224). We evaluate for 100 rollouts and report the final coverage of the target area (maximum 1).
(d) LIBERO Goal [29]: The environment consists of 10 manipulation tasks with a dimensional action space simulated Franka arm and gripper. The dataset has 500 demon- stration trajectories total, per task goal. The observation space RGB images of size (224, 224) from fixed external camera, and wrist-mounted camera. We evaluate
5

---

(4) Franica Kitchen
(b) Block Pushing
Push-T
LIBERO Gioa
Allegro Monipulation
XArm Kitchen
Figure 4: We evaluate DynaMo on four simulated benchmarks Franka Kitchen, Block Pushing, Push-T, and LIBERO Goal, and two real-world environments Allegro Manipulation, and xArm Kitchen.
goal-conditioned policy for 100 rollouts in total, 10 per task goal, and report the average success rate (maximum 1).
(e) Allegro Manipulation: A real-world environment with an Allegro Hand attached to Franka arm. We evaluate on three tasks: picking up sponge (6 demonstrations), picking up teabag (7 demonstrations), and opening a microwave (6 demonstrations). The observation space is RGB images of size (224, 224) from fixed external camera. The action space is 23-dimensional, consisting of the Franka pose (7), and Allegro hand joint positions (16).
(f) xArm Kitchen: A real-world multi-task kitchen environment with an xArı robot arm and gripper. The environment consists of five manipulation tasks. The dataset includes 65 demonstrations across five tasks. The observation space is RGB images of size (128, 128) from three fixed external cameras, and an egocentric camera attached to the gripper. The action space is 7-dimensional with the robot end effector pose and the gripper state.
4.2 Does DynaMo improve downstream policy performance?
We evaluate each representation by training an imitation policy head on the frozen embeddings, and reporting the downstream task performance on the simulated environments. We use Vector- Quantized Behavior Transformer (VQ-BeT) [1] for the policy head. For xArm Kitchen, we use a goal-conditioned BAKU [38] with VQ-BeT action head. MAE-style baselines (VC-1, MVP, MAE) use a VIT-B backbone. All other baselines and DynaMo use a ResNet18 backbone.
For environments with multiple views, we concatenate the embeddings from all views for the downstream policy. Further training details are in Appendix B. Table provides comparisons of DynaMo pretrained representations with other self-supervised learning methods, and pretrained weights for vision and robotic manipulation:
Random, ImageNet, R3M: ResNet18 with random, ImageNet-IK, and R3M [9] weights.
VC-1: Pretrained weights from Majumdar al. [11].
MVP: Pretrained weights from Xiao et [8].
BYOL: BYOL [16] pretraining on demonstration data.
BYOL-T: BYOL temporal contrast [32]. Adjacent frames ot, 0r41 sampled positive pairs, in addition to augmentations.
MoCo-v3: MoCo [33] pretraining demonstration data.
RPT: RPT [39] trained observation tokens.
TCN: Time-contrastive network [31] pretraining on demonstrations. MV: multi-view objective; SV: single view objective.
MAE: Masked autoencoder [30] pretraining on demonstrations.
DynaMo: DynaMo pretraining on demonstrations.
The best pretrained representation is underlined and the best self-supervised representation is bolded. We find that our method matches prior state-of-the-art visual representations on Franka Kitchen, and outperforms all other visual representations on Block Pushing, Push-T, and LIBERO Goal.

---

Table Downstream policy performance frozen visual representation four simulated benchmarks Franka Kitchen, Blocking Pushing, Push-T, and LIBERO Goal. observe that DynaMo matches or significantly outperforms prior work on all simulated tasks.
Franka Kitchen
Block Pushing
Push-T
LIBERO Goal
Method
(-/4)
(-/2)
(-/1)
(-/1)
Random
3.32
0.07
0.07
0.80
ImageNet
3.01
0.12
0.41
0.93
Pretrained
R3M
2.84
0.11
0.49
0.89
representations
VC-1
2.63
0.05
0.38
0.91
MVP
2.31
0.00
0.20
0.88
BYOL
3.75
0.09
0.23
0.28
BOL-TT
3.33
0.16
0.34
0.28
Self-supervised
MoCo-v3
3.28
0.03
0,57
0.70
methods
RPT
3.54
0.52
0.56
0.17
TCN-MV
0.07
0.69
TCN-SV
2.41
0.07
0.07
0.76
MAE
2.70
0.00
0.07
0.59
DynaMo 3.64
0.65
0.66
0.93
Table evaluate DynaMo on eight tasks across two real-world environments: Allegro Manipulation, and xArm Kitchen. Results are presented as (successes/total), We observe that DynaMo significantly outperforms prior representation learning methods on real tasks.
Task
BYOL
BYOL-T
MoCo-v3
DynaMo
Sponge
2/10
4/10
5/10
7/10
Allegro
Tea
1/10
0/10
2/10
5/10
Microwave
2/10
3/10
1/10
9/10
Put yogurt
4/5
4/5
2/5
5/5
Get yogurt
0/5
4/5
4/5
5/5
xArm Kitchen
Put ketchup
5/5
3/5
5/5
4/5
Get tea
2/5
2/5
3/5
5/5
Get water
0/5
0/5
3/5
3/5
4.3 Do representations trained with DynaMo work real robotic tasks?
We evaluate the representations pre-trained with DynaMo on two real-world robot environments: the Allegro Manipulation environment, and the multi-task xAmm Kitchen environment, For the Allegro environment, we use k-nearest neighbors policy [40] and initialize with ImageNet-IK features for all pretraining methods, the dataset is relatively small with around 1 (00 frames per task. In the xArm Kitchen environment, we use the BAKU [38] architecture for goal-conditioned rollouts across five tasks. For real-robot evaluations, baselines from our simulated experiments (see Table 1). The results are reported in Table We observe that DynaMo outperforms the best baseline by 43% the single-tase Allegro hand and by 20% on the multi-task xArm Kitchen environment. Additionally, as shown in Table 3, DynaMo exceeds the performance of pretrained representations by 50% the Allegro hand. These results demonstrate that DynaMo is capable learning effective robot representations in both single-task and multi-task settings.
compare DynaMo against the strongest performing
4.4 Is DynaMo compatible with different policy classes?
On the Push-T environment [3], we compare all pretrained representations across four policy classes: VQ-BeT [1], Diffusion Policy [3], MLP (with action chunking [2]), and k-nearest neighbors with locally weighted regression [40]. We present the results in Table 4. We find that DynaMo representa-
7

---

Table We evaluate the compatibility of DynaMo with different policy classes for downstream policy learning on the Push-T simulated benchmark. We report target coverage achieved (maximum 1) and demonstrate that DynaMo significantly outperforms prior representation learning methods across all policy classes.
Method
VQ-BeT
Diffusion
MLP (chunking)
kNN
Random
0.07
0.04
0.07
0.01
ImageNet
0.41
0.73
0.24
0.09
Pretrained
R3M
0.49
0.63
0.27
0.08
representations
VC-1
0.38
0.63
0.22
0.07
MVP
0.20
0.49
0.11
0.08
BYOL
0.23
0.40
0.11
0.04
BYOL-T
0.34
0.50
0.16
0.04
Self-supervised methods
MoCo v3
0.57
0.67
0.30
0.07
RPT
0.56
0.62
0.30
0.07
TCN-SV
0.07
0.14
0.07
0.01
MAE
0.07
0.06
0.07
0.02
DynaMo
0.66
0.73
0.35
0.12
Table 5: We evaluate the ability of DynaMo to finetune an ImageNet-pretraine ResNet-18 encoder across benchmarks. We demonstrate that using a pretrained encoder can further improve the performance of DynaMo.
Franka Kitchen
Block Pushing
Push-T
LIBERO Goal
Representation
(-/4)
(-/2)
(-11)
(-/1)
ImageNet
3.01
0.12
0.41
0.93
DynaMo (random init)
3.64
0.65
0.66
0.93
DynaMo (ImageNet one-tuned)
3.82
0.67
0.50
0.90
tions improve downstream policy performance across policy classes compared to prior state-of-the-art representations. We also note that our representation works on the robot hand in $4.3 with a nearest neighbor policy.
4.5
Can pretrained weights be fine-tuned in domain with DynaMo?
We fine-tune an ImageNet- IK K-pretrained ResNet18 with DynaMo for each simulated environment, and evaluate with downstream policy performance the frozen representation as descrihed in $4.2. The results shown in Table 5. We find that Dy- naMo is compatible with ImageNet initialization, and can be used to fine-tune out-of-domain pre- trained weights to further improve in-domain task performance. We also note that our method works in the low-data regime with ImageNet initialization on the real Allegro hand in Table 2.
Table Pretrained baselines on Allegro
Method
Sponge
Tea
Microwave
ImageNet
4/10
1/10
0/10
R3M
1/10
1/10
5/10
DynaMo
7/10
5/10
9/10
4.6 How important is each component in DynaMo?
In Table 6, we ablate each component in DynaMo and measure its impact on downstream policy performance simulated benchmarks.
Forward dynamics prediction: We replace the one-step forward prediction target si with the same-step target - I- To prevent the model from trivially predicting given 5r, replace the forward dynamics input (+.6-1. 1; 2 with only -- The ablated objective is essentially variant of autoencoding 8t. We observe that removing forward dynamies prediction degrades performance across environments.

---

Table 6: Ablation analysis of downstream performance relative the full architecture (100%)
Ablations
Kitchen
Block
Push-T
LIBERO
No forward
34%
8%
44%
33%
No inverse
72%
35%
97%
41%
No bottleneck
92%
22%
9%
75%
No cov. reg.
94%
62%
85%
59%
No stop grad.
1%
5%
9%
0%
Short context
100%
75%
88%
89%
Table Variants with ground truth actions, downstream performance relative to base model (100%)
Variants
Kitchen
Block
Push-T
LIBERO
Inverse dynamics only
100%
54%
70%
11%
DynaMo + action labels
97%
29%
94%
86%
Inverse dynamics transition latent: As described in $3.1, the forward dynamics loss assumes that the transition is unimodal and requires an inferred transition latent. We observed that removing the latent from the forward dynamics input results in significant performance drop.
Bottleneck on the transition latent dimension: embedding s, we find that having dim = < dim stabilizes training. Here set dim == dim 8, and find that our model can still learn reasonable representation in some environments, but training can destabilize, leading
For the transition latent and the observation
high variance in downstream performance.
Covariance regularization: We find that covariance regularization from Bardes et al. [36] im- proves performance across environments. Training still converges without it, but the downstream performance slightly worse
Stop gradient on target embeddings: We observe that removing techniques like momentum encoder [33, 16] and stop gradient [37] leads to representation collapse [41, 16, 36).
Observation context:
The dynamics objective requires at least 2 frames of observation context. For Franka Kitchen, we find that context of 2 frames works best. For the other environments, longer observation context (5 frames) improves downstream policy performance. Details of hyperparameters used for DynaMo visual pretraining can be found in Appendix B.I.
4.7 Variants with access to ground truth actions
In Table 7, we compare with two variants of DynaMo where we assume access to ground truth action labels during visual encoder training.
Only inverse dynamics to ground truth actions: as proposed in Brandfonbrener et al. [26], we train the visual encoder by learning an inverse dynamics model to ground truth actions, with covariance regularization, and without forward dynamics.
Full model inverse dynamics to ground truth actions: we train the full DynaMo model plus an MLP head to predict the ground truth actions given the transition latents inferred by the inverse dynamics model.
We observe that in both cases, having access ground truth actions during visual pretraining does not seem to improve downstream policy performance. We hypothesize that this is because the downstream policy already has access the actions for imitation learning.
Related works
This work builds on large body of research self-supervised visual representations, learning from human demonstrations, neuroscientific basis for learning dynamics for control, predictive models for decision making, learning from videos for control, and visual pretraining for control.

---

Self-supervised visual representations: Self-supervised visual representations have been widely studied since the inception of deep learning. There are several common approaches sulf-viie visual representation learning. One approach is to recover the ground truth from noise-degraded
samples using techniques like denoising autoencoders [42, 43] and masked modeling [44, 45, 30). Another approach is contrastive learning, which leverages data augmentation priors [41, 16, 33, 36, 37] or temporal proximity [31, 46] to produce contrastive sample pairs. A third self-supervised method is generative modeling [47-49], which learns to sequentially generate the ground truth data. More recently, self-supervision the latent space rather than the raw pixel space has proven effective, as seen in methods that predict representations in latent space [50, 51].
Learning from human demonstrations is well-established idea in robotics [52-55]. With the advances in deep learning, recent works such as [3, 2, 5, 4, 1, 56] show that imitation learning from human demonstrations has become a viable approach for training robotic policies in simulated and real-world settings.
Learning from demonstrations:
Neural basis for learning dynamiis: It is widely believed that animals possess internal dynamics models that facilitate motor control. These models learn representations that are predictive of sensory inputs for decision making and motor control [57-60]. Early works such as [17-20] propose that there exists an internal model of the motor apparatus in the cerebellum for motor control and planning. [21, 22] propose that the central nervous system uses forward models that predict motor command outcomes and model the environment. Learning forward and inverse dynamics models also helps with generalization to diverse task conditions [23, 24].
Predictive models for decision making: Predictive model learning for decision making is well- established in machine learning. Learning generative models that can predict sequential inputs has achieved success across many domains, such natural language processing [61], reinforcement learning [62-64], and representation learning [46, 65]. Incorporating the prediction of future states as intrinsic reward has also been shown to improve reinforcement learning performance [66-68]. Moreover, recent work demonstrates that world models trained to predict environment dynamics can enable planning in complex tasks and environments [69-73].
Learning from video for control: Videos provide rich spatiotemporal information that can be
leveraged for self-supervised representation learning [74-79]. These methods have been extended to decision-maker making through effective downstream policy learning [7-11, 6]. Further, recent work also enables learning robotic policies directly from in-domain human demonstration videos by incorporating some additional priors [80-84], as well as learning behavioral priors from actionless demonstration data [85-87].
Visual representation for control: Visual representation learning for control has been active area of research. Prior work has shown that data augmentation improves the robustness of Icarned representations and policy performance in reinforcement learning domains [88, 89]. Additionally, pretraining visual representations on large out-of-domain datasets before fine-tuning for control tasks has been shown to outperform training policies from scratch [10, 12, 9, 11, 90, 8, 91]. More recent work has shown that in-domain self-supervised pretraining improves policy performance [92-95] and enables non-paramttric downstream policies [40].
6 Discussion and Limitations
In this work, we have presented DynaMo, a self-supervised algorithm for robot representation learning that leverages the sequential nature of demonstration data. DynaMo incorporates predictive dynamics modeling to learn visual features that capture the sequential structure of demonstration observations. During pretraining, DynaMo jointly optimizes the visual encoder with dynamics models extract task-specific features. These learned representations can then be used for downstream control tasks, leading to more efficient policy learning compared to prior approaches. We believe that training DynaMo on larger unlabeled datasets could potentially improve generalization. Additionally, while promising for control tasks, more research is needed to evaluate DynaMo's effectiveness on robotic manipulation outside of lab settings.
10

---

Acknowledgements
We would like to thank Ademi Adeniji. Alex Wang. Gaoyue Zhou, Haritheja Etukuru, Irmak Güzzy, Mahi Shafiullah, Nikhil Bhattasali, Raunaq Bhirangi, Seungjae (Jay) Lee, and Ulyana Piterbarg for their valuable feedback and discussions. This work was supported by grants from Honda, Google, NSF award 2339096 and ONR awards N00014-21-1-2758 and N00014-22-1-2775. LP is supported by the Packard Fellowship.
References
[1] Seungjae Lee, Yibin Wang. Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403.03/86 2024. 1, 6, 7, 10,
[2] Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705 2023. 1, 7, 10
[3] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137. 2023. 2, 5. 7, 10, 17
[4] Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah. and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210 10. 10047, 2022. 1, 10
[5] Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning modes with one stone. Advances in neural information processing systems, 35:22955 -29668, 2022. 1, 4, 10
[6] Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint arXiv:2103. 16817, 2021. 1, 10
[7] Yecheng Jason Ma. Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kurnar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.000334 2022. 10
[8] Tete Xiao, Llija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06117 2022. 2, 6, 10
[9] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.1260) 2022. 6. 10
[10] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsur- prising effectiveness of pre-trained vision models for control. In international conference on machine learning, pages 77359-17371. PMLR, 2022. 10
[11] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances Neural Information Processing Systems, 36, 2024. 1, 2, 3, 6, 10
[12] Sudeep Dasari, Mohan Kumar Srirama, Unnat Jain, and Abhinav Gupta. An unbiased look at datasets for visuo-motor pre-traine pre-training. In Conference on Robot Learning, pages 1183-1198.
PMLR, 2023. 1, 3, 10
[13] Sridhar Pandian Arunachalam, Irmak Güzey, Soumith Chintala, and Lerrel Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5962-5968 IEEE, 2023. 1
[14] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv: 2022 1
11

---

[15] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9640-9649, 2021. 2
[16] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 2020. 2, 3, 5, 6, 9, 10
[17] Daniel M Wolpert, Zoubin Ghahramani, and Michael Jordan. An internal model for sensori- motor integration. Science, 26965232)11880-1888 1995. 2, 10
[18] Daniel M Wolpert, R Chris Miall, and Mitsuo Kawato. Internal models in the cerebellum. Trends in cognitive sciences, 2(9):338-344 1998.
[19] M Shidara, K Kawano, H Gomi, and M Kawato. Inverse-dynamics model eye movement control by purkinje cells in the cerebellum. Nature, 365(6441):50-522 1993.
[20] Shigeru Kitazawa, Tatsuya Kimura, and Ping-Bo Yin. Cerebellar complex spikes encode both destinations and errors in arm movements. Nature, 392(6675):49449997 1998. 10
[21] Chris Miall and Daniel M Wolpert. Forward models for physiological motor control. Neural networks, ((8):1265-127))) 1996. 10
[22] Michael Jordan and David E Rumelhart. Forward models: Supervised learning with a distal teacher. Cognitive Science, 16(3):307-354, 1992. 10
[23] J Randall Flanagan and Alan M Wing. The role of internal models in motion planning and control: evidence from grip force adjustments during movements of hand-held loads. Journal of Neuroscience, 17(4)1559-1122, 1997. 10
[24] Masahiko Haruno, Daniel M Wolpert, and Mitsuo Kawato. Multiple paired forward- -nnerse models for human motor learning and control. Advances in neural information processing systems, 11, 1998. 2, 10
[25] William Whitney, Rajat Agarwal, Kyunghyun Cho, and Abhinav Gupta. embeddings. arXiv preprint arXiv:/908.0990 2019.
Dynamics-aware
[26] David Brandfonbrener, Ofir Nachum, and Joan Bruna. Inverse dynamics pretraining learns good representations for multitask imitation. Advances in Neural Information Processing Systems, 36, 2024. 2.9
[27] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint 1910. 2019, 2. 5, 17
[28] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning. pages 158-168 PMLR, 2022. 2, 3, 5, 17
[29] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024. 5, 17
[30] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and putlern recognition, pages 16000-16009, 2022. 3, 6, 10
[31] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1134-1141. IEEE, 2018 3, 6, 10
12

---

[32] Sarah Young, Jyothish Pari, Pieter Abbeel, and Lerrel Pinto. Playful interactions for representa- tion learning. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 992-999, IEEE, 2022. 3,
[33] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738, 2020. 3, 5, 6, 9, 10
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 4
[36] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variaace-invvaiince-cvvva ization for self-supervised learning. arXiv preprint arXiv:2105.040000 2021. 5, 10
regular-
[37] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021. 5, 9, 10
[38] Siddhant Haldar, Zhuoran Peng, and Lerrel Pinto. Baku: An efficient transformer for multi-task policy learning, arXiv preprint arXiv:2406.07533, 2024. 6, 7
[39] Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. In Conference on Robot Learning, pages 683 (83-693. PMLR, 2023. 6
[40] Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pandian Arunachalam, and Lerrel Pinto. The surprising effectiveness of representation learning for visual imitation. arXiv preprint
arXiv:2112.01511 2021. 7, 10
[41] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020. 9, 10
[42] Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15812, 2023. 10
[43] Vladimiros Sterzentsenko, Leonidas Saroglou, Anargyros Chatzitofis, Spyridon Thermos, Nikolaos Zioulis, Alexandros Doumanoglou, Dimitrios Zarpalas, and Petros Daras. Self- supervised deep depth denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1242-1251, 2019. 10
[44] Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04005, 2018. 10
[45] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprint urXiv:2106.0825: 2021. 10
[46] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1807v:1807.0377 2018. 10
[47] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020. 10
[48] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pages 1747-1756. PMLR, 2016.
13

---

[49] Trieu H Trinh, Minh-Thame Thang Luong, and Quoe V Le. Selfie: Self-supervised pretraining for image embedding. arXiv preprint arXiv: 02940, 2019. 10
[50] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint- embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619-15629, 2023. 10
[51] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning.
2023. 10
[52] Nathan Delson and Harry West. Robot programming by human demonstration: Adaptation and inconsistency in constrained motion. In Proceedings of IEEE International conference on Robotics and Automation, volume 1, pages 30-36. IEEE, 1996. 10
[53] Michael Kaiser and Rüdiger Dillmann. Building elementary robot skills from human demonstra- tion. In Proceedings of IEEE International Conference on Robotics and Automation, volume 3, pages 2700-2705, IEEE, 1996.
[54] Sheng Liu and Haruhiko Asada. Teaching and learning of deburring robots using neural networks. In / [1993] Proceedings IEEE International Conference on Robotics and Automation, pages 339-345. IEEE, 1993.
[55] Haruhiko Asada and Boo-Ho Yang. Skill acquisition from human experts through pattern processing of teaching data. Journal of The Robotics Society of Japan, 8(1):17-24, 1990. 10
[56] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprini urXiv:2304.02532 2023. 10
[57] Richard S Sutton and Andrew G Barto. Toward modern theory of adaptive networks: expecta- tion and prediction. Psychological review, 88(2):135, 1981. 10
[58] Hermann Von Helmholtz. Handbuch der physiologischen Optik, volume 9. Voss, 1867.
[59] Andre M Bastos, W Martin Usrey, Rick A Adams, George R Mangun, Pascal Fries, and Karl J Friston. Canonical microcireuits for predictive coding. Neuron, 76(4):695-711, 2012.
[60] Lisa Feldman Barrett and W Kyle Simmons. Interoceptive predictions in the brain. Nature reviews neuroscience, 16(7):419 429, 2015. 10
[61] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9, 2019. 10
[62] Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning, pages 19561-19579. PMLR, 2022. 10
[63] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05999, 2020.
[64] Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 34: 126869 2021. 10
[65] Karl Schmeckpeper, Annie Xie, Olch Rybkin, Stephen Tian, Kostas Danilidiis, Sergey Levine, and Chelsea Finn. Learning predictive models from observation and interaction. In European Conference on Computer Vision, pages 708-725. Springer, 2020. 10
[66] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity -diven exploration by self-supervised prediction. In International conference on machine learning, pages 2778- 2787. PMLR, 2017. 10
14

---

[67] Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint 1rXiv:1612.07755 2016.
[68] Zhaohan Guo, Shantanu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Jean-Bastien Grill, Yunhao Tang, et al. Byol-explore: Exploration by bootstrapped prediction. Advances in neural information processing systems, 35:31855-31870, 2022. 10
[69] Julian Schrittwisser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604 60006690466600,0 2020.
10
[70] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2565. PMLR, 2019.
[71] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1XX2:0912..0603 2019.
[72] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02190 193, 2020.
[73] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
10
[74] Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learning of spatiotemporally coherent metrics. In Proceedings of the IEEE international conference on computer vision, pages
2015. 10
[75] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual repre- sentations from video. arXiv preprint UrXiv:2404.088 08471, 2024.
[76] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1299-3309 2021.
[77] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2566-25561 2576, 2019.
[78] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1801-1810, 2019.
[79] Sören Pirk, Mohi Khansari. Yunfei Bai, Corey Lynch, and Pierre Sermanet. Online object representations with contrastive learning. arXiv preprint 19X6:19906.0312, 2019. 10
[80] Shikhar Bahl, Abhinay Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. arXiv preprint arXiv:2207. 09450, 2022. 10
[81] Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhinav Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In Conference on robot learning, pages 906-915. PMLR, 2018
[82] Boyuan Chen, Pieter Abbeel, and Deepak Pathak. Unsupervised learning of visual 3d keypoints for control. In International Conference on Machine Learning, pages 1539-1549, PMLR, 2021.
[83] Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang. Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference Computer Vision, pages 570-587. Springer, 2022.
15

---

[84] Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube. arXiv preprint arXiv:2202.10445, 2022. 10
[85] Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International conference machine learning, pages 1755-1763.
PMLR, 2019. 10
[86] Dominik Schmidt and Minqi Jiang. Learning to act without actions. arXiv:2312.10812, 2023.
arXiv preprint
[87] Weirui Ye, Yunsheng Zhang. Pieter Abbeel, and Yang Gao. Become proficient player with limited data through watching pure videos. In The Eleventh International Conference on Learning Representations, 2022. 10
[88] llya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020. 10
[89] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 111111988899955 2020. 10
[90] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pages 416-426. PMLR, 2023. 10
[91] Abhishek Padalkar, Acom Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08 088644 2023. 10
[92] Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. On bringing robots home. arXiv preprint 1rX6022311.1609, 2023.
[93] Gaoyue Zhou, Victoria Dean, Mohan Kumar Srirama, Aravind Rajeswaran, Jyothish Pari, Kyle Hatch, Aryan Jain, Tianhe Yu, Pieter Abbeel, Lerrel Pinto, et al. Train offline, test online: A real robot learning benchmark. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9197-9203. IEEE, 2023.
[94] Irmak Guzey, Ben Evans, Soumith Chintala, and Lerrel Pinto. Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play. arXiv preprint arXiv:2303. /2076, 2023.
[95] Ruijie Zheng, Yongyuan Liang, Xiyao Wang. Shuang Ma, Hal Daumé III Huazhe Xu, John Langford, Praveen Palanisamy, Kalyan Shankar Basu, and Furong Huang. Premier-taco: Pretraining multitask representation via temporal action-driven contrastive loss. arXiv preprint arXiv:2402.061877 2024. 10
[96] Aadhithya Iyer, Zhuoran Peng, Yinlong Dai. Irmak Guzey, Siddhant Haldar, Soumith Chintala, and Lerrel Pinto. Open teach: A versatile teleoperation system for robotic manipulation, 2024.
17, 18
[97] Andrej Karpathy. nanogpt. http://ggthubb..cc/aaaaaaaaaaaan 2023. Accessed: 2024-05-20. 20
16

---

A Environment and dataset details
A.1 Franka Kitchen
The Franka Kitchen environment introdued by Gupta et al. [27] consists of a Franka arm with 9-dimensional action space. This environment includes seven tasks and dataset of 566 human- collected demonstrations. While the original environment is state- state-based, we created an image-based variant by rendering the states to 224 x 224 RGB images.
A.2 Block Pushing
In the Block Pushing environment introduced by Florence et al. [28], the objective is for the robot to push two colored blocks (red and green) into two target squares (also red and green). The training dataset consists of 1 000 trajectories, evenly distributed among the four possible combinations of block target and push order. These trajectories were collected by a scripted expert controller.
A.3 Push-T
In the Push-T environment introduced by Chi et al. [3], the goal is to push T-shaped block to designated target position on table. The dataset for this environment contains 206 demonstrations collected by human operators. The action space in this environment is two-dimensional end-effector position control. Similar to the Franka Kitchen environment, we have created an image-based variant by rendering demonstrations to 224 x 224 RGB images.
A.4 LIBERO Goal
In the LIBERO Goal environment introduced by Liu et al. [29], there are 10 manipulation tasks, each with 50 teleoperated demonstrations for goal-conditioned policy benchmarking. The environment has a 7-dimensional action space and an observation space of 224 x 224 RGB images from two cameras (fixed external view, and wrist-mounted egocentric view).
A.5 Allegro Manipulation
The environment consists of an Allegro hand attached to a Franka arm, and fixed camera for image observations. The observation space is 224 x 224 RGB images, The action space is 23-dimensional, consisting of Cartesian position and orientation of the Franka robot arm (7 DoF), and 16 joint positions of the Allegro Robot Hand. The demonstrations are collected at 50Hz for Franka, and 60Hz for the Allegro hand. The learned policies are rolled out at 4Hz.
We evaluate on three contact-rich dexterous manipulation tasks that require precise multi multi-finger control and arm movement, described in detail below.
Sponge picking: This task requires the hand to reach to the position of the sponge, grasp the sponge, and lift the sponge from the table. We collect 6 demonstrations via Open'eeck [96] for the task, starting from different positions, with 543 frames in total. The task is considered successful if the robot hand can grasp the sponge from the table within 120 seconds.
Teabag picking:
This task is similar to the previous task, but more difficult with a smaller task object. We collect 7 demonstrations via OpenTeach with 034 frames in total. In this task, the robot needs reach the teabag, grasp the teabag with two fingers, then pick it up. The task is considered successful if the robot hand can grasp the teabag from the table within 240 seconds.
Microwave opening: This task requires the hand to reach the microwave door handle, grasp the handle, and pull down the door. We collect 6 demonstrations via Open'Teach with 735 frames in total. The task is considered successful if the robot hand can open the door within 240 seconds.
A.6 xArm Kitchen
This is real-world multi-task kitchen environment comprising Ufactory xArm 7 robot with an xArm Gripper. The policies are trained on RGB images of size 128 x 128 obtained from four different camera views, including an egocentric camera attached to the robot gripper. The action space
17

---

comprises the robot end effector pose and the gripper state. We collect a total of 65 demonstrations across tasks, depicted Figure The demonstrations were collected using OpenTeach [96] at 30Hz. The learned policies are deployed at 10Hz. Figure 5 shows real-world task rollouts for the multitask policy learned for tasks.
yogurt bottle in fridge door: Pick up bottle yogurt and place the door the fridge.
Fetch yogurt bottle from fridge door: Take the bottle of yogurt oul from the door of the fridge.
Put ketchup bottle inside fridge: Pick up the bottle tomato ketchup and put inside the fridge.
Fetch tea bottle from fridge door: Take the bottle of green from the door of the fridge.
Fetch water bottle from fridge: Take the bottle of vitamin water out the fridge.
Figure 5: xArm Kitchen environment tasks
18

---

B Hyperparameters and implementation details
B.1 Visual encoder training
We present the DynaMo hyperparameters below.
Table 8: Envinoomeent-depeedent hyperparameters for DynaMo pretraining, random init
Obs. context
EMA B
Forward dynamics dropout
Transition latent dim
Franka Kitchen
SimSiam
Block Pushing
0.99
0.3
16
Push-T
SimSiam
0
LIBERO Goal
5
SimSiam
0
32
xArm Kitchen
5
0.99
0
64
Table Shared hyperparameters for DynaMo pretraining, random init
Name
Value
Optimizer
A 10
Learning rate
Weight decay
0.0
Betas
(0.9, 0.999)
Gradient clip norm
0.1
Covariance reg. coefficient
0.04
Epochs
40
Batch size
64
Table 10: Envinommentt-depeddent hyperparameters for DynaMo fine-tuning from ImageNet weights
Obs. context
EMA 3
Transition latent dim
Franka Kitchen
2
SimSiam
64
Block Pushing
5
0.99
16
Push-T
5
SimSiam
8
LIBERO Goal
5
0.99
32
Allegro
5
SimSiam
32
Table 11: Shared hyperparameters for DynaMo fine-tuning
Name
Value
Optimizer
AdamW
Learning rate
10-5
Forward dynamics dropout
0.0
Weight decay
0.0
Betas
(0.9, 0.999)
Gradient clip norm
0.1
Covariance reg. coefficient
0.04
Epochs
40
Batch size
64
For Block Pushing and xArm kitchen, we use an EMA encoder with the beta schedule from the MoCo-v3 official repo. For DynaMo training, we constant learning rate schedule for LIBERO
19

---

Goal, and a cosine learning rate decay schedule with 5 warmup epochs on all other environments. For DynaMo fine-tuning, we use cosine learning rate decay schedule with warmup epochs on all environments.
We the following official implementation repos:
MAE: httpp//s/ggtthhbc.c///aaaaararre R3M:
MVP: http://g..gggghtbbccoo////ebbbbr VC-1: htWp::tpppe//ggttooo..o/////////
We base our transformer encoder implementation on nanoGPT [97] https://github... com/
karpathy/nanoGPT
For the Allegro Manipulation environment, fine-tune MoCo and BYOL from ImageNet- IK weights for 1 100 epochs. For all other environments, we train MoCo and BYOL for 200 epochs, MAE for 400 epochs, all from random initialization. The hyperparameters used for training these models are detailed Table 12.
Compute used for training DynaMo:
Franka Kitchen: hours on 1x NVIDIA A100.
Block Pushing: hours on 1x NVIDIA A100.
Push-T: hour on 1x NVIDIA A100.
LIBERO Goal: 2 hours on Ix NVIDIA H100.
Allegro Manipulation: 3 minutes on 1x NVIDIA RTX A6000 for the sponge task, minutes for the teabag task, and minutes for the microwave task.
xArm kitchen: 4 hours on Ix NVIDIA RTX A6000.
Table 12: SSL Hyperparameters
(a) MoCo Hyperparameters
(b) BYOL Hyperparameters
Name
Value
Name
Value
Optimizer
LARS
Optimizer
LARS
Batch size
1024
Batch size
512
Learning rate
0.6
Learning rate
0.2
Momentum
0.9
Momentum
0.9
Weight decay
10-6
Weight decay
1.5 x 10-6
(c) MAE Hyperparameters
Name
Value
Optimizer
AdamW
Batch size
64 2.5 x 10-5
Learning rate
Weight decay
0.05
B.2 Downstream policy training
Table 13, and 15 detail the downstream policy hyperparameters for VQ-BeT, Diffusion Policy and MLP training for the simulated environments.
20

---

For VQ-BeT, we use the implementation from the original paper [1] with the recommended hyperparameters. For Diffusion Policy, we use the implementation https:// thub. com/ real- frall-sttanfodddffffufuuiii pollcic with transformer-based noise prediction network with the recommended hyperparameters. We use Adam W as optimizer for the three policy heads.
Compute used for downstream policy training:
Franka Kitchen VQ-BeT: 8.5 hours on 1x NVIDIA A4000.
Block Pushing VQ-BeT: 4 hours 1x NVIDIA A100.
Push-T VQ-BeT: 7 hours on 1x NVIDIA A100.
Push-T Diffusion Policy: 8 hours on 1x NVIDIA A100.
Push-T MLP: 2 hours on 1x NVIDIA A100.
LIBERO Goal VQ-BeT: 5 hours on Ix NVIDIA A4000.
xArm Kitchen VQ-BeT: 6 hours on 1x NVIDIA A4000.
Table 13: Hyperparameters for VQ-BeT training
Parameter
Franka Kitchen
Block Pushing
Push-T
LIBERO Goal
Batch size
2048
64
512
64
Epochs
1000
300
5000
50
Window size
10
10
Prediction window size
1
Learning rate
5.5 10
10
5.5
10 10-5
5.5 10-5
Weight decay
2 x 10-4
0
2 x 10-4
10
Table 14: Hyperparameters for Diffusion Policy Training
Parameter
Push-T
Batch size
256
Epochs
2000
Learning rate
10
Weight decay
Observation horizon
Prediction horizon
Action horizon
Table 15: Hyperparameters for MLP Training
Parameter
Push-T
Batch size
256
Epochs
2000
Learning rate
10
Weight decay
0
Hidden dim
256
Hidden depth
8
Observation context
5
Prediction context
21

---

C Real robot environment rollouts
Lift the sponge off the table (success)
Lift the sponge off the table (fallure)
Lift the teabag off the table (succass)
the teabag the table (fallure)
Open the microwave door (success)
Figure 6: Rollouts on Allegro Manipulation with our pynaMo--retrained encoder.
22

---

Take bottle of fridge (success)
Take the vitamin water bottle out of the fridge (success)
Take the vitamin water bottle out of the fridge (failure)
Take the yogurt bottle out of the fridge (success)
Put yogurt bottle inside fridge (success)
Put the ketchup bottle inside the fridge (success)
Put the ketchup bottle inside the fridge (failure)
Figure 7: Rollouts xArm Kitchen with our DynaMo-pretrained encoder.
D Broader Impacts
In this work, we present DynaMo, a new self-supervised learning method for pretraining in-domain visual representations for downstream policy learning. Our work takes important step towards improves data efficiency by explicitly modeling the dynamics the demonstration observations, improving upon prior state-offthh-an self-supervised methods, especially in the low-data regime, which can be valuable for robotics and visuomotor policy learning research with limited in-domain data.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We state the claims in the abstract and Section 1, and support them with experimental results in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: will release code and data the website for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification: we will release code and data on the website for reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips..
swiiiisssoooooiioo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececccccccccct httpa//gggiieee/eeeuuuuiiiiiiiii
/CodeSubmic ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: details can be found in Section 4, and Appendix A, B.2, and B.1. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: visual pretraining is immpute-intensive, and we have limited compute.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: details can be found in Appendix B.2, and B.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: / /ttp///eerpp//eeeee cc htcppppiic/tthiicooooeellllel
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:
[Yes]
Justification:
we conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: discuss broader impacts in Appendix D.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: this work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we credit all authors in our paper and respect license and terms use in our codebase.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
28

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: we will be releasing them on the website. All assets have been anonymized. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: this paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: this paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "ggroeyzvk5", "reviewer": "Reviewer_96in", "summary": "This paper presents DynaMo, using in-domain data for self-supervision. It jointly learns a latent inverse dynamics model and a forward dynamics model over a sequence of image embeddings. The", "questions": "Currently, I believe this paper has significant flaws in its experimental design, both in simulation and real-robot settings. As such, my initial score is 4, with the real-robot experiments being a notable strength. However, the existing experiments do not sufficiently support the claims made in the paper. If the authors can provide additional experiments based on my suggestions above, and if the results substantiate their claims, I would be willing to raise my rating.", "limitations": "No.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "This paper is easy to follow.", "weaknesses": "Simplified Real-World Setup:\nThe real-robot experiments appear overly simplistic. Objects seem to be fixed in place, indicated by the red marker on the table, suggesting a lack of randomization in object placement. This setup makes the task easier for conventional imitation learning methods like diffusion policy and act, potentially allowing them to achieve a 100% success rate.\n\nSuggestion: Introduce spatial randomization to the scene. Conduct additional experiments under these conditions to demonstrate Dynamo's superiority in more complex and varied scenarios.\n\n2) Unfair Comparisons in Simulation: \nIn Table 1, Dynamo is compared with several baselines that use different backbones, which makes the comparison potentially unfair.\nImpact: The difference in backbones could skew the performance results, making it difficult to accurately assess Dynamo's relative performance. \n\nSuggestion: Include more experiments of Dynamo with various backbones such as ViT and ResNet-50. Compare these results against the baselines to provide a fairer and more comprehensive evaluation.\n\n3) The motivation for using SSL in this context is unclear. Typically, SSL is advantageous due to its ability to learn from massive datasets without human labels. However, in the field of robotics, in-domain data are often scarce. This could make the application of SSL less persuasive and potentially less effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive comments. We are glad that you consider our robot experiments a notable strength. We will address each of your concerns below.\n\n**Simplified real-world setup**: We would like to clarify that the red marker on the table is for setup reference only. At test time, the object is randomly placed within the convex hull of the demonstration starting positions. We have updated the paper website with all 10 rollout videos for each Allegro task, as well as visualizations of all rollout starting positions in Figure 1 in the PDF. We have also added a kitchen task with more variations in the object starting position in global comment (5).\n\n**Unfair comparisons in simulation**: We would like to kindly note that only MAE-based baselines (VC-1, MVP, MAE) use ViTs as they are incompatible with ResNets. All other baselines use a ResNet18 backbone.\n- We have added MAE as an in-domain SSL baseline with the ViT-B backbone. Please see global comment (1), and Table 1 in the PDF for more details.\n- We have also run DynaMo (ViT-B) on Push-T. It outperforms MAE (ViT-B) by 28%, but underperforms DynaMo (ResNet18) by 22% while taking 4 days (vs. 1 hour) to train. Overall, we find that ViTs trained on small in-domain datasets perform significantly worse than small ResNets, as vision transformers require much more data to perform well.\n- For consistency, we have updated MVP to use the ViT-B backbone, such that all MAE-based baselines use the same backbone. We find that MVP (ViT-S) and MVP (ViT-B) have similar performance across environments.\n\n**Motivation for SSL in the low-data regime**: As you have mentioned, scarce in-domain data is a real problem in robotics. We elaborate in detail our motivation for SSL in the low-data regime in the global comment. To clarify, our work is designed to tackle the problem of learning efficiently from small-scale decision-making data, often with only a few hundred or less trajectories. DynaMo improves downstream performance, is compatible as an in-domain fine-tuning step for other pretrained encoders, and can be trained on much larger datasets, but we have academic compute constraints.\n\nWe hope this addresses your concerns and questions. We would be keen to discuss any remaining points that stand between us and a higher score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cj6jAkKuMw", "reviewer": "Reviewer_kzSJ", "summary": "This paper presents a self-supervised learning method for robot learning that learns representations by using data from demonstrations. The objective is based on learning latent actions from inverse dynamics, and learning forward dynamics model that uses such latent actions as inputs. Several techniques are utilized to prevent the model from finding trivial solutions and thus collapsing. Experiments are conducted in both real-world and simulation environments.", "questions": "See Weaknesses", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear writing with good figures\n- Real-robot experiments!\n- Focuses on the important problem of pre-training representations from demonstrations, as utilizing such limited but in-domain data can be crucial in the context of robot learning where in-domain data is scarse but important especially for fine-grained control tasks.", "weaknesses": "- As other self-supervised learning models are trained on non-standard robotic datasets, it is not clear whether they are trained well with good hyperparameters -- for instance without collapses -- is there a way to ensure that baseline methods are well-tuned?\n- I understand that the main focus of this paper is to introduce a self-supervised learning method and compare its performance to other baselines. But what would the performance look like if you consider the full fine-tuning setup that uses gradients from behavior cloning for updating the encoder? Can we squeeze more information and maybe performance boost from fully fine-tuning the encoder? How would all the methods perform in this setup? This could further strengthen the claims of this paper that we should focus on extracting more information from demonstrations.\n- One important missing baseline is [1] that pre-trains (optionally causal) transformer with masked modelling objective. Even though it uses a pre-trained visual encoder, using features from the causal transformer can be still a baseline Moreover, it's a bit awkward that MAE trained on demonstrations is missing from the baseline even though MVP is selected as a pre-trained representation baseline. Including MAE, maybe optionally its multi-view variant [2], can make results be more convincing.\n\n[1] Radosavovic, Ilija, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. \"Robot learning with sensorimotor pre-training.\" In Conference on Robot Learning, pp. 683-693. PMLR, 2023.\n\n[2] Seo, Younggyo, Junsu Kim, Stephen James, Kimin Lee, Jinwoo Shin, and Pieter Abbeel. \"Multi-view masked world models for visual robotic manipulation.\" In International Conference on Machine Learning, pp. 30613-30632. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review, and pointers to missing baselines. We are glad that you find our in-domain visual pretraining setting important. We will address each of your concerns below.\n\n**\"Is there a way to ensure that baseline methods are well-tuned?\"**: We monitor the observation embeddings for representation collapse during training. For all SSL baselines, we start with the recommended hyperparameters in the paper or official repo, and tune hyperparameters when there seems to be representation collapse. We will release the hyperparameter details for reproduction with the public release of this work.\n\n**\"What would the performance look like if you consider the full fine-tuning setup\"**: We have added end-to-end fine-tuning results on the Push-T environment. Please see global comment (6) and PDF Table 3 for detailed experiment results. We find that fine-tuning further improves the performance of pretrained encoders, and that pretrained representations significantly outperform training the encoder and policy jointly from scratch. We would like to note though fine-tuning the full model takes 4x longer to train.\n\n**\"Missing baseline… transformer with masked modelling objective\"**: Thank you for suggesting RPT as a baseline. We added an observation-only variant of RPT with a ResNet18 backbone. We found RPT to be a strong baseline, outperforming most other SSL baselines, but falling short of DynaMo by 7% across the sim environments. Please see global comment (2) and Table 1 in the PDF for detailed results.\n\n**\"MAE trained on demonstrations is missing from the baseline\"**: Thank you for pointing this out. We have added MAE as an in-domain SSL baseline, discussed in global comment (1). In summary, MAE underperforms DynaMo by an average of 33% across sim environments, and completely fails to solve Block Pushing.\n\nWe hope this addresses your concerns and questions. We would be keen to discuss any remaining points that stand between us and a higher score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mfh7M71rVv", "reviewer": "Reviewer_B6gm", "summary": "This paper presents a self-supervised model, DynaMo, for pretraining visual encoders adopted for visuo-motor control. The targeted downstream task is imitation learning for robotic manipulation. Instead of using an out-of-domain dataset for pretraining and then transferring to a new domain using alternative techniques, the authors propose exploiting sequences of observations from in-domain demonstrations to pretrain three different models, a visual encoder, a forward and an inverse model. Once this is done a policy can be learned with observations encoded using the pre-trained visual encoder.", "questions": "* Are the visual encoder, inverse and forward models only trained on the demonstrations from the respective datasets? Even if this is only for pretraining, the demonstrations, at least the real-world ones, are very few compared to the complexity of the tasks learned. Why not exploit all possible sequences available on the same embodiment, even for tasks that will eventually not be of interest? \n* How restrictive is the assumption that forward models are unimodal? Has this become a weakness during the experiments?\n* Since both the inverse and forward models seem to be ignored after pretraining, what is the motivation for a separation between the two? Why not train a network to predict the next encoded observation from earlier ones, essentially with the inverse and forward models merged into one? Why is the latent action representation needed at all?", "limitations": "Yes, some limitations related the real-world experiments and the unimodality of models are brought up in the conclusions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The most important benefit of DynaMo is that a visual encoder can be trained with limited risk of suppressing data dimensions necessary for visuomotor control, an otherwise frequently occurring problem.\n\nEven if similar models that combine training of forward and inverse models have existed in literature before, the action representation is assumed unobserved in the proposed model, which has rarely been the case before. The literature on imitation learning from observed state sequences is vast, with little cited in the paper. However, the way this is done for pretraining in the proposed model is innovative and easily applicable to a practical scenario.  \n\nThe experiments are rather exhaustive with five different settings and embodiments tested, two of which are real-world scenarios. In experiments that compare to alternative self-supervised methods and pretrained representations, the proposed visual embeddings are shown to be very competitive. It is also shown that DynaMo can be used to finetune an encoder pre-trained on ImageNet for even better results while being relatively insensitive to the choice of policy class.", "weaknesses": "The paper is written as if there were no research in the area before the deep learning boom. Only one citation out of 70 citations is older than 10 years. The paper suggests that training exclusively on in-domain data is new, even if this used to be the way it was typically done before the arrival of data-hungry deep-learning-based models, models that forced people to a greater extent to rely on offline training on out-of-domain data with data augmentation, contrastive learning, etc. \n\nThe idea to train pairs of inverse and forward models online has existed in psychology and robotics for at least 25 years, such as in the works of Wolpert et al [1]. Using similar models, imitation learning has been a common theme over the years, with [2] being just an example. Without this connection back to earlier research, this paper gives the impression of trying to reinvent the wheel, and it becomes unclear what the contributions really are. \n\nEven if the experiments suggest that DynaMo can be beneficial also in real-world settings, the presented experiments are too few to be conclusive. The real world is way more diverse with more than just a small selected set of objects that can be manipulated. However, this weakness is pointed out in the conclusions, which makes it less problematic.  \n\n[1] Wolpert and Kawato, “Multiple paired forward and inverse models for motor control”, Neural Networks, 11, 1998.\n\n[2] Demiris and Hayes, “Imitation as a dual-route process featuring predictive and learning components: a biologically plausible computational model”, in Imitation in Animals and Artifacts, MIT Press, 2002.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and for suggesting these papers. We are glad that you found our action-free assumption innovative. After reading these papers, it is clear that our work and indeed many others in the field of representation learning and imitation learning have been inspired by these seminal works before the deep learning boom. We apologize for the omission and will include them and further related works in our manuscript. The idea of dynamics and predictive coding is well-established in earlier literature. And for imitation learning in robotics, we see early work such as [1][2][3][4], and many others pioneer the currently prevalent setup of learning from human demonstration data for robotic manipulation. We will add an additional subsection in the Introduction and Related Works sections to highlight this inspiration.\n\n**\"It becomes unclear what the contributions really are\"**: Our paper focuses on training a good visual encoder for visual imitation learning. A major problem is that in-domain demonstration data is expensive to collect, and underutilized by common training approaches: the visual encoder is usually either trained end-to-end jointly with the policy from scratch, or pretrained on massive out-of-domain datasets that may not generalize to the task at hand. **Our contribution is twofold.** One, we show that simply modeling dynamics on videos, without any augmentations or actions, is a feasible visual pretraining objective. We empirically show that it improves downstream imitation learning performance on simulated and real robot environments, outperforming prior methods. Two, we show that our method is compatible as an in-domain fine-tuning step. Starting with a strong visual encoder trained on large out-of-domain datasets, our method can further improve its performance by fine-tuning on a small task-specific dataset.\n\n**\"Experiments are too few to be conclusive… the real world is way more diverse\"**: Thank you for raising this important point. We hope that our additional experiments showcase the applicability of our method in various settings. We fully acknowledge that lab settings are rather limited compared to real-world environments. As a community, we need to conduct more experiments outside of labs. Benchmarking the robustness of these encoders in more diverse and realistic environments would be an important direction for future work.\n\n**\"Why not exploit all possible sequences available on the same embodiment\"**: Please see global comment (4) for a detailed discussion, as well as an additional kitchen experiment exploring encoder generalization to an unseen task. To summarize, DynaMo can be trained on much larger datasets, but unfortunately we have limited compute. Nevertheless, we show that an encoder trained with DynaMo on existing tasks can be used to train a policy on a completely new task, although at this small scale, directly pretraining on the new task improves performance. We hypothesize that generalization would improve if we trained on much larger datasets. We also note that DynaMo can be used to fine-tune other models pretrained on Internet-scale datasets like ImageNet for improved performance.\n\n**\"How restrictive is the assumption that forward models are unimodal? Has this become a weakness during the experiments?\"**: This is equivalent to assuming deterministic environment transitions. Given the previous state, and an action, we assume that there is a well-determined next state. We believe this is a reasonable assumption for most real-world manipulation tasks like opening a door, putting away toys, etc. This assumption can fail when the environment is stochastic or otherwise hard to predict (e.g. making a sand dune, playing adversarial sports). In that case, we can easily relax this assumption by predicting a distribution of next states instead. To our best knowledge, the simulated environments are deterministic, and the real environments are essentially deterministic (approximately rigid-body Newtonian dynamics).\n\n**\"Why is the latent action representation needed at all?\"**: Given the same previous state, an agent can act in multiple ways, leading to a distribution of next states, even in a deterministic environment (e.g. picking up a mug by the handle, or by the body). So we first observe both the previous and next states to “fix” the latent action, then predict the deterministic next state given the previous state, and the latent action. We show that this is crucial in the ablation section (§4.6). We will make this clearer in our next revision.\n\nWe hope this addresses your concerns and questions. We would be keen to discuss any remaining points that stand between us and a higher score.\n\n[1] N. Delson and H. West. Robot programming by human demonstration: Adaptation and inconsistency in constrained motion. In Proceedings of IEEE International conference on Robotics and Automation, volume 1, pages 30–36. IEEE, 1996.\n\n[2] M. Kaiser and R. Dillmann. Building elementary robot skills from human demonstration. In Proceedings of IEEE International Conference on Robotics and Automation, volume 3, pages 2700–2705. IEEE, 1996.\n\n[3] S. Liu and H. Asada. Teaching and learning of deburring robots using neural networks. In Proceedings of IEEE International Conference on Robotics and Automation, volume 3, pages 339–345. IEEE, 1993.\n\n[4] H. Asada and B.-H. Yang. Skill acquisition from human experts through pattern processing of teaching data. In Proceedings of IEEE International Conference on Robotics and Automation, volume 3, pages 1302–1307. IEEE, 1989.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TPArBz3Pfa", "reviewer": "Reviewer_MZTu", "summary": "This paper presents a way of pre-training vision encoder for robot control. Specifically, instead of using vanilla contrastive or masked autoencoder approaches, this method creates two models: 1) an inverse dynamics model that estimates the transition latent (actions) and 2) a forward dynamics model that takes in the current encoded visual latent and the transition latent and predicts the next latent observation. The results suggest that the method improves upon existing visual pre-training methods for robotics.", "questions": "The reviewer wants to ask for two sets of experiments to address weakness 1:\n1. How do masked pre-training methods compare to DynaMo when they are trained on the same data? I.e. train two networks analogously with the method provided in VC-1 and MVP on the task-specific dataset, and evaluate their task performance. This experiment would demonstrate that even for specific tasks, pre-training with DynaMo outperforms existing visual pre-training methods on task-specific datasets. \n2. How does DynaMo generalize to unseen tasks (in the sense that it can generalize to tasks outside of the dynamics seen in training)? I.e. pre-train DynaMo on (1) Put Yogurt (2) Get yogurt (3) Get Tea and evaluate on (1) Put ketchup (2) Get Water.", "limitations": "Limitation section is present.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Few of the past works on visual pretraining for robotics consider the time / action but only focus on the visual observation aspect. This work presents a method that attempts to improve visual pretraining by modeling the dynamics present in the dataset. \n2. The results suggest that the method improves upon existing visual pre-training baselines", "weaknesses": "1. Prior visual pre-training for robotics operates under the premise that we have an image or video dataset, where we pre-train on these datasets and then finetune for a particular task. However, this method performs pre-training on the task-specific dataset, which is better aligned with the downstream tasks. Instead of having pre-training and fine-tuning using the same dataset and solving the same task, the objective of visual pretraining (exemplified by MAE, MoCo, etc.) is that if we train on a mass amount of data, we can finetune to a specific task (i.e. ImageNet pre-training then COCO segmentation finetuning). \n2. A few prior works [1,2] have tried to model forward and inverse dynamics concurrently. [1] also uses forward and inverse dynamics to train a visual encoder. The key difference between these works is that here action is modeled as a latent variable. Why ground truth action values are not used in pre-training (especially when pre-training and fine-tuning happen on the same task) is not justified in the manuscript. It would be quite convincing if pre-training is done on natural videos, or large-scale robot datasets where action spaces cannot be standardized, and then shows improved finetuning performance. \n\n[1] Agrawal, Pulkit, Ashvin V. Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. \"Learning to poke by poking: Experiential learning of intuitive physics.\" Advances in neural information processing systems 29 (2016).\n\n[2] Fragkiadaki, Katerina, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. \"Learning visual predictive models of physics for playing billiards.\" arXiv preprint arXiv:1511.07404 (2015).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive feedback. We are glad that you found our approach to visual pretraining novel. We will address each of your concerns below.\n\n**\"Instead of having pre-training and fine-tuning using the same dataset… train on a mass amount of data… finetune to a specific task\"**: We discuss the motivation for in-domain vs. large-scale SSL pretraining in detail in the global comment. In particular, we show that DynaMo is compatible as an in-domain SSL fine-tuning step for Internet-scale pretrained weights like ImageNet (paper Table 5). To clarify, our work specifically tackles the problem of learning efficiently from small-scale decision-making data, often with only hundreds of demonstration trajectories. In this low-data regime, DynaMo improves downstream performance and outperforms pretrained weights and other SSL methods, as shown in Table 1 in the paper. And in principle, DynaMo can be trained on much larger datasets, but unfortunately we have academic compute constraints.\n\n**\"Why ground truth action values are not used in pretraining… if pre-training is done on natural videos, or large-scale robot datasets\"**: This is indeed a very exciting direction. In fact, the motivation for modeling the action as a latent is to make DynaMo applicable to a wide range of datasets including natural videos, and show that simply modeling dynamics on videos, without any augmentations or actions, is a feasible visual pretraining objective for visuomotor control. We do not have the compute to run on massive datasets in an academic setting, but we hope that this opens up a direction for industry labs to explore how dynamics pretraining can scale to Internet-scale datasets. We will make this clearer in our next revision.\n\n**\"How do masked pre-training methods compare to DynaMo… on the same data\"**: Thank you for pointing out the missing baseline. We have added MAE as an in-domain SSL baseline, discussed in global comment (1). In summary, MAE underperforms DynaMo by an average of 33% across sim environments, and completely fails to solve Block Pushing.\n\n**\"How does DynaMo generalize to unseen tasks\"**: we have added a new kitchen task (picking up a bread) to test encoder generalization, detailed in global comment (4). In summary, we use the old DynaMo encoder pretrained on existing kitchen tasks to train a new policy on the unseen task, and find that the policy still manages to complete the task, although pretraining a fresh encoder can improve performance in this low-data regime. We hypothesize that encoder generalization could improve if pretrained on significantly larger datasets, which is an exciting direction that we hope industry labs could explore.\n\nWe hope this addresses your concerns and questions. We would be keen to discuss any remaining points that stand between us and a higher score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for your insightful and constructive comments, and for finding our robot results strong (96in, kzSJ) and the approach novel (B6gm, MZTu). To address your concerns, we have run all requested experiments within our compute budget. You can find our detailed response in individual replies to your review, as well as a summary of shared concerns, additional results and revisions here:

**SSL on in-domain data vs. massive out-of-domain data (reviewers 96in, MZTu, B6gm)**: We would like to clarify that this work is tackling efficient learning with small-scale decision-making data. Several prominent works [1][2][3] deal with demonstration datasets in the few hundred or less trajectories. Our SSL method, DynaMo, is created and designed to accelerate policy learning in the low-data regime. DynaMo improves downstream policy performance with in-domain SSL pretraining, and significantly outperforms learning the encoder and policy from scratch (see (6) below). We also show that DynaMo can also be used as an in-domain fine-tuning step for encoders pretrained on large out-of-domain datasets like ImageNet. Finally, DynaMo is in principle compatible with Internet-scale datasets, but unfortunately we do not have the requisite compute in an academic lab. Running DynaMo on large datasets is an exciting direction that we hope industry labs can follow-up and explore.

**Comparison with more baselines, and additional experiments**:
1. **Comparisons with MAE (reviewers kzSJ, MZTu)**: We train MAE with a ViT-B backbone on all environments with the official implementation. We find that MAE underperforms DynaMo by 24% on Franka Kitchen, 51% on Push-T, and completely fails to solve the task on Block Pushing. (PDF Table 1)
2. **Comparison with RPT (reviewer kzSJ)**: We train an observation-only variant of RPT on all environments with a ResNet18 backbone. We find RPT to be a strong baseline, outperforming most other SSL baselines, but falling short of DynaMo by 7% across the sim environments. (PDF Table 1)
3. **Baseline backbones (reviewer 96in)**: We would like to clarify that only MAE-based baselines (VC-1, MVP, MAE) use ViTs, as they are incompatible with ResNets. All other baselines use a ResNet18 backbone, which we will make clearer in our next revision. We have unified all ViTs to use the ViT-B backbone. We have also run DynaMo (ViT-B) on Push-T. It outperforms MAE (ViT-B) by 28%, but underperforms DynaMo (ResNet18) by 22% while taking 4 days (vs. 1 hour) to train. Overall, we find that ViTs trained on small in-domain datasets perform significantly worse than small ResNets, as larger models require much more data to perform well. (PDF Table 1)
4. **Generalization (reviewers B6gm, MZTu)**: We have added a new real kitchen task (picking up a bread) to test whether the old encoder trained on existing kitchen tasks can be used to train a policy on the new task. We train a policy head with the old DynaMo encoder, and also with the best baseline (old MoCo encoder). We find that policies trained with the existing encoders still manage to complete the task (DynaMo 4/10, MoCo 3/10; successes/total rollouts). We have also trained DynaMo and MoCo encoders from scratch on the new task only, and evaluated likewise. We find that policies trained with the new in-domain encoders exhibit improved performance (DynaMo 7/10, MoCo 5/10). We hypothesize that encoder generalization will improve if trained on much larger datasets, but we have limited academic compute. (PDF Table 2)
5. **Variations in starting positions (reviewer 96in)**: In the task above in (4), the starting positions of the task object are varied across the workspace (\~20x15cm in size). We find that our encoder completes the task 7/10 times, outperforming MoCo (5/10). We would also like to clarify that for the Allegro hand environment, the task objects do have significant variations in their starting positions (\~25x15cm for the sponge, and \~20x10cm for the teabag). We have included a visualization (PDF Figure 1) of the hand task starting positions, and updated the rollout videos on the website. (PDF Table 2)
6. **End-to-end fine-tuning after encoder pretraining (reviewer kzSJ)**: We fine-tune DynaMo-pretrained, MoCo-pretrained, ImageNet-pretrained, and randomly initialized ResNet18. We find that fine-tuning can further improve the performance of pretrained encoders by up to 12%, and that pretrained representations significantly outperform training the encoder and policy jointly from scratch. (PDF Table 3)

We hope that these updates inspire further confidence in our work. At the same time, we invite any further questions or feedback that you may have on our work.

[1] S. Lee, Y. Wang, H. Etukuru, H. J. Kim, N. M. M. Shafiullah, and L. Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403.03181, 2024.

[2] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.

[3] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.

Meta-review: DynaMo is a self-supervised learning approach for robotics that jointly learns inverse and forward dynamics models using in-domain data. This approach avoids suppressing important data dimensions for visuomotor control. The model assumes unobserved action representations, which is novel. The paper could benefit from a better discussion of prior art and context of this technical advance 
Empirical The paper presents its application in both simulated and real-world robot experiments. 
While the approach shows promise, the review highlights concerns about the experimental setup, including simplified real-world scenarios and potentially unfair comparisons in simulations. Furthermore other reviewers requested well-tuned baselines and exploring full fine-tuning setups, as well as additional experiments with more complex scenarios and varied backbones to better demonstrate DynaMo's effectiveness and provide a fairer evaluation.

However, the rebuttal did a good job of providing clarifications with more than one reviewers updating the scores leaning more positively.  All reviewers lean accept after rebuttal phase, and the Meta-reviewer concurs on the overall agreement. 
The authors are advised to review the final comments, and update the manuscript accordingly.. In addition to including clarifications in the main paper, the authors are also advised to describe the possible limitations of this work in more detail.
Decision: Accept (poster)