Spatiotemporal Predictive Pre-training for Robotic Motor Control

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 04:18:46

Abstract: Robotic motor control necessitates the ability to predict the dynamics of environments and interaction objects. However, advanced self-supervised pre-trained visual representations (PVRs) in robotic motor control, leveraging large-scale egocentric videos, often focus solely on learning the static content features of sampled image frames. This neglects the crucial temporal motion clues in human video data, which implicitly contain key knowledge about sequential interacting and manipulating with the environments and objects. In this paper, we present a simple yet effective robotic motor control visual pre-training framework that jointly performs spatiotemporal prediction with dual decoders, utilizing large-scale video data, termed as \textbf{STP}. STP adheres to two key designs in a multi-task learning manner. First, we perform spatial prediction on the masked current frame for learning content features. Second, we utilize the future frame with an extremely high masking ratio as a condition, based on the masked current frame, to conduct temporal prediction of future frame for capturing motion features. This asymmetric masking and decoder architecture design is very efficient, ensuring that our representation focusing on motion information while capturing spatial details. We carry out the largest-scale BC evaluation of PVRs for robotic motor control to date, which encompasses 21 tasks within a real-world Franka robot arm and 5 simulated environments. Extensive experiments demonstrate the effectiveness of STP as well as unleash its generality and data efficiency by further post-pre-training and hybrid pre-training. Our code and weights will be released for further applications.

Main paper: 
 introduction: 
 In NLP and CV, adapting pre-trained foundation models from large-scale data to various downstream tasks has seen great success. For example, pre-trained visual representations using self-supervised [38, 15, 67, 2, 93] or weakly-supervised [71, 25, 55] methods exhibit strong generalization ability for visual understanding. However, in robot learning, due to data scarcity and homogeneity, some groundbreaking methods [53, 1] resort to training from scratch only using domain-specific data.Recently, inspired by the success of transfer learning in CV, many works [69, 73, 65, 58, 59, 19] have explored developing a pre-trained visual representation (PVR) using large-scale out-of-domain data for various robotic motor control tasks. Currently, one successful paradigm [73, 99, 59, 19] is to use large-scale egocentric video datasets [29] and train vanilla vision transformers (ViT) [22] based on MAE [38] , which exhibits excellent learning efficiency and generalization ability for learning policy from raw pixel. Among them, the Ego4D [29] dataset offers numerous first-person human-object interaction scenes and good motion clues. We argue that although learning static spatial structure priors from task-relevant pre-training data sources is crucial, designing a more relevant self-supervised proxy task for motor control should not be overlooked. Therefore, in this paper, we aim to develop a more relevant self-supervised proxy task for robotic motor control representation learning.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.Robotic motor control typically requires fine-grained spatial localization and relatively dense semantics. With its ability to effectively capture low-level geometry and space structure, MAE [38] pre-training excels at this task. However, is dense spatial content sufficient for robotic motor control? Some neuroscientific studies [50, 21, 88] suggest the brain's different areas or cells show specialization. Some are dedicated to processing the information of temporal object motion, while others focus on static spatial details. Their combination results in subjective pattern perception. Inspired by this finding, we hypothesize that an effective robotic motor control pre-training proxy task should require joint learning of spatial content features and temporal motion features. However, current methods [73, 59, 19] use MAE pre-training with image frames from human videos, capturing only static content features. They overlook the temporal motion clues in human videos, which implicitly contain key knowledge about sequential interaction with environment and manipulation of objects.Therefore, we aim to bridge this gap by incorporating these motion clues into our proxy task.Based on the analysis above, the most critical challenge is the absence of action annotations in human video data for modeling object motion. To model interaction and manipulation actions from actionless video data, we propose to implicitly capture them by predicting future frame pixels based on current frame. However, predicting the future frame without any conditions could contain high uncertainty and be extremely difficult. Therefore, we propose to use the future frame with an extremely high masking ratio as a prompt condition, specifically 95%, which serves to reveal some behavior and dynamic priors, i.e. what to do and how to do it. In the experiments section, we will further explore different condition alternatives, including language narration and their combination. Additionally, directly and simply executing temporal prediction could lead the model to overlook static spatial details, and it is also not efficient enough. Therefore, another technical contribution of STP is to jointly perform spatial prediction by masking the current frame with 75% masking ratio. In summary, we present STP, a multi-task self-supervised pre-training framework through spatiotemporal predictive learning. Our STP asymmetrically mask the current frame and future frame from a video clip, using a spatial decoder to conduct spatial prediction for content learning and a temporal decoder to conduct temporal prediction for motion learning. This asymmetric masking and decoder architecture design ensures that our pre-trained encoder focusing on motion information while capturing spatial details. Subsequently, we establish our evaluation scheme. Currently, how to adapt pre-trained visual representations for robotic motor control still remain an open question. Considering the expensive cost of robot data collection or exploration, we employ a data-efficient paradigm of few-shot behavior cloning by learning from demonstrations (Lfd). To demonstrate the generalization ability of visual representation, our primary evaluation scheme involves freezing the visual encoder during policy training. Additionally, considering that fine-tuning ViT with few demonstrations might lead to overfitting and masked modeling exhibits excellent data efficiency [86, 102, 52] in domain-in data, we further follow the post-pre-training [7, 93, 59] paradigm to perform STP pre-training with taskspecific data to achieve better results. It is noteworthy that different tasks do not share representation in this setting. Finally, we conduct the largest-scale BC evaluation of PVRs for robotic motor control to date to demonstrate the effectiveness of STP, which encompasses 21 tasks ( 2 real-world tasks and 19 simulation tasks across 5 environments). These simulation tasks are derived from the union of manipulation and locomotion tasks from prior works [65, 59] .We make the following four contributions: (1) We present STP, a self-supervised visual pretraining framework for robotic motor control, which jointly conducts spatiotemporal prediction with asymmetric masking and decoder architecture design for content and motion features learning. (2) We further expand STP by performing hybrid pre-training with ImageNet-MAE and post-pre-training with task-specific data, unleashing its generality and data efficiency. (3) To our best knowledge, we conduct the largest-scale BC evaluation of PVRs for robotic motor control to date to demonstrate the effectiveness of STP. (4) Our experiments yield some insightful observations. In temporal prediction, language does not significantly enhance performance. Instead, single-modality self-supervised paradigm achieves the best results. This finding is highly encouraging for self-supervised robotic motor control representation learning. Moreover, in the few-shot BC setting, naively scaling up model size does not necessarily lead to improved outcomes. Finally, incorporating more diverse data and domain-in data into the pre-training can further enhance performance. 
related work: 
 Pre-trained Visual Representation Learning. Large-scale visual representation pre-training are continually empowering computer vision. The primary supervised learning methods include learning image recognition [40, 87] from ImageNet [20] and learning multi-modal alignment [71] from imagetext pairs. Currently, self-supervised learning methods are enjoying significant popularity, primarily falling into two main categories. The first category utilizes contrastive learning [39, 15, 14] technique or joint-embedding architecture [13] to learn view-invariance. The second category performs masked modeling [7, 38, 100, 95, 4, 2] and predict the pixel or representation of invisible parts in space. In addition, some methods [106, 67, 8] have also proposed to combine different optimization objectives in a multi-task learning manner. Recently pre-trained visual representation learning for robotic motor control have bee rapidly developing [69, 65, 73, 99, 58, 57, 46, 59, 19] . These methods cover different backbones (ResNet [40] , ViT [22] ), different policy learning methods (reinforcement learning [99] ,behavior cloning [69, 65, 59] , reward function [58] and task specification [42] ), different adaptation schemes (linear probing [69, 65, 46, 59] , fine-tuning [19] and designing adapters [78, 56] ), and different evaluation environments (diverse simulation benchmarks). At present, it is still unclear how these factors collectively influence the performance. In this paper, we choose scalable vanilla vision transformer [22] as our backbone and data-efficient few-shot behavior cloning paradigm to conduct policy learning, while ensuring the backbone is frozen during policy training.Temporal Predictive Learning. Early works once explored representation learning through future prediction, encompassing image [61] , video [35, 80] and audio [66] . VideoMAE [86, 93] extend MAE [38] to 3D video architecture. Recently TrackMAE [17] and SiamMAE [33] predict the masked future frame based on unmasked current frame, leading to a better capture of temporal correspondence and achieving outstanding performance in object tracking and segmentation tasks. In robot learning, predicting future visual states primarily serves as a transition dynamic model such as World Models [62, 77] and Dreamer [76] . [85, 9] predict the future visual states using goal image in robot data. GR-1 [97] conducts language-conditioned video prediction for policy model pre-training in a frozen visual representation space. [96] proposed dynamics-aware representation learning, and [82, 72] employed forward dynamics for self-supervised pre-training. Some works explored to train video prediction models and utilize visual foresight [32] , inverse dynamics models [18] , goal-conditioned policy learning [23] , and geometry estimation [51] methods for motor control, respectively. [92] fine-tuned pre-trained representations into dynamic and functional distance modules for manipulation tasks. Unlike these works, we utilize the public large-scale egocentric video data and employ masked spatiotemporal predictive learning as a self-supervised proxy task (without any language or action annotations) for robotic motor control representation learning, instead of designing elaborate architectures or methods for specific predictive tasks [28, 37] .Vision-based Robot Learning. Vision-based robot learning plays a crucial role in robotics community. Recently some related works focus on studying model architectures [44, 12, 47] , observation spaces [107] , downstream policy learning methods [41] , sim-to-real transfer [79] , designing adapters [78, 56] , learning-from-scratch baseline [36] , and affordance model [6, 105, 45, 60] , in visuo-motor representation learning. Other related works [70, 5, 91, 101, 48] attempt to learn manipulation skills from small-scale and in-domain human videos. In addition, language-conditioned vision robot learning has received significant attention. Some works scale multimodal robotic data [42, 11, 34, 90, 24, 68, 84] or introduce Internet data and knowledge [81, 103, 10, 54, 43, 94, 64] for end-to-end robot learning. In our study, we pre-train a off-the-shelf visual representation from large-scale egocentric video datasets for robotic motor control tasks. Our method is more simple and general for different downstream tasks of motor control. 
method: 
 In this section, we describe our method in details. First, we give an overview of our spatiotemporal predictive pretraining (STP) framework. Then, we give a technical description on our core components during pre-training: the masked image encoder and dual decoders scheme. Finally, we describe how to adapt our pre-trained encoder to downstream robotic motor control tasks.As illustrated in Figure 1 , our STP aims to pre-train an image encoder for robotic motor control from video datasets. This pre-trained image encoder is subsequently frozen and directly transferred to solve motor control tasks. Specifically, given a video dataset D, our goal is to learn an image encoder Φ enc , that maps images to the visual representations. During pre-training and post-pre-training, D represents large-scale out-of-domain videos and task-specific demonstration videos, respectively. After pretraining, we reuse Φ enc for downstream motor control policy learning. Specifically, the downstream task will require an agent to make sequential action decisions based on visual observations O. Instead of using the raw observation images as direct input like end-to-end policy learning from pixel, the agent will employ the pre-trained Φ enc to extract its visual state representation Φ enc (O) for the subsequent policy learning module.We first introduce the pipeline of our image encoder. Our image encoder processes image frames using a vanilla vision transformer [22] . Given a image I ∈ R C×H×W , we initially process it by the patch embedding layer to obtain its token sequences T, where T = {P i } N i=1 and N is the the total token number, (e.g., N = 196 for a 224 × 224 image with a patch size of 16 × 16). Then we add the fixed 2D sine-cosine positional embeddings for all tokens. Following this, we mask and remove a part of tokens, according to a randomly generated masking map M(ρ), where ρ is the masking ratio.The encoder applies several transformer blocks (consisting of a global self-attention layer and a FFN layer) on all unmasked tokens: Z = Φ enc (T u ), whereT u = {T i } i∈(1-M(ρ)). During this process, a[CLS] token is added at the beginning.Then we describe our encoding process during pre-training. We randomly sample two frames from a video clip based on an interval: the current frame I c and the future frame I f . Following the above pipeline, we randomly generate two asymmetric masking maps for the current frame and the future frame, denoted as M c = M c (ρ c ) and M f = M f (ρ f ), respectively. Each of these maps has a different masking ratio. We then use these maps to separately process the two frames and obtain their features, Z c and Z f . As analyzed above, our STP aims to jointly learn content and motion features by spatiotemporal predictive learning. For content feature learning, we follow MAE [38] , masking a portion of the current frame based on M c , with ρ c = 75%, and predict the masked parts during the decoding process. This encourages the model to learn spatial and geometric structure priors from the current frame data through spatial reasoning. For motion feature learning, we establish an objective to predict the future frame based on the masked current frame. However, predicting the future frame without any conditions could be meaningless and extremely challenging. Therefore, we use the future frame with an extremely high masking ratio as a condition, specifically ρ f = 95%, which reveals some behavior and dynamic priors. In the experiments section, we will further discuss different condition schemes, including language narration and the combination between them. In summary, our encoding process during pre-training can be formally described as follows: EQUATIONTo jointly capture static content and object motion features for better spatiotemporal understanding, our STP present a dual decoders scheme to predict both the pixel of current and future frame simultaneously in a multi-task learning manner. As shown in Figure 1 , our dual decoder scheme includes a spatial decoder Φ dec_s for spatial prediction and a temporal decoder Φ dec_t for temporal prediction. We firstly give a technical description on them, respectively. Then we describe how we combine them into our final method.Spatial Decoder. To capture static content features, our spatial decoder is solely utilized for processing the current frame visual feature. Specifically, after obtaining the masked current frame visual feature Z c , we concatenate it with some learnable masking tokens, leading to the formation Temporal Decoder. To capture motion features, our temporal decoder jointly processes the current frame and the future frame which serves as the temporal prediction condition. To elaborate, we firstly obtain the masked current frame feature Z c and the masked future frame feature Z f . We then concatenate Z f with the masking tokens that have the positional embedding added, resulting in Z d f .of Z d c = Z c ∪ {M i } i∈Mc ,Following this, Z d f and Z c interact within the temporal decoder for decoding. The architecture of our temporal decoder block is in alignment with the standard transformer decoder block [89] , consisting of a self-attention layer, a cross-attention layer, and a FFN layer, as shown in Figure 2 (b). During decoding, the self-attention layer and FFN are solely used to process Z d f . For the cross-attention layer, Z d f is continuously updated as the query, while Z c , acting as the key and value, is kept constant.Compared to standard architecture, it ensures that the past frame representation space will not be updated in the temporal decoder and are specifically used for temporal correlation and prediction.This asymmetric interact architecture not only achieves more efficient training but also produces better results. Finally, with the decoded token sequence Z d f , our temporal decoder predicts the invisible tokens of the future frame Îd f , operating under the future frame masking map M f .Multi-task Predictive Learning. As mentioned above, our STP jointly conducts spatiotemporal prediction by asymmetric masking ratio and dual decoders scheme, the whole decoding pipeline can be formally described as follows:EQUATIONOur loss function is the mean squared error (MSE) loss between the normalized masked pixels and the predicted pixels. So our loss function ℓ is as follows:EQUATIONTo enable data and computation efficiency during the policy learning process, we adopt the paradigm of few-shot behavior cloning by learning from demonstrations (Lfd), and we keep the image encoder Based on the S, we train a policy mdoel, π θ (a|C(Φ enc (o))), parameterized by θ, which maps from robot's state representations to actions. Here, C represents an optional concatenation operation that effectively fuses multi-view and multi-frame visual features, along with the robot's proprioceptive state in the channel dimension. We optimize the π θ through a standard behavior cloning MSE loss:EQUATION4 Experiments 
implementation on pre-training: 
 We execute pre-training with data from EgoVLP [55] for comprehensive ablation and fair comparison.It processes untrimmed videos of Ego4D and filters out that miss language narrations and belong to validation or test sets, resulting in a total of 3.8 million clips, called as Egoclip. In pre-training, we sample a frame pair from each clip for training. As for all experiments, we employ ViT [22] as backbone. Additionally, we maintain consistency with prior works [73, 59] , directly using the [CLS] token as the global representation. The pre-training hyperparameters can be found in section A.3.Evaluation Scheme. Following popular settings on PVRs for robotic motor control [65, 46, 59] , for each task, we learn a single policy π which is structured as a MLPs network. The policy models utilize both the history of visual observation embeddings and optional robot proprioceptive as inputs, subsequently generating executable actions as outputs.Simulation Tasks. We select the union of manipulation and locomotion tasks from prior works [65, 59] for evaluation, encompassing 19 tasks across 5 simulated environments. These inclue Meta-World [104] (Assembly, Bin-Picking, Button-Press, Drawer-Open, and Hammer), Franka-Kitchen [31] (Sliding Door, Turning Light On, Opening Door, Turning Knob, and Opening Microwave), Adroit [74] (Relocate and Reorient-Pen), DMControl [83] (Finger-Spin, Reacher-Hard, Cheetah-Run, Walker-Stand, and Walker-Walk), and Trifinger [98] (Reach-Cube and Push-Cube).More detailed simulation evaluation details can be found in section A.4.Real-World Tasks. In our real-world experiments, we evaluate contact-rich picking and pouring tasks using a Franka Emika Research 3 robot arm in a tabletop environment, ensuring no duplication with simulation Franka-Kitchen [31] . In this section, we mainly analyze the performance of some pre-trained image representations on reproducible simulation tasks. Specifically, we first evaluate the following models: (1) public DINOv2 [67] that combines masked image modeling with self-distillation on large-scale image datasets;(2) public CLIP [71] that conducts contrastive learning on large-scale image-text pairs;(3) R3M trained based on Egoclip [55] ; (4) public VC-1 [59] ; (5) MAE trained based on Egoclip;(6) STP trained based on Egoclip. ( 7) STP that conducts hybrid pre-training with initialization using ImageNet-MAE [59] . Among them, ( 1) and ( 2) achieve excellent performance on core visual understanding tasks using zero-shot or linear probing evaluation settings. ( 3) and ( 4) utilize egocentric videos for robotic motor control. ( 5), ( 6) and ( 7) are used for fair comparison and exploring the potential benefits of STP from more diverse image data, respectively. The experimental results are presented in Table 1 . Consistent with prior findings [41, 59] , there is not a universal foundation model that performs optimally across all benchmarks. However, on the whole, the MAE method is superior due to its effective modeling of low-level geometry and spatial structure, especially for the MetaWorld tasks that demand fine-grained control. Another intriguing observation is that MAE underperforms in the Franka-Kitchen and Adroit tasks. We believe that this could be due to its relatively weaker semantic representation. Under a fair comparison, our STP outperforms MAE by 4.1 (59.6 → 63.7), and additionally benefits from a more diverse image data, improving by 0.5 (63.7 → 64.2). This is attributed to that our STP not only captures static content features but also effectively models motion information by extracting temporal clues from videos of interactions and manipulations with the environment and objects. Additionally, we provide the visualization of the attention maps (model ( 5) and ( 6)) of several specific tasks in Figure 4 . The results indicate that, on top of effectively capturing spatial information, our method further encourages the model to focus on motion areas or objects, thereby providing a more sparse and compact representation for downstream low-data BC paradigm.Next, we also evaluate and compare the adaptation results of our representations to downstream motor control tasks. Specifically, we evaluate following settings: scaling up model capacity does not necessarily lead to performance gains. In the few-shot BC setting, there is a risk of overfitting in both policy and backbone training.In this section, we perform extensive ablation studies to further demonstrate the effectiveness of our joint spatial and temporal prediction, as well as temporal prediction condition design. In addition, we also study the influence of temporal decoder architecture design and future frame sampling strategy.Current frame masking. The design of the current frame masking is crucial. On one hand, similar to MAE [38] , masking some patches and predicting the missing parts can effectively promote the learning of image content features. On the other hand, the visible patches of the current frame need to interact with the condition to predict the future frame. Specifically, we mask the current frame at masking rates of 75%, 50%, and 0%, respectively, and optionally predict the missing parts through the spatial decoder. The results are shown in Table 2 (a). From results, we see that the masking ratio of 75% and performing spatial prediction still lead to the best performance. This demonstrates the importance of retaining MAE [38] for content features learning, especially for low-level manipulation in Meta-World, while a current frame with a high masking ratio (75%) is sufficient to interact with other conditions to predict the future frame.Temporal prediction condition design. Subsequently, we discuss the influence of temporal prediction condition design. We implicitly model motion in actionless video data by predicting the pixels of the future frame. A direct and simple idea is to use language narration as a condition. The text tokens can be flexibly utilized as inputs to ViT [22] , forming a multimodal encoder. Language narration provides a high-level behavior description, but lacks low-level visual dynamic priors for pixel-level prediction. However, leaking part of the future frame can effectively provide these priors. In order to explore how to construct a more meaningful temporal prediction proxy task, we compare the following schemes: (1) only language narration, (2) masking 95% of the future frame, (3) masking 90% of the future frame, (4) masking 95% of the future frame and language narration, and (5) masking 95% of the future frame and language narration, but the language is added in the temporal decoder, instead of being fused with the visible image patches in the multimodal encoder. We tokenize all language narration by pre-trained DistilBERT [75] . The results are shown in Table 2 (b). From results, we see that using only language as a prediction condition leads to a significant decline in performance, while leaking a small amount of future frame (masking 95%) in the temporal decoder can achieve competitive results. As for joint conditions of language and future frame with 95% masking ratio, adding language in the encoder is better than in the decoder. Additionally, adding language performs better on DMControl (64.1 vs. 62.1) and Trifinger (70.8 vs. 69.3), while not adding language performs better on Meta-World (92.0 vs. 91.0), Franka-Kitchen (40.9 vs. 37.7) and Adroit (48.0 vs. 46.7). We speculate the reasons for language hurts performance are as follows: (i)The input gap (multi-modal and single-modal) between upstream and downstream; (ii) Extra language in ViT may result in the loss of some fine-grained information capture. Furthermore, the latter does not require language supervision, and can provide a more scalable self-supervised solution.Temporal decoder design. We also investigate the impact of the temporal decoder design. Specifically, we consider two types of decoder blocks. One is the joint-self architecture, as shown in Figure 2 (a), and similar joint architecture are adopted in [26, 102] . The other is the self-cross architecture, as shown in Figure 2 (b), and similar cross architecture are adopted in [3, 33] . We consider the following settings: (1) 8 joint-self decoder blocks, (2) 12 joint-self decoder blocks, (3) 8 self-cross decoder blocks. Among them, setting ( 2) and ( 3) have similar amounts of parameters for a fairer comparison.The results are shown in Table 2 (c). The results demonstrate the importance of maintaining a fixed representation space of the past frame during temporal prediction.Frame sampling strategy. Finally, we investigate the impact of the sampling strategy between the current frame and future frame. The difficulty of temporal prediction is directly proportional to the frame interval values. We establish four settings where we fix the sampling intervals at 8, 16, and 24 respectively, and for the fourth setting, we randomly select an interval within the range of [8, 24] .The results are shown in Table 2 (d). The results show that an interval of 16 achieves the best balance for building temporal prediction proxy task.Table 3 : Performance comparations on real-world tasks.Method Picking Pouring Average It can more accurately align with the moving bowl and the pot. In addition, although MAE and STP have a same success rate in picking tasks, STP tends to execute grasping in a better position. This indicates that the trend and conclusion of our STP are consistent in both simulation and the real-world, which also aligns with the findings of [79] . 
conclusion: 
 In this work, we have proposed the STP, a simple, efficient and effective self-supervised visual representation pre-training framework for robotic motor control. Our STP jointly performs spatiotemporal predictive learning on large-scale videos within a multi-task learning manner. Our STP captures content features by predicting the invisible areas within the masked current frame, and simultaneously captures motion features by using a future frame with an extremely high masking ratio as a condition to predict the invisible areas within that future frame. We carry out the largest-scale BC evaluation of PVRs for robotic motor control to date to demonstrate the effectiveness of STP. Furthermore, as for pre-training data, we also prove that extending STP to hybrid pre-training and post-pre-training could further unleash its generality and data efficiency. In regards to the STP post-pre-training, we utilize data that aligns with the policy training, and the specific architecture hyperparameters correspond to those listed in Table 4 . Depending on the specific demonstration data, we adjust the values of total epochs, warmup epochs, effective batch size, and the frame interval, as shown in Table 5 .As for policy training and evaluation schemes, we primarily refer to the publicly available code 2and training data of VC-1 [59] for Metaworld [104] , DMControl [83] , Adroit [74] and Trifinger [98] .Similarly, for Franka-Kitchen [31] , we follow the public code 3 and training data of R3M [65] .Specifically, the policy training hyperparameters and evaluation schemes are shown in Table 6 and Table 7 , respectively. About policy training, we completely follow the setting of prior works [65, 59] when freezing the encoder; when performing end-to-end fine-tuning, we make appropriate adjustments to the batch size and learning rate. About evaluation details, similar to prior works [65, 59] , we establish all evaluation details such as the number of expert demonstrations and test trajectories, environmental viewpoints, optimization hyperparameters, base seeds, history windows size, and the use of robot proprioceptive. In Table 7 , the term 'prop.' stands for whether proprioceptive information is used or not, and 'history window size' signifies the number of frames received by the policy model at each step, with features between frames being fused through concatenation.'Number of trajectories' represents the quantity of trajectories evaluated. For tasks in Meta-World, Franka-Kitchen, Adroit, and Trifinger, we report the maximum success rate, whereas for tasks in DMControl, we report the maximum reward score, rescaling to be in the range of [0, 100] by dividing by 10. We report the average metric across tasks for each environment. In addition, it is worth noting that the metrics we report are the average value across all base seeds and camera viewpoints.Finally, we also report the results of our post-pre-training STP (ViT-B/16) on each task in Table 8 .In addition, we emphasize that different random seeds primarily affect the rendering of the initial frame in the sampled trajectories, as shown in Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We will release all codes and model weights on github.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specify all the training and test details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: In our experiments, different seeds are primarily used for rendering different initial frames. Therefore, our evaluatation is comprehensive and sufficient, while our comparisons are absolutely fair.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide sufficient information on the computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer:[Yes]Justification: Our paper aligns with these.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks. 
 
 Appendix: In this section, we outline the details of our real-world setup and evaluation scheme. As depicted in Figure 8 , our real-world scenario includes four camera viewpoints: top, left, right, and wrist. It includes two Kinect DK and two RealSense cameras. An example of four views is shown in Figure 9 .Specifically, we utilize four different camera views and resize their resolution uniformly to 224 × 224.To effectively model the complex and multimodal action distribution in our real-world tasks, we select diffusion policy [16] as our policy model. In accordance with this approach, we concatenate the visual embeddings of all views from two sequential frames. Following the approach in [27] , we collect robot data using a VR tele-operation setup. In this way, we collect 100 continuous trajectories for each task. It is worth noting that the quality of these demonstrations leaves room for improvement and contains a lot of noise. During the evaluation phase, we primarily evaluate two contact-rich tasks that have not appeared in Franka-Kitchen [31] • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://github.com/facebookresearch/maehttps://github.com/facebookresearch/eai-vc/tree/main/cortexbenchhttps://github.com/facebookresearch/r3m/tree/eval/evaluation 


Decision: The paper presents a spatio-temporal pretraining pre-training method, STP, for robotic motor control. While the authors have addressed initial concerns with additional experiments, there are still concerns about the approach's novelty and the marginal performance improvements over existing techniques. A significant issue is the need for more extensive and varied real-world robotic experiments. Considering the feedback, the paper's limited distinction from existing methods, modest performance improvements, and the need for more comprehensive real-world robotic testing, the manuscript does not meet the acceptance criteria. The decision is to reject the paper.

Reviews: review: paper_summary: paper_summary: In this paper, we present a self-supervised pre-trained visual representation in robotic motor control, with spatiotemporal prediction with dual decoders, utilizing large-scale video data. The spatial prediction follows a standard MAE pipeline, and the temporal prediction tries to predict the future based on the current frame. The trained encoder is applied to downstream tasks and real-world robot task for better sample efficiency.
strength_weakness: strength_weakness: 1. The major concern is the novelty of the previous methods, considering several related papers that leverage human data and visuals pertaining to downstream tasks have been proposed [1-3].

2. The experiment only contains imitation learning experiments in downstream tasks, while the reinforcement learning framework with sub-optimal data is not considered. 

[1] Learning Manipulation by Predicting Interaction. RSS 2024

[2] Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning. https://arxiv.org/html/2402.14407

[3] Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation. https://arxiv.org/abs/2312.13139
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes STP, a visual representation learning method for robotic motor control. Trained on human videos, STP uses masked auto-encoders for spatial-temporal prediction. The spatial decoder predicts the current frame from its representation with 75% of patches masked. The temporal decoder predicts the future frame using the representations of 75%-masked current frame and the 95%-masked future frame. Experiments on various simulation and real-world tasks show the effectiveness of STP compared with baselines.
strength_weakness: strength_weakness: 1. Many works have considered temporal information for robot visual representation learning. This paper should mention these and highlight the differences. For example, R3M [1] uses temporal contrastive learning, while VIP [2] and V-PTR [3] use temporal difference.
2. Though STP outperforms the baselines in many benchmarks, the performance gap is not significant (Table 1). The slight performance difference may be due to hyperparameter selection and randomness, as the paper did not provide error bars over multiple seeds.

[1] R3m: A universal visual representation for robot manipulation, 2023
[2] Vip: Towards universal visual reward and representation via value-implicit pre-training, 2022
[3] Robotic Offline RL from Internet Videos via Value-Function Pre-Training, 2023
questions: questions: 1. I think VIP and V-PTR should be included as baselines.
2. What is the evaluation protocol for downstream tasks? Does all evaluation use an expert dataset and perform imitation learning to learn a policy? How did you collect the dataset for real-world experiments?
limitations: limitations: The authors have discussed the limitations. These cannot be addressed within the scope of this paper.
paper_summary: paper_summary: The paper presents a new spatio-temporal pretraining algorithm for representation learning for robotics. The authors propose using masked autoencoding for reconstructing the current frame (for spatial reasoning) and a future frame (for temporal reasoning). The authors provide extensive experimentation across simulated and real-world settings and provide ablation studies to justify their design choices.
strength_weakness: strength_weakness: - It is unclear where the diverse image data for STP trained with Ego+I in Table 1 is obtained from. Some information about this would be helpful.
- The real-world experiments seem limited with only two real-world tasks where the MAE also performs reasonably well.
- The authors must include comparisons with prior works using MAE for spatiotemporal learning [1].

[1] Feichtenhofer, Christoph, Yanghao Li, and Kaiming He. "Masked autoencoders as spatiotemporal learners." Advances in neural information processing systems 35 (2022): 35946-35958.
questions: questions: It would be great if the authors could address the “Weaknesses” listed above.
limitations: limitations: The limitations have been addressed adequately.
paper_summary: paper_summary: This paper studies how to extract useful visual features from out-of-domain and action-free human videos to enhance robotic visualmotor control. Specifically, the authors argure that naively extracting spatial features via MAE is insufficient for robotics control, in contrast, jointly captureing spatial control and temporal movement will be more effective. To do so, the authors propose STP, a new self-supervised learning method, that simutaneously performs MAE on current frame to extract spatial information and predict furture frames to extract temporal motion clues. The overall motivation, idea and method are straightforward and reasonable. The authors evaluate STP on diverse benchmarks including 21 tasks spanning from simulation to real world tasks using imitation learning.
strength_weakness: strength_weakness: 1. The high costs associated with evaluating real-world tasks using different random seeds make it challenging to report variances. However, assessing the impact of multiple random seeds in simulated tasks could provide more reliable statistical insights. As shown in Table 1, STP's performance improvement over baselines is marginal (STP 63.7 vs. VC-1 61.8, and STP-L/16(Post PT) 78.4 vs. MAE-L/16(Post PT) 76.7). Given the inherent stochastic nature of imitation learning and reinforcement learning, evaluations across multiple episodes and various random seeds are crucial to validate the proposed methods effectively.

2. Some previous methods also consider the temporal movements when extracting the visual features. For instance, the video-language alignment loss in R3M [1] tries to align language with correct visual transitions, which can extract semantic informations about visual movements. Voltron[2] and DecisionNCE [3] also try to extract the semantic features of the temporal movements between two frames. VIP[3] and LIV[4] use RL to extract visual features, which may also capture long-term movements via bootstrapping. Therefore, the authors could strengthen their paper by highlighting these related works, demonstrating awareness of existing methods, situating their contributions and highlighting the differences between STP and these baselines.

[1] R3M: A Universal Visual Representation for Robot Manipulation. CoRL 2023

[2] Language-Driven Representation Learning for Robotics. RSS 2023.

[3] DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning. ICML 2024.

[4] VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training. ICLR 2023.

[5] LIV: Language-Image Representations and Rewards for Robotic Control. ICML 2023
questions: questions: Please refer to the weakness for details.
limitations: limitations: The authors have properly discussed the limiations in the Appendix.

Comments: title: Thanks for your comments
Thanks for your positive response
Thank you for the clarifications
Discussion on the difference with Voltron
More in-depth discussion about Voltron should be provided
comment: Thank you for your comments and acknowledgement of our responses. We primarily follow the setting of a series of previous works [1,5,6,7,8], employing a **computation and data-efficient** paradigm of **few-shot behavior cloning by learning from demonstrations (Lfd)** to verify the effectiveness of the visual representation. After carefully considering your suggestions, we have reported some preliminary reinforcement learning results in the review, and we will further explore a more comprehensive evaluation in the future. We hope that the insights gained from our experiments will contribute to further improvements in reinforcement learning for robotics motor control in future work.

[5] The (Un)Surprising Effectiveness of Pre-Trained Vision Models for Control.

[6] Real-World Robot Learning with Masked Visual Pre-training.

[7] Language-driven representation learning for robotics.
 
[8] An unbiased look at datasets for visuo-motor pre-training.
Thanks for the response. The sub-optimal setting with RL requires further investigation. I keep the original evaluation.
Thanks for your comments and the recognition of our responses.
Thank you for your reply. I appreciate your efforts in running experimental results for baselines and for multiple seeds. While STP underperforms compared to VIP in some benchmarks, it excels in others. I have decided to raise my score to 6.
Thanks for your quick reply and acknowledgement to our response.
I thank the authors for the clarifications and the additional results. After considering the rebuttal, I would like to keep my score.
Thanks for your comments and the recognition of our responses.
I agree with the authors that this paper conducts more comprehensive ablations and evaluations to support the effectiveness of each design choices. In this sense, I am very happy to increase my score. Meanwhile, considering the similarity to Voltron (the authors provide more detailed techinique insights about the differences to voltron, but the high-level differences are like what I pointed out). I decide to increase to a 5 (boardline accept).
Thanks for your prompt reply to our response and acknowledgement to our evaluation efforts. On the difference of our STP with Voltron, we would like to make the following clarifications and hope this can well address your concern.

1. Our STP shares a simpler design than Voltron just as you mentioned from multi-modal setting to uni-modal setting, which makes our STP a more scalable approach than Voltron as uni-modal data is more easily obtained.

2. The key difference is that our STP decouples the current frame and future frame for separate modeling, while Voltron employs the MAE-ST (VideoMAE) pre-training to jointly model the whole clip (2 frames). Our decoupled design leads to several important differences in technique design:

a. The encoder design is different. Our STP encoder processes each frame independently and there is no attention operation between frames. However, the Voltron encoder operates on a clip to learn a video-level representation and there is cross-frame attention operations. Our design aims to encourage our encoder to learn an image-level representation that is temporally-sensitive for prediction. We find this image-level encoder is more friendly for the downstream adaption compared with the video-level representation (video encoder has higher computational cost).

b. The decoder design is different. Our STP has two decoders: one for spatial prediction and the other for temporal prediction, to treat current frame prediction and future frame prediction separately. For the spatial decoder, we use `joint self-attention` to process the current frame; for the temporal decoder, we add `cross-attention` to capture the interaction between the current frame and future frame (See Fig. 2 in the paper). `This design ensures the predictive property of STP, in sense that the representation of current frame will not see the future and acts as condition for future prediction`. The Voltron only employs a single decoder to reconstruct the whole video directly.

c. The masking strategy is different. The decoupled design allows STP to assign different masking ratios to current frame and future frame. Specifically, we use ratio of `75%` and `95%` for them, while Voltron uses the same ratios for all frames.

3. The ablation studies in in Tab 2(a), Tab 2(b) and Tab 2(c) demonstrate that these different technique designs as mentioned above are `crucial` for achieving excellent performance. Meanwhile, during rebuttal, we add a direct comparison with MAE-ST (The architecture is the same with Voltron without language input). Our STP is better (63.7 vs. 52.6, see response to Reviewer QdYc).

4. Finally, our evaluation is more comprehensive than Voltron on the backbone scaling, pre-training data, and robotic motor control downstream tasks. In terms of pre-training data, Voltron only uses `small-scale` Something-Something-v2 dataset for pre-training. At the same time, the model size of Voltron (V-gen) is only `small`. In contrast, STP uses larger scale training data and trains the `ViT-Large`.  Our self-supervised and uni-modal settings ensure the scalability and generality of STP. 

If you have further concerns, please feel free to comment. We would like to answer your question.
Thanks for the efforts and these detailed responses! I acknowledge the evaluation efforts the reviewer made to solidify their claims. However, it seems that this paper shares many similarities with Voltron: `they both use masked reconstruction on current and future frames to extract the temporal as well as the spatial features for downstream robot learning`. 

In my view, the core differences are three folds:
1. `Settings`: Voltron studies the multi-modal setting, but this paper studies the uni-modal setting.
2. `Methods`: Voltron uses the same mask ratios for both current and future frames, but this paper adopts different ratios.
3. `Methods`:Voltron adopts the same transformer decoder to reconstruct current and future frames, but this paper designs a spatial and temporal decoder to decode them separately. 

So, for me, it looks like this paper degenerates from more complex multi-modal settings in Voltron to simpler uni-modal setups, with the overall objective (reconstructing both the current and future frames) very similar.  In this sense,  the technical contributions look more like some detailed implementation improvements. Therefore, it would be better for the authors to discuss more on the very similar Voltron.