Humanoid Locomotion as Next Token Prediction
Authors: Ilija Radosavovic
Bike Zhang
Baifeng Shi
Jathushan Rajasegaran
Sarthak Kamat
Trevor Darrell
Koushil Sreenath
Jitendra Malik
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Real-World Humanoid Control
Next Token Prediction
Primary area: robotics
TL;DR: We cast real-world humanoid control as a next token prediction problem.

Abstract: We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Our model is a causal transformer trained via autoregressive prediction of sensorimotor sequences. To account for the multi-modal nature of the data, we perform prediction in a modality-aligned way, and for each input token predict the next token from the same modality. This general formulation enables us to leverage data with missing modalities, such as videos without actions. We train our model on a dataset of sequences from a prior neural network policy, a model-based controller, motion capture, and YouTube videos of humans. We show that our model enables a real humanoid robot to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained on only 27 hours of walking data, and can generalize to commands not seen during training. These findings suggest a promising path toward learning challenging real-world control tasks by generative modeling of sensorimotor sequences.

Humanoid Locomotion as Next Token Prediction
Ilija Radosavovic
Bike Zhang* UC Berkeley
Baifeng Shi' UC Berkeley
Jathushan Rajasegaran" UC Berkeley
UC Berkeley
Sarthak Kamat UC Berkeley
Trevor Darrell
Koushil Sreenath UC Berkeley
Jitendra Malik UC Berkeley
UC Berkeley
Abstract
We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Our model is causal transformer trained via autoregressive prediction of sensorimotor sequences. To account for the multi- modal nature of the data, we perform prediction in modality-alignel way, and for each input token predict the next token from the same modality. This general formulation enables without actions. We train our model on a dataset of sequences from a prior neural network policy, model-based controller, motion capture, and YouTube videos humans. We show that our model enables real humanoid robot to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained only 27 hours of walking data, and can generalize to commands not seen during training. These findings suggest a promising path toward learning challenging real-world control tasks by generative modeling of sensorimotor sequences.
leverage data with missing modalities, such as videos
1
Introduction
The last decade of artificial intelligence (AI) has shown that large neural networks trained on diverse datasets from the Internet lead to impressive results across different settings. The core enablers of this wave of Al have been large transformer models [37] trained by generative modeling of massive quantities of language data from the Internet [26, 6, 27, By predicting the word, these models acquire rich representations of language that can be transferred to downstream tasks [26], perform multi-task learning [27, 25), and learn few-shot manner [4].
Are such modeling techniques exclusive to language? Can we learn powerful models of sensory and motor representations in similar fashion? Indeed, have seen that can learn good representa- tions of sigh-dimensional visual data by autoregressive modeling [5] and related masked modeling approaches [] While there has been positive signal learning sensorimotor representations in the context of manipulation [28], this area remains largely unexplored.
In this paper, we cast humanoid control as data modeling of large collections of sensorimotor sequences. Like in language, we train general transformer model to autoregressively predict the sequences. In contrast to language, the nature of data in robotics is different. It is high-dimensional and contains multiple input modalities. Different modalities include sensors, like joint encoders or inertial measurement units, well motor commands. These give rise sensorimotor sequences which we view as the sentences of the physical world. Adopting this perspective suggests a simple instantiation of the language modeling framework in the robotics context. We tokenize the input sequences and train a causal transformer model to predict future tokens. Importantly, we predict complete input sequences, including both sensory and motor tokens. In other words, the joint data distribution as opposed to the conditional action distribution.
modeling
* Authors with stars and daggers listed in arbitrary order. Author contributions listed at the end of the paper.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Figure I: A humanoid that walks in San Francisco. We deploy our model to the robot and test it in various locations in San Francisco over the course of one week. Please see our project page for videos. We show that our model enables the humanoid walk over different surfaces including walkways, concrete, asphalt, tiled plazas, and sanded roads. Our model can follow omnidirectional velocity commands well and enables deployment in challenging city environment like San Francisco.
2

---

Our core observation is that if a sequence is incomplete, i.e., some of the sensory or motor information missing, we can still lear from by predicting whatever information is present and replacing the missing tokens with learnable mask tokens. The intuition is that if the model has learned to make good predictions, even in the absence of information, will have acquired better model of the physical world. An important source of such data are human videos from the Internet. Namely, we can observe human movement in videos but we do not get access to the motor commands or complete sensory inputs. We demonstrate that our method can learn from such data sources effectively.
To validate our method, we apply it to the challenging problem of real-world humanoid locomotion using a full-sized humanoid robot. We first construct a dataset of sensorimotor sequences in simula- tion. These include complete sequences from a neural network policy trained with reinforcement learning [29], as well as incomplete sequences from three different sources: (i) a model-based controller, (ii) motion capture of humans, and (iii) videos of humans. We reconstruct human videos using computer vision techniques and approximately retarget both the motion capture and video sequences via inverse kinematics. We then train an autoregressive transformer to model this dataset. At test time, we execute the predicted motor commands and ignore the sensory predictions.
We demonstrate that our model can be deployed in the real world zero-shot. Specifically, we deploy our model to a range of different locations and surfaces in San Francisco over the course of one week. Please see Figure for example images and project page for videos. To quantitatively evaluate different aspects of our approach, we perform experiments in simulation. We find that our models trained via sequence modeling alone can be comparable to the state-of-the-is reinforcement learning approach [29] in the settings where data is available. We further find that our approach can benefit from incomplete data, has favorable scaling properties, and can generalize to unseen commands. We encourage the readers to see the arXiv version of this work for additional experiments.
These findings suggest a promising path toward learning challenging real-world robotic control tasks by generative modeling of large collections of sensorimotor sequences.
2 Related Work
Generative models. Generative models have been studied extensively, starting from Shannon's 's work [33] to the modern era of large language models [4]. Various such models emerged over the last decade. Examples include GAN [10] and Diffusion models [35, 14] for generating pixels, LSTM [15] and GPT [26] for generating language tokens. These models have been adopted for other modalities as well [23, 9, 38). Among these, autoregressive transformer models became the front runner, due to the impressive scaling behaviors [17] and ability to learn from in-context examples [3]. They have been successfully extended to other modalities as well, such as vision [5] and vision-language [32]. We explore autoregressive generative models in the context of real-world humanoid locomotion.
Transformers in robotics. Following the success of transformer models [37] in natural language processing [26, 6, 27, 3] and computer vision [7. 111, over the last few years, there has been an increased interest in using transformer models in robotics. We have seen a number of works showing that transformers can be effective with behavior cloning. For example, [34] learns multi-task transformer policies with language, [2] trains language-condee conditioned manipulation policies from large- scale data, [8] trains language models with embodied data, and [1] trains policies conditioned on image goals. We have also seen that transformer models can be effective for large-scale reinforcement
learning [29]. Related to ours, [28) learns sensorimotor representations with masked prediction. Like this body of work, we share the goal of using transformer models for roboties but focus on autoregressive modeling of sensorimotor sequences in the context of humanoid locomotion.
Humanoid locomotion. Legged locomotion has been long-standing challenge in robotics. Hu- manoid locomotion is particularly challenging and has been studied extensively over the the past several decades [18, 13]. Stable locomotion behaviors have been demonstrated through model-based control approaches [30. 16], while optimization- based methods can further enable highly dynamic humanoid motions [19]. Although significant progress has been made, learning- based approaches have the potential to facilitate future progress due to their potential to improve from data and general- ize to new environments. Recently, we have seen that purely learning based approach trained with large-scale reinforcement learning in simulation can enable real-world humanoid locomotion [29]. We use the same architecture, an autoregressive transformer, but propose a different training procedure. Specifically, we train the model with sequence modeling instead of reinforcement learning.
3

---

Data
Training
Deployment
Neard semuors polcy
Mocel bused contoder
Mettor copeame
Interetstiios
Transformer
Figure 2: Humanoid locomotion as next token prediction. We collect a dataset on trajectories from various sources, such from neural network policies, model-based controllers, human motion capture, and YouTube videos of humans. Then we use this dataset to train transformer policy by autoregressive modeling of observations and actions. Our transformer allows humanoid to walk zero-shot on various terrains around San Francisco. Please see our project page for video results.
3 Approach
In this section, we assume dataset D of sensorimotor trajectories T and describe our approach.
3.1 Objective
Each sensorimotor trajectory is sequence of observations, such joint positions, and actions: T = (01, d1 02, 02 OT, ar). We first tokenize the sequence into tokens to obtain / = (4 42, ... (x). Our goal is to train neural network to model the density function p(() autoregressively:
K
p(t)
II k=1
We train our model by minimizing the negative ligeliikl trajectory dataset, assuming Gaussian distribution with constant variance:
I.
log
LED
Instead of regressing the continuous values of token elements, we could quantize each dimension into bins or perform vector quantization. However, we found the regression approach to work reasonably well in practice and opt for it in our experiments for simplicity.
3.2 Missing modalities
In the discussion so far we have assumed that each trajectory sequence of observations and actions. Next, show how our framework be generalized to sequences with missing modalities, like trajectories extracted from human videos that do not have actions. Suppose we are given a trajectory of observations without the actions T (0), 02, OT). Our core observation is that can treat a trajectory without actions like a regular trajectory with actions masked. Namely, we can insert mask tokens [M] to obtain
(01, [M], 02, [M] ...,...c or, [M]). This trajectory now has the same format as the regular trajectories and can thus be processed in a unified way. We ignore the loss limited to actions and applies to any other modality as well, such as partially missing sensory observations.
for the predictions that correspond to the masked part of inputs. Note that this principle
3.3 Aligned prediction
Rather than predicting the next token in modality-agnosti way, we make predictions in a modality- aligned way. Namely, instead of predicting the next token the sequence, for each input token we predict the next token of the same modality. Please see Figure 3 for diagrams.

---

Training with complete data
Training with missing
Transformer
Transformer
Figure 3: A general framework for training with different data sources. Our data modeling allows us to train our transformer with multiple modes training. In the case of observation-action pairs being available, we train our transformer to predict the next pair of observation-action. When there is no action data available, with MoCap and internet data, we only train our transformer to predict the next observations by masking the actions with mask token. These two models of training allow model to utilize both types of data, and this enables us to scale our training in terms of data.
3.4
Joint training
We have two options for training on collections that contain diverse trajectories terms of noise levels or modality subsets. We can either train jointly with all data at once, including complete and incomplete trajectories. Alternatively, we can first pre-train on noisy and incomplete trajectories. This can be viewed as providing good initialization for then training fine-tune the model on complete data with actions. We find that both approaches work comparably in setting and opt for joint training in the majority of the experiments for simplicity.
complete trajectories. And then
3.5 Model architecture
Our model is transformer [37] with a fairly standard architecture. Given the trajectories from either complete incomplete data, we first tokenize the trajectories into tokens. We learn separate linear projection layers for each modality but share weights across time. To encode the temporal information use positional embeddings. Let assume o, R and a, R", then:
= concat(o;,a;).
h Wt+
where W E Rdd((m+n) is a linear projection layer to project concatenated observation and action modalities d dimensional embedding vector. The superscript indicates the embedding at 0-th layer, ie., the input layer. When action is unavailable, we use mask token [M] E R" to replace a, and [M] is initialized as random vector and learned end-to-end with the whole model. The model takes the sequence of embedding vectors Ho = {4 ......, hD] as input.
The transformer architecture contains L layers, each consisting of multi-head self-attention module and an MLP module. Assume the output the layer / is Hi, then the layer output is computed follows:
H, LayerNorm(HH
(5)
H. H. MHSA(HH)
(6)
H4. H. MLP(
Here, the multi-head self-attention has causal masking. where the token only attends to itself and the past tokens. Once the tokens are processed through all the layers, we project the embedding to predicted states and actions, by learning linear projection layer W R(m+n)xd,
frt1 Why
(8)
0 = (f41)om
(9)
6 = (Ft1))
(10)
5

---

Neural Net Controller
Model based Controller
MoCap
Internet Videos
Figure 4: Training dataset. To train our model, we construct dataset of trajectories coming from four different sources. (i) neural network policy: provides trajectories with complete observations and actions. (ii) model-based controller: produces trajectories from the same robot morphology but without actions. (iii) motion capture of humans: does not contain actions and is approximately retargeted onto the robot. (iv) internet videos of humans: sequences of human poses are first reconstructed via computer vision techniques and then approximately retargeted onto the robot.
Then train the transformer with the objective in (2). In the cases where the token is masked, do not apply any losses. We train our transformer with both types of data, as shown in Figure This allows us use various sources of data, thus enabling scaling in terms of data.
3.6 Model inference
At inference time, our transformer model always has access to observation actioon pairs. In this setting. we apply our transformer model autoregressively for each observation-action pair token. By conditioning on past observations and actions, we predict the next action (or observation-acee -aationnnaaiinn pair) and execute the predicted action. We then record the ground-truth observation from the robot and discard the predicted observation. We use the ground-trude observation and predicted action as the next set of tokens and append them to the past tokens to predict the next observation-acee -action pair.
Dataset
Our approach motivates building a dataset of trajectories for training our model. We construct dataset of trajectories from four different sources: (i) neural network policy, (ii) a model-based controller, (iii) human motion capture, and (iv) human videos from YouTube. illustration of trajectories from different data sources is shown in Figure 4. We describe each data source next.
4.1 Neural network trajectories
As the first source of training trajectories, we use a state-of-the-art neural network policy trained with large-scale reinforcement learning in simulation [29]. Specifically, this policy trained with billions of samples from thousands of randomized environments in Isaac Gym [22]. We run this policy in the physies simulator developed by Agility Roboties and collect 10k trajectories of 10s each on flat ground, without domain randomization. Each trajectory is conditioned on velocity command sampled from clipped normal distribution follows: linear velocity forward [0.0, 1.0] m/s, linear velocity sideways (0.5, 0.5) m/s, and turning angular velocity [-0.5, 0.5] rad/s.
Since have access to the data generation policy, as well as the exact actions that the model predicted. We use this set as our source of complete sensorimotor trajectories that have complete observations well ground truth actions.
able to record complete observations
4.2 Model-based trajectories
As the second source of trajectories, we use the model-based controller developed by Agility Robotics. is the controller that is deployed on Digit humanoid robot and available in the simulator provided flat ground of 10 seconds
by Agility Robotics well. We collect 20k trajectories of walking each, where we sample the velocity commands as follows: linear velocity forward [-1.0, 1.0] m/s, linear velocity sideways [-1.0, 1.0] m/s, and turning angular velocity [-1.0, 1.0] rad/s.

---

The model based controller outputs joint torques, which are not consistent with our joint position action space. Thus, we only record the observations without the actions. This data serves as source of trajectories with accurate observations from the same robot morphology but without the actions.
4.3 Human motion capture trajectories
As the next source of trajectories, we use the motion capture (MoCap) recordings of humans from the KIT dataset [24] distributed via the AMASS repository [21]. This data was recorded using optical marker-based tracking in a laboratory setting. The dataset consists of ~4k trajectories. We use a subset of ~1k standing and walking trajectories. We exclude motions like dancing and jumping.
In addition to not containing the ground truth actions, the MoCap trajectories come with an additional challenge: different morphology. Namely, MoCap trajectories capture human keypoint positions in 3D and over time. In order to use these trajectories for training a robot, we solve an inverse kinematics problem to approximate the corresponding robot poses. Please see the arXiv for additional details.
4.4 Trajectories from YouTube videos
Internet videos people doing various activities are potentially a vast source of training data for humanoid robots. However, the raw pixels have no information about the state and actions of the human. To recover this, we first we run a computer vision tracking algorithm PHALP [31] to extract sequences of 3D human poses. This provides estimate of the 3D joints of the human body SMPL [20] parameters and a noisy estimate of the human joints in the world coordinates.
We use the human body joint positions to retarget the motion to the humanoid robot using the inverse kinematics optimization, like the case of motion capture data discussed previously. After we retarget the motions from human videos to humanoid robot trajectories, we filter out the trajectories with high reconstruction error. Note that the scale of this data comes with the cost of being noisy.
Experiments
We evaluate our approach on the challenging task of humanoid locomotion. We perform outdoor experiments real hardware and systematic evaluations in simulation. We encourage the readers to see the extended version of this work on arXiv, which includes additional experiments and ablations.
5.1 Experimental setup
Robot platform. We use the Digit humanoid robot developed by Agility Roboties. It is full-sized humanoid that is 1.6m tall and weighs 45 kilograms. The robot has 36 degrees of freedom including the floating base, 20 of which are actuated. Due its high dimensionality and the four-bar linkage structure, it is challenging to simulate accurately which makes it particularly interesting for learning approaches like ours that learn from data, without requiring explicit model simulator.
Model. Our model has a hidden size of 192 dimensions, with 4 layers of self-attention layers and MLP layers. Each self-attention has 4 heads. We use LayerNorm before each attention layer and ReLU activation after the MLP layer. We use a BatchNorm layer to process the input before the transformer model. When predicting token time k. keep the context length at reasonable size, we only keep the past 16 steps in input. In Section 5.6, we show the model is able to scale up to more parameters and longer context length and achieve higher performance. We train on 4 NVIDIA A A100s.
5.2 Real-world deployment
We begin by reporting the results from the real-world experiments. Specifically, we deploy our model the robot and evaluate at various locations in San Francisco over the course of one week. Please see Figure for examples and project page for videos. We find that our model is able to walk over variety of surfaces including walkways, concrete, asphalt, tiled plazas, and dirt roads. Note that the deployment in large city environment, like San Francisco, is more challenging than in constrained laboratory environments. The city environment crowded with pedestrians, less controlled, and less forgiving. This makes the error tolerance low and requires model that works consistently well.
7

---

RL
Ours
Policy
Pas. Trocking Error (m)
0.40
Urkkceled
30
30
0.35
0.30
3
0.25
30" (m)
330
0.20
0.4
Yow Command (radis)
Yow Command (radis)
Figure 5: Comparison to state of the art, tra- jectory adherence. The robot is commanded to walk starting from the origin with fixed heading command of 0.5 m/s and varying yaw commands in [-0.4, 0.4] rad/s. We plot the desired (dotted) and actual (solid) trajectories for our policy and a reinforcement-leamee learnnng trained policy (RL).
Figure Tracking error comparisons. Left: We find that our model can follow yaw commands more accurately than the state-of-the-ar RL pol- icy [29]. Right: We see that our model can benefit from unlabeled trajectories without actions which is promising signal for scaling our method to large datasets of diverse trajectories.
5.3 Evaluation metrics
Next, we evaluate the models in simulation under two metries: tracking error and prediction error. Tracking error measures how accurately the robot follows specific command. The prediction error is the next token prediction loss measured on separate set of validation data. We introduce two metries with details as follows and show that two metrics can consistently predict locomotion performance.
Tracking error. In all experiments, the robot starts from rest in simulated environment and is issued a constant natural walking command consisting of a desired heading velocity sampled in [0.35, 0.70] m/s, angular velocity sampled in [-0.4, 0.4] rad/s, and zero lateral velocity. We compute x" (t), the ideal robot base position trajectory that fully satisfies the velocity command v (t) at all time steps. To measure the accuracy of command tracking, we define the position tracking error as Ei= [|x(()= (1)]. Each trajectory lasts for a duration of 10 seconds. For all evaluation experiments, we use the MuloCo simulator [36] which is able to simulate the Digit robot accurately.
Prediction error. Since the model trained with the next token prediction, evaluate the prediction error on a set of validation data that is held out from training data and contains state-action trajectories collected from the RL policy. This similar the language modeling evaluation for large language models [12]. We test both state and action prediction errors and add them together as the error metric.
5.4 Comparison to the state of the art
Trajectory adherence. We compare our model to a neural network controller trained with re- inforcement learning (RL) [29]. Figure 5 presents visual comparison of the trajectory adher- ence of our controller against these state-of-the-art baselines. Starting with a robot at the ori- gin, we plot the actual trajectory of the robot with eleven different yaw commands selected from (0.00, +0005, +-0.10 10, +0.20, +0.30, +-0.40] rad/s. For each model, we jointly plot the desired and actual path traced by the robot base. Our model exhibits better tracking than the RI. controller all turning speeds, and achieves close to perfect tracking for straight-line walking.
Quantitative evaluation. In Figure 6, left, repeat the above comparison the RL controller (N = 245), with the full range of heading and yaw velocities mentioned in Section 5.3. We plot the mean position tracking error, binned by the commanded angular yaw. While both models have lower tracking errors at lower yaw, ours consistently outperforms the baseline RL policy. Note that our model was trained on dataset that includes the trajectories from the same baseline RL policy.
5.5 Training with action-free data
One of the benefits of our approach is that it can be applied to trajectories from diverse sources, including with missing information such as actions, like in the case of human videos. In Figure 6, right, we compare the performance of training only with complete trajectories to joint training with both complete and incomplete trajectories. We observe that including incomplete trajectories
8

---

Training Data
Context Length
Model Size
0.37
0.31
0.33
Tracking
0.31
0.29
0.31
0.28
0.28
0.30
Trajectories (K)
Timesteps
Parameters (M)
Figure 7: Scaling studies. We find that our approach has favorable scaling properties, and that the performance improves with the dataset size (left), context length (middle), and model size (right).
consistently leads to better performance. This suggests that incomplete trajectories can still provide useful signal and is a promising signal for scaling our approach to large datasets of diverse trajectories.
5.6 Scaling studies
Training data. In Figure 7, left, we report the performance of our model as the size of the training dataset increases. We find that training on more trajectories results in a considerably lower position tracking error. This is a promising signal for scaling our approach larger datasets in future work.
Context length. In Figure 7, middle, we study the effect of increasing the number of tokens used in the context window of the transformer model. We observe that the larger context windows result in better performance which suggests that our model is able to benefit from the additional context.
Model size. In Figure 7, right, we compare the models of increasing size (IM, 2M, 8M) by varying embedding dimension (144, 192, 384), number of attention heads (3, 4, 12), and number of transformer blocks (4, 4, 6). We see that the error decreases monotonically with model size.
5.7 Prediction error correlates with performance
We collect 14 models trained with different train- ing recipes, model architectures, data size and types, and compute the tracking and prediction errors for each of them. In Figure 8, we re- port the tracking and prediction errors of all the models in single scatter plot. We tracking and prediction eтror are highly corre- lated with Pearson coefficient suggests that the prediction error is predictive of performance, and that models with lower pre- diction error on the validation set are likely to follow input commands with higher accuracy.
0.28
Pos. Tracking Error (m)
0.32
that
0.34
0.36
0.87. This
r=0.87
0.38
1.3
1.2
1.1
1.0
0.9
Prediction Loss (10-2)
Figure 8: Prediction error correlation.
6 Discussion
Limitations. Our approach still lags behind state-of-the-ar MPC controllers and is weaker than the state-of-the-ar RL controllers in some regards, most notably robustness. In principle, we could scale our model to millions of YouTube trajectories. However, a major obstacle for doing this is the reliability of computer vision techniques which require considerable amount of curation in practice.
Conclusion. We cast real-world humanoid locomotion as next token prediction. Our model is trained sensorimotor sequences, which come from a neural network policy, model-based controller, human motion capture, and videos of humans. We show that our model enables humanoid robot to walk in the real world zero-shot. These findings suggest a promising path toward learning challenging real-world robotic control tasks by generative modeling of large collections of sensorimotor sequences.

---

Contributions
Ilija Radosavovic conceptualized and led the project. Generated robot datasets, implemented initial codebase and models, designed and performed experiments, and led the writing of the paper.
Bike Zhang worked on retargeting, performed experiments, and contributed to writing.
Baifeng Shi designed and implemented improved labeled training, trained and evaluated models, implemented improved unlabeled training, performed experiments, and contributed to writing.
Jathushan Rajasegaran designed and implemented improved model architectures, trained and evaluated models, generated human datasets, performed experiments, and contributed to writing.
Sarthak Kamat performed experiments and contributed to writing.
Trevor Darrell advised BS and SK and provided guidance.
Koushil Sreenath advised BZ, provided feedback, and robot equipment.
Jitendra Malik advised IR and JR and was the senior advisor to the project.
Acknowledgements
This work was supported in part by DARPA Machine Common Sense program and ONR MURI program (N00014-21-1-280) for JM, DGX A100 donation from NVIDIA, the Hong Kong Centre for Logistics Roboties and The AI Institute for KS, and BAIR's industrial alliance programs. We thank Saner Cakir and Vikas Ummadisetty for help with retargeting, and Alexei Efros for discussions.
References
[1] Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, al. Robocat: A self- improving foundation agent for robotic manipulation. arXiv:2306. 11706, 2023.
[2] Authony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv:2212.068877 2022
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[4] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 2020.
[5] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.
[6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HCT, 2019.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2020.
[8] Danny Driess, Fei Xia, Mehdi SM Sajaadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. 2rX3v:2303.03337. 2023.
[9] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis.
2019.
[10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
10

---

[11] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. drXiv:2/11.06377. 2021.
[12] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 2XX1::0009033000 2020.
[13] Kazuo Hirai, Masato Hirose, Yuji Haikawa, and Toru Takenaka. The development of honda humanoid robot. In ICRA, 1998.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[15] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
[16] Shuuji Kajita, Fumio Kanehiro, Kenji Kaneko, Kazuhito Yokoi, and Hirohisa Hirukawa. The 3d linear inverted pendulum mode: simple modeling for a biped walking pattern generation. In IROS, 2001.
[17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. wxXv:2201.0836/1 2020.
[18] Ichiro Kato. Development of wabot 1. Biomechanism, 1973.
[19] Scott Kuindersma. Recent progress atlas, the world's most dynamic humanoid robot, 2020.
[20] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, 2023.
[21] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In ICCV, 2019.
[22] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. In NeurlPS, 2021.
[23] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. 16X9v 2016.
[24]
Matthias Plappert, Christian Mandery, and Tamim Asfour. The KIT motion-language dataset. Big Data, 2016.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In ICML, 2021.
[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative tre-training. 2018.
[27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. 2019.
[28] Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. In CoRL, 2023.
[29] Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 2024.
[30] Mare H Raibert. Legged robots that balance. MIT press, 1986.

---

[31] Jathushan Rajasegaran. Georgios Pavlakos, Angjoo Kanazawa, and Jitendra Malik. Tracking people by predicting 3d appearance, location and pose. CVPR, 2022.
[32] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[33] Claude E Shannon. Prediction and entropy of printed english. Bell system technical journal, 1951.
[34] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: multi-task transformer for robotic manipulation. CoRL, 2022.
[35] Jascha DicksDiccs,,in, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICML, 2015.
[36] Emanuel Todorov. Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[38] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilistic latent space of object shapes via 3d generative-adversarial modeling. In NewrIPS, 2016.
12

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [No]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
ootssSubbmssooooooiooooooooooooo Polic)lyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
CCodeeubeeeeeeeeee / //iips.coo. wwtpccccccpgpppaaaaceceaaooocapt /ttpa////ggiieeeeeeeuuubiiiiii ceabbbmsssionnl ssionnolll seiinnnylcy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : //nuurrrss..../ cc cc/ppbllitt/thhhchuddeeeeiiie
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
16

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi sneerswwthcodee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
17

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
18


Reviews:
[{"review_id": "0MOH1MtMGl", "reviewer": "Reviewer_rJUi", "summary": "This work view the robot locomotion control problem as an next token prediction problem. A causal transformer is trained autoregressively on various sources of data. The performance on a full-sized humanoid robot's locomotion indicates that this formulation can be a promising path for complex robotic control problems to incorporate diverse sources of data.", "questions": "1. In Table 1(c), why no  Pred. Err. for staged training?\n\n2. In Sec. 4.3, can the authors detail the inverse kinematics problem they solve?\n> In order to use these trajectories for training a robot, we solve an inverse kinematics problem to\n183 find the corresponding robot poses.\n\n3. In Sec. 4.4, can the authors detail on the filter strategy?\n> Once we retarget the motion from the Internet videos to humanoid trajectories, we filter the trajectories with the low optimization cost.\n\n4. How many Internet data are used? This seems not stated in the paper.", "limitations": "As the authors noted, data from human video come with the cost of being noisy as a result of our progress on the computer vision techniques. It would be helpful to elaborate on current results of using Internet videos.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This work presents a promising way to scale robotic learning in terms of data via generative modeling.\n2. Real-world experiments, and some metrics indicate the comparable performance of the proposed method to RL-based / model-based controller.", "weaknesses": "1. In Sec. 3.6 Model inference, the first step might not follow the statement.\n> At inference time, our transformer model will always have access to observation-action pairs. \n\n2. It lacks description on the inference frequency, considering the large scale pretraining with real-world locomotion task.\n\n3. It might be beneficial to ablate on how different sources of the data affect the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and suggestions! Please find our responses below:\n\n> In Sec. 3.6 Model inference, the first step might not follow the statement.\n\nThanks for pointing this out. Indeed, at the first step we have the current observation but not the action from the previous step. We use zero padding to fill in the missing actions for the initial step in both training and inference. We will clarify this in the next version of the paper.\n\n> It lacks description on the inference frequency\n\nDuring inference, our neural network model predicts desired joint positions as well as PD gains at 50 Hz. These are used as targets for a low-level PD controller that runs at 2000 Hz.\n\n> It might be beneficial to ablate on how different sources of the data affect the performance\n\nThank you for the suggestion. We perform detailed ablations below:\n\n|          |            |\n| -------- | ---------- |\n| Data     | Pred. Err. |\n| NN       | 4.17       |\n| NN+MPC   | 2.83       |\n| NN+MoCap | 2.66       |\n| NN+Video | 3.19       |\n\n|                    |            |\n| ------------------ | ---------- |\n| Data               | Pred. Err. |\n| NN                 | 4.17       |\n| NN+MPC             | 2.83       |\n| NN+MPC+MoCap       | 2.28       |\n| NN+MPC+MoCap+Video | 2.23       |\n\nWe find that each of the data sources leads to gains in performance individually (first table) and in aggregate (second table).\n\n> In Table 1(c), why no Pred. Err. for staged training?\n\nFor stage training, the model is pre-trained to predict observations in the first stage, and then fine-tuned with action prediction in the second stage. Since the prediction error counts in both observation prediction and action prediction, and the model is not supervised with observation prediction in the second stage, it is probably not fair to compare with the model that is trained jointly on observation and action prediction. For reference, the prediction error for stage pre-training is 42.22 which is considerably higher than the prediction error of joint training (0.88). \n\n> In Sec. 4.3, can the authors detail the inverse kinematics problem they solve?\n\nWe formulate an inverse kinematics optimization problem as follows:\n\n$$\n\\begin{align}\n    \\min_{\\substack{\\mathbf{q}[t], \\mathbf{\\dot{q}}[t]}} ~ & \\sum_{t=1}^{N} \\varphi^{\\text{traj}}[t] + \\varphi^{\\text{reg}}[t] \\\\\n    \\text{s.t.} ~\n    & \\mathbf{q}[t+1] = \\mathbf{q}[t] + \\frac{\\mathbf{\\dot{q}}[t+1] + \\mathbf{\\dot{q}}[t]}{2} dt, \\\\\n    & \\mathbf{q} \\in \\mathcal{Q}, \\mathbf{\\dot{q}} \\in \\mathcal{V}\n\\end{align}\n$$\n\nwhere $\\mathbf{q}$ is the robot state in the generalized coordinates, and $N$ and $dt$ are the optimization horizon and sampling time. The optimization variables include $\\mathbf{q}$, $\\mathbf{\\dot{q}}$.  For constraints, we include the Euler integration of posture $\\mathbf{q}$, constrain the range of $\\mathbf{q}$ and $\\mathbf{\\dot{q}}$ to their admissible sets $\\mathcal{Q}$ and $\\mathcal{V}$. In the cost function, $\\varphi^{\\text{traj}}$ tracks keypoint locations from human trajectories, and $\\varphi^{\\text{reg}}$ represents the regularization costs that include joint velocity minimization and smoothness.\n\n> In Sec. 4.4, can the authors detail on the filter strategy?\n\nWe empirically set a threshold on the reconstruction loss of the inverse kinematics retargeting and filter out the trajectories below the threshold.\n\n> How many Internet data are used? This seems not stated in the paper.\n\nWe use 1k human videos, which is the same number of trajectories we use for MoCap data.\n\n> It would be helpful to elaborate on current results of using Internet videos.\n\nPlease see the additional ablations reported above. We believe that this is a promising signal for using video data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qlduHDigQt", "reviewer": "Reviewer_FS4P", "summary": "This paper trained an autoregressive transformer for humanoid robot walking control using four types of data. The data sources included trajectories rolled out by RL Policy and Scripted Method, existing datasets, and human poses from YouTube videos. This work successfully enabled the humanoid robot to walk in urban environments without falling. The study explored how to train using data with missing actions, specifically by utilizing observation data and observation-action data and demonstrated the effectiveness of training with missing data. The robot in this work does not have visual capabilities.", "questions": "1. How to control the robot's walking?\n2. What exactly is Observation, and how is it transformed into R^m?\n3. From my understanding, the Neural Net Controller is the RL policy, the Model-based Controller is a scripted policy (non-learning method), and MoCap is a robot dataset, but not for the used Agility Robotics. Is my understanding correct?\n4. Is it possible to conduct ablation experiments on each type of data?\n5. What is the control frequency, what GPU and model size are used during deployment, and is the GPU placed on the robot or on a nearby PC?", "limitations": "1. One key limitation of the article is the lack of visual input, which is not discussed in the paper.\n2. Another limitation of the article is that it does not show how the real humanoid turns; it appears that the robot in the video only walks randomly.\n3. Is it compliant to use YouTube videos, especially those featuring humans as the main subject?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research problem in this work is very novel, and it has been verified on real humanoid. The effects shown in the video are excellent.\n2. This work is dedicated to applying autoregressive Transformers to humanoid locomotion, which is a promising direction.\n3. This work explores the usefulness of training with action-free data.", "weaknesses": "1. Claiming that training with video might be overclaiming; in fact, it only uses human poses from the video, which is quite different from training with video.\n2. Pre-training with action-free data is not a novel contribution.\n3. Based on the model architecture and training data, it appears that the robot can only walk randomly without controlling speed or turning. However, in Figure 5, it seems to control the robot’s walking. How is this achieved? What exactly is the observation? It doesn't seem to be the observed images. How is the observation transformed into R^m?\n4. For the four types of data collected—Neural Net Controller, Model-based Controller, MoCap, Internet Videos—the authors did not conduct ablation experiments to verify the usefulness of each type of data. Especially the MoCap and Internet Videos from inverse dynamics are highly questionable in their usefulness. Considering that the authors did not prove the usefulness of data from inverse dynamics, the paper would degrade to merely fitting some existing policies with a transformer, significantly limiting its practical value.\n5. The paper does not specify the control frequency or how the control frequency changes with model size, nor does it explain how to deploy the robot in the field. Where is the GPU located?\n6. The pre-training method used in GR1[1] is very similar to this work. The only difference is that GR1 uses two-stage training while this paper uses joint training. Both output observation-action, but this paper does not discuss this.\n\n[1] UNLEASHING LARGE-SCALE VIDEO GENERATIVE PRE-TRAINING FOR VISUAL ROBOT MANIPULATION\n\nWriting suggestions:\n1. Figure 3 is currently unclear. I suggest adding sequence numbers to the tokens to distinguish their order and using superscripts to differentiate predicted tokens from given tokens.\n2. Equations 5-10 are not very necessary and could be condensed into 1-2 lines. More space could be used to explain the training data for the transformer, which is more important.\n3. Using subplots to redraw Figures 6 and 7 would be clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions! Please find our responses below:\n\n> Claiming that training with video might be overclaiming; in fact, it only uses human poses from the video, which is quite different from training with video.\n\nWe pre-process human videos using a pre-trained transformer model to extract human poses. We will clarify in the revised manuscript.\n\n> Pre-training with action-free data is not a novel contribution.\n\nWe did not intend to claim that training with action-free data is unique to our work. Indeed, there has been prior work that studied this problem (we will include a discussion in the revised version of the manuscript). In this work, we show that a benefit of our approach is that it enables us to leverage action-free data for humanoid locomotion.\n\n> it appears that the robot can only walk randomly without controlling speed or turning. However, in Figure 5, it seems to control the robot’s walking. How is this achieved?\n\nOur controller supports omni-directional walking at varying speeds of [-1,1] m/s. We will include additional videos in the revised supplementary materials. The walking speed and direction are specified as a 3-dim vector containing the desired linear velocity on x-axis and y-axis, and the desired angular velocity around the z axis. We specify the command as a part of the observation vector. For real-world deployment, we vary the command in real time via a joystick.\n\n> What exactly is the observation? It doesn't seem to be the observed images. How is the observation transformed into R^m?\n\nThe observation is a 102-dim vector which is the concatenation of the joint positions of each joint (26-dim), the velocities of each joint (26-dim), the linear and angular velocity of the body (6-dim), the gravity (3-dim), the clock input (2-dim), the command (3-dim) and the previous action containing the commanded position of each joint (20-dim) and the updated P and D gains for the PD controller (16-dim). The robot is blind and does not take visual information as input. Observations are projected using a single linear layer into the embedding dimension.\n\nWe will include the additional details in the revised version of the manuscript.\n\n> the authors did not conduct ablation experiments to verify the usefulness of each type of data. \n\nFollowing reviewer suggestion, we perform the ablations below:\n\n|          |            |\n| -------- | ---------- |\n| Data     | Pred. Err. |\n| NN       | 4.17       |\n| NN+MPC   | 2.83       |\n| NN+MoCap | 2.66       |\n| NN+Video | 3.19       |\n\n|                    |            |\n| ------------------ | ---------- |\n| Data               | Pred. Err. |\n| NN                 | 4.17       |\n| NN+MPC             | 2.83       |\n| NN+MPC+MoCap       | 2.28       |\n| NN+MPC+MoCap+Video | 2.23       |\n\nWe see that each of the data sources leads to gains in performance individually and in aggregate.\n\n> The paper does not specify the control frequency or how the control frequency changes with model size, nor does it explain how to deploy the robot in the field. Where is the GPU located?\n\nFor all our experiments and all the model sizes we try in this work (1M, 2M, and 8M), the model is run at 50Hz during inference. The robot does not have a GPU on board and the model is run on the CPU of the on-board Intel NUC computer. In all of our outdoor experiments, everything is running on board. We use the 2M model by default for all experiments.\n\n> Related Work (GR1).\n\nThank you for the reference. While we focus on a different task (humanoid locomotion vs. manipulation) and use a different training strategy (staged training vs. joint training), there are some shared observations such as predicting observation can help action prediction. We will include the discussion in the revised version of the paper.\n\n> Writing and formatting.\n\nThanks for the suggestions. We will incorporate.\n\n> How to control the robot's walking?\n> What exactly is Observation, and how is it transformed into R^m?\n\nPlease see our comments above.\n\n> From my understanding, the Neural Net Controller is the RL policy, the Model-based Controller is a scripted policy (non-learning method), and MoCap is a robot dataset, but not for the used Agility Robotics. Is my understanding correct?\n\nNeural network controller is an RL policy from [33]. Please note that the Model-based Controller is a state-of-the art classical controller from Agility Robotics (for a dynamic task like humanoid locomotion it would be hard to script a policy). MoCap is not a robot dataset but a dataset of humans from [28, 24] (we retarget the human pose trajectories to humanoid locomotion trajectories via inverse kinematics).\n\n> One key limitation of the article is the lack of visual input, which is not discussed in the paper.\n\nWe focus on blind locomotion in this work which is a challenging problem. We intend to extend our approach to vision in future work. We will include a discussion in the revised manuscript.\n\n> Another limitation of the article is that it does not show how the real humanoid turns;\n\nOur controller supports omni-directional walking and can turn in any direction. Please see above for additional discussion. We will include videos in the updated version.\n\n> Is it compliant to use YouTube videos, especially those featuring humans as the main subject?\n\nThe videos we use are a subset of public videos from research datasets (Kinetics, PoseTrack) and we pre-process the videos to extract poses which anonymizes the individuals in the videos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0UrfnyTXWH", "reviewer": "Reviewer_qt3e", "summary": "Rather than training an RL model to learn how to walk, this paper focuses on an SSL based approach towards humanoid locomotion. Using autoregressive prediction of actions and sensor data, they pre-train and deploy zero shot to the real world. The results are strong, performing better than RL at times.", "questions": "* Did the authors do experiments with quantization/VQ? If so it would be nice to see the results for that.\n* Ablations for masked token?\n* I did not understand this comment: \"Rather than predicting the next token in a modality-agnostic way, we make predictions in a modality aligned way. Namely, for each input token we predict the next token of the same modality.\"\n* Ablations on which way to pre-train--noisy and incomplete or joint?\n* Not much information on the modalities themselves/examples modalities. If there are tons of motors/sensory information used at any given point are all of these predicted at once, with the same model? In that case, how is the causal mask constructed?\n* What is the prediction error in Table 1c?\n* Was predicting both \\hat{o}_{i+1} and \\hat{a}_{i+1} at the same time tried?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The idea of using a masked token for unknown modalities is smart and works well\n* Generalization to walking backwards is impressive\n* The approach is scalable and outperforms RL in many cases", "weaknesses": "* There was a lack of details/comprehensiveness on some of the experiments. For example, it was not clear to me how many trials were used for Figure 5. The authors also did not compare to existing MPC and only compared to RL based approaches. Limited details for hparams were given.\n* There are more ablations that could be done to further confirm design decisions (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions! Please find our responses below:\n\n**Additional details on some experiments.**\n\nOur goal in Figure 5 is to give a qualitative sense and we use one trial for each command. The following table shows a quantitative comparison between our model, RL, and MPC models. We report the mean velocity tracking error of different models under different yaw commands (250 different yaw values sampled in total).\n\n|             |           |           |           |           |           |\n| ----------- | --------- | --------- | --------- | --------- | --------- |\n| Yaw (rad/s) | 0.0       | 0.1       | 0.2       | 0.3       | 0.4       |\n| MPC         | 0.108     | 0.112     | 0.112     | 0.123     | 0.119     |\n| RL          | 0.082     | 0.083     | 0.090     | 0.105     | 0.110     |\n| Ours        | **0.070** | **0.077** | **0.078** | **0.081** | **0.089** |\n\nWe can see that MPC has the highest tracking error among the three methods while our model has the better performance across all the commands. We will update the comparisons to MPC, as well as the hyperparameters, in the next version of our paper.\n\n**Ablation on quantization.**\n\nOur model uses MSE loss on continuous observations and actions. We compare it to using cross entropy (CE) loss on quantized observations and actions. We use uniform binning for quantization, i.e., for each dimension of the observations and actions, we split the value of that dimension into N uniform bins and the observation/action prediction is formulated as a classification problem of predicting which bin the value of that dimension falls into. We use N=100 for all dimensions. The results are shown below:\n\n|                  |                       |                     |\n| ---------------- | --------------------- | ------------------- |\n|                  | Continuous (MSE loss) | Quantized (CE loss) |\n| Prediction Error | **1.39**              | 10.41               |\n\nWe can see that using MSE loss has a better performance. Note that here we only use a vanilla quantization method of uniform binning and using more advanced approaches such as non-uniform binning or vector quantization might provide better performance.\n\n**Ablation on mask token.**\n\nOur model uses mask token to replace the missing modalities in model-based, MoCap, and human video data. We compare using a leanable mask token with using a constant vector (e.g., zero vector) to fill in the missing modalities. Results are shown below:\n\n|                  |                     |                 |\n| ---------------- | ------------------- | --------------- |\n|                  | Learnable mask token | Constant vector |\n| Prediction Error | **1.39**                | 1.42            |\n\nWe can see that using mask token is slightly better than using a constant vector.\n\n**Clarification on modality-aligned prediction.**\n\nGiven a sequence of interleaved observation and action tokens [$o_1$, $a_1$, …, $o_{t-1}$, $a_{t-1}$, $o_t$, $a_t$, …], the naive next token prediction will predict $o_t$ from $a_{t-1}$ and then $a_t$ from $o_t$. For modality-aligned prediction, we predict $o_t$ from $o_{t-1}$ and $a_t$ from $a_{t-1}$, i.e., we always predict the token from the previous token that belongs to the same modality.\n\n**Ablation on noisy pre-training vs. joint training.**\n\nWe compare two ways of training our model: 1) first pre-training on noisy observation prediction on MPC, MoCap, and human video data, and then fine-tuning on action prediction on RL data. 2) training the model jointly on observation and action prediction on all data. Results are shown below:\n\n|                |                    |                |\n| -------------- | ------------------ | -------------- |\n|                | Noisy pre-training | Joint training |\n| Tracking Error | 0.311              | **0.310**      |\n\nWe can see that joint training is slightly better. The results are also reported in Table 1c in the paper.\n\n**Details on the modalities used.**\n\nThere are in total two modalities: observation and action. The observation is a 102-dim vector which is the concatenation of the joint positions of each joint/motor, the velocities of each joint, the velocity of the body, the gravity, the command, etc. The action is a 36-dim vector which contains the commands it will send to each motor for the next step. During inference, at each step the model predicts both the observation and action vector for this step. Assuming the input is the history of observations and actions [$o_1$, $a_1$, …, $o_{t-1}$, $a_{t-1}$], the model predicts $o_t$ and $a_t$ at the t-th step. The causal mask is constructed such as any modality in each step (e.g., $o_{t-1}$) will only see other modalities in the same step ($a_{t-1}$) as well as all the modalities in the previous steps (e.g., $o_{t-2}$ and $a_{t-2}$). \n\n**The prediction error for stage training (Table 1c).**\n\nFor stage training, the model is pre-trained to predict observations in the first stage, and then fine-tuned with action prediction in the second stage. Since the prediction error counts in both observation prediction and action prediction, and the model is not supervised with observation prediction in the second stage, it is probably not fair to compare with the model that is trained jointly on observation and action prediction, which is why we didn’t report the number. For reference, the prediction error for stage pretraining is 42.22 which is way higher than the prediction error of joint training (0.88). \n\n**Was predicting both observation and action at the same time tried?**\n\nOur model predicts both observation and action at the same time. Please also see the clarification on modality-aligned prediction above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aKAip9yUuw", "reviewer": "Reviewer_GH97", "summary": "This paper proposes to use next-token-prediction as a learning objective and train a casual transformer for humanoid locomotion. Compared to previous RL-based methods, the advantage of this method is to fuse data from different sources, including mocap data, videos, RL controller, and MPC controller. The authors successfully show that their model has lower tracking errors when trained with unlabeled data. The trained model is deployed on a Digit humanoid robot and shows robust locomotion across diverse real-world areas.", "questions": "See weaknesses.", "limitations": "The authors have mentioned the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper introduces the next-token-prediction framework into humanoid robots. The intuition is simple yet very interesting.\n2. The usage of different data sources, such as the RL controller and Internet videos, is novel.\n3. The real-world experiments are very solid. It is exciting to see such a framework would bring benefits on real-world robot learning.", "weaknesses": "1. As shown in Figure 8, the prediction error is correlated to the tracking error. I am curious about how much data the authors need to make these two errors correlated, since from my knowledge, the real world is much more complex and it is usually hard to have such a correlation,\n2. Recently, there have been numerous impressive advancements in humanoid robotics. While these developments are exciting, it is unfortunate that most of these works do not open-source their code. I understand the reasons behind keeping the code proprietary, but open-sourcing it would provide significant benefits to the broader community. I believe there are a lot of engineering details behind the simple framework, both in the algorithm and in the real world deployment, which however might not be shown in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions! Please find our responses below:\n\n> As shown in Figure 8, the prediction error is correlated to the tracking error. I am curious about how much data the authors need to make these two errors correlated, since from my knowledge, the real world is much more complex and it is usually hard to have such a correlation,\n\nWe use a total of 30k trajectories for training. Since we evaluate the prediction error on a held out validation set we think that the errors may be correlated in a lower data regime as well (less training data may lead to higher prediction error and higher tracking error as well). Please let us know if this answers your question.\n\n> Recently, there have been numerous impressive advancements in humanoid robotics. While these developments are exciting, it is unfortunate that most of these works do not open-source their code. I understand the reasons behind keeping the code proprietary, but open-sourcing it would provide significant benefits to the broader community. I believe there are a lot of engineering details behind the simple framework, both in the algorithm and in the real world deployment, which however might not be shown in the paper.\n\nThank you for the suggestion. We commit to open-source the code, models, and data used in this study to facilitate future research in this area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel approach to humanoid locomotion control by framing it as a next token prediction problem using a causal transformer model. The authors train their model on a diverse dataset including simulated trajectories, motion capture data, and human poses extracted from YouTube videos. Trained on this diverse collection of data, the model achieves zero-shot transfer to a real humanoid robot, enabling successful locomotion in complex urban environments.

The reviewers were generally very positive about the paper. They highlighted several key strengths: the novelty of formulating a complex humanoid locomotion problem as a next token prediction problem, the ability to leverage diverse data sources including action-free data like human videos, strong real-world results demonstrating zero-shot transfer to a physical robot, performance that outpaces reinforcement learning baselines in some scenarios, and generalization to unseen behaviors like walking backwards. These contributions represent a significant advancement for humanoids, and essentially validate the effectiveness of scaling to this locomotion problem.

However, the reviewers also identified some weaknesses. These included limited ablation studies and analysis of different data sources, lack of clarity around some experimental details and hyperparameters. The authors provided thorough responses to these concerns in their rebuttal, including additional ablation results and clarifications on model details. While there are some slight limitations in terms of analysis depth, these do not significantly detract from the overall contribution. There were also some slight concerns highlighted in the context that while posing humanoid locomotion as a next token prediction problem constitutes a very interesting application, it is objectively less complex than a domain like manipulation which is held to a higher standard, where it's expected to combine several modalities such as language, multiple sensors etc. But I do think this paper demonstrates a strong pipeline in its own way in a relatively new area, specifically through the fact that the dataset was constructed from diverse sources.

To improve the final version, the authors should expand the related work section, particularly highlighting recent works on autoregressive pre-training for robotics (which has been tackled quite a bit in recent times in varied domains), include the additional ablation results provided in the rebuttal, clarify experimental details and hyperparameters. 

In conclusion, this paper represents a good contribution to the field of humanoid robotics and is well-suited for presentation at NeurIPS. Its novel formulation of humanoid locomotion control and strong real-world results have the potential to spark new research directions in the field. The authors' commitment to open-sourcing their work further enhances its potential impact on the research community, and I would highly recommend the authors do so.
Decision: Accept (spotlight)