Humanoid Locomotion as Next Token Prediction

Authors: ilija  radosavovic
jathushan  rajasegaran
baifeng  shi
bike  zhang
sarthak  kamat
koushil  sreenath
trevor  darrell
jitendra  malik
u c berkeley
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:33:04

Abstract: We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Our model is a causal transformer trained via autoregressive prediction of sensorimotor sequences. To account for the multi-modal nature of the data, we perform prediction in a modality-aligned way, and for each input token predict the next token from the same modality. This general formulation enables us to leverage data with missing modalities, such as videos without actions. We train our model on a dataset of sequences from prior neural network policies, model-based controllers, motion capture, and YouTube videos of humans. We show that our model enables a real humanoid robot to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained on only 27 hours of walking data, and can generalize to commands not seen during training. These findings suggest a promising path toward learning challenging real-world control tasks by generative modeling of sensorimotor sequences.

Main paper: 
 introduction: 
 The last decade of artificial intelligence (AI) has shown that large neural networks trained on diverse datasets from the Internet can lead to impressive results across different settings. The core enablers of this wave of AI have been large transformer models [37] trained by generative modeling of massive quantities of language data from the Internet [26, 6, 27, 4] . By predicting the next word, these models acquire rich representations of language that can be transferred to downstream tasks [26] , perform multi-task learning [27, 25] , and learn in a few-shot manner [4] .Are such modeling techniques exclusive to language? Can we learn powerful models of sensory and motor representations in a similar fashion? Indeed, we have seen that we can learn good representations of high-dimensional visual data by autoregressive modeling [5] and related masked modeling approaches [11] . While there has been positive signal on learning sensorimotor representations in the context of manipulation [28] , this area remains largely unexplored.In this paper, we cast humanoid control as data modeling of large collections of sensorimotor trajectories. Like in language, we train a general transformer model to autoregressively predict shifted input sequences. In contrast to language, the nature of data in robotics is different. It is highdimensional and contains multiple input modalities. Different modalities include sensors, like joint encoders or inertial measurement units, as well as motor commands. These give rise to sensorimotor sequences which we view as the sentences of the physical world. Adopting this perspective suggests a simple instantiation of the language modeling framework in the robotics context. We tokenize the input sequences and train a causal transformer model to predict shifted tokens. Importantly, we predict complete input sequences, including both sensory and motor tokens. In other words, we are modeling the joint data distribution as opposed to the conditional action distribution. Figure 1 : A humanoid that walks in San Francisco. We deploy our model to the robot and test it in various locations in San Francisco over the course of one week. Please see our project page for videos. We show that our model enables the humanoid walk over different surfaces including walkways, concrete, asphalt, tiled plazas, and sanded roads. Our model can follow omnidirectional velocity commands well and enables deployment in a challenging city environment like San Francisco.Our core observation is that if a sequence is incomplete, i.e., some of the sensory or motor information is missing, we can still learn from it by predicting whatever information is present and replacing the missing tokens with learnable mask tokens. The intuition is that if the model has learned to make good predictions, even in the absence of information, it will have acquired a better model of the physical world. An important source of such data are human videos from the Internet. Namely, we can observe human movement in videos but we do not get access to the motor commands or complete sensory inputs. We demonstrate that our method can learn from such data sources effectively.To validate our method, we apply it to the challenging task of real-world humanoid locomotion using a full-sized humanoid robot Digit. We first construct a dataset of sensorimotor trajectories in simulation. These include complete trajectories from a neural network policy trained with reinforcement learning [29] , as well as incomplete trajectories from three different sources: (i) model-based controller, (ii) motion capture of humans, and (iii) videos of humans. We reconstruct human videos by using computer vision techniques and approximately retarget both the motion capture and video sequences via inverse kinematics. We then train an autoregressive transformer to model this dataset. At test time, we execute the actions autoregressively and ignore the sensory predictions.We demonstrate that our model can be deployed in the real world zero-shot and walk on different surfaces. Specifically, deploy our model across a range of different locations in San Francisco over the course of one week. Please see Figure 1 for examples and our project page for videos. To quantitatively evaluate different aspects of our approach, we perform an extensive study in simulation. We find that our autoregressive models trained from offline data alone can be comparable to the state-of-the-art approaches that use reinforcement learning [29] in tested settings. We further find that our approach can readily benefit from incomplete trajectories and has favorable scaling properties. We encourage the readers to see the arXiv for additional experiments and ablations.These findings suggest a promising path toward learning challenging real-world robot control tasks by generative modeling of large collections of sensorimotor trajectories.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification:Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Generative models. Generative models have been studied extensively, starting from Shannon's work [33] to the modern era of large language models [4] . Various such models emerged over the last decade. Examples include GAN [10] and Diffusion models [35, 14] for generating pixels, LSTM [15] and GPT [26] for generating language tokens. These models have been adopted for other modalities as well [23, 9, 38] . Among these, autoregressive transformer models became the front runner, due to the impressive scaling behaviors [17] and ability to learn from in-context examples [3] . They have been successfully extended to other modalities as well, such as vision [5] and vision-language [32] . We explore autoregressive generative models in the context of real-world humanoid locomotion.Transformers in robotics. Following the success of transformer models [37] in natural language processing [26, 6, 27, 3] and computer vision [7, 11] , over the last few years, there has been an increased interest in using transformer models in robotics. We have seen several works showing that transformers can be effective with behavior cloning. For example, [34] learns multi-task transformer policies with language, while [2] trains language-conditioned manipulation policies from largescale data. [8] trains language models with embodied data. We have also seen that transformer policies can be effective for large-scale reinforcement learning [29] as well. [28] learns sensorimotor representations with masked prediction. [1] trains goal-conditioned policies from demonstrations. Like this body of work, we share the goal of using transformer models for robotics but focus on autoregressive modeling of diverse trajectories for real-world humanoid locomotion.Humanoid locomotion. Legged locomotion has been a long-standing challenge in robotics. Humanoid locomotion is particularly challenging and has been studied extensively over the the past several decades [18, 13] . Stable locomotion behaviors have been achieved through model-based control approaches [30, 16] , while the optimization-based methods can further enable highly dynamic humanoid motions [19] . Although significant progress has been made, learning-based approaches have the potential to facilitate future progress due to their potential to improve from data and generalize to new environments. Recently, we have seen that a purely learning based approach trained with large-scale reinforcement learning in simulation can enable real-world humanoid locomotion [29] . We use the same architecture, an autoregressive transformer, but propose a different training procedure. Specifically, we train the model with sequence modeling instead of reinforcement learning. Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
approach: 
 In this section, we assume a dataset D of sensorimotor trajectories T and describe our approach.Each sensorimotor trajectory is a sequence of observations, such as joint positions, and actions: T = (o 1 , a 1 , o 2 , a 2 , ..., o T , a T ). We first tokenize the sequence into K tokens to obtain t = (t 1 , t 2 , ..., t K ).Our goal is to train a neural network to model the density function p(t) autoregressively:EQUATIONWe train our model by minimizing the negative log-likelihood over our trajectory dataset, assuming a Gaussian distribution with constant variance:EQUATIONInstead of regressing the continuous values of token elements, we could quantize each dimension into bins or perform vector quantization. However, we found the regression approach to work reasonably well in practice and opt for it in our experiments for simplicity.In the discussion so far we have assumed that each trajectory is a sequence of observations and actions. Next, we show how our framework can be generalized to sequences with missing modalities, like trajectories extracted from human videos that do not have actions. Suppose we are given a trajectory of observations without the actions T = (o 1 , o 2 , ..., o T ). Our core observation is that we can treat a trajectory without actions like a regular trajectory with actions masked. Namely, we can insert mask tokens[M] to obtain T = (o 1 , [M], o 2 , [M], ..., o T , [M]). This trajectory now has the same format as the regular trajectories and can thus be processed in a unified way. We ignore the loss for the predictions that correspond to the masked part of inputs. Note that this principle is not limited to actions and applies to any other modality as well, such as partially missing sensory observations.Rather than predicting the next token in a modality-agnostic way, we make predictions in a modalityaligned way. Namely, instead of predicting the next token in the sequence, for each input token we predict the next token of the same modality. Please see Figure 3 for diagrams. In the case of observation-action pairs being available, we train our transformer to predict the next pair of observation-action. When there is no action data available, with MoCap and internet data, we only train our transformer to predict the next observations by masking the actions with a mask token. These two models of training allow our model to utilize both types of data, and this enables us to scale our training in terms of data.We have two options for training on collections that contain diverse trajectories in terms of noise levels or modality subsets. We can either train jointly with all data at once, including complete and incomplete trajectories. Alternatively, we can first pre-train on noisy and incomplete trajectories. This can be viewed as providing a good initialization for then training on complete trajectories. And then fine-tune the model on complete data with actions. We find that both approaches work comparably in our setting and opt for joint training in the majority of the experiments for simplicity.Our model is a transformer [37] with a fairly standard architecture. Given the trajectories from either complete or incomplete data, we first tokenize the trajectories into tokens. We learn separate linear projection layers for each modality but share weights across time. To encode the temporal information we use positional embeddings. Let us assume o i ∈ R m and a i ∈ R n , then:EQUATIONEQUATIONwhere W ∈ R d×(m+n) is a linear projection layer to project concatenated observation and action modalities to d dimensional embedding vector. The superscript 0 indicates the embedding at 0-th layer, i.e., the input layer. When action is unavailable, we use a mask token [M] ∈ R n to replace a i , and [M] is initialized as a random vector and learned end-to-end with the whole model. The model takes the sequence of embedding vectors H 0 = {h 0 1 , h 0 2 , ..., h 0 t } as input. The transformer architecture contains L layers, each consisting of a multi-head self-attention module and an MLP module. Assume the output of the layer l is H l , then the layer l + 1 output is computed as follows:EQUATIONEQUATIONHere, the multi-head self-attention has causal masking, where the token only attends to itself and the past tokens. Once the tokens are processed through all the layers, we project the embedding to predicted states and actions, by learning a linear projection layer W ∈ R (m+n)×d : Then we train the transformer with the objective in (2). In the cases where the token is masked, we do not apply any losses. We train our transformer with both types of data, as shown in Figure 3 . This allows us to use various sources of data, thus enabling scaling in terms of data.EQUATION)EQUATIONEQUATIONAt inference time, our transformer model always has access to observation-action pairs. In this setting, we apply our transformer model autoregressively for each observation-action pair token. By conditioning on past observations and actions, we predict the next action (or observation-action pair) and execute the predicted action. We then record the ground-truth observation from the robot and discard the predicted observation. We use the ground-truth observation and predicted action as the next set of tokens and append them to the past tokens to predict the next observation-action pair.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [No] Justification: Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
dataset: 
 Our approach motivates building a dataset of trajectories for training our model. We construct a dataset of trajectories from four different sources: (i) a neural network policy, (ii) a model-based controller, (iii) human motion capture, and (iv) human videos from YouTube. An illustration of trajectories from different data sources is shown in Figure 4 . We describe each data source next.As the first source of training trajectories, we use a state-of-the-art neural network policy trained with large-scale reinforcement learning in simulation [29] . Specifically, this policy was trained with billions of samples from thousands of randomized environments in Isaac Gym [22] . We run this policy in the physics simulator developed by Agility Robotics and collect 10k trajectories of 10s each on flat ground, without domain randomization. Each trajectory is conditioned on a velocity command sampled from a clipped normal distribution as follows: linear velocity forward [0.0, 1.0] m/s, linear velocity sideways [-0.5, 0.5] m/s, and turning angular velocity [-0.5, 0.5] rad/s. Since we have access to the data generation policy, we are able to record complete observations as well as the exact actions that the model predicted. We use this set as our source of complete sensorimotor trajectories that have complete observations as well as ground truth actions.As the second source of trajectories, we use the model-based controller developed by Agility Robotics. It is the controller that is deployed on the Digit humanoid robot and available in the simulator provided by Agility Robotics as well. We collect 20k trajectories of walking on a flat ground of 10 seconds each, where we sample the velocity commands as follows: linear velocity forward [-1.0, 1.0] m/s, linear velocity sideways [-1.0, 1.0] m/s, and turning angular velocity [-1.0, 1.0] rad/s. The model-based controller outputs joint torques, which are not consistent with our joint position action space. Thus, we only record the observations without the actions. This data serves as a source of trajectories with accurate observations from the same robot morphology but without the actions.As the next source of trajectories, we use the motion capture (MoCap) recordings of humans from the KIT dataset [24] distributed via the AMASS repository [21] . This data was recorded using optical marker-based tracking in a laboratory setting. The dataset consists of ∼4k trajectories. We use a subset of ∼1k standing and walking trajectories. We exclude motions like dancing and jumping.In addition to not containing the ground truth actions, the MoCap trajectories come with an additional challenge: different morphology. Namely, MoCap trajectories capture human keypoint positions in 3D and over time. In order to use these trajectories for training a robot, we solve an inverse kinematics problem to approximate the corresponding robot poses. Please see the arXiv for additional details.Internet videos of people doing various activities are potentially a vast source of training data for humanoid robots. However, the raw pixels have no information about the state and actions of the human. To recover this, we first we run a computer vision tracking algorithm PHALP [31] to extract sequences of 3D human poses. This provides an estimate of the 3D joints of the human body SMPL [20] parameters and a noisy estimate of the human joints in the world coordinates.We use the human body joint positions to retarget the motion to the humanoid robot using the inverse kinematics optimization, like in the case of motion capture data discussed previously. After we retarget the motions from human videos to humanoid robot trajectories, we filter out the trajectories with high reconstruction error. Note that the scale of this data comes with the cost of being noisy. 
experiments: 
 We evaluate our approach on the challenging task of humanoid locomotion. We perform outdoor experiments on real hardware and systematic evaluations in simulation. We encourage the readers to see the extended version of this work on arXiv, which includes additional experiments and ablations.Robot platform. We use the Digit humanoid robot developed by Agility Robotics. It is a full-sized humanoid that is 1.6m tall and weighs 45 kilograms. The robot has 36 degrees of freedom including the floating base, 20 of which are actuated. Due to its high dimensionality and the four-bar linkage structure, it is challenging to simulate accurately which makes it particularly interesting for learning approaches like ours that can learn from data, without requiring an explicit model or a simulator.Model. Our model has a hidden size of 192 dimensions, with 4 layers of self-attention layers and MLP layers. Each self-attention has 4 heads. We use LayerNorm before each attention layer and ReLU activation after the MLP layer. We use a BatchNorm layer to process the input before the transformer model. When predicting a token at time k, to keep the context length at a reasonable size, we only keep the past 16 steps in input. In Section 5.6, we show the model is able to scale up to more parameters and longer context length and achieve higher performance. We train on 4 NVIDIA A100s.We begin by reporting the results from the real-world experiments. Specifically, we deploy our model to the robot and evaluate it at various locations in San Francisco over the course of one week. Please see Figure 1 for examples and project page for videos. We find that our model is able to walk over a variety of surfaces including walkways, concrete, asphalt, tiled plazas, and dirt roads. Note that the deployment in a large city environment, like San Francisco, is more challenging than in constrained laboratory environments. The city environment is crowded with pedestrians, less controlled, and less forgiving. This makes the error tolerance low and requires a model that works consistently well. We find that our model can follow yaw commands more accurately than the state-of-the-art RL policy [29] . Right: We see that our model can benefit from unlabeled trajectories without actions which is a promising signal for scaling our method to large datasets of diverse trajectories.Next, we evaluate the models in simulation under two metrics: tracking error and prediction error. Tracking error measures how accurately the robot follows a specific command. The prediction error is the next token prediction loss measured on a separate set of validation data. We introduce two metrics with details as follows and show that two metrics can consistently predict locomotion performance.Tracking error. In all experiments, the robot starts from rest in a simulated environment and is issued a constant natural walking command consisting of a desired heading velocity sampled in [0.35, 0.70] m/s, angular velocity sampled in [-0.4, 0.4] rad/s, and zero lateral velocity. We compute x * (t), the ideal robot base position trajectory that fully satisfies the velocity command v * (t) at all time steps. To measure the accuracy of command tracking, we define the position tracking error as 1 T T t=0 ∥x(t) -x * (t)∥. Each trajectory lasts for a duration of 10 seconds. For all evaluation experiments, we use the MuJoCo simulator [36] which is able to simulate the Digit robot accurately.Prediction error. Since the model is trained with the next token prediction, we evaluate the prediction error on a set of validation data that is held out from training data and contains state-action trajectories collected from the RL policy. This is similar to the language modeling evaluation for large language models [12] . We test both state and action prediction errors and add them together as the error metric.Trajectory adherence. We compare our model to a neural network controller trained with reinforcement learning (RL) [29] . Figure 5 presents a visual comparison of the trajectory adherence of our controller against these state-of-the-art baselines. Starting with a robot at the origin, we plot the actual trajectory of the robot with eleven different yaw commands selected from {0.00, ±0.05, ±0.10, ±0.20, ±0.30, ±0.40} rad/s. For each model, we jointly plot the desired and actual path traced by the robot base. Our model exhibits better tracking than the RL controller at all turning speeds, and achieves close to perfect tracking for straight-line walking.Quantitative evaluation. In Figure 6 , left, we repeat the above comparison to the RL controller (N = 245), with the full range of heading and yaw velocities mentioned in Section 5.3. We plot the mean position tracking error, binned by the commanded angular yaw. While both models have lower tracking errors at lower yaw, ours consistently outperforms the baseline RL policy. Note that our model was trained on a dataset that includes the trajectories from the same baseline RL policy.One of the benefits of our approach is that it can be applied to trajectories from diverse sources, including with missing information such as actions, like in the case of human videos. In Figure 6 , right, we compare the performance of training only with complete trajectories to joint training with both complete and incomplete trajectories. We observe that including incomplete trajectories Figure 7 : Scaling studies. We find that our approach has favorable scaling properties, and that the performance improves with the dataset size (left), context length (middle), and model size (right). consistently leads to better performance. This suggests that incomplete trajectories can still provide useful signal and is a promising signal for scaling our approach to large datasets of diverse trajectories.Training data. In Figure 7 , left, we report the performance of our model as the size of the training dataset increases. We find that training on more trajectories results in a considerably lower position tracking error. This is a promising signal for scaling our approach to larger datasets in future work.Context length. In Figure 7 , middle, we study the effect of increasing the number of tokens used in the context window of the transformer model. We observe that the larger context windows result in better performance which suggests that our model is able to benefit from the additional context.Model size. In Figure 7 , right, we compare the models of increasing size (1M, 2M, 8M) by varying the embedding dimension (144, 192, 384) , number of attention heads (3, 4, 12) , and number of transformer blocks (4, 4, 6) . We see that the error decreases monotonically with model size. We collect 14 models trained with different training recipes, model architectures, data size and types, and compute the tracking and prediction errors for each of them. In Figure 8 , we report the tracking and prediction errors of all the models in a single scatter plot. We can see that tracking and prediction error are highly correlated with Pearson coefficient r = 0.87. This suggests that the prediction error is predictive of performance, and that models with lower prediction error on the validation set are likely to follow input commands with higher accuracy. 
discussion: 
 Limitations. Our approach still lags behind state-of-the-art MPC controllers and is weaker than the state-of-the-art RL controllers in some regards, most notably robustness. In principle, we could scale our model to millions of YouTube trajectories. However, a major obstacle for doing this is the reliability of computer vision techniques which require a considerable amount of curation in practice.We explore real-world humanoid locomotion. Our model is trained on a collection of trajectories, which come from prior neural network policies, model-based controllers, human motion capture, and YouTube videos of humans. We show that our model enables a full-sized humanoid to walk in the real-world zero-shot. These findings suggest a promising path toward learning challenging real-world robot control tasks by generative modeling of large collections of trajectories.NeurIPS Paper Checklist 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This paper presents a novel approach to humanoid locomotion control by framing it as a next token prediction problem using a causal transformer model. The authors train their model on a diverse dataset including simulated trajectories, motion capture data, and human poses extracted from YouTube videos. Trained on this diverse collection of data, the model achieves zero-shot transfer to a real humanoid robot, enabling successful locomotion in complex urban environments.

The reviewers were generally very positive about the paper. They highlighted several key strengths: the novelty of formulating a complex humanoid locomotion problem as a next token prediction problem, the ability to leverage diverse data sources including action-free data like human videos, strong real-world results demonstrating zero-shot transfer to a physical robot, performance that outpaces reinforcement learning baselines in some scenarios, and generalization to unseen behaviors like walking backwards. These contributions represent a significant advancement for humanoids, and essentially validate the effectiveness of scaling to this locomotion problem.

However, the reviewers also identified some weaknesses. These included limited ablation studies and analysis of different data sources, lack of clarity around some experimental details and hyperparameters. The authors provided thorough responses to these concerns in their rebuttal, including additional ablation results and clarifications on model details. While there are some slight limitations in terms of analysis depth, these do not significantly detract from the overall contribution. There were also some slight concerns highlighted in the context that while posing humanoid locomotion as a next token prediction problem constitutes a very interesting application, it is objectively less complex than a domain like manipulation which is held to a higher standard, where it's expected to combine several modalities such as language, multiple sensors etc. But I do think this paper demonstrates a strong pipeline in its own way in a relatively new area, specifically through the fact that the dataset was constructed from diverse sources.

To improve the final version, the authors should expand the related work section, particularly highlighting recent works on autoregressive pre-training for robotics (which has been tackled quite a bit in recent times in varied domains), include the additional ablation results provided in the rebuttal, clarify experimental details and hyperparameters. 

In conclusion, this paper represents a good contribution to the field of humanoid robotics and is well-suited for presentation at NeurIPS. Its novel formulation of humanoid locomotion control and strong real-world results have the potential to spark new research directions in the field. The authors' commitment to open-sourcing their work further enhances its potential impact on the research community, and I would highly recommend the authors do so.

Reviews: review: paper_summary: paper_summary: This work view the robot locomotion control problem as an next token prediction problem. A causal transformer is trained autoregressively on various sources of data. The performance on a full-sized humanoid robot's locomotion indicates that this formulation can be a promising path for complex robotic control problems to incorporate diverse sources of data.
strength_weakness: strength_weakness: 1. In Sec. 3.6 Model inference, the first step might not follow the statement.
> At inference time, our transformer model will always have access to observation-action pairs. 

2. It lacks description on the inference frequency, considering the large scale pretraining with real-world locomotion task.

3. It might be beneficial to ablate on how different sources of the data affect the performance.
questions: questions: 1. In Table 1(c), why no  Pred. Err. for staged training?

2. In Sec. 4.3, can the authors detail the inverse kinematics problem they solve?
> In order to use these trajectories for training a robot, we solve an inverse kinematics problem to
183 find the corresponding robot poses.

3. In Sec. 4.4, can the authors detail on the filter strategy?
> Once we retarget the motion from the Internet videos to humanoid trajectories, we filter the trajectories with the low optimization cost.

4. How many Internet data are used? This seems not stated in the paper.
limitations: limitations: As the authors noted, data from human video come with the cost of being noisy as a result of our progress on the computer vision techniques. It would be helpful to elaborate on current results of using Internet videos.
paper_summary: paper_summary: This paper trained an autoregressive transformer for humanoid robot walking control using four types of data. The data sources included trajectories rolled out by RL Policy and Scripted Method, existing datasets, and human poses from YouTube videos. This work successfully enabled the humanoid robot to walk in urban environments without falling. The study explored how to train using data with missing actions, specifically by utilizing observation data and observation-action data and demonstrated the effectiveness of training with missing data. The robot in this work does not have visual capabilities.
strength_weakness: strength_weakness: 1. Claiming that training with video might be overclaiming; in fact, it only uses human poses from the video, which is quite different from training with video.
2. Pre-training with action-free data is not a novel contribution.
3. Based on the model architecture and training data, it appears that the robot can only walk randomly without controlling speed or turning. However, in Figure 5, it seems to control the robot’s walking. How is this achieved? What exactly is the observation? It doesn't seem to be the observed images. How is the observation transformed into R^m?
4. For the four types of data collected—Neural Net Controller, Model-based Controller, MoCap, Internet Videos—the authors did not conduct ablation experiments to verify the usefulness of each type of data. Especially the MoCap and Internet Videos from inverse dynamics are highly questionable in their usefulness. Considering that the authors did not prove the usefulness of data from inverse dynamics, the paper would degrade to merely fitting some existing policies with a transformer, significantly limiting its practical value.
5. The paper does not specify the control frequency or how the control frequency changes with model size, nor does it explain how to deploy the robot in the field. Where is the GPU located?
6. The pre-training method used in GR1[1] is very similar to this work. The only difference is that GR1 uses two-stage training while this paper uses joint training. Both output observation-action, but this paper does not discuss this.

[1] UNLEASHING LARGE-SCALE VIDEO GENERATIVE PRE-TRAINING FOR VISUAL ROBOT MANIPULATION

Writing suggestions:
1. Figure 3 is currently unclear. I suggest adding sequence numbers to the tokens to distinguish their order and using superscripts to differentiate predicted tokens from given tokens.
2. Equations 5-10 are not very necessary and could be condensed into 1-2 lines. More space could be used to explain the training data for the transformer, which is more important.
3. Using subplots to redraw Figures 6 and 7 would be clearer.
questions: questions: 1. How to control the robot's walking?
2. What exactly is Observation, and how is it transformed into R^m?
3. From my understanding, the Neural Net Controller is the RL policy, the Model-based Controller is a scripted policy (non-learning method), and MoCap is a robot dataset, but not for the used Agility Robotics. Is my understanding correct?
4. Is it possible to conduct ablation experiments on each type of data?
5. What is the control frequency, what GPU and model size are used during deployment, and is the GPU placed on the robot or on a nearby PC?
limitations: limitations: 1. One key limitation of the article is the lack of visual input, which is not discussed in the paper.
2. Another limitation of the article is that it does not show how the real humanoid turns; it appears that the robot in the video only walks randomly.
3. Is it compliant to use YouTube videos, especially those featuring humans as the main subject?
paper_summary: paper_summary: Rather than training an RL model to learn how to walk, this paper focuses on an SSL based approach towards humanoid locomotion. Using autoregressive prediction of actions and sensor data, they pre-train and deploy zero shot to the real world. The results are strong, performing better than RL at times.
strength_weakness: strength_weakness: * There was a lack of details/comprehensiveness on some of the experiments. For example, it was not clear to me how many trials were used for Figure 5. The authors also did not compare to existing MPC and only compared to RL based approaches. Limited details for hparams were given.
* There are more ablations that could be done to further confirm design decisions (see questions).
questions: questions: * Did the authors do experiments with quantization/VQ? If so it would be nice to see the results for that.
* Ablations for masked token?
* I did not understand this comment: "Rather than predicting the next token in a modality-agnostic way, we make predictions in a modality aligned way. Namely, for each input token we predict the next token of the same modality."
* Ablations on which way to pre-train--noisy and incomplete or joint?
* Not much information on the modalities themselves/examples modalities. If there are tons of motors/sensory information used at any given point are all of these predicted at once, with the same model? In that case, how is the causal mask constructed?
* What is the prediction error in Table 1c?
* Was predicting both \hat{o}_{i+1} and \hat{a}_{i+1} at the same time tried?
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes to use next-token-prediction as a learning objective and train a casual transformer for humanoid locomotion. Compared to previous RL-based methods, the advantage of this method is to fuse data from different sources, including mocap data, videos, RL controller, and MPC controller. The authors successfully show that their model has lower tracking errors when trained with unlabeled data. The trained model is deployed on a Digit humanoid robot and shows robust locomotion across diverse real-world areas.
strength_weakness: strength_weakness: 1. As shown in Figure 8, the prediction error is correlated to the tracking error. I am curious about how much data the authors need to make these two errors correlated, since from my knowledge, the real world is much more complex and it is usually hard to have such a correlation,
2. Recently, there have been numerous impressive advancements in humanoid robotics. While these developments are exciting, it is unfortunate that most of these works do not open-source their code. I understand the reasons behind keeping the code proprietary, but open-sourcing it would provide significant benefits to the broader community. I believe there are a lot of engineering details behind the simple framework, both in the algorithm and in the real world deployment, which however might not be shown in the paper.
questions: questions: See weaknesses.
limitations: limitations: The authors have mentioned the limitations.

Comments: title: Thank you
Thank you
Response by Authors
Response by Authors
Thank you for the rebuttal.
Kind reminder for feedback - 1 day left for discussion
Response by Authors
Author response summary
Kind reminder for feedback - 2 days left for discussion
Added videos of the robot turning to the project page
Response by Authors
Response by Authors
Thank you
Thank you for the reply
Response by Authors
comment: I believe the score I have given is appropriate. This paper represents yet another success of Next Token Prediction in a different domain, a success that has been replicated many times. There are few new techniques introduced. Essentially, papers in this paradigm primarily involve preparing data, training, and testing, without much technical complexity. I also took into account that this paper does not achieve significant breakthroughs; previous methods could also achieve tasks such as walking and turning, just not using next token prediction.

A notable drawback is that the scenarios considered are particularly simple, with only 100 dimensions of the robot's body state being considered as observations. The challenge might lie in processing and unifying large amounts of data. The primary reason this paper achieves such a high score maybe is that it effectively addresses the problem of studying humanoid locomotion using AI methods and works well in the real world. If the same approach were applied to robotic manipulation, it might not score as highly. However, the manipulation problem involves more complex considerations than humanoid locomotion, such as language instructions, visual perception, and the robot's state. 
Despite some limitations, I still believe this work is valuable because it demonstrates that next token prediction works well in such high degrees of freedom, given the appropriate data.
Considering the author's commitment to open-sourcing their work, which could significantly advance the progress of humanoid locomotion—a field that has been relatively unexplored—I believe a score of 7 is reasonable.
Thank you very much for your response and review. We are glad to hear that your concerns have been addressed.
Thank you very much for your response. After reading the author's response, my concerns have been addressed, and I believe that there are no ethical issues with this paper.
Thank you for your response and for increasing the score. We think that the challenge of balancing model size and inference speed in real-time settings like ours is very interesting. Please find some comments below.

The minimal frequency for functioning is determined by the highly dynamic nature of the problem (unstable bipedal robot with a big upper body with a lot of mass and inertia). We might be able to reduce it a bit (e.g., 40Hz) but probably not much. Our current models are relatively small (up to 8M parameters) and can run within 50Hz on the on-board CPU computer.

We think that it is likely that the improvements in inference software (e.g., low precision inference), on-board compute (e.g., access to a GPU), training recipes (e.g., distillation), and modeling (e.g., sparse architectures) will offset these challenges and enable us to keep using larger models within the inference speed constraints.

We find the modeling angle to be a particularly interesting direction for future work. For example, one could imagine architectures with more parameters than flops, where different parameters are activated at different frequencies and different parts of the model are executed asynchronously.

We will include the additional discussion and an analysis of the model size and inference speed in the revised manuscript. Thank you for the suggestions.
Thank you for your response. We are glad to hear that we have addressed your remaining concerns regarding robot capabilities and appreciate your acknowledgement of our results.

We are more than willing to include a more detailed autoregressive models for robotics section. We will incorporate the suggestion in the revised manuscript.

We believe that we have addressed all of your concerns from the initial review and subsequent discussions. Given these and in light of the strengths you and all other reviewers have highlighted, we respectfully ask you to consider increasing the overall score to better reflect our contributions.
Thank you for your response. We appreciate your acknowledgement of our contributions and results. We would like to clarify a few points below:

- Autoregressive pre-training is not a primary contribution of our work. While our approach can be used with pre-training, our method does not involve pre-training. We are "just" training a model from scratch and deploying it zero shot, without multi-stage training or fine-tuning.

- We are more than willing to expand the “Transformers for Robotics” section with work on autoregressive pre-training in robotics to provide a more comprehensive context.

- Our primary contribution lies in formulating real-world humanoid locomotion as a joint distribution learning problem, enabling training with noisy or missing data and achieving very strong results in a particularly challenging domain.

Thank you for your consideration.
Thank you for your response. I appreciate your excellent experiments; it's fascinating to see how they can be applied to real-world robotics.

I agree with qt3e's point—a more detailed autoregressive model for the robotics section would better reflect the contributions of this paper.
Thank authors for the detailed response. My questions are mostly addressed. My concern left is that will the largest model work at 50Hz, i.e. whether the increased model size with generative pretraining, hinders the inference. It would be great to discuss the minimum required frequency for functioning, and its relation to model size.

I have raised the score according to the detailed open-source commitment of authors.
I appreciate the response. Perhaps, this was a mistake on my end, as while reading the paper I believed that the idea of autoregressive pre-training for robotics was a primary contribution. Adding to this, I think that the lack of inclusion of autoregressive pre-training approaches in the "Transformer for Robotics" section is of slight concern, as *autoregressive pre-training approaches for robotics are the most similar to this work.* (I say lack of inclusion because all approaches listed were either not pre-training or not autoregressive as far as I could tell). Together, these enhanced my initial beliefs regarding the merit of this work.

I acknowledge the contributions discussed and the excellent results. Hence, I think a score of 8 is fitting. This updated score is based off of the planned inclusion of more autoregressive pre-training approaches in the related works.
Thank you again for your time and effort spent on providing the ethics review.

We clarified that we use videos from standard academic datasets, we are not distributing videos, our training data is derived from abstract geometric points that preserve privacy, and one cannot infer original videos from our models.

Please let us know if there are any specific questions or points that require further clarification. We would be happy to address them.
Thank you for your response. We are glad to hear that we addressed all of your initial comments. We would like to offer some clarifications regarding the related work discussion.

We believe that reviewer FS4P’s suggestion about an additional reference in robotic manipulation was a minor point, which we agreed to include. Reviewer FS4P acknowledged that this concern was fully addressed in their response and also noted the novelty of our work in their review.

We want to emphasize that we do not claim "Autoregressive Pre-training for Robotics" as our primary contribution. In fact, we already have the "Transformers for Robotics" section in the related work that discusses a number of papers on autoregressive pre-training in robotics. We are happy to expand this section further.

The novelty of our work lies in casting real-world humanoid locomotion as a joint distribution learning problem with transformers. We further show that this formulation enables training with noisy or missing data (e.g., derived from videos) and achieves excellent results in the real world.

While our work can have excellent impact on multiple areas, the paper and title focus on humanoid locomotion. We chose this problem as a test bed because it is particularly challenging due to its highly dynamic, complex nature. The success of our method in this domain is particularly noteworthy (few prior approaches used learning and the ones that did rely on RL).

Given these clarifications and the original review, we believe the original score more accurately reflects our contribution and that it would be unfair to reduce it. We respectfully ask you to reconsider maintaining your initial score.
Thank you for addressing my comments, I am happy with the author’s rebuttal.

However, the lack of comprehensiveness in discussing related works is slightly concerning. I was not knowledgeable about the work described by reviewer FS4P, which makes the contribution of this work less meritous. I believe the addition of an “Autoregressive pre-training for robotics” header in the related works section would be appropriate. Thus, I am lowering my score slightly. 

It’s worth noting that I still believe the work has a lot of merit, such as through masking actions and through applying autoregressive pre-training to humanoid locomotion. The results were also very impressive. I also agree with other reviewers that the eventual open source release is essential for this work. Thanks for addressing my concerns!
We appreciate your positive review and helpful suggestions. We wanted to provide a summary of the additional information provided in our response for your reference.

Additional experiments on:
- MPC comparisons - ours > RL > MPC
- Quantization ablation - continuous > discrete
- Mask token ablation - learnable > constant
- Pre-training ablation - joint > staged

Additional details on:
- Modality prediction - predict next token of the same modality
- Modalities used - observations and actions
- Prediction error - provided for reference
- Simultaneous prediction - used in our models

Thank you again for your valuable feedback. We would be happy to provide any further information if needed.
Thank you again for your time and effort spent on providing a careful review of our paper.

We performed new experiments on how different data sources affect performance. We find that (1) each of the data sources improves the performance individually and (2) that different data sources are complementary and lead to greater gains in aggregate. These detailed ablations demonstrate that our approach can benefit from different data sources.

We also provided additional details on:
- model inference
- inference frequency
- prediction errors
- inverse kinematics
- filtering strategy
- number of videos

To ensure we have fully addressed all of your concerns, please let us know if there are any specific questions or points that you feel remain unresolved or require further clarification. We would be happy to address them. 

Many thanks
We have resolved the hosting issue with GitHub and our anonymous project page is accessible again via the link provided in the original manuscript.

As requested, we have included videos of the robot turning in the real world under the “Turning in the Real World” section at the bottom of the page.

Please let us know if all your concerns are now fully addressed. We would be happy to provide any additional information.
Thank you for providing the review. We are committed to maintaining the highest standards of ethics. Please find our responses below:

- The videos that we use are a subset of two standard datasets in machine learning, Kinetics [1] and PoseTrack [2]. These datasets are in major use in the literature and have collectively been cited over 16,000 times. We have followed the standard practice in the research community and have not scraped any videos from YouTube.

- We will not distribute or make any of these videos available (we did not source, collect, or scrape these videos).

- We have extracted coarse human poses from these videos for deriving data for training our models. Specifically, we extract the geometric locations of 4 points (locations of the feet and the hands) and use them to derive robot poses for training. We would only provide the robot poses from which one cannot recover the original videos and which preserves the privacy of the individuals in the original videos.

- Our models are trained on robot poses that are derived from abstracted geometric points derived from videos. Therefore our models never see real videos and one would not be able to reverse-engineer the original videos from our models.

- We will include clarifications on the nature of our data, pre-processing, and model inputs, and include an extended discussion on ethics in the revised manuscript.

Please let us know if this addresses your concerns and/or if you have any additional questions. We would be happy to answer them.

---

[1] Carreira and Zisserman, Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, CVPR 2017

[2] Andriluka et al, PoseTrack: A Benchmark for Human Pose Estimation and Tracking, CVPR 2018
Thank you for your response and for raising the score. Please find our responses below:

> The project page doesn't include videos of the robot turning; can it actually turn in the real world or walk for an extended period of time? However, the project page seems to be inaccessible now. It would be interesting to see its obstacle avoidance capabilities under control.

Our policy can turn in all directions and walk for extended periods of time, up to 2 hours on a single battery charge. We have extensively tested these capabilities during a week-long deployment in San Francisco. Thanks to the responsive and accurate command following, our robot can avoid obstacles and navigate busy city streets with pedestrians.

As an additional data point, our omnidirectional walking capabilities are comparable to the state-of-the-art RL approach in Figure 3 of [33], with improved command following as demonstrated in Figures 5 and Figure 6 (left) of our manuscript.

We apologize for the inaccessible project page. It seems that our GitHub-hosted anonymous website was suspended by GitHub, which we are working to resolve with GitHub. Per the rebuttal instructions, it seems that we are not allowed to provide a new link to external pages in the rebuttal. We welcome any suggestions on how we might share the videos demonstrating these capabilities within the constraints of the rebuttal process and commit to include them in the release.

> Additionally, if the robot needs to go uphill or downhill slightly, would it fail? And it will be fine to analyze its robustness.

Our policy can handle gentle slopes, and we have tested it in the real world on inclines of up to 8.7% (5 degrees). For example, Figure 1 (row 3, column 1) shows the robot walking up a 7% slope in San Francisco.

Regarding robustness, our approach trained purely on offline trajectories without any domain randomization has shown surprising effectiveness in walking in the real world. However, as we acknowledge in the limitations section, our policies may be less robust to sudden, large disturbances compared to RL policies.

In follow-up experiments, we found that pre-training policies with our proposed framework and fine-tuning them with a small amount of RL leads to excellent results, enabling rapid acquisition of new capabilities like walking on steeper slopes and robustness to large disturbances.

> If this method were extended to enable humanoid robots to go up and down stairs, what challenges would arise?

To extend our method to enable the robot to go up and down stairs, we would need to incorporate vision to guide the foot placement. Our method can be readily extended to achieve this by incorporating images (potentially pre-processed using a vision encoder) as part of observations in addition to proprioception.

One challenge would be in collecting training trajectories that contain visual inputs. This would be relatively straightforward for trajectories collected using the same robot body (e.g., via prior RL or MPC controllers). To incorporate human trajectories from mocap or videos, we would need to include ego vision inputs where available (e.g., first-person videos) and use our strategy for training with missing data for trajectories without ego vision (e.g., third-person videos).

> Some of the suggestions may go beyond the scope of this paper

Thank you for raising these questions. We will include the discussion of capabilities, limitations, and future work in the revised version of the manuscript with an extended discussion section.
Thank you for your response and for increasing the score. Please let us know if you have any further questions. We would be happy to address them.
Thank you for your response.
My remaining concerns mainly focus on the actual capabilities of this robot:
The project page doesn't include videos of the robot turning; can it actually turn in the real world or walk for an extended period of time?
However, the project page seems to be inaccessible now. It would be interesting to see its obstacle avoidance capabilities under control.
Additionally, if the robot needs to go uphill or downhill slightly, would it fail? And it will be fine to analyze its robustness.
If this method were extended to enable humanoid robots to go up and down stairs, what challenges would arise?

Some of the suggestions may go beyond the scope of this paper, but considering the authors’ commitment to open-source, I have raised my score.
Thank authors for the reply. My questions are mostly addressed.

I have raised the score according to the open-source commitment of authors.
Thank you for your response. Please find our responses below:

> I appreciate the effort in addressing my questions, and most of them have been satisfactorily resolved

We are glad to hear you found our responses satisfactory. To ensure we have fully addressed all of your concerns, please let us know if there are any specific questions or points that you feel remain unresolved or require further clarification. We would be happy to address them.

> However, I have some concerns regarding the reproducibility of the paper. It seems challenging to obtain the four datasets mentioned in the paper based solely on the descriptions provided. While I believe the paper offers valuable insights, the limited details, absence of detailed information in the appendix, and lack of accompanying code might affect its overall impact on the community.

We commit to release all of the materials to fully reproduce the paper, including the four datasets, accompanying code, and detailed documentation. We will further include detailed descriptions of the datasets in the appendix, covering data collection methods, pre-processing steps, and any other relevant details. Please let us know if there is any additional information that you found lacking, and we would be happy to include it in the revised manuscript.
Thank you very much for your response. I appreciate the effort in addressing my questions, and most of them have been satisfactorily resolved. However, I have some concerns regarding the reproducibility of the paper. It seems challenging to obtain the four datasets mentioned in the paper based solely on the descriptions provided. While I believe the paper offers valuable insights, the limited details, absence of detailed information in the appendix, and lack of accompanying code might affect its overall impact on the community. Considering these factors, I believe it may be best to maintain my previous score. Thank you once again for your thoughtful response.