EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
Authors: Ryan Hoque
Peide Huang
David J. Yoon
Mouli sivapurapu
Jian Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: egocentric video
manipulation
embodied ai
robotics
Primary area: datasets and benchmarks
TL;DR: We present EgoDex, a large-scale dataset for learning dexterous human manipulation.

Abstract: Imitation learning for manipulation has a well-known data scarcity problem. Unlike natural language and 2D computer vision, there is no Internet-scale corpus of data for dexterous manipulation. One appealing option is egocentric human video, a passively scalable data source. However, existing large-scale datasets such as Ego4D do not have native hand pose annotations and do not focus on object manipulation. To this end, we use Apple Vision Pro to collect EgoDex: the largest and most diverse dataset of dexterous human manipulation to date. EgoDex has 829 hours of egocentric video with paired 3D hand and finger tracking data collected at the time of recording, where multiple calibrated cameras and on-device SLAM can be used to precisely track the pose of every joint of each hand. The dataset covers a wide range of diverse manipulation behaviors with everyday household objects in 194 different tabletop tasks ranging from tying shoelaces to folding laundry. Furthermore, we train and systematically evaluate imitation learning policies for hand trajectory prediction on the dataset, introducing metrics and benchmarks for measuring progress in this increasingly important area. By releasing this large-scale dataset, we hope to push the frontier of robotics, computer vision, and foundation models. EgoDex is publicly available for download at https://github.com/apple/ml-egodex.

Published as a conference paper at ICLR 2026
EGODEX: LEARNING DEXTEROUS MANIPULATION FROM LARGE-SCALE EGOCENTRIC VIDEO
Ryan Hoque", Peide Huang", David J. Yoon', Mouli Sivapurapu, Jian Zhang
Apple
EgoDex
3D Skeleton Annotation
300K Episodes
90M Frames
Caman pota
Upper bosy
800 Hours
30 FPS
1080p
500 Objects
Language Annotation
Figure 1: EgoDex is a large- scale egocentric dataset that focuses human dexterous manipulation.
ABSTRACT
Imitation learning for manipulation has a well-known data scarcity problem. Un- like natural language and 2D computer vision, there is no Internet-scale corpus of data for dexterous manipulation. One appealing option is egocentric human video, a passively scalable data source. However, existing large-scale datasets such Ego4D) do not have native hand pose annotations and do not focus on object manipulation. To this end, we use Apple Vision Pro collect EgoDex: the largest and most diverse dataset of dexterous human manipulation to date. EgoDex has 829 hours egocentric video with paired 3D hand and finger tracking data collected at the time of recording, where multiple calibrated cameras and device SLAM can be used to precisely track the pose of every joint of each hand. The dataset covers wide range of diverse manipulation behaviors with everyday household objects in 194 different tabletop tasks ranging from tying shoelaces folding laundry. Furthermore, we train and systematically evaluate imitation learning policies for hand trajectory prediction on the dataset, introducing metrics and benchmarks for measuring progress in this increasingly important area. By releasing this aarge-scale dataset, we hope to push the frontier of robotics, com- puter vision, and foundation models. EgoDex publicly available for download at htttt:://// / htw.bttttbttttt/////// hub. com/ apple/ -egodex
"qqual contribution.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
The "bitter lesson" Sutton 2019) of recent breakthroughs in large language models and large vision models is that the simple recipe of supervised learning with vast amounts of data far more effective than competing approaches. Two key challenges have prevented the application of the bitter lesson to the longstanding challenge of autonomous robot manipulation: (1) it is unclear what data should be collected, and (2) it is unclear how such data can be collected at the requisite scale.
The leading approach to data collection for robot imitation learning is telcoperation, in which human operators provide demonstrations by directly controlling robot hardware. Recent works such as Open X-Embodiment (Padalkar et al.] 2024) and DROID (Khazatsky et al., (024) pio- neer community-wide efforts pool together hundreds of hours of teleoperation data. While such datasets can be used for pretraining robot control policies, teleoperation bottlenecked by physi- cal robot operation, and it is unclear how continue scaling this paradigm beyond its current size. Other works explore learning visual representations from existing in-the-wild Internet videos and images (Radosavovic et 2023 Ma al., 2023). In this case, while large-scale data is available, unstructured video data lacks the precise annotation necessary to learn dexterous manipulation.
We explore middle path between the two: egocentric human video with paired 3D hand pose annotations. As suggested by recent work (Kareer et al. [024]] Qiu et al. 2025), such an approach passively scalable, similar to text and images on the Internet. Effectively learning from such data is critical in future where wearable headsets and smart glasses may be omnipresent. Data is a crucial component for doing so; before AlexNet (Krizhevsky et al. 2012) must come ImageNet Russakovsky et al. 2015
To this end, we introduce EgoDex: a large-scale dataset and benchmark for learning dexterous manipulation from large-scale egocentric video. EgoDex consists of 829 hours of 30 FPS video and paired skeletal data with a total of 90 million frames and 338000 task demonstrations across 194 tabletop manipulation tasks. To knowledge, the EgoDex dataset is the largest and most diverse dataset dexterous human manipulation to date.
There are several key properties of the proposed data that make it more suitable for dexterous ma- nipulation than existing alternatives:
EgoDex is passively scalable, unlike robot teleoperation and other approaches that require de- liberate effort for data collection. EgoDex suggests the human hand as a common embodiment, unlike teleoperation and other approaches that collect data that is only compatible with specific robot hardware platforms.
EgoDex has 30 FPS 1080p egocentric video with a wide field view, capturing much of what human sees while manipulating objects. It has precise and highly detailed 3D pose informa- tion for the user's head, arms, wrists, and each joint of each finger from on-device SLAM and calibrated cameras, containing critical dexterous manipulation data unlike in-the-wild Internet videos and Ego4D (Grauman et al. 2022).
EgoDex consists of extremely diverse behaviors beyond simple pick-and-place such unscrew- ing bottle eap, flipping pages of book, and plugging a charger into a socket. It consists entirely of active manipulation, unlike existing large egocentric video datasets such Ego4D.
We systematically evaluate imitation learning policies for hand trajectory prediction to assess the state of the art and identify challenges for future research. We hope that EgoDex will not only accelerate progress robot manipulation but also be useful more broadly in applications such as computer vision, video generation, and world modeling.
RELATED WORK
2.1
LARGE-SCALE MANIPULATION DATASETS
Several prior works introduce large-scale open-source robot teleoperation datasets, including Robo- Turk (Mandlekar et al., 2019), BridgeData (Walke et al. 2023), RT-X (Padalkar et al.) (024), and DROID (Khazatsky et al. 2024). While such datasets contain up hundreds of hours of valuable
2

---

Published as a conference paper at ICLR 2026
Dutuset
# Traj.
Tasks
# Frames
Lang. Annot.
Cum. Ext.
Dexterous Annot.
Collection Method
Turls Mandlekar et al.
2019
2k
3
12M
X
teleoperation
RoboNet Dasan et al. 2019
162k
n/a
15M
X
scripted
BridgeData V2 Walke di al
2023
60k
13
2M
x
X
lelouptscripied
DROID Khazarsky et ml.
21744
76k
86
19M
X
teleoperation
EgoMiiic Karoer ml.
2024
2k
3
0.4M
X
X
egucentric vidoo
EPIC-KITCHENS (Damen et
2018
40k
125
12M
X
egocentric video
HOID Liu et all 20221
4k
54
2M
egocentric video
Ego4D (TOD) (Gruman et al. 2022
89k
n/a
21M
X egocentric video
EgoDex (ours)
338k
194
90M
egocentric video
Table I: Comparison of different robot manipulation datasets (above the middle line) and human manipulation datasets (below the middle line). Ego4D (HOI) considers the subset of Ego4D that involves hand-object interaction. EgoDex has the largest amount trajectories, tasks, and frames by large margin and has language annotation, camera extrinsics, and dexterous annotation. "Dex- terous annotation" " is defined here as labels for multi-finger hand poses, which do not include lower fidelity pose data like parallel jaw robot grippers or wrist-only tracking.
manipulation data, it is not clear how scale the paradigm further than its present scale. Robot telcoperation is extremely labor-intensive and cesource-constraneee requiring an operational phys- ical robot and a human teleoperator actively controlling the robot perform each desired task. Furthermore, it is not clear to what degree such datasets can generalize beyond the set of hardware embodiments and camera viewpoints with which they were collected, even when the datasets consist of samples collected across multiple different embodiments.
Other large-scale datasets such as Ego4D (Grauman et al., 2022) and EPIC-KITCHENS (Damen et al. 2018) consist of egocentric video recording humans performing various activities. While such datasets are more scalable and not limited to particular hardware platforms, they typically do not focus on manipulation and do not have paired 3D annotations for dexterous manipulation.
There is also large body of work that considers hand-object interaction (Liu et al. 2022 Banerjee al. 2025 Chao et al. 2021] Zhou et al.). While these datasets often include 3D hand pose annotations, they are orders magnitude smaller than EgoDex due to manual annotation. Moreover, their emphasis is primarily on grasping rather than diverse and long- horizon manipulation tasks.
2.2 SCALABLE METHODS FOR ROBOT DATA COLLECTION
Recent work identifies the data scarcity problem in robot imitation learning and proposes innovative techniques for scalable data collection. CCi et al. (2024) propose the "universal manipulation in- terface": handheld grippers that enable human teachers provide demonstrations without physical robots. Wang et al. (2024) introduces a portable data collection system with motion capture gloves. Others propose collecting robot-free demonstrations by simulating robot hardware in augmented reality CChnn et al. 2024b Park et al. 2024 Nechyporenko et al. 2024).
These approaches all face similar pitfall: they require active data collection. While they may make it easier to collect data than teleoperation, human demonstrators must still be incentivized to intentionally collect the data. Such approaches face significant uphill battle in approaching the scale of Internet datasets, where text and images are not deliberately collected but rather passive byproduct of human interaction with the Internet.
2.3 LEARNING FROM HUMAN VIDEO
Video data is abundant on the Internet. Prior work explores representation learning on unstructured large-scale image and video data for pretraining visual encoders (Radosavovic et al., 2023 Ma ct al.) 2023) and grasp affordances (Bahl et al. 2023) for downstream manipulation. However, raw unstructured video data faces prohibitively large gap between its image distribution and that of a dexterous manipulation task. Moreover, such videos are not labeled with corresponding motor actions with which to train policy.
One option is to postprocess the unstructured video data with 3D hand prediction networks such HaMeR (Pavlakos et al. 2024), recently explored by Ren al. (025). However, the prediction
3

---

Published as a conference paper at ICLR 2026
Ego
DROID
Distribution of distinct verbs, sorted by frequency. The horizontal axis is verbs of EgoDex. The orange plot is taken from DROID (Khazatsky et al.) 2024]. While many verbs in DROID are below the 10 mark, most verbs in EgoDex are above the 10" mark. Bottom: Distribution of distinct objects. The clustering is suggested by GPT-4.
Figure 2: Distribution of EgoDex dataset. Top:
quality of these networks can suffer without multiple viewpoints and detailed knowledge of the camera extrinsics at all times, usually unavailable with raw Internet video. In contrast, the EgoDex dataset includes 3D head and hand tracking at the time of collection, where multiple cameras on the Vision Pro, known intrinsics and extrinsics, and production-gren grade hand prediction network all contribute precise annotation.
Most similar to our work is EgoMimic (Kareer al. 2024), which proposes the collection of ego- centric video and paired 3D hand tracking. The primary difference is scale: while EgoMimic collects around 4 hours of data, we collect 829 hours with a much broader data and task distribution. We also collect more dexterous annotations, critical for downstream manipulation: 3D positions and orien- tations for the upper body including the head, shoulders, arms, and 25 joints in each hand, whereas EgoMimic collects only the wrist positions.
3 EGODEX DATASET
The EgoDex dataset contains 829 hours of 1080p. 30 Hz egocentric video with 338000 episodes across 194 tasks. This is a total of 90 million frames. The dataset also has rich and structured annotations including natural language, camera extrinsics, and hand pose annotations (Section (32)) The full dataset takes 2.0 TB of storage on disk. We compare EgoDex to existing manipulation datasets in Table []
3.1 DATA COLLECTION
All data is collected with Apple Vision Pro running visionOS 2. The high-resolution and high- frequency passthrough and wide field of view enable intuitive egocentric data collection, where the collector can observe the environment unobstructed as if with their own eyes, and the camera data records precisely what the collector sees without any pose offsets (unlike, for instance, head- mounted camera). We use ARKit, a production- -gredd pose tracking software, to collect natural demonstrations with bare hands and without any additional hardware apparatus.
To streamline data collection, data is recorded in sessions: approximately 10-15 minute segments that consist of many individual episodes, where episode boundaries indicated by "pause" and subsequent "resume" of recording from the data collection app. Raw video is compressed to facil- itate data transfer, upload, download, and storage. Without the use of modern video compression
4

---

Published as a conference paper at ICLR 2026
righthommm
JefFoream
Figure 3: Left: Joints captured by EgoDex. Right: Examples of dexterous manipulation behaviors. Tracked fingertips are highlighted in distinct colors and show 0.5 seconds of motion before the current frame. From left to right, top to bottom, the tasks are: unzipping Ziploc bag, removing a from fixture, folding t-shirt, decluttering, opening
book from bookshelf, removing case, unscrewing a bottle cap, tying shoelaces, and washing a cup.
algorithms, the data would take over 500 TB of disk space, about 250x its current size. At training time, data is loaded efficiently with Py PyTorch torchcodec (Team, 2024), which only decodes the desired frames in the sampled batch of data.
3.2 MODALITIES
The data consists of the following: 1) Egocentric RGB video with 1920 1080 resolution at 30 Hz frequency. 2) Camera intrinsics and extrinsics at Hz. 3) 3D position and orientation of all upper body joints (including 25 joints for each hand) at 30 Hz. 4) Confidence values for pose predictions at 30 Hz. 5) Natural language annotation of the manipulation.
The metadata annotated by data collectors includes the task name, brief task description in natural language, details about the enviroument, and details about the object(s) that are manipulated. Since the metadata can be noisy, these fields provided as input to GPT-4 (OpenAl al. (2024), which combines this information into a single detailed natural language description.
Confidence values scalars between 0 and indicating the ARKit prediction confidence per skele- tal joint. A confidence of zero indicates that the joint is fully occluded from view. See Appendix A.3 A.. for comprehensive list of all the joints and more information.
3.3 TASK TYPES
EgoDex consists of 3 types of tasks:
Reversible tasks are pairs of tasks that are the inverse of each other. The distribution of final states for one task within the distribution initial states for its inverse. For example, connecting charger to device and removing charger from the device.
Reset-free tasks are tasks with a final state distribution that falls within its own initial state distri- bution. For example, throwing ball in the air and catching it (where gravity acts the reset).
Resel tasks are tasks in which the environment must be reset to the initial state distribution after each demonstration.
Reversible and reset-free tasks enable higher yield from data collection as they eliminate costly resets, which are not included in the recorded data.
3.4 DIVERSITY
Prior works (Khazatsky et al., 2024 Padalkar et al., (024) identify several potential axes demon- stration diversity: viewpoint diversity, task diversity, scene diversity, object diversity, and more. In
5

---

Published as a conference paper at ICLR 2026
EgoDex, the emphasis is on diversity in dexterous manipulation behaviors. Tasks and objects vary such that the required dexterity ranges far beyond pick-and-place, the primary behavior in most robot teleoperation datasets. For example, tasks include tightening screw, tying shoelaces, dealing cards, flipping pages, catching tennis balls, and slotting hatteries. The task distribution covers a wide range of everyday household manipulation tasks that be performed on tabletop surface. There is also a significant amount of basic pick-and-place with diverse objects, as well as the benchmark tasks from the FurnitureBench assembly benchmark (Heo al. 2023). The full list of 194 tasks is provided in Appendix A.2
To get sense of the spread of the task distribution as in prior work, we plot the distribution of de- duplicated verbs in Figure 2 We observe that the distribution is much wider than prior works such as DROID (Khazatsky et al. 2024), where large fraction of verbs have less than 10 demonstrations and sometimes only single demonstration; in contrast, most of the verbs in EgoDex have more than 10 demonstrations.
Still, the verb distribution does not capture the full diversity of manipulation behaviors or tasks. For example, "assemble" can involve radically different behaviors in the context different objects and tasks. See Figure 3 for examples of different dexterous manipulation behaviors captured in the dataset.
While the scene diversity in EgoDex is limited to tabletop environments, the Cartesian product of scene and behavior is not the focus of our work, which focuses on behavioral diversity. Scene diversity be introduced with modern visual data augmentation methods such as image-to-image generative models (Yuan et al. 2025 Chen et al. 2024a).
EGODEX BENCHMARKS
4.1 ACTION REPRESENTATION
Given the full set of skeletal joints in the EgoDex dataset, many action representations are possible: wrist positions, wrist orientation, positions of fingertips, and so on. Since we focus on dexterous manipulation, we choose representation that captures sufficient bimanual dexterity. Specifically, the action a, time represented as the 3D position of each wrist, the 6D orientation each wrist (as suggested by Zhou al. 2018)), and the 3D position of each fingertip. Thus, each action has total dimensionality of 2 hands x (3 6 + (3 x fingertips)) = 48. In practice, actions are predicted in chunks over a fixed time horizon. Poses are expressed in the current camera frame (Kareer al. 2024). and each action chunk is a relative trajectory (Chi et al. 2024).
4.2 BENCHMARKS
We propose two benchmark tasks for EgoDex. The first is dexterous trajectory prediction: from the egocentric image observations, skeletal joint poses, and natural language description, the task is to predict the trajectories of the hands for a given time horizon following the observations. Specifically, we seek to train the following estimator:
fo So..,, Ac++
where o are the egocentric image observations up to and including time t, so..t are skeletal pose observations up to and including time f, is a natural language description, â H is the predicted action chunk, and H is the prediction horizon.
Since multimodality can be very severe in natural human motion, the second benchmark is inverse dynamics: from the image observations and skeletal poses up to time t as well goal image observation at the end of the time horizon, the task is to predict the trajectories of the hands in between the start and end observations. In this case, train the following estimator, which can be interpreted as visually goal- conl-conditioned policy:
So (Op..t, So... 1,, a
Each of these benchmarks parameterized by prediction horizon H. For example, short-horize trajectory prediction task may set H = 30 (1 second), while more difficult long-horizon task may set H 90 (3 seconds).

---

Published as a conference paper at ICLR 2026
Figure 4: Model prediction visualizations for Dec BC test images with 2 second horizon. Blue trajectories are ground truth and red trajectories are predictions, where darker colors and closer the current frame and lighter colors further in the future. The points shown are the wrist and fingertip positions projected into the camera frame (a total of 12 trajectories).
Unlike typical robot hardware experiments that vary across physical enviromments, the EgoDex benchmarks are fully reproducible with fixed training and test set. We set aside 1% of the EgoDex dataset fixed held-out test set for evaluations, where the remaining 99% can be split across training and validation as desired.
4.3 EVALUATION METRICS
Since trajectory prediction for natural human motion is inherently multimodal, evaluating a single predicted trajectory against the ground truth sample may be insufficient for measuring correctness. For example, for simple task of placing fruit basket, could placed at variable locations within the basket, moved at variable speeds, and moved in different but equally valid trajectories from the initial position to the basket.
Thus, for each benchmark task evaluate performance with "best of K" metric. For each data point in the full test set, we sample the trained model K times to capture different possible modes. We then compute the distance between the ground truth trajectory and the trajectory closest to it out of the samples, where "distance" calculated as the Euclidean distance between predicted 3D keypoint positions and their ground truth 3D counterparts, averaged over each timestep in the predicted chunk and each of the 12 keypoints (i.e., the wrist and fingertips of each hand). Intuitively, this value can be interpreted the average positional error in 3D space between ground truth and prediction in meters. The final value is averaged over the full test set. For deterministic models, the value is the same regardless of K; for stochastic models, the value improves K increases, as the model gets more chances to sample the ground truth mode.
EXPERIMENTS
We train and evaluate tfate-of-the-art imitation learning policies from the X-IL framework (Jia et al.) 2025) on the benchmarks from Section Specifically, we train two Transformer model architectures (encoder-decoder and decoder-only) and three policy representations (behavior cloning, denoising diffusion, and flow matching). We also run experiments to evaluate the effect of prediction horizon, visual goal-conditioning, dataset size, and model size. In total we train and evaluate 14 different models. We train all models for 50,000 gradient steps with a batch size of 2048 parallelized across NVIDIA A100 GPUs. To make the train-test split, we randomly sample a 1 1% subset of each task and set it aside as a held-out set for evaluation. Since this test set does not contain out-of-din out-of-distributio (OOD) tasks, provide additional OOD experiments in Appendix A.1 Additional training and

---

Published as a conference paper at ICLR 2026
Avg Distance (m)
Final Distance (m)
Model
K =
K 5
K 10
K = 1
K
K
10
Dec + BC
0.045
0.045
0.045
0.062
0.062
0.062
Dec + DDPM
0.053
0.044
0.041
0.071
0.050
0.044
Dec + FM
0.052
0.042
0.040
0.071
0.049
0.043
EncDec + BC
0.044
0.044
0.044
0.060
0.060
0.060
EncDec + DDPM
0.052
0.042
0.039
0.071
0.048
0.043
EncDec + FM
0.051
0.041
0.038
0.070
0.047
0.041
Table Evaluations for different models on trajectory prediction with a second horizon.
Avg Distance (m)
Final Distance (m)
Model
H 30 (Is)
H 60 (2s) H 90 (3s)
H = 30 (1s)
H 60 (2s) H 90 (3s)
Dec BC
0.031
0.045
0.053
0.049
0.062
0.069
Table 3: Results for models trained and evaluated with different prediction horizons. As expected, accuracy falls as the prediction horizon increases. H = 60 values are repeated from Table 2 for convenience.
model details are provided in Appendix A.4 5 and summarized below.
The results are presented in Tables 2
and Figure
Encoder-decoder architectures outperform decoder-only. In Table 2 we observe that all encoder- decoder ("EneDec") models consistently outperform their decoder-only ("Dec"" counterparts by a small margin.
Different policy representations excel in different settings. In Table we observe that the encoder-decoder flow matching ("FM") model outperforms the other models for K = 5 and K 10 by up to 34%. As expected, denoising diffusion ("DDPM") and FM evaluations improve as K in- creases, while behavior cloning ("BC") remains the same independent of K it is deterministic. Note however that for the K 1 setting, BC outperforms both diffusion and flow-matching by about 15%. This suggests that the average prediction of BC better than DDPM and FM, while the best prediction of DDPM and FM is better than BC's single prediction.
Performance degrades as the prediction horizon increases. In the remaining experi- ments we vary different properties while fixing the model the simplest policy: decoder-only behavior cloning. In Table we see that reduc- ing the horizon from 2 seconds to second im- proves average and final distance by 31% and 21 % respectively, while increasing the horizon from to seconds worsens average and final distance by 18% and 11% respectively. Intu- itively, accurate prediction becomes more chal- lenging the horizon increases the model must predict 48-de di-dimensionnl dexterous actions farther into the future. Appendix AopendixAA.Tssheee ad- ditional experiments using the encoder-decoder with flow matching (EncDec FM) model, which shows similar trend.
Data Scaling
0.000
Blstance
Dxtanc (m)
5.50
0.045
5%
10%
25%
50%
300%
Dasasitt Skze
Figure 5: Distance metrics w.r.t. training dataset size, where size is plotted on a log-scale. Perfor- mance improves as the dataset gets larger.
Visual goal-conditioning significantly im- proves performance. In Table 4 we observe that visual coal-conditioning reduces average distance by 22% and final distance by 53%. Intu- itively, visual goal provides visual "anchor" to ground the endpoint of the predicted trajectory and mitigate multimodality. This yields a baseline score for the inverse dynamics benchmark speci- fied in Section 4
8

---

Published as a conference paper at ICLR 2026
Model
Avg Distance (m) Final Distance (m)
Dec + BC
0.045
0.062
Dec + BC w/ goal image
0.035
0.029
Table 4: Visual goal-conditioning results. Training a model with visual goal conditioning reduces average distance by 22% and final distance by 53%.
Medium-size model capacity is sufficient for the current dataset size. We train and evaluate larger Dec+BC model with 500 million parameters as opposed to the default 200 million parameters. The larger model attains average distance 0.045 and final distance 0.062, exactly the same as the default 200 million parameter model. This may increase accessibility for the EgoDex benchmarks, medium-size models fit comfortably on commodity GPU hardware.
Performance scales with dataset size. In Figure 5 we observe that average and final distance improve dataset size increases. Results suggest that performance scales with data, motivating the collection of large-scale egocentric datasets like EgoDex.
6 RESEARCH USE CASES
Robotics While significant progress has been made in the development of robot hardware with humanoid morphologies and dexterous hands, there remains prohibitive embodiment gap between humans and today's 's robots. Some options for bridging the embodiment gap include 1) co-training with scall-scale robot dataset, as demonstrated by Kareer et al.) (2024); Qiu et al. (2025); 2) pre- training with large-scale human data and supervised fine-tuning with smaller-scale robot data, simi- lar to the training recipe large language models; 3) training visual encoder on the human data for more data-efficient imitation learning downstream (Nair et al.) 2022); ) 4) learning robot manipu- lation priors from the human-object interaction trajectories and then fine-tuning with reinforcement
learning or imitation learning (Singh al. 2024 Gavryushin et al. 2025).
Perception EgoDex can be used for learning tasks such as action recognition and human-object interaction detection. Datasets like EPIC-KITCHENS (Damen et al. 2018) have demonstrated the value of egocentric video for recognizing and anticipating daily actions, and even more challenging tasks like detecting active objects and predicting state changes from egocentric video. Researchers can also study which objects are involved in each action and how. For example, one could model the contact points, grasps, and trajectories when using a tool (screwdriver, scissors, etc.). related task is learning object affordances, i.e., understanding what actions each object supports.
Video Generation and World Models Recent advances large-scale diffusion models have sig- nificantly enhanced the capabilities of language- conditioood video generation, producing temporally consistent and semantically accurate visual narratives from natural language inputs (Li et al. [2024] Peng al.] 20255 NVIDIA al. 2025 Zhao al., (025). These generative frameworks have demonstrated potential not only in creating realistic and detailed video content but also as world models for decision making tasks, supporting reinforcement learning agents by simulating future outcomes based on predicted visual dynamics (Wang et al. 22255 Bruce et al. (024: Yang et al.) 2023 Hafner et al. 2019). Despite these impressive advancements, there remains a substantial research gap in video generation and world modeling from egocentric viewpoint. Egocentric perspectives present unique challenges, including managing significant viewpoint variability, main- taining temporal and spatial consistency amid frequent camera movements, and accurately reflect- ing agent-centric interactions and intentions. Since EgoDex provides large-scale video data with 3D pose and language annotations, it enables the training of an egocentric world model.
peciiiio-mmaiing
7 CONCLUSION
We introduce EgoDex, massive dataset of egocentric video paired with 3D pose annotations in wide range of dexterous manipulation tasks. We train and evaluate imitation learning policies for hand trajectory prediction on this data.

---

Published as a conference paper at ICLR 2026
While EgoDex has significant diversity across tasks and manipulation behaviors, it is limited in background and scene diversity. The dexterous annotations can also be imperfect, especially during heavy occlusion (e.g., towel folding) or very high speed motions, they are themselves model predictions. Future work involves procedural background randomization on the existing data Yuaan al. 2025) as well as data collection in more diverse environments.
REFERENCES
Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from hu- man videos a versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18778-13799 June 2023.
Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang. Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and object tracking in 3D from egocentric multi-view videos. JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2025.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative inter- active environments. In Forty-first International Conference on Machine Learning, 2024.
Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar lqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCH A benchmark for capturing hand grasping of objects. In IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2021.
Lawrence Yunliang Chen, Chenfeng Xu, Karthik Dharmarajan, Muhammad Zubair Irshad, Richard Cheng. Kurt Keutzer, Masayoshi Tomizuka, Quan Vuong, and Ken Goldberg. Rovi-aug: Robot and viewpoint augmentation for cross-embodiment robot learning. Conference on Robot Learning (CoRI.), 2024a.
Sirui Chen, Chen Wang, Kaden Nguyen, Li Fei-Fei, and C Karen Liu. Arcap: Collecting high- quality human demonstrations for robot learning with augmented reality feedback. arXiv preprint arXiv:2410.08464, 2024b.
Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching with- out in-the-wild robots. In Robotics: Science and Systems (RSS), 2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scal- ing egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. Conference on Robot Learning (CoRL), 2019.
Alexey Gavryushin, Xi Wang, Robert J. S. Malate, Chenyu Yang, Xiangyi Jia, Shubh Goel, Da- vide Liconti, René Zurbrügg, Robert K. Katzschmann, and Mare Pollefeys. MAPLE: Encod- ing Dexterous Robotic Manipulation Priors Learned From Egocentric Videos. arXiv preprint arXiv:2504.06084, 2025.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car- tillier, Sean Crane. Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer. Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis,
10

---

Published as a conference paper at ICLR 2026
Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Ku- mar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Mod- hugu, Jonathan Munro, Tullic Murrell, Takumi Nishiyasu. Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawa- har, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torre- sani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the World in 3,000 Hours of Egocentric Video. In JEEE/CVF Computer Vision and Pultern Recognition (CVPR), 2022.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representations (ICLR), 2019.
Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. Furniturebench: Reproducible real- world benchmark for long-horizon complex manipulation. Robotics: Science and Systems (RSS), 2023.
Xiaogang Jia, Atalay Donat, Xi Huang, Xuan Zhao, Denis Blessing, Hongyi Zhou, Han A. Wang. Hanyi Zhang, Qian Wang, Rudolf Lioutikov, and Gerhard Neumann. X-il: Exploring the design space of imitation learning policies, 2025.
Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoff- man, and Danfei Xu. Egomimic: Scaling imitation learning via egocentric video. arXiv preprint arXiv/4100242221, 2024.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale. Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pan- nag Sanketi, Archit Sharma, Cody Simpson, Quan Vuong. Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Bai- jal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jack- son, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mir- chandani, Daniel Morton, Tony Nguyen, Abigail O'Neil,, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang. Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martin, Sub- ramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manip- ulation dataset. Robotics: Science and Systems (RSS). 2024.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, 2012.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 56424-56445, 2024.
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human object interac- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21013-21022, June 2022.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implies pre-training. In International Conference on Learning Representations (ICIR), 2023.
11

---

Published as a conference paper at ICLR 2026
Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Scaling robot supervision to hundreds of hours with robo- turk: Robotic manipulation dataset through human reasoning and dexterity. In IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), 2019.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A uni- versal visual representation for robot manipulation. In Conference on Robot Learning (CoRL), 2022.
Nataliya Nechyporenko, Ryan Hoque, Christopher Webb, Mouli Sivapurapu, and Jian Zhang. Ar- mada: Augmented reality for robot manipulation and robot-free data acquisition. arXiv preprint arXiv:2412. 10631, 2024.
NVIDIA, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji. Erik Barker, Tiffany Cai, Prithvi- jit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge. Yunhao Gc. Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leai-Taixe, Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung-Y Yi Lin, Huan Ling, Ming-Yu Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo. Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page. De- spoina Paschalidou. Zeeshan Patel, Lindsey Pavao. Morteza Ramezanali, Fitsum Reda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng, Jibin Varghese, Hao Wang. Haoxiang Wang, Heng Wang, Ting-Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang. Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingging Zhao, and Artur Zolkowski. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575. 2025.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapirg Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hocschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Eukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Rejichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe,, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny,
12

---

Published as a conference paper at ICLR 2026
Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry. Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https: //tpivv.aaiivvv/aa wwt.aaa../..ooooooooeo.o.aa/.aa/ abs/ 230//2333.20 08774
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
Younghyo Park, Jagdeep Singh Bhatia, Lars Ankile, and Pulkit Agrawal. Dexhub and dart: Towards internet scale robot data collection, 2024. URL
http:///xxivvvooo httpr///b///21..2222 02214
Georgios Pavlakos, Dandan Shan, Iija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 31) with transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Young, Xinying Guo, Binluo Wang, Hang Xu, Hongxin Liu, Mingyan Jiang, Wenjun Li, Yuhui Wang. Anbang Ye, Gang Ren, Qianran Ma, Wanying Liang, Xiang Lian, Xiwen Wu, Yuting Zhong, Zhuangyan Li. Chaoyu Gong, Guo- jun Lei, Leijun Cheng, Limin Zhang, Minghao Li, Ruijie Zhang. Silan Hu, Shijie Huang, Xi- aokang Wang, Yuanheng Zhao, Yuqi Wang, Ziang Wei, and Yang You. Open-sora 2.0: Training commercial-level video generation model in 200k, 2025.
Ri-Zhao Qiu, Shiqi Yang. Xuxin Cheng, Chaitanya Chawla, Jialong Li, Tairan He, Ge Yan, David J. Yoon, Ryan Hoque, Lars Paulsen, Ge Yang. Jian Zhang, Sha Yi, Guanya Shi, and Xiaolong Wang. Humanoid policy human policy, 2025. URL http:::tt////ttt :/apxii:////xx...//// http:://wwwwwv..aaaaaaaa abs 2503.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021. URL https:: https://pp..mmmtttccco.... www.aniccccccool.........
Ilija Radosavovic, Tete Xiao, Stephen James, Picter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. Conference on Robot Learning, pages 416-426. PMLR, 2023.
Juntao Ren, Priya Sundaresan, Dorsa Sadigh, Sanjiban Choudhury, and Jeannette Bohg. Motion tracks: A unified representation for human-robot transfer in few-shot imitation learning, 2025.
URL https: www. hbcg:/bs///500
06994
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 1553):211-252 2015.
Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel, and Jitendra Malik. Hand-object interaction pretraining from videos. arXiv preprint arXiv:2409.08273 2024.
13

---

Published as a conference paper at ICLR 2026
Richard S. Sutton. The bitter lesson, 2019. URL http:/ /wwp///www..... www..ncompleeeeedaaa......
[ncldaas/BitterLeesonn html Accessed: 2025-02-26.
PyTorch Team. torchcodee: Easy and efficient video decoding for pytorch. https://githib:
ppt.orcch/torrccoeeecccccoo 2024, Accessed: 2025-04-01.
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: dataset for robot learning scale. In Conference on Robot Learning (CoRL), 2023.
Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C. Karen Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. Robotics: Science and Systems (RSS), 2024.
Lirui Wang, Kevin Zhao, Chaoqi Liu, and Xinlei Chen. Learning action---io dynamics with heterogeneous masked autoregression, 2025. URL URLAttpss//aaxii.or///b/////
04296
Mengjiao Yang. Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbecl. Learning interactive woll-word simulators. arXiv preprint arXiv:2310.06114, 2023.
Chengbo Yuan, Suraj Joshi, Shaoting Zhu, Hang Su, Hang Zhao, and Yang Gao. Roboengine: Plug- and-play robot data augmentation with semantic robot segmentation and background generation, 2025. URL htt.hhtss://arrt
abs 22003. 18738
Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao, Weikai Chen, and Xiaoguang Han. Taste-rob: Advancing video generation of task-oriented hand-object interaction for generalizable robotic manipulation. In Proceedings of the Computer Vision and Pattern Recognition Confer- ence, pages 27683-27693, 2025.
Bohan Zhou, Yi Zhan, Zhongbin Zhang, and Zongqing Lu. Megohand: Multimodal egocentric hand-object interaction motion generation. The Thirty-ninth Anmual Conference on Neural Information Processing Systems.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7738-5746, 2018. URL https:: sew....satttsschol..........
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 ADDITIONAL. EXPERIMENTS
The EgoDex dataset also includes of entirely out-of-distribution (OOD) tasks in the dataset under a separate folder (titled extra). We ran additional experiments testing the decoder-only behav- ior cloning (Dec+BC) model on these OOD tasks. We observe that some OOD tasks are comparable to in-distribution performance, while tasks that are further out of distribution have worse perfor- mance. This suggests EgoDex models can generalize to OOD tasks that are at least somewhat similar to in-distribution tasks.
Model
Avg Distance (m)
Final Distance (m)
In-Distribution Avg
0.045
0.062
(OOD) Jigsaw Puzzle
0.047
0,065
(OOD) Tetra Board
0.060
0.082
(OOD) Knit Scarf
0.064
0,093
(OOD) Play Reversi
0.065
0,096
(OOD) Blowdry Hair
0.083
0.118
(OOD) Stamp Paper
0.099
0.162
Table 5: Additional experimental results out-of-distribution tasks.
main experiments, we chose to use decoder-only BC model to evaluate the effect of increasing the prediction horizon. We additionally ran experiments to evaluate the effect of the prediction horizon with the encoder-decoder with flow matching (EncDec+FM) model for further clarity. The trend is consistent with the Dec+BC model, where performance degrades the horizon increases.
Avg Distance (m) K =
Final Distance (m)
Model
=
K = 10
K =
K
K = 10
H = 30 (Is)
0.036
0.028
0.026
0.055
0.037
0.033
60 (2s)
0.051
0.041
0.038
0.070
0.047
0.041
H = 90 (3s)
0.061
0.050
0.047
0.079
0.053
0.046
Table Ablation prediction horizon for the encoder-decoder with flow matching (EncDec+FM) model.
A.2 COMPLETE LIST OF TASKS
We provide a complete list of task names here, labeled as they appear in the dataset and separated by task type (reversible, reset-free, or reset, with definitions in Section 3.3). Recall that each reversible task is actually a pair of two tasks. There are total of 194 tasks. See Figure [ for a visual of subset of the objects used the various manipulation tasks.
For users interested in robot deployment, the casic.pick.pp plece task in particular has large amount of very diverse pick-and-place data as well as high-quality language annotation.
Reversible (76 x 2 total tasks):
braid_unbreia
charge__nnchargeeairpdd
deal_gather_cards
fry_bread
15

---

Published as a conference paper at ICLR 2026
told_unffold_ppper_bbit
gather_roll_dice
Insert_remov__arroos
insert_remove_draveer
insert_eemoe__Shiiiiinnttttu
insert_remove__ab
Lcce_eiipense__ce
pit_ewwsy_eett_g_poorrrdageee
sccey_usccer_ffingess_iiuuuuuu
aleeve_uusleeve_carrs
stack_unstaakk_cpp
threas__mthrreddbbba___eeeee.e
tle_and__mtie___hhelaeee
Insert_reeovv__tmmisjaallll
gee__clos__ineett_eerrre...a
pick_place__food
put_ihn_take_out__glaasss
scree_uusscreyaallee__ffxoor
setuup_.deenppp.chhessboorr
slot_batteries
stack_unstack_booll
stack_unsstcck_ttppperrrr
throv_colllectoojjecct
vertiical_p.cc_place
vash_put__wwa__dihhes
add_remove__id
arringe_toopee_comnnee
assemble_disassmmmle_eeggss
assemlee_ddacccmmlla_oot___o__
asemtlle_ddiaesmmmle_sstttttture
aseetble__diaassemmeee......
boil_serve_egg
leg
charge_inccargge_dviieee
clip_uncllpppppprs
crumple_flattee_ppper
fry_egg
fel
16

---

Published as a conference paper at ICLR 2026
fold_
Insert_remov__bagging
insert_eenor___log__cceee
Insert_remov__utensils
lock_unlock_key
per__closs_imeert_reeevvo_ooo
sccop_damp_ice
screv_unscre__bottle.caa
setup_cleanpp_table
stock_unstock_frridde
stack_unstack__pltee
tlrow_anddcattchbball
tie
jee_untiierrubbrbann
wrap_unwrap_food
zip_unzip_bag
zip_unzip_case
stack_uussaack_eetrabboorrdd
stack_renove_jergaa
insert_ddup__loocks
rate_smoott_een__aardn
play_rraset_commettfoor
insert_reeove_looksheeel
Reset-free (28 total tasks):
color
fidget_mande
sddeet_agpericc_ppiiner.. rings
measure_objects
staple_paper
use_rubiks_cube
wash_kiitcen_ddishes
wipe_screen
knead_slime
point_am__cllck_reemttt
type_keyboard
clean_surface
dry_hands
play_mancala
flip_coin
faip_pages
paint_clean_brush
play_piano
17

---

Published as a conference paper at ICLR 2026
push_pop_toy
put_cocthpatte_oo_ttttbtbbbt
wash_fruit
wipe_kkicchen_surfacs
stamp_paper
blowdry_haiz
knit_scarf
makeup
write
clean_cups
roll_ball
Reset (14 total tasks):
clean_tableware
declutter_desk
basic_pick_place
stack
make_sandwich
peel_plac__stickrr
sweep_dustpan
wrap
assemble_jenga
basic_fold
pour
sort_beads
use_chopsticks
play_reversi
A.3 COMPLETE LIST OF SKELETAL JOINTS
The annotations consist of SE(3) poses (represented as 4 x 4 homogeneous transformation matrices) for each of the following joints, labeled by their names as they appear in the dataset:
Upper Body:
hip, spinel, spine2, spine3, spine4, spine5, spines, spine7, neckl, neck2, neck3, neck4, leftShoulder, leftArm, leftForearm, leftHand, rightshoulder, rightArm, rightForearm, rightHand
Left Hand:
leffRiingiinngerrttttccrpppl leftRingFingerTli
18

---

Published as a conference paper at ICLR 2026
Figure Some of the objects used in the various manipulation tasks.
Lef:Thhambbnntemmddaaeeesss, aag.raaaaeecaa...crcrreeaemeaaar lettChunbbintermciiattttii, leftThumbknuckle,, leftThumbTip
Right Hand:
righttattlleeeingeeKknkk Eigetaattteetiinseeesttttaaaaara
hig.l:atiininiggiiiiiiiinnnnninn LigattThumbbItteemmdiittttttttt
hhg.:///we....aa.a...oo.o.....nn Thumbknuckk right ThumbTip
Note that left and right refer the wrists. Note also that the joint confidence values in the data behave differently for the wrists and the hands. Wrist confidence values (for leftHand and rightHand Hand) indicate whether each hand detected as whole, while finger joint confidence values indicate confidence relative to the wrist. If. for instance, the left index fingertip has high confidence but the left wrist has confidence, it is unlikely that the left index fingertip is reliable.
A.4 TRAINING DETAILS
In the experiments section we train and evaluate different models: combinations of architec- tures and policy optimization methods, 4 additional models with different training dataset sizes, additional models with different prediction horizons, model with larger model size, and model with visual goal-conditioning. See Figure for intuition on the model architecture.
Each model is trained and evaluated on a single node with 96 logical CPUs (48 physical CPUs) and 8 NVIDIA A100 GPUs each with 80GB RAM. Training is run for 50,000 gradient steps with a batch size 2048 (256 per GPU with data parallelism). at which point training and validation loss plateau. The full training Adam and learning rate le-4. Image observations are resized to 224 x 224 and sent through pretrained ResNet encoder, while language annotations passed through frozen CLIP (Radford
takes approximately 72 hours. The models optimized with
19

---

Published as a conference paper at ICLR 2026
Pollcy Representations
BC
DDPM
Flow-Matching
Architectures
Decoder
Encoder
Decoder
Pestior
CUP
HLP
Longuege Instruction "hut spsta in baskee"
Prep. Dise
Figure 7: Model architectures.
al. 202T) encoder. Only the current image observation and proprioceptive state are passed as input to the policy (i.e., no history); adding history may improve performance. DDPM and FM models are trained and evaluated with sampling steps. All other hyperparameters are the defaults from the X-IL codebase (Jia et al. 2025).
20


Reviews:
[{"review_id": "dWLSISCTAz", "reviewer": "Reviewer_9Nm4", "summary": "The paper introduces EgoDex, a massive egocentric dataset for dexterous manipulation: ~829 hours, 90M frames, 338k demonstrations, covering ~194 tabletop tasks, recorded at 30 FPS/1080p with paired 3D head/upper-body/hand (25 joints/hand) poses via Apple Vision Pro (with on-device SLAM + calibrated cameras to improve the estimation accuracy). The authors also define two benchmarks—trajectory prediction and inverse dynamics—with a best-of-K evaluation, and report baselines across encoder–decoder vs. decoder-only and BC/DDPM/Flow-Matching models, plus ablations on horizon, goal conditioning, model/dataset size. Visual goal conditioning notably improves final error (−53%), and performance scales with more data. A fixed held-out test split is provided for reproducibility. Lastly, the authors promise to open source all data in the future.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths lie in the benchmark design and the extensive effort to collect a well-calibrated dataset. Specifically, they are\n\n- Big and diverse. Way larger than prior human/robot sets, with language, camera extrinsics, and dense dexterity labels—covering ~200 tasks and ~500 objects.\n\n- Clean paired signals. Synced ego RGB + full 3D skeleton (wrists/fingertips/head/arms) at 30 Hz, which is much cleaner than post-hoc hand pose from internet videos.\n\n- Benchmark-ready. Two well-defined tasks (trajectory prediction, inverse dynamics) with a fixed held-out test split, so results are easy to compare apples-to-apples.\n\n- Useful takeaways. The experiments surface a few clear patterns—for example, goal conditioning helps consistently.", "weaknesses": "The weaknesses include:\n\n- Benchmark scope is narrow. The evaluation focuses on human motion prediction only, without assessing the robot side (e.g., retargeting quality and policy performance). Human trajectory prediction is only an intermediate signal; good imitation error does not necessarily translate to task success—this disconnect has been noted before [1], and that’s based on robot data—all the more so for human trajectory prediction as the retargeting might also amplify the error. A more direct metric like robot task success rate (in sim or on hardware) would better capture end-to-end utility. I agree trajectory metrics are reproducible, but they’re not decisive for manipulation.\n\n[1]. Mandlekar, Ajay, et al. \"What matters in learning from offline human demonstrations for robot manipulation.\" arXiv preprint arXiv:2108.03298 (2021).\n\n- Practicality of goal images. Minor but important: how is the goal image obtained at inference? If it’s required, please discuss practical acquisition (e.g., user-provided snapshot, retrieval from a library, last-frame of a setup video) and report results (or at least ideas and any preliminary explorations) with vs. without a goal image to gauge real-world usability during inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XW3Apn64o6", "reviewer": "Reviewer_cxJH", "summary": "This paper introduces a large-scale egocentric manipulation video dataset. It includes comprehensive annotations and is more suitable for robotics task. The data is used to train and evaluate imitation learning policies.", "questions": "The reviewer believes that some more pilot studies will strengthen the paper's significance, but may not influence its rating to be accepted.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and well-organized. \n2. The proposed dataset is very useful. \n3. The dataset statistics are extensively provided.", "weaknesses": "1. Some more pilot experiments should be provided, as mentioned in Section 6. It would be great to see that the downstream experiments are *actually* deployed, not just discussed, to prove the significance of the dataset.  \n2. More baseline or backbone networks should also be evaluated in the benchmark experiment. \n3. Two recent works [1, 2] should be discussed and compared, especially [1].\n\n*Refs*:  \n[1] MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation. Zhou et al.  \n[2] TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation. Zhao et al.  CVPR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Aq3ywf1fs", "reviewer": "Reviewer_kYXT", "summary": "This paper introduces EgoDex, a new large-scale egocentric video dataset focused on human dexterous manipulation. Comprising over 300,000 episodes, 90 million frames, and 800 hours of video across 200 tasks and 500 objects, the dataset is meticulously labeled with 3D hand skeleton annotations and calibrated camera parameters. The data collection focused on spontaneous, natural human actions rather than scripted or unnatural movements. This resource is intended to foster advancements in video understanding, embodied AI, and robot manipulation by providing a rich, high-quality foundation for learning dexterity.", "questions": "1. **Collection Rigor**: Provide exact details on the number of subjects and the protocol for handling and filtering noisy or failed 3D hand pose annotations in the dataset.\n\n2. **Depth/3D Utility**: Conduct a small-scale pilot study demonstrating the utility of the dataset for a 3D-related task (e.g., 3D lifting or affordance modeling) to justify the investment in 3D labels.\n\n3. **Robotics Validation**: Present a clear, quantitative baseline experiment on a robotic manipulation task (e.g., using a SOTA method fine-tuned on EgoDex) to solidify the dataset's direct value to the robotics community.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Largest Scale and Rich Annotation**: EgoDex represents the largest-scale egocentric video dataset of human hand manipulation to date. The inclusion of precise, quantified camera parameters and 3D hand pose labels is a significant contribution, providing an invaluable resource for both video understanding and the development of embodied agents.\n\n2. **Uniquely Naturalistic Data**: The data is collected from spontaneous, active human execution rather than unnatural, deliberately posed or slow-moving scripted actions. This naturalistic quality gives the dataset a unique advantage and is crucial for training robust models that generalize to real-world human behavior.\n\n3. **Novel and Engaging Task Formulation**: The proposed task of predicting human hand trajectories from an egocentric viewpoint is highly intriguing. This task is both academically interesting and potentially beneficial for mutual promotion and integration with robotic motion prediction and control tasks in the future.", "weaknesses": "1. **Limited Auxiliary Sensor Data**: The current collection is primarily focused on RGB video. The study could have been significantly enhanced by incorporating additional hardware information during data collection, such as depth maps or synchronized foreground/background segmentation masks. This supplementary data would facilitate lifting observations to 3D and enable subsequent researchers to confidently develop tasks involving procedural background randomization/generation.\n\n2. **Insufficient Detail on Data Collection Protocols**: The manuscript lacks rigor in specifying certain critical aspects of the data collection procedure. Key missing details include the exact number of subjects involved, the strictness of the policy regarding the hand always remaining within the field of view, and a statistical analysis of the duration distribution of tasks. Furthermore, the handling of noisy or erroneous keypoint predictions (e.g., jitter or catastrophic failure) is not discussed.\n\n3. **Inadequate Downstream Task Validation**: The true value of a dataset lies in its utility for downstream tasks. Despite mentioning the positive impact EgoDex could have on robotic manipulation, affordance learning, or video generation, the research does not provide sufficiently solid experimental evidence to prove these claims. For a top-tier conference like ICLR, this lack of rigorous downstream validation significantly weakens the overall persuasiveness of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1crhwIl4v1", "reviewer": "Reviewer_7T6A", "summary": "This paper presents a large-scale egocentric human video dataset that captures diverse manipulation behaviors in everyday scenarios, annotated with 3D hand poses and tracking information. To demonstrate the dataset’s utility for robot manipulation learning, an imitation learning policy is trained to predict hand trajectories, and a set of benchmark tasks and evaluation metrics are introduced.", "questions": "What are the failure modes of the learned models? Could the authors provide more qualitative results to illustrate these cases? \n\nBeyond the trajectory prediction task, do the authors plan to include additional evaluation tasks to further assess the dataset’s utility?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Leveraging human videos to learn dexterous manipulation policies is a promising direction, especially since collecting teleoperation data is often costly and time-consuming. \n\nThe authors’ plan to open-source the dataset will further benefit the research community and foster future developments in this area.", "weaknesses": "The evaluation is limited, focusing solely on the trajectory prediction task. \n\nThe paper lacks experiments demonstrating that progress on trajectory prediction tasks can effectively transfer to real-world robotic manipulation. Additional empirical studies validating this transferability would further strengthen the work and better justify the practical usefulness of the proposed dataset.  \n\nMoreover, based on the provided sample data, the captured human behaviors appear somewhat unnatural---for instance, frequent use of pinch grasps---and the backgrounds are relatively clean and uncluttered, which may limit the dataset’s realism and representativeness of everyday human activities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)