EgoWorld: Translating Exocentric View to Egocentric View using Rich Exocentric Observations
Authors: Junho Park
Andrew Sangwoo Ye
Taein Kwon
Venue: ICLR 2026 Poster (2026)
Keywords: Egocentric Vision
Diffusion Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce EgoWorld, a novel framework that reconstructs egocentric world from rich exocentric observations, including point clouds, 3D hand poses, and textual descriptions.

Abstract: Egocentric vision is essential for both human and machine visual understanding, particularly in capturing the detailed hand-object interactions needed for manipulation tasks. Translating third-person views into first-person views significantly benefits augmented reality (AR), virtual reality (VR) and robotics applications. However, current exocentric-to-egocentric translation methods are limited by their dependence on 2D cues, synchronized multi-view settings, and unrealistic assumptions such as the necessity of an initial egocentric frame and relative camera poses during inference.  To overcome these challenges, we introduce *EgoWorld*, a novel framework that reconstructs an egocentric view from rich exocentric observations, including point clouds, 3D hand poses, and textual descriptions. Our approach reconstructs a point cloud from estimated exocentric depth maps, reprojects it into the egocentric perspective, and then applies diffusion model to produce dense, semantically coherent egocentric images. Evaluated on four datasets (*i.e.,* H2O, TACO, Assembly101, and Ego-Exo4D), *EgoWorld* achieves state-of-the-art performance and demonstrates robust generalization to new objects, actions, scenes, and subjects. Moreover, *EgoWorld* exhibits robustness on in-the-wild examples, underscoring its practical applicability. 
Project page is available at https://redorangeyellowy.github.io/EgoWorld/.

Published as a conference paper at ICLR 2026
EgoWorld:
TRANSLATING EXOCENTRIC VIEW TO EGOCENTRIC VIEW USING RICH EXOCENTRIC OBSERVATIONS
Junho Park Andrew Sangwoo Ye², Taein Kwon AT Lab, LG Electronics, *KAIST, Visual Geometry Group, University of Oxford
Exocentric View
Observations
Egocentric View *
Figure 1: Ego World translates single exocentric view into egocentric view. By leveraging rich multi-modal exocentric observations, such as point clouds, 3D hand poses, and textual descriptions, EgoWorld is able generate high-quality egocentric views, even in unseen scenarios. Each ob- served modality provides complementary information that contributes to the accurate and realistic reconstruction of the egocentric view.
ABSTRACT
Egocentric vision is essential for both human and machine visual understanding, particularly in capturing the detailed hand-object interactions needed for manip- ulation tasks. Translating third-persoon views into first-person views significantly benefits augmented reality (AR), virtual reality (VR) and robotics applications. However, current pooenntrictooegocentti translation methods are limited by their dependence on 2D cues, synchronized multi-view settings, and unrealistic as- sumptions such as the necessity of an initial egocentric frame and relative camera poses during inference. To overcome these challenges, we introduce Ego World, novel framework that reconstructs an egocentric view from rich exocentric ob- servations, including point clouds, 3D hand poses, and textual descriptions. Our approach reconstructs point cloud from estimated exocentric depth maps, repro- jects it into the egocentric perspective, and then applies diffusion model to pro- duce dense, semantically coherent egocentric images. Evaluated four datasets (iee, H2O, TACO, Assembly101 101, and Ego-Exo4D). EgoWorld achieves state-of- the-art performance and demonstrates robust generalization to new objects, ac- tions, scenes, and subjects. Moreover, EgoWorle exhibits robustness on in-the- wild examples, underscoring its practical applicability. Project page is available at https::/redoraaggeet
qithub io No/EgoWorld//
INTRODUCTION
Egocentric vision plays a crucial role in advancing visual understanding for both humans and in- telligent systems (Ardeshir & Borji 2018 Grauman al. 2024 Kwon et al. 2021, Sener et al.
Corresponding author.

---

Published as a conference paper at ICLR 2026
Egocentric views are particularly valuable for capturing detailed hand-object interactions, which are essential in skill-intensive tasks such as cooking, assembling, or playing instruments. However, most existing resources are recorded from third-person perspectives, primarily due to the limited availability of head-mounted cameras and wearable recording devices. Consequently, the ability to generate or predict egocentric images from exocentric inputs holds significant promise for enhancing instructional videos and applications in augmented reality (AR), virtual reality (VR), and robotics, where perception is inherently egocentric. For example, instructional videos are often recorded from third-person viewpoint, which can be challenging for viewers to follow due to the mismatched perspectives. Translating these videos into a first-person view enables more intuitive guidance by clearly showing detailed finger placements during a task. Moreover, this translation capability unlocks the development of robust, user-centered world models (Wong et al., 2022 Chen et al. 2023 Gao et al. (023) that capture the spatial and temporal details necessary for real-time perception, planning, and interaction at scale.
2022
Although exocentric-t0-egooentte view translation holds great promise, it remains particularly difficult challenge in computer vision. The main obstacle stems from the substantial visual and geometric differences between and objects with the fine detail necessary for precise manipulation, whereas exocentric views offer wider context and kinematic cues but lack emphasis on these intricate interactions. Bridging these views is fundamentally under-constrained and cannot be addressed by geometric alignment alone, due to factors such as occlusions, restricted fields of view, and appearance changes across different viewpoints. For instance, elements like the inner pages of a book may be completely obscured in an exocentric perspective but still need to be realistically inferred in the egocentric output. Moreover, reconstructing background details in the egocentric view, which are invisible from the exocentric perspective, a nontrivial task.
and first-person views. Egocentric views focus hands
Recently, the impressive achievements of diffusion models (Rombach et al. 2022 Ho et al. 2020) have opened up new possibilities for applying generative techniques to the task of exocentric-to- egocentric view translation. However, many existing approaches rely on restrictive input conditions, such as multi-view images (Liu et al., 2024a), known relative camera pose (Cheng et al. 2024), or a reference egocentric frame to generate subsequent ones (Xu et al. 2025), making them impractical for scenarios where only single view images are available. More closely, Exo2Ego (Luo et al. 2024b) attempts to generate egocentric views from single exocentric image. Yet, it depends heavily on accurate 2D hand layout predictions for structure transformation, which can be unreliable in cases of occlusion, viewpoint ambiguity, or cluttered environments. Furthermore, it struggles to generalize to novel environments and objects, often overfitting to the training dataset. Overall, current methods lack the detailed understanding of exocentric observations necessary to synthesize precise and realistic hand-object interactions from first-person view.
To address the limitations of current approaches, we propose EgoWorld, a novel framework for translating exocentric views into egocentric views using rich exocentric observations, as illustrated in Fig. [ Our method employs a two-stage pipeline to reconstruct the egocentric view: (1) ex- tracting diverse observations from the exocentric view, including projected point clouds, 3D hand poses, and textual descriptions; and (2) reconstructing the egocentric view based on these extracted cues. In the first stage, we construct a point cloud by combining the input exocentric RGB image with the estimated exocentric depth map, which is scale-aligned by the 3D exocentric hand pose for spatial calibration. This point cloud is then transformed into the egocentric view using a transforma- tion matrix computed from the predicted 3D hand poses in both views. After the projection of the point cloud, sparse egocentric image is obtained and it is subsequently reconstructed into dense, high-quality egocentric image using a diffusion-based inpainting model. To further enhance the se- mantic alignment and visual fidelity of the hand-object reconstruction, we incorporate the predicted exocentric text description and estimated egocentric hand pose during the reconstruction process.
We evaluate the effectiveness of Ego World through extensive experiments conducted on four datasets (i.e., H2O (Kwon et al. 2021). ). TACO (Liu et al. 2024b). Assembly 101 (Sener et a 2022). and Ego-Exo4D (Grauman et al. 20244). ) which provide well-annotated exocentric and egocentric ofee-----hh-r performance on these benchmarks. Owing to its end-to-end design, WgoWorld demonstrates strong generalization across various scenarios, including unseen objects, actions, scenes, and subjects. Moreover, evaluations unlabeled real-world data further confirm its strong in-the-wild generalization ability.
video pairs. Our method achieves
2

---

Published as a conference paper at ICLR 2026
Our main contributions can be summarized as follows:
We introduce EgoWorld, novel end-to-end framework that reconstructs high-fidelity ego- centric views from a single exocentric image by leveraging rich multi modal cues, includ- ing projected point clouds, 3D hand poses, and textual descriptions.
Our two-stage pipeline uniquely integrates geometric reasoning with semantic information and diffusion-based inpainting model that significantly enhances hand-object interaction fidelity and semantic alignment for generating egocentric images.
We demonstrate the strong generalization capability of EgoWorld through extensive exper- iments on H2O, TACO, Assembly101, and Ego-Exo4D datasets. Our approach achieves state-of-the-an performance across diverse and previously unseen scenarios (i.e.. unseen objects, actions, scenes, and subjects). Additionally, we show EgoWorld's real-world ap- plicability with an-the-wild examples.
RELATED WORK
2.1
EGGCENTTIC TRANSLATION
Egocentric vision has also been scaling up particularly due to the introduction of benchmarks (Damen et al.) 2018 8 Kwon et al.] 2021 Grauman et al.] 2022, 2022 Grauman ct al. 2024). Recently, research on wecccnttrccto-coccnnie
Damen et al., 2022] Sener et al.) (and vice versa) trans- 2024a Xu et al. 2025) has also gained lation (Luo et al.) 20244a by Cheng et al. 2024 Liu al.
Luo et al. 2024a) proposed an intention-driven ego-to-exo video generation framework that leverages head trajectory and action descriptions to guide content- consistent and motion-aware video synthesis. Exo2Ego (Luo et al., (024b) introduced a two-stage generative framework for pescentrrc-toeeoceettee view translation that leverages structure trans- formation and diffusion-based hallucination with hand layout priors. 4Diff (Cheng et al., (2024) proposed a 3D-aware diffusion model for translating exocentric images into egocentric views using egocentric point cloud rasterization and 3D-aware rotary cross-attention Exo2Ego-V V (Liu et al. 2024a) presented diffusion-based method for generating egocentrie videos from sparse 360 ex- ocentric views of skilled daily-life activities, addressing challenges like viewpoint variation and motion complexity. EgoExo-Gen (Xu et al. (025) addressed cross-view video prediction by gen- erating future egocentric frames from exocentric video, the initial egocentric frame, and textual instructions, using hand-object interaction dynamics as key guidance. However, these works have fatal limitations: dependency of 2D layouts, pre- pre-nefined relative camera pose, multi-view consec- utive sequences inputs, and the challenge of integrating multiple external modalities, such as textual description and pose map.
significant attention. Iiiention-Egouaas
2.2
IMAGE COMPLETION
Image completion is fundamental problem in computer vision, which aims to fill missing regions
with plausible contents (Pathak et al. 2016, Liu et al., 2019; Xiong et al. 2019) Song ct al. 2018
Zhao et al. 2021 Suvorov et al. 2022 Li et al. 2022). For example, MAT (L al. 2022) proposed
transformer-base based model for hole image inpainting that combines the strengths of transform-
ers and convolutions to efficiently handle high-resolution images. On the other hand, masked image
encoding methods learn representations from images corrupted by masking (Vincent et al. 2010
Pathak et al., 2016)] 2016 Chen et al.) 2020) Dosovitskiy et al.) 2020)) Bao et al. 202 He et al. (022). For example, MAE (He et al. 2022) masks random patches of an input image and learns to reconstruct
the missing regions. However, these studies have a limitation that they rely solely on the information surrounding the pixels to restore missing area. With the advent of foundational diffusion models (Ho) et al. 2020: [Song et al. 2020), it has become possible to perform image completion based on vari- ous types conditions. Specifically, latent diffusion model (Rombach et al. 2022) supports flexible conditioning such as text or bounding boxes and enable high-resolution image synthesis, achieving sfate-of-the-an results in inpainting, class- conditinni generation, and other tasks by incorporating -ross-attention. Furthermore, the value of diffusion-burs based models has been demonstrated across wide range of challenging domains, such as hand- hand or hand-object interaction image generation (Zhang al. 2024 Park et al. 2024), and motion generation (Cha al. 2024 Huang al. 2025).
3

---

Published as a conference paper at ICLR 2026
Exccentric View Observation Par
Egocentric Paye
nranoall
Catmatoo
ID Hand Plose
Exicuenle
Expcentric Image lass
Language Mari
Egocantric Image lops
Ceser
Figure 2: Overall framework of Ego World. EgoWorld has two-stage pipeline (1) Exocentric view observation & ezoo which extracts diverse observations from the exocentric view, including pro- jected point clouds, 3D hand poses, and textual descriptions; and (2) egocentric view reconstruction Pego, which reconstructs the egocentric view based on cues from the exocentric view observation.
METHOD
3.1 PROBLEM FORMULATION
EgoWorld consists of two stages: exocentric view observation Pezo and egocentric view recon- struction 4 Pego, as shown in Fig. 2 First, given single exocentric image Lero E [ Lezo predicts corresponding sparse egocentric RGB map Sego E #xx333 3D egocentric hand pose Pego R. Nx3 and textual description Texo- H and W indicates height and width of Texos and N indicates the number of keypoints of the hand. Then, in T Pego: an egocentric image lego Rxxxxx is generated based on the observations predicted in Pezo. Therefore, EgoWorld is formulated as follows:
Segor P Pegor Texo Bexo (exo(Texo)
Lego = $ (Seor P Pegor )ewo).
(2)
3.2 EXOCENTRIC VIEW OBSERVATION
Exocentric view observation Lezo takes various real-world observations, such as sparse egocentric RGB map Sego, 3D egocentric hand pose Pego- and textual description Tewo, from the single exo- centric image Lezo. These observations essential for the egocentric view reconstruction & egoo
First, with an off-the-shelf depth estimator (Wang et al. 2025), an exocentric depth map Dexo E HH is extracted from Texo- Obtaining Dezo is essential, because in Pega- the reconstruction process relies on Sego, which serves as a crucial hint. Specifically, when pixel information from an exocentric view is transformed into an egocentric view, provides partial observations of the hand, object, or scene, and this serves as strong basis for approaching the problem from an inpainting perspective.
Next, a 3D exocentric hand pose Pezo E Nx3 is extracted from Levo with an off-the-shelf hand pose estimator (Yu al.) 2022).. As Dewo provides only relative depth and is inherently affected by scale ambiguity, it is crucial to leverage Pero for reasonable scale fitting. Specifically, it is possible extract metrically-scat scaled Pazo and an exocentric hand depth map Dhand E I.. HxW from the estimated MANO(Romen) et 2017 mesh of Pemo- We define hand region Phand which a pixel-level valid area determined by Drand, and compute a global scale factor 8* by comparing it with Deno follows:
Dnand (u, v)
median (0.,2) Pras Dexo (t, v) +
(3)
where u, indicate the pixel coordinate of depth maps, and is small constant prevent division by zero. Applying s* yields a metrically-calibrated exocentric depth map D' D'ezo = s* Deaoo There- fore, with Iezo and an exocentric camera intrinsic parameter Kewo E 33x3 which is estimated from the off-the the-shee depth estimator, D' Dizzo is utilized to obtain point cloud Coro RR ( xXx))
4

---

Published as a conference paper at ICLR 2026
To project Cero in the egocentric view, need an matrix X E R4x4 which can be computed through a transformation between Pezo and Pego- How- ever, to the best of our knowledge, there is model that predicts Pago directly from Lezo. Thus, 3D egocentric hand pose estimator &ego. which is designed with a
view transformation
we build a powerful-but-simpee simple architecture consisting of (Dsovvvivis al., (0200)basedd backhone Obackhone and an MLP-based regressor Øreg- Specifically, after extracting an image feature from Lezo with @backbones is fed through Øreg to obtain P' Pego. We optimize dego with an L2 loss function.
From the obtained Pezo and Pego, we calculate X between them with the Umeyama algorithm (Umeyama 199I), which estimates transformation matrix as follows:
Xego-texo = (s, R. t), such that Pezo ~ sRPego + t.
(4)
Here, 8, R. and t. are the estimated scale, rotation, and translation matrices. Since both Pero and Pego are in metric units, 8 expected to close to 1. The transformation from exocentric to egocentric view is given by X (Xego- Therefore, we translate Cewo with X into Cegor project it into egocentric view with an egocentric camera intrinsic parameters Kego 3 and obtain the sparse egocentric RGB map Sego-
Finally, Tezo is extracted with off-the-shelf vision-language model (VLM) (Bai al. 2023). For example, when Texo and a user-provided question (i.e. "Describe in detail about the scene and the object that the person interacting with using their hands.") are given, VLM outputs the corresponding answer Tezo. Since Texo contains both the overall contextual information present in the exocentric view and specific details about actions and objects, it significantly aids Pego for reconstructing the faithful egocentric view for unseen scenarios.
3.3 EGOCENTRIC VIEW RECONSTRUCTION
Since Sego only contains partial information observed from the exocentric view, is necessary to reconstruct the missing regions. Thus, leveraging the powerful latent diffusion model (LDM)
(Rombach et al. 2022). we exploit exocentric observations Sego- Pegos and T Tezo for $ Pego-
Following the LDM, input images are encoded into the latent embedding using a frozen VAE en- coder (Esser et al. 2021), and the denoised latent embedding is decoded into an output image using the frozen VAE decoder. Specifically, we encode Sego to sparse embedding Sego E Reix6ix1 with VAE encoder. We obtain a 2D egocentric hand pose map P2D E 6122x122x by projecting P Pego with Kego- encode P2D to 4-channels embedding with VAE encoder, and reduce the number of channels of 4-channels embedding to 1-channel via a channel reduction layer. This layer consists of one convolutional layer, which inputs 4-channel embedding and outputs 1-channel embedding. Therefore, we obtain a I-channel pose embedding Pego E 64644441
During training, the ground-trumt truth egocentric image Lego E 1512551223 is also encoded to a clean latent 2p E Relx6ix1 through the VAE encoder, and the noise é E RR6166111 is added to 2g to make noisy embedding 2 E _64x6644 with timestep f follows:
(5) where og denotes the noise level t. By concatenating Sego- Pegos and 2,, we obtain 9-channel latent embedding z/ E R64x64 which is fed into the input of a pre-trained U-Net. Simultaneously, a textual description Tezo is passed through CLIP (Radford et al.) 2021) to obtain text embedding Cezo E 777668 which serves as guidance for the U-Net of LDM. In this manner, the forward and reverse processes for the denoising network Eo carried out to predict e, with the following objective:
24 = Vāy 2g + vI ā. - C. N(.
L Ezo Pe ||ee -
During sampling, we start the denoising process from random Gaussian noise 2T NN(0, I) with well-trained cg. We concatenate ær E 264x64x4 with Sego and Pego. and feed to eg to obtain the predicted latent En E R61x64x4 by reversing the schedule in Eq. at each timestep E [1, 7). We adopt classifier-free guidance (CFG) (Ho & Salimans] 2022) to strengthen textual guidance as follows:
E = (1 + w) c0(2,, Cexo) - w ca(z,, t, t)). where indicates the scaling factor in CFG, and Ø means unconditional. To the end, the final generated egocentric image Tego is obtained from &n by passing the VAE decoder.
(7)
5

---

Published as a conference paper at ICLR 2026
Table 1: Comparisons with state-of- the--rrrttt unseen scenarios (i.e., objects, actions, scenes, and subjects) in H2O Woon et al., 2021). Compared to state-of state-of-the-arts (i.e., pix2pixHD (Wang al., 2018), pixelNeRF Yu al. (021), and CFLD (Lu et al., 2024)), EgoWorld outperforms for all unseen scenarios in all metrics (i.e. FID, PSNR, SSIM, LPIPS, PA-MPJPE, and CLIPScore).
Scenanos
Usseen Orjects
Unseen Actios
Medhods
PA-MPIPEL
CLiPScore
LPIPS)
PA PA-MPIPEL
CLIPScor!
pis2ytHH Warg al 2013
456.25
25.012
0.2993
0.6057
18,007
0.2302
211.10
24.420
0.2854
0.6127
17.754
0.3450
498.23
26.357
0.3887
0.5372
15.746
0.2270
251.76
27.961
0.2950
0.8159
14:656
0,2311
CFLD La drain
59.655
25.922
0.4307
0.4539
7.997]
0.2656
50,953
28.529
0.4324
0.4593
8,1199
0.2699
Eao (ouss
41.354
31.171
0.4514
0,3476
7.3178
0.2731
33,384
31,620
0.4566
0,3780
7,3602
0.2834
Secnatios
Uascen Secnes
Unsess Subjects
PDD]
PSNKT
551M1
LPIPS.
MPJPIL
CLIPScore
F1D1
PSNR1
SSIMI
LFIPS]
MP/PEL
CLIPScore!
pis2pedH Ware allll.200
18,567
0.2425
0.2159
452.15
0.7236
21.35 31.357
0,2311
pitelNekill
0.2574
17,085
0.2397
893.13
0.6838
0,2263
CILD
2014
118,10
3,3896
787066
0.2506
129.30
0.6299
9.5600
0.2461
Ago
90393
31,084
0,8819
7.4087
0,2585
96.
24,851
0,4685
06138
8,1031
0,2583
Input (Exocentris)
Ground-truth (Egocentric)
pix2pixHDD
pixelNeRF
CFLD
Ours
Urssen Objects
Unseen Actions
Unseen
Scands
Unseen Subjects
Figure 3: Comparisons with state-of-the-arlss and subjects) in H2O (Kwon ef al.) 2021). Compared to State-of-the-ard (i.e., pix2pixHD (Wang al.) 22018), pixelNeRF (Yu al. 2021). and CFLD (Lu et al. (2024)) EgoWorld outperforms the image reconstruction quality with respect to hand-object interaction and background regions for all unseen scenarios.
scenarios (iee,, objects, actions, scenes,
EXPERIMENTS
4.1 DATASETS
To evaluate exocentric-to-egocenii translation models including EgoWorld, select H20 (won al. (02T). which contains diverse scenarios such as unseen objects, actions, scenes, and subjects. Following previous work (Luo al. 2024b), split four unseen settings to evaluate generalization as follows: (1) unseen objects, where we train with six objects and test with novel two objects, (2) unseen actions, where
train with first 80% frames and test with last 20% frames, (3) unseen scenes, where we train with four scenes and test with novel two scenes, and (4) unseen subjects, where we train with one subject and test with novel subject. To further demonstrate the general- izability of our method, we also evaluate it on TACO (Liu et al. 2024b). Assembly 101 (Sener et al.) 2022), and Ego-Exo4D (Grauman tion sequences involving 15, 1,380, and 689 actions respectively, we adopt them as unseen actions scenario, which allows for a general and comprehensive evaluation of generalization performance.
2024) datasets. Since they provide hand-object interac-
4.2 EVALUATION METRICS
Following previous works (Luo et al. 2024b Liu al. 2024a), we adopt comprehensive set of evaluation metrics to assess reconstruction quality and generalization: (1) Fréchet Inception
Distance (FID) (Heusel et al. 2017). which Inception-v3 (Salimans et al. 2016) features to
measure the distributional distance between generated and real images; (2) Peak Signal-to-Noise
Ratio (PSNR), pixel-wise fidelity metric that quantifies the ratio between the maximum possible

---

Published as a conference paper at ICLR 2026
Table 2: Comparisons with state-of-the-arts on unseen actions in TACO (Liu al. 2024b), Assembly101 101 (Sener et al., 2022), and Ego-Exo4D (Grauman et al., 2024). Compared to state- of-the-arts (i.e., pix2pixHD Wang et al. 2018), pixelNeRF (Yu et al. (021). and CFLD (Lu al.) 2024h,, EgoWorld outperforms for all unseen scenarios in all metrics (i.e, FID, PSNR, SSIM, LPIPS, PA-MPJPE, and CLIPScore).
Methods
FIDI
PSNR1
SSIM
LPIPS)
PAIMPIPE CLIPScore
TACO
[Liu
al. 2024bb
Di Wang et 2018
227.87
25.875
0.2806
0.7037
19.054
0.2509
RixelNeRF
0.2251
CFLD 2024
61.357
28.769
0.4009
0.5033
7.9078
0.2715
Ego rOurs)
37.191
30.155
0.4237
0.4025
7,3590
0,2828
Assembly101 (Sener all
2022
pix2pixHID Wang et al. 2018
350.97
17.107
0.3587
0.6578
21.967
0.2114
piscINeRF Yo al 2021
356.44
19.037
0.3761
0.6019
19.658
0.2070
CFLD (Lu ara
53,931
20.998
0.3988
0.5566
11.108
0.2458
EgoWord (Dmn))
25.365
0.4101
Ego-Exo4DD
(Grauman al.
2024
pix2pixIID Wang et 2018
401.48
14.792
0.3065
0.6899
25.082
0.2203
20243
17.347
CFLD (Lu 2024
70.476
21.578
0.3614
0.5975
15,010
0.2670
Egolkond (OBrs)
61.231
24.985
0.3986
0.5482
13.992
0.2862
Inout (Exocentric)
Ground-inuth (Egocentrcc
CRLD
Ours
Input (Exocentric)
Ground-Iruth (Egocentric)
CFLD
Curs
TACO
Assembbyy
Egp-EooOO
Figure 4: Comparisons with state-of-the-art on unseen actions scenario TACO (Liu et al. 2024b), Assembly101 (Sener et al., 2022), and Ego-Exo4D (Grauman et al., 2024). Compared to state-of-ther the-art (i.e. CFLD (Lu et al. 2024)), WgoWorld outperforms the image reconstruction quality with respect to hand-object interaction and background regions on more challenging scenarios than H2O (Kwon et al. 2021).
pixel value and the mean squared error (MSE) between a reconstructed image and its ground-truth counterpart; (3) Structural Similarity Index Measure (SSIM) (Wang et al.] 2004). which evalu- ates image similarity by comparing luminance. contrast, and structural information to better reflect human visual perception; (4) Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al.
2018), which employs deep neural network (Simonyan & Zisserman 2014) trained on human
7

---

Published as a conference paper at ICLR 2026
Input
Sparse Map
Pose Map
CFLD
Ours
'A peraco shiing tvble will the toble in from' fhem: The ssing tosh thoir Anents intereet with the box The penson's left hand sightly ratend, and their right hand pioced aighty lower on box This posttoning of the hands gives sense of accelaais emploseir on the bex. The person booking divecty the dus as was), winch fatther empptunuse ees ther interaction with
with aide the bottle. The bottle center the table akmout entraey amount halon
the boftom. The man espears elterr using the lotion mayoe just placing hiss hands near Y sorre reason."
hand posdiored book, presssing dhum on the top the book This interaaiion suggostt fluif the winner angaging with the contert the book demondtratng somothing related to the book."
oook. The
Figure 5: Real-world comparisons with state-of-the-art. Compared to state-of-the-art (i.e., CFLD (Lu et al. 2024)), EgoWorld significantly outperforms with respect to hand-object interaction and background regions for in-the-wild scenarios.
judgments to assess perceptual similarity: (5) Procrustes Analysis Mean Per Joint Position Error (PA-MPJPEE, which measures the average Euclidean distance between predicted and ground-truth 3D hand joints after Procrustes alignment (i.e., scale, rotation, and translation normalization) to evaluate hand generation accuracy, where the predicted 3D hand joints are obtained using HaMeR Pavlakos et al., 2024); and (6) CLIPScore (Hessel et al. (202I), which computes the similarity be- tween image and text embeddings obtained from CLIP (Radford et al. 202I) to assess object-level generalization.
4.3 RESULTS
4.3.1
COMPARISONS ON BENCHMARKS
To compare EgoWorld with related works, we consider several state-of-the-arts: (1) pix2pixHD (Wang et al. 2018), a single-view images-to-image translation model, (2) pixeINeRF (Yu et al.) 2021 generalizable neural rendering method that synthesizes novel views from or few images by combining pixel-aligned features with NeRF-style volume rendering. and (3) CFLD (Lu et al.] 2024), coarse-to-fine latent diffusion framework that decouples pose and appearance information at different stages of the generation process. Due to absence of source code Exo2Fgo (Luo et al., 2024b), which estimates egocentric hand layout and generated egocentric image based on the hand layout, we adopt CFLD. Since CFLD assumes ground-truth hand layouts as input, it is an upper- bound reference for Exo2Ego. In addition, since the source code of 4Diff (Cheng al. 2024), which generates images using only point clouds without hand poses or textual descriptions, is not available, we simulate it by removing pose and text from our model. Experimental results of 4Diff setting can be found in Tab.
Based on experiments on H2O across four unseen scenarios, our method achieves state-of-the-ard performance on all evaluation metrics shown in Tab. [ pix2pixHD and pixelNeRF perform sub- stantially worse across scenarios, while CFLD serves as the strongest baseline. Compared to CFLD, Ego World achieves consistent improvements in all unseen settings. On unseen objects, FID is duced from 59.615 to 41 41.334 (30% relative reduction), with PSNR improving by over 5 dB (25. 25,922 to 31.171). A similar trend is observed for unseen actions, where FID decreases by 35% (50.953 to 33.284) and PSNR increases by more than 3 dB. Even in the more challenging unseen scene setting, requiring accurate global context reconstruction, Ego World reduces FID by 23% (118.1 to 90.893).
8

---

Published as a conference paper at ICLR 2026
Input (Exocentric)
Ground-truth (Egocentric)
Ours Pose w/o Text
Ours w/o Text
Ours Pose w/ Text
Ours w/ Text
Sparse Map
Pose Map
surfac fis mode sturtyy posebly cantioert or phootic and feor aryss deage weth co wable sardinge grapaics The persin or teading the bose their bet twnr war appeers to be abour spen of plas on auriacos The bor Lsing herd ascuray and supeare cordller.
"hee peronn in hrtecucing with
wich
shanw
penson is holding orango oos, appeare Irerrciing
The studly eptertt possily condbcadt orango colo, hard and bas ineghtic in indesstpp ltaf if ner be Uner or Agriiigth The persoets onteraction with the onr suppeuts that chey Are witer urbbsong eskmssing be contents which could be niltted io product nONeA, demonathaion, acore alter Eype cuntent creation
thalrtandd,
rightyy
Figure 6: Ablation study for conditioning modalities. World generates more reasonable im- ages when conditioned on both pose maps and text, compared to using only one or none,
Ground-Iruth
Sparse Map
Ground-truth
Sparse Map
The persos is Inturuiingg with crsall wOodNs oblec whoch agosers be tor a ond éier, to to maniuubtiii posably mowigg slvin surfice or auishing la posiion The object made of woot and has natural erpantc mhl clear speciic fundtton could
"The parest image fr man beart who anting dosing rable, ae aearing witicn appears to de Starbuckk coffee App ryiig plai something from the por with As furst otier fooking
The person hntomciing wit our snack apechicaly Pringhe. Pringee is holchng the Prispler can in thee wi hand and teachin the the can with thei ngn hend They are also macing i plice of
"The crage shoas labllke dest sehup whore perton rnslding wit bott hands
slanta with mountbd carorr. Noty uove for spponshng cosorvation Bell the iipodd restangslor and puppo rott ls ploced the dest, Stattver apess the doar are sepprst acee cbjects
hand and are procese of patking it
spherical
tecking
The red-colored texts represent in- correct descriptions, which are reflected as conditioning inputs for Ego World generate egocentric images.
Figure 7: Ablation study for incorrect textual description.
Although the gain in PA-MPJPE is moderate (e.g.. 7.9971 to 7.3178 on unseen objects), substantial improvements in FID and LPIPS (0.4539 to 0.3476) indicate enhanced perceptual realism beyond pose alignment. The consistent increase in CLIPScore further suggests improved semantic consis- tency between generated egocentric views and underlying interactions.
As shown in Fig. 3 pix2pixHD produces noisy artifacts, while pixeINeRF generates blurry outputs lacking fine-grained details. pix2pixHI is not well-suited for exocentri--to-ggoeenti translation due to large geometric discrepancies, and pixelNeRF is primarily designed for multi-view synthesis. Although CFLD reconstructs hands effectively, struggles with detailed object appearance and global scene context, resulting in unrealistic backgrounds. Therefoe, EgoWorld achieves robust performance even in challenging unseen scenarios, by leveraging complementary cues from the exocentric view, including pose, text, and sparse maps.
Moreover, as shown in Tab. 2 and Fig. 4 Egold generalizes effectively other datasets with increasing real-world complexity. On TACO, our method reduces FID by approximately 39% com- pared to CFLD (61.357 to 37.191) and improves PSNR by 1.4 dB. On Assembly101 although the absolute FID margin is smaller (53.931 to 50.232). EgoWorld consistently outperforms CFLD across all metrics, including a 4.3 dB improvement PSNR On Ego-Exo4D, which exhibits substantial real-world variability, our method reduces FID by 13% and improves PSNR by over 3 dB while also lowering PA-MPJPE by more than mm. Across all datasets and unseen settings, these consis- tent improvements demonstrate that EgoWorld effectively reconstructs both local hand details and global scene structure, achieving strong perceptual fidelity, semantic alignment, and pose accuracy in diverse cross-view generation scenarios.

---

Published as a conference paper at ICLR 2026
4.3.2 COMPARISONS ON REAL- WORLD EXAMPLES
Furthermore, evaluate real-world generalization in in-the-wild settings, we conduct experiments on Ego Woold using a state-of-the-ar baseline model for comparison. We collect in-the-wild images of people interacting with arbitrary objects using their hands. Note that our method relies solely on single RGB image captured using a smartphone (iPhone 13 Pro), and we apply the complete pipeline withoul any additional inputs. As shown Fig. 5 CFLD produces egocentric images that appear unnatural and biased toward patterns observed in the training data, resulting in inconsis- tencies when applied to novel interaction scenarios. In contrast, EgoWorld generates coherent and realistic egocentric views by effectively leveraging the sparse structural map, demonstrating strong generalization to unseen real-world examples. These results suggest that Ego gooWold maintains robust performance even in unconstrained environments. With further training on more diverse datasets, the proposed framework has the potential to support practical real-world applications.
4.3.3 ABLATION STUDY FOR CONDITIONING MODALITIES
To analyze the contribution of each condition- ing modality, perform ablation study by selectively enabling pose and text inputs, as shown in Tab. 3 Without pose text, the model achieves an FID of 56.120 and PSNR of 27.054. Adding pose alone yields only marginal improvement (FID 55.016), whereas incorporating text alone substantially reduces FID to 44.240 (21% relative reduction) and im- proves PSNR to 28.565, highlighting the im- portance of semantic cues for object and scene reconstruction. The best performance is achieved when both pose and text are jointly used, further reducing FID to 41.334 and increasing PSNR to 31.171 (+2.6 dB over text-only), while also lowering PA-MPJPE to 7.3178. As shown in Fig, 6 removing text leads to incorrect object reconstruction, whereas pose guidance improves hand con- figuration realism, demonstrating complementary roles of semantic (text) and structural (pose) con- ditioning. We additionally simulate the 4Diff (Cheng et al. 2024) setting by removing both pose and text conditions (first row), which results in clear pertormance degradation, indicating that realistic egocentric reconstruction requires both semantic and geometric guidance.
Table 3: Ablation study for conditioning modal- ities. EgoWorld achieves higher scores when con- ditioned on both pose maps and text, compared to using only one or none.
Pose Test | Finj
PSNRT
SSIM
[PIPS]
PA-MPIE]]
CLIPSSaed
56.120
27,054
0,4660
0.4454
7,8022
0.2713
53.016
27.544
0,4489
0.4121
7,8007
0.2720
44.240
28.565
0.4733
0.3821
1.7452
0.2729
41.34
31.171
0.8814
0.3476
7.3178
0.2751
4.3.4 ABLATION STUDY FOR INCORRECT TEXTUAL DESCRIPTION
To evaluate the influence of textual guidance on egocentric reconstruction, we intentionally pro- vide textual descriptions that partially mismatch the exocentric image. As shown in Fig. 7 the textual input modulates appearance-level attributes of objects, subjects, and the overall scene in the reconstructed egocentric view. Importantly, despite the semantic mismatch, EgoWorld consis- tently preserves the underlying geometric structure (e.g., table slope) encoded in the sparse map. This behavior indicates that textual guidance affects semantic and appearance components, while the structural layout remains grounded in geometric observations. These results demonstrate that Ego World can flexibly integrate multi-moda. cues, enabling controllable semantic modulation with- out compromising geometric consistency, even under previously unseen combinations of visual and textual inputs.
CONCLUSION
In this work, we introduce EgoWorld, a novel framework that translates exocentric observations into egocentric views by leveraging rich multi-modal cues. Our two-stage design first extracts informa- tive exocentric observations and then reconstructs realistic egocentric images from sparse egocentric maps through a diffusion model conditioned on pose and text. Through extensive experiments on four datasets (i.e., H2O, TACO, Assembly 101, and Ego-Exo4D), we demonstrate that Ego World out- performs existing methods and proves highly effective. Beyond benchmark performance, Ego World exhibits strong generalization to real-world samples, highlighting its potential for deployment in diverse and unconstrained scenarios.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENT
This research funded by an SNSF Postdoc.Mobility Fellowship P5200T.225550..
REFERENCES
Shervin Ardeshir and Ali Borji. An exocentric look at egocentric actions and vice Computer Vision and Image Understanding, 171:61-68, 2018.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. 100.:236..196, 2023,
Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint wwXiv22166008244 2021.
Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Back. Text2hoi: Text-guided 3d motion generation for hand-object interaction. In IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 1577-1585, 2024.
Joya Chen, Difei Gao, Kevin Qinghong Lin, and Mike Zheng Shou. Affordance grounding from demonstration video to target image. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6799 6808, 2023.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691 1703, 2020.
Feng Cheng, Mi Luo, Huiyu Wang, Alex Dimakis, Lorenzo Torresani, Gedas Bertasius, and Kristen Grauman. 4diff: 3d-aware diffusion model for third-to-first viewpoint translation. In European Conference Computer Vision, pp. 407-425, 2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision, pp. 720-736, 2018.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Fumaar, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocen- tric vision: Collection, pipeline and challenges for Booo--icchens-O00 International Journal of Computer Vision, pp. 1-23, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. International Conference on Learn- ing Representations, 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873-
12883, 2021.
Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306.0 086-000000006000 2023.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos
Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19383-19400, 2024.

---

Published as a conference paper at ICLR 2026
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems, 30:6626 1166626-6637, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-688, 6851, 2020.
Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin Liang, Haoyu Ma, Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, et al. Hoigpt: Learning long-sequence hand-object interaction with language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7136-7146, 2025.
Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Mare Pollefeys. H2o: Two hands ma- nipulating objects for first person interaction recognition. In IEEE/CVF International Conference on Computer Vision, pp. 10148, 2021.
Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. Mat: Mask-aware transformer for large hole image inpainting. In IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp.
2022.
Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. Coherent semantic attention for image inpainting. In IEEE/CVF International Conference on Computer Vision, pp. 1170-4179, 2019.
Jia-Wei Liu, Weijia Mao, Zhongcong Xu, Jussi Keppo, and Mike Zheng Shou. Exocentric-to- egocentric video generation. Advances in Neural Information Processing Systems, 37:136149- 136172, 2024a.
Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual -oo--actio--obect understanding. In JEEE/CVF Confer- on Computer Vision and Pattern Recognition, pp. 21740-21751, 2024b.
Yanzuo Lu, Manlin Zhang, Andy J Ma, Xiaohua Xie, and Jianhuang Lai. Coarse-to-fie latent diffusion for pose-guided person image synthesis. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6420-6429, 2024.
Hongchen Luo, Kai Zhu, Wei Zhai, and Yang Cao. Intention-driven ego-to-exo video generation. arXiv preprint arXiv:2403.09 091444 2024a.
Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. Put myself in your shoes: Lifting the egocentric perspective from exocentric videos. In European Conference on Computer Vision, pp. 407-425, 2024b.
Junho Park, Kyeongbo Kong, and Suk-Ju Kang. Attentionhand: Text-driven controllable hand image generation for 3d hand reconstruction the wild. European Conference on Computer Vision, 2024.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2536-256 2544, 2016.
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands 3d with transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9826-9836, 2024.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- fional Conference on Machine Learning, pp. 8748-8763, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Javier Romero, Dimitris Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, 36(6), 2017.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016.
Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101, A large-scale multi-view video dataset for understanding procedural activities. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2 21096- 21 21106, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 0rXiv:1409.1556 2014.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, and C-C Jay Kuo. Spg-net: Segmen- tation prediction and guidance network for image inpainting. arXiv preprint urXhv.1805.03356. 2018.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. IEEE/CVF Winter Con- ference Applications of Computer Vision, pp. 2149-211111111 2159, 2022.
Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Puttern Analysis and Machine Intelligence, 13(04))376-880 1991.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pitore-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of Machine Learning Research, 11(12), 2010.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. arXiv preprint arXiv:2503. 16551, 2025.
Ting- Ting-Chun Wang, Ming- Yu Liu, Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In IEEE/VV Con- ference on Computer Vision and Pattern Recognition, pp. 8798-8807, 2018.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions Image Processing, 13(4): 600-612, 2004.
Benita Wong, Joya Chen, You Wu, Stan Weixian Lei, Dongxing Mao, Difei Gao, and Mike Zheng Shou. Assistq: Affordance-centric question-driven task completion for egocentric assistant. In European Conference on Computer Vision, pp. 485-501, 2022.
Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo. Foreground- aware image inpainting. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5840-5848, 2019.
13

---

Published as a conference paper at ICLR 2026
Jilan Xu, Yifei Huang, Baoqi Pei, Junlin Hou, Qingqiu Li, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. Egoexo-gen: Ego-centric video prediction by watching exo-centric videos. In International Conference Learning Representations, 2025.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578 47884457,.0 2021.
Zhengdi Yu, Shaoli Huang, Fang Chen, Toby P. Breckon, and Jue Wang. Acr: Attention collaboration-baset regressor for arbitrary two-hand reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2023.
Mengqi Zhang, Yang Fu, Zheng Ding, Sifei Liu, Zhuowen Tu, and Xiaolong Wang. Hoidiffusion: Generating realistic 3d hand-object interaction data. In IEEE/CVF Conference Computer Vi- sion and Pattenn Recognition, pp. 8521-8531, 2024.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0165586999 595, 2018.
Shengyu Zhao, Jonathan Cui. Yilun Sheng. Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint arXiv:2103. 10428, 2021.
14


Reviews:
[{"review_id": "lBw5ycoKR2", "reviewer": "Reviewer_pZGe", "summary": "This paper addresses the task of exocentric-to-egocentric cross-view translation. The authors propose a two-stage training framework to tackle this problem: (1) extracting point clouds, 3D hand poses, and textual descriptions from the exocentric view, and (2) reconstructing the egocentric view based on these extracted cues. The key idea is to leverage multimodal information, including textual supervision, 3D geometric cues, and hand pose data to better guide the exo-to-ego synthesis process. The proposed approach is evaluated on four datasets: H2O, TACO, Assembly101, and Ego-Exo4D.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. \n\n- The idea of leveraging multimodal information—including rich textual supervision, 3D geometric cues, and hand pose data—is simple, intuitive, and effective. \n\n- The evaluation is comprehensive, covering four representative ego–exo datasets.", "weaknesses": "- **Qualitative Results**: Most of the qualitative examples are drawn from the H2O dataset except for Fig 4. It would be beneficial to include more examples from the other three datasets—TACO, Assembly101, and Ego-Exo4D—to showcase a broader range of scenarios beyond desktop activities. This would help readers gain a more comprehensive understanding of the proposed approach’s generalizability across diverse environments.\n\n- **Ablation Study**: In Table 3, the presented ablation results are somewhat limited. Including a more complete ablation table in the supplementary material would provide a clearer and more comprehensive analysis of the contribution of each component within the framework.\n\n- **Backbone Model**: To further improve reconstruction quality, it may be beneficial to adopt more recent video diffusion models, such as Wan or CogVideoX, as the backbone. The current choice, LDM (published in 2022), appears somewhat outdated compared to recent advancements in video generation.\n\n- **Quantitative Evaluation**: The quantitative evaluation primarily focuses on image-based quality metrics. It would strengthen the analysis to include additional metrics assessing hand generation accuracy and object-level generalization, providing a more holistic evaluation of the model’s performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v8uodY15oT", "reviewer": "Reviewer_pmAo", "summary": "The paper proposes a new framework, EgoWorld, for translating third-person views into first-person (ego) view images. Unlike prior methods that depend on multi-view inputs or known ego-camera poses, EgoWorld introduces a two-stage framework. It first extracts multimodal information—such as 3D hand pose, textual description, and depth—from a single third-person image using existing specialized models. These modalities are then integrated into a diffusion-based image reconstruction model to generate the first-person view. Experiments on four public datasets (H2O, TACO, Assembly101, and Ego-Exo4D) demonstrate that EgoWorld achieves state-of-the-art performance and shows promising generalization to unseen scenarios.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The concept of using textual cues to enhance visual synthesis across viewpoints is intuitive and promising.\n\n- The combination of sparse maps as partial observation with textual description for completion is interesting.", "weaknesses": "1. It would be better to highlight the difference compared with 4Diff or Exo2Ego. 4Diff adopts depth maps/point clouds while Exo2Ego adopts hand layout. In addition to depth maps and hand poses, the proposed method adopts an additional textual description for LDM as a condition, which like a combination.\n\n2. The introduction of textual cues is the key of the proposed method and provides semantic alignment. However, the quality of the text and its contribution to the results is unclear. It would be better to include metrics like clip score to measure the alignment between text and generated images. Moreover, in Fig.C, only the incorrect textual description for hand-held objects is shown, which is insufficient. Most importantly, it would be interesting to see the balance between sparse maps and incorrect textual description like Fig.D. Also, those analysis are important and should in the main paper.\n\n3. The comparison with baselines seems unfair. For example, CFLD is primarily designed for pose-guided person image synthesis in general human view translation, not specifically for hand-centric first-person generation. It would be better to incorporate more directly relevant comparisons (e.g., Exo2Ego or EgoExo-Gen). \n\n4. It would be better to provide qualitative results and more analysis for unseen objects or novel environments and to show its strong generalization capability. Based on L.81-84 (\"Yet, it depends heavily...overfitting to the training dataset\"), overfitting is easy to happen and can be alleviated by exocentric observations. However, Fig.4 shows that the images generated by the proposed method are close to the groundtruth even in unknown regions, which also raises the concerns about overfitting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IKXosBsDgH", "reviewer": "Reviewer_u9sS", "summary": "This paper proposes EgoWorld, a two-staged framework that generates an egocentric view from external camera views. EgoWorld extracts multimodal cues, such as textual descriptions, hand pose, and depth map from external views. Then, the egocentric view is estimated sparsely from the pointcloud generated from depth map and relative transform between egocentric and exocentric view. This is further refined using predicted hand pose and a textual description via a latent diffusion model to produce an egocentric image. Method is evaluated on H2O, TACO, Assembly101, and Ego-Exo4D and achieves state-of-the-art performance.", "questions": "* Some experiments in the appendix should instead be in the main paper (e.g., results using prediction instead of GT for H2O).\n* Previous exocentric-to-egocentric translation approaches can be evaluated with the same setup and framework, or partially ablated to use the same input (one camera, single frame) to demonstrate the advantage of the proposed framework.\n* In particular, Exo2Ego's hand-object interaction layout requirement doesn't significantly differ from using hand pose in the proposed method. They can be easily compared by providing a hand-only layout estimated in the same way as the proposed method.\n* The evaluation should show whether using hand pose for relative camera pose estimation is more advantageous than a wider variety of baselines (despite one naive comparison in the appendix). The claim of being camera parameter-free is not significant without it.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Framework utilizes multi-modal information, such as textual description, from an exocentric view in a plausible way.\n* The overall proposed method is well presented with clarity.", "weaknesses": "* Evaluation compares the proposed framework to novel view synthesis methods, many of which are outdated and not specialized for the egocentric setup. This is particularly problematic when there are methods that focus on an egocentric view.\n* Egocentric view transform is estimated based on the egocentric hand pose predicted from the external view. This would be highly dependent on the learned camera setup and the device's visual from an external view. Using ground-truth data in some evaluation further weakens the claim that such a framework is advantageous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "56uctqhZcR", "reviewer": "Reviewer_jkpb", "summary": "This paper introduces EgoWorld, a two-stage framework to translate a single exocentric image into a high-fidelity egocentric view. Stage one extracts geometric and semantic cues (a sparse reprojected point cloud, 3D egocentric hand pose, and text) from the exocentric image. Stage two uses a diffusion model to inpaint and reconstruct the final egocentric view, conditioned on these cues. The method shows great performance across four datasets.", "questions": "See weaknesses for questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses the challenging task of translating third-person to first-person views, a problem with clear applications in AR/VR and robotics.\n- The method design makes a lot of sense to me. It logically separates 3D geometric reprojection (creating a sparse map) from generative inpainting (using diffusion with multimodal cues) to reconstruct the final image.\n- EgoWorld achieves strong results on four different datasets, with thorough testing against unseen objects, actions, scenes, and subjects, including in-the-wild examples. The provided ablation study is helpful.", "weaknesses": "- I feel the method's success critically depends on the 3D egocentric hand pose estimator ($\\phi_{ego}$), which predicts the *egocentric* hand pose from the *exocentric* image. This key component is only briefly described in the main paper. It's unclear what insights make this difficult cross-view prediction possible. More importantly, the paper should justify why it uses hand poses ($P_{exo}$ and $P_{ego}$) for the Umeyama alignment instead of simply predicting the egocentric *head pose*. Using the head pose seems like a much more direct way to find the camera's transformation matrix, and the benefit of the hand-based alignment is not explained.\n- The proposed method is image-based, but the paper's motivation, applications, and datasets are all video-centric. It would be helpful if the authors discuss how this single-frame approach could be extended to a full video-to-video translation framework. A discussion on the expected challenges, such as maintaining temporal consistency across generated frames, would be a valuable addition.\n- The name of the first stage, \"Exocentric View Observation\" is confusing. This stage *outputs* an \"egocentric RGB map $S_{ego}$\" and a \"3D egocentric hand pose $P_{ego}$\". A name like \"Exocentric-to-Egocentric Cue Translation\" would be more accurate and less confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)