Exocentric-to-Egocentric Video Generation
Authors: Jia-Wei Liu
Weijia Mao
Zhongcong Xu
Jussi Keppo
Mike Zheng Shou
Venue: NeurIPS 2024 poster (2024)
Keywords: Exocentric-Egocentric Vision
Video Generation
Viewpoint Translation
Primary area: generative_models
TL;DR: A novel exocentric-to-egocentric video generation method for challenging daily-life skilled human activities.

Abstract: We introduce Exo2Ego-V, a novel exocentric-to-egocentric diffusion-based video generation method for daily-life skilled human activities where sparse 4-view exocentric viewpoints are configured 360° around the scene. This task is particularly challenging due to the significant variations between exocentric and egocentric viewpoints and high complexity of dynamic motions and real-world daily-life environments. To address these challenges, we first propose a new diffusion-based multi-view exocentric encoder to extract the dense multi-scale features from multi-view exocentric videos as the appearance conditions for egocentric video generation. Then, we design an exocentric-to-egocentric view translation prior to provide spatially aligned egocentric features as a concatenation guidance for the input of egocentric video diffusion model. Finally, we introduce the temporal attention layers into our egocentric video diffusion pipeline to improve the temporal consistency cross egocentric frames. Extensive experiments demonstrate that Exo2Ego-V significantly outperforms SOTA approaches on 5 categories from the Ego-Exo4D dataset with an average of 35% in terms of LPIPS. Our code and model will be made available on https://github.com/showlab/Exo2Ego-V.

Exocentric-to--gooennri Video Generation
Jia- Wei Liu'r,
Weijia Maol*, Zhongeong Xu', Jussi Keppo", Mike Zheng Shou
Show Lab, National University of Singapore
Figure I: Given sparse 4 exocentric videos configured 360 around daily-life skilled human activities such as playing basketball (upper), CPR training (lower), our Exo2Ego-V V can generate corresponding egocentric videos with the same activity and environment as the exocentric videos. We encourage readers to click and play the video clips in this figure using Adobe Acrobat.
Abstract
We introduce Exo2Ego-V, V, novel poocentrii-to-egocentee diffusion- video generation method for daily-life skilled human activities where sparse 4-view exo- centric viewpoints are configured 360 around the scene. This task is particularly challenging due to the significant variations between exocentric and egocentric viewpoints and high complexity of dynamic motions and real-world daily-life environments. To address these challenges, we first propose a new diffusion- based multi-view exocentric encoder to extract the dense multi-scale features from multi-view exocentric videos as the appearance conditions for egocentric video generation. Then, we design exocentric-too-ggceniii view translation prior to provide spatially aligned egocentric features as concatenation guidance for the input of egocentric video diffusion model. Finally, we introduce the temporal attention layers into our egocentric video diffusion pipeline improve the tem- poral consistency cross egocentric frames. Extensive experiments demonstrate that Exo2Ego-V V significantly outperforms SOTA approaches on categories from the -xo-Exx4D dataset with an average of 35% in terms of LPIPS, Our code and model will be made available on /
com/ http:/hhowlbbbBa aaw.oolllaaxxxeooooooooo V
Introduction
When people observe and learn skills such as cooking or playing basketball from an exocentric (third- person) perspective, they can easily envision themselves executing these skills from egocentric (first-person) perspective [2]. This
eereentric-eoccntrrii (Exo-Ego) Ego) translation remains the foundation
of visual learning [ for both human beings and AI robots 88 3 37 35). and unleashes new
opportunities for AI assistant [54] 13 and augmented reality 1. However, it remains particularly challenging for computer vision algorithms achieve such exocentric egocentric video generation for difey-life skilled human activities, primarily due to 1) significant variations between exocentric
Equal Contribution
Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and egocentric viewpoints, and 2) high complexity of dynamic motions and daily-life environments, as illustrated in Fig. !
Existing view translation approaches mainly focus on the task of novel view synthesis and have made remarkable progress particularly since the introduction of Neural Radiance Fields (NeRF) [36]. While initially limited to reconstructing static 3D scenes, subsequent studies have extended NeRF to address the challenges of dynamic view synthesis 42 38
39
22 30] However, these 50
27
approaches are limited to per-scene regressive optimization and require dozens or hundreds of views as input. As a result, they fall short in the exocentric egocentric video generation task due to the sparse yet highly variable viewpoints and significant occlusions.
The remarkable success of powerful image diffusion models [45] provides new opportunities for introducing such generative models in the task of exo-ego generation. Recent attempt (33 leverages action intention consisting of human movement and action description for Ego2Exo video generation. However, it strictly requires the first exocentric frame priori which largely simplifies the ego-to- exo generation task to optical flow prediction and exocentric frame warping, highly limiting its applications in general exo-ego generation tasks, On the other hand, Exo2Ego [34] attempts to tackle with the exo2ego view translation by first transferring exo hand pose ego, and then learning the conditional distribution of the target ego image given a single exo image and the predicted ego hand pose. Despite promising, it is limited image-level translation and requires a carefully designed capturing setup with the exocentric camera configured close to the hand-object region and thus is restricted desktop activities with simple environments.
In contrast, we contend that it is necessary for exo-ego translation algorithms to be resilient to the complexity and diversity of daily-life scenarios such cooking in kitchens, playing basketball in courts, etc. The introduction of Ego-Exo4D [15] opens new opportunities and challenges for exo-ego translation by providing a large-scale simultancously-cappureee egocentric and exocentric videos of daily-life skilled human activities. In order to capture the complete and complex human-environnen activities, they configure 4 exocentric cameras in 360" around the dynamic scene, resulting in new challenges of significant variations between exocentric and egocentric viewpoints, as well as complex dynamic motions and daily-life environments, as shown in Fig. [
To tackle with these challenges, we propose novel Exocentric-to-egoeene dubbed as Exo2Ego-V. We address the significant challenges of large viewpoint variations and com- plex environments from two aspects: exo appearance conditions and ego translation prior. Firstly, we propose deffiusion-bssed multi-view exocentric encoder to extract the multi-scate exocentric features as the appearance conditions for egocentric video generation. We achieve this by concatenating ego hidden states with exo features for self-altention computation, so that the ego hidden states can attend to both the egocentric features as well as the multi-view exocentric features through the self-attention mechanism. In addition, we inject the relative position information into our exocentric encoder by adding exocentric latents with relative Exo2Ego relative camera pose embedding. Our exocentric encoder can extract dense human activity and environment information to guide the appearance of egocentric video generation pipeline. Secondly, we introduce an Exo2Ego view translation prior based on PixeINeRF (60] to provide coarse yet spatially aligned egocentric features as concatenation guidance for the input of egocentric video diffusion model. Finally, to improve the temporal dynamic motion consistency of egocentric video contents, we insert temporal layers into our egocentric video diffusion pipeline to encode the temporal information across ego frames.
video diffusion pipeline
We extensively evaluate our Exo2Ego-V V on 5 categories of skilled human activities from the chal- lenging igo-Exx4DD [15] dataset and H2O dataset [26). As shown in Fig. [ our Exo2Ego-V V can generate the corresponding egocentric videos given multi-view exocentric videos, and significantly outperforms SOTA approaches with an average of 35% in terms of LPIPS.
To summarize, the major contributions of our paper are:
We present a novel framework of Exo2Ego-V, V, the first work to achieve exocentric-to- egocentric video generation for daily-life real-world skilled human activities.
We propose a new diffusion- based multi-view exocentric encoder and an Exo2Ego view translation prior that can extract dense exocentric features and spatially aligned egocentric features conditions for egocentric video diffusion pipeline.
Extensive experiments show that Exo2Fgo-V V significantly outperforms SOTA approaches on the challenging Ego-Exo4D [15] dataset with an average of 35% in terms of LPIPS.
2

---

2 Related work
2.1 Egocentric-exocentric vision
Tremendous progress has been made for exocentric vision with various visual perception and genera-
tion tasks due to the large amount of dataset captured at thrid-person views [1] 23 48). Recently, egocentric vision has also been scaling up particularly since the introduction of EPIC- Kitchens [0]
and Ego4D 14]. Previous attempts for joint egocentric and exocentric vision explore learning the ego-eXo view-invariant features on paired small-scale dataset [2] 61] or through unpaired learning [59]. Another line of research focuses on egocentric human localization from exocentric videos [37] 52]. as well as egocentric human pose estimation from exocentric videos [5T]. More recently, the introduction of Ego-Exo4D dataset [15] opens up new opportunities for joint egocentric and exocentric vision with large-scale synchronized multi-view ego-eXo videos with multi-modality annotations.
2.2 Egocentric-exocentii cross-view translation
Ego-exo view translation. There is limited prior work on ego-exo cross-view translation. Early attempt [2 explores the exo-to-ego image generation with novel parallel generative adversarial network to learn shared features of exo and ego images. STA-GAN [29 further extends P-GAN [28
to Exo2Ego video synthesis with a spatial temporal attention fusion module. However, they are limited to simple activities such as walking where most contents in egocentric and exocentric views More recently, IDE [33 leverages action intention consisting of human movement and action description for ego2exo video generation. However, it requires the first exocentric frame a priori which largely simplifies the ego-to-exo generation task to optical flow prediction and exocentric frame warping. On the other hand, Exo2Ego (34) attempts to tackle with the exo2ego view translation by first transferring exo hand pose to ego, and then learning the conditional distribution of the target ego image given single promising, it is limited to image-level translation and requires a carefully designed capturing setup with the exocentric camera configured close to the hand-object region and thus is restricted desktop activities with simple environments.
are static environments [29
image and predicted ego hand pose. Despite
Novel view synthesis (NVS). Ego-Exo view translation is also related to NVS, which has made remarkable progress particularly since the introduction of NeRF (36). While initially limited to reconstructing static 3D scenes, NeRF has been extended to modelling dynamic scenes 42 38
39 50 27
22 32]. However, these approaches are limited to per- scene regressive optimization and require dozens or hundreds of views as input. On the other hand, generalizable scene reconstruction methods [60 fall short in the exocentric egocentric video generation task due to the sparse yet highly variable viewpoints and significant occlusions.
dynamic humans 41
53
(66 are still limited to static scenes. As a result, they
2.3 Video generation
Recent works have extended the power of image diffusion models to video editing [55
43 31 and
generation 6 62 18 20]. Tune-A-Vided (55) inflates the image diffusion with cross- frammf
attention and fine-tunes the source video, aiming to implicitly Iearn the source motion and transfer it to the target video. Video Diffusion Models (DDM) [00] designs a factorized space-tine paace-time UNet to generate videos. Stable Video Diffusion S introduces a systematic data curation workflow, enabling the training of a state-of-the th--te--rttttt-- text-to-vided video and image-to- magge-o--video models. AnimateDiff [16] proposes a plug-and and-play motion module on temporal layers for personalized eext-to-image animation. Other approaches [21 58 introduce such video generation architectures to human image animation and achieve faithful performances. Our Exo2Fgo-V V is new video diffusion pipeline for the challenging Exo2Ego generation on daily-life skilled human activities.
3 Method
3.1
Preliminaries
Latent diffusion models (LDDMs). L.Mss encode input images to latent representation using pretrained variational auto-encoder (VAE) and operate the diffusion and denoising process following
3

---

(b) Multi-View Exocentric Encoder
Cameras
Posd
Edexo
Egu Viewpoints
Concatenation
Exo Spatial Arta
Ego Spatial Artn
Ego Temporal Attn
(c)
Exo2Ego Prior
Ego Feature Fago
Noisy Latents Migo
Cdugp
Figure 2: Overview of Exo2Ego-V. Given 4 exocentric videos configured 360 around daily-life skilled human actívities such cooking (a), multi-view exocentric encoder (b) extracts the multi-scale exocentric features as the appearance conditions for egocentric video generation, and our Exo2Ego view translation prior (c) predicts the egocentric features the concatenation guidance for the egocentric noisy latents input. With these information, our egocentric video diffusion pipeline (d) generates the egocentric vidcos with the same activity and environment as the exocentric vidcos.
denoising diffusion probabilistic models (DDPMs) 19 (see Sec. Sec. A.6 for more details) in the reduced- dimension latent space, and finally decode the denoised latents to the image space. Our model an extention the pretrained text-to-image latent diffusion model (45) that utilizes the UNet architecture [49] for denoising noise prediction with multiple down, middle, and up blocks. Each block consists of ResNet2D layer, a self-attention layer, and cross-attention layer.
3.2 Overall framework
exo configured 360 around dife skilled human activities, our objective to generate their corresponding egocentric video V ago. as shown in Fig. [[a). Since these daily-life activities happen naturally in real-world scenarios such as kitchens, baskethall courts, bike stores, etc, our task features the diversity of the real world with complex activities and environments. Therefore, the sparse 4 exocentric cameras have to be configured evenly in 360 around the dynamic scene order capture both the complex human activities and real-world environments [15), resulting in significant variations between exocentric and egocentric viewpoints. Furthermore, such real- wolld skilled human activities such as cooking, repairing bike, and playing basketball are also highly challenging in terms of the complexity of dynamic motions and real-world environments.
Task definition. Given 4 exocentric videos Vexo V exo
V2
V
V
exn
exo
Overall framework. In order to tackle with the above challenges, we propose novel exocentric- to-egocentric video diffusion pipeline dubbed as Exo2Ego-V, V, as shown in Fig. 2 To address the significant challenges of large viewpoint variations and complex environments,
propose diffusion- based multi view exocentric encoder (Fig. b)) extract the multi-scale multi-view exocentric features as the appearance conditions for egocentric video generation. In addition, we design an exocentri-too-egoeenti view translation prior (Fig. (() based on PixelNeRF (60) to provide coarse yet spatially aligned egocentric features as a concatenation guidance for egocentric video generation. Finally, we introduce the temporal attention layers into our egocentric video diffusion pipeline to improve the temporal consistency cross egocentric frames (Fig. d))
3.3 Multi-view exocentric encoder
Motivation, Our task is featured with the significant variations of Exo2Ego viewpoints and complex environments. The core to tackle with this challenge is to fully explore the multi-view exocentric information for the purpose of guiding the egocentric video generation. A naive solution to utilize the CLIP 44 image encoder to extract latent features from low-resolution images. However, such semantic-level CLIP image features fall short extracting the dense and fine-grained detail information from exocentric videos. Another naive solution is to train 4D dynamic scene reconstruc-

---

tion model for each multi- view exocentric sequences, but it requires high computation and storage resources and current methods cannot handle sparse views complex dynamic scene reconstruction.
Framework. Inspired by recent reference image animation methods [58 21 that extract dense image features with reference UNet preserve reference human identity, we propose multi- view exocentric encoder with a different purpose of extracting the dense multi-view exocentric intricate details as appearance conditions for egocentric video generation. Specifically, our exocentric encoder creates a trainable copy of the base Ego UNet and inject the relative camera poses from exocentric viewpoints to the egocentric viewpoints as additional embeddings, as shown in Fig. 2 We additionally explore adding temporal layers for exocentric encoder in our ablation study.
Given Vexo E RN HxW and their relative camera poses P E N encoder computes the multi-view multi-scale appearance condition features Fexo for the egocentric video generation at denoising step f 0:
exocentric
F Edexoo (Zexc
V exos P, t)
where N 4 is the number of exocentric views and F is the number of frames for each video. Fexo are the normalized attention features for the downsampling, middle, and upsampling blocks of Exo UNet. We set 0 to preserve the appearance details of the noise-free exocentric videos.
Then, the exocentric features F Fixo are utilized as the appearance conditions for egocentric video generation by concatenating Fexo with the corresponding egocentric UNet hidden states Zogo for the self-attention layers in every block at each denoising step t:
Q2 = W bg K, w.K. [Zigo.d: F Fexo,b] Vo = W. [
Lgoobb F Fexo,
(2)
where [-] denotes concatenation operation, and the Ego UNet self- is: Softmax
Qu 9.-K!
.Ve.
The queried egocentric noisy latents can attend to both the egocentric features as well as the multi- view exocentric features through the self-attention mechanism, and thus translate the appearance of complex human skill activities and environments from exocentric views to the egocentric view.
Camera pose. In order to inject the relative position information into our Exo2Ego generation pipeline, inspired by MVDream [47], we embed the relative Exo2Ego camera poses with a 2-layer MLP and add the embedding with the denoising timestep embedding for our multi-vies exocentric encoder. Since we set for our Exo UNet, the relative camera pose embeddings are the main embedding to inject the relative position information into the exocentric feature extraction.
3.4
view translation prior
Motivation. Although our multi-view exocentric encoder can extract dense and intricate appearance details, it entirely relies on the self-attention mechanism to explore the correspondences from the egocentric contents to the exocentric features, which are still challenging viewpoints variations. To tackle with this, we design an Exo2Ego view translation prior based on PixeINeRF [60] to generate a coarse yet spatially aligned egocentric latent feature as the concatenation guidance for egocentric video generation pipeline.
our scenarios with large
Framework. Given multi-view exocentric videos V Vexo. exo camera poses P Pexas egocentric videos V egoo and ego camera poses P ego. we learn an Exo2Ego view translation prior V by training generalizable PixeINeRF (60] for all timesteps. For each synchronized timestep of the exocentric videos and 1 egocentric video, at each iteration we extract 4 exo frames and ego frame and randomly sample rays from these 5 images for optimization. Inspired by ReconFusion (56), we utilize a light PixeINeRF with 6-layer MLPs for higher efficiency. Please see Sec. A.I for more details.
As shown in Fig. cc). with this Exo2Ego translation prior, we can render both the egocentric features F Fego and egocentric pixels Iego given the multi-view exocentric videos, exo camera poses, and queried ego camera pose:
(V exo P
Pexo:
P Pggo)
Iego)
(3)
ego+
Inspired by ReconFusion (56] we design our translation prior to render egocentric features F Fogo at
the egocentric viewpoint with the same spatial resolution as the egocentric latents, so that F oggo is spatially aligned with the noisy egocentric latents. Therefore, we concatenate zego with F Fego along channel dimension the input to the egocentric video diffusion model to predict the noise Cego at
5

---

each denoising timestep t. In addition, we extract the CLIP image feature of the rendered ego images Lgo as the cross-attention information for our egocentric video diffusion pipeline:
Cogo
(F
F Fego]: Fexos Iego t).
3.5 Temporal dynamic motion layer
To improve the temporal dynamic motion consistency of egocentric and exocentric video contents, we follow common practice [8] 20 insert temporal attention layers within the 2D UNet blocks, as shown in Fig. 2 d). Specifically, we insert the temporal layers on the egocentric video generation pipeline and we additionally ablate on inserting the temporal layers on the exocentric encoder. As such, the input egocentric latents zego E RRNxCxFx xW are first reshaped to (N xC for computing the spatial attentions with egocentric and exocentric features in spatial layers, and then reshaped to x(NHW))FFXC to compute the temporal tross-framt information in temporal layers.
3.6 Optimization
We optimize our Exo2Ego-V in a 2-stage training strategy. In the first stage, we remove the temporal layers and optimize the Exo2Ego spatial appearance translation modules, including the multi-view exocentric encoder, Exo2figo view translation prior, as well as the Ego UNet. In the second stage, we only optimize the Ego temporal layers for temporal consistency and freeze other modules.
Exo2Ego spatial appearance translation. We first pre-train our Exo2Ego view translation prior with the pixel-level reconstruction loss CREC. Then, we alternately finetune the Exo2Ego view translation prior with the reconstruction loss CREC, and multi-view exocentric encoder and the Ego UNet with the noise prediction loss Ls.
CREC
Rrender Rggl
Ls = Egg Vexi Pex the
w (t) ||c
Cego Co--
(5)
where Rrender is the rendered ray pixels sampled randomly from exocentric and egocentric frames, Rg the corresponding ground-trutt pixels. is the ego noise sampled from (0. 1). w(t) is weighting function that depends on the noise level t.
Temporal motion finetuning. In the second stage, we freeze the translation prior and Exo and Ego UNets, and finetune the pretrained temporal layers from AnimateDiff [16] on our egocentric and exocentric videos with F frames in temporal dimension.
CT E L.F V. deeee [w" (f) Ile" - eegoll2)
(6)
Experiments
4.1
Dataset
We evaluate our method on 5 categories of Ego-Exo4D dataset [1] featuring both exocentric and egocentric human activities: Cooking, Covid Test, Basketball, CPR, and Bike. Each category contains synchronized captured exo and ego videos of different participants performing these activities at different locations around the world. Specifically, Cooking captures people preparing various dishes in kitchens. Baskethall captures participants playing basketball in courts. Covid Test captures individuals conducting covid tests for themselves in various scenes. CPR captures scenes where participants perform cardiopulmonary resuscitation on a CPR model. Bike captures scenes of participants repairing bikes in bike stores. We the the number of temporal frames to 8 and spatial resolution to 480 x 270 and 256 x 256 for exocentric and egocentric videos, respectively. For each video from the above categories, we extract frames 7.5 fps and split them into multiple action clips according to the action annotations and turning timesteps where the participant's head pose turns more than 45e within second. We retain the videos that contain both ego and exo intrinsic and extrinsic parameters. Finally, we processed 489 videos from Cooking category, 909 videos from Basketball category, 127 videos from Covid Test category, videos from Bike category. The videos lengths vary between 3 ~ 15 minutes for different categories. We also evaluate our method on the H2O dataset (6). which provides synchronized multi-view Exo-Ego images for desktop activities.
videos from CPR category, and 359

---

We train our Exo2Ego-V V (see Sec. A.2 for training details) and baselines for each category and utilize the following 3 test evaluation: (1) Unseen action: We split each video into multiple clips based on the action annotations, so that each clip features different action. We use 80% of action clips as our train set and the remaining 20% unseen action clips test set. (2) Unseen take: Each take refers to complete human activity video. Each participant conduct 2 ~ 4 takes for an activity. We randomly select one take as our test set and use the remaining takes for training. (3) Unseen scenes: Each
category is captured in multiple different scenes around the world. We randomly select one entire scene with multiple takes out of a category as our test set and use the remaining scenes for training.
4.2 Comparisons with SOTA approaches
COOKING
COVID TEST
Ego GT
Ours
SVD
SD
PixeINeRF
[-
t-2
[-4 r-6
[-0
1-2
[=4 1-6
Figure Qualitative comparisons of our method against SOTA approaches on unseen actions.
Baselines. We compare our Exo2Ego-V V with three baselines. (1) Stable Video Diffiusion (SVD) [5], recent state-of-the-ar image-to-video diffusion model. (2) Stable Diffusion (SD) [46), a powerful text-to-image diffusion model. (3) PixelNeRF (60). general 3D scene reconstruction model. We adapt SVD and SD for our Exo2Ego generation task by inputting exо views as their conditions and train the models to generate ego views. Specifically, we first use VAE model to obtain the latent contents of each exo view and concatenate ego noisy latent with these 4 exo latents along the channel dimension as input. Then, we use the CLIP model obtain the exo image CLIP features as the cross-attention information for SVD and SD. We train these three baselines for each categories.
Quantitative results. We report PSNR, SSIM, and LPIPS with AlexNet 25 that measure the differences between generated ego frames and groundtruth on Tab. [ Our Exo2Ego-V achieves the best performance for both unseen actions and unseen takes in terms of all metrics. It noted PixelNeRF 600 achieves good PSNR scores since PSNR favors blurry images 33 as shown in Fig. 3 and 4 Most importantly, our Exo2Ego- significantly outperforms SOTA approaches on all categories with an average of 35% in terms of LPIPS, which clearly demonstrates the superiority of our Exo2ligo-
Table 2: Quantitative compar- ison of our method against SVD on H2O dataset.
PSNR SSIMI LPIPS
SVD Ours
16.530 0.468
0.271
18.600 0.581 0.19
V. We additionally evaluate our Exo2Ego-V and best-performing baseline SVD 5 on the H20 dataset (26) in Tab. 2 Our model still achieves the best performance, which demonstrate the generalizability of our method on different datasets.

---

CPR
COVID TEST
Ego GT
Ous
SVD
SD
PixeINeRF
F
F-2
F6
F-
F-6
Figure 4: Qualitative comparisons of our method against SOTA approaches on unseen takes.
Table 1: Averaged quantitative evaluation different categories. We color code each cell as best
UNSBEN ACTION
COOKING
BASKETRALL.
COVID TEST
CPR
BIKE
PSNR1
SSIM1
LPIPS]
PSNR1
SSIM LPIPS]
PSNRT
SSIM LPIPS.
PSNR1
SSIM1
LPIPS]
PSNR1
SSIM
LPIPS
PixeINcRF 60
17.278 0.412 0.610
08:051
0.512 0.599
19.707
0.548 0.543
18.41
0.561
0.555
16.070
0.351
0.079
SD (6
12.167 0.313
0.583
12.480
0.100 0.005
14,200
0.413
0.538
16.513
0.373
0.454
12.510
0.289
0.577
SVD
14.318
0.407
0.519
15.529
0.491 0.533
16.584
0.507 0.477
17.807
0.630
0.397
14.541
0.364
0.516
17.367
0.408
0.249
21.462 0.668 0.235
0.486
UNSBEN TAKE
COOXING
BASKETBALL
COVID
CPR
BIKE
PSNR1 SSIMI LPIPS]
PSNRT SSIM; LPIPS]
PSNRT SSIM LPIPS
PSNR1
SSIMI
LPIPS]
PSNRT
SSIM
LPIPS.]
PixcINeRF 60
17.177 0.426 0.041
18.744 0.349 0.000
20,402
0.561 0.331
16.674
0.634
0.476
16.129 0.393
0.003
SD
12.512 0.324 0.582
13.018
0.433 0.612
14.651
0.416
0.541
18.143
0.667
0.356
12.510 0.314
0.581
SVD 5
14.554 0.416 0.532
16.
16.563 0.546 0.535
16.875
0.500
0. 0.490
18.302
0.680
0.353
14. 192 0.383
0.527
Ours
17.712 0.504 0.456
21.177 0.616 0.300
21.300
0.687 0.245
18.473
0.711 0.219
16.343 0.441 0.489
Qualitative results. Fig. 3 and visualizes the qualitative comparison of Exo2Ego-V over SOTA
approaches on unseen actions and unseen takes, respectively, where Exo2Ego-V V achieves substantially better egocentric videos quality than other approaches for all categories (see Sec. AA.3 for more results). SVD [5] and SD [46] encounter significant difficulties by conditioning on the highly semantic exo images features to generate egocentric videos. In addition, SD [46] falls short in temporal consistency due to its 2D image-level generation. PixelNeRF [60] renders very blurry results due the significant difficulty of sparse yet highly variable viewpoints and large occlusions. In addition, Fig. 6 visualizes the comparison of our method against SVD (5) on the H2O dataset (26). Our method achieves the best performance and generates photorealistic hand-object interactions. Please see supplementary video for more results on video comparisons, which demonstrates the superiority of our Exo2Ego-V on both much higher spatial appearance quality and temporal consistency compared to other methods.
Table 3: Averaged quantitative evaluation on different categories against baselines for unseen scenes.
UNSEEN SCENE
COORING
BASKETBALL
COVID TEST
CPR
PSNRT
SSIMI LPIPSL
PSNR1
SSIMI LPIPS) PSNR SSIM? LPIPS.
PSNR1
SSIM1
LPIPS
PixeINeRF 60
13.393
0.355 0.700
16,091
0.418 0.876
14.85
0.138
0.697
14.349
0.348
0.745
SD 46
10.617
0.268
0.587
11,686
0.334
0.640
13.072
0.409
0.624
14.937
0.428
0.586
SVD S
11.900
0.321
0.553
14.468
0.385
0.586
14.392
0.484
0.627
14,931
0.447
0.580
Ours
13.926
0.389
0.602
16 201
0. 162
0.300
14.127
0.387
0.004
15.387
0.533
0.651

---

Comparisons on unseen scenes. We con- duct additional experiments on unseen scenes of our Exo2Ego-V V and baselines, and report PSNR, SSIM, and LPIPS with AlexNet [25 that measure the differences between generated ego frames and groundtruth on Tab. 3 We do not conduct experiments on Bike category since it is only captured on different scenes, which are too few to generalize to scenes. Fig. visualizes the qualitative comparison of Exo2Ego-Y V over SOTA approaches seen seenes, Our Exo2Fgo-V V achieves the best performance most metrics in Tab. B and substantially better egocentric videos quality than other approaches shown in Fig. Pix- elNeRF [60] still renders very blurry results but gets good PSNR values since PSNR favors blurry images (38. We also find that it very challenging for all methods to evaluate on the unseen scenes due to the significant variance of new environments compared the training set, and the lack of large-scale scene diversity from the training data.
Ego GT
Chtr
SVD
SD
PixeiNoRF
F-0
1-6
Figure 5: Qualitative comparisons of method
against SOTA approaches on unseen scenes.
H2O
H20
EgoGT
Ours
SVD
2
H
1
Figure Qualitative comparisons of our method against SOTA approaches on H2O dataset.
Ego GT
Ego GT
Ours(full)
exo2ego prior
encoder
temporal layer
F-
t-2
[-5
H
F
[-d
[-6
Figure 7: Qualitative ablation results of our method for cooking category unseen actions.

---

Ours views
Ours views
Ego GT
Ours
views
Ours view
Ours wiExo Clip Feature
Ours Temporal-Spatial Modules
Figure 8: More ablation results of our method for cooking category on unseen actions.
4.3 Ablation study
We conduct ablation studies on the cooking category from Ego-Exo4D dataset [5] We ablate on the proposed multi-view exocentric encoder, Exo2Ego view translation prior, and the temporal layers of multi-view exocentric encoder. As shown in Tab. $ our full model achieves the best performance in terms of PSNR and SSIM. In addition, we provide the qualitative results of our ablations in Fig. 7 which further demonstrates the effectiveness of our designs. Removing exocentric encoder results in inferior performance than full model, which clearly proves its capability in extracting dense and multi-scale exo features for ego video generation. Although removing exo2ego prior achieves the best LPIPS, it results in cleaner but inaccurate egocentric video due to the lack of egocentric guidance, which improves the LPIPS but gets a lower PSNR. As shown Fig. 7 removing exo2ego prior results in the missing of right arm. In addition, we ablate adding temporal layers for our multi-view exocentric encoder and evaluate the temporal consistency by computing the CLIP image embeddings on our generated ego clips and report the average cosine similarity between all pairs of clip frames. Adding the exo temporal layers achieves a higher averaged temporal score 0.924 compared to 0.918 of full model for unseen actions, demonstrating higher temporal consistency but the expense of inferior image-level quality in Tab. Thus, we disgard the exo temporal layer in final model.
We ablate on the number of exo views and replacing our exocentric feature encoder with CLIP features in Tab. 4 and Fig. 8 Our model with exo views achieves the best performance, and our method achieves much better perfor- mance compared to the one using CLIP fea- tures. We also ablate first performing tem-
Table 4: Ablation results of our method.
COOKING
UNSEEN ACTION
UNSEEN TAKE
PSNR1 SSIM1
LPIPS
PSNR1
SSIMI LPIPS
W/ temporal layer
16.702 0.473
0.507
16.754
0.483
0.584
wio Exoccnltic eucodar
16.777
0.155
0.411
16.731
0.150
0.199
Exo2ego pricr
17.268 0.493
[0.364
17.668
0.502
0.401
17.230 0.486
0.383
17.400
0.425
w/ Views
16.930 0.474
0.309
17.020
0.479
0.445
poral attention and then spatial attention for our model. The spatial-temporal model is slightly better than the temporal-spatial model in terms of PSNR and SSIM, and slightly worse for LPIPS. We follow the spatial-temporal attentions [16 5)
w/ Views
17.020 0.478
17.20
0.439
w/ Exo CLIP
16.540
0.156
0. 0-425
16.
0.415
0.180
w/ Temporal sputial
17.000
0.484
0.402
17.290 0.490
0.443
Holl)
0.198
17.712
0.501
5 Conclusion
We introduced a novel framework of Exo2Ego-V V, the first work to achieve Exo2Ego video generation for daily-life real-world skilled human activities. To tackle the challenges, first proposed the new diffusion-baat based multi-view exocentric encoder to extract the dense multi-scale exocentric features as the appearance conditions. Then, we introduced Exo2Ego view translation prior to provide coarse yet spatially aligned egocentric features as a concatenation guidance. Finally, we inserted temporal layers into Ego Unet for improved temporal consistency across ego frames. Exo2Ego-V produced significant improvements on challenging Ego-Exo4D dataset [5 over SOTA approaches.
Limitations and future work. Exo2Ego-V V focuses Exo2Ego video generation several categories of skilled human activities. It remains challenging but is worthwhile researching on more general activities. Exploring Gaussian Splatting as translation prior is also promising direction.
6
Acknowledgment
This project is supported by the Mike Zheng Shou's Start-Up Grant from NUS. Jia- Wei Liu is also supported by NUS IDS-ISEP scholarship. Thanks to Zihang Xia for helpful discussions.
10

---

References
[1] Michael Abrash. Creating the future: Augmented reality, the next suman-machine interface. In 2021 IEEE International Electron Devices Meeting (IEDM), pages 1-2. IEEE, 2021.
[2] Shervin Ardeshir and Ali Borji. An exocentric look at egocentric actions and vice versa. Computer Vision and Image Understanding, 171:61-68, 2018
[3] Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. arXiv preprint arXiv:2207.09453 09400,, 2022.
[4] Homanga Bharadhwaj. Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Zero-shot robot manipulation from passive human videos. arXiv preprint arXiv:2302.02011, 2023.
[5] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint
2023.
[6] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563-22575, 2023.
[7] Joya Chen, Difei Gao, Kevin Qinghong Lin, and Mike Zheng Shou. Affordance grounding from demonstration video to target image. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 6799-6808, 2023.
[8] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 27X1v:330..04137, 2023.
[9] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler. Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pages 720-736, 2018
[10] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100 International Journal of Computer Vision, pages 1-23, 2022.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[12] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 5712-5721, 2021.
[13] Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306. 08640, 2023.
[14] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentrie video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18995-19012, 2022.
[15] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyl- los Afouras, Kumar Ahutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311.18259, 2023.

---

[16] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Ani- matediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprini arXiv:2307.04775, 2023.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18) Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023303 2022.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NewrIPS, 18684006651, 2020.
[20] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646 2022.
[21] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117. 2023.
[22] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from single video. In European Conference Computer Vision, pages 402-418. Springer, 2022.
[23] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 1705.06900 2017.
[24] Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXi::1412.6980, 2014.
[25] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[26] Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Mare Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 101 10138-10148, 2021.
[27] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498-6508, 2021.
[28] Gaowen Liu, Hao Tang, Hugo Latapie, and Yan Yan. Exocentric to egocentric image gener- ation via parallel generative adversarial network. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1843-1847. IEEE, 2020.
[29] Gaowen Liu, Hao Tang, Hugo M Latapie, Jason Corso, and Yan Yan. Cross-view exocentric to egocentric video synthesis. In Proceedings of the 29th ACM International Conference on Multimedia, pages 974-982, 2021.
[30] Jia-Wei Liu, Yan-Pei Cao, Weijia Mao, Wenqiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Devrf: Fast deformable voxel radiance fields for dynamic scenes. arXiv preprint arXiv:2205. 15723, 2022.
[31] Jia-Wei Liu, Yan-Pei Cao. Jay Zhangjie Wu, Weijia Mao, Yuchao Gu, Rui Zhao, Jussi Keppo, Ying Shan, and Mike Zheng Shou. Dynvideo-e: Harnessing dynamic nerf for large-scale and view-change human-centric video editing. arXiv preprint arXiv:2310.
2023.
[32] Hei Liu, Pan Cao, Tianyuan Yang, Eric Zhongcong Xu, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Hosnerf: Dynamic human-ohject-scene neural radiance fields from 2023.
single video. arXiv preprint
12

---

[33] Hongchen Luo, Kai Zhu, Wei Zhai, and Yang Cao. Intention-driver ego-to-exo video generation. arXiv preprint 24Xx:22003.0994,, 2024.
[34] Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. Put myself in your shoes: Lifting the egocentric perspective from exocentric videos. arXiv preprint arXiv:2403.06333 2024.
[35] Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning, pages 651-661. PMLR, 2022.
[36] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99- 106, 2021.
[37] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 2022.
[38] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Bruail. Nerfies: Deformable neural radiance fields. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[39] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint wrXiv:2106.13222 2021.
[40] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[41] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9054-9063, 2021.
[42] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[43] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang. Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2.33.009533 2023.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10674-10685. IEEE, 2022.
[47] Yichun Shi, Peng Wang. Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi- view diffusion for 3d generation. arXiv preprint arXiv:2308. 165/2, 2023.
[48] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[49] Springer. U-net: Convolutional networks for biomedical image segmentation, 2015.
13

---

[50] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12959-12970, 2021.
[51] Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, Diogo Luvizon, and Christian Theobalt. Estimating egocentric 3d human pose in the wild with external weak supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13157-13166, 2022.
[52] Yangming Wen, Krishna Kumar Singh, Markham Anderson, Wei-Pang Jan, and Yong Jae Lee. Seeing the unseen: Predicting the first-person camera wearer's location and pose in third-person scenes. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3446-3455, 2021.
[53] Chung- Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16210-16220, 2022.
[54] Benita Wong. Joya Chen, You Wu, Stan Weixian Lei, Dongxing Mao, Difei Gao, and Mike Zheng Shou. Assistq: Affordince-centricc guestion-driven task completion for egocentric assistant. In European Conference on Computer Vision, pages 485-501. Springer, 2022.
[55] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint 2rr.Xiv:22121111 1565, 2022.
[56] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul Srinivasan, Dor Verbin, Jonathan Barron, Ben Poole, reconstruction with diffusion priors. arXiv preprint arXiv:23 12..222,, 2023.
Reconfusion: 3d
[57] Mingze Xu, Chenyou Fan, Yuchen Wang, Michael S Ryoo, and David Crandall. Joint person segmentation and identification in synchronized first-and third-person videos. Proceedings of the European Conference Computer Vision (ECCV), pages 637-652, 2018.
[58] Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint 1xiv:2331.166990 2023.
[59] Zihui Sherry Xue and Kristen Grauman. Learning finengrained view-invariant representa- tions from unpaired ego-exo videos via temporal alignment. Advances in Neural Information Processing Systems, 36, 2024.
[60] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[61] Huangyue Yu, Minjie Cai, Yunfei Liu, and Feng Lu. First-and third-person video co-analysis by learning spatial-temporal joint attention. IEEE Transactions Partern Analysis and Machine Intelligence, 45(6):6631 16646, 2020.
[62] David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation, 2023.
[63] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.110/6 2022.
14

---

A Supplemental material
A.1 Implementation details
Exo2Ego translation prior details. Inspired by ReconFusion [56]. our Exo2Ego translation prior V/ is based a light PixelNeRF with 6-
(x)
+ f (x)
layer MLPs (Fig. 9 for higher efficiency to-
Point position
Latent feature
gether with a ResNet34 [17] pretrained on the ImageNet dataset to extract exo image features. For each synchronized timestep of the 4 exocen- tric videos and 1 egocentric video, we extract 4 frames and 1 ego frame and randomly sample 128 pixel rays from these 5 images and sample 3D points for each iteration. Then, add positional embedding to these points (x) and query their latent features f (x) by project- ing them on the latent images. Then we con- catenate them with viewing direction as input to our translation prior as in Fig. 19] We conduct volumetric rendering at the third layer for the latent features, and then input the latent features the last 3 layers for final pixel color and latent features.
128
View direction
4
128
129
Density
128
128
131
RGB Color
Ego feature
Figure 9: Network details for our Exo2Ego trans- lation prior.
For egocentric video generation, only sample rays from 32 32 ego frames that the rendered ego features are spatially aligned with the noisy ego latent. Therefore, we concatenate the rendered ego features with the noisy ego latent the input to our egocentric video diffusion pipeline.
Egocentric camera unprojection. The ego camera from Ego-Exo4D dataset [15] utilizes the FisheyeRad ThinPPrsm (Fisheye624) model, which accounts for thin-prism distortion. This model includes four additional coefficients: 8o, $1, 82, $3. The projection function is:
u fx * (u, + (up, vz) + tpx (Ur, vz)) + Cx fy (vr tx(u,, v,) + (ur, vr)) + Cy,
tpx (ur, Vz) Sor(0) + +(r()))
tpy Up, Vr) 52r(0)² + sa)(
(7)
r(0) = (u- /x2 (x==xx²//F + (v -cy)2/Fg 3/13.
arctan((u - cx) Ex, (v - -y)/fy))
u, V are the camera pixel coordinates and d, 0 are the world point. fx, fy are the focal lengths. Its parametrization contains additional coefficients: So. S1, S2, S3. Firstly we use the Newton method to calculate the ur, Vr and then calculate o. 0 using the above unprojection method. Finally, sample 3D points along the calculated directions d,
A.2 Training details
We optimize our Exo2Ego-V V using Adam optimizer (24). We set the learning rate of the multi-view exocentric encoder and the egocentric diffusion model as 0.00001, and we set the learning rate of view translation prior as 0.0001. We first train the translation prior with 500K iterations on a single A100 GPU for 36 hours, and then optimize our Exo2Fgo spatial appearance translation with 500K iterations on A100 GPUs for hours, and finally finetune our temporal motion module with 100K iterations on 8 A.00 GPUs for 40 hours, all using the PyTorch [40] deep learning framework.
15

---

BASKETBALL
BIKE
Ego GT
Ours
SVD
PixeINeRF
[-()
F-2
E4
F6
F=0
F-2
F4
F
Figure 10: More qualitative comparisons of our method against SOTA approaches on unseen actions.
A.3
Additional qualitative results
We provide more qualitative comparisons of Exo2Ego-V over SOTA approaches on unseen actions in Our method achieves the best performance across all approaches.
Fig. 10
A.4
Feature Visualization
COVID TEST
C72
In Fig. we present our ego feature visual- ization results using the Exo2Ego prior. The visualization results clearly represent the con- tents of the ego views. This indicates that our Exo2Ego prior can effectively extract and trans- mit the important information from ego views to the multi-view exocentric encoder.
Exo2ligo Prior Renderl Featuees
EuddEgo Priur Rendered Piuds
Figure 11: Exo2Ego prior feature visualization.
A.5 Reasoning Efficiency
We provide the inference time comparison shown in Tab. where the inference time of our method to generate an 8-frame egocentric video is 9.06 second, which is comparable with other baselines. We believe is feasible to use our model offline applications generate egocentric videos from the exocentric videos, such as capturing exocentric cooking videos and generating corresponding egocentric videos offline for cooking skills learning. Improving the inference speed towards real-time is very promising and we leave it as future works.
Table 5: Inference time of our method in comparison with baselines.
Ours SVD [
PixeINeRF 60
Inference time (sccond) 9.06
4.26
6.91
5.65
A.6 Preliminary denoising diffusion probabilistic models (DDPMs)
DDPMs [19 are generative frameworks designed synthesize data by reproducing consistent forward Markov chain 21,..., The process begins from random noise distribution and progres-
16

---

sively denoise the noisy contents to the clean data. Considering a data distribution as g(2o), the
Markov transition (zz, is conceptualized Gaussian distribution by variance 8, E (0, 1):
(x-1) = (Mx;; v - B, B,Il), 1= 1
(8)
Under the Bayes and Markov principles, the conditional probabilities can be derived as:
(00))=0) = (a Varto, (1 T.) (0 N ( mo): 3,11), L = 1,
7.
(9)
where O₁ = 1 - Br, ai IT. Os. 3, - Bt, (2t, = Varc T-a; To + vas(1-ār-1)
l-
It.
DDPMs utilize a reverse approach to synthesize the chain L1, "TT.
po(#t-1(æt) = ((x (x f), (x f)), T, I.,.
(10)
The model parameters are optimized ensure the synthesized reverse sequence aligns with the forward sequence.
A.7 Broader impacts
Our work generate egocentric videos from exocentric videos. Since the scale of egocentric dataset still much less than the exocentric dataset, our method has the potential to improve the egocentric vision such egocentric perception by generating more egocentric data from the exocentric data. Our Exo2Fgo-V also support applications on Al assistant and augmented reality by generating egocentric videos from exocentric videos. We believe our method will not bring negative societal impacts.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main contribution of our paper is clearly described in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This paper does not contain any theoretical proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide dataset details in Sec. 4.1 and implementation details in Sec. A.] We use publicly released Ego-Exo4]] dataset for experiments, and our code and model will be made public if this paper gets accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We use publicly released Ego-Exo4D dataset for experiments, and our code and model will be made public if this paper gets accepted. We also provide dataset details in Sec. 4.1 and implementation details in Sec. A.1
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
aubllic/guudess/odeeSuiiiiss sow.:ssubbbsiiooooooocooo. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//abps.cc/public/uuiiee htarbb/ii/ggdddd//odeeeeeeeeeeee
swssssssiinyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experimental setting in Sec. 4.1 implementation details in Sec, [A.] and training details in Sec. A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We omitted error bars from our analysis due the excessive computational expense involved in enumerating all experimented categories in the dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss computer resource information in Sec. A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurrps.. wttpuurriipccppppupeeeeeeee hwtpp///bbcct/iicccciiceeeeeeoee
Answer: [Yes]
Justification: We conduct experiments with publicly released dataset Ego-Exo4D that preserve anonymity of participants. Our research conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential societal impacts in Sec. A.7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not have such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly cited papers and sources for existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide asset documentation alongside our code and model when release them.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We do not involve crowdsourcing nor research with human subjects. We only use publicly released Ego-Exo4D dataset for experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not involve crowdsourcing nor research with human subjects. We only use publicly released Ego-Exo4D dataset that already handled IRB approvals for experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "SxYrNOHBYc", "reviewer": "Reviewer_W4kZ", "summary": "This paper deals with the task of exocentric-to-egocentric video generation. It presents a diffusion-based framework of exo2ego-v to tackle the challenges of the significant variations between exocentric and egocentric viewpoints and high complexity of dynamic motions and real-world daily-life environments. It propose a multi-view exocentric encoder to extract the multi-scale multi-view exocentric features as the appearance conditions for egocentric video generation. It also designs an exocentric-to-egocentric view translation prior based on PixelNeRF to provide coarse yet spatially aligned egocentric features as a concatenation guidance for egocentric video generation. Experiments on a part of Ego-Exo4D dataset show the effectiveness of the proposed method.", "questions": "1) Do you train one Exo2Ego prior model for each video, or for each synchronized timestep of a video? It is not clear for me how regular the Exo2Ego prior model need be retrained.  \n2) What is the dimension of the encoded features for multi-view exocentric videos, and how are they combined with the camera poses which have different dimensions?  \n3) How are the egocentric camera poses determined during testing? Do you directly use the poses provided by the dataset?  \n4) In Figure 5, the same results of EgoGT are shown twice. Please have a check. \n5) Current evaluation metrics are averaged over the generated video. I wonder how the performance would change over time.", "limitations": "The paper can also show the runtime of the video generation framework.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper presents a diffusion-based framework to first address the challenging task of exocentric-to-egocentric video generation for real-world skilled human activities.  \n2) To tackle the challenges of exocentric-to-egocentric video generation, the paper proposes a new diffusion-based multi-view exocentric encoder and an Exo2Ego view translation prior that can extract dense exocentric features and spatially aligned egocentric features as conditions for the egocentric video diffusion pipeline.  \n3) Experiments on on 5 categories of skilled activities on the Ego-Exo4D dataset show that the proposed framework achieves best performance most evaluation cases.", "weaknesses": "1) The presentation of the paper needs further improvement.  \n    1.1 For the method part, the network architecture is not clear for the exocentric encoder and the egocentric video diffusion model. As a result, the feature dimension of many notations such as $F_{exo}$, $Z^{t}_{exo}$ are not known. It is also difficult to understand the procedure within Equation (1) and (4).  \n    1.2 For the experiment part, the qualitative results have taken too much space (e.g., Figure 3). Space should be saved for the presentation of the results on the unseen scenes (which are moved to appendix).  \n    1.3 If the test scenes are not seen during training, Exo2Ego translation prior (i.e., pixelNeRF) need be trained for the new scenes. During testing, egocentric video frames are not available, while it is said in the paper that four exocentric frames and one egocentric frame are used to train the pixelNeRF.  \n2) As shown by the results on the more challenging unseen scenes (Table 3), the proposed method sometimes gets worse performance than the baseline methods. The authors should give more analysis and discussion for such results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for recognizing our work and the valuable comments.**\n# W1.1: Network architecture.\nThe base network architecture of the exocentric encoder and the spatial modules of the egocentric video diffusion model are extensions of the Stable Diffusion [44]. Therefore, they consist of **four downsample layers, one middle layer, and four upsample layers, where each layer includes 2D convolution, self-attention, and cross-attention**. Since Stable Diffusion operates in the latent space, the feature dimension of input encoded noisy latents is $\\mathbb{R}^{N\\cdot F\\times C\\times\\frac{H}{8}\\times\\frac{W}{8}}$.\n\nThe exocentric attention features’ dimensions are different for different downsample, middle, and upsample layers with channel dimension varying at [320, 640, 1280], and the spatial dimensions $\\frac{H}{8}\\times\\frac{W}{8}$ are downsampled by 2x2 or upsampled by 2x2 for each downsample and upsample layers, respectively. Eq. (1) and (4) are the **denoising processes of diffusion models** that predict the added noise of input noisy latents, except that in Eq. (1) we extract the exocentric features from each denoising layer. We will add these details to the main paper.\n# W1.2: Appendix results.\nThanks for your advice. We will move the unseen scenes results to the main paper and move some qualitative results to the appendix.\n# W1.3, Q1: Exo2Ego prior training.\n**We only train one single generalizable Exo2Ego prior model for all exocentric videos and egocentric videos from the training set.** **For each training iteration, we randomly extract synchronized 4 exocentric frames and 1 egocentric frame to supervise our Exo2Ego prior (L197-L200).** During the complete training, each iteration will load different Exo-Ego videos from the training set to train our single generalizable Exo2Ego prior. Therefore, we only need to train one single Exo2Ego prior from all videos from the training set.\n\nDuring testing, our Exo2Ego translation prior can **generalize to test scenes that are not seen during training without retraining.** Therefore, we do not need to train the Exo2Ego translation prior for novel test scenes. As shown in our experiments, we evaluate our Exo2Ego-V on unseen actions, unseen takes, and unseen scenes without retraining our Exo2Ego prior on any of the test sets, which is very challenging. We will explain it clearer in our final paper.\n# W2: Unseen scenes results.\nThe evaluation of the unseen scenes is particularly challenging for two reasons. 1) **We only train one single Exo2Ego-V model**, including the Exo2Ego prior, from the training dataset and evaluate its performance on the proposed 3 challenging test setups without any retraining or finetuning. 2) **The number of different scenes from Ego-Exo4D is relatively limited, but the geographic diversity of the captured scenes is particularly significant across the world**. Therefore, all methods perform worse on the unseen scene evaluation.\n\n**Despite these challenges, our method still outperforms the baseline methods in most metrics for the unseen scene evaluation.** As shown in Tab. 3 of the main paper, PixelNeRF only achieves the best PSNR metric for the Covid Test category, but it generates very blur results as shown in Fig. 7 since PSNR favors blurry images [37]. SVD occasionally achieves better metrics results but it only generates wrong scenes without any human-object interactions, as shown in Fig. 7 of main paper. In comparison, **our method is the only method that tries to generate both scenes and human-object interactions, such as the background and human arms and basketball in Fig. 7**. Although the performance on unseen scenes is not yet optimal, we believe jointly training our method with a more diverse multi-view dataset has the potential to improve performance and we leave this as future work.\n# Q2: Exocentric feature dimension.\nThe exocentric attention features’ dimensions are different for different downsample, middle, and upsample layers with channel dimension varying at [320, 640, 1280], and the spatial dimensions $\\frac{H}{8}\\times\\frac{W}{8}$ are downsampled by 2x2 or upsampled by 2x2 for downsample and upsample layers.\n\nAs explained in L183-L187, to encode the relative camera poses information, we do not directly concatenate the camera poses with exocentric features. Instead, **we first encode the camera pose to 1280 dimension with a 2-layer MLP, and then add it to the denoising timestep embedding as the residuals for the noisy latents** in the following denoising blocks. In these following denoising blocks, there are additional MLP layers to map the channel dimension of the timestep and camera embedding to the same dimension as the corresponding noisy latents, so that these embeddings are added to the noisy latents as residuals.\n# Q3: Egocentric camera poses.\nYes, we directly use the camera poses provided by the Ego-Exo4D dataset.\n# Q4: EgoGT Results.\nThanks for your advice, we will remove the repeated EgoGT which was intended for symmetrical layout.\n# Q5: Performance over time.\nFollowing your suggestion, we compute the PSNR for each frame of the generated videos over time, and then average these metrics across all test videos for each time step respectively. We evaluate this on the three test sets of the Cooking category and compute all metrics. We report PSNR due to space limits, and SSIM and LPIPS show similar results. As shown in following table, **the performance remains relatively stable over time, and the intermediate frames perform slightly better than the final frames due to larger motion in the latter.**\n\n*Tab. 1 Quantitative metrics (PSNR) over time for Cooking category.*\n||Frame 1|Frame 2|Frame 3|Frame 4|Frame 5|Frame 6|Frame 7|Frame 8|\n|-|-|-|-|-|-|-|-|-|\n|Unseen Action|17.35|17.42|17.44|17.44|17.43|17.35|17.38|17.27|\n|Unseen Take|17.63|17.78|17.81|17.76|17.79|17.86|17.71|17.60|\n|Unseen Scene|14.01|14.07|14.07|14.02|13.95|13.88|13.80|13.69|\n# L1: Inference time.\nPlease refer to common response 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j0NdGJa5cs", "reviewer": "Reviewer_ANQv", "summary": "This paper proposes a novel diffusion-based video generation method that translates exocentric views to egocentric view. Overall, I think the idea is novel and the method performs well on multiple daily human activities.", "questions": "1. The proposed method outperforms the baseline SVD that performs video generation given the first frame as condition. It is not clear to me whether the authors also adopted first frame as the condition while training the model since the first frame is a very strong prior. If not, could the authors explain why the proposed method can outperform SVD without using such strong prior, when both methods are given the same exocentric views as conditions.  \n2.  It is not clear whether the proposed method can be generalized to other egoexo datasets that also contain camera poses, e.g. H2O [1], Assembly101[2]. \n\n[1] Kwon et al. H2O: Two Hands Manipulating Objects for First Person Interaction Recognition. ICCV21.\n\n[2] Sener, Fadime, et al. Assembly101: A large-scale multi-view video dataset for understanding procedural activities.CVPR 2022.", "limitations": "Please refer to Weaknesses and Questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe application of view translation using Nerf-based approach seems interesting, despite minor improvement on several metrics.  \n2.\tMultiview exocentric encoder and temporal modeling is proven effective in supporting view transfer.\n3.\tThe paper is well written and easy to follow.", "weaknesses": "1.\tIn Line 157-159, the authors mentioned that encoding multiview exocentric videos using CLIP image features lacks fine-grained details. However, there is no sufficient evidence in the experiments. For example, comparing the proposed method with global CLIP features or intermediate spatial CLIP features as conditions. \n2.\tGenerally, as most of the takes in EgoExo4d contain 4 exocentric views, which are significantly fewer than dozens of views used in 3D reconstruction approaches. The effectiveness of reconstructing a 3D scene using such sparse viewpoints is questionable, not to mention the significant camera pose differences between egocentric and exocentric views.\n3.\tIn the experiments, the rendered pixels of PixelNerf are extremely blurry, and thus it is not very convincing to use the CLIP features of rendered egocentric image in Eq.(4). To prove the effectiveness of exo2ego prior, the authors are encouraged to show some visualization results of the rendered egocentric features using PixelNerf.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for recognizing our work and the valuable comments.**\n\n# W1: Ablation on Exocentric CLIP features.\nThanks for your advice. We conduct additional ablation study by replacing our exocentric feature encoders with the CLIP exocentric features. As shown in the following table, **our method achieves much better performance compared to using CLIP features across all metrics on the unseen action and unseen take evaluations.** This clearly demonstrates the superiority of our multiview exocentric encoder in maintaining fine-grained details compared to the CLIP features. We also provide **qualitative ablation comparison on Fig. 4 of rebuttal pdf**, where our method with Exo encoder outperforms the one with CLIP features on many detailed regions such as left and right human arm and objects. We will add this ablation result to our final paper.\n\n|                    | Unseen Action |       |       | Unseen Take |       |       |\n|--------------------|---------------|-------|-------|-------------|-------|-------|\n|                    | PSNR↑          | SSIM↑  | LPIPS↓ | PSNR↑        | SSIM↑  | LPIPS↓ |\n| Ours w/ Exo Encoder | **17.37**    | **0.493** | **0.408** | **17.71** | **0.504** | **0.456** |\n| Ours w/ Exo CLIP    | 16.54         | 0.456 | 0.425 | 16.41       | 0.445 | 0.480 |\n\n# W2: Only 4 exocentric views are very challenging.\nThe 4 sparse views and significant differences between exocentric and egocentric cameras are **exactly the key challenges for our Exo2Ego video generation task**. Therefore, **previous 3D reconstruction methods such as PixelNeRF fail in this task**, as shown in the baseline PixelNeRF results in Fig. 3 of main paper. To tackle these challenges, we propose our complete Exo2Ego-V pipeline with both multi-view exocentric encoder to extract multi-scale exocentric features and Exo2Ego translation prior to render coarse yet spatially aligned egocentric feature maps. **These two designs complement each other: the exocentric features are fine-grained but not spatially aligned with the ego input, while the egocentric features are a bit coarse but they are spatially aligned with the ego input.** Our experiments demonstrate that our Exo2Ego-V pipeline achieves much better performance than previous reconstruction and generation baselines for such a challenging task with large Exo-Ego viewpoint differences.\n\n# W3: Visualization of rendered egocentric features.\nAs shown in Fig. 6 of appendix, our Exo2Ego prior rendered both the egocentric features and images. We want to clarify that we utilize two types of egocentric features: the egocentric features rendered directly from the Exo2Ego prior, and the CLIP features of the egocentric images rendered from the Exo2Ego prior. The first one is spatially aligned with the egocentric input latent and is directly concatenated with the input to provide spatial guidance, and the latter is utilized as the cross attention computation to maintain the original diffusion architecture.\n\nFollowing your suggestino, we provide more **visualizations of rendered egocentric images and rendered egocentric features on Fig. 2 of rebuttal pdf**. As shown in the figure, **the rendered feature maps can accurately encode the geometric information such as the desktop shape, and the CPR models**. Our rendered features are spatially aligned with the egocentric inputs and are complemented with the fine-grained exocentric features extracted from our exocentric encoder. Although the rendered feature maps could be coarse for complex scenarios, but they are spatially aligned egocentric features (as mentioned in L150-L151 of main paper) can provide spatial guidance for our egocentric generation pipeline,  and our ablation studies also demonstrate the effectiveness of our proposed Exo2Ego prior. \n\n**We conducted an ablation study on our proposed Exo2Ego translation prior in Tab. 2 and Fig. 5 of the main paper, where removing Exo2Ego prior results in worse performance for PSNR and SSIM**. In addition, **through the visualization of Fig. 5, removing Exo2Ego prior (second row, second column) results in the missing of right human arm.** These ablations **demonstrate the effectiveness of our Exo2Ego prior.**\n\n# Q1: SVD training.\n**For fair comparisons, we provide the same 4 exocentric videos as inputs for our method and all baseline methods including SVD.** Therefore, **we modify the condition blocks of SVD to take 4 exocentric videos as input and finetune the pretrained SVD model on the Ego-Exo4D dataset.** As such, all methods are given the same exocentric videos as inputs to generate corresponding egocentric videos, and they are all trained and tested on the same dataset to achieve fair comparisons among each other.\n\n# Q2: Generalization on other EgoExo datasets.\n\nThanks for your advice! Our method can be generalized to other EgoExo datasets. Following your suggestion, **we additionally train and evaluate our method on the H2O dataset, and compare our method with the SOTA baseline - SVD on the following table**. As shown in the following table, **our method achieves much better performance than SVD, with significant 30.3% improvement over SVD in terms of LPIPS**. **We provide qualitative comparisons of our method against SVD on Fig. 1 of rebuttal pdf**, where **our method achieves much more photorealistic and accurate results than SVD**, such as the details of human hands, interacted objects, and the environments. **These can demonstrate our method can be generalized to other datasets**, and we will add this result on the final paper.\n\n|      | PSNR↑      | SSIM↑      | LPIPS↓     |\n|------|-----------|-----------|-----------|\n| Ours | **18.60** | **0.581** | **0.189** |\n| SVD  | 16.53     | 0.468     | 0.271     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VyhxmNxP9x", "reviewer": "Reviewer_qWdm", "summary": "This paper proposes a novel method for generating egocentric videos from multi-view exocentric videos using diffusion-based techniques. This method addresses the challenges of viewpoint variation and dynamic motion by employing a multi-view exocentric encoder and a view translation prior, along with temporal attention layers to enhance temporal consistency. Experiments results on the Ego-Exo4D dataset verify the effectiveness of the proposed method.", "questions": "After reviewing the response letter, some of my concerns have been partially addressed. However, the paper's novelty, primarily inspired by ReconFusion, appears to have a weak technical contribution. Additionally, the evaluation is limited by the dataset used. Consequently, I have adjusted my score to a the borderline reject.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe logic of the paper is reasonable.\n\n2.\tThe motivation is interesting.", "weaknesses": "1.\tThe exocentric-to-egocentric view translation prior is very similar to the existing work ReconFusion, and the authors need to clarify the differences.\n\n2.\tIn the egocentric video generation pipeline, why perform spatial attention before temporal attention instead of following the TimeSformer? Are there relevant experiments for further clarification?\n\n3.\tThe reasoning efficiency of the whole process should be evaluated. This determines whether it can be practically applied.\n\n4.\tHow does the number of exocentric videos affect performance? In real scenarios, it is difficult to capture 4 time-synchronized exocentric videos at the same time. Besides, do the 4 exocentric videos have to be evenly distributed around the scene?\n\n5.\tThe evaluation dataset is so single that it is impossible to validate the generalizability of the method on more diverse data. Furthermore, Ego-Exo4D contains a wide range of human activities, why only 5 of these categories are selected for the experiment?\n\n6.\tThe baselines compared are too few to adequately validate the superiority of the method.\n\n7.\tWriting needs further improvement, tenses should be correct (Line 185, Page 5) and contextual expressions need to be consistent (Unet or UNet).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for helpful comments.**\n# W1: Differences with ReconFusion.\nOur Exo2Ego prior is inspired by ReconFusion and based on PixelNeRF (see L199-L200, L204-L206 of main paper), but it significantly differs from ReconFusion in following aspects.\n1) **The task of our method is significantly different from ReconFusion.** ReconFusion targets 3D static scene reconstruction, and thus it trains a diffusion prior on 3D static datasets and explores how to distill their diffusion prior into a 3D model through regularization losses. In contrast, our method is the first work to approach the challenging exocentric to egocentric dynamic video generation for skilled human activities with significant differences between exocentric and egocentric viewpoints.\n2) **The design of our Exo2Ego prior is different from ReconFusion’s diffusion prior.** ReconFusion directly trains a diffusion prior that includes a PixelNeRF and diffusion UNet, and then explores distilling from the pretrained diffusion prior. In contrast, our method first trains a PixelNeRF-based Exo2Ego prior, and then we integrate this Exo2Ego prior into our complete Exo2Ego video generation pipeline and finetune Exo2Ego prior to provide the coarse yet spatially aligned egocentric features for our egocentric video generation. In addition, the PixelNeRF from the diffusion prior of ReconFusion is a 2-layer MLP, while our Exo2Ego prior is a 6-layer MLP.\n3) **We further propose the multi-view exocentric encoder to provide fine-grained multi-scale exocentric features for egocentric video generation. Our multi-view exocentric encoder and Exo2Ego prior complement with each other.** Our complete Exo2Ego video generation model is the first video-level model for Exo2Ego video generation, and achieves new SOTA. \n# W2: Ablation on temporal-spatial.\nWe conduct an ablation to first perform temporal attention and then spatial attention for our model. As shown in the following table, the original spatial-temporal model is slightly better than the temporal-spatial model in terms of PSNR and SSIM, and slightly worse for LPIPS. Since previous video generation methods such as AnimateDiff [16], SVD [5] all follow the spatial-temporal pipeline, our method also follows such spatial-temporal attentions. It is noted that since our temporal layers are finetuned from AnimateDiff [16] that is a spatial-temporal pipeline, our spatial-temporal model performs slightly better than temporal-spatial pipeline. **We provide qualitative results on Fig. 4 of rebuttal pdf where ours generates more accurate human-objects.**\n||Unseen Action|||Unseen Take|||\n|-|-|-|-|-|-|-|\n|| PSNR↑| SSIM↑| LPIPS↓| PSNR↑| SSIM↑| LPIPS↓|\n|Spatial-Temporal| **17.37**| **0.493**|0.408|**17.71**|**0.504**|0.456|\n|Temporal-Spatial| 17.00| 0.484|**0.402**|17.29|0.490|**0.443**|\n# W3: Reasoning efficiency.\nPlease refer to common response 2.\n# W4: Number of exocentric videos.\n**We conduct additional ablations on the number of exocentric views of 4, 3, 2, 1 views on the following table**. Our method’s performance slightly drops with the reduction of exocentric views. We provide qualitative comparisons on Fig. 5 of rebuttal pdf. **Even with only one exocentric view, our performance is still much better than other baselines, which significantly demonstrates the effectiveness of our approach with various numbers of exocentric views. This avoids the necessity of capturing time-synchronized exocentric videos at the same time, making our method more applicable for real-world applications**.\n||Unseen Action|||Unseen Take|||\n|-|-|-|-|-|-|-|\n||PSNR↑|SSIM↑|LPIPS↓|PSNR↑|SSIM↑|LPIPS↓|\n|Ours w/ 4 Views|**17.37** |**0.493**|0.408|**17.71**|**0.504**|0.456|\n|Ours w/ 3 Views|17.23|0.486|**0.383**|17.40|0.489|**0.428**|\n|Ours w/ 2 Views|16.93|0.474|0.399|17.02|0.479|0.445|\n|Ours w/ 1 View|17.02|0.478|0.395|17.20|0.476|0.439|\n# W5: Evaluation dataset.\nWe selected Ego-Exo4D dataset [15] as our main evaluation dataset due to its significant challenges of the complexity and diversity of daily-life scenarios, large differences between exo and ego viewpoints, complex human-object interactions, and its largest Ego-Exo data scale with 1286 hours of video. **Ego-Exo4D has 8 domains: Cooking, Health, Bike Repair, Music, Basketball, Rock Climbing, Soccer, Dance. It is different from Ego4D and does not contain a wide range of activities**. Since our task focuses on Exo2Ego video generation, **We select 5 categories that significantly emphasize both egocentric and exocentric activities, such as human-object interactions in both Ego and Exo viewpoints (see L235-L238 of main paper)**. Others like Dance, Soccer, Rock Climbing are mainly exocentric activities and contain very few human or human-object interactions in the egocentric view, which weakens the usefulness of generating egocentric views and thus are not included in our experiments.\n\n**Our method can be generalized to other EgoExo datasets. Please refer to common response 1 for more results on H2O dataset.**\n# W6: Baselines.\nOur method is the first work to achieve Exo2Ego video generation for daily-life real-world skilled human activities. **This task is particularly challenging and there are no available baselines to compare with.** Therefore, we design three baselines including the **SOTA open-sourced video generation model** SVD, the **image generation method** SD, and the **3D-based method** PixelNeRF. We modify the input and condition modules of these baselines and train and evaluate them on the Ego-Exo4D dataset. We selected SVD because it is the SOTA open-sourced video generation model. **Therefore, we believe our baselines are adequate to demonstrate the superiority of our method. We additionally inference two more pre-trained image-to-video and video-to-video generation models on the Ego-Exo4D dataset as shown in Fig. 6 of rebuttal pdf, where these methods totally fail to generate egocentric videos from exocentric videos input**.\n# W7: Writing.\nWe have corrected such typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qxGcD8BjIF", "reviewer": "Reviewer_aauy", "summary": "This paper introduces a novel method for generating egocentric videos from exocentric videos of daily-life skilled human activities. This task is very challenging because of the significant viewpoint variations, sparsely posed exo videos, and dynamic environments. The authors propose to use a PixelNeRF-informed diffusion method to address the problem and achieves good visual performance and SOTA accuracy.", "questions": "Could you provide a failure case analysis? Do you think the fine-grained hand-object interactions causes most of the failure cases?", "limitations": "Limitation included", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper addresses a challenging problem and achieves great performance. The proposed method is novel and reasonable that effectively leverages the camera poses and multi-view video features. The experiments show that it achieves the SOTA accuracy.", "weaknesses": "The proposed method lacks specific designs tailored for the \"Exocentric-to-Egocentric\" task. \"exo\" and \"ego\" perspectives are relative to \"the person\" in the video, yet the proposed method does not include any design elements focused on the person. The method actually has broad applicability to novel view video synthesis with significant viewpoint variations. One of the goals of the \"Exocentric-to-Egocentric\" task is to generate dynamic hand motions and object state changes. However, the visualization results in the paper show bad hand-object motions synthesis result. In summary, while the model excels at reconstructing the geometry of the surrounding environment, it lacks focus on hand-object dynamics which is important in egocentric video.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for recognizing our work and the valuable comments.**\n\n# W1: Person-related design.\n\nThanks for your advice. We totally agree with you that person-related designs such as human hand pose prior or object states information would be very useful to improve the performance of Exo2Ego video generation. However, our method is the first work to achieve exocentric-to-egocentric video generation for daily-life real-world skilled human activities. For such challenging daily-life activities in real-world environments such as cooking in the kitchen, playing basketball on the court, **the exocentric cameras have to be placed far from the human** in order to capture the complete human activities. As a result, it is particularly difficult to capture detailed human hand-object regions on the exocentric cameras. **Therefore, it is particularly challenging to obtain human hand pose prior information or object state information from exocentric cameras from preprocessing**, and **generating human-object interactions is also one of the particular challenges for our Exo2Ego video generation task.**\n\nTherefore, **our method focuses on the general exocentric to egocentric video generation and also has broad applicability to novel view video synthesis with significant viewpoint variations as you suggested.** In addition, **our method achieves much better performances on the human-object regions compared to the baselines, as shown in Fig. 3 of rebuttal pdf.** Our method can accurately generate the human hands that are operating the CPR on the model, while SVD generates much worse results. We agree that adding more person-related designs can further improve the performance and leave this as faithful future directions.\n\n# Q1: Failure case analysis.\n\nThanks for your advice. **We provide two failure cases on Fig. 7 of rebuttal pdf.** Yes, most of our failure cases are caused by the fine-grained human-object interactions, **due to the significant differences between exocentric and egocentric viewpoints, as well as far-away placed exocentric cameras in the daily-life environments.** Although we achieve much better performance than baselines, we believe there is room for improvements especially based on your advice of person-related designs. We leave this as faithful future directions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful and valuable comments. We also appreciate that the core contributions and the quality of our results are recognized in the review:

1.  **First work** to address the **challenging** task of Exo2Ego video generation. **New** diffusion-based multi-view exocentric encoder and an Exo2Ego view translation prior. **Best performance** on most evaluation cases. (Reviewer W4kZ)
    
2.  The application of view translation using the Nerf-based approach seems **interesting**. Multiview exocentric encoder and temporal modeling is proven **effective** in supporting view transfer. The paper is **well written and easy to follow**. (Reviewer ANQv)
    
3.  The task is **very challenging**. The proposed method is **novel and reasonable** that effectively leverages the camera poses and multi-view video features. Achieves **SOTA accuracy**. (Reviewer aauy)
    
4.  **The motivation is interesting**, and the logic of the paper is **reasonable**. (Reviewer qWdm)
    

  

**We have included additional figures in the global rebuttal PDF. Please refer to the PDF for further qualitative results.**

Below we first address the common advice from Reviewer ANQv and Reviewer qWdm to demonstrate the generalization of our method on other EgoExo datasets, as well as the common advice from Reviewer W4kZ and Reviewer qWdm on the inference time comparison.

For other concerns and comments, please refer to individual responses to each reviewer.

  

# Common 1: Generalization on other EgoExo datasets (Reviewer ANQv, qWdm).

  

Thanks for your advice. Our method can be generalized to other EgoExo datasets. Following your suggestion, **we additionally train and evaluate our method on the H2O dataset [1], and compare our method with the SOTA baseline - SVD on the following table**. As shown in the following table, **our method achieves much better performance than SVD, with significant 30.3% improvement over SVD in terms of LPIPS**. **We provide qualitative comparisons of our method against SVD on H2O dataset [1] on Fig. 1 of rebuttal pdf**, where **our method achieves much more photorealistic and accurate results than SVD**, such as the details of human hands, interacted objects, and the environments. **These can demonstrate our method can be generalized to other datasets**, and we will add this result on the final paper.

*Tab. 1 Quantitative comparison of our method against SVD on H2O dataset [1].*

|      | PSNR↑      | SSIM↑      | LPIPS↓     |
|------|-----------|-----------|-----------|
| Ours | **18.60** | **0.581** | **0.189** |
| SVD  | 16.53     | 0.468     | 0.271     |

  

# Common 2: Reasoning efficiency (Reviewer W4kZ, qWdm).


We provide the inference time comparison on the following table, where the inference time of our method to generate an 8-frame egocentric video is 9.06 second, which is comparable with other baselines. We believe it is feasible to use our model in offline applications to generate egocentric videos from the exocentric videos, such as capturing exocentric cooking videos and generating corresponding egocentric videos offline for cooking skills learning. Improving the inference speed towards real-time is very promising and we leave it as future works. We will include the inference time comparison in our final paper.

*Tab. 2 Inference time of our method in comparison with baselines.*
|                         | Ours | SVD  | SD   | PixelNeRF |
|-------------------------|------|------|------|-----------|
| Inference Time (second) | 9.06 | 4.26 | 6.91 | 5.65      |

Meta-review: The paper introduces Exo2Ego-V, a diffusion-based video generation method for daily-life skilled human activities with sparse 4-view exocentric viewpoints. It proposes a multi-view exocentric encoder, designing an exocentric-to-egocentric view translation prior, and introducing temporal attention layers. It performs well and shows improvements. Most concerns are addressed, though there is also some discussion about the difference between this work and ReconFusion. After reading the paper, reviews, and responses, the AC agrees that this paper proposed a sound method and made interesting improvements. However, the revision suggested should be made in the final version.
Decision: Accept (poster)