Exocentric-to-Egocentric Video Generation

Authors: jia-wei  liu
weijia  mao
zhongcong  xu
jussi  keppo
mike zheng shou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:39:19

Abstract: We introduce Exo2Ego-V, a novel exocentric-to-egocentric diffusion-based video generation method for daily-life skilled human activities where sparse 4-view exocentric viewpoints are configured 360° around the scene. This task is particularly challenging due to the significant variations between exocentric and egocentric viewpoints and high complexity of dynamic motions and real-world daily-life environments. To address these challenges, we first propose a new diffusion-based multi-view exocentric encoder to extract the dense multi-scale features from multi-view exocentric videos as the appearance conditions for egocentric video generation. Then, we design an exocentric-to-egocentric view translation prior to provide spatially aligned egocentric features as a concatenation guidance for the input of egocentric video diffusion model. Finally, we introduce the temporal attention layers into our egocentric video diffusion pipeline to improve the temporal consistency cross egocentric frames. Extensive experiments demonstrate that Exo2Ego-V significantly outperforms SOTA approaches on 5 categories from the Ego-Exo4D dataset with an average of 35% in terms of LPIPS. Our code and model will be made available on https://github.com/showlab/Exo2Ego-V.

Main paper: 
 introduction: 
 When people observe and learn skills such as cooking or playing basketball from an exocentric (thirdperson) perspective, they can easily envision themselves executing these skills from an egocentric (first-person) perspective [2] . This exocentric-egocentric (Exo-Ego) translation remains the foundation of visual learning [15] for both human beings and AI robots [8, 4, 3, 37, 35] , and unleashes new opportunities for AI assistant [54, 7, 13] and augmented reality [1] . However, it remains particularly challenging for computer vision algorithms to achieve such exocentric to egocentric video generation for daily-life skilled human activities, primarily due to 1) significant variations between exocentric and egocentric viewpoints, and 2) high complexity of dynamic motions and daily-life environments, as illustrated in Fig. 1 .Existing view translation approaches mainly focus on the task of novel view synthesis and have made remarkable progress particularly since the introduction of Neural Radiance Fields (NeRF) [36] . While initially limited to reconstructing static 3D scenes, subsequent studies have extended NeRF to address the challenges of dynamic view synthesis [42, 38, 39, 50, 27, 12, 30] . However, these approaches are limited to per-scene regressive optimization and require dozens or hundreds of views as input. As a result, they fall short in the exocentric to egocentric video generation task due to the sparse yet highly variable viewpoints and significant occlusions.The remarkable success of powerful image diffusion models [45] provides new opportunities for introducing such generative models in the task of exo-ego generation. Recent attempt [33] leverages action intention consisting of human movement and action description for Ego2Exo video generation. However, it strictly requires the first exocentric frame a priori which largely simplifies the ego-toexo generation task to optical flow prediction and exocentric frame warping, highly limiting its applications in general exo-ego generation tasks. On the other hand, Exo2Ego [34] attempts to tackle with the exo2ego view translation by first transferring exo hand pose to ego, and then learning the conditional distribution of the target ego image given a single exo image and the predicted ego hand pose. Despite promising, it is limited to image-level translation and requires a carefully designed capturing setup with the exocentric camera configured close to the hand-object region and thus is restricted to desktop activities with simple environments.In contrast, we contend that it is necessary for exo-ego translation algorithms to be resilient to the complexity and diversity of daily-life scenarios such as cooking in kitchens, playing basketball in courts, etc. The introduction of Ego-Exo4D [15] opens new opportunities and challenges for exo-ego translation by providing a large-scale simultaneously-captured egocentric and exocentric videos of daily-life skilled human activities. In order to capture the complete and complex human-environment activities, they configure 4 exocentric cameras in 360°around the dynamic scene, resulting in new challenges of significant variations between exocentric and egocentric viewpoints, as well as complex dynamic motions and daily-life environments, as shown in Fig. 1 .To tackle with these challenges, we propose a novel exocentric-to-egocentric video diffusion pipeline dubbed as Exo2Ego-V. We address the significant challenges of large viewpoint variations and complex environments from two aspects: exo appearance conditions and ego translation prior. Firstly, we propose a diffusion-based multi-view exocentric encoder to extract the multi-scale exocentric features as the appearance conditions for egocentric video generation. We achieve this by concatenating ego hidden states with exo features for self-attention computation, so that the ego hidden states can attend to both the egocentric features as well as the multi-view exocentric features through the self-attention mechanism. In addition, we inject the relative position information into our exocentric encoder by adding exocentric latents with relative Exo2Ego relative camera pose embedding. Our exocentric encoder can extract dense human activity and environment information to guide the appearance of egocentric video generation pipeline. Secondly, we introduce an Exo2Ego view translation prior based on PixelNeRF [60] to provide coarse yet spatially aligned egocentric features as a concatenation guidance for the input of egocentric video diffusion model. Finally, to improve the temporal dynamic motion consistency of egocentric video contents, we insert temporal layers into our egocentric video diffusion pipeline to encode the temporal information across ego frames.We extensively evaluate our Exo2Ego-V on 5 categories of skilled human activities from the challenging Ego-Exo4D [15] dataset and H2O dataset [26] . As shown in Fig. 1 , our Exo2Ego-V can generate the corresponding egocentric videos given 4 multi-view exocentric videos, and significantly outperforms SOTA approaches with an average of 35% in terms of LPIPS.To summarize, the major contributions of our paper are:• We present a novel framework of Exo2Ego-V, the first work to achieve exocentric-toegocentric video generation for daily-life real-world skilled human activities.• We propose a new diffusion-based multi-view exocentric encoder and an Exo2Ego view translation prior that can extract dense exocentric features and spatially aligned egocentric features as conditions for our egocentric video diffusion pipeline.• Extensive experiments show that Exo2Ego-V significantly outperforms SOTA approaches on the challenging Ego-Exo4D [15] dataset with an average of 35% in terms of LPIPS.2 Related work 
egocentric-exocentric vision: 
 Tremendous progress has been made for exocentric vision with various visual perception and generation tasks due to the large amount of dataset captured at thrid-person views [11, 23, 48] . Recently, egocentric vision has also been scaling up particularly since the introduction of EPIC-Kitchens [9, 10] and Ego4D [14] . Previous attempts for joint egocentric and exocentric vision explore learning the ego-exo view-invariant features on paired small-scale dataset [2, 61] or through unpaired learning [59] .Another line of research focuses on egocentric human localization from exocentric videos [57, 52] , as well as egocentric human pose estimation from exocentric videos [51] . More recently, the introduction of Ego-Exo4D dataset [15] opens up new opportunities for joint egocentric and exocentric vision with large-scale synchronized multi-view ego-exo videos with multi-modality annotations.Ego-exo view translation. There is limited prior work on ego-exo cross-view translation. Early attempt [28] explores the exo-to-ego image generation with a novel parallel generative adversarial network to learn shared features of exo and ego images. STA-GAN [29] further extends P-GAN [28] to Exo2Ego video synthesis with a spatial temporal attention fusion module. However, they are limited to simple activities such as walking where most contents in egocentric and exocentric views are static environments [29, 28] . More recently, IDE [33] leverages action intention consisting of human movement and action description for ego2exo video generation. However, it requires the first exocentric frame a priori which largely simplifies the ego-to-exo generation task to optical flow prediction and exocentric frame warping. On the other hand, Exo2Ego [34] attempts to tackle with the exo2ego view translation by first transferring exo hand pose to ego, and then learning the conditional distribution of the target ego image given a single exo image and predicted ego hand pose. Despite promising, it is limited to image-level translation and requires a carefully designed capturing setup with the exocentric camera configured close to the hand-object region and thus is restricted to desktop activities with simple environments.Novel view synthesis (NVS). Ego-Exo view translation is also related to NVS, which has made remarkable progress particularly since the introduction of NeRF [36] . While initially limited to reconstructing static 3D scenes, NeRF has been extended to modelling dynamic scenes [42, 38, 39, 50, 27, 12] , dynamic humans [41, 53, 22, 32] . However, these approaches are limited to perscene regressive optimization and require dozens or hundreds of views as input. On the other hand, generalizable scene reconstruction methods [60, 56] are still limited to static scenes. As a result, they fall short in the exocentric to egocentric video generation task due to the sparse yet highly variable viewpoints and significant occlusions.Recent works have extended the power of image diffusion models to video editing [55, 43, 31] and generation [6, 62, 63, 18, 20] . Tune-A-Video [55] inflates the image diffusion with cross-frame attention and fine-tunes the source video, aiming to implicitly learn the source motion and transfer it to the target video. Video Diffusion Models(VDM) [20] designs a factorized space-time UNet to generate videos. Stable Video Diffusion [5] introduces a systematic data curation workflow, enabling the training of a state-of-the-art text-to-video and image-to-video models. AnimateDiff [16] proposes a plug-and-play motion module on temporal layers for personalized text-to-image animation. Other approaches [21, 58] introduce such video generation architectures to human image animation and achieve faithful performances. Our Exo2Ego-V is a new video diffusion pipeline for the challenging Exo2Ego generation on daily-life skilled human activities.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations in Conclusion. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Latent diffusion models (LDMs). LDMs encode input images to a latent representation using a pretrained variational auto-encoder (VAE) and operate the diffusion and denoising process following denoising diffusion probabilistic models (DDPMs) [19] (see Sec. A.6 for more details) in the reduceddimension latent space, and finally decode the denoised latents to the image space. Our model is an extention of the pretrained text-to-image latent diffusion model [45] that utilizes the UNet architecture [49] for denoising noise prediction with multiple down, middle, and up blocks. Each block consists of a ResNet2D layer, a self-attention layer, and a cross-attention layer.Task definition. Given 4 exocentric videosV exo = V 1 exo , V 2 exo , V 3 exo , V 4exo configured 360°a round daily-life skilled human activities, our objective is to generate their corresponding egocentric video V ego , as shown in Fig. 2(a) . Since these daily-life activities happen naturally in real-world scenarios such as kitchens, basketball courts, bike stores, etc, our task features the diversity of the real world with complex activities and environments. Therefore, the sparse 4 exocentric cameras have to be configured evenly in 360°around the dynamic scene in order to capture both the complex human activities and real-world environments [15] , resulting in significant variations between exocentric and egocentric viewpoints. Furthermore, such real-world skilled human activities such as cooking, repairing bike, and playing basketball are also highly challenging in terms of the complexity of dynamic motions and real-world environments.Overall framework. In order to tackle with the above challenges, we propose a novel exocentricto-egocentric video diffusion pipeline dubbed as Exo2Ego-V, as shown in Fig. 2 . To address the significant challenges of large viewpoint variations and complex environments, we propose a diffusionbased multi-view exocentric encoder (Fig. 2(b) ) to extract the multi-scale multi-view exocentric features as the appearance conditions for egocentric video generation. In addition, we design an exocentric-to-egocentric view translation prior (Fig. 2(c )) based on PixelNeRF [60] to provide coarse yet spatially aligned egocentric features as a concatenation guidance for egocentric video generation. Finally, we introduce the temporal attention layers into our egocentric video diffusion pipeline to improve the temporal consistency cross egocentric frames (Fig. 2(d) ).Motivation. Our task is featured with the significant variations of Exo2Ego viewpoints and complex environments. The core to tackle with this challenge is to fully explore the multi-view exocentric information for the purpose of guiding the egocentric video generation. A naive solution is to utilize the CLIP [44] image encoder to extract latent features from low-resolution images. However, such semantic-level CLIP image features fall short in extracting the dense and fine-grained detail information from exocentric videos. Another naive solution is to train a 4D dynamic scene reconstruc-tion model for each multi-view exocentric sequences, but it requires high computation and storage resources and current methods cannot handle sparse 4 views complex dynamic scene reconstruction.Framework. Inspired by recent reference image animation methods [58, 21] that extract dense image features with a reference UNet to preserve reference human identity, we propose our multiview exocentric encoder with a different purpose of extracting the dense multi-view exocentric intricate details as appearance conditions for egocentric video generation. Specifically, our exocentric encoder creates a trainable copy of the base Ego UNet and inject the relative camera poses from exocentric viewpoints to the egocentric viewpoints as additional embeddings, as shown in Fig. 2(b) . We additionally explore adding temporal layers for exocentric encoder in our ablation study.Given V exo ∈ R N ×C×F ×H×W and their relative camera poses P ∈ R N ×4×4 , our exocentric encoder computes the multi-view multi-scale appearance condition features F exo for the egocentric video generation at denoising step t = 0:EQUATIONwhere N = 4 is the number of exocentric views and F = 8 is the number of frames for each video.F exo are the normalized attention features for the downsampling, middle, and upsampling blocks of Exo UNet. We set t = 0 to preserve the appearance details of the noise-free exocentric videos.Then, the exocentric features F exo are utilized as the appearance conditions for egocentric video generation by concatenating F exo with the corresponding egocentric UNet hidden states z t ego for the self-attention layers in every block b at each denoising step t:EQUATIONwhere [•] denotes concatenation operation, and the Ego UNet self-attention is: SoftmaxQ b •K T b √ d •V b .The queried egocentric noisy latents can attend to both the egocentric features as well as the multiview exocentric features through the self-attention mechanism, and thus translate the appearance of complex human skill activities and environments from exocentric views to the egocentric view.Camera pose. In order to inject the relative position information into our Exo2Ego generation pipeline, inspired by MVDream [47] , we embed the relative Exo2Ego camera poses with a 2-layer MLP and add the embedding with the denoising timestep embedding for our multi-view exocentric encoder. Since we set t = 0 for our Exo UNet, the relative camera pose embeddings are the main embedding to inject the relative position information into the exocentric feature extraction.Motivation. Although our multi-view exocentric encoder can extract dense and intricate appearance details, it entirely relies on the self-attention mechanism to explore the correspondences from the egocentric contents to the exocentric features, which are still challenging for our scenarios with large viewpoints variations. To tackle with this, we design an Exo2Ego view translation prior based on PixelNeRF [60] to generate a coarse yet spatially aligned egocentric latent feature as the concatenation guidance for our egocentric video generation pipeline.Framework. Given multi-view exocentric videos V exo , exo camera poses P exo , egocentric videos V ego , and ego camera poses P ego , we learn an Exo2Ego view translation prior Ψ by training a generalizable PixelNeRF [60] for all timesteps. For each synchronized timestep of the 4 exocentric videos and 1 egocentric video, at each iteration we extract 4 exo frames and 1 ego frame and randomly sample rays from these 5 images for optimization. Inspired by ReconFusion [56] , we utilize a light PixelNeRF with 6-layer MLPs for higher efficiency. Please see Sec. A.1 for more details.As shown in Fig. 2(c ), with this Exo2Ego translation prior, we can render both the egocentric features F ego and egocentric pixels I ego given the multi-view exocentric videos, exo camera poses, and queried ego camera pose:EQUATIONInspired by ReconFusion [56] , we design our translation prior to render egocentric features F ego at the egocentric viewpoint with the same spatial resolution as the egocentric latents, so that F ego is spatially aligned with the noisy egocentric latents. Therefore, we concatenate z t ego with F ego along channel dimension as the input to the egocentric video diffusion model to predict the noise ϵ ego at each denoising timestep t. In addition, we extract the CLIP image feature of the rendered ego images I ego as the cross-attention information for our egocentric video diffusion pipeline:EQUATIONTo improve the temporal dynamic motion consistency of egocentric and exocentric video contents, we follow common practice [18, 20] to insert temporal attention layers within the 2D UNet blocks, as shown in Fig. 2(d ). Specifically, we insert the temporal layers on the egocentric video generation pipeline and we additionally ablate on inserting the temporal layers on the exocentric encoder. As such, the input egocentric latents z t ego ∈ R N ×C×F ×H×W are first reshaped to R (N F )×H×W ×C for computing the spatial attentions with egocentric and exocentric features in spatial layers, and then reshaped to R (N HW )×F ×C to compute the temporal cross-frame information in temporal layers.We optimize our Exo2Ego-V in a 2-stage training strategy. In the first stage, we remove the temporal layers and optimize the Exo2Ego spatial appearance translation modules, including the multi-view exocentric encoder, Exo2Ego view translation prior, as well as the Ego UNet. In the second stage, we only optimize the Ego temporal layers for temporal consistency and freeze other modules.Exo2Ego spatial appearance translation. We first pre-train our Exo2Ego view translation prior with the pixel-level reconstruction loss L REC . Then, we alternately finetune the Exo2Ego view translation prior with the reconstruction loss L REC , and multi-view exocentric encoder and the Ego UNet with the noise prediction loss L S .EQUATIONwhere R render is the rendered ray pixels sampled randomly from exocentric and egocentric frames, R gt is the corresponding ground-truth pixels. ϵ is the ego noise sampled from N (0, 1). w(t) is a weighting function that depends on the noise level t.Temporal motion finetuning. In the second stage, we freeze the translation prior and Exo and Ego UNets, and finetune the pretrained temporal layers from AnimateDiff [16] on our egocentric and exocentric videos with F frames in temporal dimension.EQUATION4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We use publicly released Ego-Exo4D dataset for experiments, and our code and model will be made public if this paper gets accepted. We also provide dataset details in Sec. 4.1 and implementation details in Sec. A.1.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
dataset: 
 We evaluate our method on 5 categories of Ego-Exo4D dataset [15] We also evaluate our method on the H2O dataset [26] , which provides synchronized multi-view Exo-Ego images for desktop activities. Quantitative results. We report PSNR, SSIM, and LPIPS with AlexNet [25] that measure the differences between generated ego frames and groundtruth on Tab. 1. Our Exo2Ego-V achieves the best performance for both unseen actions and unseen takes in terms of all metrics. It is noted PixelNeRF [60] achieves good PSNR scores since PSNR favors blurry images [38] as shown in Fig. 3 and 4 .Most importantly, our Exo2Ego-V significantly outperforms SOTA approaches on all categories with an average of 35% in terms of LPIPS, which clearly demonstrates the superiority of our Exo2Ego-V. We additionally evaluate our Exo2Ego-V and best-performing baseline SVD [5] on the H2O dataset [26] in Tab. 2. Our model still achieves the best performance, which demonstrate the generalizability of our method on different datasets. Qualitative results. Fig. 3 and 4 visualizes the qualitative comparison of Exo2Ego-V over SOTA approaches on unseen actions and unseen takes, respectively, where Exo2Ego-V achieves substantially better egocentric videos quality than other approaches for all categories (see Sec. A.3 for more results). SVD [5] and SD [46] encounter significant difficulties by conditioning on the highly semantic exo images features to generate egocentric videos. In addition, SD [46] falls short in temporal consistency due to its 2D image-level generation. PixelNeRF [60] renders very blurry results due to the significant difficulty of sparse yet highly variable viewpoints and large occlusions. In addition, Fig. 6 visualizes the comparison of our method against SVD [5] on the H2O dataset [26] . Our method achieves the best performance and generates photorealistic hand-object interactions. Please see supplementary video for more results on video comparisons, which demonstrates the superiority of our Exo2Ego-V on both much higher spatial appearance quality and temporal consistency compared to other methods. Comparisons on unseen scenes. We conduct additional experiments on unseen scenes of our Exo2Ego-V and baselines, and report PSNR, SSIM, and LPIPS with AlexNet [25] that measure the differences between generated ego frames and groundtruth on Tab. 3. We do not conduct experiments on Bike category since it is only captured on 4 different scenes, which are too few to generalize to new scenes. Fig. 5 visualizes the qualitative comparison of our Exo2Ego-V over SOTA approaches on unseen scenes. Our Exo2Ego-V achieves the best performance on most metrics in Tab. 3 and substantially better egocentric videos quality than other approaches as shown in Fig. 5 . Pix-elNeRF [60] still renders very blurry results but gets good PSNR values since PSNR favors blurry images [38] . We also find that it is very challenging for all methods to evaluate on the unseen scenes due to the significant variance of new environments compared to the training set, and the lack of large-scale scene diversity from the training data. We conduct ablation studies on the cooking category from Ego-Exo4D dataset [15] . We ablate on the proposed multi-view exocentric encoder, Exo2Ego view translation prior, and the temporal layers of multi-view exocentric encoder. As shown in Tab. 4, our full model achieves the best performance in terms of PSNR and SSIM. In addition, we provide the qualitative results of our ablations in Fig. 7 , which further demonstrates the effectiveness of our designs. Removing exocentric encoder results in inferior performance than full model, which clearly proves its capability in extracting dense and multi-scale exo features for ego video generation. Although removing exo2ego prior achieves the best LPIPS, it results in a cleaner but inaccurate egocentric video due to the lack of egocentric guidance, which improves the LPIPS but gets a lower PSNR. As shown in Fig. 7 , removing exo2ego prior results in the missing of right arm. In addition, we ablate on adding temporal layers for our multi-view exocentric encoder and evaluate the temporal consistency by computing the CLIP image embeddings on our generated ego clips and report the average cosine similarity between all pairs of clip frames. Adding the exo temporal layers achieves a higher averaged temporal score of 0.924 compared to 0.918 of full model for unseen actions, demonstrating higher temporal consistency but at the expense of inferior image-level quality in Tab. 4. Thus, we disgard the exo temporal layer in final model. We ablate on the number of exo views and replacing our exocentric feature encoder with CLIP features in Tab. 4 and Fig. 8 . Our model with 4 exo views achieves the best performance, and our method achieves much better performance compared to the one using CLIP features. We also ablate on first performing temporal attention and then spatial attention for our model. The spatial-temporal model is slightly better than the temporal-spatial model in terms of PSNR and SSIM, and slightly worse for LPIPS. We follow the spatial-temporal attentions [16, 5] . 
conclusion: 
 We introduced a novel framework of Exo2Ego-V, the first work to achieve Exo2Ego video generation for daily-life real-world skilled human activities. To tackle the challenges, we first proposed the new diffusion-based multi-view exocentric encoder to extract the dense multi-scale exocentric features as the appearance conditions. Then, we introduced an Exo2Ego view translation prior to provide coarse yet spatially aligned egocentric features as a concatenation guidance. Finally, we inserted temporal layers into Ego Unet for improved temporal consistency across ego frames. Exo2Ego-V produced significant improvements on challenging Ego-Exo4D dataset [15] over SOTA approaches.Limitations and future work. Exo2Ego-V focuses on Exo2Ego video generation on several categories of skilled human activities. It remains challenging but is worthwhile researching on more general activities. Exploring Gaussian Splatting as translation prior is also a promising direction. 
acknowledgment: 
 This project is supported by the Mike Zheng Shou's Start-Up Grant from NUS. Jia-Wei Liu is also supported by NUS IDS-ISEP scholarship. Thanks to Zihang Xia for helpful discussions. Exo2Ego translation prior details. Inspired by ReconFusion [56] , our Exo2Ego translation prior Ψ is based on a light PixelNeRF with 6layer MLPs (Fig. 9 ) for higher efficiency together with a ResNet34 [17] pretrained on the ImageNet dataset to extract exo image features.For each synchronized timestep of the 4 exocentric videos and 1 egocentric video, we extract 4 exo frames and 1 ego frame and randomly sample 128 pixel rays from these 5 images and sample 3D points x for each iteration. Then, we add positional embedding to these points γ (x) and query their latent features f (x) by projecting them on the latent images. Then we concatenate them with viewing direction as input to our translation prior as in Fig. 9 . We conduct volumetric rendering at the third layer for the latent features, and then input the latent features to the last 3 layers for final pixel color and latent features.For egocentric video generation, we only sample rays from 32 × 32 ego frames so that the rendered ego features are spatially aligned with the noisy ego latent. Therefore, we concatenate the rendered ego features with the noisy ego latent as the input to our egocentric video diffusion pipeline.Egocentric camera unprojection. The ego camera from Ego-Exo4D dataset [15] utilizes the FisheyeRadTanThinPrism (Fisheye624) model, which accounts for thin-prism distortion. This model includes four additional coefficients: s 0 , s 1 , s 2 , s 3 . The projection function is:u = fx * (u r + t x (u r , v r ) + tp x (u r , v r )) + c x , v = fy * (v r + t x (u r , v r ) + tp x (u r , v r )) + c y , tp x (u r , v r ) = s 0 r(θ) 2 + s 1 r(θ) 4 , tp y (u r , v r ) = s 2 r(θ) 2 + s 3 r(θ) 4 , r(θ) = (u -c x ) 2 /f 2 x + (v -c y ) 2 /f 2 y , ϕ = arctan((u -c x )/f x , (v -c y )/f y ) (7)u, v are the camera pixel coordinates and ϕ, θ are the world point. fx, fy are the focal lengths. Its parametrization contains 4 additional coefficients: s 0 , s 1 , s 2 , s 3 . Firstly we use the Newton method to calculate the u r , v r and then we calculate ϕ, θ using the above unprojection method. Finally, we sample 3D points along the calculated directions ϕ, θ. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We omitted error bars from our analysis due to the excessive computational expense involved in enumerating all experimented categories in the dataset.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work does not have such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly cited papers and sources for existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide asset documentation alongside our code and model when we release them. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We optimize our Exo2Ego-V using Adam optimizer [24] . We set the learning rate of the multi-view exocentric encoder and the egocentric diffusion model as 0.00001, and we set the learning rate of view translation prior as 0.0001. We first train the translation prior with 500K iterations on a single A100 GPU for 36 hours, and then optimize our Exo2Ego spatial appearance translation with 500K iterations on 8 A100 GPUs for 48 hours, and finally finetune our temporal motion module with 100K iterations on 8 A100 GPUs for 40 hours, all using the PyTorch [40] deep learning framework. We provide more qualitative comparisons of Exo2Ego-V over SOTA approaches on unseen actions in Fig. 10 . Our method achieves the best performance across all approaches.A. In Fig. 11 , we present our ego feature visualization results using the Exo2Ego prior. The visualization results clearly represent the contents of the ego views. This indicates that our Exo2Ego prior can effectively extract and transmit the important information from ego views to the multi-view exocentric encoder.We provide the inference time comparison as shown in Tab. 5, where the inference time of our method to generate an 8-frame egocentric video is 9.06 second, which is comparable with other baselines. We believe it is feasible to use our model in offline applications to generate egocentric videos from the exocentric videos, such as capturing exocentric cooking videos and generating corresponding egocentric videos offline for cooking skills learning. Improving the inference speed towards real-time is very promising and we leave it as future works.Table 5 : Inference time of our method in comparison with baselines.Ours SVD [5] SD [46] PixelNeRF [60] Inference time (second) 9.06 4.26 6.91 5.65DDPMs [19] are generative frameworks designed to synthesize data by reproducing a consistent forward Markov chain x 1 , . . . , x T . The process begins from a random noise distribution and progres-sively denoise the noisy contents to the clean data. Considering a data distribution as x 0 ∼ q(x 0 ), the Markov transition q(x t |x t-1 ) is conceptualized as a Gaussian distribution by a variance β t ∈ (0, 1):EQUATIONUnder the Bayes and Markov principles, the conditional probabilities can be derived as:EQUATIONα t = 1 -β t , ᾱt = t s=1 α s , βt = 1-ᾱt-1 1-ᾱt β t , μt (x t , x 0 ) = √ ᾱtβt 1-ᾱt x 0 + √ αt(1-ᾱt-1)1-ᾱtx t . DDPMs utilize a reverse approach to synthesize the chain x 1 , . . . , x T :EQUATIONThe model parameters θ are optimized to ensure the synthesized reverse sequence aligns with the forward sequence.Our work can generate egocentric videos from exocentric videos. Since the scale of egocentric dataset is still much less than the exocentric dataset, our method has the potential to improve the egocentric vision such as egocentric perception by generating more egocentric data from the exocentric data. Our Exo2Ego-V also support applications on AI assistant and augmented reality by generating egocentric videos from exocentric videos. We believe our method will not bring negative societal impacts.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main contribution of our paper is clearly described in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper introduces Exo2Ego-V, a diffusion-based video generation method for daily-life skilled human activities with sparse 4-view exocentric viewpoints. It proposes a multi-view exocentric encoder, designing an exocentric-to-egocentric view translation prior, and introducing temporal attention layers. It performs well and shows improvements. Most concerns are addressed, though there is also some discussion about the difference between this work and ReconFusion. After reading the paper, reviews, and responses, the AC agrees that this paper proposed a sound method and made interesting improvements. However, the revision suggested should be made in the final version.

Reviews: review: paper_summary: paper_summary: This paper deals with the task of exocentric-to-egocentric video generation. It presents a diffusion-based framework of exo2ego-v to tackle the challenges of the significant variations between exocentric and egocentric viewpoints and high complexity of dynamic motions and real-world daily-life environments. It propose a multi-view exocentric encoder to extract the multi-scale multi-view exocentric features as the appearance conditions for egocentric video generation. It also designs an exocentric-to-egocentric view translation prior based on PixelNeRF to provide coarse yet spatially aligned egocentric features as a concatenation guidance for egocentric video generation. Experiments on a part of Ego-Exo4D dataset show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1) The presentation of the paper needs further improvement.  
    1.1 For the method part, the network architecture is not clear for the exocentric encoder and the egocentric video diffusion model. As a result, the feature dimension of many notations such as $F_{exo}$, $Z^{t}_{exo}$ are not known. It is also difficult to understand the procedure within Equation (1) and (4).  
    1.2 For the experiment part, the qualitative results have taken too much space (e.g., Figure 3). Space should be saved for the presentation of the results on the unseen scenes (which are moved to appendix).  
    1.3 If the test scenes are not seen during training, Exo2Ego translation prior (i.e., pixelNeRF) need be trained for the new scenes. During testing, egocentric video frames are not available, while it is said in the paper that four exocentric frames and one egocentric frame are used to train the pixelNeRF.  
2) As shown by the results on the more challenging unseen scenes (Table 3), the proposed method sometimes gets worse performance than the baseline methods. The authors should give more analysis and discussion for such results.
questions: questions: 1) Do you train one Exo2Ego prior model for each video, or for each synchronized timestep of a video? It is not clear for me how regular the Exo2Ego prior model need be retrained.  
2) What is the dimension of the encoded features for multi-view exocentric videos, and how are they combined with the camera poses which have different dimensions?  
3) How are the egocentric camera poses determined during testing? Do you directly use the poses provided by the dataset?  
4) In Figure 5, the same results of EgoGT are shown twice. Please have a check. 
5) Current evaluation metrics are averaged over the generated video. I wonder how the performance would change over time.
limitations: limitations: The paper can also show the runtime of the video generation framework.
paper_summary: paper_summary: This paper proposes a novel diffusion-based video generation method that translates exocentric views to egocentric view. Overall, I think the idea is novel and the method performs well on multiple daily human activities.
strength_weakness: strength_weakness: 1.	In Line 157-159, the authors mentioned that encoding multiview exocentric videos using CLIP image features lacks fine-grained details. However, there is no sufficient evidence in the experiments. For example, comparing the proposed method with global CLIP features or intermediate spatial CLIP features as conditions. 
2.	Generally, as most of the takes in EgoExo4d contain 4 exocentric views, which are significantly fewer than dozens of views used in 3D reconstruction approaches. The effectiveness of reconstructing a 3D scene using such sparse viewpoints is questionable, not to mention the significant camera pose differences between egocentric and exocentric views.
3.	In the experiments, the rendered pixels of PixelNerf are extremely blurry, and thus it is not very convincing to use the CLIP features of rendered egocentric image in Eq.(4). To prove the effectiveness of exo2ego prior, the authors are encouraged to show some visualization results of the rendered egocentric features using PixelNerf.
questions: questions: 1. The proposed method outperforms the baseline SVD that performs video generation given the first frame as condition. It is not clear to me whether the authors also adopted first frame as the condition while training the model since the first frame is a very strong prior. If not, could the authors explain why the proposed method can outperform SVD without using such strong prior, when both methods are given the same exocentric views as conditions.  
2.  It is not clear whether the proposed method can be generalized to other egoexo datasets that also contain camera poses, e.g. H2O [1], Assembly101[2]. 

[1] Kwon et al. H2O: Two Hands Manipulating Objects for First Person Interaction Recognition. ICCV21.

[2] Sener, Fadime, et al. Assembly101: A large-scale multi-view video dataset for understanding procedural activities.CVPR 2022.
limitations: limitations: Please refer to Weaknesses and Questions.
paper_summary: paper_summary: This paper proposes a novel method for generating egocentric videos from multi-view exocentric videos using diffusion-based techniques. This method addresses the challenges of viewpoint variation and dynamic motion by employing a multi-view exocentric encoder and a view translation prior, along with temporal attention layers to enhance temporal consistency. Experiments results on the Ego-Exo4D dataset verify the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.	The exocentric-to-egocentric view translation prior is very similar to the existing work ReconFusion, and the authors need to clarify the differences.

2.	In the egocentric video generation pipeline, why perform spatial attention before temporal attention instead of following the TimeSformer? Are there relevant experiments for further clarification?

3.	The reasoning efficiency of the whole process should be evaluated. This determines whether it can be practically applied.

4.	How does the number of exocentric videos affect performance? In real scenarios, it is difficult to capture 4 time-synchronized exocentric videos at the same time. Besides, do the 4 exocentric videos have to be evenly distributed around the scene?

5.	The evaluation dataset is so single that it is impossible to validate the generalizability of the method on more diverse data. Furthermore, Ego-Exo4D contains a wide range of human activities, why only 5 of these categories are selected for the experiment?

6.	The baselines compared are too few to adequately validate the superiority of the method.

7.	Writing needs further improvement, tenses should be correct (Line 185, Page 5) and contextual expressions need to be consistent (Unet or UNet).
questions: questions: After reviewing the response letter, some of my concerns have been partially addressed. However, the paper's novelty, primarily inspired by ReconFusion, appears to have a weak technical contribution. Additionally, the evaluation is limited by the dataset used. Consequently, I have adjusted my score to a the borderline reject.
limitations: limitations: yes
paper_summary: paper_summary: This paper introduces a novel method for generating egocentric videos from exocentric videos of daily-life skilled human activities. This task is very challenging because of the significant viewpoint variations, sparsely posed exo videos, and dynamic environments. The authors propose to use a PixelNeRF-informed diffusion method to address the problem and achieves good visual performance and SOTA accuracy.
strength_weakness: strength_weakness: The proposed method lacks specific designs tailored for the "Exocentric-to-Egocentric" task. "exo" and "ego" perspectives are relative to "the person" in the video, yet the proposed method does not include any design elements focused on the person. The method actually has broad applicability to novel view video synthesis with significant viewpoint variations. One of the goals of the "Exocentric-to-Egocentric" task is to generate dynamic hand motions and object state changes. However, the visualization results in the paper show bad hand-object motions synthesis result. In summary, while the model excels at reconstructing the geometry of the surrounding environment, it lacks focus on hand-object dynamics which is important in egocentric video.
questions: questions: Could you provide a failure case analysis? Do you think the fine-grained hand-object interactions causes most of the failure cases?
limitations: limitations: Limitation included

Comments: title: Could you kindly let us know whether we address all your concerns?
Could you kindly review our response?
Could you kindly review our response?
Response to Reviewer qWdm (2/2)
Response to Reviewer qWdm (1/2)
Reminder: author-reviewer discussion will end soon!
Response to Reviewer ANQv
Official Comment by Reviewer ANQv
Author-Reviewer Discussion Begins!
comment: Thanks for the authors' detailed responses which have addressed my initial concerns. After reading all the responses and comments from other reviewers, I would like to increase the rating to "Weak Accept".
Dear Reviewer, 

Thank you once again for your feedback. We provided additional responses towards your remaining concerns. As the rebuttal period is nearing its conclusion, could you kindly review our response to ensure it addresses your concerns? We appreciate your time and input. 

Best regards,

Authors of 10425
Dear Reviewer, 

Thank you once again for your feedback. As the rebuttal period is nearing its conclusion, could you kindly review our response to ensure it addresses your concerns? We appreciate your time and input. 

Best regards,

Authors of 10425
Dear Reviewer, 

Thank you once again for your feedback. As the rebuttal period is nearing its conclusion, could you kindly review our response to ensure it addresses your concerns? We appreciate your time and input. 

Best regards,

Authors of 10425
# Evaluation dataset.


We explained in our **rebuttal W5** that we selected Ego-Exo4D dataset [15] as our main evaluation dataset due to its **significant challenges of the complexity and diversity of daily-life scenarios, large differences between exo and ego viewpoints, complex human-object interactions, and its largest Ego-Exo data scale with 1286 hours of video.** We selected the 5 out of 8 categories from the Ego-Exo4D dataset that significantly emphasize both egocentric and exocentric activities, such as human-object interactions in both Ego and Exo viewpoints.
  
**We have demonstrated that our method can be generalized to other EgoExo datasets such as H2O dataset in common response 1**, where our method achieves much better performance than SVD, with significant 30.3% improvement over SVD in terms of LPIPS. **We are glad that Reviewer ANQv, who suggested this H2O dataset evaluation, has responded that our rebuttal has solved all Reviewer ANQv’s concerns, and we will add these results to provide more insights to readers.**

  
**Therefore, we believe our dataset evaluation is very challenging and extensive to evaluate the superiority of our proposed method.**
Dear Reviewer qWdm,

Thank you for updating your comment in the Reviews Questions. We are glad that we have addressed some of your concerns. Here we further respond to your remaining concerns about the paper novelty and evaluation dataset.

# Paper novelty.

**We respectfully disagree with your statement that our paper’s novelty is primarily inspired by ReconFusion.** We provided detailed responses about the differences between our Exo2Ego prior with ReconFusion in our rebuttal to address your concern about weakness 1. **Here we further clarify the overall novelty and technical contributions of our paper in comparison with ReconFusion.**

1. **Our method is the first work to achieve exocentric-to-egocentric video generation for daily-life real-world skilled human activities.** This task is particularly challenging in terms of the significant differences between the exocentric and egocentric viewpoints, as well as complex human-object interactions and real-world environments. In contrast, ReconFusion focuses on traditional 3D static scene reconstruction. **Therefore, our method is fundamentally different from ReconFusion.**
 
2. **To address the above challenges of Exo2Ego video generation, we propose a new diffusion-based multi-view exocentric encoder to extract fine-grained multi-scale exocentric features, as well as an Exo2Ego view translation prior to render spatially aligned geocentric features.** These two modules complement each other with fine-grained exocentric features and spatially aligned egocentric features to provide conditional information for **our egocentric video generation pipeline.** Our complete Exo2Ego video generation model is **the first video-level model for Exo2Ego video generation, and significantly outperforms previous approaches by a large margin of 35% for LPIPS.**

3. **Therefore, the technical contributions of our method are three-folds: 1) Diffusion-based multi-view exocentric encoder, 2) Exo2Ego view translation prior, 3) The first Exo2Ego video generation pipeline.** We conducted extensive ablation studies to demonstrate the effectiveness of each component in our main paper.

As discussed in our main paper (L199-L200, L204-L206) and rebuttal W1, **only the Exo2Ego view translation prior is partially inspired by ReconFusion — both our Exo2Ego prior and ReconFusion are based on PixelNeRF, but our Exo2Ego view translation prior is significantly different from ReconFusion. Furthermore, our proposed first exocentric-to-egocentric video generation model, our multi-view exocentric encoder, and our challenging Exo2Ego video generation task definition are all fundamentally different from ReconFusion.** **Below we further discuss the detailed technical differences between our Exo2Ego translation prior with ReconFusion.**

1. **The design and functionality of our Exo2Ego prior is different from ReconFusion’s diffusion prior.** Our method first trains a PixelNeRF-based Exo2Ego prior, and then we integrate this Exo2Ego prior into our complete Exo2Ego video generation pipeline and finetune Exo2Ego prior to provide the coarse yet spatially aligned egocentric features for our egocentric video generation. In contrast, ReconFusion directly trains a diffusion prior that includes a PixelNeRF and diffusion UNet, and then explores distilling from the pretrained diffusion prior. 

2. **Our Exo2Ego prior is specially designed for rendering egocentric views from exocentric inputs.** To achieve this, we differentiate the egocentric views and exocentric views for our Exo2Ego prior model – it only extracts exocentric views’ ResNet features as condition signals to render the pixel colors and features. **Our exocentric and egocentric viewpoints are sparse and significantly different from each other. In contrast, ReconFusion does not differentiate between views and is trained on dense multi-view 3D static scenes datasets that contain hundreds of views for each 3D static scene.**

3. **Our Exo2Ego prior’s model architecture is different from ReconFusion.** We employ a 6-layer MLP as our Exo2Ego prior base model to handle the complex Exo2Ego translation with large Exo-Ego viewpoint differences, while ReconFusion only adopts a 2-layer MLP. 

4. **Our Exo2Ego prior’s training strategy is different from ReconFusion.** We first pretrain our Exo2Ego prior on the Ego-Exo4D dataset. Then, we integrate our Exo2Ego prior into our Exo2Ego video generation pipeline and we alternatively finetune Exo2Ego prior and the complete pipeline during training. This strategy ensures that Exo2Ego prior renders spatially-aligned feature maps necessary for egocentric video generation, as well as keeps the original scene reconstruction capability.
Dear reviewers,

The authors have responded to your questions and comments. Please propose your post-rebuttal comments.

Thank Reviewer ANQv for your positive discussions!

Best, Your AC
Dear Reviewer ANQv,

**Thanks for your valuable comments! We are glad that our rebuttal solves all your concerns! We will add these additional experiments to the final paper to provide more insights to readers!**

Regarding your additional question, “Given the GT ego frame/rendered feature map/rendered rgb image, is it possible to map one pixel (e.g. center of the table) back to each of the four exocentric frames given the ego and exo camera poses?”

**Yes, it is possible to map one pixel from the ego frame back to each of the four exocentric frames given the ego and exo camera poses.** To achieve that, we also require egocentric depth information $\mathbf{D}$ which can be conveniently obtained by additionally rendering the sampled points depths with their density values $\sigma$ along each ray using our Exo2Ego prior. 

Specifically, we can first backproject one egocentric pixel $\left(u, v\right)$ to the 3D space, denoted as $\mathbf{X}$, using egocentric cameras pose (to provide projection direction) together with the egocentric depth at that pixel (to provide the projection distance) as follows,

$\mathbf{X}=E_{\mathrm{ego}}^{-1}\cdot K_{\mathrm{ego}}^{-1}\cdot\mathbf{D}\left[u, v\right]\cdot\left[u, v, 1\right]^{\mathrm{T}}$,

where $E_{\mathrm{ego}}$ and $K_{\mathrm{ego}}$ are the extrinsics and intrinsics of egocentric camera, and $\mathbf{D}$ is the depth map for egocentric frame.

After that, we can project $\mathbf{X}$ to each of the four exocentric frames using the exocentric camera poses $E_{\mathrm{exo}}$ and $K_{\mathrm{exo}}$. 

$\left(u_{\mathrm{exo}}, v_{\mathrm{exo}}, 1\right)=K_{\mathrm{exo}}\cdot E_{\mathrm{exo}}\cdot\mathbf{X}$,

where $\left(u_{\mathrm{exo}}, v_{\mathrm{exo}}\right)$ are the projected exocentric pixels. 

As you suggested, this could provide more fine-grained understandings of how each pixel in the rendered feature map/rgb image leverages the pixel-wise exocentric information from each exo camera.

Thank you very much for recognizing our work and valuable comments!

Best regards,

Authors of 10425
Dear authors,

Thanks for your responses and explanations. Overall, the rebuttal solves all my concerns. With these additional experiments, this paper would provide more insights to readers.


Regarding the response to W3 (visualization of rendered egocentric features), I have a few more questions that do not affect the rating.


The visualization results in Figure 2 show that the rendered feature maps/pixels generally capture the low-frequency of the visual content, e.g. table. , and these maps/pixels have exact spatial correspondence to the GT ego frame as mentioned by authors. 


Given the GT ego frame/rendered feature map/rendered rgb image, is it possible to map one pixel (e.g. center of the table) back to each of the four exocentric frames given the ego and exo camera poses? Could you briefly introduce the solution if possible?


If the mapping is available, this could provide more fine-grained understandings of how each pixel in the rendered feature map/rgb image leverages the pixel-wise exocentric information from each exo camera. I think this is useful when the field-of-ego-view is under occlusion in one/two exocentric cameras, and we would like to know which exo camera(s) provide valuable visual clues to support generation in the ego view.
Dear authors,

Thanks for your hard work and detailed responses.

Dear reviewers,

the authors have responded to your questions and comments, please read them and further provide feedback: Whether your concerns addressed? After reading all the reviews and responses, are there more questions about the clarity, contributions, results, etc?

Thanks!

Best, Your AC