View-Invariant Policy Learning via Zero-Shot Novel View Synthesis

Authors: stephen  tian
blake  wulfe
kyle  sargent
katherine  liu
sergey  zakharov
vitor  guizilini
jiajun  wu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 08:20:33

Abstract: Large-scale visuomotor policy learning is a promising approach toward developing generalizable manipulation systems. Yet, policies that can be deployed on diverse embodiments, environments, and observational modalities remain elusive. 
In this work, we investigate how knowledge from large-scale visual data of the world may be used to address one axis of variation for generalizable manipulation: observational viewpoint. Specifically, we study single-image novel view synthesis models, which learn 3D-aware scene-level priors by rendering images of the same scene from alternate camera viewpoints given a single input image.
For practical application to diverse robotic data, these models must operate *zero-shot*, performing view synthesis on unseen tasks and environments. We empirically analyze view synthesis models within a simple data-augmentation scheme that we call View Synthesis Augmentation (VISTA) to understand their capabilities for learning viewpoint-invariant policies from single-viewpoint demonstration data. Upon evaluating the robustness of policies trained with our method to out-of-distribution camera viewpoints, we find that they outperform baselines in both simulated and real-world manipulation tasks.

Main paper: 
 related work: 
 Learning viewpoint-robust robotic policies. Learning deep neural network policies that can generalize to different observational viewpoints has been discussed at length in the literature. One set of approaches effectively augment the input data to a learned policy or dynamics model with additional 2D images rendered from differing camera viewpoints. These renderings are often obtained from simulators [5, 6] or by reprojecting 2D images [9] . Augmenting training with simulator data can improve robustness on simulation environments, but these methods must then address the challenge of sim-to-real transfer for deployment on real systems. In this work, we study methods for learning invariant policies directly using robot data from the deployment setting, including real robot trajectories. Existing work [10] performs view augmentation of real-world wrist camera images; however, this is performed with the goal of mitigating covariate shift as opposed to improving camera pose robustness, and requires many views of a static scene to generate realistic novel views.Another line of work forms explicit 3D representations of the scene such as point clouds or voxels to leverage equivariance properties [11, 12, 13, 14, 15] , or projects from these representations to 2D images originating from canonical camera poses [16] . While these approaches have been shown to be robust to novel camera views [3] , they require well-calibrated camera extrinsics, which can be practically challenging and time-consuming to collect, and are not present in all existing datasets (for example, the majority of datasets in Open X-Embodiment [1] do not provide camera extrinsics).Rather than rely on explicit 3D representations, a related body of work learns latent representations that are robust to variations in camera pose. These methods often use view synthesis or contrastive learning as a pretraining or auxiliary objective [17, 18, 19, 6] , and also often require accurate extrinsics, can be computationally expensive to run at inference time, or impose restrictive requirements on the latent space that can make multi-task learning challenging.A technique that has shown promise in reducing the observational domain gap in robotic learning settings is the use of wrist-mounted or eye-in-hand cameras as part of the observation space [20, 21] . However, this does not obviate the need for third-person observations as it only provides information local to the gripper. We corroborate in our experiments that wrist-mounted camera observations are helpful but not solely sufficient for learning viewpoint-robust policies, and further that the use of wrist cameras can yield improvements orthogonal to the use of augmentation for third-person views.Single-image novel view synthesis. Single-image novel view synthesis methods aim to reconstruct an image from a target viewpoint given a single image from a source viewpoint. One set of methods for novel view synthesis infers neural radiance fields from one or a few images [22, 23] . Another recent line of work trains diffusion models on images to perform novel view synthesis, and then distills 3D representations from these models [24] . These approaches have been extended to scenelevel view synthesis [8, 25, 26] , making them amenable to robotic manipulation settings. They have been largely developed, trained, and evaluated on large video datasets; however, to our knowledge, their application in robotic policy learning remains relatively unexplored.Generative image models in robotics. Pretrained image generation models have been applied in the context of robotic manipulation via semantic data augmentation [27, 28] , where the goal is for the policy to better generalize to unseen backgrounds or objects as opposed to camera viewpoints. Similar generative models have also been applied to improve cross-embodiment transfer of policies [29] and as high-level planners using an image subgoal interface [30, 31, 32, 33, 34] . Overall, these methods address different challenges and are largely complementary to our method. The techniques we discuss can be flexibly applied to many visuomotor policy learning settings; however, for a systematic and computationally constrained evaluation, we choose to study them in the context of visual imitation learning. 
problem statement: 
 Perturbation Quarter Circle ArcWe frame each robotic manipulation problem as a discrete-time partially observed Markov decision process (POMDP), with state space S, action space A, transition function P , reward function R, and observation function O. This observation function maps states and actions into the observation space conditioned on extrinsic parameters E. We assume access to a dataset D consisting of M expert demonstrations τ 0:M : τ i = (o 0 , a 0 , . . . , o t , a t , . . . , o T , a T ) where T is the total number of timesteps in a particular demonstration. Concretely, the observation o consists of both low-dimensional observations in the form of robot proprioceptive information, as well as RGB image observations o I ∈ R H×W ×3 captured by a fixed third-person camera with extrinsics E orig .The objective is to learn a policy π(a|o) that solves the task, where observed images o I are captured by a camera with extrinsics E test sampled from a distribution E test . Critically, we do not assume access to the environment or the ability to place additional sensors at training time.We define the single-image novel view synthesis (NVS) problem as finding a function M(I context , f, E context , E target ) that, given an input image I context ∈ R H×W ×3 of a scene captured with camera extrinsics (e.g., camera pose) E context and simplified intrinsics represented by a field of view f , renders an image of the same scene I context captured with camera extrinsics E target .To extend this setting to zero-shot novel view synthesis, we further assume that the image I orig depicts a robotic task that was not seen when training M. As we will describe in Section 5, we conduct experiments on both models that have never seen robotic data, as well as models finetuned on robotic data from simulated pre-training tasks and large-scale, real-world data. 
learning view invariance with zero-shot novel view synthesis models: 
 In this section, we describe VISTA, the data augmentation scheme for view-invariant policy learning that we study in the remainder of the paper. The method is summarized in Figure 3 .To learn viewpoint-invariance, some prior works augment experience with images rendered from virtual cameras in simulation [5, 6] . However, we wish to learn viewpoint-invariant policies directly from existing offline datasets, which could be from inaccessible simulated environments or data collected in the real world. Furthermore, many robotic datasets do not contain the multiview observations or depth images needed for 3D reconstruction. Thus, we explore using single image novel view synthesis methods to perform augmentation.Concretely, given a single-image novel view synthesis model M(I context , f, E context , E target ), VISTA uses M to replace each frame of a demonstration trajectory with a synthesized frame with independently randomly sampled target extrinsics E target ∼ E train . That is, we independently replace each observation-action pair (o, a) with (M(o I , f, E context , E target ), a). For the sake of systematic evaluation, in our simulated experiments, we assume knowledge of both the initial camera pose E context and the target distribution E target . However, the novel view synthesis models we study use only the relative poses between E context and E target ; absolute poses are not required and are not used in real-world experiments. We assume that the field of view is known.VISTA has several appealing properties. First, while methods that form explicit 3D representations must either use multi-view images or assume static scenes when performing structure-from-motion, it avoids the computational expense of 3D reconstruction and takes advantage of the fact that a scene is static at any slice in time. Second, VISTA does not add additional computational complexity at inference time, as the trained policy's forward pass remains the same. Lastly, VISTA inherits improvements in the modeling and generalization capability of novel view synthesis models.We center our analysis around a particular novel-view synthesis model, ZeroNVS [8] . ZeroNVS is a latent diffusion model that generates novel views of an input image given a specified camera transformation. It is initialized from Stable Diffusion [35] and fine-tuned on a diverse collection of 3D scenes, therefore achieving strong zero-shot performance on a wide variety of scene types. Moreover, as a generative model, it tends to generate novel views which are crisp and realistic, mitigating the domain gap between generated and real images.Although ZeroNVS provides reasonable predictions even in zero-shot settings, we found that it also has failure modes that generate images that appear to contain extreme close-ups of objects in the scene, potentially due to poor extrinsics in the training dataset. To partially address these scenarios, we simply reject and resample images that have a perceptual similarity (LPIPS) [36] distance larger than a value η from the input image, which we found to slightly improve performance.While many techniques for imitation learning have been proposed, as a strong baseline that fits our computational budget, we use the implementation of behavior cloning with a Gaussian mixture Figure 4 : Qualitative examples of novel views rendered on robotic tasks. All images are synthesized zero-shot; that is, models have not been previously trained on data from that task. We observe that finetuning on robotic datasets improves image fidelity, particularly for robot appearances.model output from robomimic [37] in our simulated experiments. In real-world experiments, we instead train diffusion policies [38] due to their success in learning policies for real robots.For additional implementation details and pseudocode of VISTA, please see Appendix A. 
experimental analysis: 
 In this section, we perform empirical analyses to answer the following questions:Q1: Can policies trained with data augmented by novel view synthesis models trained on largescale out-of-domain datasets improve robustness to novel viewpoints? How do these models compare to existing alternatives? Q2: Can finetuning novel view synthesis models on robotic data improve the performance of VISTA when applied to unseen tasks with larger viewpoint changes? Q3: How do methods providing augmented third-person viewpoints interact with strategies for reducing the observational domain gap, such as adding wrist-mounted cameras? Q4: Can VISTA be applied to learn policies on real robots directly using real-world data? How does finetuning view synthesis models on diverse real robot data affect downstream performance?Simulated experimental setup. We perform simulated experiments using the robomimic framework built on the MuJoCo simulator [39] , with additional tasks introduced in MimicGen [40] . For the Lift and Nut Assembly tasks, we use the proficient-human expert demonstration datasets from robomimic for training. For the remainder of the tasks, we train using the first 200 demonstrations of D0 datasets and evaluate using the D0 environment variants as defined by MimicGen.To contextualize the results, we introduce the following baseline methods:• Single view. To represent the performance of a model trained without view-invariance, this performs behavioral cloning using only the source demonstration dataset. • Simulator (oracle). As an upper bound of the performance of per-frame random augmentation for learning viewpoint invariance, this baseline directly uses the simulator to render novel viewpoints. • Depth estimation + reprojection. This baseline follows the augmentation scheme described in Section 4. It synthesizes novel views from RGB images using a three-stage pipeline. Because we do not assume access to depth, it first performs metric depth estimation using an off-the-shelf model [41] . Next, it lifts the RGBD information into a 3D point cloud with a pinhole camera model and then renders the point cloud into an RGB image at the target camera extrinsics. Finally, because this reprojection often produces partial images, we perform inpainting of "holes" and outpainting of image boundaries using a pretrained diffusion model [35] . • PixelNeRF. To evaluate differences between novel view synthesis models, we evaluate a method that performs per-frame viewpoint augmentation using a PixelNeRF [22] model trained on the same mixture of datasets as ZeroNVS. PixelNeRF uses a convolutional encoder to condition a neural radiance field [42] , which is then rendered from the novel viewpoint.Further details regarding baseline implementations and hyperparameters are in Appendix A.Q1. Performance of pre-trained novel view synthesis models. First, we seek to evaluate the performance of view synthesis models that rely on large-scale, diverse pretraining. We test a distribution of test camera poses denoted perturbations, that are representative of incremental changes, for instance, that of a physical camera drifting over time or subject to unintentional physical disturbance. Specifically, this distribution is parameterized by a random translation ∆t ∼ N (0, diag(σ 2 t )) and rotation around a uniformly randomly sampled axis, where the magnitude is sampled from N (0, σ r 2 ). Samples from this range are visualized in Figure 2 , and example observations are in Appendix B.The results are presented in Table 1 . First, we note that the oracle simulator augmentation scheme is able to reclaim a significant portion of policy performance that is lost by only training on the original data (single view). We find that the depth estimation + reprojection method is able to consistently provide modest improvements to the performance on test viewpoints. Among the fully neural methods, PixelNeRF does not synthesize views with sufficient fidelity, and causes even a drop in performance compared to not doing augmentation. We thus omit this baseline in further evaluations. However, we find that a pretrained ZeroNVS model, despite (likely) having never seen an image of a robotic arm during training, is able to improve novel view performance even further.Q2. Effect of finetuning view synthesis models on in-domain data. Next, we investigate whether finetuning these novel view synthesis models on in-domain data can yield improved performance when applied to unseen tasks. To test this, we study a more challenging distribution of camera poses with a real-world analogue to constructing another view of a given scene. We first compute a sphere centered at the robot base and containing the initial camera pose. We then sample camera poses on the sphere at the same z height and within a 90°azimuthal angle of the starting viewpoint. The radius of the sphere is further randomly perturbed with Gaussian noise with variance σ 2 r . We call this distribution quarter circle arc, with samples shown in Figure 2 and more details in Appendix B.We finetune the ZeroNVS model on a multi-view dataset generated using eight MimicGen tasks: stack three, square, three piece assembly, mug cleanup, pick place can, nut assembly, kitchen, and coffee prep. Additional finetuning details can be found in Appendix B. We then test the model, denoted ZeroNVS (MimicGen), when used for augmentation on datasets of held-out tasks, which we categorize below by their relationship with the finetuning tasks.• Unseen Object: Tasks contain objects that are not present in any finetuning tasks.• Shared Object: Tasks contain objects that are present in one or more finetuning tasks, but in the context of a set of different objects or scenes. • X-Embodiment: The same task is present in the finetuning data, but is performed by a different robot (Rethink Sawyer instead of Franka Panda).Quantitative results are presented in Table 2 . In this more challenging setting, improvements from the depth estimation + reprojection baseline are much more limited, likely because many requested novel viewpoints are outside the original camera's viewing frustum. The pretrained ZeroNVS model yields modest improvements on all tasks. We see the best performance when using the model finetuned on the MimicGen data, often doubling the success rate of the next best method.Qualitatively, as seen in Figure 4 , we find that the ZeroNVS model finetuned on MimicGen data produces higher fidelity images, particularly with respect to the robotic arm's appearance. We find as per expectation that wrist observations improve performance across the board, as they are agnostic to third-person viewpoint. These improvements complement those of view augmentation methods.Q3. Use of wrist-mounted cameras to reduce domain gap. Wrist-mounted cameras are a popular and effective approach to improving visuomotor policy performance and reducing domain shift due to changes in visual observations [20] . In this experiment, we examine the effect of using wrist-camera observations in conjunction with augmented third-person views. The results are shown in Figure 5 . We see that adding wrist camera observations slightly improves performance on the threading task for all augmentation techniques, suggesting that methods for view-invariance for third-person views can be complementary to the use of wrist cameras. For the threading task, the performance of a policy using solely wrist observations, which are unperturbed at test time, is 58%. This is better than even our strongest policy using a thirdperson view augmentation model. However, the performance of wrist-camera-only policies may be limited for many tasks [20] . For instance, in threading, the oracle augmentation + wrist camera policy achieves a 73% success rate using the original third-person viewpoint.Q4. Performance on real robots. We further investigate the performance of VISTA when training policies on real-world data. Critically, we also seek to validate whether finetuning NVS models on large-scale real multi-view robotic data can improve performance for real-world policies.To test this, we first finetune a ZeroNVS model on a subset of the DROID [7] dataset, which contains over 75k trajectories of a variety of tasks in diverse environments. We randomly sample a subset of 3000 trajectories in DROID and sample 10 random timesteps within each trajectory as "scenes" for finetuning, using the four external views from two stereo cameras as paired data. We then collect a dataset consisting of 150 expert demonstrations on a Franka Emika Panda robot for the task place cup on saucer from a single camera viewpoint. We train diffusion policies [38] on this data following the configuration in DROID [7] with four policy variants as follows: Original Table 3 : Real robot policy success rates. We evaluate each rollout's success on two stages of the task:Reaching the cup and positioning the gripper for a grasp as determined by a human rater, and Completing the full place cup on saucer task. Camera 5 results for "original data + wrist" are omitted as the policy exhibited qualitatively essentially random behaviors for novel views. We find that the ZeroNVS augmentation improves viewpoint robustness, and using the DROID-finetuned NVS model yields additional gains.data + wrist uses the third-person camera view and wrist view as policy inputs. ZeroNVS aug + wrist additionally performs augmentation on the third-person view using the ZeroNVS model from the original paper. ZeroNVS (DROID) + wrist uses the NVS model finetuned on DROID data instead. Finally, representing an alternative approach that sidesteps the viewpoint shift problem entirely by only using the wrist camera, which is always fixed related to the end effector, wrist only is a baseline that does not use third-person camera inputs. We evaluate these policies when the external observations are captured from the original viewpoint and four novel views (see Figure 6 ). Full finetuning and real world experimental setup details are in Appendix C.The results, presented in Table 3 , indicate that VISTA is also effective in improving policy viewpoint robustness in the real world settings. Further, we see a performance gain from finetuning the NVS model on a large, diverse robotic dataset. In contrast, the policy trained on the single viewpoint data struggles to reliably reach toward the cup under viewpoint changes. 
conclusion and limitations: 
 Limitations. While VISTA is effective at improving the viewpoint robustness of policies, it does have certain limitations. First, the computational expense of generating novel views can be significant. Second, augmenting views during policy training can increase training time and therefore computational expense. Third, sampling views during data augmentation requires some distribution of poses from which to sample. This distribution must cover the reasonable space of views expected at deployment time. Fourth, single-image novel view synthesis models often perform poorly when the novel view is at a camera pose that differs dramatically from the original camera pose, and this limits the distribution from which views may be sampled during data augmentation. 
test on novel observation viewpoints: 
 A foundation model for robotic manipulation must be able to perform a multitude of tasks, generalizing not only to different environments and goal specifications but also to varying robotic embodiments. A particular robotic embodiment often comes with its own sensor configuration and perception pipeline. This variety is a major challenge for current systems, which are often trained and deployed with carefully controlled or meticulously calibrated perception pipelines. One approach to training models that can scale to diverse tasks as well as perceptual inputs is to train on a common modality, such as third-person RGB images, for which diverse data are relatively plentiful [1] .A challenge in using these data is that policies learned by current methods struggle to generalize across perceptual shifts for single RGB images. In this paper, we study one ubiquitous and practically challenging shift: when the camera viewpoint is altered. Prior studies have found that policies trained on RGB images collected from fixed viewpoints are consistently unable to generalize to visual inputs from other camera poses [2, 3, 4] .Existing approaches to learning viewpoint invariance include training using augmented data collected at scale in simulation [5, 6] or physically varying camera poses when collecting large-scale real robot datasets [7] . However, these strategies require resolving the additional challenges of simto-real transfer and significant manual human effort, respectively.In this work, we leverage the insight that 3D priors can be obtained by generative models from large-scale (potentially robot-free) data and used to make robot policies more robust to changes in camera pose. We take a simple data augmentation approach to this problem by sampling views from a 3D-aware image diffusion novel view synthesis (NVS) model during policy training time.In training on these augmented views, the policy becomes robust to images from out-of-distribution camera viewpoints. We refer to this approach as View Synthesis Augmentation (VISTA).VISTA has several advantages. First, it can leverage large-scale 2D image datasets, which are more diverse than existing robotic interaction datasets with explicit 3D observations. Second, if in-domain robotic data is available, performance may be further improved via finetuning. Third, neither depth information nor camera calibration is required. Fourth, no limitations are placed on the form of the policy. While we focus on imitation learning, VISTA can also be applied to other robotic learning paradigms. Lastly, policy inference time is not impacted, as we do not modify inference behavior.We first investigate the performance of a diffusion-based novel view synthesis model, ZeroNVS [8] , when applied using our VISTA data augmentation scheme, and perform an empirical analysis of its performance with respect to various viewpoint distributions. Then, we investigate how finetuning an NVS model with in-domain data of robotic tasks can improve downstream policy robustness for held-out tasks. Finally, we show that these models can be used to learn viewpoint-robust policies from real robotic datasets. We demonstrate the potential for NVS models trained on large diverse robotic data to provide these priors across robot tasks and environments, finding that finetuning Ze-roNVS models on the DROID dataset [7] can improve downstream real-world policy performance. 
 
 Appendix: In this paper, we presented VISTA, a simple yet effective method for making policies robust to changes in camera pose between training and deployment time. Using 3D priors from single image view synthesis methods trained on large-scale data, VISTA performs data augmentation to learn policies invariant to camera pose in an imitation learning context. Experiments in both simulated and real world environments demonstrated improved robustness to novel viewpoints of our approach over baselines, particularly when using view synthesis models finetuned on robotic data (though applied zero-shot with respect to tasks). There are a number of promising directions for future work, but of particular interest is studying the performance of this data augmentation scheme at scale across a large dataset of robotic demonstrations.Please see our project website at https://s-tian.github.io/projects/vista for code implementations, pretrained model weights, videos, and additional visualizations.All novel view synthesis methods that we consider generate novel views at a resolution of 256 × 256 given RGB images at a resolution of 256 × 256. The synthesized images are later downsampled for policy training. To clarify how these models are used in a policy learning pipeline, we provide pseudocode of the VISTA algorithm in Algorithm 1.Algorithm 1 VISTA: Learning View Invariant Policies via Novel View Synthesis.Require: Dataset D consisting of trajectories τ 0:M : τ i = {(o 0 , a 0 , • • • , o t , a t , • • • o T , a T )},D ′ ← D ′ ∪ (o ′ t , a t )▷ Add augmented transition to buffer.else 15:D ′ ← D ′ ∪ (o t , a t )▷ Add original transition to buffer. 16: ▷ Learn policy on augmented dataset, in our case, imitation learning via behavior cloning. ◁ 17: π ← LEARNPOLICY(D ′ ) 18: return πZeroNVS is a latent diffusion model that generates novel views of an input image given a specified camera transformation. It is initialized from Stable Diffusion and then fine-tuned on a diverse collection of 3D scenes, and therefore achieves strong zero-shot performance on a wide variety of scene types. Moreover, as a generative model, it tends to generate novel views which are crisp and realistic, mitigating the domain gap between generated and real images. This distinguishes ZeroNVS from methods such as PixelNeRF [22] , which are trained with regression-based losses and tend to produce blurry novel views even for small camera motion.We use the implementation and pretrained checkpoint provided by Sargent et al. [8] . As mentioned in Section A, although ZeroNVS largely produces reasonable views even zero-shot, it can sometimes produce images with significant visual artifacts. To filter these out, we reject and resample images that have a LPIPS [36] distance larger than a hyperparameter η from the input image. We set η = 0.5 for all simulated experiments and η = 0.7 for real experiments. We do not extensively tune this hyperparameter. If the model fails to produce an image with distance < η after 5 tries, the original image is returned.ZeroNVS also requires as input a scene scale parameter. To determine the value of the scene scale for simulated experiments, we perform view synthesis using the pretrained ZeroNVS checkpoint on a set of 100 test trajectories for the lift and threading environments, and compute the LPIPS score between the ZeroNVS rendered images and ground truth simulator renders for values {0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95}. We find that the lowest error across the tasks is achieved at 0.6 and thus use 0.6 for all environments, including the real robot experiments.While the behavior of the ZeroNVS model is somewhat sensitive to scene scale, we believe this may be alleviated by selecting a wider viewpoint randomization radius at training time, which is corroborated by our real robot experiments.When sampling, we perform 250 DDIM steps and use a DDIM η of 1.0. We use a field of view (FOV) of 45 degrees for simulated experiments (obtained from the simulator camera parameters) and FOV of 70 degrees for the real world experiments (obtained from the Zed 2 camera datasheet).Sampling the diffusion model for NVS is roughly similar to sampling from the vanilla Stable Diffusion model; it takes on average 8.7 seconds to generate a single 256 × 256 image with ZeroNVS using these settings on a single NVIDIA RTX 3090 GPU.This baseline represents a geometry-based approach that leverages depth estimation models trained on large-scale, diverse data.First, we use ZoeDepth (ZoeD NK) [41] , an off-the-shelf model, to perform metric depth estimation on the input RGB image. Next, we deproject the images into pointclouds using a pinhole camera model. We rasterize an image from the points using the Pytorch3D point rasterizer [43] , setting each point to have a radius of 0.007 and 8 points per pixel. Finally, we use a publicly available Stable Diffusion inpainting model (https://huggingface.co/runwayml/ stable-diffusion-inpainting) to inpaint regions that are empty after rasterization. We use 50 denoising steps as per the defaults. It takes on average 2.8 seconds to generate a single 256 × 256 image with this baseline on a single NVIDIA RTX 3090 GPU.For PixelNeRF [22] , we use the implementation from the original authors at https://github. com/sxyu/pixel-nerf. We use a pretrained model trained on the same datasets as ZeroNVS [8] . It takes on average 5.8 seconds to generate a single 256 × 256 with PixelNeRF on a single NVIDIA RTX 3090 GPU.Here we provide details regarding the simulated experimental setup. As a high level goal, we aim to minimize differences from our setup from existing robotic learning pipelines to demonstrate how this augmentation technique can be generally and easily applied across setups.Our simulated experiments use environments created in the MuJoCo simulator and packaged by the robosuite [44] and MimicGen [40] frameworks.For the Lift, PickPlaceCan, and Nut Assembly tasks, the training datasets are the Proficient-Human datasets for those tasks from robomimic [37] and consist of 200 expert demonstrations each. For all MimicGen tasks (Threading, Hammer Cleanup, Coffee, Stack) the datasets consist of the first Original View 200 expert trajectories for the "core" MimicGen-generated datasets, downloaded from https:// github.com/NVlabs/mimicgen_environments.We use the same distribution of viewpoints at training time for augmenting the dataset and when testing the policies. Note, however, that images generated by novel view synthesis models are not guaranteed to actually be from the target viewpoint -only the oracle that uses the simulator to render the scene satisfies this.Due to the lack of widely adopted testing settings for testing robotic policies on novel views and that the effect of a particular view distribution is highly environment dependent, the hyperparameters for the view distribution were selected by hand by the authors to approximate reasonable distributions that a robot learning practitioner may encounter in practice. We hope these distributions may also be reasonable testing settings for evaluating future methods on these tasks.This set of viewpoints are representative of incremental changes, for instance, that of a physical camera drifting over time or subject to unintentional physical disturbance. Specifically, this distribution is parameterized by a random translation ∆t ∼ N (0, diag(σ 2 t )) and rotation around a uniformly randomly sampled 3D axis, where the magnitude is sampled from N (0, σ r 2 ). Specifically, we set σ t = 0.03 m and σ r = 0.075 rad. Samples of observations taken from viewpoints drawn from this distribution are shown in Figure 7 .This is a more challenging distribution of camera poses with a real-world analogue to constructing another view of a given scene. We first compute a sphere centered at the robot base and containing the initial camera pose. We then sample camera poses on the sphere at the same z height and within a 90°azimuthal angle of the starting viewpoint. The radius of the sphere is further randomly perturbed with Gaussian noise with variance σ 2 r . Specifically, the radius of the sphere is 0.7106 m for all simulated environments, which is the distance between the camera and the robot base in the Lift task, and σ r = 0.05 m. camera extrinisics from the DROID dataset, and use simplified intrinsics assuming a camera FOV of 68 degrees for all cameras, which we obtained from a single randomly sampled camera in the dataset. In reality, the FOV differs slightly for each camera due to hardware differences, and slightly better results may be obtained by using per-camera intrinsics.As in the simulated finetuning experiments, we again use the training settings from the original ZeroNVS paper but change the optimizer from AdamW to Adam and decrease the learning rate to 2.5e-5, and decrease the batch size to 512 due to computational constraints. We use 29000 scenes for training and 1000 for validation. As an attempt to reduce overfitting, we mix in a single shard of 50 scenes each from the CO3D and ACID datasets which are sampled for each training sample with probability 0.025 each. DROID data is sampled with probability 0.95. We did not extensively validate the effect of this data mixing due to computational cost of finetuning the model repeatedly, and it is likely unnecessary. We finetune the model for 14500 steps using four NVIDIA L40S GPUs. This takes approximately 50 hours of wall clock time.Training augmentation viewpoints. For the real world experiments, we do not have access to the test viewpoint distribution. To sample viewpoints for ZeroNVS data augmentations for these experiments, we sample from a distribution parameterized in the same way as the "perturbation" range in the simulated experiments, but with a vastly increased variance in translation and rotation magnitude intending to cover a wide range of possible test viewpoints. This distribution is parameterized by a random translation ∆t ∼ N (0, diag(σ 2 t )) and rotation around a uniformly randomly sampled 3D axis, where the magnitude is sampled from N (0, σ r 2 ). Specifically, we set σ t = 0.15 m and σ r = 0.375 rad. Policy learning. For policy learning on the real robot, we train diffusion policies [38] . Specifically, we use the implementation from the evaluation in the DROID paper [7] with language conditioning removed. The input images are of size 128 × 128, and both random color jitter and random crops (to 116 × 116) are applied to the images during training. We train all policies for 100 epochs (50000 gradient steps), using 2 NVIDIA RTX 3090 or RTX A5000 GPUs.In the experiments presented in Section 5, we perform data augmentation via novel view synthesis by doing offline preprocessing of the dataset, augmenting and replacing each transition with a single augmented transition. However, many random data augmentation strategies for neural network training perform augmentation "on-the-fly", applying augmentations to each particular batch. This increases the effective dataset size. Augmentation with novel view synthesis methods is too computationally expensive to apply per batch with our computational budget, but we are still interested in understanding how the performance of trained policies is affected by increasing the number of augmented trajectories for each original dataset trajectory.For the threading task with viewpoints sampled from the "quarter circle arc" distribution, we trained policies on dataset containing The results are shown in Figure 10 . We find that increasing the number of augmented transitions per original dataset transition yields modest improvements with both models, although there is a surprising dip in performance when using 5 augmented transitions for the ZeroNVS (MimicGen finetuned) model.Here we conduct a baseline comparison to the multi-view masked autoencoding (MV-MAE) method from Multi-View Masked World Models (MV-MWM) [6] . While MV-MWM has a similar motivation to our work, it has a very different problem setting compared to ours: they assume training-time access to a multi-view dataset, while we assume only access to an offline dataset of trajectories captured from a single viewpoint. Thus we adapt the imitation learning approach described in the MV-MWM work to our finetuning experimental setup. Specifically, we train a MV-MAE, using all hyperparameters from Seo et al. [6] on the finetuning dataset from Experiment Q2. Of particular note, we use a higher rendering resolution for this baseline (96 × 96 compared to 84 × 84 for policies trained using our novel view augmentation) to match the image resolution used by Seo et al. [6] . We train for 5000 steps, performing early stopping as we observe overfitting by monitoring the validation loss on datasets for the coffee, hammer, stack, and threading tasks. We then freeze the pre-trained encoder and use it to train policies on single-view datasets, testing on the quarter circle arc test view distribution. We show the results in Table 5 . Our approach significantly outperforms the multi-view masked autoencoding method.In Figure 11 , we provide additional qualitative results of novel views synthesized for the real world cup on saucer task. We show synthesized images for views sampled from the training view distribution described in Appendix C.3.To understand how different combinations of observation viewpoints qualitatively affect learned policies, we additionally conduct an analysis of saliency maps of learned policies trained on thirdperson views only compared to combined third-person and wrist cameras.Specifically, we visualize saliency maps of convolutional layers of both simulated and real-world policies using GradCAM++ [45, 46] in Figure 12 . We find that wrist camera observations tend to have salient features at locations corresponding to objects nearby or underneath the gripper. Policies with only third-person camera views as input tend to have more salient features corresponding to the robotic arm or gripper itself.For GradCAM++, we choose the target layer to be the common choice [46] of the last convolutional layer in ResNet18 or ResNet50 for simulated and real policies respectively. For the threading policies the target model output is the mean of the output Gaussian mixture model action distribution with the highest probability of being selected (largest logit). For the real-world cup on saucer policies the target model output is the mean of the output denoised action sequence. We visualize saliency maps on data from viewpoints sampled from the same test distributions as in Experiment Q2 (quarter-circle arc) and Experiment Q4 respectively.Figure 11 : Additional view synthesis results on real robot datasets. The column labeled "Original" is the input view, and random poses are sampled to render the images in the other columns. Note that the ZeroNVS model finetuned on DROID data (rightmost column) is consistently able to generate more crisp, realistic images than the other models, particularly with the robotic arm's visual appearance.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: The reviewers found some strengths in this submission, but also clearly articulated some questions for the rebuttal phase. 
Thank you for your responses.

Reviews: review: title: title: Simple Idea for data augmentation that show promising directions for view invariant learning
paper_summary: paper_summary: Propose a data augmentation scheme for improving generalization to novel view points for Imitation Learning Policies
main_review: main_review: The authors propose an approach for using Novel View Synthesis to perform data augmentation to improve viewpoint invariant Imitation Learning to new views during execution time. The authors use ZeroNVS, a zero-shot novel view synthesis approach that they investigate with and without fine-tuning to understand its performance on downstream tasks. Specifically, the authors augment the dataset with images rendered from novel views that are then used to train an Imitation Learning policy on the downstream task. The proposed approach achieves a higher performance on testing from unseen viewpoints which shows the efficacy of the approach both in simulation and on real robot experiments. Additionally, the authors also show the incorporation of wrist camera images to (slightly) improve the generalization capabilities.

Drawbacks:
- A lack of results of the view synthesis on the real robot would help make the results better to understand.
- A better analysis of the wrist camera improvements, would make it better to understand how it affects the task, for example via attention/saliency/convolutional maps. Especially on the real robot setting.
- There is no talk about consistency of a given view along a trajectory. It would be important to discuss how to ensure consistent systhesis along the trajectory for different views.
questions: questions: - Have the NVS trained along with the policy learning to learn some task-oriented representations, Or alternatively, any multiview similarity-based policies rather than generating augmentation data from scratch.
- How do the results of Table 3 look like without the wrist camera augmentation? From the results of Q3, while there is a general trend of increased performance due to the wrist camera, it does not seem significantly better.
- Does NVS help improve cross embodiment generalization, especially with the wrist camera view
limitations: limitations: 3
review_summary: review_summary: Simple Idea, nice execution, leads to some performance improvement
title: title: Submission 512 Review
paper_summary: paper_summary: This work leverages generative models with 3D priors in order to generate novel viewpoints for a given robot scene in a 0-shot fashion. As a result, a given dataset with demonstrations can be augmented and the policies trained on it can exhibit view-invariance. The authors preset experiments both in simulation and on real robots.
main_review: main_review: The paper is very-well written with a clearly-stated hypotheses - i.e. can we use 3D generative models to generate novel views of the same scene and how useful is that for training more robust robot policies. Both hypotheses are tested with sufficient experiments and results are presented and discussed extensively.

Strengths:
* Very high level of clarity.
* Experiments are done for both simulated and real-world robots.
* The problem for making robot learning more robust is very relevant to the field
* Results are demonstrated in video form too - further eases the reader.
* Extensive list of method's limitations.

Weaknesses (some of these are discussed but still):
* Some of the hyper-parameter values - e.g. for the novel view distribution - are not explained.
* No code is mentioned - any plans to release it later?
* It is not quite clear how do we involve the wrist-mounted cameras in the overall paper narrative. Yes, its yet another view of the scene, demonstrating that the more views the better but it doesn't have much to do with the overall thesis which revolves around *view synthesis* with generative models and can thus be somewhat distracting from the main story (as per the title).
questions: questions: 1) Can you give some more details around the diffusion models used - how easy is it to sample them, fine-tune them (how much resources are required) and how easy is it to add them to the training process. I.e. is training policies with your method 2-3 times slower, for example (or more/less)?
2) Table 3 has missing entries and it's unclear why - can you comment? E.g. for wrist-only model we have dashes everywhere and then 16/20 and 5/20 at the end.
3) The distribution of camera parameters for sampling new views - how do we arrive at those practically. Is there some way of procedurally knowing when we go outside of the distribution of camera parameters that the generative model supports? You mention LPIPS but that is going to be high even if we take the same picture and do affine transforms on it let alone render a new view. Maybe some other metrics from the generative models literature that can give a sense of how real the novel views are - e.g. https://arxiv.org/abs/2002.09797
limitations: limitations: 3
review_summary: review_summary: I recomment weak accept and am happy to update to strong accept depending on the discussion and answers.
title: title: Official Review of Paper512 by Reviewer EcF3
paper_summary: paper_summary: This paper proposes a method to learn policies robust to changes in camera pose between training and deployment time.  A diffusion-based novel view synthesis model,  ZeroNVS, is used for generating images of the same scene from alternate camera viewpoints given a single input image. Then they replace the original images in the demonstration trajectories with the generated images of the novel view in the training process. Therefore, the trained policy can be applied to novel visual configurations in a zero-shot manner. Experiments demonstrate good zero-shot transfer performance to novel camera views in both the simulation and the real world.
main_review: main_review: Strength:

- Studies an important problem (adaptation ability of visuomotor policies to novel camera pose) and introduction motivates the problem well.

- Great to see real robot results supporting the simulation results!

- Experiment results show meaningful improvements in performance compared to baselines chosen in this paper.


Weakness:

- I think the algorithm presented in this paper is rather vague and unclear. To let the author easily get the whole pipeline of the proposed algorithm, providng clear pseudocode of it will be of great help.

- The baseline methods chosen in this paper is not strong enough. I would like to see the performance comparison of the proposed method with Multi-View Masked World Model (MV-MWM).

- The presentation of the Zero-Shot Novel View Synthesis in section 3.2 is too brief. ZeroNVS is the major technical component of the proposed method, and not everybody in the robotics community is familiar with ZeroNVS. I would like to see more details that reveal the mechanism of the ZeroNVS. How it works? Why it works better than the other baselines such as PixelNeRF?
questions: questions: Please see the list of weaknesses in the section above.
limitations: limitations: 3
review_summary: review_summary: The paper is well motivated with a good set of experiments in both the simulation and the real world. Still experiments have room for growth (can add a stronger baseline for comparison) and writing has room for improvement (can have a better presentation of the proposed algorithm).