From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos
Authors: Matthew Wallingford
Anand Bhattad
Aditya Kusupati
Vivek Ramanujan
Matt Deitke
Aniruddha Kembhavi
Roozbeh Mottaghi
Wei-Chiu Ma
Ali Farhadi
Venue: NeurIPS 2024 poster (2024)
Keywords: Novel View Synthesis
3D
Video
360 Video
Large-Scale
Data
Scene Generation
Primary area: machine_vision
TL;DR: Empowered by the largest real-world, multi-view dataset to date, we present the first model to reasonably synthesize real-world 3D scenes and reconstruct their geometry conditioned on a single image.

Abstract: Three-dimensional (3D) understanding of objects and scenes play a key role in humans' ability to interact with the world and has been an active area of research in computer vision, graphics, and robotics. Large scale synthetic and object-centric 3D datasets have shown to be effective in training models that have 3D understanding of objects. However, applying a similar approach to real-world objects and scenes is difficult due to a lack of large-scale data. Videos are a potential source for real-world 3D data, but finding diverse yet corresponding views of the same content have shown to be difficult at scale. Furthermore, standard videos come with fixed viewpoints, determined at the time of capture. This restricts the ability to access scenes from a variety of more diverse and potentially useful perspectives. We argue that large scale ODIN videos can address these limitations to provide scalable corresponding frames from diverse views.  In this paper we introduce 360-1M, a 360° video dataset consisting of 1 million videos, and a process for efficiently finding corresponding frames from diverse viewpoints at scale. We train our diffusion-based model, ODIN, on 360-1M. Empowered by the largest real-world, multi-view dataset to date, ODIN is able to freely generate novel views of real-world scenes. Unlike previous methods, ODIN can move the camera through the environment, enabling the model to infer the geometry and layout of the scene. Additionally, we show improved performance on standard novel view synthesis and 3D reconstruction benchmarks.

From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos
Matthew Wallingford
Anand Bhattad
Aditya Kusupati
Vivek Ramanujan Matt Deitke
Sham K Aniruddha Kembhavi Roozbeh Mottaghi Wei-Chiu Ma Ali Farhadì
University of Washington Toyota Technological Institute at Chicago
Harvard University AAllen Institute for AI
mcw2440cs waw.iiesswwainntoooooo edu
Abstract
Thrree-dmenniooal (3D) understanding of objects and scenes play a key role in humans' ability interact with the world and has been an active area of research computer vision, graphics, and robotics. Large scale synthetic and object-centric 3D datasets have shown to be effective in training models that have 3D understanding of objects. However, applying a similar approach to real- wolld objects and scenes is difficult due lack of large-scale data. Videos are a potential source for real- world 3D data, but finding diverse yet corresponding views of the same content has shown to difficult at scale. Furthermore, standard videos come with fixed viewpoints, determined at the time of capture. This restricts the ability to access scenes from variety of more diverse and potentially useful perspectives. We argue that large scale 360 videos can address these limitations to provide: scalable corresponding frames from diverse views. this paper, we introduce 360- 1M, 360 video dataset, and a process for efficiently finding corresponding frames from diverse viewpoints at scale. We train our diffusion-buat model, ODIN', on 360-1M. Empowered by the largest real- world, multi-view dataset to date, ODIN is able to freely generate novel views of real-world scenes. Unlike previous methods, ODIN move the camera through the environment, enabling the model to infer the geometry and layout of the scene. Additionally, we show improved performance standard novel view synthesis and 3D reconstruction benchmarks.
1 Introduction
Humans have the ability to understand and reason about the 3D geometry the world, which is key
for everyday tasks such navigation and object manipulation [14, 67, 62, 48]. In machine learning, 3D perception and reasoning has been a wangsstandng goal for researchers with broad applications in roboties [58, 20, 61], vision [27, 25], and graphics [22, 57]. Fueled by large-scale datasets of synthetic objects [10, 11], recent generative models have shown impressive understanding of 3D) objects [24, 44, 31]. While these models' ability to generate synthetic objects impressive, enabling 3D generative models for real world scenes and objects remains an open challenge.
One intuitive source for scalable data has been video as it implicitly contains rich information about the 3D world. However, learning 3D modeling from video has been elusive despite impressive effort [66, 36, 19, 37, 38]. The key problem has been how to consistently transform video into form amenable to learning about the 3D world. Existing 3D models learn from multi-view data, a
In Norse mythology, Odin uses his ravens, Huginn and Muninn, as his eyes to fly throughout the world and relay what they see.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Generated 3D Scene
Starting Input View
Synthesized View
Synthesized View
Translate: Right 20m Rotate: 5" left
Synthesized View
Translate: Forward 30m, Right 5m Rotato: 20% left, 5 Upward
Translate: Back 20m, Right 15m Rotate: 15 left
Figure I: By learning from the largest real-world, multi-view dataset to date, our model ODIN, rich scenes from single input image with free camera movement
synthesize novel views of throughout the scene. We can then reconstruct the 3D scene geometry from these geometrically consistent generations.
collection of images of scenes or objects and their respective camera pose. Creating such multi-view datasets from video requires finding sets of corresponding frames that capture similar parts of the scene but from different locations (Figure 2).
This search for corresponding frames in video has proven difficult at scale for a few reasons. First, correspondences are sparsely distributed throughout the video because the trajectory of the camera is fixed at the time of capture. Ideally, the camera operator would focus on specific object or portion of the scene while moving around However, in-the- wild videos are far from this ideal. For example, if a person records themselves walking in the park, it is rare that they consistently focus the camera on the same object such as bench as they walk towards, past, and away from it. Second, the computational cost of checking whether frames form correspondence is expensive [42, 40, 5], therefore searching extensively is infeasible. Given these limitations, the largest real-world multi-view datasets to date [34, 65] utilize Amazon Mechanical Turkers to manually record video clips of objects, and are limited to 50 and 238 object categories respectively.
To address these limitations, we collect one million 360 videos from YouTube, introduce process to efficiently transform 360 video into multi-view data, and train diffusion-baasn novel-view synthesis (NVS) model on the dataset. Our model named ODIN, is the first to reasonably synthesize real-world 3D scenes and reconstruct their geometry conditioned on a single image. Quantitatively we evaluate our method on standard novel view synthesis benchmarks (DTU and MipNeRF360) and find improved performance compared to existing models without fine-tuning our own. Additionally, compare ODIN to existing methods for 3D reconstruction on Google Scanned Objects well

---

as a held-out set of 360- IM and show significantly improved performance, especially on complex real-world scenes. We will open-source our model and dataset.
2
Related Work
Novel View Synthesis. NeRF [26] optimizes a volumetric scene function using sparse 2D images, representing the scene as a continuous 5D function. MipNeRF [3] extends NeRF with multi-scale representation to enhance detail and reduce aliasing. Plenoctree [63] combines NeRF principles with an octree structure for efficient rendering. DIVeR [55] proposes a deterministic volumetric rendering for NeRF Gaussian Splatting [18] uses Gaussian functions and splatting techniques for detailed scene representation and rendering. Unlike these methods which rely densely sampled multi-view images and known camera poses, our approach captures extensive real-world scenes from widely varying camera views. PixelNeRF [64] and DietNeRF [16] extend NeRF to handle sparse imput views but only for controlled scttings.
Recent works leverage powerful generative (diffusion) models [35] for novel view synthesis of objects [31, 21, 53, 6], and more recently for scenes [6, 39, 8]. ZeroNVS uses a 3D-aware diffusion model with novel camera conditioning to generate 360-degree views from single image, focusing on depth-scale ambiguity and background diversity with synthetic and real-world datasets. Diffussio with Forward Models [47] integrates forward model into the diffusion process for unsupervised training on partial observations, solving inverse problems like view synthesis without direct signal supervision. ReconFusion [56] combines NeRFs with diffusion priors to enhance 3D reconstruction from limited views, improving geometry and texture plausibility with real and synthetic multi- view datasets. LucidDreamer [8] and RealmDreamer [45] use a multi-step pipeline involving point cloud guidance and Gaussian splats to generate detailed 3D scenes from text or image prompts but lacks physical realism and has limited control over viewpoint changes. In contrast, leverages a large-scale collection of 360-degree YouTube videos to train a diffusion- based model, enabling the synthesis of diverse real-world 3D scenes and reconstruction from single image, thus accommodating significant camera view changes and broader range of scenarios.
method
Camera Pose Estimation and Structure from Motion. Estimating camera pose and structure- from-motion (SfM) have rich history in computer vision [2, 43, 50]. Camera pose estimation consists of estimating the 6 degrees of freedom of cameras from which images were taken and the camera intrinsics. The process typically involves finding corresponding key-points between multiple images of a scene, and using their apparent motion within the images to infer the 3D geometry of the scene and relative location of each camera. For multi-view datasets and novel view synthesis, works typically use COLMAP [42, 40] or SLAM variant [46, 28]. We choose to use the recent method Dust3R [52] as it is computationally faster and allows for as few as 2 images whereas most SIM methods require dozens. This enables us to scan much more quickly through videos for frame correspondences and create the scrge-scale dataset from 360 video.
Multi-View Datasets. Existing multi-view datasets such as MVImageNet [65]. CO3D [34], RealEsttae [68], ACID [23], Epic- Sitic-Kithhens [9], MipNeRF-360 [4], and Epiic-Fielss [49] provide valuable multi-view sequences of real-world scenes and objects but are often constrained by the specific environments objects they capture. MVImageNet is the largest multi-view dataset to date with over 200,000 video clips captured of 238 object categories. Though this effort is impressive, using Mechanical Turkers to manually capture videos of objects is difficult to scale further and limits content diversity. In Figure we show examples of correspondences within MVImageNet which can be compared to correspondences of 360-1M in Figure 8. Large-scale 3D object datasets like Objaverse [10], Objaverse-XL [11]. and infinigen [32] focus on detailed 3D object assets to generate synthetic objects and scenes. Autonomous driving and 3D reconstruction datasets such as Kitti [15], DTU [1], ShapeNet [7], and Google Scanned Objects [13] offer multi-view data for specific tasks like driving scenarios, 3D modeling, and object classification.
In contrast, our dataset leverages a large-scale collection of 360-degree YouTube videos, providing a vastly more diverse and extensive source of real-world data. Our dataset accommodates significant camera view changes and broader real-world applications, going beyond the constraints of controlled multi-view datasets and specific domain focuses.
3

---

Standard Video Trajectory
360 Video Trajectory
Possible
Viown
Fland Frustrum
Selected
Views
Figure 2: Left: An illustrative trajectory of standard video with the view point fixed at the time of capture. The fixed view point makes finding corresponding frames challenging. Right: The trajectory of 360 video through the scene. The controllable camera enables alignment of views different frames of the video.
3 Multi-View Data from 360 Video
There are key elements missing from current multi-view datasets: scale and real-world data. Various datasets and works have managed [34, 65, 10, 11] to make progress along these dimensions individually, however, current datasets afford both aspects.
The key challenge collecting large-scale, multi-view datasets derives from the difficulty of finding high-quality frame correspondences, and estimating their relative poses. Existing structur-fromm motion algorithms, such as COLMAP [41] and HLOC [40], are slow and require many images of the same scene. In this section we detail our process for efficiently transforming 360 video into high-quality multi-view data.
3.1 Scalable Correspondence Search
There are two properties of corresponding video frames that are necessary for training novel view synthesis models: sufficiently differing viewpoints and overlapping content. In manually collected novel view synthesis (NVS) datasets this is accomplished by taking a video while circling the object. Finding frames that fit these criteria from in-the-wild video much more difficult.
A major reason is that high-quality correspondences are sparsely distributed in standard videos. For example, someone taking a video while walking down the street often keeps their camera view facing their direction of travel. So while they may capture a parked car on the side of the road while walking towards they likely will not pan their camera capture it from many angles while walking past or away from it. Therefore, it is difficult to obtain paired images of the scenes or objects from distant locations and diverse views scale. One solution to this problem 360 nature allows the views of frames to be rotated such that they contain overlapping content. Therefore given two frames that are close enough in spatial location, in theory we can align the views to look at the various regions of the scene to form multiple view correspondences.
leverage 360 videos. The
Now describe how we operationalize this approach. We begin by sub-sampling frames of the 360% video at = frame-per-second. We find empirically this to be sufficiently fast frame rate given the movement speed of the camera. The computation of the correspondence search scales with r², therefore we judicially select the frame rate. Next we perform pairwise comparison between frames within frame window of length, equirectangular projection E Φ) where 0 is the pitch, Φ is the yaw, and I is the image. We map the panoramic image four different views E[I.j * x/2, 0) for j E (I..... 4]. Thus a panoramic We then pass all pairs within the time window to the Dust3r model [52] which outputs relative pose estimate, P and confidence map. C. We take the mean confidence over the spatial components of the confidence map with height and width w. He hw Eeeo C, and filter out frames below threshold, T = 4. higher mean confidence means that the frames must be overlapping the model can accurately estimate the pose.
20. We map the 360 panoramic frames using an
frame, F; at time t produces four frames FF.,, Ft../2..... F,,3+5/2).

---

Once the correspondences have been found we refine the relative pose between them by performing gradient descent on the pitch and yaw of both equirectangular projections with respect pa. Intu- itively, we can think of this as rotating the cameras to maximize the overlap (Figure 2). After all correspondences have been found, we discard pairs with relative translation less than .25 m because they provide minimal information for training the model.
3.2 Correspondence Propagation
Computing relative pose between frames, especially for video, has been computationally prohibitive and major bottle-neck for arre-scalee multi-view datasets [9, 34, 65]. An exhaustive search between all frames a video would incur a cost of s²y² where s the number of seconds, and r the frame rate. A common approach is to limit the search with window of size L to reduce the cost to L however this limits the pairs to short-range correspondences.
We propose hybrid approach that enables finding long-range correspondences with limited additional compute. After the initial frames have been found as detailed section 3.1, we create a graph in which the nodes are frames and an edge exists if two frames have correspondence. We then perform the same procedure outlined in section 3.1 for all connected frames each sub-graph. Intuitively, two frames share corresponding third frame (connected in the graph)then the two are also likely to share a correspondence.
This approach allows us to maintain a small search window, while still finding long-range correspon- dences. We show examples of such long-range correspondences in Figures 8 and 9.
3.3 Resolving Scale Ambiguity
Dust3R and other structure-froote frotmmmo-ioo methods [29, 41, 33] output relative camera poses in dimensionless quantities, therefore need to calibrate them universal scale. We do by fusing the depth map estimates, D, from an off-the-shelff depth estimator [60], with the point map, X E hxxx predicted by Dust3R. A pointmap is a correspondence between each pixel j) and the point in 3D where the ray from pixel (i,j) intersects the scene. We anchor the dimensionless pointmap to the depth math D by optimizing for scale factor, a. in the following equation:
w ΣV 1-
h
arg min
CCslozzi Diy)b
where ziy is the depth component of Xij and Ciy is the confidence map output by Dust3R, Ciy is close to Ô for points which the model has high uncertainty and acts filter for points with poor estimates. We choose I.1 distance to limit the effect of outliers. Once we recover this scale factor, we multiply the translation of the estimated camera pose (R, t) by to obtain the metric pose estimate.
4 Dataset Collection and Statisties
To leverage the proposed scalable correspondence search (Section 3) for generating large-scale multi-view dataset, we collect the largest 360" video dataset to date. 360- IM, consisting of over 1 million 360 videos. In this section, describe the collection process and statistics for the dataset.
4.1 Collecting 360 Video
We collect all meta-data from YouTube in order to filter 360 videos. The meta-data provides information duration, view count, format, and subject category among other fields. We filter for the equirectangular format which indicates 360 video and results in 1,076,592 total videos. We then download the videos in the equirectangular format at the best quality available. We will release the deta-data for the 360 videos alongside the dataset.
We filter the downloaded videos for empty, and duplicate videos. We remove duplicated videos with a deduplication model [17] run on the thumbnails the videos. This does not guarantee the contents of the video are unique, however running over all frames is computationally infeasible.
5

---

Imput
(Ours)
Figure 3: Qualitative comparison of novel view synthesis on real-world scenes. The left and right images are conditioned on camera views from the left and right respectively. In the middle scene of the kitchen, ODIN accurately models the geometry of the table counter and chairs well as unseen parts of the scene such as the living room.
4.2 Dataset Statistics
360-1M consists of 80,567,325 unique frames extracted from 1,076,592 videos with average of 74.83 unique frames per video. The average video length 6.3 minutes and distributed in long-tail fashion (Figure 5). When searching for correspondences, we sample the videos at 1 FPS. The videos are distributed evenly across 15 subject categories, with the most popular category being Travel and Events (149,534 videos) and the least popular being Pets and Animals (8802 vidcos) (Figure 6). We find 363,417,730 total frame correspondences along with their relative camera poses.
Method
Our final goal is to generate images along viewpoint trajectory conditioned on single image of scene task known as novel view synthesis (NVS). Note that our task differs from tradition novel view synthesis work such [26] which aim to generate novel views after training many images of a single scene. Similar to prior works [39, 24], we leverage a diffusion-basee models have shown impressive capabilities in learning priors from large-scale data. An alternative is a NeRF based approach which mainly effective small-scale settings.
model. This class of
5.1 Viewpoint Conditioned Diffusion
Given a single image, E hxxx3 of a scene, our objective is to generate sequence of images, 4 from different viewpoints, (R, t) where R is the relative rotation and the translation between views. Following [24], we use latent diffusion architecture which consists of an encoder E, denoiser U-net for and decoder D. The standard diffusion training objective is:
min E II(e- ( fo(z,, f(r (,t))
Our modeling objective differs from previous work in that we condition on both rotation, R, and translation f. The long-range correspondences in our training data afford much freer camera movement throughout the scene compared to previous works. Due to the limitations of previous training data, other methods can only rotate about center point of the object or scene.
5.2 Motion Masking
learning how to perform novel view synthesis from videos poses challenge assumes the scene itself does not vary with time when generating images from novel viewpoints. Previous approaches

---

Single Input Images
Generated 3D Scenes
BA 111
Figure 4: Examples of generated 3D scenes using ODIN. The blue dot indicates the location of the input image and the red lines indicate the trajectory of the camera which generated the images. ODIN is capable of long-range generation of geometrically consistent images. In the bottom scene, we see the model accurately infers the geometry of the unseen cathedral ceiling and the long hallway.
have addressed this challenge by training solely on videos of static scenes such as only indoor houses [68] manually filtering videos [34, 65]. However, such approaches limit the diversity and scale of the data. Therefore, to learn from in-the-wild videos, we propose motion masking, an approach for handling dynamic objects.
Motion masking consists of predicting a dense mask of values between 0 and which we apply to the output by the U-net fe through elementwise multiplication. This soft mask allows the model to filter out portions of the scene which may be difficult to predict due to object movement. To produce the motion mask we add an additional channel to the U-Net denoiser, which outputs dense mask with values which we clamp between and 1. During training, this mask filters dynamic elements from the loss function.
Formally, let M gax denote the dense mask generated by the decoder. The modified loss function, incorporating temporal masking, is given by:
C = |[(E ca Solz, R,1))) MI
However, directly optimizing this loss leads to degenerate solution where all elements of the scene are filtered from the loss. To address this, we introduce an auxiliary loss term that incentivizes the mask to be non-zero:
Cauxiliary -
Σ Mij
Incorporating motion masking and the auxiliary loss enables the model focus on static elements in dynamic scenes while training for novel view synthesis.
7

---

5.3 3D Reconstruction
Our model is trained to output a single image given an input image and target view, a popular approach which provides flexibility in the type of data that can be trained on, while still allowing for 3D reconstruction and multi-view generation. This flexibility is particularly crucial for training from video data, where obtaining a full collection of frames for given scene from in-the-wild videos may not be possible.
Naively, the image image paradigm has the drawback that generating multiple views does not guarantee consistency across views. To address this, we follow the approach of previous works [39, 31, 24] which employ various techniques to induce consistency across multiple generations. We adopt a trajectory- sampling approach similar to [39] where images are sampled along a smooth trajectory, though in case generations are conditioned on the previous generation, E,2 = fo R, t). Once multiple views are generated reconstruct the scene using Dust3r [52].
not restricted to simple rotations. While sampling, subsequent
6 Experiments
In this section we benchmark our model, ODIN, against existing methods for novel view synthesis and 3D reconstruction. We improve performance on standard benchmarks which consist of relatively simple scenes with minimal camera translation, all without fine-tuning on the target task. Qualitatively we find that ODIN has new capahilities in generating real-world scenes from long-range novel views.
6.1 Experimental Setup
We evaluate our model on the standard novel view synthesis (NVS) benchmarks, DTU [1], and Mip-NeRF 360 [4]. DTU consists of table-top items and Mip-NeRF 360 consists of scenes with views rotated 360% around a point. We report the standard NVS metrics, PSNR, LPIPS, and SSIM. As noted by previous literature, PSNR and SSIM are not well correlated with human evaluation so we primarily focus on LPIPS and qualitative comparison. Furthermore, to showcase the novel capabilities of our model, we evaluate our method held-out set of 160-1M constructed from one-thousand 360 videos.
For 3D reconstruction compare with Zerol-to-3 [24], MCC, SJC-I, and Point-E on Google Scanned Objects (GSO) and ZeroNVS on our held-out set of 360-1M. For 360-1M we derive the pseudo-ground truth from Dust3R model which is trained all ground truth views of the scene given by the video. We report Chamfer- Distance for 360-1M in addition to volumetric IoU for GSO. The 3D reconstructions for our model are created by generating images along trajectories then using Dust3r to reconstruct the scene.
Table 1: Comparison with other novel view synthe- sis models on the DTU benchmark which consists of single objects placed on table tops.
Table 2: Comparison of various novel view synthe- sis models on the MipNeRF 360 benchmark [39,4] As noted by previous work [39], PSNR and SSIM are unreliable metrics for novel view synthesis so focus LPIPS.
NVS
LPIPS 4
PSNR
SSIM
PixeINeRF [64]
0.535
15.55
0.537
NVS
LPIPS +
PSNR t
SSIM f
SinNeRF [59]
0.525
16.52
0.560
DietNeRF [16]
0.487
14.24
0.481
PixeINeRF [64]
0.718
16.50
0.556
NeRDi [12]
0.421
14.47
0.465
Zero-1-to-3 [24]
0.667
11.70
0.196
ZeroNVS [39]
0.380
13.55
0.469
ZeroNVS [39]
0.625
13.20
0.240
ODIN (Ours)
0.378
16.67
0.525
ODIN (Ours)
0.587
16.84
0.537
The models we benchmark against are trained on a variety of 2D and multi-view data sources. The diffusion- based methods, Zero -to-3 [24] and ZeroNVS [39] start from a StableDiffusion pretrained model. Ztro1-to-3 [24] fine-tunes on Objaverse [10], while ZeroNVS [39] fine-tunes on Co3D [34], ACID [23], and Real-Estate100 1kk [68]. When possible we evaluate the models provided by the original works. Most closely related to our work in architecture is Z [24] with the key difference being addition of motion masking for training on video.
8

---

6.2 Novel View Synthesis
We observe improved performance on DTU and Mip-NeRF 360 on the standard NVS metrics (Tables I and 2). Our improvement on DTU is relatively small which is to be expected as the dataset consists of simple objects, with black backgrounds. On Mip-NeRF 360, which consists of real-world scenes, we see significant improvement. In particular, the other methods struggle to generate reasonable images from views that differ significantly from the input view. In Figure we compare qualitatively to other recent works. We observe that 7erol-to-3 cannot generate full scenes and struggles to generate real objects as expected due its training data. ZeroNVS generates more plausible views, but still considerably worse for more complex scenes.
6.3 3D Reconstruction
We present 3D scenes reconstructed from ODIN generated images along a trajectory (Figure 4). Quantitative comparison can be found in Table 3. For Google Scanned Objects [13] our method comparable Zero-1 1-to-1 [24] and outperforms other methods. Comparable performance to Zero-1-to-3 is expected as it was designed for synthetic objects. We compare with ZeroNVS for scene reconstruction on a held-out set of 360- IM (Table Appendix). Other methods are not capable of generating scenes therefore we only benchmark against this method.
Table 3: 3D reconstruction results on Google Scanned Objects [13].
Method
MCC [54]
SJC-I [51]
Point-E [30]
Zero-1-to-3 [24]
ODIN (Ours)
Chamfer Distance |
0.1230
0.2245
0.0804
0.0717
0.0697
IoU f
0.2343
0.1332
0.2944
0.5052
0.5328
7
Limitations and Broader Impact
The framework and method we presented in this work are a promising step towards large-scale 3D> models, however there are some limitations to approach. From modeling perspective, the motion mask allows us to filter portions of scenes which have dynamic elements, however ideally we would like to learn to model the dynamic elements well. Some progress has been made on 4D NeRF models which can move the camera view in both time and space, however generalized 4D models largely unexplored.
Our work may have positive societal impact in the creation of 3D assets for AR and VR or downstream applications such as robotic navigation. From negative perspective our work could be used to create fake images or inappropriate scenes.
Conclusion
In this work, we propose scalable approach to constructing real-world multi-view data and show the merits our model, ODIN, trained on the largest multi-view dataset, 360-1M, to date. Enabled by the scale, diversity, and long-range correspondences in 360-1M, ODIN demonstrates capabilities beyond those of previous methods in generating 3D-consistent novel views of real-world scenes with free camera movement. On novel view synthesis and 3D reconstruction benchmarks ODIN outperforms existing methods without fine-tuning to the target data. While ODIN shows impressive results, we believe that there is further potential in the use of 360- 1M and 360 video for novel view synthesis as well as other domains such as video generation. For novel view synthesis an exciting next step would be model dynamics to generate 4D scenes. We will open-source source our code, models, and dataset.
Acknowledgments and Disclosure of Funding
We would like to thank Kuo-Hao Zeng for his feedback on the manuscript. We acknowledge funding from NSF IIS 1652052, IIS 1703166, DARPA 166001-19-2-4031, DARPA 195-55- and gifts from Allen Institute for Artificial Intelligence, Google and Apple. Sham Kakade acknowledges

---

funding from the Office of Naval Research under award N00014-22-1-237, This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
References
[1] H. Aanæs, R. R. Jensen, G. Vogiatzis, E. Tola, and A. B. Dahl. Large-scale data for multiple-view
stereopsis. International Journal of Computer Vision, pages 1-16, 2016. 3,
[2] S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S. M. Seitz, and R. Szeliski. Building rome in a day. Communications of the ACM, 54:105-112, 2011. 3
[3] T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021. 3
[4] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022. 3,8
[5] E. Brachmann, J. Wynn, S. Chen, T. Cavallari, Á. Monszpart, D. Turmukhambetov, and V. A. Prisacariu. Scene coordinate reconstruction: Posing of image collections via incremental learning of a relocalizer. arXiv preprint arX1v:2404.14333 2024. 2
[6] E. R. Chan, K. Nagano, M. A. Chan, A. W. Bergman, J. J. Park, A. Levy, M. Aittala, S. D. Mello, T. Karras, and G. Wetzstein, GeNVS: Generative novel view synthesis with 3D-aware diffusion models. In ICCV, 2023.
[7] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. ShapeNet: An information-rich 3D model repository. arXiv preprint arXiv:1512.0301 2015.
[8] J. Chung, S. Lee, H. Nam, J. Lee, and K. M. Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384 2023. 3
[9] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti,
J. Munro, T. Perrett, W. Price, et al. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):4125-411 4141, 2020. 3.5
[10] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: universe annotated 3D objects. arXiv preprint arXiv:2212.0805: 2022. 1, 4,
[11] M. Deitke, R. Liu, M. Wallingford, H. Ngo. O. Michel, A. Kusupati, A. Fan, C. Laforte, V. Voleti, Y. Gadre, E. VanderBilt, A. Kembhavi, C. Vondrick, G. Gkioxari, K. Ehsani, L. Schmidt, and A. Farhadi. Objaverse-XL: A universe of 10M+ 3D objects. arXiv preprint arXiv:2307.05665
2023. 1,3,4
[12] C. Deng, C. Jiang, C. R. Qi, X. Yan, Y. Zhou, L. Guibas, D. Anguelov, et al. NeRDi: Single-view NeRF synthesis with language- guided diffusion as general image priors. In CVPR, 2022. 8
[13] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh, and V. Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553-2560 IEEE, 2022. 3, 9, 14
[14] D. Fox, W. Burgard, F. Dellaert, and S. Thrun Monte carlo localization: Efficient position estimation for mobile robots. AAAI, 1999. 1
[15] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR), 2013. 3
[16] A. Jain, M. Tancik, and P. Abbeel. Putting NeRF on a diet: Semantically consistent few-shot view synthesis. In ICCV, 2021. 3,8
10

---

[17] T. Jain, C. Lennan, Z. John, and D. Tran. Imagededup. https://github. http://gittuub.cooomddel..o/ imagededup, 2019. 5
[18] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4):1-14, 2023.
[19] T. Kipf, G. Elsayed, A. Mahendran, A. Stone, S. Sabour, G. Heigold, R. Jonschkowski, A. Dosovitskiy, and K. Greff. Conditional object-centric learning from video. arXiv preprint arXiv:211112294, 2021.
[20] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv, 2017. 1
[21] C.-H. Lin, J. Gao, L. Tang. T. Takikawa, X. Zeng. X. Huang, K. Kreis, Fidler, M.-Y. Liu, and T.-Y Lin. Magic3D: High-resolution text-to-3D content creation. In CVPR, 2023. 3
[22] H. Lin. Robotic manipulation based on 3d vision: A survey. Proceedings of the 2020 International Conference on Pattern Recognition and Intelligent Systems, 2020. URL https://age http//ppp...mmatccsschhollaa. org/ http/CorppuID::21499933
[23] A. Liu, Tucker, V. Jampani, A. Makadia, N. Snavely, and Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In ICCV, 2021. 3,
[24] R. Liu, R. Wu, B. V. Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero-1-to-2 Zero-shot one image to 3D object. In CVPR, 2023. 1, 6, 8, 9,
[25] W.-C. Ma, A. J. Yang, S. Wang, R. Urtasun, and A. Torralba. Virtual correspondence: Humans as cue for extreme-view geometry. CVPR, 2022.
[26] B. Mildenhall, Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 3, 6
[27] R. Mottaghi, C. Schenck, D. Fox, and A. Farhadi. See the glass half full: Reasoning about liquid containers, their volume and content. ICCV, 2017. 1
[28] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos. Orb-slam: versatile and accurate monocular slam system. IEEE transactions on robotics, 3177)11447-116,, 2015. 3
[29] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 31 3175:114771116,, 2015. 5
[30] A. Nichol, H. Jun, P. Dhariwal, Mishkin, and M. Chen. Point-e: system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.0875/1 2022. 9
[31] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. DreamFusion: Text-to-3D using 2D diffusion.
In ICLR, 2022. 3, 8
[32] A. Raistrick, L. Lipson, Z. Ma, L. Mei, M. Wang. Y. Zuo, K. Kayan, H. Wen, B. Han, Y. Wang. et al. Infinite photorealistic worlds using procedural generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12630-12641, 2023. 3
[33] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06115, 2022. 5
[34] J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny. Common objects in 3D: Large-scale learning and evaluation of real-life 3D category reconstruction. In ICCV, 2021. 2, 7,
[35] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. arXiv, 2021. 3
[36] M. S. Sajjadi, D. Duckworth, A. Mahendran, S. Van Steenkiste, F. Pavetic, M. Lucic, L. J. Guibas, K. Greff, and T. Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems, 35:9512-9524, 2022. 1

---

[37] M. S. Sajjadi, H. Meyer, E. Pot, U. Bergmann. K. Greff, N. Radwan, S. Vora, M. Lučié, D. Duckworth, A. Dosovitskiy, et Scene representation transformer: Geometry-fre novel view synthesis through set-latent scene representations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[38] M. S. Sajjadi, A. Mahendran, T. Kipf, E. Pot, D. Duckworth, M. Lučić, and K. Greff. Rust: Latent neural scene representations from unposed imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17297-17306, 2023. 1
[39] K. Sargent, Z. Li, T. Shah, C. Herrmann, H.-X. Yu, Y. Zhang, E. R. Chan, D. Lagun, L. Fei-Fei, D. Sun, et al. Zeronvs: sero-shot 360-degree view synthesis from a single real image. arXiv preprint arXiv:2310. 17994, 2023. 3, 6,
[40] P.-E. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12716-12725, 2019. 2, 3,4
[41] J. L. Schönberger and J.-M. Frahm. Structure-frood motion revisited. In CVPR, 2016. 4,5
[42] J. L. Schönberger and J.-M. Frahm. Computer Vision and Pattern Recognition (CVPR), 2016. 2.
revisited. In Conference on
[43] J. L.. Schonberger and J.-M. Frahm. Structure- from-rroottt revisited. Proceedings of the IEEE conference computer vision and pattern recognition, pages 4104-4113, 2016. 3
[44] Y. Shi, P. Wang, J. Ye, M. Long, K. Li, and X. Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 2023.
[45] J. Shriram, A. Trevithick, L. Liu, and R. Ramamoorthi. Realmdreamer: Text-driven 3d scene generation with inpainting and depth diffusion. arXiv preprint arXiv:2404.077199 2024.
[46] Z. Teed and Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 34:16558 14:16558-16569, 2021.
[47] A. Tewari, T. Yin, G. Cazenavette, S. Rezchikov, J. Tenenbaum, F. Durand, B. Freeman, and V. Sitzmann. Diffusion with forward models: Solving stochastic inverse problems without direct supervision. Advances in Neural Information Processing Systems, 36, 2024.
[48]
Todd. The visual perception of 3d shape. Trends in cognitive sciences, 8(3):115-13) 2004.
[49] V. Tschernezki, A. Darkhalil, Zhu, D. Fouhey, Laina, D. Larlus, D. Damen, and A. Vedaldi. Epic fields: Marrying 3d geometry and video understanding. Advances in Neural Information Processing Systems, 36, 2024. 3
[50] S. Ullman. The interpretation of structure from motion. Proceedings of the Royal Society of London. Series B. Biological Sciences, 203(1153):405 426, 1979. 3
[51] H. Wang, X. Du, J. Li, R. A. Yeh, and G. Shakhnarovich. Score Jacobian chaining: Lifting pretrained 2D diffusion models for 3D generation. arXiv preprint arXiv:2212.00772 2022. 9
[52] S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud. Dust3r: Geometric 3d vision made easy. arXiv preprint arXiv:2312.141322 2023. 4,
[53] Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and Zhu. ProlifiDDreamer: High-fidelity and di- verse text-to-3D generation with variational score distillation. arXiv preprint arXiv:2305.16213. 2023.
[54] C.-Y. Wu, J. Johnson, J. Malik, C. Feichtenhofer, and G. Gkioxari. Multiview compressive coding for 3d reconstruction. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 9065-9077 2023. 2023.
12

---

[55] L. Wu, J. Y. Lee, A. Bhattad, Y.-.. Wang. and D. Forsyth. Diver: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16200-16209, 2022.
[56] R. Wu, B. Mildenhall, P. Henzler, K. Park, R. Gao, D. Watson, P. P. Srinivasan, D. Verbin, Barron, B. Poole, and A. Holynski. Reconfusion: 3d reconstruction with diffusion priors. arXiv, 2023.
[57] H. Xia, Z.-H. Lin, W.-C. Ma, and S. Wang. Video2game: Real-time, interactive, realistic and browser-compatible environment from single video, 2024.
[58] Y. Xiang, Schmidt, V. Narayanan, and D. Fox. Poseenn: convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv, 2017. 1
[59] D. Xu, Y. Jiang, P. Wang. Z. Fan, H. Shi, and 7.. Wang. SinNeRF: Training neural radiance fields on complex scenes from a single image. In ECCV, 2022.
[60] L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of scale unlabeled data. arXiv preprint arXiv:2401.10891, 2024.
[61] Z. Yang, Y. Chen, J. Wang, S. Manivasagam, W.-C. Ma, A. J. Yang, and R. Urtasun. Unisim: A neural closed-loop sensor simulator. CVPR, 2023. 1
[62] L. Yen-Chen, P. Florence, A. Zeng, J. T. Barron, Y. Du, W.-C. Ma, A. Simeonov, A. R. Garcia, and P. Isola. Mira: Mental imagery for robotic affordances, 2022.
[63] A. Yu, R. Li, M. Tancik, Li, R. Ng, and Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752-5761, 2021. 3
[64] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixeINeRF: Neural radiance fields from one or few images. In CVPR, 2021. 3,
[65] X. Yu, M. Xu, Y. Zhang, H. Liu, C. Ye, Y. Wu, Z. Yan, C. Zhu, Z. Xiong, T. Liang, al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9150-9161, 2023. 2, 5,7,
[66] X. Zhao, A. Colbur, F. Ma, M. Bautista, J. M. Susskind, and G. Schwing. gener- alized dynamic novel view synthesis from monocular videos possible today? arXiv preprint 2r..v:2310.08587, 2023.
[67] B. Zhou, Krähenbühh, and V. Koltun. Does computer vision matter for action? Science Robotics, 2019.1
[68] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Trans. Graph. (Proc. SIGGRAPH), 37, 2018. 3, 7, 8
13

---

A Dataset Statistics
12000
Pets Animals
Nonprofits & Activism
10000
News & Politics Comedy Howto Style
Category Name
Duration (s)
8000
Gaming
Science Technology
6000
Education
Autos
Vehicles
Music
4000
Sports
Film Animation
Entertainment
Travel & Events
People & Blogs
50K
100K
150K
200K
250K
300K
350k
400K
450K
102
10*
10a
106
Count
Count
Figure Video duration distribution in 360-1M.
Figure 6: Video categories' distribution in 360-1M.
de
Language
fr
pt
ru
ko
es
en
10k
40k
100K
Count
Figure 7: Video language distribution in 360-1M M.
B Correspondence Examples
C 3D Reconstruction Evaluation
Table 4: 3D reconstruction results 360-1M [13].
Comparison with Zero 1-to-3.
Method
Zero 1-to-3
Our Method
Chamfer Distance
0.1059
0.07992
IoU f
0.3178
0.5267
D Safeguards for Data Release
Upon public release of our data we will require applications to obtain the links to the videos and data-data.
E License for Data Release
YouTube videos are under fair use for research purposes and we provide only links to videos in the dataset.
14

---

Figure 8: Example of long-range range correspondence found automatically within 360-1M
Figure 9: Example of long-range correspondence found within 360-1M.
Training Details
We train ODIN, for weeks 16 A40's for 100 epochs. We used batch size of 1024 where one sample consists of a frame correspondence. We use a learning rate of le - with a constant learning rate, In general, if otherwise specified we use the default peramerrseeserss from [24]
G Ablations
In table 6 we show performance for various values of /lambda, the coefficient for motion masking detailed section 5.2. table we show ablation over sampling various frames per second for 10k videos. We find that performance increases with higher FPS, and chose reasonable balance between performance and compute cost at FPS when scaling to the larger million datasets.
15

---

Correspondence Examples from MVImageNet
Toy mouse
Ladder
Toy tiger
Ceiling lamp
Toy rabbit
Light switch
Figure 10: General example of correspondences from MVImageNet. Previously the largest multi- view dataset.
Table 5: Evaluation of LPIPS, PSNR, and SSIM at different frame rates (FPS) of sampling.
FPS
LPIPS
SSIM PSNR
0.5 FPS
0.488
15.88
0.492
FPS
0.467
16.67
0.525
5 FPS
0.461
16.85
0.539
10 FPS
0.475
16.71
0.536
Table 6: Ablation study over À values for motion masking with novel view synthesis metrics.
LPIPS
SSIM PSNR
0.1
0.498
12.31
0.366
0.5
0.467
14.73
0.402
0.378
16.67
0.525
2
0.395
14.94
0.431
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We support our claims in the abstract and introduction with thorough experi- ments and evaluation.
Guidelines:
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
16

---

Justification:
Yes we discuss limitations.
Guidelines:
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not have theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide details to reproduce the results of our paper.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We will provide the dataset. meta-data and pose estimates upon public release of the paper. At the time of submission it is difficult to host such large files. We provide the code in the supplemental.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes we provide the implementation details.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Generating error bars by sampling various initial noise for all diffusion models compare against would be computationally infeasible. ZeroNVS requires 1.5 hours to generate one scene with one A40.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Yes we provide details in the appendix.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttps:///euripee http://wwwrrr..ccccuue/eteeteeea cctppppllcc/ttticccccildl.aaaaa
Answer: [Yes]
Justification: Yes we conform to code of ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
17

---

Answer: [Yes]
Justification:
Yes we provide it.
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We describe safeguards for the responsible release of our dataset in the appendix.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes we release only links to the videos contained in our dataset under fair use for research purposes and explicitly state as such in the appendix.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes provide documentation and analysis of the video data release.
14. Crowdsourcing and Research with Human Subjects
Answer: [NA]
Justification: No user study.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No user study.
18


Reviews:
[{"review_id": "ZNMdOiMWu1", "reviewer": "Reviewer_8VWi", "summary": "This paper addresses the challenge of training models with 3D object understanding using large-scale real data, proposing the use of 360-degree videos as a scalable and diverse data source. Its main contributions include a 360-1M video dataset, an efficient method to convert 360-degree videos into multi-view data, and a novel diffusion-based model for view synthesis. The proposed model is evaluated on NVS and 3D reconstruction tasks compared to prior approaches.", "questions": "See above. Overall, I believe the proposed dataset and the demonstrated performance of the model hold promise for future research. I look forward to the rebuttal addressing my concerns.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper propose a novel large-scale real-world dataset collected from Internet along with an efficient pipeline to extract valid correspondence between frames. This dataset should be useful for future researches in the community.  \n2. The idea of utilizing 360 degree video to provide more correspondences is interesting.  \n3. Experiments demonstrate the proposed model is able to predict novel view images along a long trajectory, which is an important capability for large-scale scene/object understanding.  \n4. The paper is overall well written and easy to follow.", "weaknesses": "1. Besides its major contribution as a large-scale and open-world training dataset compared to ZeroNVS, the model primarily aligns with Zero-1-to3, which somewhat limits its technical contributions.\n\n2.  Experiments:  \n[-] Could you provide a reason why Zero-1-to-3 is not compared on the DTU benchmark, which is suitable for Zero-1-to-3 given its focus on single objects placed on tabletops?  \n[-] I wonder how Zero-1-to-3 is applied in Fig. 3, as its camera pose involves both elevation and azimuth angles, which may differ from the video’s camera poses. Could the authors clarify how they calculate the camera pose?  \n[-] This paper presents quantitative results only for experiments on DTU and MipNeRF360 (Tables 1 and 2).  Including visual comparisons could help understanding.\n[-] Why does Table 4 in the supplementary materials show a higher Chamfer Distance compared to ZeroNVS?  \n\n2. Others  \n[-] Considering the training requirements outlined in the appendix, it’s evident that training is highly resource-intensive, which is reasonable given the dataset’s scale. I wonder if there are potential methods to expedite this process, such as initializing from a pretrained Zero-1-to-3 model?    \n[-] Could the authors elaborate on how long-range image pairs contribute to the model? It’s challenging to distinguish correspondence in Fig. 9 from the appendix, as the images appear not to overlap. While such pairs may assist the model in extrapolating unseen regions, could they potentially hinder learning the viewpoint changes of observable content?   \n[-] The visualization in Fig. 3 could be enhanced by including relative changes in viewing angles, as it’s currently difficult for humans to find correspondence between the output and input view images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review. We are glad they find our model demonstrates important capabilities for large-scale scene understanding, the idea of using 360° video is interesting, and our dataset will be useful for future researchers. We provide responses to their comments and questions below and are happy to discuss further. \n\n**Besides its major contribution as a large-scale and open-world training dataset compared to ZeroNVS, the model primarily aligns with Zero-1-to3, which somewhat limits its technical contributions.**\n\nWe contend that the technical contribution of this work goes significantly beyond a new dataset. We've enumerated our contributions below.\n \n-   We present a new technique capable of finding image-pairs from in-the-wild 360° videos. Corresponding image pairs are crucial for training novel view synthesis (NVS) models and current approaches such as colmap and hloc are not capable of extracting such pairs from large-scale video.\n-   We introduce temporal masking, a method which enables training on dynamic scenes which has been a limitation of novel view synthesis works.\n- We provide a new dataset consisting of over 1 million 360° videos, multi-view image pairs from the videos with their relative camera pose, and meta data for each video. \n-   Our model demonstrates novel capabilities in generating full, real-world scenes along long-range trajectories consisting of both rotation and translation compared to Zero1-to-3 which is limited to synthetic objects and camera rotations about the object.\n\n**Could you provide a reason why Zero-1-to-3 is not compared on the DTU benchmark, which is suitable for Zero-1-to-3 given its focus on single objects placed on tabletops?**\n\nWe originally omitted Zero-1-to-3 since it is very similar to ZeroNVS and strictly worse in performance. We have added it back to Table 1 for completeness. \n\n**I wonder how Zero-1-to-3 is applied in Fig. 3, as its camera pose involves both elevation and azimuth angles, which may differ from the video’s camera poses. Could the authors clarify how they calculate the camera pose?**\n\nIt’s true that Zero1-to-3 is limited to specific relative camera poses. We fit the azimuth, elevation angles, and radius of rotation to minimize L2 distance between the Zero1-to-3 camera pose and the target pose. Due to this limitation of Zero-1-to-3, we primarily focused our comparisons on ZeroNVS as it is designed for unconstrained camera movement, real-world scenes, and has a similar architecture. \n\n**This paper presents quantitative results only for experiments on DTU and MipNeRF360 (Tables 1 and 2). Including visual comparisons could help understanding.**\n\nWe’ve included videos of our generation for better visual understanding in our one page pdf. Additionally we will add qualitative comparison for MipNeRF360 and DTU to the main paper. \n\n\n**Why does Table 4 in the supplementary materials show a higher Chamfer Distance compared to ZeroNVS?**\n\nThanks for catching the type in the appendix. That should be a comparison with Zero-1-to-3. We’ve corrected the error.\n\n**I wonder if there are potential methods to expedite this process, such as initializing from a pretrained Zero-1-to-3 model?**\n\nWe tried initializing from Zero-1-to-3 and ZeroNVS. We found that it led to slightly faster convergence, about ~10% fewer training iterations. The performance at convergence was the same when starting from both pretrained models.\n\n**Could the authors elaborate on how long-range image pairs contribute to the model?**\n\nThe long-range image pairs in the training data are crucial for generating long-range trajectories and modeling large-scale scenes with our model such as in figure 1l. Models are limited to types of camera movements in their training distribution. We observe this with zero1-to-3 being limited to only rotating the camera around objects. \n\n**It’s challenging to distinguish correspondence in Fig. 9 from the appendix, as the images appear not to overlap. While such pairs may assist the model in extrapolating unseen regions, could they potentially hinder learning the viewpoint changes of observable content?**\n\nThe trees and the steps in the background are shared by both images and Fig. 9 shows that Dust3r is capable of accurately finding the relative camera poses. Empirically we did not find that extrapolating to unseen regions decreased performance of the model for observable content.\n\n**The visualization in Fig. 3 could be enhanced by including relative changes in viewing angles, as it’s currently difficult for humans to find correspondence between the output and input view images.**\n\nThanks for the suggestion, we will add this to figure 3 to improve the visualization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lCL46xqBjX", "reviewer": "Reviewer_Jg5T", "summary": "This paper introduces a new approach for efficiently finding corresponding frames from diverse viewpoints from YouTube videos at scale. The resulting 360-1M dataset contains over eighty million frames from approximately one million videos. Additionally, the paper also builds on the 360-1M dataset by proposing the ODIN model, which synthesizes novel views and reconstructs 3D scenes from a single input image. The ODIN model is trained by leveraging the diverse viewpoints in the dataset to handle significant camera view changes and complex real-world scenes. The authors demonstrate the utility of the 360-1M dataset as well as the benefits of their proposed ODIN model by comparing it to state-of-the-art 3D reconstruction approaches, where it outperforms the latter by a significant margin.", "questions": "Please look at the above-mentioned limitations.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The model figures are informative and especially helpful in helping the reader to understand the different stages of the data curation process as well as the intuition behind each stage. The paper is also well-organized and well-written.\n\n2) The introduced algorithm to transform 360 degrees videos into the required multi-view format is especially significant. It opens up the possibility of scaling up available 3D object and scene datasets, which are often plagued by a lack of large-scale data. Furthermore, it is especially helpful that this algorithm appears to generalize well to videos from diverse subject categories.\n\n3) The ODIN model also introduces a novel modeling objective which is conditioned on both rotation and translation. This differs from prior work which is unable to do so. This may be beneficial for generating more realistic samples of real world objects and scenes for further training.", "weaknesses": "1) The paper relies heavily on the 360-degree videos collected from YouTube, which may contain varying video qualities and resolutions. The described preprocessing steps does not fully account for this diversity in video quality, which may affect the quality of the extracted multi-view data negatively.\n\n2) It is mentioned in line 220 that optimizing Equation 2 directly results in a degenerate solution and Equation 3 is introduced to mitigate this problem. However, there is no ablation to demonstrate this. It may help make the paper more comprehensive. \n\n3) In Tables 2 and 3,  it is shown that ODIN outperforms other approaches on the LPIPS and PSNR metrics but not the SSIM metric consistently. However, there is no discussion of possible reasons for this discrepancy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review. We are glad they found our algorithm for transforming 360° video into multi-view data to be especially significant and see its potential for scaling 3D datasets. We provide responses to their comments and questions below and would be happy to engage in further discussion.\n\n**The paper relies heavily on the 360-degree videos collected from YouTube, which may contain varying video qualities and resolutions. The described preprocessing steps does not fully account for this diversity in video quality, which may affect the quality of the extracted multi-view data negatively.**\n\nWe experimented with filtering techniques such as CLIP filtering, where we fine-tuned a CLIP model to classify low quality frames from high-quality frames. For the fine-tuning data we hand-labeled 10k example frames. We found that the confidence scoring used to find image-pairs was sufficient for removing videos that were low in quality such as blank frames, corrupted or blurry frames, and static videos. Intuitively, frames that are low in quality are unlikely to be paired since relative pose and depth are difficult to estimate. For resolution we filtered out videos below a resolution of 2K. We also provide the available resolutions, view count, etc. for each video in the meta-data so users can filter videos based on various criteria. \n\n**It is mentioned in line 220 that optimizing Equation 2 directly results in a degenerate solution and Equation 3 is introduced to mitigate this problem. However, there is no ablation to demonstrate this. It may help make the paper more comprehensive.**\n\nThanks for the suggestion. We’ve included an ablation below over the hyperparameter lambda in Eq. 3 and have added discussion about choosing lambda to the appendix. We found that if the lambda value was too small, then the model would mask all objects in the scene and the reconstruction quality was poor. If the lambda value was too large, then smearing would occur near dynamic objects as the model was forced to predict the motion of the object.\n\n|                    | LPIPS | PSNR  | SSIM  |\n|--------------------|-------|-------|-------|\n| $\\lambda$ = .1      | 0.498 | 12.31 | 0.366 |\n| $\\lambda$ = .5      | 0.467 | 14.73 | 0.402 |\n| $\\lambda$ = 1       | 0.378 | 16.67 | 0.525 |\n| $\\lambda$  = 2     | 0.395 | 14.94 | 0.431 |\n\n**In Tables 2 and 3, it is shown that ODIN outperforms other approaches on the LPIPS and PSNR metrics but not the SSIM metric consistently. However, there is no discussion of possible reasons for this discrepancy.**\n\nOn line 247 we briefly note that previous works [1,2,3] have shown that SSIM and PSNR are not accurately correlated with better novel view synthesis. For example, in [3] figure 7 they show that a frame with uniformly grey pixels can outperform far more reasonable generations. In general, PSNR is sensitive to the low level pixel statistics which is tangential to the content of the image. We will better clarify this point in section 6.1.\n\n1. Zero-Shot 360-Degree View Synthesis from a Single Image\n2. Generative novel view synthesis with 3d-aware diffusion models\n3. Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "efmkFecujW", "reviewer": "Reviewer_pxup", "summary": "This paper considers novel view synthesis from a single image. The main contribution is a dataset sourced from 1 million 360 videos from youtube, which is used to create about 380 million pairs of images with corresponding relative poses. With this dataset the authors train a diffusion model similar to zero-1-to-3 [24], but for general scenes instead of objects, and real-world scenes instead of synthetic ones. State-of-the-art results are shown for novel view synthesis from single images.", "questions": "- I wonder how important the use of dust3r is for the 3d reconstruction. Is the confidence measure by dust3r used, and if so, how? I would imagine that it would be useful to get a consistent 3d reconstruction even if the views generated by ODIN are not consistent. By consistent I mean both that the views might not adhere to a specific camera model well enough for traditional sfm method to work well, or that the different views contain contradicting contents. Is that why there are some holes in the reconstructions in fig. 4 (e.g. the middle part in the leftmost 3d model of the cathedral), because it looks from the positions of the cameras that they should not be empty? What would happen if colmap was used on the generated images, would it still result in a reasonable 3d model?\n- Did you notice any failure cases related to just conditioning on the previously generated image for the 3d generation? The camera motions in fig. 4 are fairly simple with little rotations and objects do not disappear and reappear, but if there would be more rotations I would imagine that it needs to be more carefully handled, e.g. as in the paper “WonderJourney: Going from Anywhere to Everywhere” (CVPR 2024).\n- For 3d reconstruction, many methods (e.g. zeroNVS) use a NeRF model and score distillation sampling instead of direct 3d reconstruction from multiple generated images like it is done in this paper. Were any experiments like that performed?", "limitations": "This is adequately addressed", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The main contribution of the paper is that it introduces a dataset for novel view synthesis that is significantly larger (380 million image pairs extracted from 1 million 360-videos) than existing datasets for the task, and considers whole scenes rather than single objects as e.g. objaverse.\n- The authors train a diffusion model (ODIN) for novel view synthesis that is more general than existing work (zero-1-to-3 and follow up works which only consider single objects), and compared to scene based methods (zeroNVS), it is more accurate and general, permitting a larger set of possible relative poses (R,t). The improvements are mainly due to training on the larger dataset since the methodology, namely the diffusion model architecture and training, is very similar to zero-1-to-3.\n- The proposed diffusion model, ODIN, along with the 3d reconstruction method dust3r can generate realistic-looking 3d reconstruction from a single image.", "weaknesses": "- There is a lack of qualitative examples. Specifically, a few things that would have been useful to show are 1) The generated images along the trajectories in fig. 4 in the form of both images and videos. Only the final 3d reconstruction by dust3r and not the intermediately generated images by ODIN are shown, 2) Images of the renderings of different methods in table 1 and 2, and 3) Generated images and the corresponding 3d reconstructions on Google Scanned Objects, so qualitative comparison corresponding to table 3 for all methods.\n- There are a few missing references and as a result too strong claims in the paper. 1) “Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models” (Yu et al. ICCV 2023), 2) “Geometry-Free View Synthesis: Transformers and no 3D Priors” (Rombach et al. ICCV 2021), and 3) “Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image” (Ren et al. CVPR 2022). These works, and also zeroNVS (which is cited) all address novel view synthesis from a single image, though trained on smaller datasets. ZeroNVS also constructs a 3d scene by training a NeRF with Score distillation sampling. Due to this I think e.g. the claim on line 48 that ODIN is the first to reasonably reconstruct a 3d scene from a single image is too strong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review. We are glad they appreciated the diversity and scale of our proposed dataset and the new capabilities of our resulting model. We provide responses to their comments and questions below and are happy to engage in further discussion. \n\n**A few things that would have been useful to show are 1) The generated images along the trajectories in fig. 4 in the form of both images and videos. Only the final 3d reconstruction by dust3r and not the intermediately generated images by ODIN are shown, 2) Images of the renderings of different methods in table 1 and 2, and 3) Generated images and the corresponding 3d reconstructions on Google Scanned Objects, so qualitative comparison corresponding to table 3 for all methods.**\n\nGreat suggestion, we’ve included generated videos in the one page pdf for better visualization and have added 3D construction examples and further image examples to the paper.\n\n**There are a few missing references and as a result too strong claims in the paper ZeroNVS also constructs a 3d scene by training a NeRF with Score distillation sampling. Due to this I think e.g. the claim on line 48 that ODIN is the first to reasonably reconstruct a 3d scene from a single image is too strong.**\n\nThanks for the suggestion. We’ve removed this sentence from the introduction and softened the language overall. Additionally we’ve added the missing references you’ve pointed us to. \n\n**I wonder how important the use of dust3r is for the 3d reconstruction. Is the confidence measure by dust3r used, and if so, how? I would imagine that it would be useful to get a consistent 3d reconstruction even if the views generated by ODIN are not consistent. By consistent I mean both that the views might not adhere to a specific camera model well enough for traditional sfm method to work well, or that the different views contain contradicting contents. Is that why there are some holes in the reconstructions in fig. 4 (e.g. the middle part in the leftmost 3d model of the cathedral), because it looks from the positions of the cameras that they should not be empty? What would happen if colmap was used on the generated images, would it still result in a reasonable 3d model?**\n\nFor the 3D reconstruction portion colmap performs qualitatively the same. You’re correct that the hole in fig. 4 is due to low confidence from the Dust3R model in that region. In general, as long as enough images are generated,  experimentally, we found that colmap can perform similarly. To summarize, Dust3r paired with the graphical search is necessary for finding corresponding frames while constructing the dataset, but at inference the model is agnostic to the method for 3D reconstruction. \n\n**Did you notice any failure cases related to just conditioning on the previously generated image for the 3d generation? The camera motions in fig. 4 are fairly simple with little rotations and objects do not disappear and reappear, but if there would be more rotations I would imagine that it needs to be more carefully handled, e.g. as in the paper “WonderJourney: Going from Anywhere to Everywhere” (CVPR 2024).**\n\nYes, as the reviewer mentioned the most common failure case is if an object comes in and out of occlusion it can be inconsistently generated by the model. With long-range camera trajectories more parts of the scene will come in and out of view which can lead to inconsistent generations, a typical failure mode of conditional NVS models. We found that the SDS anchoring introduced by ZeroNVS was effective in improving this consistency.\n\n**For 3d reconstruction, many methods (e.g. zeroNVS) use a NeRF model and score distillation sampling instead of direct 3d reconstruction from multiple generated images like it is done in this paper. Were any experiments like that performed?**\n\nIn general the dataset and model are agnostic to the downstream 3D reconstruction method. \nWe originally chose direct 3D reconstruction so that the geometry of the scene could be inferred in real time and we found it better for large-scale scenes and inferring geometry compared to distilling into a NeRF model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "10P9Zu1u7j", "reviewer": "Reviewer_whPn", "summary": "The paper introduces a large-scale, real-world, multiview, 360-degree outward dataset designed for static novel view synthesis and 3D reconstruction. To capture these data, the study develops a pipeline capable of identifying corresponding multiview frames from 360-degree outward videos with a fixed camera trajectory. To demonstrate the dataset's contribution and impact, the work proposes a diffusion-based model that achieves state-of-the-art performance using this dataset.", "questions": "- I hope the author can verify that the viewpoints are diverse enough by discussing the difference between a multiview dataset that can capture images from different locations at the same timestamp and a dataset that can only change the camera orientation at the same timestamp.\n\n- The dataset contains a large amount of data, which may vary in quality. I hope the author can discuss the impact of the low-quality data in the dataset. Is it negligible?\n\n- Iines 178-179 mentions that it is infeasible to guarantee the uniqueness of video content. Hence, how do you make sure there is no overlap between training and testing set? How do you make sure the training dataset does not contain the testing cases in other dataset? Will it be unfair for the benchmark?\n\n- The dataset contains diverse data. How is the data distribution in terms of motion? Will high-motion data, such as scenes with rain, have a negative impact? \n\n- Lines 126-128 need experimental support. Please consider trying it on a subset of the dataset.\n\n- The work uses Depth Anything as the depth estimator. This is a monocular depth estimator with two types of checkpoints pretrained on two different datasets. I wonder if this causes a domain gap in the proposed dataset when using different checkpoints. Do you finetune the estimator on your dataset? Additionally, it is unknown how good the multiview consistency is on your data with this estimator.\n\n- What is the tradeoff between using this dataset and others? From the experiment section, we know that the dataset can increase performance. But how about the training time compared to existing datasets? Maybe the author can consider adding the training time from the appendix to the table.\n\n- How sensitive and stable is the motion masking hyperparameter in Eq. 3? How should it be tuned?\n\n- What is the performance when the input image contains many dynamic objects?", "limitations": "I agree with the limitation section in the paper.\n- One potential limitation of the large-scale dataset is to maintain the data quality and high fps.\n- Another potential limitation is to make sure there is no overlap between training and testing set.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength\n- The work introduces a real-world, multiview, 360-degree outward dataset that is significantly larger and more diverse than existing datasets for static novel view synthesis. This dataset is crucial for modern data-driven methods.\n\n- The work presents a novel and efficient pipeline that utilizes Dust3R to find corresponding frames and graphs to maintain long-range correspondence for data collection. This innovative pipeline makes the data collection process scalable.\n\n- The work proposes a diffusion-based model, ODIN, that leverages the video dataset for static novel view synthesis by incorporating motion masking and viewpoint conditions to achieve state-of-the-art performance. Additionally, the model can generate long-range novel views, further validating the dataset's positive impact.", "weaknesses": "Dateset\n- The dataset is collected from 360 YouTube videos, captured beforehand by the camera operator. The camera trajectory (x, y, z) is controlled by the camera operator, thereby limiting the flexibility to select diverse viewpoints from different positions.\n\n- The quality of the data is unknown and can be unstable across different data points because the author cannot investigate each video one by one or frame by frame. This represents a tradeoff between capturing one's own data and crawling data from the Internet. For instance, the author acknowledges in lines 178-179 that it is infeasible to guarantee the uniqueness of video content.\n\n- Although the work claims that 1 fps is sufficient in lines 126-128, this claim lacks experimental support. The author should consider trying different fps rates and show the performance gap and computational time between them.\n\n- Larger dataset may need more training time and computational resource.\n\nMethod\n- One of the most challenging aspects that prevent people from using video for static scene tasks is the impact of motion over time (multiview inconsistency). The work proposes a motion mask method to mitigate this impact. However, this aspect lacks analysis and discussion. For instance, the importance of the hyperparameter in Eq. 3 is unknown.\n\n- In Fig. 3, in the second row, the method is shown to synthesize humans as well. However, I wonder whether the motion masking has a negative impact on synthesizing dynamic objects when they are static.\n\nWriting\n- In lines 5-6, the claim may be too arbitrary. As mentioned in lines 87-90, there are some large-scale real-world datasets, such as CO3D, albeit with limited diversity. A related work, OmniObject3D, should also be mentioned.\n\n- In lines 8-12, the phrase \"...diverse views\" may be too broad and should be softened. The proposed dataset is collected from pre-captured 360 YouTube videos. The camera trajectory is decided by the camera operator beforehand, and only the camera orientation can be controlled after that. Hence, compared to general multiview datasets, which can have multiple viewpoints from different locations at the same timestamp, the views of the proposed dataset are limited.\n\n- Eq. 2 does not match the equation in Sec. 5.1. The equation in Sec. 5.1 contains an undefined term.\n\n- I hope the author can define their 360 multiview videos at the beginning since there are several kinds of 360 videos. For instance, DiVa360 captures 360-degree inward videos with multiple cameras at different locations at the same timestamp.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review. We are happy that they find our dataset crucial for modern data-driven methods and find our correspondence search pipeline to be innovative. We have addressed comments and questions below and are happy to engage in further discussion. Note that we could not address all points within the character limit and will address the remaining questions during the discussion phase in the official comments. \n\n**The camera trajectory (x, y, z) is controlled by the camera operator, thereby limiting the flexibility to select diverse viewpoints from different positions.**\n\nWe found that the increased scale and diversity of camera views in the 360-1M dataset compensates for this limitation as shown qualitatively and quantitatively in our results. The 1 million in-the-wild videos provides camera movements with extremely diverse trajectories, for example people sky-diving, climbing mountains, and walking many blocks through cities. Though each video has a fixed (x,y,z), the diversity of viewpoints across the data gives our model flexibility in generating from different viewpoints compared to models trained on previous datasets such as ObjaVerse and Co3D. \n\n**The quality of the data is unknown and can be unstable across different data points. For instance, the author acknowledges that it is infeasible to guarantee the uniqueness of video content.**\n\nWhile it’s true that we do not crawl every frame, we contend that the quality of the model is a strong indicator of the data quality. In our case we show that the scale and the quality of the 360-1M leads to improved model performance compared to previous datasets. \n\n**Although the work claims that 1 fps is sufficient in lines 126-128, this claim lacks experimental support.**\n\nWe initially ran small-scale proxy experiments at various FPS before scaling to the full dataset. Performing such experiments at 1 million scale was computationally infeasible. We found that below 1 FPS there was no performance loss. We’ve also included the computational cost difference. Below we’ve included the results of our experiments on DTU and have added this to the appendix. For this experiment we kept the window size for correspondence searching fixed and used 50k videos from 360-1M. \n\n|          | LPIPS | PSNR  | SSIM |\n|----------|-------|-------|------|\n| 0.5 FPS  | 0.488 | 15.88 | 0.492|\n| 1 FPS    | 0.467 | 16.67 | 0.525|\n| 5 FPS    | 0.461 | 16.85 | 0.539|\n| 10 FPS   | 0.475 | 16.71 | 0.536|\n\n**Larger dataset may need more training time and computational resource.**\n\nWe agree that considering computational cost is important. The task of finding corresponding pairs in video and estimating their relative pose was the main computational cost for us. By providing the image pairs and relative pose, other researchers can benefit from large-scale multi-view datasets without the computational burden of parsing a million videos.\n\n**One of the most challenging aspects is the impact of motion over time. This aspect lacks analysis. For instance, the importance of the hyperparameter in Eq. 3 is unknown.**\n\nWe’ve included an ablation below over the hyperparameter lambda in Eq. 3 and have added discussion about choosing lambda to the appendix. We found that if the lambda value was too small, then the model would mask all objects in the scene and the reconstruction quality was poor. If the lambda value was too large, then smearing would occur near dynamic objects as the model was forced to predict the motion of the object.\n\n|                    | LPIPS | PSNR  | SSIM  |\n|--------------------|-------|-------|-------|\n| 0.1 = $\\lambda$      | 0.498 | 12.31 | 0.366 |\n| 0.5 = $\\lambda$     | 0.467 | 14.73 | 0.402 |\n| 1 = $\\lambda$        | 0.378 | 16.67 | 0.525 |\n| 2 = $\\lambda$        | 0.395 | 14.94 | 0.431 |\n\n**I wonder whether the motion masking has a negative impact on synthesizing dynamic objects when they are static.**\n\nQualitatively we found that motion masking helped in reconstructing dynamic objects even when they are static such as parked cars. We hypothesize that due to the scale of the dataset, objects which can be both dynamic and static are seen multiple times in both settings. \n\n**As mentioned, there are some large-scale real-world datasets, such as CO3D, albeit with limited diversity. A related work, OmniObject3D, should be mentioned.**\n\nThanks, we have added OmniObject3D to the related works. We’ve changed line 5-6 to be more concrete. Co3D is impressive and relatively large-scale, but orders of magnitude smaller than our proposed dataset (20 thousand vs 1 million videos).\n\n**In lines 8-12, the phrase \"...diverse views\" may be too broad.**\n\nOften datasets such as DTU, MVImageNet and Co3D have images taken from positions close in space around an object (1-5 meters apart). By diverse views we meant varying differences in camera pose (up to 50 meters apart and 360 degree rotation) and from various locations, not only around objects or landmarks. We will make it more precise what we mean by diverse viewpoints in the abstract and throughout the paper. \n\n**I hope the author can discuss the difference between a multiview dataset that can capture images from different locations at the same timestamp and one that can only change the camera orientation.**\n\nOne limitation of a multi-view video dataset is that camera poses at different points in the (x,y,z) trajectory will have different timestamps. Learning novel view synthesis from dynamic scenes is still an open problem which temporal masking addresses to an extent, but there is still significant progress to be made. Current novel view synthesis (ZeroNVS, MegaScene) works seem capable of training on datasets where time and spatial location change simultaneously. Datasets which capture images from different locations at the same timestamp are ideal, but we believe this would be extremely difficult to manually collect at scale, and harnessing existing video data is an appealing alternative.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful reviews and feedback. We are glad that the reviewers found our dataset to be crucial for modern data-driven methods [whPn] and our proposed pipeline for constructing multi-view data from 360° video to be innovative [whPn], interesting [8VWi] and especially significant for scaling 3D datasets [Jg5T]. Additionally we appreciate that the reviewers found our model is important for large-scale scene understanding [8VWi], introduces a novel modeling objective [ Jg5T], and achieves state-of-the-art for novel view synthesis [whPn]. 


**In the one page pdf we’ve included generated videos from our model to better visualize the outputs of ODIN. We recommend that the pdf be downloaded and viewed with adobe acrobat for the best viewing experience.**


Below we’ve addressed individual questions and comments and are happy to engage in further discussion with reviewers.

Meta-review: The paper proposes a large-scale dataset of internet images and a method to extract correspondences between frames. After the rebuttal, all reviewers unanimously recommend acceptance. After reading the paper, the reviews and the rebuttal, the AC agrees that the paper should be accepted. The dataset, model, and method will be highly valuable to the community.

During the rebuttal, the authors have promised several changes and additions to the paper (e.g. include Zero-1-to-3, $\lambda$ analysis, discussion in sec 6.1, changes in the text, limitations, etc.) The AC strongly suggests that all these changes will be included in the final version of the paper.
Decision: Accept (poster)