From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos

Authors: matthew  wallingford
anand  bhattad
aditya  kusupati
vivek  ramanujan
matt  deitke
⋄  sham
kakade ∆ aniruddha
kembhavi  ‡⋄
roozbeh  mottaghi
wei-chiu  ma
ali  farhadi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-03 20:03:52

Abstract: Three-dimensional (3D) understanding of objects and scenes play a key role in humans' ability to interact with the world and has been an active area of research in computer vision, graphics, and robotics. Large scale synthetic and object-centric 3D datasets have shown to be effective in training models that have 3D understanding of objects. However, applying a similar approach to real-world objects and scenes is difficult due to a lack of large-scale data. Videos are a potential source for real-world 3D data, but finding diverse yet corresponding views of the same content have shown to be difficult at scale. Furthermore, standard videos come with fixed viewpoints, determined at the time of capture. This restricts the ability to access scenes from a variety of more diverse and potentially useful perspectives. We argue that large scale ODIN videos can address these limitations to provide scalable corresponding frames from diverse views.  In this paper we introduce 360-1M, a 360° video dataset consisting of 1 million videos, and a process for efficiently finding corresponding frames from diverse viewpoints at scale. We train our diffusion-based model, ODIN, on 360-1M. Empowered by the largest real-world, multi-view dataset to date, ODIN is able to freely generate novel views of real-world scenes. Unlike previous methods, ODIN can move the camera through the environment, enabling the model to infer the geometry and layout of the scene. Additionally, we show improved performance on standard novel view synthesis and 3D reconstruction benchmarks.

Main paper: 
 introduction: 
 Humans have the ability to understand and reason about the 3D geometry of the world, which is key for everyday tasks such as navigation and object manipulation [14, 67, 62, 48] . In machine learning, 3D perception and reasoning has been a long-standing goal for researchers with broad applications in robotics [58, 20, 61] , vision [27, 25] , and graphics [22, 57] . Fueled by large-scale datasets of synthetic objects [10, 11] , recent generative models have shown impressive understanding of 3D objects [24, 44, 31] . While these models' ability to generate synthetic objects is impressive, enabling 3D generative models for real world scenes and objects remains an open challenge.One intuitive source for scalable data has been video as it implicitly contains rich information about the 3D world. However, learning 3D modeling from video has been elusive despite impressive effort [66, 36, 19, 37, 38] . The key problem has been how to consistently transform video into a form amenable to learning about the 3D world. Existing 3D models learn from multi-view data, a Starting Input View Figure 1 : By learning from the largest real-world, multi-view dataset to date, our model ODIN, can synthesize novel views of rich scenes from a single input image with free camera movement throughout the scene. We can then reconstruct the 3D scene geometry from these geometrically consistent generations.collection of images of scenes or objects and their respective camera pose. Creating such multi-view datasets from video requires finding sets of corresponding frames that capture similar parts of the scene but from different locations (Figure 2 ).This search for corresponding frames in video has proven difficult at scale for a few reasons. First, correspondences are sparsely distributed throughout the video because the trajectory of the camera is fixed at the time of capture. Ideally, the camera operator would focus on a specific object or portion of the scene while moving around it. However, in-the-wild videos are far from this ideal. For example, if a person records themselves walking in the park, it is rare that they consistently focus the camera on the same object such as a bench as they walk towards, past, and away from it. Second, the computational cost of checking whether frames form a correspondence is expensive [42, 40, 5] , therefore searching extensively is infeasible. Given these limitations, the largest real-world multi-view datasets to date [34, 65] utilize Amazon Mechanical Turkers to manually record video clips of objects, and are limited to 50 and 238 object categories respectively.To address these limitations, we collect one million 360 • videos from YouTube, introduce a process to efficiently transform 360 • video into multi-view data, and train a diffusion-based novel-view synthesis (NVS) model on the dataset. Our model named ODIN, is the first to reasonably synthesize real-world 3D scenes and reconstruct their geometry conditioned on a single image. Quantitatively we evaluate our method on standard novel view synthesis benchmarks (DTU and MipNeRF360) and find improved performance compared to existing models without fine-tuning our own. Additionally, we compare ODIN to existing methods for 3D reconstruction on Google Scanned Objects as well as a held-out set of 360-1M and show significantly improved performance, especially on complex real-world scenes. We will open-source our model and dataset. 
related work: 
 Novel View Synthesis. NeRF [26] optimizes a volumetric scene function using sparse 2D images, representing the scene as a continuous 5D function. MipNeRF [3] extends NeRF with a multi-scale representation to enhance detail and reduce aliasing. Plenoctree [63] combines NeRF principles with an octree structure for efficient rendering. DIVeR [55] proposes a deterministic volumetric rendering for NeRF. Gaussian Splatting [18] uses Gaussian functions and splatting techniques for detailed scene representation and rendering. Unlike these methods which rely on densely sampled multi-view images and known camera poses, our approach captures extensive real-world scenes from widely varying camera views. PixelNeRF [64] and DietNeRF [16] extend NeRF to handle sparse input views but only for controlled settings.Recent works leverage powerful generative (diffusion) models [35] for novel view synthesis of objects [31, 21, 53, 6] , and more recently for scenes [6, 39, 8] . ZeroNVS uses a 3D-aware diffusion model with novel camera conditioning to generate 360-degree views from a single image, focusing on depth-scale ambiguity and background diversity with synthetic and real-world datasets. Diffusion with Forward Models [47] integrates a forward model into the diffusion process for unsupervised training on partial observations, solving inverse problems like view synthesis without direct signal supervision. ReconFusion [56] combines NeRFs with diffusion priors to enhance 3D reconstruction from limited views, improving geometry and texture plausibility with real and synthetic multiview datasets. LucidDreamer [8] and RealmDreamer [45] use a multi-step pipeline involving point cloud guidance and Gaussian splats to generate detailed 3D scenes from text or image prompts but lacks physical realism and has limited control over viewpoint changes. In contrast, our method leverages a large-scale collection of 360-degree YouTube videos to train a diffusion-based model, enabling the synthesis of diverse real-world 3D scenes and reconstruction from a single image, thus accommodating significant camera view changes and a broader range of scenarios.Camera Pose Estimation and Structure from Motion. Estimating camera pose and structurefrom-motion (SfM) have a rich history in computer vision [2, 43, 50] . Camera pose estimation consists of estimating the 6 degrees of freedom of cameras from which images were taken and the camera intrinsics. The process typically involves finding corresponding key-points between multiple images of a scene, and using their apparent motion within the images to infer the 3D geometry of the scene and relative location of each camera. For multi-view datasets and novel view synthesis, works typically use COLMAP [42, 40] or a SLAM variant [46, 28] . We choose to use the recent method Dust3R [52] as it is computationally faster and allows for as few as 2 images whereas most SfM methods require dozens. This enables us to scan much more quickly through videos for frame correspondences and create the large-scale dataset from 360 • video.Multi-View Datasets. Existing multi-view datasets such as MVImageNet [65] , CO3D [34] , RealEstate10K [68] , ACID [23] , Epic-Kitchens [9] , MipNeRF-360 [4] , and Epic-Fields [49] provide valuable multi-view sequences of real-world scenes and objects but are often constrained by the specific environments or objects they capture. MVImageNet is the largest multi-view dataset to date with over 200,000 video clips captured of 238 object categories. Though this effort is impressive, using Mechanical Turkers to manually capture videos of objects is difficult to scale further and limits content diversity. In Figure 10 we show examples of correspondences within MVImageNet which can be compared to correspondences of 360-1M in Figure 8 . Large-scale 3D object datasets like Objaverse [10] , Objaverse-XL [11] , and infinigen [32] focus on detailed 3D object assets to generate synthetic objects and scenes. Autonomous driving and 3D reconstruction datasets such as Kitti [15] , DTU [1] , ShapeNet [7] , and Google Scanned Objects [13] offer multi-view data for specific tasks like driving scenarios, 3D modeling, and object classification.In contrast, our dataset leverages a large-scale collection of 360-degree YouTube videos, providing a vastly more diverse and extensive source of real-world data. Our dataset accommodates significant camera view changes and broader real-world applications, going beyond the constraints of controlled multi-view datasets and specific domain focuses.  
multi-view data from 360 • video: 
 There are two key elements missing from current multi-view datasets: scale and real-world data. Various datasets and works have managed [34, 65, 10, 11] to make progress along these dimensions individually, however, no current datasets afford both aspects.The key challenge in collecting large-scale, multi-view datasets derives from the difficulty of finding high-quality frame correspondences, and estimating their relative poses. Existing structure-frommotion algorithms, such as COLMAP [41] and HLOC [40] , are slow and require many images of the same scene. In this section we detail our process for efficiently transforming 360 • video into high-quality multi-view data.There are two properties of corresponding video frames that are necessary for training novel view synthesis models: sufficiently differing viewpoints and overlapping content. In manually collected novel view synthesis (NVS) datasets this is accomplished by taking a video while circling the object. Finding frames that fit these criteria from in-the-wild video is much more difficult.A major reason is that high-quality correspondences are sparsely distributed in standard videos. For example, someone taking a video while walking down the street often keeps their camera view facing their direction of travel. So while they may capture a parked car on the side of the road while walking towards it, they likely will not pan their camera to capture it from many angles while walking past or away from it. Therefore, it is difficult to obtain paired images of the scenes or objects from distant locations and diverse views at scale. One solution to this problem is to leverage 360 • videos. The 360 • nature allows the views of frames to be rotated such that they contain overlapping content. Therefore given two frames that are close enough in spatial location, in theory we can align the views to look at the various regions of the scene to form multiple view correspondences. Now we describe how we operationalize this approach. We begin by sub-sampling frames of the 360 • video at r = 1 frame-per-second. We find empirically this to be a sufficiently fast frame rate given the movement speed of the camera. The computation of the correspondence search scales with r 2 , therefore we judicially select the frame rate. Next we perform pairwise comparison between frames within a frame window of length, L = 20. We map the 360 panoramic frames using an equirectangular projection E(I, θ, ϕ) where θ is the pitch, ϕ is the yaw, and I is the image. We map the panoramic image to four different views E(I, j * π/2, 0) for j ∈ {1, . . . , 4}. Thus a panoramic frame, F t at time t produces four frames {F t,0 , F t,π/2 , . . . , F t,3 * π/2 }. We then pass all pairs within the time window to the Dust3r model [52] which outputs relative pose estimate, P and confidence map, C. We take the mean confidence over the spatial components of the confidence map with height h and width w, µ c = 1 hw c∈C c, and filter out frames below threshold, τ = 4. A higher mean confidence means that the frames must be overlapping as the model can accurately estimate the pose.Once the correspondences have been found we refine the relative pose between them by performing gradient descent on the pitch and yaw of both equirectangular projections with respect to µ c . Intuitively, we can think of this as rotating the cameras to maximize the overlap (Figure 2 ). After all correspondences have been found, we discard pairs with relative translation less than .25 m because they provide minimal information for training the model.Computing relative pose between frames, especially for video, has been computationally prohibitive and a major bottle-neck for large-scale multi-view datasets [9, 34, 65] . An exhaustive search between all frames of a video would incur a cost of s 2 r 2 where s is the number of seconds, and r is the frame rate. A common approach is to limit the search with a window of size L to reduce the cost to L 2 , however this limits the pairs to short-range correspondences.We propose a hybrid approach that enables finding long-range correspondences with limited additional compute. After the initial frames have been found as detailed in section 3.1, we create a graph in which the nodes are frames and an edge exists if two frames have correspondence. We then perform the same procedure outlined in section 3.1 for all connected frames in each sub-graph. Intuitively, if two frames share a corresponding third frame (connected in the graph)then the two are also likely to share a correspondence. This approach allows us to maintain a small search window, while still finding long-range correspondences. We show examples of such long-range correspondences in Figures 8 and 9 .Dust3R and other structure-from-motion methods [29, 41, 33] output relative camera poses in dimensionless quantities, therefore we need to calibrate them to a universal scale. We do so by fusing the depth map estimates, D, from an off-the-shelf depth estimator [60] , with the point map, X ∈ R h×w×3 , predicted by Dust3R. A pointmap is a correspondence between each pixel (i, j) and the point in 3D where the ray from pixel (i, j) intersects the scene. We anchor the dimensionless pointmap to the depth math D by optimizing for a scale factor, σ, in the following equation:EQUATIONwhere z ij is the depth component of X ij and C ij is the confidence map output by Dust3R. C ij is close to 0 for points which the model has high uncertainty and acts as a filter for points with poor estimates. We choose L1 distance to limit the effect of outliers. Once we recover this scale factor, we multiply the translation of the estimated camera pose (R, t) by σ to obtain the metric pose estimate. 
dataset collection and statistics: 
 To leverage the proposed scalable correspondence search (Section 3) for generating a large-scale multi-view dataset, we collect the largest 360 • video dataset to date, 360-1M, consisting of over 1 million 360 • videos. In this section, we describe the collection process and statistics for the dataset.We collect all meta-data from YouTube in order to filter 360 • videos. The meta-data provides information on duration, view count, format, and subject category among other fields. We filter for the equirectangular format which indicates 360 • video and results in 1,076,592 total videos. We then download the videos in the equirectangular format at the best quality available. We will release the meta-data for the 360 • videos alongside the dataset.We filter the downloaded videos for empty, and duplicate videos. We remove duplicated videos with a deduplication model [17] run on the thumbnails of the videos. This does not guarantee the contents of the video are unique, however running over all frames is computationally infeasible. In the middle scene of the kitchen, ODIN accurately models the geometry of the table counter and chairs as well as unseen parts of the scene such as the living room.360-1M consists of 80,567,325 unique frames extracted from 1,076,592 videos with an average of 74.83 unique frames per video. The average video length is 6.3 minutes and is distributed in a long-tail fashion (Figure 5 ). When searching for correspondences, we sample the videos at 1 FPS. The videos are distributed evenly across 15 subject categories, with the most popular category being Travel and Events (149,534 videos) and the least popular being Pets and Animals (8802 videos) (Figure 6 ). We find 363,417,730 total frame correspondences along with their relative camera poses. 
method: 
 Our final goal is to generate images along a viewpoint trajectory conditioned on a single image of a scene -a task known as novel view synthesis (NVS). Note that our task differs from tradition novel view synthesis work such as [26] which aim to generate novel views after training on many images of a single scene. Similar to prior works [39, 24] , we leverage a diffusion-based model. This class of models have shown impressive capabilities in learning priors from large-scale data. An alternative is a NeRF based approach which is mainly effective in small-scale settings.Given a single image, x ∈ R h×w×3 , of a scene, our objective is to generate a sequence of images, xi from different viewpoints, (R, t) where R is the relative rotation and t is the translation between views. Following [24] , we use a latent diffusion architecture which consists of an encoder E, a denoiser U-net f θ , and decoder D. The standard diffusion training objective is:min θ E z∼E(x),t,ϵ∼N (0,1) ∥(ϵ -f θ (z t , t, f (x, R, t)))∥ 2 2Our modeling objective differs from previous work in that we condition on both rotation, R, and translation t. The long-range correspondences in our training data afford much freer camera movement throughout the scene compared to previous works. Due to the limitations of previous training data, other methods can only rotate about a center point of the object or scene.Learning how to perform novel view synthesis from videos poses a challenge as it assumes the scene itself does not vary with time when generating images from novel viewpoints. Previous approaches have addressed this challenge by training solely on videos of static scenes such as only indoor houses [68] or manually filtering videos [34, 65] . However, such approaches limit the diversity and scale of the data. Therefore, to learn from in-the-wild videos, we propose motion masking, an approach for handling dynamic objects.Motion masking consists of predicting a dense mask of values between 0 and 1, which we apply to the output by the U-net f θ through elementwise multiplication. This soft mask allows the model to filter out portions of the scene which may be difficult to predict due to object movement. To produce the motion mask we add an additional channel to the U-Net denoiser, which outputs a dense mask with values which we clamp between 0 and 1. During training, this mask filters dynamic elements from the loss function.Formally, let M ∈ R h×w denote the dense mask generated by the decoder. The modified loss function, incorporating temporal masking, is given by:L = ∥(ϵ -ϵ θ (z t , t, f θ (x, R, t))) • M ∥ 2 2(2) However, directly optimizing this loss leads to a degenerate solution where all elements of the scene are filtered from the loss. To address this, we introduce an auxiliary loss term that incentivizes the mask to be non-zero:EQUATIONIncorporating motion masking and the auxiliary loss enables the model to focus on static elements in dynamic scenes while training for novel view synthesis.Our model is trained to output a single image given an input image and target view, a popular approach which provides flexibility in the type of data that can be trained on, while still allowing for 3D reconstruction and multi-view generation. This flexibility is particularly crucial for training from video data, where obtaining a full collection of frames for a given scene from in-the-wild videos may not be possible.Naively, the image to image paradigm has the drawback that generating multiple views does not guarantee consistency across views. To address this, we follow the approach of previous works [39, 31, 24] which employ various techniques to induce consistency across multiple generations. We adopt a trajectory-based sampling approach similar to [39] where images are sampled along a smooth trajectory, though in our case we are not restricted to simple rotations. While sampling, subsequent generations are conditioned on the previous generation, ϵ i,t = f θ (x i , R, t). Once multiple views are generated we reconstruct the scene using Dust3r [52] . 
experiments: 
 In this section we benchmark our model, ODIN, against existing methods for novel view synthesis and 3D reconstruction. We improve performance on standard benchmarks which consist of relatively simple scenes with minimal camera translation, all without fine-tuning on the target task. Qualitatively we find that ODIN has new capabilities in generating real-world scenes from long-range novel views.We evaluate our model on the standard novel view synthesis (NVS) benchmarks, DTU [1] , and Mip-NeRF 360 [4] . DTU consists of table-top items and Mip-NeRF 360 consists of scenes with views rotated 360 • around a point. We report the standard NVS metrics, PSNR, LPIPS, and SSIM.As noted by previous literature, PSNR and SSIM are not well correlated with human evaluation so we primarily focus on LPIPS and qualitative comparison. Furthermore, to showcase the novel capabilities of our model, we evaluate our method on a held-out set of 360-1M constructed from one-thousand 360 • videos.For 3D reconstruction we compare with Zero1-to-3 [24] , MCC, SJC-I, and Point-E on Google Scanned Objects (GSO) and ZeroNVS on our held-out set of 360-1M. For 360-1M we derive the pseudo-ground truth from a Dust3R model which is trained on all ground truth views of the scene given by the video. We report Chamfer-Distance for 360-1M in addition to volumetric IoU for GSO. The 3D reconstructions for our model are created by generating images along trajectories then using Dust3r to reconstruct the scene. The models we benchmark against are trained on a variety of 2D and multi-view data sources. The diffusion-based methods, Zero1-to-3 [24] and ZeroNVS [39] start from a StableDiffusion pretrained model. Zero1-to-3 [24] fine-tunes on Objaverse [10] , while ZeroNVS [39] fine-tunes on Co3D [34] , ACID [23] , and Real-Estate10k [68] . When possible we evaluate the models provided by the original works. Most closely related to our work in architecture is Zero1-to-3 [24] with the key difference being our addition of motion masking for training on video.We observe improved performance on DTU and Mip-NeRF 360 on the standard NVS metrics (Tables 1 and 2 ). Our improvement on DTU is relatively small which is to be expected as the dataset consists of simple objects, with black backgrounds. On Mip-NeRF 360, which consists of real-world scenes, we see significant improvement. In particular, the other methods struggle to generate reasonable images from views that differ significantly from the input view. In Figure 3 we compare qualitatively to other recent works. We observe that Zero1-to-3 cannot generate full scenes and struggles to generate real objects as expected due its training data. ZeroNVS generates more plausible views, but is still considerably worse for more complex scenes.We present 3D scenes reconstructed from ODIN generated images along a trajectory (Figure 4 ). Quantitative comparison can be found in Table 3 . For Google Scanned Objects [13] our method is comparable to Zero-1-to-3 [24] and outperforms other methods. Comparable performance to Zero-1-to-3 is expected as it was designed for synthetic objects. We compare with ZeroNVS for scene reconstruction on a held-out set of 360-1M (Table 4 in Appendix). Other methods are not capable of generating scenes therefore we only benchmark against this method. The framework and method we presented in this work are a promising step towards large-scale 3D models, however there are some limitations to our approach. From a modeling perspective, the motion mask allows us to filter portions of scenes which have dynamic elements, however ideally we would like to learn to model the dynamic elements as well. Some progress has been made on 4D NeRF models which can move the camera view in both time and space, however generalized 4D models are largely unexplored.Our work may have positive societal impact in the creation of 3D assets for AR and VR or downstream applications such as robotic navigation. From a negative perspective our work could be used to create fake images or inappropriate scenes. 
conclusion: 
 In this work, we propose a scalable approach to constructing real-world multi-view data and show the merits of our model, ODIN, trained on the largest multi-view dataset, 360-1M, to date. Enabled by the scale, diversity, and long-range correspondences in 360-1M, ODIN demonstrates capabilities beyond those of previous methods in generating 3D-consistent novel views of real-world scenes with free camera movement. On novel view synthesis and 3D reconstruction benchmarks ODIN outperforms existing methods without fine-tuning to the target data. While ODIN shows impressive results, we believe that there is further potential in the use of 360-1M and 360 • video for novel view synthesis as well as other domains such as video generation. For novel view synthesis an exciting next step would be to model dynamics to generate 4D scenes. We will open-source our code, models, and dataset.A Dataset Statistics  
 
 Appendix: Upon public release of our data we will require applications to obtain the links to the videos and meta-data.YouTube videos are under fair use for research purposes and we provide only links to videos in the dataset. We train ODIN, for 2 weeks on 16 A40's for 100 epochs. We used a batch size of 1024 where one sample consists of a frame correspondence. We use a learning rate of 1e -4 with a constant learning rate. In general, if not otherwise specified we use the default hyper-parameters from [24] G AblationsIn table 6 we show performance for various values of /lambda, the coefficient for motion masking detailed in section 5.2. In table 5 we show ablation over sampling various frames per second for 10k videos. We find that performance increases with higher FPS, and chose a reasonable balance between performance and compute cost at 1 FPS when scaling to the larger 1 million datasets.In Norse mythology, Odin uses his ravens, Huginn and Muninn, as his eyes to fly throughout the world and relay what they see. 


Decision: The paper proposes a large-scale dataset of internet images and a method to extract correspondences between frames. After the rebuttal, all reviewers unanimously recommend acceptance. After reading the paper, the reviews and the rebuttal, the AC agrees that the paper should be accepted. The dataset, model, and method will be highly valuable to the community.

During the rebuttal, the authors have promised several changes and additions to the paper (e.g. include Zero-1-to-3, $\lambda$ analysis, discussion in sec 6.1, changes in the text, limitations, etc.) The AC strongly suggests that all these changes will be included in the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper addresses the challenge of training models with 3D object understanding using large-scale real data, proposing the use of 360-degree videos as a scalable and diverse data source. Its main contributions include a 360-1M video dataset, an efficient method to convert 360-degree videos into multi-view data, and a novel diffusion-based model for view synthesis. The proposed model is evaluated on NVS and 3D reconstruction tasks compared to prior approaches.
strength_weakness: strength_weakness: 1. Besides its major contribution as a large-scale and open-world training dataset compared to ZeroNVS, the model primarily aligns with Zero-1-to3, which somewhat limits its technical contributions.

2.  Experiments:  
[-] Could you provide a reason why Zero-1-to-3 is not compared on the DTU benchmark, which is suitable for Zero-1-to-3 given its focus on single objects placed on tabletops?  
[-] I wonder how Zero-1-to-3 is applied in Fig. 3, as its camera pose involves both elevation and azimuth angles, which may differ from the video’s camera poses. Could the authors clarify how they calculate the camera pose?  
[-] This paper presents quantitative results only for experiments on DTU and MipNeRF360 (Tables 1 and 2).  Including visual comparisons could help understanding.
[-] Why does Table 4 in the supplementary materials show a higher Chamfer Distance compared to ZeroNVS?  

2. Others  
[-] Considering the training requirements outlined in the appendix, it’s evident that training is highly resource-intensive, which is reasonable given the dataset’s scale. I wonder if there are potential methods to expedite this process, such as initializing from a pretrained Zero-1-to-3 model?    
[-] Could the authors elaborate on how long-range image pairs contribute to the model? It’s challenging to distinguish correspondence in Fig. 9 from the appendix, as the images appear not to overlap. While such pairs may assist the model in extrapolating unseen regions, could they potentially hinder learning the viewpoint changes of observable content?   
[-] The visualization in Fig. 3 could be enhanced by including relative changes in viewing angles, as it’s currently difficult for humans to find correspondence between the output and input view images.
questions: questions: See above. Overall, I believe the proposed dataset and the demonstrated performance of the model hold promise for future research. I look forward to the rebuttal addressing my concerns.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces a new approach for efficiently finding corresponding frames from diverse viewpoints from YouTube videos at scale. The resulting 360-1M dataset contains over eighty million frames from approximately one million videos. Additionally, the paper also builds on the 360-1M dataset by proposing the ODIN model, which synthesizes novel views and reconstructs 3D scenes from a single input image. The ODIN model is trained by leveraging the diverse viewpoints in the dataset to handle significant camera view changes and complex real-world scenes. The authors demonstrate the utility of the 360-1M dataset as well as the benefits of their proposed ODIN model by comparing it to state-of-the-art 3D reconstruction approaches, where it outperforms the latter by a significant margin.
strength_weakness: strength_weakness: 1) The paper relies heavily on the 360-degree videos collected from YouTube, which may contain varying video qualities and resolutions. The described preprocessing steps does not fully account for this diversity in video quality, which may affect the quality of the extracted multi-view data negatively.

2) It is mentioned in line 220 that optimizing Equation 2 directly results in a degenerate solution and Equation 3 is introduced to mitigate this problem. However, there is no ablation to demonstrate this. It may help make the paper more comprehensive. 

3) In Tables 2 and 3,  it is shown that ODIN outperforms other approaches on the LPIPS and PSNR metrics but not the SSIM metric consistently. However, there is no discussion of possible reasons for this discrepancy.
questions: questions: Please look at the above-mentioned limitations.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper considers novel view synthesis from a single image. The main contribution is a dataset sourced from 1 million 360 videos from youtube, which is used to create about 380 million pairs of images with corresponding relative poses. With this dataset the authors train a diffusion model similar to zero-1-to-3 [24], but for general scenes instead of objects, and real-world scenes instead of synthetic ones. State-of-the-art results are shown for novel view synthesis from single images.
strength_weakness: strength_weakness: - There is a lack of qualitative examples. Specifically, a few things that would have been useful to show are 1) The generated images along the trajectories in fig. 4 in the form of both images and videos. Only the final 3d reconstruction by dust3r and not the intermediately generated images by ODIN are shown, 2) Images of the renderings of different methods in table 1 and 2, and 3) Generated images and the corresponding 3d reconstructions on Google Scanned Objects, so qualitative comparison corresponding to table 3 for all methods.
- There are a few missing references and as a result too strong claims in the paper. 1) “Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models” (Yu et al. ICCV 2023), 2) “Geometry-Free View Synthesis: Transformers and no 3D Priors” (Rombach et al. ICCV 2021), and 3) “Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image” (Ren et al. CVPR 2022). These works, and also zeroNVS (which is cited) all address novel view synthesis from a single image, though trained on smaller datasets. ZeroNVS also constructs a 3d scene by training a NeRF with Score distillation sampling. Due to this I think e.g. the claim on line 48 that ODIN is the first to reasonably reconstruct a 3d scene from a single image is too strong.
questions: questions: - I wonder how important the use of dust3r is for the 3d reconstruction. Is the confidence measure by dust3r used, and if so, how? I would imagine that it would be useful to get a consistent 3d reconstruction even if the views generated by ODIN are not consistent. By consistent I mean both that the views might not adhere to a specific camera model well enough for traditional sfm method to work well, or that the different views contain contradicting contents. Is that why there are some holes in the reconstructions in fig. 4 (e.g. the middle part in the leftmost 3d model of the cathedral), because it looks from the positions of the cameras that they should not be empty? What would happen if colmap was used on the generated images, would it still result in a reasonable 3d model?
- Did you notice any failure cases related to just conditioning on the previously generated image for the 3d generation? The camera motions in fig. 4 are fairly simple with little rotations and objects do not disappear and reappear, but if there would be more rotations I would imagine that it needs to be more carefully handled, e.g. as in the paper “WonderJourney: Going from Anywhere to Everywhere” (CVPR 2024).
- For 3d reconstruction, many methods (e.g. zeroNVS) use a NeRF model and score distillation sampling instead of direct 3d reconstruction from multiple generated images like it is done in this paper. Were any experiments like that performed?
limitations: limitations: This is adequately addressed
paper_summary: paper_summary: The paper introduces a large-scale, real-world, multiview, 360-degree outward dataset designed for static novel view synthesis and 3D reconstruction. To capture these data, the study develops a pipeline capable of identifying corresponding multiview frames from 360-degree outward videos with a fixed camera trajectory. To demonstrate the dataset's contribution and impact, the work proposes a diffusion-based model that achieves state-of-the-art performance using this dataset.
strength_weakness: strength_weakness: Dateset
- The dataset is collected from 360 YouTube videos, captured beforehand by the camera operator. The camera trajectory (x, y, z) is controlled by the camera operator, thereby limiting the flexibility to select diverse viewpoints from different positions.

- The quality of the data is unknown and can be unstable across different data points because the author cannot investigate each video one by one or frame by frame. This represents a tradeoff between capturing one's own data and crawling data from the Internet. For instance, the author acknowledges in lines 178-179 that it is infeasible to guarantee the uniqueness of video content.

- Although the work claims that 1 fps is sufficient in lines 126-128, this claim lacks experimental support. The author should consider trying different fps rates and show the performance gap and computational time between them.

- Larger dataset may need more training time and computational resource.

Method
- One of the most challenging aspects that prevent people from using video for static scene tasks is the impact of motion over time (multiview inconsistency). The work proposes a motion mask method to mitigate this impact. However, this aspect lacks analysis and discussion. For instance, the importance of the hyperparameter in Eq. 3 is unknown.

- In Fig. 3, in the second row, the method is shown to synthesize humans as well. However, I wonder whether the motion masking has a negative impact on synthesizing dynamic objects when they are static.

Writing
- In lines 5-6, the claim may be too arbitrary. As mentioned in lines 87-90, there are some large-scale real-world datasets, such as CO3D, albeit with limited diversity. A related work, OmniObject3D, should also be mentioned.

- In lines 8-12, the phrase "...diverse views" may be too broad and should be softened. The proposed dataset is collected from pre-captured 360 YouTube videos. The camera trajectory is decided by the camera operator beforehand, and only the camera orientation can be controlled after that. Hence, compared to general multiview datasets, which can have multiple viewpoints from different locations at the same timestamp, the views of the proposed dataset are limited.

- Eq. 2 does not match the equation in Sec. 5.1. The equation in Sec. 5.1 contains an undefined term.

- I hope the author can define their 360 multiview videos at the beginning since there are several kinds of 360 videos. For instance, DiVa360 captures 360-degree inward videos with multiple cameras at different locations at the same timestamp.
questions: questions: - I hope the author can verify that the viewpoints are diverse enough by discussing the difference between a multiview dataset that can capture images from different locations at the same timestamp and a dataset that can only change the camera orientation at the same timestamp.

- The dataset contains a large amount of data, which may vary in quality. I hope the author can discuss the impact of the low-quality data in the dataset. Is it negligible?

- Iines 178-179 mentions that it is infeasible to guarantee the uniqueness of video content. Hence, how do you make sure there is no overlap between training and testing set? How do you make sure the training dataset does not contain the testing cases in other dataset? Will it be unfair for the benchmark?

- The dataset contains diverse data. How is the data distribution in terms of motion? Will high-motion data, such as scenes with rain, have a negative impact? 

- Lines 126-128 need experimental support. Please consider trying it on a subset of the dataset.

- The work uses Depth Anything as the depth estimator. This is a monocular depth estimator with two types of checkpoints pretrained on two different datasets. I wonder if this causes a domain gap in the proposed dataset when using different checkpoints. Do you finetune the estimator on your dataset? Additionally, it is unknown how good the multiview consistency is on your data with this estimator.

- What is the tradeoff between using this dataset and others? From the experiment section, we know that the dataset can increase performance. But how about the training time compared to existing datasets? Maybe the author can consider adding the training time from the appendix to the table.

- How sensitive and stable is the motion masking hyperparameter in Eq. 3? How should it be tuned?

- What is the performance when the input image contains many dynamic objects?
limitations: limitations: I agree with the limitation section in the paper.
- One potential limitation of the large-scale dataset is to maintain the data quality and high fps.
- Another potential limitation is to make sure there is no overlap between training and testing set.

Comments: title: Growth Rate (Open question)
Post-rebuttal
Thanks For the Discussion and Score Increase
Continued Response to Reviewer whPn
comment: I just realized that my discussion is not public to the authors. Anyway, I copy and paste it here...

I have an open question regarding data maintenance. The dataset is large-scale because it includes existing data accumulated over time from YouTube, in addition to the data collected through your pipeline. My concern is whether the data growth rate might slow down in the future. Perhaps the authors could consider plotting the total amount of data over time or by year, so that readers can understand the data growth rate. This could also help the authors estimate the expected time needed to collect a new large amount of data and update the dataset to a second version.
Thank you very much for your comprehensive efforts in addressing my concerns. In particular, I find your response on how and why the long-range image pairs are used particularly helpful. I also appreciate your efforts on the additional ablation experiments, given the time and computational constraints on your end as well as empirically quantifying the effects of using 1 FPS in other responses. After reading all of the reviewers' feedback as well as the corresponding responses from the authors, I will retain my original rating.
I thank the authors for providing these answers. I keep my accept rating.
I thank the authors' efforts in addressing my concerns, especially over the experiment details. After reading the opinions from other reviewers, I raise my rating to weak accept.
That sounds good. About the long-term maintenance, I don't have too much experience with "super large-scale" dataset. I am not sure whether it is easy to find a place to store such a huge dataset in long term. Also, when I am taking about the maintenance, I am also talking about the update. Just, for example, Waymo released 9+ versions of its dataset [1] from 2019 to 2024, each with additional data or labels. But, anyway, this is not part of my reviews. I believe you all can figure out a way for the dataset.

[1] Scalability in perception for autonomous driving: Waymo open dataset, CVPR 2020
We appreciate your diligence as a reviewer and the insightful feedback for improving our work! Our current plan is to host the links to the videos, the relative camera poses, the video meta-data, and the pretrained models through Hugging Face. Additionally we plan to release the data processing pipeline through github along with the model training code. If you have advice for long-term maintenance of datasets we appreciate your input. 

Also, we thank you for the rating increase.
I thank the authors for addressing all of my questions and concerns in the rebuttal. I also read the other reviews. I understand how challenging it is to propose a dataset. I personally consider the dataset pipeline to be part of the technical contribution. I am glad that the authors ran some experiments on a subset with ground truth to ensure the quality.

If the authors are willing to share, I am curious about the plan to maintain this dataset. For dataset paper, I think long-term maintenance is much more important than the paper itself sometimes.

After reading the rebuttal, I decide to change my score from weak accept to accept.
Below we've included responses to the remaining questions and comments from reviewer whPn. We appreciate their detailed feedback and questions. 

**The dataset contains a large amount of data, which may vary in quality. I hope the author can discuss the impact of the low-quality data in the dataset. Is it negligible?**

We found that low quality video frames such as blank frames, blurry frames, low resolution frames, or those with minimal camera movement were automatically filtered out in the Dust3r scoring phase. If the frames were low quality they naturally received a low score because the depth could not be well estimated. We verified this through manual inspection by randomly sampling frames which were selected by the Dust3r scoring mechanism.

**Iines 178-179 mentions that it is infeasible to guarantee the uniqueness of video content. Hence, how do you make sure there is no overlap between training and testing set?**

We ran deduplication over video titles, URLs, and thumbnails for all 1 million videos though this does not guarantee videos contain overlapping clips. The primary utility of a large multi-view dataset is for training models. We believe that datasets such as Mip-NeRF are better for evaluation. 

**Lines 126-128 [in theory we can align the views to look at the various regions of the scene to form multiple view correspondences] need experimental support. **

Figure 8 and 9 in the appendix show qualitative paired frames from the dataset. Quantitatively we found 80,567,325 paired frames from the videos by aligning the views. Upon manual inspection we found that the frames were aligned to contain overlapping content. We will add more qualitative paired frames from the dataset to the appendix for readers view. 

**The work uses Depth Anything as the depth estimator. I wonder if this causes a domain gap in the proposed dataset.**

We used the outdoor version of Depth Anything off-the-shelf. We found that the depth estimation did not need to be extremely precise, as small errors in the scale of the scene did not impact training the model. 

**What is the tradeoff between using this dataset and others?**

We believe our dataset is complementary to other datasets. For example, ZeroNVS combines multiple datasets to train their model. We see no reason why our dataset could not be similarly combined with other multi-view datasets to train even better models. In terms of training, the training time scales proportionally with dataset size so more data means longer training. We will add a FLOPS comparison to the appendix.