WildCAT3D: Appearance-Aware Multi-View Diffusion in the Wild
Authors: Morris Alper
David Novotny
Filippos Kokkinos
Hadar Averbuch-Elor
Tom Monnier
Venue: NeurIPS 2025 poster ()
Keywords: novel view synthesis
diffusion models
computer vision
computer graphics
Primary area: applications
TL;DR: WildCAT3D learns multi-view diffusion from images in the wild by explicitly modeling global appearance variations.

Abstract: Despite recent advances in sparse novel view synthesis (NVS) applied to object-centric scenes, scene-level NVS remains a challenge. A central issue is the lack of available clean multi-view training data, beyond manually curated datasets with limited diversity, camera variation, or licensing issues. On the other hand, an abundance of diverse and permissively-licensed data exists in the wild, consisting of scenes with varying appearances (illuminations, transient occlusions, etc.) from sources such as tourist photos. To this end, we present WildCAT3D, a framework for generating novel views of scenes learned from diverse 2D scene image data cap tured in the wild. We unlock training on these data sources by explicitly modeling global appearance conditions in images, extending the state-of-the-art multi-view diffusion paradigm to learn from scene views of varying appearances. Our trained model generalizes to new scenes at inference time, enabling the generation of multiple consistent novel views. WildCAT3D provides state-of-the-art results on single-view NVS in object- and scene-level settings, while training on strictly fewer data sources than prior methods. Additionally, it enables novel applications by providing global appearance control during generation.

WildCAT3D: Appearance-Aware
Multi-View Diffusion in the Wild
Morris Alper!.2 David Novotny²
Filippos Kokkinos
Hadar Averbuch-Elor' Tom Monnier
Tell Aviv University
Meta Al
*Cornell University
https://vildcattdl github.io
Training image collections in the wild
REURRE
Novel-view synthesis from single image
Novel-view synthesis with appearance control
"a spring day with clear blue sky"
Figure 1: WildCAT3D. (top) We large image collections captured in the wild to train our feed-forward novel-view synthesis model. (middle) At inference time, WildCAT3D can generate full level novel views from single image of new (never-before encountered) scene. (bottom) It can also be used to control the appearance of the generated views, e.g. via a text prompt.
Abstract
Despite recent advances in sparse novel view synthesis (NVS) applied to object- centric scenes, scene-level NVS remains a challenge. A central issue the lack available clean multi-view training data, beyond manually curated datasets with limited diversity, camera variation, or licensing issues. On the other hand, an abundance of diverse and germissively-liceneee scenes with varying appearances (illuminations, transient occlusions, etc.) from sources such tourist photos. To this end, we present WildCAT3D, framework for generating novel views of scenes learned from diverse 2D scene image data cap- tured in the wild. We unlock training these data sources by explicitly modeling global appearance conditions in images, extending the state-of ofte--of-theaar the-art multi-view diffusion paradigm learn from scene views of varying appearances. Our trained model generalizes to new scenes at inference time, enabling the generation of multiple consistent novel views. WildCAT3D provides state-of-the-an results on single-view NVS in object- and scene-level settings, while training on strictly fewer data sources than prior methods. Additionally, it enables novel applications by providing global appearance control during generation.
data exists in the wild, consisting
Work done during Morris's internship at Meta.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1 Introduction
Imagine observing the Golden Gate Bridge from afar at different viewpoints as the weather conditions change in fog, bright sunlight, in rain and at night, From these observations, one can intuit the 3D structure of the Golden Gate Bridge, and can likely imagine how similar bridges might appear from various viewpoints. Our work applies a similar intuition to novel view synthesis (NVS)- the task of predicting a new 2D view of a scene which has been partially observed - to generate views of new scenes by learning from observations differing in global appearance. While recent progress in NVS has been achieved by leveraging powerful multi-view diffusion models, as popularized by CAT3D Gao al. 2024), these models typically have poor applicability to full scenes due to scarce clean multi-view data. As such, these models typically build off pretrained image generation models which are fine-tuned limited datasets of synthetic renderings or crowd-sourced videos capturing isolated objects, with poor applicability to full scenes like the MegaScenes collection (Tung image
ct al. 2024]. On the other hand, such scenes are abundantly covered in permissivel-licenen collections captured in the wild from the Internet, in which scene views differ greatly in appearance (e.g., aspect ratios, lighting, weather conditions transient occlusions), making them incompatible with existing multi-view diffusion architectures. In this work, we unlock their ability to learn from this scalable source of readily- available, diverse, and permissively licensed scene data.
We propose WildCAT3D, a multi-view diffusion model à la CAT3D which can be learned from in-the-wild Internet data through appearance awareness. Our key insight is that inconsistent data can be leveraged during multi-view diffusion training to learn consistent generation, by specifically decoupling content and appearance when denoising novel views. More concretely, starting from the standard multi-view diffusion framework of CAT3D, we propose to explicitly integrate a feed- forward and generalizable appearance model whose goal is capture the appearance properties of the input views. We do so by adding an appearance encoding branch to the model, designed to produce low-dimensional appearance embeddings that are used as an extra conditioning signal for the niew diffusion model. This branch is trained simultaneously with the diffusion model and custom classifier-free guidance mechanism is applied to avoid oversaturation artifacts. During inference, the appearance embedding from the source view is injected to the target views, which allows us to preserve the appearance across the generated views. Intuitively, these design choices allow our model to "peek" at coarse appearance signals such as weather condition and aspect ratio, without leaking too much information about the target views to denoise.
In addition, in order to improve the viewpoint consistency, we augment our appearance-aware model by adapting a warp conditioning mechanism the context of our multi-view diffusion framework. More specifically, for each target view to denoise, pixels from the source view are warped following the target viewpoint using a known depth map. and the resulting image is injected as an additional conditioning signal into the diffusion model. Intuitively, such a mechanism approximately indicates the correct placement of the scene, thus resolving the scale ambiguity that inherent to the single-view NVS problem.
We exhaustively compare our method standard NVS benchmarks and demonstrate superior performance, while training on fewer curated data sources than prior works. Importantly, this highlights the strength of our work in leveraging larger set of samples from existing, permissively licensed imagery captured in-the-wild, rather than relying on heavily curated datasets. Our results also show strong performance on diverse scenes captured in tourist photos, including static video generation from single input frames and custom camera trajectories. In addition, our explicit modeling of appearance both allows us to learn from in tn-the-wild data successfully generate consistent views of full scenes and also enables novel applications such as interpolation between views of differing appearances or NVS with appearance control via text, as showcased in Figure 1
In summary, our key contributions are three-fold:
A new appearance-awar multi-view diffusion model able to learn from in-the-wild images, Performance superior state-of-the-art methods on single-view NVS benchmarks,
Novel applications enabled by NVS with controlled appearances.

---

2
Related Work
NVS with Difftusion Models. Following recent progress in using diffusion models for generative modeling of image data, a line of works has successfully applied new-conditioned diffusion to NVS. Earlier works are limited to in-distribution object views with masked backgrounds and spherical camera poses [Watson et al., 2022] [Zhou and Tulsiani [023]. The more recent Zero-1-to-3 [Liu jet al. 2023a] and ZeroNVS [Sargent et al., [023] train diffusion model to generate a new view conditioned on observed view and new camera pose, using curated multi-view image data as strong supervision to learn generalizable NVS. More recently, following works demonstrating diffusion models with multi-view prior [Shi et al. 2023a Li ct al.
2023 Wang and Shi 2023 Shi et al. 2023b Yang et al., 2024al Liu et al. 2023b]. CAT3D [ao et al. 2024] has shown SOTA NVS performance by leveraging this multi-view prior, allowing for multiple observed and/or output views to be processed in parallel. These approaches yield high-quality novel views, which may be used for tasks such as downstream 3D asset reconstruction. However, they are constrained by limited available training data, mostly covering single objects captured in crowd-sourced videos or synthetic renderings. Moreover, key sources of synthetic data may have contested licensing status. In contrast to these, our work enables training a multi-view diffusion model on a freely-licensed and abundant source of data in-the-wild, whose appearance variations are incompatible with these prior methods.
Another line of work leverages diffusion models with a warp-and hurp-and-inpaint pipeline to enforce 3D consistency in unbounded scenes (Fridman et al., 2024 Yu et al. 2024a (Shriram et al., 2024 Chung ict al. [202] Yu et al. 2024b]. While this ensures 3D consistency, it often accumulates errors from depth estimation leading to inaccurate warps. By contrast, our method uses warps conditioning signal and not a strict constraint, allowing our model to correct such inaccuracies.
NVS from in-the- Wolthe-Wild Image Data. The abundance of Internet photo-touriste images has inspired research on extracting 3D structure from such data for tasks such as NVS. While such work predates modern neural methods (Snavely et al. 2006 Agarwal et al., 201 recent works have used deep learning methods applied large-scale photo collections which have been processed with SfM pipelines [Li and Snavely 20188 Tung et al.) [202]. A number of works have concentrated on enhancing NVS and 3D reconstruction pipelines with explicit modeling of appearance variation between photos captured in-the-wild [Meshry et al. 2019 Li et al. 2020 Martin-Brualla al., 2021 Chen et al. 2022 Kulhanek et al. 2024). These methods perform test-time optimization on single le scene, with appearance representations extracted from pixel-level features or learned as directly optimized vectors. By contrast, our method trains a generalizable encoder which extracts appearance representations from image latents. Moreover, our framework uses these representations as conditioning signals for diffusion-based generation, which requires training- and inference- time adaptations generate consistent, high-quality novel scene views. As such, our trained model is able to generalize to novel scenes without lengthy optimization and may directly use appearance information to condition high-quality 2D view generation, unlike existing works.
3 WildCAT3D
We proceed to define the WildCAT3D framework. We begin by providing background on the CAT3D [Gao et al. 2024] framework (Section (.1) which our method extends. We then describe WildCAT3D's explicit modeling of appearance conditions (Section 3.2) and its scene scale disam- biguation via warp conditioning (Section 3.3 Our full pipeline is illustrated in Figure
3.1 Background: CAT3D
CAT3D is multi-view diffusion model, adapted and fine-tuned from text-to-image Latent Diffusion Model (LDM) [Rombach et al., 2022] in order to generate multiple views of a scene conditioned on source view(s) and source and target camera poses. Denoting the latent dimension as k and the spatial resolution of latents as n n, the input noise (originally ie x n x n) first expanded to accept e slots corresponding to observed or unobserved views. Then, the input I of shape x (k + 7) x n X n consists of ground-truth latent for observed views and noise for unobserved views, concatenated channel-wise with 7 additional channels including a binary mask for observed and unobserved views (copied over all n x n spatial locations) and a x n x n-dimensional Plücker raymap (Sajjadi et al., 2022 Zhang et al. 2024] parametrizing the cameras for each view. To process these inputs, the
3

---

Pixal Warping
Source view
Camera
Camera
TTS
Plucker Embedding
VAE Encoder *
Plucker Embedding
VAE Enooder +
VAE Encoder +
Appearance Encoder
Appearance Encoder
MM
moise
VAE Decoder *
Dencieed larget
diffusion model
Figure 2: Overview. WildCAT3D learns to synthesize novel views by denoising target views of inconsistent appearances from a source view. Given a batch of source (blue) and target (red) views, we first compute camera embeddings and VAE latents. The latter are then fed to encoder computing small appearance vector copied across spatial locations, allowing the model to "peek" at appearance conditions. Finally, for target views, compute additional warping embeddings using the VAE applied to warped source images calculated from an estimated depth map. These signals are channel- wise concatenated and fed the diffusion model. During training (depicted above), noise is added target view latents. During inference, target view latents are replaced by random noise while their appearance channels are copied from the source view branch.
LDM''s self-attention layers are expanded to 3D attention, by selecting queries, keys, and values from all e views. The original text-based cross-attention layers are removed (discarding the text encoder). At each denoising pass, ground-truth clean latents passed in observed positions of the input, and the denoising loss in training is applied to slots for unobserved views only. CAT3D may be trained with any number (S v) of observed and unobserved views, integrating single- and multiple-view NVS in a single model. During inference, CAT3D generates multiple novel scene views parallel.
This may be formulated mathematically as follows. A generative model for single images estimates the probability distribution p([[t) of images conditioned on some conditioning signal I. CAT3D models the distribution p(I" I c²) where ,,,,a indicate observed/inoossevedd camera information for a given view. This is parametrized by model weights and Gaussian noise z" (O, 1))" (i.id. noise for each unobserved view), and images parametrized by the VAE.
views, and the
3.2 Appearance-Aware Multi-Vies Diffusion
In-the-wild photo collections show appearance variations such as differing aspect ratios, lighting conditions, seasonal weather, transient occlusions, etc., and naively training CAT3D such data results in similar inconsistencies at inference time (shown in our ablations). We thus propose to explicitly model appearance variations while encouraging consistency inference time as follows.
Generalizable Appearance Encoder. We augment CAT3D with feed-forward appearance encoder module, implemented as shallow convolutional network applied to image latents. By compressing each view into single, low-dimensional vector, this serves as an information bottleneck. encoding coarse global appearance variations without the capacity to leak fine- tne-grained details in images. During training. the d-dimensional appearance embedding of each view copied across each n spatial position. The resulting & x d x n n tensor is concatenated to the input channels of CAT3D, allowing model "peek" at global appearances of both observed and unobserved views during training (despite unobserved views' latents themselves being noised). The appearance encoder is jointly trained with the model's denoising objective at train time; it is generulizable
can be applied to new scenes at inference (unlike test-time methods such Chen et al. (2022

---

Using the notation from Section 3.1 we formulate this approach as follows. We assume that each image possesses a latent appearance variable a, representing the conceptual space of appear- ance variations for the same underlying scene. WildCAT3D models the conditional distribution p(T"[[o, ca, a", w²), parametrized by model weights 0, Gaussian noise z (N(0, 1))², appearance variables a", and warp information w (see Section 3.3
We also assume that appearance can be derived given an image; as such, we learn encoder A with weights & that parametrizes a as Ag (I). Thus we model p(1" c², AI")), w"), with the predicted distribution determined by weights 6, Φ. We implement A with a hight-weight convolutional network to introduce an inductive bias towards global appearance. Note that Ag is a lossy (bottleneck) function, compressing an image into a single, low-dimessional vector. Therefore, I" cannot be directly reconstructed from A,(I") and thus modeling this distribution is non-trivial.
We note that this derives appearance representations directly from images, rather than using ac- companying textual metadata. This has several advantages: Images may not be accompanied by descriptive text, conditioning on text would require additional novel components to bridge text and visual appearance spaces, and important appearance details may be poorly reflected in text, such as precise image aspect ratios and lighting conditions.
Our encoder design follows standard convolutional principles with gradually decreasing spatial dimen- sions to create now-dimensional appearance bottleneck. The architecture requires dindeneniional output to serve as an effective information bottleneck. Overall, this uses a minimal network avoiding additional design choices parameters that would require computationally intensive testing. Further details are provided in the appendix.
Appearance-Awarg Conditioning, Following CAT3D, we apply Hassifier-free guidance (CFG) (Ho and Salimans| 2022] to achieve high visual quality; this drops conditioning signals (latents and camera rays for observed views) for unconditional training, and extrapolating between conditional and unconditional predictions inference. However, further naively applying CFG to the appear- ance conditions (i.e. masking a" in unconditional training) ) results in oversaturation artifacts. We hypothesize that this is because appearance embeddings are tied to image lighting and color bal- ance, and CFG is known to be prone to oversaturation when applied to components controlling "gain" of intensity values in images (Sadat et al. 2024). Therefore we propose an appearance-awar conditioning method achieve the benefits of CFG without such artifacts. We adapt the "uncondi- tional" setting of standard CFG by keeping appearance conditions while dropping other observed view conditions: ( (T"]c", Ag (I"),, i.e. we still condition all appearance embeddings As (19) = (a",a") a)) which includes those of observed views. In other words, WildCAT3D "peeks" at the global appearances of all views in both conditional and unconditional training settings of CFG.
Inference There is an inherent gap between the training objective of WildCAT3D (denoising views of varying appearances) and its use during inference (generating views fully consistent with an input view). To produce outputs that are consistent despite training on inconsistent data, select the first observed view Ig. calculate its appearance embedding ag = A (In), and copy it into the appearance embedding channel of each unobserved view, then generating using appearance-aware CFG as described above. Intuitively, this conditions the generated views on the same overall appearance as the first observed view, and our results show that this indeed succeeds in matching its appearance characteristics (lighting, style, etc.). Interestingly, this solution also enables the preservation of the aspect ratio, resulting in generated views that are consistent enough to generate smooth and appealing videos given single image.
Ippearance-Conditionn
Since the appearance embedding used at inference is arbitrary. we can actually use a completely external image to compute the desired appearance embedding and thus perform appearance transfer or appearance-controlle to concatenate model with text-to-image retrieval model. More specifically, given a text prompt, we use CLIP Radford et al. (021) to compute its similarity with pre-computed image embeddings | from given database. In practice, we perform this retrieval over approximately 10K images from MegaScenes, covering varying weather and lighting conditions similar to the full dataset.
generation. In order to support appearance control through text, we propose
3.3 Warp Conditioning
central challenge for single-view NVS systems is the inherent scale ambiguity of single image input: given observed view and the relative camera pose of a desired unobserved view, the scale
5

---

of the translation vector between the cameras' extrinsic poses is unknown itt al. 2022]. We resolve this via the observation from [Tung et al., 2024 that scene scale can be injected by an image warped according to its depth aligned to the extrinsic camera scale. To adapt this to our multi-view diffusion setting, we warp the first observed view to the camera of each of the slots of CAT3D using estimated depth map aligned to the scene's SfM pointcloud, concatenating the VAE latents of each warp as additional conditioning channels. Note that this is fully compatible with in-the- wild data since warps encode the correct pose even when differing in global appearance from target views. We show in our ablations that this warp conditioning is necessary for accurate viewpoint consistency.
Ranftl et al. 2020 Yin
To summarize, total WildCAT3D has input of shape u (2k + d d n x n, where the k + input channels of CAT3D (k latents, 6 camera embeddings, 1 binary mask) are extended with d channels for appearance embeddings and & channels for warp latents.
3.4 Implementation Details
For WildCAT3ID's pretrained image generative backbone, we use open-source LDM similar to [Rombach et al. 2022]. We first expand it CAT31) model and train on the standard curated CO3D [Reizenstein et al. 2021] and Re10K (Zhou et al.] 2018) multi-view datasets; we then expand to a full WildCAT3D model and fine-tune on MegaScenes (Tung et al. [2024] and CO3D. Our appearance module is a fully-convolutiona network applied to image latents, outputting embedding of dimension
for each image. This is copied to every spatial location and concatenated as additional channels to the denoiser network's 's input at each denoising step. By default we use 8 (slots for views), training with one observed and seven unobserved randomly- scene views. For video results (supp. mat.), we increase this to
16 slots at inference.
The additional input channels are handled through a lightweight 1xl convolution projection layer, making them compatible with the lower input dimension expected by the LDM's denoising network while adding negligible extra parameters. See the appendix for further details.
To calculate warp images for warp conditioning, we unproject the first observed view to 3D point cloud, and then render it from each camera pose. Following Tung al. (2024), we unproject using depth calculated with DepthAnything (Yang et al. (024b) and aligned with RANSAC to the COLMAP point cloud (corresponding the cameras' extrinsic scale). We render warps by rendering this point cloud from each view, with points' RGB values derived from the first view.
4 Experiments
4.1 Novel View Synthesis Results
NVS metrics are provided in Table I comparing to the recent SOTA MegaScenes NVS model (MS NVS) along with prior models reproduced from [Tung et al., 2024 Our method trained on MegaScenes directly (unlike the aggressive filtering used to train MS NVS) mostly achieves superior performance across the board, on both reference-basee based and generative metrics, and on out-of-distribution datasets (object-centric DTU and scene-centric Mip-NeRF 360). This is despite MS NVS and other prior models being trained on additional sources of multi-view data (see supp.). Visual comparisons are provided in Figures 6a and in our supmat, showing that WildCAT3D generally maintains consistency with observed views while hallucinating plausible content for unseen regions.
On the MegaScenes benchmark itself, our quantitative results are comparable to the MS NVS baseline. However, we observe a significant qualitative improvement when applied to views with novel trajectories, as seen in Figure We suspect this reflects the MegaScenes baseline model being trained on image pairs selected using the same filtering method used to construct the MegaScenes test set, which consists of (ground truth, target) image pairs, while our method not exposed to such filtering at train time. Hence, the baseline model may be partially overfit to the data format of this benchmark, Consistent with this, our model shows significantly better metrics on out-of-distribution NVS benchmarks including the challenging scene-level Mip-NeRF 360 benchmark (Table I.

---

DTU
Jensen al.
2014
Mip-NeRF 360
Barron et al.
2022
Method
PSNR
SSIM t LPIPS + FID + KID'
PSNR
SSIM
LPIPS
FID
KID*
4
Zero-I-to-3 (released)
6.872
0.210
0.565
128.9
0.297
10.72
0.287
0.526
171.2
1.126
ZeroNVS (released)
5.799
0.111
0.648
160.0
0.352
6.999
0.124
0.669
137.0
0.537
Zero-I-to-3 (MS)
7.637
0.276
0.516
101.9
0.223
12.92
0.383
0.443
67.65
0.163
ZeroNVS (MS)
8,019
0.307
0.483
87.41
0.158
13.78
0.412
0.406
60.68
0.139
SD-Inpaint
9,946
0.369
0.495
214.4
1.067
12.92
0.400
0.456
150.1
0.792
MS NVS
8.795
0.393
0,400
85.96
0.163
14.06
0,441
0.381
64.41
0.142
WildCAT3D (Ours)
10.77
0.426
0.388
57.32
0.039
14.77
0.445
0.352
42.17
0.050
Re10K Zhou et al.
2018
MegaScenes
TTng al. 2024
Method
PSNR 1 SSIM t LPIPS 4 FID 4 KID"
PSNR
SSIM
LPIPS
FID
KID
Zero-1-to: (released)
11.63
0.438
0.405
160.2
0.725
9.090
0.241
0.548
86.89
0.634
ZeroNVS (relcased)
9.487
0.353
0.456
123.0
0.352
7.471
0.151
0.616
69.10 19.10
0.487
Zero-I-to-3 (MS)
14.64
0.570
0.272
68,91
0.024
12.16
0.367
0.429
9.784
0.023
ZeroNVS (MS)
16.02
0.630
0.205
61.12
0.024
12.90
0.401
0.386
9.838
0.024
SD-Inpaint
15.54
0.643
0.269
118.9
0.396
12.36
0.392
0.425
38.48
0.242
MS NVS
17.22
0.666
0.177
60.01
0.023
13.40
0.445
0.344
11.58
0.040
WildCAT3D (Ours)
21.58
0.758
0.131
24.70
-0.001
13.92
0.439
0.355
9.871
0.015
Table I: Novel-view synthesis benchmarks. We evaluate the single-view setup and compare our
results prior works as reported by Tung et al.] [024]: Zero-I-to-3 [Liu et al. 2023a ZeroNVS Sar
gent et al. [023), naive inpainting method dubbed SD-Inpaint, and MegaScenes N VS model Tung al. [2024]. Our method achieves superior performance compared to baseline methods while using strictly fewer data sources and not requiring aggressive data filtering. KID indicates KID values multiplied by ten for readability. Best results are in bold.
4.2 Additional Applications
In Figures we illustrate additional applications of WildCAT3D. By injecting the appearance embedding of an external image during infereence, we may generate novel views a scene while editing its appearance (e.g. generating nighttime scene observed from daytime photo). By concatenating this with CLIP text-to-image retrieval [Rafford et al. 202T], we may use text to control in the figure, text-based retrieval may effectively find images capturing the desired overall appearance for subsequent injection. Finally, by fine- neeetunnng WildCAT3D with two observed input views, can interpolate between views of a scene with differing appearances to produce static video with a consistent appearance starting and ending at the two respective poses. At inference time, this uses camera poses along interpolated trajectory between the two input cameras, along with the appearance either the start or the end pose injected into each slot.
the edit (e.g. "sunset"))
4.3 Analysis of Appearance Embeddings
To further interpret our results, we analyze the appearance embeddings produced by WildCAT3D's trained appearance encoder module. We cluster embeddings of approximately 20K MegaScenes (val and test) images with K-Means (k 100). Figure 7 illustrates such clusters projected into two dimensions via PCA, and random exemplars from different clusters. Clusters contain images with similar aspect ratios, lighting conditions, and other global appearance factors (e.g. indoor vs. blue sky nighttime), providing interpretability to the appearance encoder's functionality.
4.4 Ablations
We ablate key elements of our framework in Table 2 and Figure ob leads to spatial misalignments due to scene scale ambiguity. Further removing appearance modeling, i.e. fine-tuning CAT3D directly on in-the wild data, leads to appearance inconsistencies. By contrast, our full model improves on all benchmarks except Re10K from training on in-the-wild data over the base CAT3D, due to our careful modeling of appearance and scene scale. Note that Re10K has very limited diversity and camera movement, and is in-distribution for the base CAT3D; we thus interpret the stronger performances of the latter on Re10K as overfitting effeect.
Removing warp conditioning
7

---

Input View
Generated Novel Views
Ours
MS NVS
CIMA
Ours
MS
Figure 3: Qualitative comparison on MegaScenes with novel trajectories. Using single images as input (left), we show results for WildCAT3D and MegaScenes NVS model (MS NVS) scenes unseen during training, conditioned on continuous camera trajectory. We see that our model significantly outperforms prior SOTA generating consistent and high-quality sequences from single views. We encourage the reader to check our video results our supmat to further assess the quality gap.
Input View
Appearance*
Generated Novel Views
Figure 4: Application: appearance-controllee generation. Starting from a source view (left) and an additional image with specific appearance (middle), our model is able to synthesize novel views that are not only consistent with the source view content and the desired viewpoints, but also consistent with the appearance style the additional image (right). We perform text- -ext-guidance by concatenating model with text-to-image retrieval model. *Retrieved with text prompts "sunset" and "a spring day with a clear blue sky" respectively.

---

Start View (Input)
End View (Input)
Generated Interpolation Sequences
Figure 5: Application: in-the-wild interpolation. When fine-tuned with two observed views, WildCAT3D can interpolate between scene views with differing appearances. Injecting the appearance embedding of either the start the end pose yields generated views with consistent appearances.
Ground-truth
MS NVS
Ours
GT
Maaa
(a) Comparison with prior work
(b) Ablation study
Figure 6: Qualitative NVS results. (a) We compare predicted views to the ground-truth for MegaScenes NVS model from Tung al. 2024 (MS NVS) and Our model shows greater consistency with target poses as well as better visual quality, consistent with our quantitative results. (b) Removing warp conditioning (-warp) results misalignment relative to ground-truth camera poses. Training CAT3D directly on in-the-wild data (i.e. without warps and appearance embeddings, -warp-app) yields inconsistent output images.
Figure 7: Appearance embedding analysis. A subset of K- Means clusters visualized with 2D PCA and random cluster images. They tend to show similarities in appearance and aspect ratio.

---

DTU
Jensen
2014
Mip-NeRF 360
Barron al.
2022
Method
PSNR
SSTM
LPIPS 4 FID 4 KID"
PSNR
SSIM
LPIPS 4 FID 4 KID*
WildCAT3D (Ours)
10.77
0.426
0.388
57.32
0.039
14.77
0.445
0.352
42.17
0.050
-warp
9.795
0.374
0.439
54.89
0.042
13.98
0.404
0.395
44,82
0.056
-warp-app
8,699
0.281
0.491
68.80
0.069
13.90
0.390
0.417
46,80
0.064
Base CAT3D
10.25
0.399
0.424
53.96
0.042
13.92
0.399
0.410
57.96
0.205
Re10K
Zhou et al
2018
MegaScenes SSIM
Tung et
2024
Method
PSNR
SSIM
LPIPS 4 FID 4 KID
PSNR
LPIPS + FID 4. KID*
WildCAT3D (Ours)
21.58
0.758
0.131
24.70
-0.001
13.92
0.439
0.355
9.871
0.015
-warp
18.97
0.670
0.182
28.71
0.003
13.45
0.410
0.379
9,934
0.014
-warp-app
17.73
0.625
0.216
31.99
0.008
12.63
0.368
0.422
12.41
0.026
Base CAT3D
21.89
0.751
0.127
21.53
-0.004
12.91
0.390
0.415
19.16
0.116
Table 2: Quantitative ablation study, We evaluate ablating key parts of our model, namely warp ("warp"" and appearance components ("app"), when trained on in-the-wild data. We also report the performance the base CAT3D without any in- to-the-wild finetuning. KID' indicates KID values multiplied by ten for readability. Best results are in bold.
5 Conclusion
We have presented the WildCAT3D framework for generalizable novel view synthesis learned from images in-the-wild. By explicitly modeling appearance variations, WildCAT3D unlocks training on abundant and permissively-licensee appearance conditions of generated views. Our results have shown its superior performance on NVS benchmarks and novel applications, while training on strictly fewer data sources than prior methods and more fully leveraging existing open web data capturing full scenes. Our experiments, while on limited scale, demonstrate that modeling appearance enables learning from unfiltered, in-he-wil data, laying a foundation for web-scale NVS training.
photo-tourism data, as well as allowing control over the global
Limitations and Future Work. Generated views are not guaranteed be fully consistent, unlike methods using explicit 3D representations. While our method mitigates degradation due to training on mutually inconsistent data, some visual artifacts persist, such as mild flickering and saturation changes. Our architecture could be enhanced by incorporating a video prior, textual conditioning, automatic generation of camera poses, or explicit modeling of transient occlusions. Additional components may help model semantic variations between images (e.g. holiday decorations) or view-dependent effects such as reflections. Our method relies on an existing depth estimation pipeline, and emrors in predicted depth may propagate to novel views (see appendix). Finally, while our text- based conditioning via image retrieval may provide coarse appearance conditioning, future work could add explicit textual conditioning allow more fine-grained editing control over semantic content of scenes.
Societal Impact. While scene generation shows promise for positive applications in entertainment and education, we acknowledge the inherent risks of visual generative models to produce disinformation or undesired hallucinations. The central aim of our work encourage the adoption of open data for state-of-the-ard NVS, while still requiring the same caution in responsible usage as existing generative methods.
Acknowledgments and Disclosure of Funding
This work was sponsored by Meta AI. We thank Kush Jain and Keren Ganon for providing helpful feedback.
10

---

References
Ruiqi Gao, Alcksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. Advances in Neural Information Processing Systems, 2024.
Joseph Tung, Gene Chou, Ruojin Cai, Guandao Yang, Kai Zhang, Gordon Wetzstein, Bharath Hariharan, and Noah Snavely. Megascenes: Scene-level view synthesis scale. In ECCV, 2024.
Daniel Watson, William Chan, Ricardo Martin- Brriin,, Jonathan Ho, Andrea Tagliasacchi, and Mo- hammad Norouzi. Novel view synthesis with diffusion models. arXiv preprint arXiv:2210004628, 2022.
Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12588-12597, 2023.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1- Zero-shot one image 3d object. In Proceedings of the IEEE/CVF international 2023a.
conference on computer vision, pages
Kyle Sargent, Zizhang Li. Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. Zeronvs: Zero-shot 360-degree view synthesis from single real image. arXiv preprint arXiv:2310. 103310.1790 17994, 2023.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023a.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.062/4 2023.
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201, 2023.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. 7ero123++: single image consistent multi-view diffusion base model. arXiv preprint 23Xiv22310.15/10 2023h.
Jiayu Yang, Ziang Cheng. Yunfei Duan, Pan Ji, and Hongdong Li. Consistnet: Enforcing 3d consistency for multi-view images diffusion. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 7079-7088, 2024a.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consiseee images from a single-view image. arXiv preprint 00X.0:2309.03453, 2023b.
Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. Advances in Neural Information Processing Systems, 36, 2024.
Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, et al. Wonderjourney: Going from anywhere to everywhere. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6658-6667, 2024a.
Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ramamoorthi. Realmdreamer: Text-driven 3d scene generation with inpainting and depth diffusion. arXiv preprint arXiv:2404.07/99 2024 2024.
Jacyoung Chung, Suyoung Lee, Hycongjin Nam, Jaerin Lee, and Kyoung Mu Lee, Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384 2023.
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint 24Xiv:2409.02044 2024b.
11

---

Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. ACM Transactions on Graphies (TOG), (5(3):835-846 2006.
Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in day. Communications of the ACM, 54(10): 105-112, 2011.
Zhengqi Li and Noah Snavely, Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2041-2050 2018.
Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla. Neural rerendering in the wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6878-6888 2019.
Zhengqi Li, Wenqi Xian, Abe Davis, and Noah Snavely. Crowdsampling the plenoptic function. In Computer VCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16. pages 178-196. Springer, 2020.
Ricardo
Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, Xuan Wang, and Jue Wang. Hallucinated neural radiance fields in the wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 12943-12952, 2022.
Jonas Kulhanek, Songyou Peng, Zuzana Kukelova, Mare Pollefeys, and Torsten Sattler. WildGaus- sians: 3D gaussian splatting in the wild. arXiv, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 10684-10695, 2022.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6229-6238, 2022.
Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang. Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. arXiv preprint arXiv:2402. 148/7, 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
Seyedmorteza Sadat, Otmar Hilliges, and Romann M Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. arXiv preprint arXiv:2410.02416, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 1433::62331133) 2020.
Wei Yin, Jianming Zhang, Oliver Wang. Simon Niklaus. Simon Chen. Yifan Liu, and Chunhua Shen. Towards accurate reconstruction of 3d scene shape from single monocular image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):6480-644) 2022.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEF/CVF international conference on computer vision, pages 10901-10911, 2021.
12

---

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely, Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 07X1v:1805.09817,,0000 2018.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024b.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 406 413,41132222 2014.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5470-5479, 2022.
Jianyuan Wang. Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 31686-21697, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühher, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4), July 2023. URL https://repo- sam inria. fr wtw.////app////////ae/aa caaussannssplattiie
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005//2 2022.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings the IEEE/CVF winter conference on applications of computer vision, pages 5404-5411, 2024.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi. and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern 2023a.
Recognition, pages
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:2307.0566 2023b.
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14458-
14467, 2021.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] proovided proper justification is given (c.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction summarize our work, fully in accordance with the details described the remainder the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are explicitly discussed in the paper's conclusion section.
Guidelines:
14

---

The answer NA means that the paper has no limitation while the answer No means that the puper has limitations, but those are not discussed the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. general. empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relics upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed description of all experimental methodology to enable reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data provided or not.
15

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. novel architecture, describing the architecture
For example, if the contribution fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be a way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soure source models, it may be that access to the model is limited in some way (e.g., to registered users), but should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While we do not release our code or model weights, we provide full experi- mental details to enable reproduction of our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guidest httgl///uuutess/oCdeSbbisssssss sembbbiissiooooccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https://nips... cc/
http://aauaimisssoooooooioooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
16

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All such details are thoroughly described in our main paper and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due compute limitations (each training requiring approximately one week on our system) we do not re-run training multiple times to estimate the variance in our results due to the stochastic nature of training.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard emror of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If eтror bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe these required compute resources in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each the individual experimental runs well estimate the total compute.
17

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurs wttprripp.ccppuuupe wctppcccccccccccccccccccccclcllt http://rrppppppppoooo/////aaaaaa
Answer: [Yes]
Justification: Our work does not infringe on any of the items covered in these guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses potential societal impact of work in the conclusion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We cannot release our code data as discussed above, this concern does not apply.
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We explicitly credit models and sources of data used in our paper, along with licensing and terms of use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com /dtassess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
datasets.paperssw
For existing datasets that are re-packaged. both the original license and the license of the derived asset (if it has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does release new code, data, models. Our supplementary material contains videos displaying model inference results, which are fully documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
19

---

Justification:
We do not use crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research
with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMSs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used in this research.
Guidelines:
The answer NA means that the core method development this research does not involve LLMs any important, original, non-standard components.
Please refer our LLM for what should or should not be described.
20

---

A Additional Results
A.1 Video Qualitative Results
Please see the results viewer on our project page for additional results of WildCAT3D inference and applications on validation and test inputs. Videos include examples of vanilla WildCAT3D inference for novel view synthesis, and examples of applications (appearance-coonrollledd generation, in-the-wild interpolation). We also provide the images used as input views. For vanilla inference, results include various hard-coded trajectories (lateral turns, zoom-outs, and NeRF-like circular paths). For each scene in the interpolation results, two interpolations are provided: one using the start view's 's appearance and the other using the end view's appearance throughout. For the appearance conditioned generation application, the images used for appearance embedding injection are also provided.
A.2 Image Qualitative Results
illustrate further examples of our outputs in- and -uttof-distribution NVS bench- marks, showcasing our strong performance at predicting novel views from a single observation.
Figures and 9
A.3 MegaScenes-Only Ablation
In Table 3 compare training end-to-end using data in-the-wild from MegaScenes the only source of multi-view data (rather than including CO3D and Re10K in multi-view training). While this underperforms
full model including these curated sources of multi-view data, still achieves competitive performance overall, indicating that MegaScenes alone may be used to learn strong prior on consistent novel views of scenes despite itself containing inconsistencies between scene views due to appearance variations.
A.4 3D Reconstruction
While the main focus of our work is generating novel 2D views of scene, we also show that these outputs may be used for downstream 3D reconstruction. In particular, we feed the 15 generated novel views of a scene (using u = 16 slots for WildCAT3D) into VGGSIM [Wang et al. 2024]. beed-forward 3D reconstruction pipeline that recovers camera poses and the 3D geometry of the scene, and use reconstructed sparse scene geometry to initialize a 3D Gaussian Splatting representation Kerbl et al. 22033) Examples are illustrated in Figure 10 showing that model's outputs are sufficiently high-quality and consistent to recover underlying 3D scene geometry. Additionally, the recovered camera trajectories generally match the trajectories used for conditional generation (lateral, circular, and straight trajectories respectively in the examples shown).
state-of-the-an
DTU [Jensen al. 20144)
Mip-NeRF 360 [Barron et 2022
Method
PSNR
SSIM t LPIPS 4 FID 1. KID"
PSNR
SSIM LPIPS 1 FID | KID"
WildCAT3D (Ours)
10.77
0.426
0.388
57.32
0.039
14.77
0.445
0.352
42.17
0.050
MS-only
9.679
0,362
0.453
70.42
0.080
13.68
0.405
0.403
47.56
0.063
Re10K
Zhou al.
2018
MegaScenes SSIM
Tung al. 2024
Method
PSNR SSIM 1 LPIPS 4 FID J KID",
PSNR
LPIPS 4 FID J. KID'
WildCAT3D (Ours)
21.58
0.758
0.131
24.70
-0.001
13.92
0.439
0.355
9.871
0.015
MS-only
18.83
0.673
0.188
29.15
0.005
13.30
0.415
0.378
10.03
0.016
Table 3: MegaScenes-only ablation. We compare our full model to using MegaScenes as the only source of multi-view data ("MS-only" above), discussed in Section A.3 KID" indicates KID values multiplied by ten for readability.
21

---

Input
Target Views
GT
Outputs
GT
Outputs
GT
Outputs
GT
Outputs
GT
Outputs
Figure 8: In-distribution qualitative results. Given single input image (left), WildCAT3D can generate consistent novel views from target viewpoints (right). We here show results on test scenes from in-distribution data (MegaScenes, CO3D), OOD results can be found Figure 9 Despite training in-the-wil wild data with appearance variations, produce views with consistent appearances (aspect ratio, global lighting, etc.) by using the appearance embedding from the input view.
A.5 Depth Estimation Error Propogation
As our method relies on an existing depth estimation pipeline (monocular depth estimation aligned to a scene's SfM pointcloud). it may suffer from error propagation when depth is incorrectly estimated. Examples are illustrated in Figure suggesting that more robust depth estimation may further improve results.
22

---

Input
Target Views
Input
Target Views
GT
Outputs
GT
Outputs
GT
Outputs
GT
Outputs
GT
Outputs
EU
Figure 9: Out-of-distribution (OOD) qualitative results. Results similar Figure applied to OOD data sources (DTU, Mip-NeRF 360).
B
Additional Implementation Details
Image Resolution
We train 512 pixel resolution (64 64 latent resolution); images with other aspect ratios resized so the longest edge 512 and padded to square. Such padding is also used by prior work and evaluation benchmarks (c.g. Gao et al. [2024] Tung et al. (2024]); ); unlike square cropping, avoids losing information from the image periphery and allows for generation of images with different aspect ratios in inference. For metric calculations, resize predictions to x56x256 resolution.
B.2 Diffusson Process Details
Following Salimans and Ho 2022 Lin et al. [2024], we adjust noise scheduling to enforce zero terminal SNR and train with velocity prediction and loss in order allow for generation of images with varying overall brightness levels. During inference, we generate images using CFG scale 3,
B.3 Camera Representation Details
Following CAT3D Gao al. 2024], we remove degree ambiguity by transforming all camera poses to be relative to the first, observed pose. During training, normalize camera scale using
23

---

Input
Sample Generated Views
3D Reconstructions (Point Cloud Cameras)
3DGS Novel Rendered Views and Depth Maps
Figure 10: 3D Reconstruction from Generated Views. We show 3D reconstruction results applied WildCAT3D generations, as described Appendix A.4 Reconstructed point clouds and camera positions are shown above, generally matching the expected scene geometry and camera trajectories used for generation. While two generated views are shown for conciseness, 3D reconstruction results are calculated from 15 views. We also show an example of novel views and their normalized depth maps generated using a 3D Gaussian Splatting (3DGS) representation initialized with the sparse reconstruction of the first scene.
the 10th quantile of COLMAP sparse depth visible from the first view, following ZeroNVS Sargent et al. 2023
We calculate Plücker raymaps follows: given camera origin and pixel p (vectors in world coordinates), its raw 6-dimensional coordinates are given by (d, o x d), where d = d is its displacement from the camera origin. As these are homogeneous coordinates describing its associated ray, we unit-normalize by dividing by the scalar factor / d |o x d This provides numerical stability by ensuring all coordinates are bounded (as extreme values may have been introduced into cameras' extrinsic matrices from translation vector scaling mentioned above). These coordinates calculated for each 64 x 64 spatial position provide 6 x 64 x 64 channel coordinates used as input channels.
B.4 Model Architecture
The appearance encoder module has the following architecture: It is made up of alternating convolu- tional layers (filter size 3, same padding) and 2 x 2 max pooling, with filter dimensions 16, 16, 16, 4, respectively. This converts 64 x 64 image VAE latents to 2 x 2-dimensional embeddings, which are finally flattened to an embedding of dimension 8.
In order to be compatible with the dimensionality of added channels concatenated to latents, we add a projection layer (1 x convolution) to reduce the x 64 x (2k: + d + 7) concatenated channels to dimension 64 x 64 x 4 before being input to the LDM's UNet.

---

Input
Warp
Generated View
Target View
Figure 11: Propagation of Depth Estimation Errors. As our method relies existing depth estimation module to calculate warps, errors in depth estimation may propagate to novel views. This is illustrated above, as incorrect warps correspond to errors in depth estimation, which are seen to cause resulting novel generated views to deviate from the correct targets.
Curated Data
In-the-Wii
Method
Synth.* * ACID CO3D Re10K MegaScenes
Zero-1-to-3
x
X
ZeroNVS
MS baseline
V
K
WildCAT3D (Ours)
X
V
MS-only ablation
/
Table 4: Training data sources used. Curated Data refers to fully consistent multiview data obtained from synthetic renderings or heavily curated videos. v filtered for matching time metadata and aspect ratios; MS MegaSceees, 'Objaverse(-XL)
B.5 Training Data
Table 4 shows the sources of multi-view data used to train WildCAT3D, as well as those used in the baseline models we compare (Zero-I 3 Liu al. [023a ZeroNVS (Srooet Sargent al. [2023), MegaScenes baseline NVS Tung et al. (2024). )) As seen in the table, our model uses strictly fewer data
sources for multi-view training (neither using Objaverse (Deitke al., 2023a b) ACID [Liu et al.
2021]), and does not require aggressive filtering of in-the-wild data to avoid views with differing appearances.
Our approach successfully utilizes the full MegaScenes dataset without aggressive filtering, consisting of approximately 430K scenes and 2M images. This is large relative curated multi-view datasets (e.g. CO3D contains 19K videos, with 1.5M total frames). This highlights a key strength of method its ability fully utilize this new, scalable source of data.
Our training data sources are all licensed under permissive licenses: Re10K and Megascenes under CC BY 4.0, and CO3D under CC CC-N- 4.0.
B.6 Training Procedure and Compute Resources
For model training, we use batch size 64 (where each sample in mini-batch is itself set of eight scene views), distributed over 32 NVIDIA A100 GPUs (using approximately 80GB of memory on each) on our internal cluster. The initial CAT3D model initialized from an open-source LDM trained for 200K iterations, followed by 60K WildCAT3D fine-tuning iterations. End-to-end model training takes approximately one week to complete. The datasets used require several terabytes of storage, such as the 3.2 TB used to store the original images from MegaScenes, although this could be
25

---

reduced by only storing images resized to the 512 x 512 resolution used by our model. Preliminary experiments and each our ablations used similar compute resources.
Compared to the leading alternative method - the MegaScenes baseline NVS model our imple- mentation has larger computational footprint during training, requiring several times more GPU memory and days of training. (MegaScenes NVS was trained on 6 NVIDIA A6000 GPUs for 1-2 days.) In particular, our approach is trained generate several high-resolution (512x512) images in parallel, unlike MegaScenes NVS (which is trained to generate one 256x256 image). Our approach could be adjusted for computational efficiency in training by reducing the number of generated views and their resolution.
B.7 Experimental Details
For evaluation benchmarks, apply WildCAT3D with = 8 input slots (matching its training procedure). As benchmarks pair one source view to single unobserved target view, we apply WildCAT3D by grouping target views together that share the same source view. We split these into groups of seven unobserved views, padding with extra duplicated targets as needed to match the number of input slots.
To calculate generative metrics (FID, KID) on MegaScenes, we use random 15K-item subset the test set to make their calculation computationally feasible.
For our interpolation application (generating interpolated views between two views of a scene with differing appearances), we generate a camera trajectory as folllows: We use camera intrinsics from the first view, and interpolated extrinsics. In particular, we linearly interpolate between the camera translation vectors, and use spherical linear interpolation (slerp) to interpolate between the camera rotation matrices.
26


Reviews:
[{"review_id": "3fqgS1HavQ", "reviewer": "Reviewer_TYQ2", "summary": "The paper proposes WildCAT: A multiview diffusion model capable of NVS, that can be trained on in-the-wild image collections while also allowing for coarse appearance control over the generated images. The primary new capability introduced by the work is the ability to train a multiview diffusion model like CAT3D on inconsistent in-the-wild collections such as natural scenes for which plenty of data is available on the web. The paper enables this by proposing two main contributions: The first is an appearance encoder that extracts coarse appearance features from an input image, and the second is warp conditioning mechanism that warps pixels to their expected target locations using monocular depth estimates to better guide the image generation and resolve scale ambiguities in NVS. The core architecture of the paper builds on top an existing multiview diffusion model (CAT3D) and concatenates additional input channels for appearance and warp conditioning. The appearance embedding is a global embedding of an image; obtained through a shallow convolutional encoder applied on the image latent, and is duplicated spatially. The warp conditioning is a spatial latent map obtained by warping the input image to the target view using estimated depth from DepthAnything. Experiments show that with these improvements, a multiview diffusion model capable of NVS and coarse appearance control can be trained on inconsistent image collections.", "questions": "The number of additional channels required by WildCAT seems to be quite a lot more than CAT3D (d channels for the appearance, another k channels for the spatial warp). I’m curious how much this affects the size of the model and the training speed? Is it only the first layer where the capacity increases? Or did the authors increase the overall capacity of the LDM to assimilate the additional information?", "limitations": "There are significant temporal artifacts in the generated NVS and appearance editing results. The visual quality of the results is not very impressive.  However I do recognize and appreciate that this is the first installment of the method which introduces new capabilities and can be improved in future work. The authors address this in their work already.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "*Strengths*\n- The paper demonstrates interesting first results in novel view synthesis and controllable image generation, adding new capabilities to models like CAT3D. Results in the supplemental material for appearance/view interpolation show promise. This work has implications for world generation where the user can control the appearance of the generated scene to some extent. \n\n*Weakness*\n- There is significant flicker in the generated results. This makes me wonder if applying the ideas proposed in this paper over a method like Cat4D, which also has temporal priors, is a more appropriate extension of the state of the art. \n- The proposed appearance and warp conditioning mechanism are reasonably straightforward and are not groundbreaking in terms of novelty.", "was_revised": true, "final_justification": "The authors addressed my concerns in their rebuttal. I continue to hold the opinion that is an interesting first method that promises to unlock web-scale training data for NVS. Given that this is the first installment of such a method, I'm willing to cut some slack for the poor visual quality of the current results. This learnings from this work have the potential to be a very valuable contribution to the community and as such I increase my rating to 5.", "rebuttal": "We thank the reviewer for their constructive feedback and their positive assessment.\n\nRegarding visual quality and temporal artifacts: We acknowledge this as a significant challenge posed by training on in-the-wild data, which contains inconsistencies not found in perfectly curated datasets. In this setting, we significantly improve over previous methods training on such data (Table 1, Fig 3), including the previously SOTA MegaScenes baseline method, which also suffer from this trade-off in order to train on abundant web data capturing full scenes. As the reviewer suggests, future improvements could include video priors (L265; such as CAT4D) or distillation from 3D priors/reconstruction to further enhance temporal consistency. We will discuss this more explicitly as a limitation of our approach and direction for future improvement in our revision.\n\nRegarding model capacity and training efficiency: We clarify that the additional input channels are handled through a lightweight 1×1 convolution projection layer (Appendix L70, Section B.4) that reduces these channels before processing. This adds very few new parameters and does not noticeably slow training or require increasing the overall LDM capacity. We will add a clarification of this important implementation detail.\n\nRegarding novelty: While individual components may seem straightforward, their combination enables a fundamentally new capability: training multiview diffusion models on inconsistent in-the-wild data. This opens access to vast amounts of previously unusable web data, representing a significant step toward scalable NVS systems. The technical challenge was not just developing these components, but making them work together effectively to handle real-world data inconsistencies that break existing methods."}, {"review_id": "RtAjsN658y", "reviewer": "Reviewer_MpEA", "summary": "WildCAT's key innovation is to explicitly model and disentangle appearance from 3D content. It extends the CAT3D framework by introducing two main components: 1) a generalizable Appearance Encoder that learns a low-dimensional representation of an image's global appearance, which is then used as a conditioning signal for the diffusion model, and 2) a Warp Conditioning mechanism that uses a depth-guided warp of the source view to resolve scale ambiguity and improve geometric consistency. This design allows WildCAT to train on inconsistent data while generating novel views that are consistent in both appearance and geometry at inference time. The paper demonstrates state-of-the-art performance on standard NVS benchmarks and showcases novel applications like appearance transfer and text-controlled appearance editing.", "questions": "1. Could you elaborate on the impact of the depth estimation quality on the warp conditioning? Have you experimented with different depth estimators or analyzed how failures in depth (e.g., on reflective surfaces or thin structures) affect the final NVS results?\n2. Regarding the claim of using \"strictly less data sources,\" could you provide a more direct comparison of the total training data volu\n3. The appearance-controlled generation via CLIP retrieval is a very compelling application. Could you discuss the diversity and coverage of the database used for retrieval (MegaDepth)? Does the quality of the text-to-appearance transfer depend heavily on finding a very close match in the database, or can it generalize from moderately similar images?", "limitations": "This is an excellent paper that makes a significant and practical contribution to the field of novel view synthesis. The problem of learning from in-the-wild data is highly relevant, and the proposed WildCAT framework is a novel, elegant, and effective solution. The paper is well-written, the experimental results are strong and comprehensive, and the demonstrated applications are impressive. The work successfully \"unlocks\" a new and valuable data source for training powerful generative NVS models. The weaknesses are minor and relate mostly to requests for additional details and clarifications, which do not detract from the core strength and impact of the contribution.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. The paper tackles a significant real-world problem: how to leverage abundant but \"messy\" internet-scale data for NVS. By moving beyond the reliance on curated datasets, this work opens up a path towards more scalable, diverse, and generalizable NVS models, which is a crucial step for the field.\n2. The proposed solution is both clever and well-motivated. The idea of an appearance encoder that allows the model to \"peek\" at appearance conditions during training is an elegant way to handle inconsistent data. Combining this with warp conditioning to enforce geometric consistency provides a comprehensive solution. The modifications to the classifier-free guidance to handle appearance are also a thoughtful and important detail.\n\nweaknesses：\n1. The warp conditioning module, which is shown to be critical for performance, relies on an external depth estimation model (Depth Anything). The quality and potential biases of this off-the-shelf model could significantly impact WildCAT's performance. The paper would be strengthened by a discussion on the sensitivity of the results to the accuracy of the estimated depth maps. For example, how do errors in depth estimation propagate and affect the final rendered views?\n2. The paper repeatedly claims to train on \"strictly less data sources than prior methods.\" While this may be true in terms of the number of distinct datasets used (e.g., not using Objaverse), it could be misleading. The MegaScenes dataset used for fine-tuning is itself massive. A more precise comparison of the total number of training images or training hours would provide a clearer picture of the data and computational efficiency compared to baselines like MS NVS.\n3. The appearance encoder is a central component, yet its architecture is described simply as a \"shallow convolutional network.\" Given its importance as an \"information bottleneck,\" providing more specific details (e.g., number of layers, filter sizes, activation functions) would improve reproducibility and allow for a better understanding of its capacity and generalization capabilities.", "was_revised": true, "final_justification": "Most questions have been clarified, and i appreciate this paper.", "rebuttal": "We thank the reviewer for their constructive feedback and their positive assessment.\n\nRegarding depth estimation sensitivity: Depth estimation errors can indeed propagate to the warp conditioning, and hence to the final views. Specifically, during our experiments, we have observed cases where incorrect depth (e.g. misplaced foreground objects in one Mip-NeRF360 scene) leads to poor alignment in novel views. We will include this failure case analysis in our revision. Note that we selected DepthAnything as it was a state-of-the-art method at the time of the submission and the same choice made by the MegaScenes baseline. Testing alternative depth estimators was computationally prohibitive given the heavy training requirements, but represents worthwhile future work.\n\nRegarding appearance encoder architecture: We provide detailed specifications in Appendix Section B.4. The design follows standard convolutional principles with gradually decreasing spatial dimensions to create a low-dimensional appearance bottleneck. We will include additional architectural motivation in our revision.\n\nRegarding data comparison and computational efficiency: Firstly, we will clarify that our use of “strictly less data sources” is intended to highlight our ability to leverage the full extent of existing, open web-scale data with less use of curated datasets which may be less scalable or have contested licensing; it is not intended to be a claim regarding sample or compute efficiency (and we will state this explicitly). We demonstrate clear improvements over the MegaScenes baseline, which used only a small, aggressively-filtered subset of available in-the-wild data. Our approach successfully utilizes the full MegaScenes dataset without aggressive filtering, consisting of approximately 430k scenes and over 2M images, showing that our method can effectively handle inconsistent appearance conditions. This is indeed large relative to curated multi-view datasets (e.g. CO3D contains 19k videos, with 1.5M total frames); we view it as a strength of our method that it is able to utilize this new, scalable source of data.  Regarding further compute and dataset details, training time is approximately one week (Appendix L86), and we will add specific comparisons of training images, dataset sizes, and computational requirements versus other methods in our revision.\n\nRegarding CLIP retrieval coverage: We use ~10k images from MegaScenes, providing the same diversity as the full dataset (tourist scenes with varying weather, lighting, and conditions). This approach primarily captures global lighting variations and weather conditions as discussed in Section 4.3. We find that this retrieval system works well in practice to capture such conditions, and we will add more explicit discussion and demonstration of these aspects.\n\nWe will also correct the formatting issues and typos you identified."}, {"review_id": "aLju1xYTHl", "reviewer": "Reviewer_tPri", "summary": "This paper proposes an appearance-aware multi-view diffusion model that can utilize multi-view data of the same scene under different weather conditions for training. Specifically, it explicitly constructs a global appearance condition network to decouple content and appearance. During inference, an image can be designated to provide the appearance condition, enabling the generation of multi-view images with different appearances. The authors validate the proposed method through extensive ablation studies and provide many visualizations in the supplementary materials.", "questions": "When looking at the visualizations in the supplementary materials, I noticed that the generated multi-view images seem to have somewhat high contrast and sharpness, and the overall lighting appears unnatural. Have you explored the possible reasons behind this issue?", "limitations": "The authors have already explained the limitations of their method well in the main paper.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The idea of decoupling appearance and content is very novel and impressive, and the results also validate this approach, meeting expectations.  \n2. Due to this decoupling, the proposed algorithm supports training with multi-view data of different appearances, and also allows specifying the appearance during inference to generate multi-view images with different appearances.\n### Weaknesses\n1. The authors use a fully convolutional network as the appearance model. What are the design principles of this network? For example, do factors such as its depth or number of output channels significantly influence the final results?\n\n2. The authors use an image generator as the backbone of CAT3D. Have they tried using a video diffusion model to see its effect on the results? Additionally, what would happen if the backbone were replaced with a pure DiT structure instead of a Unet structure?", "was_revised": true, "final_justification": "The authors’ proposed approach to decouple appearance and content is both interesting and novel, and the experimental results support their claim. Moreover, this decoupling enables training with multi-view data of different appearances, which is a significant advantage. The authors carefully addressed my concerns in the rebuttal. Although they did not conduct additional experiments due to time constraints, their explanations are reasonable. Therefore, I recommend acceptance.", "rebuttal": "We thank the reviewer for their constructive feedback and their positive assessment.\n\nRegarding appearance encoder design principles: The architecture requires low-dimensional output to serve as an effective information bottleneck, with gradually decreasing spatial dimensions following standard convolutional network design. Due to the substantial computational requirements of retraining the full model, extensive hyperparameter tuning of layer sizes was not feasible. Overall, we opted to design a minimal network in order to avoid additional design choices or parameters that would require testing.  We will provide more explicit discussion of the design rationale in our revision.\n\nRegarding video backbone integration: Computational constraints make this challenging for the current work, but we see this as promising future work as mentioned (L265).\n\nRegarding visual quality: As noted in our response to reviewer TYQ2, the current artifacts represent a trade-off for leveraging abundant web data, with future improvements possible through video priors or 3D distillation techniques. Addressing the reviewer’s point about high contrast and sharpness, we suspect that this relates to the use of classifier-free guidance, which is known to tend towards such artifacts in general [Sadat et al. 2024]. As we discuss (L155-167), this may be exacerbated due to appearance conditioning, and we design a strategy for mitigating this issue. We will explicitly note that this may persist to some degree as a limitation of our method and direction for future improvement."}, {"review_id": "PF7dN2OtpE", "reviewer": "Reviewer_DnZH", "summary": "This paper tries to train the CAT3D models with images captured under different conditions (lighting, occlusions). The key idea is that this pipeline could utilize a large amount of online data and has the potential to achieve a better quality if we fully utilize the online in-the-wild condition-changed multiview images. To achieve this, an additional appearance encoder is applied to let the diffusion process peak on the appearance for a better generation. To address the scale ambiguity of single-view scene generation, the method estimates an additional depth map and applies warping as another constraint in the generation. Experiments demonstrate reasonable performance improvement over the original baseline.", "questions": "1. About the ablation study, there could be an alternative way to describe the scene with some texts and use the texts as the appearance codes. How about the performance of such an alternative baseline, which is more straightforward?\n2. Could we demonstrate 3DGS or NeRF reconstruction on the generated multiview images? Because the original CAT3D allows generating the 3D scene. \n3. An alternative way is to generate the appearance-inconsistent images directly and then reconstructt them with some robust NeRF/GS methods that allow appearance changes among different images.", "limitations": "The paper mentions some limitations, but I think the main limitation is the true scalability of the proposed method. Because no experiments have been conducted to truly improve the performance with online in-the-wild multiview data.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n1. The paper is well written and clear.\n2. The idea of utilizing online in-the-wild multiview images is interesting and novel. This has the potential to further scale up the training data.\n\nWeakness:\n1. The proposed method does not really demonstrate that scaling up with these in-the-wild multiview data could improve the results of CAT3D. Instead, the experiments only show that we could get some reasonable results with the proposed pipeline.\n2. The resulting novel-view-synthesis quality is not so good, with visible strong flickering artifacts in the submitted videos.\n3. The training still involves multiview images under the same lighting and capturing conditions. To prevent the appearance feature vector from containing too strong information about details, the proposed method applies the appearance code of the first frame as input to the network to generate other views of the same condition.", "was_revised": true, "final_justification": "The paper explores an interesting direction, which may inspire other researchers in the field, even though the paper does not demonstrate scalability. I keep my original score with a borderline accept of this paper.", "rebuttal": "We thank the reviewer for their constructive feedback and their positive assessment.\n\nRegarding scalability demonstration: We first note that CAT3D and other recent approaches are mostly trained on object-centric data, with poor applicability to full scenes (L28), which is a central motivation of introducing training on Internet image collections in-the-wild  [Tung et al. 2024]. In this vein, our results show meaningful improvements over the MegaScenes baseline, representing an important first step toward web-scale training on previously unusable data. In particular, the MegaScenes baseline was trained only on a small, aggressively-filtered subset of in-the-wild data, while our key contribution is showing that with appropriate conditioning mechanisms (appearance encoder + warp conditioning) we can successfully train on the full, unfiltered dataset with improved results.\n\nRegarding text-based appearance alternatives: This would be less straightforward since our training procedure learns visual appearance encodings without text supervision. Implementing text-driven appearance control would require developing additional novel components to bridge text and visual appearance spaces. Additionally, we note that our appearance module learns cues such as aspect ratio that are specific to our task, which are not directly reflected in text typically accompanying images. We will include this discussion in our revision. \n\nRegarding 3D reconstruction from generated views, we demonstrate that our method can produce coherent 3D point clouds (Appendix A.4), suggesting potential for downstream 3D reconstruction applications. These can be used to initialize a 3DGS reconstruction, which we will illustrate in our revision. Regarding the suggestion to reconstruct appearance-inconsistent views directly followed by robust volumetric reconstruction, these methods typically require scene-specific tuning and a large number of input images for successful reconstruction (more than the sparse views our method outputs in a single pass). Our method performs inference in a generic fashion on novel scenes and directly outputs mutually-consistent views without requiring additional post-processing."}]

Author rebuttal:

Meta-review: This paper proposes WildCAT, which extends a multiview diffusion model like CAT3D to enable training on inconsistent, in-the-wild image collections while supporting coarse appearance control for novel view synthesis. Its key innovations include (1) a generalizable Appearance Encoder that disentangles global appearance from 3D content, and (2) a Warp Conditioning mechanism using monocular depth to resolve scale ambiguities and improve geometric consistency. Strengths include: (1) novel decoupling of appearance and 3D geometry, enabling training on in-the-wild data; (2) the proposed appearance encoder and warp conditioning that improve view consistency; and (3) state-of-the-art performance on NVS with flexible appearance control, advancing scalable 3D content generation. 

Reviewers noted that while the empirical results are compelling, the technical innovations are incremental, and the quality of generated data has room for improvement. Despite these limitations, all reviewers and the AC agree that the paper’s contributions outweigh its weaknesses. We recommend acceptance and encourage the authors to address all reviewers' comments in their manuscript revisions.
Decision: Accept (poster)