Title: WildCAT3D: Appearance-Aware Multi-View Diffusion in the Wild
Venue: NeurIPS 2025
Accepted: yes
Keywords: novel view synthesis

diffusion models

computer vision

computer graphics
Abstract: Despite recent advances in sparse novel view synthesis (NVS) applied to object-centric scenes, scene-level NVS remains a challenge. A central issue is the lack of available clean multi-view training data, beyond manually curated datasets with limited diversity, camera variation, or licensing issues. On the other hand, an abundance of diverse and permissively-licensed data exists in the wild, consisting of scenes with varying appearances (illuminations, transient occlusions, etc.) from sources such as tourist photos. To this end, we present WildCAT3D, a framework for generating novel views of scenes learned from diverse 2D scene image data cap tured in the wild. We unlock training on these data sources by explicitly modeling global appearance conditions in images, extending the state-of-the-art multi-view diffusion paradigm to learn from scene views of varying appearances. Our trained model generalizes to new scenes at inference time, enabling the generation of multiple consistent novel views. WildCAT3D provides state-of-the-art results on single-view NVS in object- and scene-level settings, while training on strictly fewer data sources than prior methods. Additionally, it enables novel applications by providing global appearance control during generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tries to train the CAT3D models with images captured under different conditions (lighting, occlusions). The key idea is that this pipeline could utilize a large amount of online data and has the potential to achieve a better quality if we fully utilize the online in-the-wild condition-changed multiview images. To achieve this, an additional appearance encoder is applied to let the diffusion process peak on the appearance for a better generation. To address the scale ambiguity of single-view scene generation, the method estimates an additional depth map and applies warping as another constraint in the generation. Experiments demonstrate reasonable performance improvement over the original baseline.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is well written and clear.\n2. The idea of utilizing online in-the-wild multiview images is interesting and novel. This has the potential to further scale up the training data.\n\nWeakness:\n1. The proposed method does not really demonstrate that scaling up with these in-the-wild multiview data could improve the results of CAT3D. Instead, the experiments only show that we could get some reasonable results with the proposed pipeline.\n2. The resulting novel-view-synthesis quality is not so good, with visible strong flickering artifacts in the submitted videos.\n3. The training still involves multiview images under the same lighting and capturing conditions. To prevent the appearance feature vector from containing too strong information about details, the proposed method applies the appearance code of the first frame as input to the network to generate other views of the same condition.\n\nQUESTIONS:\n1. About the ablation study, there could be an alternative way to describe the scene with some texts and use the texts as the appearance codes. How about the performance of such an alternative baseline, which is more straightforward?\n2. Could we demonstrate 3DGS or NeRF reconstruction on the generated multiview images? Because the original CAT3D allows generating the 3D scene. \n3. An alternative way is to generate the appearance-inconsistent images directly and then reconstructt them with some robust NeRF/GS methods that allow appearance changes among different images.\n\nLIMITATIONS:\nThe paper mentions some limitations, but I think the main limitation is the true scalability of the proposed method. Because no experiments have been conducted to truly improve the performance with online in-the-wild multiview data.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an appearance-aware multi-view diffusion model that can utilize multi-view data of the same scene under different weather conditions for training. Specifically, it explicitly constructs a global appearance condition network to decouple content and appearance. During inference, an image can be designated to provide the appearance condition, enabling the generation of multi-view images with different appearances. The authors validate the proposed method through extensive ablation studies and provide many visualizations in the supplementary materials.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The idea of decoupling appearance and content is very novel and impressive, and the results also validate this approach, meeting expectations.  \n2. Due to this decoupling, the proposed algorithm supports training with multi-view data of different appearances, and also allows specifying the appearance during inference to generate multi-view images with different appearances.\n### Weaknesses\n1. The authors use a fully convolutional network as the appearance model. What are the design principles of this network? For example, do factors such as its depth or number of output channels significantly influence the final results?\n\n2. The authors use an image generator as the backbone of CAT3D. Have they tried using a video diffusion model to see its effect on the results? Additionally, what would happen if the backbone were replaced with a pure DiT structure instead of a Unet structure?\n\nQUESTIONS:\nWhen looking at the visualizations in the supplementary materials, I noticed that the generated multi-view images seem to have somewhat high contrast and sharpness, and the overall lighting appears unnatural. Have you explored the possible reasons behind this issue?\n\nLIMITATIONS:\nThe authors have already explained the limitations of their method well in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWildCAT's key innovation is to explicitly model and disentangle appearance from 3D content. It extends the CAT3D framework by introducing two main components: 1) a generalizable Appearance Encoder that learns a low-dimensional representation of an image's global appearance, which is then used as a conditioning signal for the diffusion model, and 2) a Warp Conditioning mechanism that uses a depth-guided warp of the source view to resolve scale ambiguity and improve geometric consistency. This design allows WildCAT to train on inconsistent data while generating novel views that are consistent in both appearance and geometry at inference time. The paper demonstrates state-of-the-art performance on standard NVS benchmarks and showcases novel applications like appearance transfer and text-controlled appearance editing.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. The paper tackles a significant real-world problem: how to leverage abundant but \"messy\" internet-scale data for NVS. By moving beyond the reliance on curated datasets, this work opens up a path towards more scalable, diverse, and generalizable NVS models, which is a crucial step for the field.\n2. The proposed solution is both clever and well-motivated. The idea of an appearance encoder that allows the model to \"peek\" at appearance conditions during training is an elegant way to handle inconsistent data. Combining this with warp conditioning to enforce geometric consistency provides a comprehensive solution. The modifications to the classifier-free guidance to handle appearance are also a thoughtful and important detail.\n\nweaknesses：\n1. The warp conditioning module, which is shown to be critical for performance, relies on an external depth estimation model (Depth Anything). The quality and potential biases of this off-the-shelf model could significantly impact WildCAT's performance. The paper would be strengthened by a discussion on the sensitivity of the results to the accuracy of the estimated depth maps. For example, how do errors in depth estimation propagate and affect the final rendered views?\n2. The paper repeatedly claims to train on \"strictly less data sources than prior methods.\" While this may be true in terms of the number of distinct datasets used (e.g., not using Objaverse), it could be misleading. The MegaScenes dataset used for fine-tuning is itself massive. A more precise comparison of the total number of training images or training hours would provide a clearer picture of the data and computational efficiency compared to baselines like MS NVS.\n3. The appearance encoder is a central component, yet its architecture is described simply as a \"shallow convolutional network.\" Given its importance as an \"information bottleneck,\" providing more specific details (e.g., number of layers, filter sizes, activation functions) would improve reproducibility and allow for a better understanding of its capacity and generalization capabilities.\n\nQUESTIONS:\n1. Could you elaborate on the impact of the depth estimation quality on the warp conditioning? Have you experimented with different depth estimators or analyzed how failures in depth (e.g., on reflective surfaces or thin structures) affect the final NVS results?\n2. Regarding the claim of using \"strictly less data sources,\" could you provide a more direct comparison of the total training data volu\n3. The appearance-controlled generation via CLIP retrieval is a very compelling application. Could you discuss the diversity and coverage of the database used for retrieval (MegaDepth)? Does the quality of the text-to-appearance transfer depend heavily on finding a very close match in the database, or can it generalize from moderately similar images?\n\nLIMITATIONS:\nThis is an excellent paper that makes a significant and practical contribution to the field of novel view synthesis. The problem of learning from in-the-wild data is highly relevant, and the proposed WildCAT framework is a novel, elegant, and effective solution. The paper is well-written, the experimental results are strong and comprehensive, and the demonstrated applications are impressive. The work successfully \"unlocks\" a new and valuable data source for training powerful generative NVS models. The weaknesses are minor and relate mostly to requests for additional details and clarifications, which do not detract from the core strength and impact of the contribution.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes WildCAT: A multiview diffusion model capable of NVS, that can be trained on in-the-wild image collections while also allowing for coarse appearance control over the generated images. The primary new capability introduced by the work is the ability to train a multiview diffusion model like CAT3D on inconsistent in-the-wild collections such as natural scenes for which plenty of data is available on the web. The paper enables this by proposing two main contributions: The first is an appearance encoder that extracts coarse appearance features from an input image, and the second is warp conditioning mechanism that warps pixels to their expected target locations using monocular depth estimates to better guide the image generation and resolve scale ambiguities in NVS. The core architecture of the paper builds on top an existing multiview diffusion model (CAT3D) and concatenates additional input channels for appearance and warp conditioning. The appearance embedding is a global embedding of an image; obtained through a shallow convolutional encoder applied on the image latent, and is duplicated spatially. The warp conditioning is a spatial latent map obtained by warping the input image to the target view using estimated depth from DepthAnything. Experiments show that with these improvements, a multiview diffusion model capable of NVS and coarse appearance control can be trained on inconsistent image collections.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n- The paper demonstrates interesting first results in novel view synthesis and controllable image generation, adding new capabilities to models like CAT3D. Results in the supplemental material for appearance/view interpolation show promise. This work has implications for world generation where the user can control the appearance of the generated scene to some extent. \n\n*Weakness*\n- There is significant flicker in the generated results. This makes me wonder if applying the ideas proposed in this paper over a method like Cat4D, which also has temporal priors, is a more appropriate extension of the state of the art. \n- The proposed appearance and warp conditioning mechanism are reasonably straightforward and are not groundbreaking in terms of novelty.\n\nQUESTIONS:\nThe number of additional channels required by WildCAT seems to be quite a lot more than CAT3D (d channels for the appearance, another k channels for the spatial warp). I’m curious how much this affects the size of the model and the training speed? Is it only the first layer where the capacity increases? Or did the authors increase the overall capacity of the LDM to assimilate the additional information?\n\nLIMITATIONS:\nThere are significant temporal artifacts in the generated NVS and appearance editing results. The visual quality of the results is not very impressive.  However I do recognize and appreciate that this is the first installment of the method which introduces new capabilities and can be improved in future work. The authors address this in their work already.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}