Title: Generative Perception of Shape and Material from Differential Motion
Venue: NeurIPS 2025
Accepted: yes
Keywords: generative perception

diffusion models

mid-level vision

perceptual ambiguity

shape reconstruction

material estimation
Abstract: Perceiving the shape and material of an object from a single image is inherently ambiguous, especially when lighting is unknown and unconstrained. Despite this, humans can often disentangle shape and material, and when they are uncertain, they often move their head slightly or rotate the object to help resolve the ambiguities. Inspired by this behavior, we introduce a novel conditional denoising-diffusion model that generates samples of shape-and-material maps from a short video of an object undergoing differential motions. Our parameter-efficient architecture allows training directly in pixel-space, and it generates many disentangled attributes of an object simultaneously. Trained on a modest number of synthetic object-motion videos with supervision on shape and material, the model exhibits compelling emergent behavior: For static observations, it produces diverse, multimodal predictions of plausible shape-and-material maps that capture the inherent ambiguities; and when objects move, the distributions converge to more accurate explanations. The model also produces high-quality shape-and-material estimates for less ambiguous, real-world objects. 
By moving beyond single-view to continuous motion observations, and by using generative perception to capture visual ambiguities, our work suggests ways to improve visual reasoning in physically-embodied systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a generative model for the estimation of shape and material parameters from a single image or from a short video sequence. The method relies on a 3D diffusion transformer model, which can diffuse the material properties in a generative fashion (multiple possible outputs from the same input). The results of the paper show that the method is less uncertain about its predictions once a short video sequence is provided, in contrast to using a single image as input. The method is evaluated on synthetic and real-world data, and compared against previous methods and baselines, showing the increased effectiveness of the method with respect to previous work.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n- This paper tackles a very important problem in computer graphics and vision, and does it in an ambitious way. I believe these findings can spark further work in the literature. \n- The paper is generally well written (with some exceptions), it is very well motivated and it is easy to follow. \n- The proposed method is sound, and makes sense in the context of the problem. Learning 3D object representations with transformers under few-shot scenarios is a really difficult task and the proposed diffusion model make sense and is smart. Data generation and augmentation is also done in a smart way. \n- The paper provides code, which should enhance reproducilibility. \n- Supplementary material provides additional videos and results, which enhance the understanding of the paper, as still images cannot fully convey the domain tbis method is operating on.\n- The paper is somewhat well evaluated, with appropriate ablation studies. \n\n\nWeaknesses:\n- The method is evaluated on very synthetic data, well prepared for the model to disambiguate easily between object and background. There is a single object in each scene. This is a shared limitation of related projects in the literature, however, it is worth acknowledging, as this type of data is not common in the real world and therefore limit the scope of the \"physically-embodided systems\" the paper targets. \n- The method is exclusively studied on opaque objects, with some to little metallicness on them and with a very simple reflectance model. While this is somewhat fine in the context of the literature, the fact that phenomena like transparency, anisotropy, subsurface scattering or translucency are not studied, strongly limit the scope upon which this model is learning to do general-purpose perception of objects. \n- It would have been interesting to study the inheret uncertainty of capturing material properties from single images or from few-shot imagery, as in UMat: Uncertainty-Aware Single Image High Resolution Material Capture (CVPR) or Uncertainty for SVBRDF Acquisition using Frequency Analysis (SIGGRAPH). This line of research is very relevant for this paper and should be analyzed in the related work section. I believe that the paper overstates its contribution in the uncertainty-awareness side, as it does not evaluate this part of the problem, and ignores related work on it.\n- The metrics used for this paper should be expanded. More perceptual metrics must be used to evaluate this method, including: LPIPS, FLIP, and ideally more render-aware metrics, as in Perceptual quality of BRDF approximations: dataset and metrics (EG). \n- The paper relies on specific motions for it to work. The impact of this is not clear, and it may be the case that t will not generalize properly to new motions, or to moving light sources. This is an important limitation.\n- I think the paper sometimes overuses forward references (eg in Line 35, there is a reference to Figure 7a). This reduces the quality of the reading experience.  \n- The choice of model design would benefit from additional explanation. \n- It seems like the model struggles in the single-image example in very specific ways: The model either places most of the variation on the geometry, or on the albedo, leaving the other somewhat constant (Fig 7a). This is unexpected behaviour and creates ill-behaved probability distributions, which maybe be less than ideal for uncertainty quantification. Can the authors elaborate on this?\n\nQUESTIONS:\n- What is the impact of the input resolution in the output? Can this model be used on higher resolution inputs?\n- What motivated the choice of the Disney BRDF? Would other BRDFs work worse? \n- Could the authors provide more insights on why the loss weighting coefficients were chosen in this way? My experience is that these inverse rendering algorithms are usually really sensitive to these weightings, and I am curious about these weights, as they are somewhat different to other weights presented in the literature. \n- Could the model be more \"active\" in its exploration? Eg, choose how to rotate the camera, thee objects or the light source. This would would better emulate how humans explore material appearance.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to predict the graphical parameters (for BSDF) and geometries (normal) from input RGB signals in a generative way. The authors propose to disentangle the ambiguity between geometry and appearance by including multi-views from differential motions, and achieve great performances in selected experiments.\n\nSTRENGTHS AND WEAKNESSES:\nI really like the idea proposed by this paper, that the shape and the material disentanglement should be tackled in a multi-view way. I conclude the strengths of this paper as follows:\n1. this paper proposes to use multiple views from differential object motions to eliminate the ambiguity between object shapes and materials.\n2. They use a unified framework to predict the shape and the material. Instead working as two separate branchs respectively, these two tasks benefits each other. \n3. Well-presented paper with every statement supported by proper citations and very clear explanations, making readers easy to follow and understand.\n\nAs for the weaknesses, I will list the following:\n1. As shown in the limitation section, the motion considered in this paper is simple. Although explicitly stated, this weakness is still unignorable. Since we cannot guarantee similar observations would still hold in more complex motion settings. \n2. The framework asserts that without reconstructing the environment light will lead to a more modular and flexible decomposition (line78-79), but this assertion (actually a strong foundation for the implementation of this work) is not well supported by experiments.\n\n\nOverall this is a good paper in my opinion. But since I am not primarily working on this field, so these weaknesses may not be that so important for this field, and also I will refer to other reviewers’ opinion to adjust my understanding.\n\nQUESTIONS:\nApart from the weaknesses, I also have the following questions. Note that some questions are not aiming to challenge the quality of the paper, but just for better understanding the potential and working scheme of the method, so I don’t list them as weaknesses. \n1. motion related:\n    1. what is exactly the definition of ‘differential object motion’ in this paper? Is only the motion caused by camera movement considered?\n    2. will the model work for deformable objects (could be slightly deformable)?\n    3. will the model work for static object but moving light?\n2. Can the authors show some results of directly working on complex scenes with multiple objects? This results may help us understand whether the mode tend to rely on global features (rely on the overall shape of objects) or local patterns. If latter, the model should work kinda reasonably for complex scenes.\n3. Can the authors show the diversity change of 10 stochastic predictions from one case, of static vs moving?\n4. I recommend the authors include some relighting results based on the decomposed materials and shapes, which can show the flexibility of the proposed methods for the downstream task and can also make the paper more attractive.\n\nLIMITATIONS:\nThe authors sufficiently discuss the limitations in the paper, and I agree with them.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a conditional video diffusion model that performs joint estimation of object shape and material from short RGB video clips capturing differential motion. Unlike prior works that focus on single images or isolated attributes, this paper tackles the ambiguity inherent in inferring both shape and material by training a conditional generative model that outputs pixelwise surface normals and material maps, conditioned on RGB frames. To overcome the high computational requirements of modeling high-resolution outputs, the method uses a hybrid architecture: full spatio-temporal attention is applied only at lower resolutions, while higher resolutions use convolutional blocks with separate local neighborhood and spatial attention modules. The paper argues that motion helps disambiguate appearance cues. The model is trained on a synthetic dataset of 100k video clips covering approximately 1100 3D models under diverse textures and lighting. Experiments demonstrate plausible shape and albedo recovery, multi-modal predictions on ambiguous inputs, and modest improvements over single-frame baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Exceptionally good writing, figures, and presentation.\n- Novel problem setting: Using short videos for conditioning shape and material estimation is novel and addresses ambiguities that single-view methods struggle with.\n- Interesting architecture: The hybrid design (global transformer at low resolution, local convolution plus attention at high resolution, channel mixing for inter-attribute communication) is well-motivated and technically sound.\n- Pixel-space diffusion: Performing diffusion in image space is a reasonable design choice to preserve fine details compared to latent diffusion pipelines (e.g. Marigold)\n- Clear qualitative results: The paper includes meaningful visualizations showing how motion resolves ambiguities (e.g. shiny vs. painted textures) and how this results in multiple diverse samples.\n- The paper and supplementary demonstrate careful ablations and explanations, including the impact of motion, channel mixing, and temporal consistency.\n\n**Weaknesses:**\n- Generalization. The method is trained from scratch on the dataset of 1100 3D models. This is in contrast to methods like Marigold, which are finetuned form rgb image diffusion, and are not limited in generalization to real-world datasets.\n- Marginal gains vs. Marigold on shape: From the numbers in Table 1, the improvement over Marigold or StableNormal for normal estimation appears modest. \n- No baseline for albedo: The paper claims better albedo estimation but does not clarify whether comparable models like Marigold or RgbX could also be extended to report the same metric under similar settings; most importantly, Marigold already supports albedo estimation.\n- Limited real-world validation: Although the paper shows some real-world photos, these are too few. There is no demonstration of real captured videos to show whether the differential motion cues generalize beyond synthetic motion. This is important given the first concern regarding limited generalization due to training on synthetic dataset.\n- Unfair comparison protocol: For stochastic methods, the paper averages the top 3 of 10 samples for its model but uses the mean prediction for the IID baseline. additionally reporting the same protocol (e.g. best-of-10 or top-3) would be fairer.\n\n**Minor:**\nLines 37–41 claim that most recent methods are non-generative and over-smooth ambiguities. This feels overstated as several recent works do address stochastic outputs; the claim could be softened.\n\nIn summary, the main concern is the limited comparison to existing baselines, real-world generalization and potential unfairness in the experimental protocol. If the authors can clarify the comparison details, report or explain concerns regarding albedo baselines, and add more real-world results, this would make the contribution stronger and i would then recommend acceptance.\n\nQUESTIONS:\n- Albedo estimation: Marigold reportedly can estimate albedo too. Why is this not reported or compared directly?\n- Baseline video input: Is it possible to adapt single-frame models to accept multiple frames (e.g. by simple frame stacking) to provide a direct baseline for video conditioning? Qualitative figures for this would also help.\n- Real videos: Could the authors show at least a few more real short videos with object motion to demonstrate robustness?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel conditional denoising-diffusion model for generating shape-and-material maps from short input videos of objects undergoing differential motions. The model, named U-ViT3D-Mixer, is designed to handle the ambiguity in perceiving shape and material from a single image by leveraging motion cues. It can produce diverse, multimodal predictions for ambiguous static observations and converge to more accurate explanations when objects move. The model is trained on synthetic object-motion videos and exhibits compelling emergent properties, such as generating plausible multimodal samples for ambiguous inputs and effectively utilizing differential object motion to resolve perceptual ambiguities. The paper also demonstrates the model's performance on real-world objects, showing high-quality shape-and-material estimates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper introduces a new method for joint shape and material perception using a conditional video diffusion model, which is a fresh perspective in the field of computer vision.\n\n2.\tFigures such as Figure 3 (benchmarking against baselines) and Figure 6 (disambiguation via motion, PCA of sample clusters) offer strong evidence of the approach’s claims, with informative visual interpretations.\n\n3.\tThe model is benchmarked quantitatively and qualitatively against strong baselines for shape and albedo estimation on challenging datasets (Table 1, Figure 3), and extensive ablations (Table 2, Figure 5) illustrate the necessity of key architectural choices.\n\n\nWeaknesses:\n\n1.While results on synthetic and some real images are promising, the model and experiments focus exclusively on isolated objects with controlled backgrounds and restricted motion types. The generalizability to multi-object scenes, unknown backgrounds, or real-world video with complex movement is not demonstrated.\n\n2.The evaluation protocol for ambiguous cases is largely qualitative, relying on visualizations or sample diversity. While these are compelling, there is little systematic, quantitative assessment of how well the sampled distributions align with human uncertainty, nor with established perceptual ambiguity benchmarks apart from visual comparisons. This leaves some doubt as to how reliably the model’s uncertainty calibration matches true perceptual ambiguities.\n\n3.The quantitative benchmarks (Table 1, Table 2) lack error bars, confidence intervals, or significance testing. The justification is that ambiguity is intrinsic, but this omission limits the rigor and interpretability of reported improvements and comparisons to baselines. For instance, gains on DiliGENT shape benchmarks (Table 1) are small and would benefit from significance estimates.\n\n4.Training the model requires significant computational resources, taking around 5 days on 4 H100 GPUs, which might be a barrier for some researchers.\n\n5.The synthetic dataset, though large and diverse, may not fully cover the complexity of real-world bidirectional reflectance distribution functions (BRDFs) or complicated indoor/outdoor lighting. Although Section 4 makes an effort in randomizing environment maps and textures, real-world scenes may involve more challenging or unmodeled conditions not validated here.\n\n6.The main baselines in Table 1 focus on single-image models or those specializing in either shape or material. There is limited direct comparison to models that address joint estimation or leverage motion—even if prior art is scarce, this could be more explicitly discussed as a limitation or open problem.\n\n7.The model is designed for isolated objects, and extending it to handle multi-object scenes with natural backgrounds would require additional training and reasoning about grouping and compositions.\n\n8.While efficient in parameters for the design, the paper does not provide a detailed analysis of inference/training speed, GPU/memory requirements, or direct comparisons of parameter count and runtime to established baselines, which is increasingly important in applied settings.\n\nQUESTIONS:\n1.How would the approach generalize to camera motion, translation, or longer or non-rigid motion sequences? Have you performed any pilot studies or experiments on more diverse video data?\n\n2.Can the model be extended to handle more complex lighting conditions, such as those with multiple light sources or non-uniform lighting?\n\n3.Would the authors consider adding error bars, confidence intervals, or repeated trials to the key quantitative results in Table 1 and 2 to strengthen claims of empirical improvement? This would clarify if improvements are significant and reproducible.\n\n\n4.How does the model perform on objects with highly specular or transparent materials, which are known to be challenging for shape and material estimation?\n\n5.What are the main failure or degradation modes when the model is used with cluttered backgrounds, occlusions, or novel, unmodeled materials/lighting?\n\n\n6.What are the specific reasons for the model's improved performance on albedo estimation compared to existing models?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}