Generative Perception of Shape and Material from Differential Motion
Authors: Xinran Han
Ko Nishino
Todd Zickler
Venue: NeurIPS 2025 poster ()
Keywords: generative perception
diffusion models
mid-level vision
perceptual ambiguity
shape reconstruction
material estimation
Primary area: applications
TL;DR: We introduce a generative perception model that, given a few frames of an object undergoing motion, produces diverse and plausible interpretations of its shape and material—including texture and reflectance—through feedforward stochastic sampling.

Abstract: Perceiving the shape and material of an object from a single image is inherently ambiguous, especially when lighting is unknown and unconstrained. Despite this, humans can often disentangle shape and material, and when they are uncertain, they often move their head slightly or rotate the object to help resolve the ambiguities. Inspired by this behavior, we introduce a novel conditional denoising-diffusion model that generates samples of shape-and-material maps from a short video of an object undergoing differential motions. Our parameter-efficient architecture allows training directly in pixel-space, and it generates many disentangled attributes of an object simultaneously. Trained on a modest number of synthetic object-motion videos with supervision on shape and material, the model exhibits compelling emergent behavior: For static observations, it produces diverse, multimodal predictions of plausible shape-and-material maps that capture the inherent ambiguities; and when objects move, the distributions converge to more accurate explanations. The model also produces high-quality shape-and-material estimates for less ambiguous, real-world objects. 
By moving beyond single-view to continuous motion observations, and by using generative perception to capture visual ambiguities, our work suggests ways to improve visual reasoning in physically-embodied systems.

Generative Perception of Shape and Material from Differential Motion
Xinran Nicole Han Harvard University xinranhan@g. harvard.
Ko Nishino Kyoto University
Todd Zickler
Harvard University
konCi ac.
zicklerdseas, hwrvarrdddc.. edu
Abstract
Perceiving the shape and material of an object from single image inherently ambiguous, especially when lighting is unknown and unconstrained. Despite this, humans can often disentangle shape and material, and when they are uncertain, they often move their head slightly or rotate the object to help resolve the ambiguities. Inspired by this behavior, we introduce a novel conditional denoising-diffuse diffusion model that generates samples of shape-and material maps from short video of object undergoing differential motions. Our parameter-efficient architecture allows training directly in pixel-space, and generates many disentangled attributes of an object simultaneously. Trained on a modest number of synthetic object-motion videos with supervision shape and material, the model exhibits compelling emer- gent behavior: For static observations, produces diverse, multimodal predictions of plausible shape-and- -nmmmerraaatti maps that capture the inherent ambiguities; and when objects move, the distributions converge to more accurate explanations. The model also produces high-quality mhate-and-material estimates for less ambiguous, real-world objects. By moving beyond single-view to continuous motion observa- tions, and by using generative perception capture visual ambiguities, our work suggests ways to improve visual reasoning in physicallyeeboodied systems
Introduction
single image an object is inherently ambiguous. The image can be mimicked by a planar surface with paint, for example, or curved mirror under suitable lighting. There are many possibilities, and they arise from intertwined roles of shape and material in image formation. We argue that vision systems should reason about shape and material jointly, and that they should embrace and model their ambiguitics, so that additional cues like motion or touch can be triggered for disambiguation. This will help embodied AI systems deal with very practical problems, such distinguishing wet floor from a glossy one, or distinguishing a painted visual cliff from an actual cliff.
Humans naturally resolve much of the uncertainty through motion. Imagine moving a shiny textured object under natural lighting: glossy highlights sweep across its surface, immediately revealing its material and geometric structure. This interplay of material, surface, and motion provides far more insight than a single static observation. Just as young children explore their environment by manipulating objects, computational vision systems can benefit from observing differential object motion even across a few video frames. These observations motivate a perception framework that (i) explicitly models shape-materia ambiguity rather than collapsing to a single estimate, and (ii) leverages object motion as a natural cue for reducing ambiguity when motion is available.
In contrast to these principles, recent learning- ceaeeng-bbsedd approaches in computer vision typically focus on inferring either shape or material, but not both, and they are often optimized to predict a single
Project page: https://xrhan, www.ergitttuu....... wwtpbi/fd//ffffoiioooomm
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

10. Unambiguous static photo
(R Continuous object motion
Shupe
Albedo
Differential Motion
shape
albedo
shape
albedo
relighting
Metillic
Ronghness
Specular
Grib, Ambiguous statk photo
Observer
CORVeCE sample
conccive simple
planee sample
Figure 1: Left: We train generative perception model jointly infer shape and materials using synthetic tnree-frame videos of objects undergoing differential motions. Right: Our model (i) generalizes captured real-world photos; (ii) leverages continuous observations to disentangle complex objects, and (iii) shows an emergent ability to provide multiple hypotheses for ambiguous static images, such convex, concave and planar 'postcard' explanations.
"best" output even for tasks that are fundamentally ambiguous. Examples include the estimation of monocular shape (e.g., depth 33 59
normals [43 60 8 and intrinsic texture color [[]]. In addition to focusing on a single attribute, such models tend to ignore the ambiguity inherent in the visual world, treating uncertainty noise to be suppressed rather than something to be modeled.
Instead, we advocate a generative perception approach, where a vision system generates diverse samples of shapes and materials that explain the input observation(s). When the input is single image, the output samples should reflect the full range of plausible interpretations, expressing the fundamental ambiguities of the scene. Enyo and Nishino (18). Koesis al. [36], and Han et al. 233 have previously explored this idea for single images and particular attributes, demonstrating generative perception, respectively, of illumination and reflectance; materials; and shape. Here, argue that generative perception should not be limited to a single view, or to a single object attribute. It should incorporate moving observations when available, and it should jointly disentangle all of shape, texture, and reflectance.
To realize this goal, we introduce a conditional video diffusion model for generative perception that jointly infers shape and material from both static and moving objects. Our denoising diffusion architecture, called U-ViT3D-Mixer operates directly in 256 x 256 pixel space and uses spatio- temporal attention and cross-channel mixing to simultancous reason about surface normals, diffuse texture and reflectance. At coarse scales it employs global attention to capture object-scale constraints, and at fine scales it employs parameter-efficient neighborhood attention to capture local space-time constraints shape and material.
We train our model from scratch on short synthetic videos of moving and static objects, and find that it generalizes to captured images and videos while exhibiting several desirable perceptual capabilities: (i) it achieves competitive results on existing static-image shape and material benchmarks; (ii) it exhibits an emergent ability to generate plausible multimodal samples when the input is ambiguous, such as the classic convex/concave ambiguity or the trivial postcard solution (see Figure D and crucially, (iii) it makes effective use of differential object motion to resolve perceptual ambiguities, improving prediction accuracy when additional motion frames are available.
Generative perception is a framework for understanding how vision systems can operate under inherent ambiguity. Our work advances this framework by extending from static images to motion videos. and by estimating shape and material together. We hope this opens new directions for cognitive modeling, and for embodied AI systems that aim to emulate human perception.
2 Related Work
We study novel problem: generative disentanglement of object shape and material from short videos. This has not been directly addressed in prior work, but it relates to several research threads.
Intrinsic image decomposition and inverse rendering. Intrinsic image decomposition aims to separate an image into texture (albedo), diffuse shading, and sometimes additional reflectance properties such metallic-ner ness and roughness, all without inferring shape [7] 36 IT. Inverse

---

rendering extends this goal by recovering the full scene properties properties-includingg shape, materials, and illumination- that are necessary for physically accurate re-rendering [4 42 6T]. Our work
differs from these by estimating shape and materials without explicit illumination, thereby focusing on all of the scene attributes that are intrinsic to an object.
Recently, Zeng et al. [61] introduced RGB-X, which finetunes latent diffusion models for single- image prediction of surface normals, material properties, and illumination, while also supporting lext-conditioned synthesis. The model differs from ours by outputting each attribute independently via a prompt selector, without modeling their interactions. Similarly, DindursionRendeer [40] finetunes Stable Video Diffusion [9] for physical attributes prediction given video observations. The model conditions on a domain embedding (e.g. "normals", "roughness") infer one target attribute at a time, In contrast, our model integrates shape and material reasoning within single backbone, allowing their representations to inform each other.
Diffusion models for conditional prediction. Generative modeling with diffusion models has gained increasing recent attention [5] 49]. Many recent works have leveraged priors from pre-trained latent diffusion models for dense prediction tasks such as estimating depth 33 26).
materials [36] [1]. These methods often suppress output diversity by 36 or regularizing variance [60], overlooking the inherent ambiguity in these tasks and leading to over-smoothed predictions that fail to capture multimodal outcomes. In contrast, recent generative perception models embrace ambiguity by using diffusion models to produce diverse samples of illumination and reflectance [88] or of shape (23 from a single image.
normals 43 46 60 [
averaging multiple samples 33
Here we extend this idea to consider motion and to infer shape and materials together.
Shape from differential motion. Prior mathematical analyses have used differential equations to study the reconstruction of shape from small motions under idcalized conditions, such as directional lighting and Lambertian or mirror-like materials [44 5 57 Belhumeur al. [6] show that for Lambertian shapes with unknown lighting there are continuous stretches and tilts (the generalized bas-relief ambiguity) that cannot be resolved by small motions. When lighting and reflectance (BRDF) are both unknown, Chandraker et al. 12] show that three observations of differential motions around three distinct axes are needed correctly reconstruct the shape along a characteristic surface curve. Thus, from a mathematical standpoint, perceptual ambiguity persists even when observing differential motion, which highlights the need for generative models that can represent the distribution of possibilities. A learning-base approach like ours also has the benefit of applying to a broader range of lights and materials that would be very difficult to characterize analytically.
3 Methodology
We consider a rigid object undergoing small motion with respect to a stationary viewer and lighting environment. The observer is represented by orthographic camera along the 2-axis in world coordinates. Given a sequence of temporally continuous RGB image observations c E R Fx XXXXXXXXX x3 with F frames, we aim infer time-varying (F-frame) shape-and-material fields X (n, M {pa, Y, ps}} from c. We use bold symbols to denote variables for multiples frames. Figure| depicts overall approach.
3.1 Shape and Material Representations
We represent shape by the viewer-centric surface normal field n E Fx The normal vector the backprojection of each image point (u, v) in single frame has unit length and is denoted by n(u, v). We represent material attributes M using the point-wise surface albedo (i.e., diffuse color/texture) pa(u, v). roughness r(u,v) metallic-ness 7(,, v) and specularity p. (,(u,), v). Each parameter is time and space varying map with the same dimension F x H x W that has either three channels (pz) or one channel (r, 7, Ps) and values in the range [0, 1]. These parameters are chosen from a subset of those of an analytical reflectance model- the Disney principled BSDF model [10]. Together they determine the bidirectional reflectance distribution function (BRDF)
fr(ws, wo: (pa, Ps. (,,r,))) = (1 - Pd (fai (w), wo) + fretro (wi, woi r))
(1)
+ Sapeo (wi, Wai Pas Psi r, T,7)-
3

---

Conditional Video
Local 3D Blocks
U-VIT3D-
frament
Predicted Noise
Locol 3D
Skip Connection
Local 3D Bodds
ResSlock (Comv2D)
Trareformer
Channal Mixer
Layeers
4
Spatiel
Noisy Attributes
concatenation
latent l of three frames
Temporal Attention
Channel Mixer
Reflectance
Figure Our parameter-efficient denoising network, U-ViT3D-Mixen takes in a channel-wise concatenation of conditional video frames and noisy chape-and-materiiy frames. At high spatial resolutions, it uses efficient local 3D blocks (middle) with decoupled spatial, temporal, and channel- wise interactions. At lower spatial resolutions, it uses global transformer layers with full 3D attention.
In an idealized setting without ambient occlusion, this BRDF determines the reflected radiance at each surface point by the physically- rendering equation 32
(u, U, w,wo))
(L)) n(u, v))dwa
where wi and wi are the incident and outgoing directions of light in the local coordinate frame of the hemisphere centered at normal n(u, v), and L, is the incident directional illumination. Unlike inverse rendering methods (e.g.. [42)), we do not infer lighting and solely focus recovering normals n and material properties M. We provide the full details of the rendering formulation in the appendix.
3.2 Generative Perception from Differential Motion
We design a conditional video diffusion model that can exploit the temporal continuity of differential motion and that supports simultaneous stochastic sampling of shape, texture, and reflectance. Our model also processes still images by treating them as statie videos.
3.2.1 Joint Inference with Diffusion Model
Our model has the unique feature of predicting shape and materials jointly using single backbone. This design choice is motivated by our preliminary probing experiment (see Appendix (A.2). where model trained to predict albedo can be probed using DPT-style probes in [17] to produce reasonable estimates surface normals. This suggests strong cross-attribute interaction and motivates a unified architecture. Our approach is very different from existing models that require tine-tuning multiple pre-trained diffusion models for shape or materials (37), or that use a text prompt to switch between
output attribute time [1 40
We use conditional diffusion probabilistic model 22 (DDPM) to generate saape-and-materri attributes q(X (e) by iterative denoising. To train the model we define 'forward process' that adds Gaussian noise to clean shape-and-material map x::
2, = 04 + ore, where e A(
We use a continuous noise schedule where zn is close clean data and 2 close to Gaussian noise, and we use variance-preserving 28 noise schedule which implies of + of 1.
We neural network with weights 0 approximate q( ((X c) by tractable distribution pe X (c). The network learns to estimate the noise in intermediate timesteps E (0, 1) for all frames. During training, each conditioning frame ef E e is associated with the same noise level o, and with different, randomly sampled Gaussian noise e!. We train the network by minimizing the prediction error averaged over all frames:
L(O)= =
Bu-uio.,,e co to
F
JEF

---

During sampling, given the conditional video frames we perform the reverse diffusion process using DDIM [5] beginning with sample 2 from the standard Gaussian distribution. Given sample 2, at intermediate timestep I, we predict the clean sample à with the noise estimate En from the denoiser neural network and project it to a lower noise level 8 < t. using
2g = at + C, t). where ã = (2 - -o,Ea)/a-
(5)
As O, obtain the shape-and- -nd--mterial prediction z for f F. The sampling process of DDIM can be made deterministic and thus the randomness only comes from the initial noise #1.
3.2.2
Spatio-temporal and Multi-attribute Interactions
Unlike prior work that fine-tunes large pretrained diffusion models for estimating either shape 33 43 19 46, 60] or materials 761 we train our model to estimate both from scratch using a modest set of synthetic videos. This choice is motivated by findings from Han et al. [23], which demonstrate that pretrained models with strong priors can lack the flexibility to provide multimodal output distributions in the presence of ambiguity. Our training strategy, along with an improved noise schedule [35 30 and our parameter- sarameter-efficient efficient architecture, allows us to investigate whether multimodal, generative perception can emerge naturally from learning, without being affected by the potential biases in large pretrained models.
As shown in Figure 2 our model takes in set of input video frames cl and concatenates it channel- wise with set of noisy per-frame shape-and and-materia maps z.f = [nf M/ f E F. This yields 12 channels per frame as the input to the denoising network. We introduce the following network components that act across spatial, temporal, and attribute dimensions to achieve joint shape-and- material perception from observations of differential motion.
Spatio-temporal interaction. Scaling pixel-space diffusion models to operate at higher resolutions (e.g., 256x256) introduces significant computational challenges. For instance, the diffusion UNet 28 and DiT 488 operate with full self-attention in each block, which becomes infeasible at higher resolutions due to a quadratic growth in token count.
To address this, we adopt hybrid design inspired by U-VIT 29 30 and Hourglass Diffusion
Transformer [14. At high spatial resolutions, we use ResNet-style convolution blocks [27 followed
by separate local spatial attention (via shift-invariant neighborhood attention [25] and temporal
attention layers. At coarser resolutions (e.g., 32x32, 16x16), we apply transformer blocks 48 with
global spatio-temporal attention and use 3D Rotary Positional Embeddings (55]
This spatoal-temporal interaction higher resolutions is particularly suited to exploiting the local space-time cues for shape, material and motion that are known to exist mathematically [12). Our ablations show that it notably improves performance for both shape and material estimation.
Multi-attribute mixer. While convolution and local attention modules extract strong spatio-temporal features, they do not promote interaction across channels representing different physical attributes. To address this, we incorporate channel mixer blocks inspired by MLP-Mixer [56] after early convolutional and local attention layers. These blocks enable explicit inter-attribute communication. accelerating convergence and improving predictions.
Since our model performs multi-attribute denoising across shape, albedo and reflectance, we apply attribute-speciie specific loss weighting coefficients Ashape Aalbedo Areflectance- We find that upweighting shape and albedo (Aahape Aatbedo 0.4) relative to metallic-ness, specular and roughness parameters (Areflectance 0.2) leads to faster convergence and better overall performance.
In summary, our architecture extends existing pixel-space image diffusion architectures [50 29
29
30 [4] video-conditioned, antri-et setting, enabling high-resolution generation with rich spatial, temporal, and cross- gross-attribute interactions.
4
Experiments
To the best of our knowledge, no existing dataset provides high-quality annotations of physical object attributes under object motion. We therefore construct synthetic dataset for training using the Mitsuba3 renderer (3T) with custom integrators to extract ground-truth shape and material. We source []] obtaining 1 1100 artist-designed models. To enhance diversity,
objects from Adobe 3D Assets
5

---

Iaput
GroWizard
StaleNoomal
Ours
Reference
Input
IDD
RGB-X
Ours
Reference
RODS
Figure 3: Qualitative comparisons of estimated shape and albedo/texture from static scene. For fair comparison of ecale-invariant albedos, we visualize the scaled albedo from each model closest the ground truth in the masked object region. Our model achieves comparable shape predictions with existing baselines that specializes at shape prediction, and achieves better quality in albedo estimation. Our model also produces greater spatial detail.
Table 1: Quantitative evaluation shape and normal estimation tasks.
Shape Estimation (DiLiGent)
Albedo Estimation (MIT-Intrinsic)
Model
Mean AE ())
Median AE (L)
SSIM (t)
RMSE (L)
PSNR (t)
Marigold-E2E-FI [46
17.74
13.86
StahleNormal 60
18.21
14.59
DSINE
20.29
16.17
GeoWizard
19
24.62
21.55
IID
36
0.61
0.13
18. 19
ColorfulShading
11
0.72
0.10
20.56
Marigold-Albed
34
0.72
0.11
19.86
RGB-X 61
34,99
32.10
0.71
0.11
20.82
Ours
18.03
12.31
0.80
0.08
23.14
each shape is rendered under two material settings: (i) its original artist-designed materials, and (ii) a randomly sampled texture from of 400 open-source assets [13 spatially-unifone reflectance from randomly sampled metallic, roughness, and specular coefficients.
15 58) combined with
All scenes are lit with one of 200 hundred randomly-sampe HDR environment maps collected from [1], with horizontal and vertical environment flips for augmentation. To simulate object motion, place the object centroid in the image center, randomly sample 3D rotation for initial pose, and apply motion rotation about the vertical image axis with a random angle sampled from (1-80- 22) 0%, [2 8}] with probability (0.4, 0.2, 0.4). This means that 20% of training samples are static scenes containing no motion. We generate 45 short video clips (5 frames each) for each object, and we split them into pairs of consecutive 3-frame clips (F" 3) for training. This results in a dataset of approximately 100K vidco-attribute pairs.
We train our model using the AdamW optimizer 45 Training requires roughly five days using four H100 GPUs. We follow the sigmoid loss reweighting strategy [35] 30) and adopt shifted-cosine continuous noise schedule [30) [53]. Complete hyperparameter settings for architecture provided in the appendix. During inference, we use DDIM sampling [52] with 50 steps, which takes about 2.7 seconds per input video single A100 GPU.
4.1 Static Object Experiments
Our model processes single images static objects by using F image copies as input, so we can assess its shape and material performance using existing single-image datasets. We evaluate our model in two ways: (i) similarity to ground truth for everyday objects that are less ambiguous, and (ii) its ability to generate plausible multimodal solutions when the input more ambiguous.

---

Figure 4: Our model exhibits multimodal shape perception ambiguous visual stimuli presented in Han et al. 23 despite only being trained on everyday objects and without specific data augmentation.
Note that all comparison models this section except RGB-X X 66 are specialized for either shape or material estimation. Also, we cannot compare to the previous generative perception approaches because they require either known shape or textureless diffuse objects.
of [1] 23
We evaluate shape accuracy using the DiliGENT dataset [51] This dataset contains photos taken under point light sources, originally designed for the photometric stereo task. Notably, this lighting is quite different from the environment lighting in our training set. For each of the 10 objects, randomly sample images and compute the mean and median angular surface normal errors within the masked object region. We compare with oate-offtheeart shape prediction methods: StableNormal (60), Marigold- -2EEFFF [46], GeoWizard [19], and DSINE B All except DSINE are fine-tuned from pre-trained image diffusion models. For stochastic methods (ours, Stable GeoWizard), report the average of the three lowest error values from among 10 predictions.
We assess albedo/texture estimation using the MIT Intrinsic Image Dataset P2 which provides object photos under laboratory lighting and ground truth albedo maps. We compare to deterministic method, ColoinulShading [11). and inffusion-based methods Intrinsic Image Diffusion (IID) (36], Marigold-Albes [3] and RGB-X X [6I] Following prior work [T] we use seale-in invariant SSIM, RMSE, and PSNR for evaluation. For the IID model, follow their recommendation by using the mean of 10 samples. For RGB-X, Malioold-Alhedo and our model, we report the average of the three lowest error values from 10 predictions.
Table shows quantitative results, and Figure shows qualitative comparisons with other diffusion- based methods. For shape estimation, our model performs competitively with existing methods that specializes shape prediction. Note that our model estimates the material in addition to the shape and is trained solely on synthetic data. For albedo estimation, our model outperforms existing models by a large margin. We attribute this improvement to (i) the use of pixel-space diffusion, which helps preserve fine details, and (ii) synthetic training dataset with diverse and challenging objects using randomized texture and reflectance.
Overall, our model matches or exceeds all of the specialized shape or material models existing benchmarks. The only other model that estimates both RGB-X, but we find that it often struggles with complex shapes and provides less competitive performance across diverse objects.
Perceptual ambiguity. Han et al. 2 introduced awareness of computational models, particularly for the convex/concave ambiguity. When evaluated on these images and as shown Figures [ (ii) and [ our model produces multimodal predictions, similar to human perception. Interestingly, due to training on textured rather than homogeneous objects, our model also occasionally exhibits 'postcard' interpretation, perceiving a flat surface with painted texture. We find the joint estimation of shape and material further enhances interpretability through disentanglement and can offer insight into the model's perceptual hypotheses.
of test images designed to probe the ambiguity-
4.2 Moving Object Experiments
We compare three ablated variants of our model: (i) U-VIT3D (base) is a bascline with convolutional layers at high resolutions and transformer blocks in the bottleneck; (ii) + Local Pemtio-Temporll Attention replaces one ResNet block per resolution level with local spatial attention and temporal attention module; and (iii + Multi-Attribute Mixer further adds lightweight channel mixers to each local 3D block.
We conduct ablation studies on a held-out synthetic evaluation set containing objects undergoing differential motion. These are generated using the same pipeline as our training data but with distinct objects to ensure generalization. We evaluate performance under both moving and static observations. For each video, draw 10 samples and report the mean of the top-3 based on joint ranking of
7

---

Frame
Relighting Exxamples
Shape
Albedo
Figure 5: Shape and material estimates from our full model. For each three- frame test video, we show one representative frame. The first four columns use randomly sampled materials, and the last four use original materials. Note that some specular objects are quite challenging: we demonstrate in the supplementary videos how motion aids disambiguation. On the right we show relighting examples using the estimated shape, albedo, and reflectance parameters under directional lighting.
Table Ablation study on shape and albedo estimation using our differential-motte evaluation set. Reported metrics are computed over mean of the top 3 of 10 stochastic samples per video. Model variants are compared under both moving and static observation conditions.
With Motion
Without Motion
Model Variant
# Params
Mean AE
SSIM
Mean AE +
SSIM f
U-ViT3D (base)
95.7M
18.757
0.778
22,471
0.760
+ Local Spatio-Temporal Attention
97.4M
14.555
0.825
18.957
0.811
+ Aulti-Attribtee Mixer
100.3M
11.262
0.832
16.886
0,823
shape (Mean Angular Error (MAE)) and albedo (SSIM). This combined metric helps capture overall performance instead of focusing solely on shape or materials.
Table shows that each module leads notable performance gains with minimal added parameters. Specifically, we find that using spatial neighborhood attention is crucial for better shape estimation. The base model with only convolutions tends produce over-smoothed results on complex inputs and often fails to reason about the shape of highly specular objects. Our model also shows better performance when it observes objects under motion instead being static, highlighting its ability to leverage motion cues. Figure 5) visualizes predictions from the full model on motion inputs.
Perceptual ambiguity. To assess model's ability adapt its hypotheses in the presence of motion, we apply it to the perceptual demonstrations from Hartung and Kersten 24 These are videos of moving objects rendered with either (a) uniform shiny material that reflects the environment, (b) spatially-varying matte material representing painted texture. The two scenarios are indistinguishable in the absence of motion, because they can produce individual frames that identical.
We probe our model's interpretations by visualizing its distribution of generated albedo samples, because the shiny interpretation implies spatially-uniff does not. The left of Figure shows an embedding of 100 albedo predictions when seeing a shiny teapot that is either static moving. The albedo samples diverse in the static case, containing both shiny and matte interpretations, and they converge to the shiny explanation when there motion.
albedo whereas the matte interpretation
The right of Figure 6 shows albedo samples for single video frame that is common across two distinet videos of the same shape rendered as shiny or matte. The frame is ambiguous isolation, so the interpretation is heavily influenced by temporal motion. We find that our model clearly distinguishes between the shiny and matte explanations, with samples forming two distinct clusters.
These experiments demonstrate that our model can effectively leverage differential motion to refine its material prediction, similar to human observers 244 Note that our dataset does not contain any objects with painted textures that are purposefully designed to look like reflected environments. Nonetheless, learns to include that possibility among its predictions for static objects, and it learns to either confirm or deny that possibility in the presence of motion.
We encourage viewing of videos httpss///errstenlabb paych. wwtp//emss//mooionnnsufeccc

---

Static
Albedo Samples
Motion (Dituse) Motion Sminyt
Albedo Samples
Motlen Bllng
CC
Figure 6: Motion disambiguation results for the Shiny or Matte Ambiguity 24 Each plot shows a PCA embedding of 100 albedo samples corresponding to the frame (inset) that is common between two distinct input videos. Left: When a shiny teapot moves, the albedo samples converge to a subser of the ones produced for the static teapot. Right: The albedo samples for a matte-rendered motion video are clearly separated from those of a shiny-rendered video. Note that spatially-uniform albedos form tighter clusters in PCA space, while highly textured ones exhibit greater variation.
4.3 Real-world Examples
Input
Sample
Sample 2
shows additional examples with online images and real-world videos. For each input, we segment the objects and mask the back- ground before passing it to the model.
Figure 7 and g
Figure iillustatte how our model jointly inter- prets shape and albedo from static inputs. In the top row, the carpet uses distorted textures and shading to mimic a 'visual cliff". In the bottom row, our model generalizes to an artist's painting, producing either a flat 'postcard' inter- pretation shaded 3D one. When the shape is perceived as three-dimensional, the predicted albedo appears more uniform.
Figure 7: Multiple shape and material hypothe- ses on an ambiguous online image. A cat on an illusion carpet painted to resemble hole. When in- terpreted as a hole (Sample 1), the predicted albedo is brighter in the center region (yellow box) than when interpreted as a plane (Sample 2).
Figure compares our model to RGB-X [6] and Reffurr [0 on real-world videos of various objects under natural outdoor or indoor lighting. We test them in two settings (a) moving objects, same as our training paradigm and (b) moving camera, which is out-of-distribution for our model. For the latter we use the catured images in the Stanford-ORB dataset [38], by feeding our model sets of three images from nearby camera views. In both scenarios, we show that our model produces better visual quality for both shape and albedo, especially for shiny and metallic objects.
To test the overall generalization of our model unseen motions, we include additional quantitative results in the appendix AA66)am and A.7
find that model generalizes quite well to out-of-distribution inputs, including to ambiguous perceptual stimuli (Fig. 44 and to captured videos that contain multiple objects or novel types of motions (Fig. [8. We attribute this to: (i) our of local, ihift-invariant attention at high resolutions, which helps avoid overfitting to global object features; and (ii) our training set, which includes 3D assets composed of multiple objects and self-occluding object-parts.
Summary. Experimentally
Limitations. Our model currently focuses on restricted classes of motions and materials. While it can handle multiple objects undergoing the same rigid motion, it does not apply to scenes that contain deforming objects, or that contain multiple distinct motions. Nor does it apply to objects that exhibit translucency transparency. Another limitation of our model is its reliance on objects being pre- pre-segmenttted from the background. Extending the model to perform perceptual grouping in addition to inferring object attributes is an important future direction.
5 Conclusion
Inspired by the interdependence of shape and material perception, and by the human ability to resolve ambiguities through motion, we introduce a generative perception approach that produces combined samples of shape, texture, and reflectance which explain a short video of differential object motion. Our model recovers complete descriptions of an object's 's shape and materials while capturing and modeling inherent ambiguities. We show that it successfully maintains a distribution

---

Input
RGB-X
DiffusionRemderer
Ours
Captured Videos (Object Motion)
Stanford-ORB (Camera Motion)
Figure 8: Outputs from captured videos of real world objects. Our model effectively uses differential motion to produce higher-quality estimations of shape and albedo compared to RGB-X and Diffu- sionRenderer.
of plausible interpretations under ambiguous static observations, and that it naturally converges to more concentrated and accurate hypotheses in the presence of motion. By introducing an effective conditional video diffusion model that operates directly in pixel space, we also provide potential new backbone for space-time ppace-time generative perception. We hope this inspires future research in ambiguity- aware perception and dynamic vision for embodied systems.
Acknowledgments
We thank Jianbo Shi reviewing the manuscript. also thank Kohei Yamashita for guidance about the data rendering pipeline and Boyuan Chen for discussion about video diffusion modeling. This work was supported part by the NSF cooperative agreement PHY-2019786 (an NSF AI Institute, http://iaifi..rg and by JSPS 21H04893 and JST JPMJAP2305.
10

---

References
[1] Adobe Stock. https://stock.. http://sttccc.dbb.coom
[2] HDRIs: poly haven. cictpa://oolyhavers horrh
[3] Gwangbin Bae and Andrew Davison. Rethinking inductive biases surface normal estimationn In IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), 2024.
[4] Jonathan Barron and Jitendra Malik. Shape, illumination, reflectance from shading. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 37(8):1670- 111116777-667,... 2014.
[5] Ronen Basri and Darya Frolova. A two-frame theory motion, lighting and shape. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-7. IEEE.
[6] Peter Belhumeur, David Kriegman, and Alan L Yuille. The bas-relief ambiguity. International Journal of Computer Vision, 35(1):33-44, 1999.
[7] Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM Transactions Graphics (TOG), 33( 2014.
[8] Yanrui Bin, Wenbo Hu, Haoyuan Wang, Xinya Chen, and Bing Wang. Normalcratter: Learnnng temporally consistent normals from video diffusion priors. arXiv preprint arXiv:2504.11227 2025.
19] Andreas Blattmann, Tim Dockhor, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffussin models to large datasets. arXiv preprint arXiv:2311.15127, 2023.
[10] Brent Burley. Physically- based Shading at Disney. In siggraph, 2012.
[11] Chris Careaga and Yağız Aksoy, Colorful diffuse intrinsic image decomposition in the wild. ACM Transactions Graphics (TOG), 43(6):1 12.0:1--12220000 2024.
[12] Manmohan Chandraker. The information available to a moving observer on shape with unknown, isotropic
BRDFs. IEEE Transactions Pattern Analysis and Machine Intelligence, 38(7):1283-1297, 2015.
[13] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Puttern Recognition (CVPR). 2014.
[14] Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z Kaplan, and Enrico Shippole. Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers. In Forty-first International Conference on Machine Learning, 2024.
[15] Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousscau, Single-image svbrdf capture with rendering-aware deep network. ACM Transactions on Gruphics (ToG), (7(5):1-15, 2018.
[16] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in neural information processing xystems, 34:8780-8794 2021.
[17] Mohamed El Banani, Amit Raj. Kevis- Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein. Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D awareness of visual foundation models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 21795-21806, 2024.
[18] Yuto Enyo and Ko Nishino. Diffusion reflectance map: Single-image stochastic inverse rendering of illumination reflectance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
[19] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang. Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3D geometry estimation from a single image. In European Conference on Computer Vision, pages 241-258. Springer, 2024.
[20] Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin P. Murphy, and Tim Salimans. Diffusion meets flow matching: Two sides of the same coin. 2024. URL htllittts///ddfuuiioooo low.gi
thub.io/
[21] Marc-André Gardner. Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and -rançFiançoi Lalonde. Learning to predict indoor illumination from single image. arXiv preprint 4704 00090, 2017.

---

[22] Roger Grosse, Micah K Johnson, Edward Н Adelson, and William T Freeman. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In 2009 IEEE 12th International Conference on Computer Vision, pages 2335-2342. IEEE, 2009.
[23] Xinran Han, Todd Zickler, and Ko Nishino. Multistable shape from shading emerges from patch diffusion. Advances in Neural Information Processing Systems, 37:34686-33 33711, 2024.
[24] Bruce Hartung and Daniel Kersten. Distinguishing shiny from matte. Journal of Vision, 2(7):511-551 2002.
[25] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6185-6194,
2023.
[26] Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaigiang Zhou, Hongbo Zhang, Bingbing Liu, and Ying-Cong Chen. Lotus: Diffusion-based visual foundation model for high-quality dense prediction. arXiv preprint arXiv:2409. 18124, 2024.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbcel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 6851, 2020.
[29] Emiel Hoogehoom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213-13232. PMLR, 2023.
[30] Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Salimans. Sim- pler diffusion (sid2): 1.5 fid on imagenet512 with pixel-space diffusion. arXiv preprint 2rXiv:2410.19324, 2024.
[31] Wenzel Jakob, Sébastien Speierer. Nicolas Roussel, Merlin Nimier-David Delio Vicini, Tizian Zeltner. Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. Mitsuba 3 renderer, 2022. https://mitsuba- renderer.org.
[32] James T. Kajiya. The Rendering Equation. SIGGRAPH, 20(4):143- 150, August 1986.
[33] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9492 9502, 2024.
[34] Bingxin Ke, Kevin Qu, Tianfu Wang, Nando Metzger, Shengyu Huang, Bo Li. Anton Obukhov. and Konrad Schindler. Marigold: Affordable adaptation of diffusion- based image generators for image analysis. arXiv preprint arXiv:2505.093: 2025.
[35] Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives the elbo with simple data augmentation. Advances Neural Information Processing Systems, 36:65484-65566, 2023.
[36] Peter Kocsis, Vineent Sitzmann, and Matthias NicBner. Intrinsic image diffusion for indoor single-view material estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5198-5208, 2024.
[37] Peter Koesis, Lukas Höllein, and Matthias Nicßner. Intrinsix: High-quality PBR generation using image priors. arXiv preprint, 2025.
[38] Zhengfei Kuang, Yunzhi Zhang, Hong-Xing Yu, Samir Agarwala, Elliott Wu, Jiajun Wu, al. Stanford- ORB: a real-world 3d object inverse rendering benchmark. Advances in Neural Information Processing Systems, 36:46938 46957, 2023.
[39] Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. SyncDiffusion: Coherent montage via synchronized joint diffusionns. Advances in Neural Information Processing Syxtems, 36, 2024.
[40] Ruofan Liang. Zan Gojeic, Huan Ling, Jacob Munkberg. Jon Hasselgren, Chih-Hao Lin, Jun Gao. Alexander Keller, Nandita Vijaykumar, Sanja Fidler, et al. Diffusion renderer: Neural inverse and forward rendering with video diffuusion models. In Proceedings of the Computer Vision and Patiern Recugnition Conference, pages 26069 20000,, 2025.
[41] Stephen Lombardi and Ko Nishino. Reflectance and Natural Illumination from a Single Image. In ECCV, pages $82-595. Springer, 2012.
12

---

[42] Stephen Lombardi and Ko Nishino. Reflectance and Illmiiaaiion Recovery in the Wild. IEEE Trans. on Pattern Analysis and Machine Intelligence, 38(1):129-14,, Jan. 2016.
[43] Xiaoxiao Long. Yuan- Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Thcobalt, ct al. Wonder3D: Single image to 3D using cross-domain diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition pages 9970-99800 9980,9980,.000 2024.
[44] M. S. Monguet-Higging Reflection and refraction at a random moving surface. i. pattern and paths of specular points. 50(9):838 844, Publisher: Optical Society of America.
[45] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICIR), 2019.
[46] Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidt, Daan de Geus, Alexander Hermans, and Bastian Leibe. Fine-tuning image-conditional diffusion models is easier than you think. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025.
[47] Geoffrey Oxholm and Ko Nishino. Shape and Reflectance Estimation in the Wild. IEEE Trans. on Pattern Analysis and Machine Intelligence, 38(2):77-3338 Feb. 2016.
[48] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resoltiion image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[50] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisse 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings. Part III 18, pages 234-241. Springer, 2015.
EideventioonMICCA.. 2015:
[51] Boxin Shi, Zhe Wu, Zhipeng Mo, Dinglong Duan, Sai-Kit Yeung, and Ping Tan, A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3707-3716, 2016.
[52] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International Conference on Learning Representations, 2021.
[53] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du Russ Tedrake, and Vincent Sitzmann. History- guided video diffusion. arXiv preprint arXiv:2502.0676. 2025.
[54] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[55] Jianlin Su. Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 127:137 2024.
[56] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. MLP-Mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24211-24272, 2021.
[57] Yuriy Vasilyev, Todd Zickler, Steven Gortler, and Ohad Ben-Shahar. Shape from specular flow: Is one flow enough? In CVPR 201 pages 2561-2568. IEEE
[58] Giuseppe Vecchio and Valentin Deschaintre. Matsynth: A modern PBR materials datasct. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[59] Lihe Yang, Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024.
[60] Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. StahleNormal: Reducing diffusion variance for stable and sharp normal, ACM Transactions on Graphics (TOG), 43 2024.
[61] Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold- Geoffrooy Yiwei Hu, Fujun Luan, Ling- Qi Yan, and Miloš Hašan. RGB+X: Image decomposition and synthesis using material- and lighting-awan aware diffusion models. In ACM SIGGRAPH 2024 Conference Papers. SIGGRAPH '24, New York, NY, USA, 2024. Association for Computing Machinery, ISBN 97984000052500 doi: 10. 10.344/6641599.3666 13.5444445..
13

---

A Technical Appendices and Supplementary Material
A.1 Additional Related Work 3D Reconstruction
Many recent learning-based multi-view scene reconstruction methods infer point clouds from im- ages [7T 69 70 68]. These models are designed for deterministic reconstruction instead modeling ambiguity, and they do not estimate material properties, so often have the surrounding illumination baked into their output point colors. Although some view-synthesis techniques based on NeRFs [67] and Gaussian splatting [63] 62] can disentangle reflectance and lighting, they require many viewpoints and per-scene optimization. In contrast, our method bed-forward. and it leverages differential motion in short videos to successfully resolve texture-lighting ambiguity without the need for many wide-baseline images and explicit camera pose estimation.
generate 3D assets from text or conditioning image, including the generation of object parts that are not visible in the input. These models are often trained on artist-desinned, cartoon-style 3D assets instead of photorealistic ones, and their output is typically mesh plus a texture that has highlights and other illumination effects baked in.
Some other recent studies 44 64 71
A.2 Probing experiment on normal estimation from albedo prediction
We conduct probing experiment with a low-resolution (64x64) diffusion UNet that is trained to infer the object albedo given a conditional image. We are interested in whether the model learns useful representations for predicting object surface normals. To do this, inspired by prior work [17). we first pre-train the albedo prediction UNet and then insert probes similar the DPT decoder [66] using multiscale convolutions at intermediate layers. We train the lightweight probes to estimate shape from latent features using a small dataset of 10K ground-trut pairs for five epochs. At test time, we directly read out from the inserted probes for surface normal estimates.
As shown in Figure 9 the model trained for albedo prediction indeed produce plausible shape estimates through probing. As a result, we hypothesize that building unified framework for both shape and material estimation leverage shared representation and enable more computationally efficient architecture.
Image
Reference
Surface Normal
Image
Reference
Surface Normal
Figure 9: A model trained to predict surface albedo learns representation useful for estimating surface shapes. We show two examples with the ground truth shape (middle) and the readout shape from simple convolutional probes (right).
A.3 Ablation Studies
In Figure 10a
compare the shape estimates from our base model (U-VIT3D) with those from full model (U- ViT3D-Mixer). The base model uses only convolutions at high resolutions, so it tends produce over-simplified geometry estimates. On the other hand, our full U-ViT3D-Mixer model captures finer details robustly, even for shiny objects.
In Figure 10b, we show the benefits of leveraging motion cues for material estimation. Texture estimation is highly ambiguous when the object is static: the appearance could come from painted texture, reflected environment, mixture of the two. Figure shows samples of our model's textures from ambiguous static images, and shows how object motion allows the model to separate texture accurately.
A.4 Extension using Temporal Consistency Guidance
Our model generates short clips (F 3 frames), but can be used to create longer sequences by simultancously generating multiple clips and using inference-time guidance (c.g.. [39 23 to enforce
14

---

base model
full model
wio motion
with motion
(a)
(h)
Figure 10: Left: Ablation of model components. The base U-ViT3D model, i.e., local attention and channel mixing layers ablated from our full model, produces smoothed results and struggles with specular surfaces due to complex rnter-reflections. Right: Ablation of motion from input images. When the shiny objects are rendered as being static, the model can sometimes provide inaccurate estimates due to the inherent ambiguity between texture and illumination. Once the model observes the object undergoing motion, this ambiguity is resolved and the model produces more accurate albedo estimates.
temporal consistency between them. During inference, we apply a reconstruction loss (e.g.. MSE loss) overlapping frame across adjacent temporal windows, encouraging the denoised predictions å to remain consistent at that frame. We find that this can provide temporally consistent shapes and materials over long time horizons, without the need for modifications additional training. On the project page, we compare our long-horizon results to StableNormal [60], which shows substantially more temporal flickering and inconsistency.
A.5 Additional evaluation results for IID 36
In the experiment section Table [ use the mean prediction for the IID baseline following the authors' recommended setting because it achieves the best performance in their original paper [36] The evaluation results with the alternative top-3 out of 10 evaluation protocol are below in Table They are consistently worse than the mean prediction, consistent with the original paper's 's claims.
Table 3: IID 36 performance under different evaluation protocols.
IID Albedo Estimation
SSIM
RMSE
PSNR
LPIPS
Mean prediction out of 10 samples Best 3 of 10 samples
0.61
0.13
18.19
0.16
0.44
0.15
17.01
0.20
A.6 Quantitative Results using the Stanford-ORB dataset
In the main paper we show qualitative results using images from the Stanford-ORB benchmark (38). where we feed our model sets of three images from nearby camera views. The effective "motion" in this experiment is very different from our training data because (i) the camera moves instead of the object; and (ii) the camera positions are separated by wide, irregular baselines. Despite this, we find that our model achieves high accuracy, especially in comparison to the two strongest comparison models-MMrrgoldd-
[46] for shape and ColorfulShading [I]] for albedo-using the same input images. Tables 4 and show results for six of the benchmark's scenes, where we see competitive performance for shape and superior performance for albedo. This is an example of our model generalizing beyond the horizontal object motion used for training.
15

---

Table 4: Stanford ORB real-world objects: shape inference results
Model
Shape (Median AE) +
Blocks
Cactus
Car
Cup
Grogu
Teapot
Marigold-c2e-ft 46
10.19
14.59
13.69
12.
13.27
13.25
Ours
11.22
19.50
13.86
7.85
10.84
15.02
Table 5: Stanford ORB real-world objects: albedo inference results
Model
Albedo (SSIM /PSNR ?)
Blocks
Cactus
Car
Cup
Grogu
Теapot
ColorfulShading
0.76/ 24.69
0.66 / 25.58
0.64 23.01
0.84 24.83
0.61 / 19.87
Ours
0.74/ 25.28
0.78 26.23
0.70 23.68
0.88 24.97
0.86 24.05
0.79 24.12
A.7
Generalization to random rotation and translation
Some examples in the main paper already suggest that model generalize to non-horizontal object motions. For instance, in the left video of Figure the teapot rotates vertically, orthogonal to the motion seen during training. To examine this, perform simulation experiment using 174 unseen objects with randomized textures, each rendered under three different motion settings:
1. vertical-axis rotation, as used for training:
2. a rotation about a randomly-chose axis; and
3. translation of up 8% of the object size in random direction.
Note that (2) and (3) are very different from training, and that (3) is equivalent to having a static image with translational misalignments between frames.
The results in Table 6 show there is limited degradation from in-distribution motions out-of- distribution ones. We attribute this to the fact that local shape and material constraints derive from the relative angles between rotation axis, normals and curvatures (see [12), and that since variety of shapes during training, the model sees many of these local relationships, each affected by local translation that depends the point's 3D position with respect to the rotation axis.
wide
Table 6: Evaluation across different types of object motions.
Shape
Albedo
Scenario
Mean AE
Median AE J
SSIM f
RMSE
In-dist. vertical-axx rotation
12.267
10.415
0.868
0.051
Out-of-dist. random rotation
15.062
13.158
0.863
0.052
Out-of-dist. random translation
16.652
14.681
0.858
0.054
A.8
Reflectance Model
We describe the full reflectance model based on the Disney principled BSDF model [10 Recall the reflectance equation
fr(w; Wo; M) 71 Pd [fase (w)) + fretro (wi, Woi r)] + fapee (w;, woi Pds Ps. r, 7).
(6)
π
where M = {Pa: 7,7 consists of the material parameters: diffuse albedo Pa. specular term, Ps roughness r, metallic-ness coefficient y and denotes the spatially varying surface normal point.
The diffuse term depends on the angle of incident light 0, and outgoing direction 0,:
fast (1 F2/2) F./2), where F; (1 cos 1,) F. (1-cos 0.)
(7)
To capture retro-reflective highlights, we add fretro = RRR(F, F, F. (Rx 1)),
where RR 2ny cos ta. cos 02 h Wi,
(8)
16

---

with h denoting the half-vector between the incident and outgoing directions.
The specular reflection is based the microfacet model with GGX distribution:
FDG
F DDG
fapec
(9)
4 (w) (Wo.n)
4 cos 0g cos 0,
Here, D is a microfacet distribution function defined by the roughness parameter 7;
D(h) (((n-h)2(r4 1) 1)2?
(10)
and G denotes the
function
2
= (w,))) (wo), where Gi(w) / r (1 n.w)? /n/
(11)
The Fresnel term blends dielectric and metallic responses:
F = (1 - 7) Edielectric + Fschlick:
(12)
with
1/ cos 0g 17 cos 8:
cos O2 - 17cos (7.coss02))) (2)) where 1 1 v0.08 Ps
2
Edieloctric
+
1,
2 cos 02 + cos 0,
cos G2 + 17 cos 0:
(13)
where 0r is the angle between the normal and the transmitted ray computed using Snell's Law and
Fschtick = Pa + pa) (1 cos 0.)
(14)
A.9 Network Architecture Details
We use the following hyperparameters for the U-ViT3D-Mixe model.
channels [96, 192, 384, 768],
block_dropout = [0, 0, 0.1, 0.1],
block_type ['Local3D' (1), 'Local3D' (1), 'Transformer' (3), 'Transf (8)),
attention_num channels 768,
woise_embedding.co
= 6,
patch_size = 2,
local_attention. Becto__widd__________
7,
channel_mixe wew.iiooeeppssionofaaaaa factor 3,
type v-prediction (MSE)
We the following training setup.
batch_size = 64,
optimizer 'AdamW'
adam_ betas (0.9, 0.99),
adam_veise wat__weighh__eeeeee 0.01
learning_rate 1e-4,
'bfloat16', mox_train_steee 400k
A.10 Inference Hardware Requirement
We compare our model to other stochastic, diffusion-based baseline models in terms of inference time GPU requirement and runtime. We use each model's FP32 version and conduct the analysis on a single A100 GPU. shown in Table our model is compact, fast and memory efficient, due its architectural design.
17

---

Table 7: Model comparison of inference requirements.
Model
Output
Model Size (FP32)
Inference Time
Inference Memory
Ours
Shape and Materials, three frames
0.4 GB
2.7 s
2.8 GB
StamaeNormal
Shape only, single-frame
3.8 GB
19.5
IID
Materials only, single-framc
6.2 GB
6.0
9.5 GB
RGB-X
Shape or Matcrials, single-frame
3.8 GB
6.0 GB
Appendix References
[62]
Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. Gaussianshader: 3D gaussian splatting with shading functions for reflective surfaces. arXiv preprint urXiv:2311.17977 2023.
[63] Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3D gaussian splatting for inverse rendering. arXiv preprint 1rX14:2311.1647, 2023.
[64] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3D mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[43] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma. Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3D: Single image to 3D using cross-domain diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1970-9980, 2024.
[66] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12179- 12188, 2021.
[67] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[68] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[69] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei Efros, and Angjoo Kanazawa. Continuous 3D perception model with persistent state. arXiv preprint arXiv:2501.12335 2025.
[70] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3R: Geometric 3D vision made easy. In CVPR, 2024.
[71] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3D latents for scalable and versatile 3D generation. arXiv preprint arXi::2412.01506, 2024.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:[Yes [Ye]
Justification: Our experiment section 4 reffect the claims in the abstract and introduction: (1) joint inference of shape and materials real and synthetic objects to suggest multiple hypotheses for ambiguous input and (3) leverage motions cues for more accurate inference.
(2) emergent ability
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[ Yes]
Justification: We discuss limitations and possible future work directions in the conclusion section Sec. 4
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include new theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include in the paper detailed description of the model architecture, with implementation and model training details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: Yes]
Justification: Yes we include those details the supplementary materials. We plan to open source the code, pretrained model well our synthetically generated dataset. We will also provide our data generation script for customized usage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.com cc/pu
blic/guides/
ssi ssionilicyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /twpps.ccppullle.ee wwtpccccpppccccceccoec
/oweee/eeeeeaaeeeeeeeeeeeeeeeeee wwwiibimsssioii ssisnnolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
Yes we include those details in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report statistical significance because of the observation that object attributes disentanglement can be inherently ambiguous. We support our quantitative results of shape and albedo estimation by drawing 10 samples and taking the top three for all stochastic methods. For evaluation on motion disambiguation, we draw 100 samples to demonstrate the distribution shift in estimated albedo.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the compute resource requirement in the experiment section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: h/tpp/r//////.../../// hwtpp//blltt.ticccc...........ee
Answer: [Yes]
Justification: Our work conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: have discussed the potential positive impact our model: our framework can serve generative perception backbone for understanding time-varying visual inputs, useful for physically embodied agents. We do not perceive immediate negative societal impact from our work.
22

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The data and model used in our work do not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes we have credited and licensed the Adobe 3D assets used for generating our synthetic dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will be releasing our dataset and codebase with proper documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer[[NA]
Justification:
The development of the core method does not involve LLM usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (fttoe////aurrrpeecccc////eeer// for what should or should not be described.
25


Reviews:
[{"review_id": "kUmoiUJHnk", "reviewer": "Reviewer_EEKb", "summary": "This paper presents a novel conditional denoising-diffusion model for generating shape-and-material maps from short input videos of objects undergoing differential motions. The model, named U-ViT3D-Mixer, is designed to handle the ambiguity in perceiving shape and material from a single image by leveraging motion cues. It can produce diverse, multimodal predictions for ambiguous static observations and converge to more accurate explanations when objects move. The model is trained on synthetic object-motion videos and exhibits compelling emergent properties, such as generating plausible multimodal samples for ambiguous inputs and effectively utilizing differential object motion to resolve perceptual ambiguities. The paper also demonstrates the model's performance on real-world objects, showing high-quality shape-and-material estimates.", "questions": "1.How would the approach generalize to camera motion, translation, or longer or non-rigid motion sequences? Have you performed any pilot studies or experiments on more diverse video data?\n\n2.Can the model be extended to handle more complex lighting conditions, such as those with multiple light sources or non-uniform lighting?\n\n3.Would the authors consider adding error bars, confidence intervals, or repeated trials to the key quantitative results in Table 1 and 2 to strengthen claims of empirical improvement? This would clarify if improvements are significant and reproducible.\n\n\n4.How does the model perform on objects with highly specular or transparent materials, which are known to be challenging for shape and material estimation?\n\n5.What are the main failure or degradation modes when the model is used with cluttered backgrounds, occlusions, or novel, unmodeled materials/lighting?\n\n\n6.What are the specific reasons for the model's improved performance on albedo estimation compared to existing models?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper introduces a new method for joint shape and material perception using a conditional video diffusion model, which is a fresh perspective in the field of computer vision.\n\n2.\tFigures such as Figure 3 (benchmarking against baselines) and Figure 6 (disambiguation via motion, PCA of sample clusters) offer strong evidence of the approach’s claims, with informative visual interpretations.\n\n3.\tThe model is benchmarked quantitatively and qualitatively against strong baselines for shape and albedo estimation on challenging datasets (Table 1, Figure 3), and extensive ablations (Table 2, Figure 5) illustrate the necessity of key architectural choices.\n\n\nWeaknesses:\n\n1.While results on synthetic and some real images are promising, the model and experiments focus exclusively on isolated objects with controlled backgrounds and restricted motion types. The generalizability to multi-object scenes, unknown backgrounds, or real-world video with complex movement is not demonstrated.\n\n2.The evaluation protocol for ambiguous cases is largely qualitative, relying on visualizations or sample diversity. While these are compelling, there is little systematic, quantitative assessment of how well the sampled distributions align with human uncertainty, nor with established perceptual ambiguity benchmarks apart from visual comparisons. This leaves some doubt as to how reliably the model’s uncertainty calibration matches true perceptual ambiguities.\n\n3.The quantitative benchmarks (Table 1, Table 2) lack error bars, confidence intervals, or significance testing. The justification is that ambiguity is intrinsic, but this omission limits the rigor and interpretability of reported improvements and comparisons to baselines. For instance, gains on DiliGENT shape benchmarks (Table 1) are small and would benefit from significance estimates.\n\n4.Training the model requires significant computational resources, taking around 5 days on 4 H100 GPUs, which might be a barrier for some researchers.\n\n5.The synthetic dataset, though large and diverse, may not fully cover the complexity of real-world bidirectional reflectance distribution functions (BRDFs) or complicated indoor/outdoor lighting. Although Section 4 makes an effort in randomizing environment maps and textures, real-world scenes may involve more challenging or unmodeled conditions not validated here.\n\n6.The main baselines in Table 1 focus on single-image models or those specializing in either shape or material. There is limited direct comparison to models that address joint estimation or leverage motion—even if prior art is scarce, this could be more explicitly discussed as a limitation or open problem.\n\n7.The model is designed for isolated objects, and extending it to handle multi-object scenes with natural backgrounds would require additional training and reasoning about grouping and compositions.\n\n8.While efficient in parameters for the design, the paper does not provide a detailed analysis of inference/training speed, GPU/memory requirements, or direct comparisons of parameter count and runtime to established baselines, which is increasingly important in applied settings.", "was_revised": true, "final_justification": "I appreciate the authors' rebuttal, which successfully addressed most of my concerns.\n\nI also read the other comments and rebuttals and raised my rating to 5. However, I was unable to raise the rating to 6 because the authors' initial manuscript did not provide results on their model for multi-object scenes, which I believe are significant and important to the paper's approach.", "rebuttal": "We thank the reviewer for finding our method novel and the experiment results extensive and convincing. \n\n## Weaknesses\n**W1**\n> **Generalizability to different motion types, multi-object scenes, unknown backgrounds, or real-world video is not demonstrated.**\n\nWe currently focus on object-centered scenarios where the background is clean or has been segmented with an off-the-shelf model like SegmentAnything2. Scaling up to uncontrolled backgrounds and more complex and dynamic scenes would be great for future work. Our model handles multiple objects just fine, because the occlusions and disocclusions in those cases are similar to the ones between object-parts (e.g., the braided sphere in Figure 5 and supp.).\n\nWe also empirically find that our model can generalize reasonably well to motions different from the ones used for training. See the additional synthetic and captured results in our responses to **Reviewer 3 - WscD - W1** and **Reviewer 2 - yKQP - W4**.\n\n**W2**\n> **Quantitative assessment with established perceptual ambiguity benchmarks.**\n\nComparison with human perceptual ambiguity is indeed an interesting avenue of future work. There are no established human perceptual ambiguity benchmarks related to joint material and shape perception, partly because before our work there were no computational models capable of capturing shape and material ambiguities like the one demonstrated in [24]. We hope that papers like ours will help inspire future work in psychophysics and inspire joint human and computational perception studies. \n\n**W3**\n> **Quantitative benchmarks lack error bars, confidence intervals, etc.**\n\nTo further support the quantitative evaluations, we repeat the same experiments for our models for 5 runs and report the mean and standard deviations here. Note that the best-performing shape baseline (*Marigold-e2e-ft*) and albedo baseline (*ColorfulShading*) are deterministic so no error bars are reported. We see consistent performance as Table 1 in the main paper. For Table 2 ablations, the results also confirm the effectiveness of proposed modules. \n\n---\n\n*Table 1*\n|       Method       | Mean AE | Median AE |  SSIM | RMSE | PSNR |\n|:------------------:|:---------------:|:-----------------:|:-------------:|:-------------:|:-------------:|\n|        Ours        | 17.782 ± 0.292  | 11.890 ± 0.418    | 0.816 ± 0.008 | 0.076 ± 0.002 | 23.817 ± 0.119 |\n\n*Table 2 -with motion*\n\n| Model                                 |     Mean AE ↓     |      SSIM ↑      |\n|:-------------------------------------|:----------------:|:---------------:|\n| U-ViT3D (Base)                           | 18.899 ± 0.324   | 0.775 ± 0.003   |\n| + Local Spatial-Temporal Attention     | 14.688 ± 0.203   | 0.826 ± 0.003   |\n| + Multi-Attribute Mixer                 | 11.832 ± 0.241   | 0.841 ± 0.002   |\n\n---\n\n**W4**\n> **Training requires significant compute (~ 5 days on 4 H100 GPUs).**\n\nThe computational resource required by our model is modest compared with other trained-from-scratch or finetuned generative models. For comparison, the IID model [35] requires ~24 A6000 GPU days for finetuning and the Diffusion Forcing Transformer [49] requires 60 H100 GPU days to train a video generation model at the same 256 by 256 resolution.\n \nOur architecture and training protocol provide a general template for creating generative models from scratch for tasks that require leveraging novel datasets or modalities and that do not have a suitable pre-trained foundation model or VAE to start from.\n\n**W5**\n> **The synthetic dataset may not fully cover complex real-world BRDFs or lighting.**\n\nOur training data exhausts the parameter space of the analytical BRDF model we adopt, heavily samples the space of environment-map illuminations, and includes global lighting effects like inter-reflections and ambient occlusion. This actually does cover real-world materials and shape, as demonstrated by the accuracy of the model on real-world objects.  For instance, the model generalized to laboratory point light sources in the DiLiGent dataset (as shown in Figure 3) as well as captured real-world videos in Figure 7(b). This ability to fully leverage synthetic data for problems in which real-data supervision is near impossible to obtain is one of the contributions of our work. \n\n**W6**\n> **The main baselines in Table 1 focus on single-image models or those specializing in either shape or material. There is limited direct comparison to models that address both or leverage motion.**\n\nTo the best of our knowledge, our work is the first to perform simultaneous estimation of  shape and material, for either static or moving objects. As such, there are no methods we can directly compare with. Instead, we compared the best shape or material models, and we included variations of our full model as comparisons in Table 2 and Figure 5 to show the effectiveness of the model’s components. We will clarify this in the paper.\n\n**W7**\n> **Extension to handle multi-object with natural backgrounds.**\n\nOur model handles multiple objects just fine (please see response to **W1**). We will show more results and discuss them in the final version. Natural backgrounds should not be a problem either as long as off-the-shelf segmentation methods work. We will test and add results. \n\n**W8**\n> **Comparison to baselines on GPU requirements and inference speed.**\n\nTraining and inference speed are discussed in the paper’s Section 4 under *Implementation*. To supplement this, the following compares to other stochastic, diffusion based models. We use each model’s FP32 version conduct the analysis on a single A100 GPU.\n\nThis shows that our model is compact, fast and memory efficient, due to its architectural design.\n\n|         Model         |               Output               | Model Size (FP32) | Inference Time | Inference GPU Memory |\n|:---------------------|:----------------------------------|:-----------------:|:--------------:|:--------------------:|\n| Ours                  | Shape and Materials, three frames |      0.4 GB       |     2.7 s      |       2.8 GB         |\n| StableNormal    | Shape only, single-frame           |      3.8 GB       |     1.1 s      |      19.5 GB         |\n| IID             | Materials only, single-frame       |      6.2 GB       |     6.0 s      |       9.5 GB         |\n| RgbX             | Shape or Materials, single-frame   |      3.8 GB       |    10.5 s      |       6.0 GB         |\n\n## Questions\n\n**Q1**\n> **How would the approach generalize to camera motion, translation, or longer or non-rigid motion sequences?**\n\n- For out-of-distribution object motions like random rotation or translation, please see the quantitative results on synthetic data in **Reviewer 3 - WscD - W1**. \n\n- For real-world camera motions that consist of both rotation and translation, please see the results on real-world captures in **Reviewer 2 - yKQP - W3**. \n\n- For longer motion sequences, please see the supplementary material, where our model provides better temporal consistency and higher quality  than the baseline (StableNormal [56]).\n\n- For non-rigid motions, our model performs reasonably well for modest deformations, but when the deformations become very large, the model behaves similar or worse than when given static inputs. Handling deformation is an interesting direction for future work. We will add discussion.\n\n**Q2**\n> **Can the model be extended to handle more complex lighting conditions?**\n\nThese are already in the test set (e.g., Figure 5 where specular objects are rendered under complex illumination). Our model accurately estimates the shape and material of objects captured under such complex, multi- and/or non-uniform lighting. The supplementary material also includes such instances. \n\nFigure 7 (b) shows results on objects taken under complex lighting conditions with multiple light sources both from indoor lighting and outdoor natural light. We will clarify by discussing these results in terms of lighting complexity. \n\n**Q3**\n\n> **Error bars, repeated trails for key results.**\n\nSee **W3**.\n\n**Q4**\n\n> **Performance on objects with highly specular or transparent materials.**\n\nThe model does great for any sort of opaque object we have tested. For examples, please see Figure 1(ii) and Figure 5. \n\nOur current model produces a flat ‘postcard’ explanation for transparent objects. Extending our model to handle transparency and translucency are our next targets for future work, and they are interesting in terms of human perception as well (see [#]).\n\n[#] Marlow and Anderson. \"The Cospecification of the Shape and Material Properties of Light Permeable Materials\", PNAS 2021.\n\n**Q5**\n\n> **Main failure mode**\n\nOur model does well for opaque objects under any environment lighting. For transparency it breaks down as described previously.\n\nOur model also handles occlusions well, either between the parts of a single object or between different objects. For example, the plate of cookies in Figure 5 contains multiple objects, and the braided sphere in that figure contains occlusions between object parts that are moving in opposite directions (please see the extended video results on the supplemental html page).\n\n**Q6**\n\n> **Reasons for improved performance on albedo estimation.**\n\nModeling in pixel space instead of a latent space (VAE-based) is important, as well as using randomized textures during training.\n\nJoint training with shape also plays a significant role. As demonstrated in the paper, inferring albedo alone can be quite ambiguous, especially from a single view. It is possible that shape-domain supervision helps encourage the model to learn better priors on shape AND albedo, allowing it to leverage the learned shape prior to jointly sample plausible solutions.  This is in line with prior studies like [*] that show the positive confluence of joint training for multiple vision tasks, especially when data is scarce.\n\n[*] Misra et al. “Cross-stitch Networks for Multi-task Learning”. CVPR 2016."}, {"review_id": "M8UqNb7h93", "reviewer": "Reviewer_yKQP", "summary": "This paper presents a conditional video diffusion model that performs joint estimation of object shape and material from short RGB video clips capturing differential motion. Unlike prior works that focus on single images or isolated attributes, this paper tackles the ambiguity inherent in inferring both shape and material by training a conditional generative model that outputs pixelwise surface normals and material maps, conditioned on RGB frames. To overcome the high computational requirements of modeling high-resolution outputs, the method uses a hybrid architecture: full spatio-temporal attention is applied only at lower resolutions, while higher resolutions use convolutional blocks with separate local neighborhood and spatial attention modules. The paper argues that motion helps disambiguate appearance cues. The model is trained on a synthetic dataset of 100k video clips covering approximately 1100 3D models under diverse textures and lighting. Experiments demonstrate plausible shape and albedo recovery, multi-modal predictions on ambiguous inputs, and modest improvements over single-frame baselines.", "questions": "- Albedo estimation: Marigold reportedly can estimate albedo too. Why is this not reported or compared directly?\n- Baseline video input: Is it possible to adapt single-frame models to accept multiple frames (e.g. by simple frame stacking) to provide a direct baseline for video conditioning? Qualitative figures for this would also help.\n- Real videos: Could the authors show at least a few more real short videos with object motion to demonstrate robustness?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- Exceptionally good writing, figures, and presentation.\n- Novel problem setting: Using short videos for conditioning shape and material estimation is novel and addresses ambiguities that single-view methods struggle with.\n- Interesting architecture: The hybrid design (global transformer at low resolution, local convolution plus attention at high resolution, channel mixing for inter-attribute communication) is well-motivated and technically sound.\n- Pixel-space diffusion: Performing diffusion in image space is a reasonable design choice to preserve fine details compared to latent diffusion pipelines (e.g. Marigold)\n- Clear qualitative results: The paper includes meaningful visualizations showing how motion resolves ambiguities (e.g. shiny vs. painted textures) and how this results in multiple diverse samples.\n- The paper and supplementary demonstrate careful ablations and explanations, including the impact of motion, channel mixing, and temporal consistency.\n\n**Weaknesses:**\n- Generalization. The method is trained from scratch on the dataset of 1100 3D models. This is in contrast to methods like Marigold, which are finetuned form rgb image diffusion, and are not limited in generalization to real-world datasets.\n- Marginal gains vs. Marigold on shape: From the numbers in Table 1, the improvement over Marigold or StableNormal for normal estimation appears modest. \n- No baseline for albedo: The paper claims better albedo estimation but does not clarify whether comparable models like Marigold or RgbX could also be extended to report the same metric under similar settings; most importantly, Marigold already supports albedo estimation.\n- Limited real-world validation: Although the paper shows some real-world photos, these are too few. There is no demonstration of real captured videos to show whether the differential motion cues generalize beyond synthetic motion. This is important given the first concern regarding limited generalization due to training on synthetic dataset.\n- Unfair comparison protocol: For stochastic methods, the paper averages the top 3 of 10 samples for its model but uses the mean prediction for the IID baseline. additionally reporting the same protocol (e.g. best-of-10 or top-3) would be fairer.\n\n**Minor:**\nLines 37–41 claim that most recent methods are non-generative and over-smooth ambiguities. This feels overstated as several recent works do address stochastic outputs; the claim could be softened.\n\nIn summary, the main concern is the limited comparison to existing baselines, real-world generalization and potential unfairness in the experimental protocol. If the authors can clarify the comparison details, report or explain concerns regarding albedo baselines, and add more real-world results, this would make the contribution stronger and i would then recommend acceptance.", "was_revised": true, "final_justification": "I have read the rebuttal and other reviewers’ comments. The authors have addressed my main concerns, in particular by providing a direct comparison to the newly released Marigold-Albedo model, where their approach shows clear improvements. They also added more real-world validation and clarified evaluation protocols, which alleviates my earlier reservations about fairness and completeness of comparisons. While the limitations regarding generalization to in-the-wild scenarios remain, I now consider the contributions (novel joint shape/material generative estimation from motion, solid technical design, and thorough ablations) to outweigh these concerns. I am therefore upgrading my recommendation to borderline accept.", "rebuttal": "We thank the reviewer for recognizing the novelty of our problem formulation and architecture and for providing detailed feedback. We have conducted additional studies to address the concerns and questions.\n\n## Weaknesses\n\n**W1**\n> **Generalization. The method is trained from scratch on the dataset of 1100 3D models. This is in contrast to methods like Marigold, which are finetuned from image diffusion, and are not limited in generalization to real-world datasets.**\n\nThanks for this question; we will add a discussion.\n\nPrior work [23] convincingly shows that Marigold fails to capture multimodal output distributions (e.g. concave/convex) for ambiguous test images and captured photos, such as those in Figure 1(iii) and Figure 4. That work also shows similar failures for other models that are fine-tuned from Stable Diffusion: The SD-based models provide unimodal distributions instead of proper multimodal distributions that reflect the inherent mathematical ambiguities and match multi-stability in humans. \n\nIn contrast, our model captures the multimodality very well, thanks to its joint estimation of shape and materials, its pixel-space diffusion architecture that avoids a VAE, its novel spatio-channel attention mechanism, and its training from scratch. Together, these innovations provide the ability to avoid overcommitting to any single shape/material explanation when inputs are truly ambiguous. \n\nIt is possible that a pre-trained foundation model could provide a shortcut to achieving results like our model, but this has yet to be demonstrated. \n\n**W2**\n> **Marginal gains vs. Marigold on shape: From the numbers in Table 1, the improvement over Marigold or StableNormal for normal estimation appears modest.**\n\nYes, but note that the Marigold-e2e-ft and StableNormal models are trained to specialize in shape estimation from a single image, while ours provides both shape and materials, not just from a single image, but also from a motion video, where it can leverage powerful motion cues. Also, as described above, our model does better at modeling multi-modal ambiguity when the input is inherently ambiguous.\n\n**W3**\n> **The paper claims better albedo estimation but does not clarify whether comparable models like Marigold or RgbX could also be extended to report the same metric under similar settings; most importantly, Marigold already supports albedo estimation.**\n\nSection 4.1 shows comparisons of albedo accuracy using three strong baseline models (IID, ColorfulShading and RgbX) and three different metrics. We did not compare to Marigold-Albedo because it wasn’t public until May 15th 2025, the same week as this NeurIPS deadline. We have since tested the new Marigold-Albedo model using the same metrics (top 3 out of 10), and we found that its performance is similar to ColorfulShading and RgbX, i.e., below our model:\n\n|      Model       | SSIM ↑ | PSNR ↑ | RMSE ↓ | LPIPS ↓ |\n|:----------------|:------:|:------:|:------:|:-------:|\n| Marigold–Albedo  |  0.72  | 19.86  |  0.11  |  0.15   |\n| Ours             |  0.80  | 23.14  |  0.08  |  0.11   |\n\nNote that a major difference between these two models (Marigold-albedo and RgbX) and ours is that they do not perform joint estimation of shape and albedo.Marigold uses two separate checkpoints for shape and albedo, and  RgbX uses a text prompt to switch between the two outputs. This is very different from our approach that directly outputs both and allows them to interact during the denoising process. We believe this is important to our accuracy gains, and our results demonstrate that it is critical for capturing ambiguities.\n\n**W4**\n> **Limited real-world validation.**\n\nPlease see our results on real videos in Figure 7(b). In addition, Figures 3, 7(a) all show results on real-world objects.\n\nWe also further tested on Stanford ORB benchmark [*], a dataset of images of real-world objects, by feeding our model sets of three masked images from nearby camera views. The effective “motion” in this experiment is very different from our training set because \n- (i) the camera moves instead of the object, and \n- (ii) the camera positions are separated by wide, irregular baselines. \n\nDespite this, we find that our model achieves high accuracy, especially in comparison to the two strongest baselines, Marigold-e2e-ft for shape and ColorfulShading for albedo using the same input images. \n\nBelow are the numbers for six of the ORB benchmark’s scenes. Like Table 1, they show competitive performance on shape and superior performance on albedo. *This is additional evidence that our model can generalize beyond the synthetic motion used for training.*\n\n---\n**Stanford ORB Real-world Objects Results**\n- Shape Inference\n\n|       Shape (Median AE ↓)        | Blocks | Cactus |   Car   |   Cup   | Grogu | Teapot |\n|:-------------------:|:------:|:------:|:-------:|:-------:|:-----:|:------:|\n| Marigold-e2e-ft [46]     | 10.19  | 14.59  |  13.69  |  12.14  | 13.27 | 13.25  |\n| Ours                | 11.22  | 19.50  |  13.86  |   7.85  | 10.84 | 15.02  |\n\n- Albedo Inference\n\n|       Albedo (SSIM ↑ / PSNR ↑)        |    Blocks    |    Cactus    |      Car     |      Cup     |     Grogu    |    Teapot    |\n|:-------------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n| ColorfulShading [10]    | 0.76 / 24.69 | 0.66 / 25.58 | 0.64 / 23.01 | 0.84 / 24.83 | 0.61 / 19.87 | 0.69 / 23.33 |\n| Ours                | 0.74 / 25.28 | 0.78 / 26.23 | 0.70 / 23.68 | 0.88 / 24.97 | 0.86 / 24.05 | 0.79 / 24.12 |\n\n---\n\nTo further demonstrate the generalization of our model to other kinds of motions, we also conduct quantitative experiments on unseen assets generated with out-of-distribution motions such as random rotations and translations. Please see results in **Reviewer 3 - WscD - W1**.\n\n[*] Kuang et al. “Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark”. NeurIPS 2023.\n\n**W5**\n> **Unfair comparison protocol: For stochastic methods, the paper averages the top 3 of 10 samples for its model but uses the mean prediction for the IID baseline. Additionally reporting the same protocol would be fairer.**\n\nWe use the mean prediction for the IID baseline following the authors’ recommended setting because it achieves the best performance in their original paper [35]. The evaluation results with the alternative top-3 out of 10 evaluation protocol are below. They are consistently worse than the mean prediction, consistent with the original paper’s claims. Nonetheless, we agree that including the same metric (top 3 out of 10) will make our description more complete. Thank you for the suggestion.\n\n|  IID [36] | SSIM ↑ | RMSE ↓ | PSNR ↑ | LPIPS ↓ |\n|:--------------------------------------------|:------:|:------:|:------:|:-------:|\n| Mean prediction out of 10 samples           |  0.61  |  0.13  | 18.19  |  0.16   |\n| Best 3 out of 10 samples |  0.44  |  0.15  | 17.01  |  0.20   |\n\n**W6**\n> **Lines 37-41 claim could be  softened.**\n\nThank you for the suggestion. We will adjust the wording.\n\n## Questions\n\n**Q1**\n> **Albedo estimation: Marigold reportedly can estimate albedo too. Why is this not reported or compared directly?**\n\nPlease see the discussion and new results in our response to **W3**. We note that by NeurIPS policy any work published after March 1st, 2025 is considered contemporaneous, and that Marigold-Albedo was not released until May. Thank you for suggesting the comparison. We will include our new results in the final version.\n\n**Q2**\n> **Baseline video input: Is it possible to adapt single-frame models to accept multiple frames (e.g. by simple frame stacking) to provide a direct baseline for video conditioning?**\n\nYes it is possible, and we included some examples (using StableNormal) in the last section of the supplemental html. As shown in those examples, the problem with simple stacking is that it leads to flickering due to the lack of temporal consistency. Without training additional temporal interactions such as 3D convolutions or temporal attention, stacked models struggle to effectively leverage the time-varying inputs.\n\nWe will add more examples to the supplement, including with albedo.\n\n**Q3**\n> **Real videos: Could the authors show at least a few more real short videos with object motion to demonstrate robustness?**\n\nSure, we can add more that complement the existing ones in Figure 7(b). Since we are not able to upload media with this rebuttal, we instead conducted a quantitative evaluation using real ‘videos’ created from the real wide-baseline images in the Stanford ORB dataset. See our response to **W4**. We will also add more real videos to the final supplement."}, {"review_id": "0BSbEaoGO0", "reviewer": "Reviewer_WscD", "summary": "This paper aims to predict the graphical parameters (for BSDF) and geometries (normal) from input RGB signals in a generative way. The authors propose to disentangle the ambiguity between geometry and appearance by including multi-views from differential motions, and achieve great performances in selected experiments.", "questions": "Apart from the weaknesses, I also have the following questions. Note that some questions are not aiming to challenge the quality of the paper, but just for better understanding the potential and working scheme of the method, so I don’t list them as weaknesses. \n1. motion related:\n    1. what is exactly the definition of ‘differential object motion’ in this paper? Is only the motion caused by camera movement considered?\n    2. will the model work for deformable objects (could be slightly deformable)?\n    3. will the model work for static object but moving light?\n2. Can the authors show some results of directly working on complex scenes with multiple objects? This results may help us understand whether the mode tend to rely on global features (rely on the overall shape of objects) or local patterns. If latter, the model should work kinda reasonably for complex scenes.\n3. Can the authors show the diversity change of 10 stochastic predictions from one case, of static vs moving?\n4. I recommend the authors include some relighting results based on the decomposed materials and shapes, which can show the flexibility of the proposed methods for the downstream task and can also make the paper more attractive.", "limitations": "The authors sufficiently discuss the limitations in the paper, and I agree with them.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I really like the idea proposed by this paper, that the shape and the material disentanglement should be tackled in a multi-view way. I conclude the strengths of this paper as follows:\n1. this paper proposes to use multiple views from differential object motions to eliminate the ambiguity between object shapes and materials.\n2. They use a unified framework to predict the shape and the material. Instead working as two separate branchs respectively, these two tasks benefits each other. \n3. Well-presented paper with every statement supported by proper citations and very clear explanations, making readers easy to follow and understand.\n\nAs for the weaknesses, I will list the following:\n1. As shown in the limitation section, the motion considered in this paper is simple. Although explicitly stated, this weakness is still unignorable. Since we cannot guarantee similar observations would still hold in more complex motion settings. \n2. The framework asserts that without reconstructing the environment light will lead to a more modular and flexible decomposition (line78-79), but this assertion (actually a strong foundation for the implementation of this work) is not well supported by experiments.\n\n\nOverall this is a good paper in my opinion. But since I am not primarily working on this field, so these weaknesses may not be that so important for this field, and also I will refer to other reviewers’ opinion to adjust my understanding.", "was_revised": true, "final_justification": "Thanks for the authors' for their detailed responses to my questions and concerns, and all of them are addressed. I have thoroughly read the rebuttal to my reviews and also to other reviews. I would say, although the authors are trying to show their model's generalizable ability to various cases, the usage of the model is still constrained to slight differential motion and object centric setting, but not for general motion complexity and normal daily scenes as in-the-wild videos. This is an apparent weakness proposed by all the reviewers.\n\nHowever, I believe a good paper does not equal to a perfect model or a good product. I'd like to advocate the paper for the following reasons:\n\n1. The basic concept of this paper is intuitive and instructive: the appearance and the geometry is ambiguous. Ambiguity means no single solution, so instead of fitting the model on a large datasest to get a single MLE guess, this work proposes to adapt their prediction to more observations from novel views, intuitively like human behavior. This is 'novel'.\n2. The underlying working scheme - matching local feature patterns across views to make better predictions, instead of relying on global understanding - shows potential to gneralize the idea to much more complex future research. So I would suggest the authors put some effort in the next version to discuss more about this part. To be more specific, the motion among objects (either deformable or not), cameras, and lights (static or moving) should perform very similar in local-feature wise. Therefore, the pipeline could benefit future larger models a lot.\n\nBased on the efforts and atitudes of the authors put on this rebuttal, I hold a positive opinion that they have a strong willing to include the experiments in the rebuttal in their final version, and are willing to include more discussion as listed above. So I will increase my Rating to 5 in my final recommendation. However, based on the unignorable weakness, I can't make a rating 6 recommendation.", "rebuttal": "We thank the reviewer for affirming that the proposed ideas are interesting and the unified framework is effective. We also thank the reviewer for the thoughtful comments and questions, which we address below. \n\n## Weaknesses\n\n**W1**\n> **The motion considered in this paper is simple and we cannot guarantee similar observations would still hold in more complex motion settings.**\n\nOur model actually performs reasonably well for motions that are different from the vertical-axis rotations used for training. For instance, in the left video of Figure 6, the teapot rotates around a horizontal axis that is orthogonal to training motion, and yet the model does fine. (See the supplemental html for a visualization.)\n\nTo quantify this, we performed an additional simulation experiment using 174 unseen objects with randomized textures, each rendered under three different settings: (1) a vertical-axis rotation, as used for training; (2) a rotation about a randomly-chosen axis; and (3) a translation of up to 8% object size in a random direction. Note that (2) and (3) are very different from training, and that (3) is equivalent to having a static image with translational misalignments between frames.\n\nThe numbers are provided below, and they show there is limited degradation from in-distribution motions to out-of-distribution ones. We attribute this to the fact that local shape and material constraints derive from the relative angles between rotation axis, normals and curvatures (see [12]), and that since we use a wide variety of shapes during training, the model sees many of these local relationships, each affected by a local translation that depends on the point’s 3D position with respect to the rotation axis.\n\n---\n\n- **Out-of-distribution Differential Motion Experiment**\n\n|                   Scenario                   | Shape – Mean AE ↓ | Shape – Median AE ↓ | Albedo – SSIM ↑ | Albedo – RMSE ↓ |\n|:--------------------------------------------:|:---------------:|:-----------------:|:-------------:|:-------------:|\n| In-distribution vertical-axis rotation       |     12.267      |      10.415       |     0.868     |     0.051     |\n| Out-of-distribution random rotation          |     15.062      |      13.158       |     0.863     |     0.052     |\n| Out-of-distribution random translation       |     16.652      |      14.681       |     0.858     |     0.054     |\n\n---\n\n**W2**\n> **The framework asserts that without reconstructing the environment light will lead to a more modular and flexible decomposition (line78-79), but this assertion is not well supported by experiments.**\n\nThank you for the great question. Our choice of omitting explicit illumination recovery is  motivated by psychophysical studies that show that humans do not perform explicit lighting estimation, or at least delay it until after shape inference [#, %].\n\nWe agree that this argument is qualitative and not based on controlled experiments, so we will adjust our language to reflect this. \n\n[#] Ostrovsky et al. “Perceiving illumination inconsistencies in scenes”, Perception, 2005.\n\n[%] Kunsberg and Zucker. “How shading constrains surface patches without knowledge of light sources.” SIAM Journal on Imaging Sciences, 2014.\n\n## Questions\n\n**Q1-1**\n> **What is exactly the definition of ‘differential object motion’ in this paper? Is only the motion caused by camera movement considered?**\n\nDifferential simply means the object motion is small. We use it to distinguish from wide-baseline images that are typically used for multiview reconstruction, and to connect with previous mathematical studies (like [12]) that use (continuous) differential equations. In practice in our data, we see typical displacements of tens of pixels as an object moves.. \n\nWhile we only train using object motion, we find that our model generalizes fairly well to cases where the camera moves instead, even when the camera movements are over large, irregular baselines. See the results in our response to **Reviewer 2 - yKQP - W4** and results on the Stanford ORB dataset.\n\n**Q1-2**\n> **Will the model work for deformable objects (could be slightly deformable)?**\n\nWe find that our model performs reasonably well for modest deformations, but when the deformations become very large, the model can no longer exploit correlations between frames and instead behaves similar or worse than when given static (i.e., zero motion) inputs. This is an interesting direction for future work and we will add it to the Conclusion and Limitations.\n\n**Q1-3**\n> **Will the model work for static object but moving light?**\n\nNo it does not. This would be a typical photometric stereo setup, for which a number of different methods exist (e.g., Ikehata, CVPR 2023 [*]). Photometric stereo is useful in laboratory settings, but it is otherwise somewhat artificial. Instead, we consider a scenario that is closer to when a natural observer picks up an object and moves it around to better understand its shape and material. In this scenario, the lighting still moves relative to the object, but the object moves relative to the camera as well.\n\n[*] Ikehata, “Scalable, Detailed and Mask-Free Universal Photometric Stereo.” CVPR 2023.\n\n**Q2**\n> **Can the authors show some results of directly working on complex scenes with multiple objects? This results may help us understand whether the mode tend to rely on global features or local patterns. If latter, the model should work kinda reasonably for complex scenes.**\n\nGreat question. The short answer is that the model makes effective use of local patterns and so handles complex scenes quite well. We attribute this to: \n- (1) our training assets including scenes with multiple objects and self-occluding object-parts (e.g., fruits on a plate) and \n- (2) our use of local, shift-invariant attention at high resolutions, which helps avoid overfitting to global object features.\n\nThere is supporting evidence in Figure 4, where the images are very abstract and different from the training set, without occluding contours or other global cues. There is also evidence in Figure 5, where the plate of cookies contains multiple objects, and the braided sphere contains self-occluding object parts that move in different directions. (See also the extended video of the braided sphere in the supplemental html.)  \n\nWe will improve our discussion of this important point in the final version. \n\n**Q3**\n> **Can the authors show the diversity change of 10 stochastic predictions from one case, of static vs moving?**\n\nWe do this in the left of Figure 6, where we visualize the embeddings of 100 predictions along with three examples for each of the static and moving cases. We will add more examples of the predictions to the final supplement, but overall they look fairly similar to the ones in Figure 6. \n\n**Q4**\n> **I recommend the authors include some relighting results based on the decomposed materials and shapes, which can show the flexibility of the proposed methods for the downstream task and can also make the paper more attractive.**\n\nWe included some relighting examples in the right-most column of Figure 5, and we will add more examples to the final supplement, including for captured objects. Thanks for the suggestion."}, {"review_id": "hfrxLnf50u", "reviewer": "Reviewer_yYrM", "summary": "This paper introduces a generative model for the estimation of shape and material parameters from a single image or from a short video sequence. The method relies on a 3D diffusion transformer model, which can diffuse the material properties in a generative fashion (multiple possible outputs from the same input). The results of the paper show that the method is less uncertain about its predictions once a short video sequence is provided, in contrast to using a single image as input. The method is evaluated on synthetic and real-world data, and compared against previous methods and baselines, showing the increased effectiveness of the method with respect to previous work.", "questions": "- What is the impact of the input resolution in the output? Can this model be used on higher resolution inputs?\n- What motivated the choice of the Disney BRDF? Would other BRDFs work worse? \n- Could the authors provide more insights on why the loss weighting coefficients were chosen in this way? My experience is that these inverse rendering algorithms are usually really sensitive to these weightings, and I am curious about these weights, as they are somewhat different to other weights presented in the literature. \n- Could the model be more \"active\" in its exploration? Eg, choose how to rotate the camera, thee objects or the light source. This would would better emulate how humans explore material appearance.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n- This paper tackles a very important problem in computer graphics and vision, and does it in an ambitious way. I believe these findings can spark further work in the literature. \n- The paper is generally well written (with some exceptions), it is very well motivated and it is easy to follow. \n- The proposed method is sound, and makes sense in the context of the problem. Learning 3D object representations with transformers under few-shot scenarios is a really difficult task and the proposed diffusion model make sense and is smart. Data generation and augmentation is also done in a smart way. \n- The paper provides code, which should enhance reproducilibility. \n- Supplementary material provides additional videos and results, which enhance the understanding of the paper, as still images cannot fully convey the domain tbis method is operating on.\n- The paper is somewhat well evaluated, with appropriate ablation studies. \n\n\nWeaknesses:\n- The method is evaluated on very synthetic data, well prepared for the model to disambiguate easily between object and background. There is a single object in each scene. This is a shared limitation of related projects in the literature, however, it is worth acknowledging, as this type of data is not common in the real world and therefore limit the scope of the \"physically-embodided systems\" the paper targets. \n- The method is exclusively studied on opaque objects, with some to little metallicness on them and with a very simple reflectance model. While this is somewhat fine in the context of the literature, the fact that phenomena like transparency, anisotropy, subsurface scattering or translucency are not studied, strongly limit the scope upon which this model is learning to do general-purpose perception of objects. \n- It would have been interesting to study the inheret uncertainty of capturing material properties from single images or from few-shot imagery, as in UMat: Uncertainty-Aware Single Image High Resolution Material Capture (CVPR) or Uncertainty for SVBRDF Acquisition using Frequency Analysis (SIGGRAPH). This line of research is very relevant for this paper and should be analyzed in the related work section. I believe that the paper overstates its contribution in the uncertainty-awareness side, as it does not evaluate this part of the problem, and ignores related work on it.\n- The metrics used for this paper should be expanded. More perceptual metrics must be used to evaluate this method, including: LPIPS, FLIP, and ideally more render-aware metrics, as in Perceptual quality of BRDF approximations: dataset and metrics (EG). \n- The paper relies on specific motions for it to work. The impact of this is not clear, and it may be the case that t will not generalize properly to new motions, or to moving light sources. This is an important limitation.\n- I think the paper sometimes overuses forward references (eg in Line 35, there is a reference to Figure 7a). This reduces the quality of the reading experience.  \n- The choice of model design would benefit from additional explanation. \n- It seems like the model struggles in the single-image example in very specific ways: The model either places most of the variation on the geometry, or on the albedo, leaving the other somewhat constant (Fig 7a). This is unexpected behaviour and creates ill-behaved probability distributions, which maybe be less than ideal for uncertainty quantification. Can the authors elaborate on this?", "was_revised": true, "final_justification": "The authors have addressed all of my concerns during the rebuttal. This paper is interesting to the computer vision community, and, while it is not free from important limitations, I think it merits acceptance. \n\nIt would still have been benefitial to incorporate real objects, more realistic material models, and more active exploration into the model capabilities and training data, but these are shared limitations of most related work. Nevertheless, the paper is valuable and the incorporation of perceptual metrics really adds value to it, making its analyses much more solid.", "rebuttal": "We thank the reviewer for recognizing the importance of our proposed problem and the effectiveness of our framework demonstrated through evaluations. We address the concerns and questions below.\n\n## Weaknesses\n\n**W1**  \n> **The method is evaluated on very synthetic data; there is a single object in each scene. This is a shared limitation of related projects, however, it is worth acknowledging, as this type of data is not common in the real world.**\n\nWe agree and will explicitly discuss this assumption about isolated objects in the final version. In practice, we find it convenient to simply use a segmentation model like SegmentAnything2 to extract foreground objects, as we did for the captured images and videos in Figures 7(a) and 7(b). Please also see our response to **Reviewer 3 - WscD - Q2** on applying the model to more complex scenes.\n\n**W2**\n> **The method is exclusively studied on opaque objects, with some to little metallicness on them and with a very simple reflectance model; Transparency, anisotropy, subsurface scattering or translucency are not studied.**\n\nOur choice of BRDF model includes spatially-varying specular, roughness and metallicness parameters in addition to texture, which is comparable to or more general than the models used in many prior works. For example, IID [35] and RgbX [57] only consider the roughness and metallicness parameters. \n\nThat said, we agree that scaling up the model to consider more complex materials like transparency or anisotropy is an exciting direction for future work. We observe that datasets for real-world evaluation on objects that have more complex materials are currently scarce, so there is need for work in that direction, too.\n\n**W3**\n>  **It would have been interesting to study the inherent uncertainty of capturing material properties from single images or from few-shot imagery, as in UMat: Uncertainty-Aware Single Image High Resolution Material Capture (CVPR) or Uncertainty for SVBRDF Acquisition using Frequency Analysis (SIGGRAPH).**\n\nThese two works are very different from ours because they assume the shape is known. In this scenario there is much less ambiguity and the output distributions tend to be unimodal. Our case is different because the output distributions can contain very distinct (e.g., concave/convex) modes. Also, our model can recover complex, non-planar geometries with self-occlusions, and it can handle global illumination effects like interreflections and ambient occlusion. We will add this discussion to the final version.\n\n**W4**\n> **The metrics used for this paper should be expanded in include perceptual metrics.**\n\nThank you for the suggestion. For geometry, we believe that angular errors are better metrics than perceptual ones. For albedo, we included three metrics: SSIM, PSNR, and RMSE. Below we have also added LPIPS using the same protocol, which we will include in the final version. It shows the same overall trend, with our model outperforming the others and with ColorfulShading being the second best. \n\nNote that we have included the new Marigold-Albedo model in the table below, in line with our response to **Reviewer 2 - yKQP - W3**.\n\n---\n*Albedo Estimation (MIT-Intrinsic)*\n\n|    Model   | IID (Mean Sample) | IID (Top 3) |  RgbX  | ColorfulShading | Marigold - Albedo |  Ours  |\n|:------------:|:-----------------:|:-----------:|:------:|:---------------:|:--------:|:------:|\n| LPIPS (↓)    |      0.163       |    0.199   | 0.166 |      0.139     |  0.149  | 0.110 |\n---\n\n**W5**\n\n> **The paper relies on specific motions for it to work. It may be the case that it will not generalize properly to new motions, or to moving light sources.**\n\nWe find experimentally that our model generalizes fairly well to motions that are different from the ones that are used for training. For instance, in the left video of Figure 6, the teapot rotates around a horizontal axis that is orthogonal to training motion, and yet the model does fine. (See the supplemental html for a visualization.) Additionally, our new results in the response to **Reviewer 3 - WscD - W1** assess generalization to random-axis object rotations and random translations, and our new results in the response to **Reviewer 2 - yKQP - W4** assess generalization to wide-baseline camera motion.\n\nWe do not explore moving light sources since that is essentially photometric stereo, which is better suited to laboratory settings and has methods that are quite mature (e.g.,[*]). Instead we focus on object motion because it is similar to situations where a natural observer picks up an object and moves it around to better understand its shape and material. This type of motion also relates directly to prior perceptual studies like [24] and mathematical studies like [12].\n\n[*] Ikehata. “Scalable, Detailed and Mask-Free Universal Photometric Stereo.” CVPR 2023.\n\n**W6** \n\n> **I think the paper sometimes overuses forward references.**\n\nWe will modify the text to improve clarity.\n\n**W7**\n> **The choice of model design would benefit from additional explanation.**\n\nPlease see Section 3.2.2 for our discussion on the model design. Our design is primarily driven by two needs: (1) improving data and compute efficiency, which is often quite challenging for pixel-space diffusion models, and (2) creating a  unified architecture for shape and materials. \n\nNeed (1) motivates our use of local shift-invariant attention modules and decoupled spatial and temporal attention at high resolutions. Need (2) motivates our introduction of channel-mixer modules to facilitate communication between shape and materials. Additionally, our use of temporal attention is motivated by previous mathematical studies (e.g., [12]) showing that local space-time patterns provide constraints between object motion and surface normals and curvatures.\n\n**W8**\n>  **It seems like the model struggles in the single-image example in very specific ways: it either places most of the variation on the geometry, or on the albedo, leaving the other somewhat constant (Fig 7a). This is unexpected and creates ill-behaved probability distributions, which maybe be less than ideal for uncertainty quantification.**\n\nThere may be a misunderstanding here. Our model’s probability distributions over shape and materials in the single-image case are not ill-behaved. They simply represent the ambiguities that inherently exist. The distribution over shapes and materials from a single image is always multi-modal, because there is always at least the planar ‘postcard’ explanation in addition to the veridical one. And in many cases, like Figure 1(iii), there are additional modes in the output distribution that correspond to convex or concave interpretations which are both mathematically correct. (See [23] for a deeper dive on these cases.)\n\nOur goal is not to develop a quantitative measure of ‘uncertainty’ for these output multimodal distributions. Instead we aim to create an effective way to draw samples from them.\n\nOur experiments show that our model provides diverse samples when the input is ambiguous and has a multimodal output distribution, such as in the single-image cases. They also show that the very same model provides more accurate and less diverse samples when there is motion to help reduce the inherent ambiguity. \n\n## Questions\n\n**Q1**\n>  **What is the impact of the input resolution? Can this model be used on higher resolution inputs?**\n\nCurrently the model works for a fixed resolution of 256x256, similar to previous diffusion-based models. There is ongoing research toward applying diffusion models at resolutions that are higher than their training resolution, for example by applying a model to overlapping regions while applying spatial regularization [%]. This may be worth exploring in future work.\n\nSeparately, our use of neighborhood attention with relative position embeddings (RoPE) [51] at the finer scales may allow other ways to apply our model to higher resolutions with the right architectural modifications. This may also be worth exploring in future work.\n\n[%] Bar-Tal et al. \"MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation\". ICML 2023.\n\n**Q2** \n>  **What motivated the choice of the Disney BRDF? Would other BRDFs work worse?**\n\nWe chose the Disney BRDF because it accurately covers a wide range of materials, including plastics and metals, smooth and rough surfaces, and retroreflective materials. We think other BRDF models could also work, provided they capture a similarly broad range of reflectance phenomena.\n\n**Q3**\n> **Could the authors provide more insights on the choice of loss weighting coefficients?**\n\nOur weighting choices are (0.4, 0.4, 0.3) for (shape, albedo, reflectance), and they are motivated by literature in multi-task learning. Our work is closer to multi-task learning because unlike inverse-rendering, we have no pixel-space rendering loss from a forward rendering equation. So we think of the shape, albedo and reflectance as being different ‘tasks’. \n \nIn general, one wants to assign slightly larger weights to  more difficult tasks and slightly smaller weights to  easier tasks.  We tried multiple weight strategies such as (1/3, 1/3, 1/3 ) or (0.7, 0.2, 0.1), and overall we found that assigning slightly higher weights to shape and albedo leads to faster convergence.\n\n**Q4**\n> **Could the model be more \"active\" in its exploration? This would better emulate how humans explore material appearance.**\n\nThis is a great suggestion for future work. Indeed, we view our proposed framework as a crucial first step toward gaining the ability for more active, human-like perception. In contrast to deterministic  models, generative approaches like ours offer a natural way to assess ambiguity by sampling multiple hypotheses. This should eventually enable perception systems that can recognize when a perceptual distribution is not unimodal, and that can query more views or initiate other modalities like touch to reduce the ambiguity."}]

Author rebuttal:

Meta-review: The paper proposed a conditional diffusion model to estimate the shape and material from a short video of object motions. The reviewers like the proposed idea which is novel and well-presented. The proposed model has also undergone careful ablation. But they also suggested more evaluation on multi-objects and real-world settings. The rebuttal provided detailed clarifications and many additional results. It resolved most of the reviewers’ concerns and made most of the reviewers increase their ratings. Now, all the reviewers expressed a positive attitude toward the paper. The AC concur with the reviewers and believe that the proposed new model could be valuable for the vision and graphics community. As suggested by the reviewers, the final version should include the results in the rebuttal and provide more discussion of the model, especially its generalization ability on the real-world settings.
Decision: Accept (poster)