Multistable Shape from Shading Emerges from Patch Diffusion
Authors: Xinran Han
Todd Zickler
Ko Nishino
Venue: NeurIPS 2024 spotlight (2024)
Keywords: shape from shading
multistable perception
diffusion models
low-level vision
Primary area: machine_vision
TL;DR: We present a bottom-up, patch-based diffusion model for monocular shape from shading that produces multimodal outputs, similar to multistable perception in humans.

Abstract: Models for inferring monocular shape of surfaces with diffuse reflection---shape from shading---ought to produce distributions of outputs, because there are fundamental mathematical ambiguities of both continuous (e.g., bas-relief) and discrete (e.g., convex/concave) types that are also experienced by humans. Yet, the outputs of current models are limited to point estimates or tight distributions around single modes, which prevent them from capturing these effects. We introduce a model that reconstructs a multimodal distribution of shapes from a single shading image, which aligns with the human experience of multistable perception. We train a small denoising diffusion process to generate surface normal fields from $16\times 16$ patches of synthetic images of everyday 3D objects. We deploy this model patch-wise at multiple scales, with guidance from inter-patch shape consistency constraints. Despite its relatively small parameter count and predominantly bottom-up structure, we show that multistable shape explanations emerge from this model for ambiguous test images that humans experience as being multistable. At the same time, the model produces veridical shape estimates for object-like images that include distinctive occluding contours and appear less ambiguous. This may inspire new architectures for stochastic 3D shape perception that are more efficient and better aligned with human experience.

Multistable Shape from Shading Emerges from Patch Diffusion
Xinran Nicole Han Harvard University xinranhan@g. harvard.
Todd Zickler
Ko Nishino Kyoto University kon@i kyoto ac. jp
Harvard University zickleréseas. harvard.
Abstract
Models for inferring monocular shape of surfaces with diffuse reflection- shape from shading- to produce distributions of outputs, because there are funda- mental mathematical ambiguities of both continuous (e-g.. bas-relief) and discrete (e.g., convex/concave) types that are also experienced by humans. Yet, the outputs of current models are limited to point estimates or tight distributions around single modes, which prevent them from capturing these effects. We introduce a model that reconstructs multimodal distribution of shapes from a single shading image, which aligns with the human experience of multistable perception. We train small denoising diffusion process to generate surface normal fields from 16 16 patches of synthetic images of everyday 3D objects. We deploy this model patch-war wise at multiple scales, with guidance from inter-patch shape consistency constraints. Despite its relatively small parameter count and predominantly bottom-up structure, we show that multistable shape explanations emerge from this model for ambigu- ous test images that humans experience being multistable. At the same time, the model produces veridical shape estimates for object-like images that include distinctive occluding contours and appear less ambiguous. This may inspire new architectures for stochastic 3D shape perception that are more efficient and better aligned with human experience.
Introduction
From chiaroscuro in Renaissance paintings to the interplay of light and dark in Ansel Adams' photographs, humans are masters at perceiving three-dimensional shape from variations of image meassitt-shaddng frodiig--oomma from single image alone. Even though our visual experience is dominated by everyday objects, our perception of shape from shading generalizes to many synthetic "non-ecological" images invented by vision scientists. Some of these images have ambiguous (e.g., convex/concave) interpretations and lead to multistable perceptions, where one's impression of 3D shape alternates between two more competing explanations.
[[gure]]shhws an example adapted from [3I] which is alternately interpreted as an indentation or a protrusion. Both explanations are physically correct because the same image can be generated from either shape under different lighting conditions.
How can a computational model acquire this human ability to capture multiple underlying shape explanations? An algorithmic suggestion comes from Marr's principle of least commitment (5), which requires not doing anything that may later have to be undone. But this is in contrast to many computer vision methods for shape from shading, including SIRFS 4 and recent neural feed-forward models [53] (66,, which are deterministic and produce a single, best estimate of shape. These types of models commit one explanation based on priors that either designed learned from dataset, and they cannot express multiple interpretations of an ambiguous image. They are unlikely to be good models for the mechanisms that underlie multistable perception.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Degth Anything
Generating shapes
SIRFS
DerenderDD
Ours
Image
Outgut normals
Integrated depth
WonderSD
Marigold
Many shapes (left) can explain the same image (middle) under different lighting, including flattened and tilted versions and convex/concave flips. The concave/convex flip in this example is also perceived by humans, often aided by rotating the image clockwise by 90 degrees. Previous methods for inferring either surface normals (SIRFS 44, Derender3D [33, Wonder3D (33) depth (Marigold (27]. Depth Anything [56]) produce a single shape estimate or a unimodal distribution. Ours produces multimodal distribution that matches the perceived flip. (Image adapted from 31].)
Figure I:
Instead, approach monocular shape inference conditional generative process, and inspired by a long history of shape from shading with local patches [8 25 bottom-up, patch-based diffusion model that mimic multistable perception for ambiguous images of diffuse shading. Notably, our model is trained using images of familiar object-like shapes and has prior experience with the ambiguous images that we for testing. It is built on small conditional diffusion process that is pre-trained to predict surface normals from 16 x 16 image patches. When we apply this patch process at multiple scales with inter-patch shape consistency constraints, and when we coordinate the sampling across patches, the model ends up capturing global ambiguities that are very similar to those experienced by humans.
29
54
20 17]. we present a
An important attribute of our model is the way it handles lighting. It builds on the mathematical observation that shape perception can precede lighting inference [30]. It also adheres to the philosophy that inferred lighting cannot, and should not, be precise because of spatially-varyingg effects like global illumination [50]. Our model achieves these aims by guiding its diffusion sampling process with a very weak constraint on lighting consistency, where each patch nominates a dominant light direction and then all patches enact their own concave/convex flips in response to those nominations.
Another critical aspect of our model is a diffusion sampling process that is coordinated across multiple scales. It involves spatially resampling the normal predictions at intermediate diffusion time steps and then adding noise before resuming the diffusion at the resampled resolutions. Our approach is inspired by previous work that uses a "V-cycle" fine-coarse-fing to avoid undesirable local extrema during MRF optimization 36). Our ablation experiments show that multi-scale sampling is crucial for finding good shape explanations that are globally consistent.
We train our patch diffusion model on images objects like those Fig. 2a and we find experimen- tally that it can generalize to new objects as well as to images like Fig. which are quite different from the training set and appear multistable to humans. This in contrast to previous diffusion-basee based monocular shape models (27 33 which cannot capture multistability and produce output that is much less diverse. Our model is also extremely efficient, only requiring a small pixel-based diffusion UNet that operates 16 x 16 patches. Our total model weights require only 10MB of storage, much less than the 2-3GB required by some of the previous (and more general) models we compare to.
2 Background
2.1
Ambiguities in Shape from Shading
Shape from shading is a classic reconstruction problem in computer vision. Since being formulated by Horn in the 1970s [23] there have been many approaches to tackle it, often by assuming diffuse Lambertian shading and uniform lighting from either a single direction g [43 (66) or as low-order
spherical harmonics (e.g.. 4 59 Almost all methods either require the lighting to be known
(e.g., for natural illumination
41
or try to estimate precisely via inverse rendering during the
optimization process [54] 59
53 and many approaches rely set of priors to constrain the possible

---

Denoise
Crop
xi
1. Guidernn /t
XC-1
Reshape
40
(a) Training
(b) Inference (single scale)
Figure 2: Training patches are cropped from synthetic images of ordinary diffuse objects, and during training, a small diffusion model learns to denoise the normal field z for patch u from a random sample z7 conditioned on the patch intensities c". During inference, the model applied in parallel to non-overlapping patches, with guidance from inter-patch shape-consistency constraints to minimize curvature smoothness loss Ls and integrability loss L1.
search space. For instance, SIRFS uses different lighting priors for natural versus laboratory conditions and a surface normal prior along occluding contours. Recent deep learning approaches like Derender3D [53] have demonstrated impressive results without being limited to Lambertian reflectance, but they similarly rely on priors internalized from their training sets and have trouble generalizing new conditions.
The main challenge shape from diffuse shading comes from the many levels of inherent ambiguity. At single Lambertian point, when lighting is unknown, there is muuti-dimennionn. manifold of surface orientations and curvatures that are consistent with the spatial derivatives of intensity at the point 29 20 Even when lighting and surface albedo are known at point, there is cone of possible normal directions. At the level of quadratic surface patch, when lighting unknown, there is a discrete four-way ambiguity corresponding convex, concave, and saddle shapes [54 29]]. At global level, when lighting and surface albedo are known, ambiguities arise from interpreting the system of polynomial equations (13). And
Lambertian shading equation a PDE (e.g., (6)
when lighting and albedo are unknown, there is an additional three-parameter global ambiguity that
corresponds to flattenings and tiltings of the global shape [5). Finally, when lighting unknown, global shape has discrete counterpart that corresponds to a global convex/concave flip.
It is important to note that all of these mathematical ambiguities models for the image formation process, such as exact Lambertian shading, perfectly uniform albedo, and most commonly, perfectly uniform lighting that ignores global illumination effects such as interreflections and ambient occlusion, which in reality have substantial effects [50]. An advantage of stochastic, learning-based approach, like the one presented here, is the potential capture all of these ambiguities as well as others that have not yet been discovered or characterized.
based certain idealized
2.2 Denoising Diffusion with Guidance
Diffusion probabilistic models 22 (DDPM) generate data by iteratively denoising samples from a Gaussian (or other) pre-determined distribution. We build on a conditional denoising diffusion model, where the condition is grayscale image patch, and the model is designed to approximate the distribution g(zolc) on 3-channel normal maps zo with a tractable model distribution pe (zolc). A 'forward process' adds Gaussian noise clean input 2o and modeled as Markov chain with Gaussian transitions for timesteps & = 0, 1, ..... T. Each step in the forward process adds noise according to q( ;= ((vI- 3 3,I), where {.0} a predetermined noise variance schedule. The intermediate noisy input x, can be written
It = Vayzo + v1 anw, w ~ NO, where
(1 3.)
(1)
The 'reverse process' 9(2+-1 [x,, e) is modeled by learned Gaussian transition p N( 140 (xt, e), T). The mean value po( (;e) can be expressed as combination of the noisy image 2; and noise prediction eo c0(2,, t; from learned model. The noise prediction model
3

---

0 can be trained by minimizing the prediction error
L( == Er [llw-ea c t; c)
(2)
as shown in 22 To sample noiseless data using the learned model, we start from an initial random Gaussian noise seed and use the learned denoiser co to compute hz-1 vi-8 (Zr
9. Vl-a, (2,,, t; + oez iteratively, where (O, I), which is stochastic process.
The denoising diffusion implicit model (DDIM) shows that the reverse procedure can be made deterministic by modeling it as a non-Markovian process with the same forward marginals 48]. This approach helps accelerate the sampling process by using fewer steps and also provides an estimate of the predicted ño at each timestep with Ty. Each denoising step combines noise and foreseen denoised version
It-1 = -a(-iif( + v1 ar (x),
(3)
where
te) = vI (2
(4)
Vat
is the predicted do at reverse sampling step L.
The DDIM formulation provides way 'guide' the process of sampling It-1 from T, by applying additional constraints or losses to the predicted (z) at intermediate sampling steps. Previous work
has used similar approaches to solve inverse problems [8] 47 to combine outputs from multiple
diffusion models for improved perceptual similarity [2). Guided denoising is achieved with
x/ = Iz 7/c v. ((vp(z)))
(5)
xt-1 = /0-
+ vI 11--a---e a1-00116((22;tt)))
(6)
in each sampling subroutine from time / to ( 1, where vy is the (possibly time-dependent) step size of the guided gradient update. The first step (5) can be repeated multiple times before applying the denoising step (
3 Multiscale Patch Diffusion with Guidance
Consider a surface represented by a differentiable height function h(2, y) over image domain (z, y) viewed by parallel projection from above. The image plane is sampled Image patches have size x d and are indexed by u, and we denote them as c" E [0, 1 dxd
square grid (pixcls).
Training occurs patches extracted from images of everyday objects, as depicted in Fig. 2a For each image patch c" there is a corresponding patch normal field z [-1, 1]axdxd, whose (1. th spatial element represents surface normal vector via
zy (i,5) | (il II(-Pij, -.,, 1)ll
(-Pij) 9..jj-4..:1
ni. E S?,
where (p,g) g) (rh/Ox, 0h/ are the surface derivatives. Training proceeds described in Sec. [2.2 with a dataset of patch tuples (c", xx)) and a UNet cø(x,, L; ctc similar to that in (22 whose four-channel input is the concatenation c" x². (Model and training details in Appendix A.2)
As depicted in Fig. 2b inference occurs over images e of size x W which are divided into their collections of non-overlapping patches ca. We assume that the underlying global normal field To E [-1, 1,1,113xx0 3x XW is continuous including at most locations on the seams between the patch normal fields atj. We formulate the prediction of global field xo reverse conditional diffusion an undirected, four-connected graph ((V,E), E). Each patch z u E V is a node, and there are edges {u, v} E between pairs of patches that horizontally or vertically adjacent.
To encourage the patch fields to form a globally coherent prediction, we use guidance as described by Eqs. 5 and 6 Our guidance includes two terms:
((re)
1
Σ [urreE
Cs + A Σ ()))
1
(8)
E

---

(optiooal
Guidod Sampling
resiz
msiae
X7
N&R
LCG
N&R
LCG
Inject Nose & Resume Sampling (Nec)
Lighting Consistonry Guidanne (LCG)
least
squares
cluster Alp
odd notse
Sameling
Figure 3: Top: Illustration multiscale sampling across two scales in fine-coarse fine "V-cycle", with conditional images omitted for simplicity. In practice, our V-cycle covers more than two scales. Left: The N & subroutine injects noise an earlier timestep 0 < T and then resumes guided sampling (Fig. 2b) at that scale. Right: Optional intermediate guidance comes from lighting consistency (LCG), where each patch nominates a dominant light direction and then some patches flip in response to those nominations. Pseudocode is in the appendix.
where L, is within -atiih continuity term that encourages integrability of the normal fields over small pixel loops, and Ls is an inter-patch spatial consistency term that encourages constant curvature across the seams between patches. Hyperparameter two terms, and 2 0 in Eq. determines the overall guidance strength.
(0, 1] controls the relative weighting of the
The integrability term follows Horn and Brooks 24 by penalizing deviation from discrete approximation to the integrability of surface normals, i.e., 0p/Oy 0g/Oz, over 2 x 2 loops of pixels. We write this as
(2(
Σ (Pr Pi+1,++1 Ps.j - Pit1s + 4.,+1 + Q.41,5+1 q. - 9+11)) (9)
where the summation is over the j.j A. implied by Eq.
pixels in patch u, and p, q are the components of
The spatial consistency term penalizes deviation from constant surface eurvature across each seam {u, v} E E in the direction perpendicular to the seam. Consider four consecutive normals in the perpendicular direction n1, n22 mi, m2 where n, belong to patch u and m, belong to patch v. We penalize the absolute angular difference between my in e and its extrapolated estimate using normals in u, i.e., n2 + (r22 - n1). Making this symmetric gives
cos
(m=
(n2 + (n2 mi)))
cos (n2 (mm (m2 m1) mm))
(10)
which we sum over the length of the seam to obtain 47).
3.1 Dominant Global Lighting Constraint
Our guidance so far enforces global coherence, but even globally coherent surfaces can contain regions that independently undergo convex/concave flips without affecting their surround [3I] The top row of Fig. 5 provides familiar example. One can hide any three of the bumps/dents and then perceive the fourth as being either concave or convex. Yet, when one is allowed to examine the image as whole and rotate it upside down. instead of perceiving 2 16 interpretations, one sees only two: the bottom-right element is bump (or dent) and the other three are its opposite. This behavior explained by lighting. one expects the dominant light direction to be similar everywhere on the surface, the four flips become tied together.
We can incorporate this notion of dominant light consistency into our model using an additional discrete guidance step, as depicted in the bottom right of Fig. 3 This step can be applied to any global sample f and has three parts: (i) patches Φy independently nominate dominant light directions i"; (ii) we identify single direction / that is most common among these nominations; and (iii) some patches perform concave/convex flip to become more consistent with i.
Specifically, each patch 2 that not too close to being planar (i.e., that non-constant ()(()) inates its dominant light direction by computing the least-squares estimate according to shadowless
(i) nom-
5

---

Lambertian shading:
(og ().2) 2 (i)
2
arg min iER
c0(1)
(11)
We create two clusters in the set {in} using k-means and choose the center of the majority cluster as the dominant global direction i. Each patch in the minority cluster undergoes concave/convex flip (p.g) + (-p. -q) by multiplying (-1) with the first two channels of ág- Since the independent flips can cause discontinuities at patch seams, we always follow this discrete lighting guidance step by an inject Noise & Resume sampling (N&R) subroutine, where we add noise to an intermediate timestep via Eq. and resume spatially-guided denoising from that timestep. Pseudocode is provided in Appendix A.I
This approach to lighting consistency has several advantages. Unlike many previous computational approaches to shape from shading, it does not assume the lighting to be known beforehand. Nor does it require the lighting to be exactly spatially uniform across the surface, which provides some resilience to global illumination effects. It imposes no prior on the dominant light direction (e.g.. 'lighting from above"), but one framework can be applied with or without lighting consistency guidance (see Appendix [A.3). it may provide mechanism in the future for modeling the way in which humans selectively enforce lighting consistency across an image
imagine extending it to do so. And because
patch-based
3.2 Multiscale Optimization
Since each local image patch can be explained by either concave or convex shapes, the terms in the spatial guidance energy (Eq. ( are multimodal, and finding a global minimum is computationally difficult. Our experiments, like the one in Fig. show that optimization at a single scale with random initial noise and gradient-descent guidance often gets trapped poor local minima. To overcome this, and also to fully leverage shading information from various spatial frequencies and scales, we draw inspiration from work Markov random fields (36) and introduce multiscale optimization scheme. This is possible because our patch diffusion UNet and guidance can be applied to any resampled resolution (s) x (sW) that is divisible by patch size
Our multiscale optimization occurs in a "V-cycle", a sequence of fine-coarse- resolutions. We begin by applying guided denoising at the highest image resolution. Then, we downsample the predicted global field to a lower resolution before injecting noise and resuming reverse sampling (N&R) that lower resolution. As depicted in the left of Fig. 3 this has the effect of generating random sample at the lower resolution that is informed by a previous sample at the higher resolution. similar process occurs when going from coarse to fine, but with the global field being upsampled before applying the &&& subroutinc.
To further reduce discontinuities at seams, we find it helpful to store and fuse the global field estimates from the final few resolutions of the fine-coarse-fif fiine cycle. We do this by computing their (p. 7) fields via 7 resampling them to the highest resolution and averaging them, and then converting the average back to a normal field.
4 Experimental Results
The input to our patch UNet is the concatenation of c and x.. It has 4 channels and spatial dimension x d with d = 16. We train using patches of size d x d extracted from rendered images of the 3D objects in (26) curated from Adobe Stock. We use Lambertian shading from random light directions, with random albedo in (0.5, 1] and without cast shadows or global illumination effects. Our dataset contains around 8000 images (256 x 256) of 400 unique objects. Some examples are shown in the left of Fig. 2 The images contain occluding contours, and for empty background pixels i, we set Io(i, j) 1-,,- 1, -1). We augment the training data by creating two convex/concave copies of each patch field z that does not contain any background. At inference time, we use the DDIM sampler (48) with 50 sampling steps and with guidance. Additional details are in the appendix.
For comparison, we consider three deterministic approaches and two stochastic models. SIRFS and Derender3D (53) are deterministic inverse-renderigg models that estimate lighting and reflectance

---

rundom sampling
+ single-scal e-aale
sisale spotial consistency
lighting consistency
Depth
Cross-section Aigrstte Dephi
spatial consistency
Image
FURTAR SERjECN EST
Figure Ablations, and comparison to human subjects using image and psychophysics data from 37 Left: Ablations demonstrate the importance of each component. Right: Depth cross- sections extracted from four (integrated) samples from the convex mode our full model exhibit relief-like variations similar to those reported across human subjects. (The dashed line is the depth that used to render the input image.)
Depth
Image
Wonder3D
SIRFS
Devender3D
Marigold
Reference
Anything
X
G
G
Figure Normals produced by our model for various synthetic test surfaces rendered with directional light sources. For depth maps, brighter is closer. "Reference" depicts the shapes- eace with con- vex/concave countcrpart- were used render the input images. We find that our reconstructions more accurate and diverse than other methods.
together with shape. They are among the few recent models that do not rely critically on having input occluding contour masks. Depth Anything (56) a recent learning-base
deterministic model for
monocular depth estimation. leverages a DINOv2 encoder [3) and a DPT decoder 44 and is trained for depth regression using 62M images. For comparisons to stochastic models, we include Marigold [27] which is derived from Stable Diffusion 455 and is fine-tuned for depth estimation. We also include Wonder3D 333, which likewise leverages prior based on Stable Diffusion. Wonder3D is trained to generate consistent multi-view normal estimates on more than 30k 3D objects, and it.
achieves state-of-the-ae results on 3D reconstruction benchmarks 12
4.1 Ablation Studies
Figure 4 analyzes the key components of our model using crop of shape and image from the lab of James Todd [37 (the complete image is in Appendix A.5). The left of the figure shows that when each patch is reconstructed independently, the resulting normals are inconsistent, because each patch may choose different concave/convex mode well as its various flattenings and tiltings. When spatial consistency guidance is applied at one scale, the global field is more consistent but suffers from discontinuous seams due to poor local minima. With multiscale sampling the seams improve, but separate bump/dent regions can still choose different modes without being consistent with any single dominant light direction. Finally, when lighting consistency is added, the output fields become more concentrated around two global modes one that globally convex (lit primarily from below) and another that is globally concave (lit primarily from above).
the right of the figure, we compare samples from our model to depth profiles that were labeled by humans on the same image 337) (These results have appeared previously in (31).) We generate four samples from our model's s globally-convex mode and integrate them to depth maps using [15]. Their cross- coss-scctions exhibit variations with similar qualitative structure.
7

---

Ours
Wonder30
Reference
Frontal plane
12:59 / 30.95
29.96 38.96
17.32/27.0 /27.88
2
Figure 6: t-SNE visualizations of normal field samples produced by our model and by Wonder3D. Plots depict 100 samples from each model, along with the two mathematical possibilities (under directional light) and the normals of trivial frontal plane. For each model report the Wasserstein distance (smaller is better) between its samples and the reference distribution, which is uniform over two possibilities. Our model is more accurate and in all cases covers both possibilities.
w/o
with lighting consistency
Nomels
Arees ligght
Inleggileed sunfane
(velit)
Point light
(b)
Figure 7: Sampled reconstructions for real images. (a) For the 'plates' image from such the indicated box exhibit independent convex/concave flips when lighting consistency is not used; but when lighting consistency enforced, only two global modes emerge. (b) Sampled reconstructions for some multistable images we captured with illumination from point area light. (Rotate them by 180" to enhance the alternative experience.) Note that half of the object in the first was painted matte, and its other half was left glossy. Despite being trained entirely on synthetic data under idealized lighting, the model exhibits some generalization by producing plausible multistable outputs for these captured scenes.
57 regions
4.2 Ambiguous Images
Figure 5 shows results from our full model for images and shapes that we intentionally design to be ambiguous, using insight from [31]. Each one can be perceived either convex concave, as shown the right-most column (Reference). Samples from our model clearly demonstrate the effectiveness of model in terms of both coverage and accuracy of the possible shapes. In contrast, we find that the two models derived from Stable Diffusion (Wonder3D and Marigold) provide less accuracy these images, and that, average, they tend to have 'lighting from above' prior baked in. For instance, they tend to interpret the third and fourth row as concave, while Depth Anything [56) interprets them as convex. Additional results included in the appendix.
Figure 6 visualizes distributions of shape reconstructions as 2D t-SNE plots (with perplexity equal to 30) sampling 100 random seeds for model and for Wonder3D. For reference, also plot the t-SNE embeddings a frontal plane, (i. j) = (0, O, 1) and of the two reference shapes. Our model covers both reference shapes whereas Wonder3D either covers only one is close to plane. These differences in coverage and accuracy are also apparent in terms of Wasserstein distance.
4.3 Real Images
We evaluate our model using few different categories of captured images. In each case, we resize the image to 256 x 256 (to accommodate the restrictions of some of the previous models), and we the multiscale schedule described in Appendix A.10

---

Ours
Ours
Wonder3D
Ours
Wonder3D
Figure 8: Left: Reconstructed normals and integrated depth for an image taken from the web. Right: Reconstructions for images in the SIS dataset of [54] with median angular error from the ground truth normals (lower is better). Our model's accuracy is on par with the best existing methods. Additional quantitative results in the appendix.
Captured ambiguous images. Inspired by the 'plates' image in [58], captured a handful of images that induce multistable perceptions for human observers. We captured these images with Sony a7SIII camera under lighting from area or point light sources, and Fig. /blshows some examples. We find that our model's ' multimodality is qualitatively well aligned with perceptual multistability.
Shape from shading dataset [54]. This dataset contains diffuse objects captured with directional light sources and dark background, along with ground truth surface normals for measuring accuracy. The right of Fig. shows that our model produces normal maps that are on par with previous methods, without knowledge of light source directions. Additional quantitative results are in Appendix A.8
Internet and astronomical images. The left of Figure g shows that model can produce detailed and plausible shape estimate for a tone-mapped sRGB image taken from the web [2]. Appendix A.6 includes two satellite images of the surface of Mars and shows that our model reproduces the so-called crater illusion.
5 Related Work
49 28 many Given the recent success of diffusion models in generating realistic images 22
works have explored the power patch-based diffusion, including for generating high-resolution panoramic images 3 32) Our method also leverages patch diffusion, but it departs from these work in two key ways. Unlike 13, 32), we do not generate patches auto-regressively or require an anchor patch. Instead, we simultaneously guide all patches (e.g.. 100 patches for a 160 x 160 image) toward a coherent output using spatial consistency guidance. A second distinction is that we do not provide global condition such text to each individual patch. Instead, each patch is conditioned only on the corresponding crop of the input grayscale image, which is why we call a bottom-up architecture. It shares the same spirit previous work inverse lighting [14, which also uses bottom-up architecture to produce a variety of explanations that can then be integrated with top-down information.
Recently, Wang et al. [52) introduced patch-based diffusion training framework that incorporates patch coordinates to reduce training time and storage cost. Patch-based diffusion has also been used for other tasks. Ozdenizei et al. [2 use overlapping patch diffusion to restore images in adverse weather, and Ding et al. [T] use it to synthesize images in higher resolution. All of these works use fairly large patches (e.g., e., 64 x 64) and some their components, such as ceature-averaging noise-averaging,
not appropriate for our shape from shading problem because of its inherent multi- modaii-mdallity.. A convex sample and a concave sample cannot simply be averaged to improve the output. These challenges motivate the novel features of model, including global consistency guidance and multiscale sampling.
6 Conclusion
Inspired by the multistable perception of ambiguous images, and by mathematical ambiguities in shape from shading, we introduce a diffusion-based bottom-up model stochastic shape inference. It learns exclusively from observations of everyday objects, and then it produces perceptually-alignel multimodal shape distributions for images that are different from training set and that appear ambiguous to human observers. A critical component of our model is sampling scheme that

---

operates across multiple scales. Our model also provides compositional control: global lighting consistency can be turned on off, thereby controlling whether regional bumps/dents can each undergo concave/convex flips independently. Our findings motivate the exploration of other multiscale stochastic architectures, for variety of computer vision tasks. They may also help improve the understanding and modeling of human shape perception.
Limitations A key limitation is our restriction to textureless and shadowless Lambertian shading. While this restriction is common in theoretical work (29 [54 20 and useful for creating ambiguous images, it is well-known that many ambiguities disappear in the presence of other cues such as glossy highlights, cast shadows, and repetitive texture. Also, since our model is predominantly bottom-up. it suffers when large regions of an image are covered by cast shadows (c.g., the shoulder region in Fig. 8 These types of regions often require non-local context like object recognition in order to be accurately completed. Incorporating more diverse materials (e.g., as in [14) and top-down signals into model are important directions for future research.
Another limitation of our model stems from its sequential V-cycle approach to multiscale sampling. It scales linearly with the number of resolutions, which is likely to be improved by optimization or parallelization that increases runtime efficiency. Also, since our multiscale approach is training free, it requires manual search to identify a good schedule. Similar previous work that restarts the sampling process from intermediate timesteps [55], ours also require choosing the timestep at which to resume sampling. Overall, further analysis is needed to better understand the structure of our model's ' latent space, and to discover more efficient and general approaches to multiscale generation.
Acknowledgments
We thank Jianbo Shi for helpful discussions and Steven Zucker for suggesting the crater illusion. We also thank James Todd, Benjamin Kunsberg, Steven Zucker and William Smith for kindly sharing their images and perceptual stimuli. This work was in part supported by JSPS 20H05951, 21 and JST JPMJAP2305. It was also in part supported by the NSF cooperative agreement PHY-2019786 (an NSF AI Institute, http://iaifi.orgg.
References
[1] Adobe Stock. https://stock: wwwba.ccbooooooooo com/
[2] Nydia, the Blind Flower Girl of Pompeii. Randolph Rogers. Photo by 7ack Jarosz. https: //wb:////www../ //wweppexe pexe
1s. http:/hhotooommmmmstttt
[3] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. MultiDiffusion: Fusing diffusion paths for controlled image generation. In International Conference Machine Learning, pages 1737-1752. PMLR, 2023.
[4] Jonathan T Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 1708)1670-11887 2014.
[5] Peter N Belhumeur, David Kriegman, and Alan I. Yuille. The bas-relief ambiguity. International Journal of Computer Vision, 35(1):33344 1999.
[6] Michacl Breuß, Emiliano Cristiani, Jean-Denis Durou, Maurizio Falcone, and Oliver Vogel. Perspective shape from shading: Ambiguity analysis and numerical approximations. SIAM Journal on Imaging Sciences, 5(1):311-344 2012.
[7] Patrick Cavanagh. The artist as neuroscientist. Nature, 4344(731)):01- 2005.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2023.
[9] Forrester Cole, Phillip Isola, William T Freeman, Frédo Durand, and Edward H Adelson. Shapecollage: occlusion-aware, example-based shape interpretation. In ECCV 2012: 12th European Conference on Computer Vision, 2012.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in neural information processing systems, 34:8780- 8794, 2021.
10

---

[11] Zheng Ding, Mengqi Zhang, Jiajun Wu, and Zhuowen Tu. Patched denoising diffusion models for high-resolution image synthesis. In The Twelfth International Conference Learning Representations, 2023.
[12] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann. Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3D scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553-2560. IEEE, 2022.
[13] Ady Ecker and Allan D Jepson. Polynomial shape from shading. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 145-152 152. IEEE, 2010.
[14] Yuto Enyo and Ko Nishino. Diffusion reflectance map: Single-image stochastic inverse rendering of illumination and reflectance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2024.
[15] Robert T. Frankot and Rama Chellappa. A method for enforcing integrability in shape from shading algorithms. IEEE Transactions on pattern analysis and machine intelligence, 10(4))439 451, 1988.
[16] John Haddon and David Forsyth. Shape representations from shading primitives. In Computer Vi- sion ECCV'98: 5th European Conference Computer Vision Freiburg, Germany, June 2 6, 1998 Proceedings, Volume II pages 415 431. Springer, 1998.
[17] Xinran Han and Todd Zickler. Curvature fields from shading fields. In NeurlPS 2023 Workshop on Symmetry and Geometry Neural Representations, 2023.
[18] Tal Hassner and Ronen Basri. Example based 3D reconstruction from single 2D images. In 2006 Conference on Computer Vision and Puttern Recognition Workshop (CVPRW'06) 2006.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Kathryn Heal, Jialiang Wang. Steven J Gortler, and Todd Zickler. A lighting-invarte invariant point processor for shading. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 94-102, 2020.
[21] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12509. 2022.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systemx, 33:6840-6851, 2020.
[23] Berthold KP Horn. Shape from shading: A method for obtaining the shape of a smooth opaque object from one view. 1970.
[24] Berthold KP Horn and Michael Brooks. Shape from shading. MIT press, 1989.
[25] Xinyu Huang, Jizhou Gao, Liang Wang, and Ruigang Yang. Examplar-hased shape from shading. In Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007), 2007.
[26] Satoshi Ikchata. Universal photometric stereo network using global lighting contexts. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12591111200, 2022.
[27] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9492-9502, 2024.
[28] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. DiffuuiinCCIIP: Text-guided diffuusion models for rohust image manipulation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pottern Recognition, pages 2426-2435, 2022.
[29] Benjamin Kunsberg and Steven W Zucker. Characterizing ambiguity in light source invariant shape from shading. arXiv preprint arXiv: 1306.5480, 2013.
[30] Benjamin Kunsherg and Steven W Zucker. How shading constrains surface patches without knowledge of light sources. SIAM Journal on Imaging Sciences, 7(2):641 2014.
[31] Benjamin Kunsberg and Steven W Zucker: From boundaries to bumps: when closed (extremal) contours critical. Journal of Vision, 21(13):7-7,

---

[32] Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. SyneDiffusion: Coherent montage via synchronized joint diffusions. Advances in Neural Information Processing Systems, 36, 2024.
[33] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingji Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3D: Single image 3D using cross-domain diffusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9970-9980, 2024.
[34] David Marr. Early processing of visual information. Philosophical Transactions of the Royal Society of London. B, Biological Sciences. 275(942):483- 519, 1976.
[35] David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
[36] Omer Mcir, Mcirav Galun, Stav Yagev, Ronen Basri, and Irad Yavnch. A multiscale variable-grouppe framework for MRF energy minimization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1805-1813, 2015.
[37] Makaela Nartker. James Todd. and Alexander Petrov. Distortions of apparent 3D shape from shading caused by changes in the direction of illumination. Journal of Vision, 17(10))324-322 2017.
[38] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[39] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. arXiv preprint 2rX1v:2304.07799, 2023.
[40] Yuri Ostrovsky, Patrick Cavanagh. and Pawan Sinha. Perceiving illumination inconsistencies in scenes. Perception, 34(1 34(11):10111131, 2005.
[41] Geoffrey Oxholm and Ko Nishino. Shape and reflectance from natural illumination. European Conference on Computer Vision (ECCV 2012), 2012.
[42] Ozan Özdenizci and Robert Legenstein. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[43] Alex Pentland. Shape information from shading: a theory about human perception. In /1988 Proceedings] Second International Conference on Computer Vision. pages 404-413, IEEE, 1988.
[44] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22179-12188, 2021.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11885, 2022.
[46] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient Attention: Attention with lincar complexities. Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531-3539, 2021.
[47] Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models hard data consistency. arXiv preprint arXiv:2307.088222 2023.
[48] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International Conference on Learning Representations, 2021.
[49] Yang Song, Jascha Sohl- Dick--Dikttin, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[50] James T Todd, Eric JL Egan, and Flip Phillips. Is the perception of 3D shape from shading based on assumed reflectance and illumination? j-Perception, 566)-477-5,4 2014.
[51] Johan Wagemans, Andrea J Van Doom, and Jan Koenderink. The shading cue in context. i-Perception, 1 (3):159-177, 2010.
[52] Zhendong Wang. Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang. Weizhu Chen, Mingyuan Zhou, et al. Patch Diffusion: Faster and more data- bata-efficient training of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[53] Felix Wimbauer, Shangzhe Wu, and Christian Rupprecht. De-rendering 3D objects in the wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[54] Ying Xiong, Ayan Chakrabarti, Ronen Basri, Steven J Gortler, David W Jacobs, and Todd Zickler. From shading to local shape. IEEE Transactions Pattern Analysis and Machine Intelligence, 37(1):67-79, 2014.
[55] Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. Advances Neural Information Processing Systems, 36: 19806-76838 2023.
[56] Lihe Yang. Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371-10381, 2024.
[57] Ye Yu and William AP Smith. Inverserendernet: Leamnngg single image inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3155 3164, 2019.
[58] Ye Yu and William AP Smith. Outdoor inverse rendering from single image using multiview self- supervision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3659-300 2021.
[59] Daniel Zoran, Dilip Krishnan, Jose Bento, and Bill Freeman. Shape and illumination from shading using the generic viewpoint assumption. Advances in Neurul Information Processing Systems, 27, 2014.
13

---

A Appendix supplemental material
A.1 Algorithms pseudocode
We provide the pseudocode for the single-scale spatial consistency guided sampling (Alg D and the lighting consistency guidance (Alg. 2 Here, we have hyperparameters À for weighting the smoothness and integrabilityy loss, 7 as guidance update weight and Jr as the number of noise update steps. The results in our paper use À = 0.5 and J = 3. The parameter is aasolutiondependde and is included with the schedule specification in Appendix(A.I. A.10
A.2 Experimental setup details
Model Architecture. We use conditional UNet architecture similar to [2] with input spatial dimensions of 16 x 16 and 4 channels. The input to the UNet is concatenation of the grayscale shading image and 3-channel normal map. We use linear attention module [46] for better time and memory efficiency. The UNet consists of 4 downsampling and upsampling stages composed of the commonly used ResNet [9 blocks, group normalization layers, attention layers, and residual
connections.
Dataset and Training Details. We train the pixel-space conditional diffusion model on a dataset that we build from the UniPS dataset [26). It contains about 8000 256 x 256 synthetic images of 400 unique objects from the Adobe3D Assets [ rendered from different viewing directions. We render the objects with the shadow-less Lambertian model using the provided ground truth normal maps in (26). randomly sampled directional light source within 60 degrees of the z-axis, and an albedo value in [0.5, 1]. The surface normal values outside of the objects are set (-1, -1,--1,11
We subdivide the image into non-overlapping patches of size 16 16 and train our model predict the noise at each sampled timestep using a smooth L1 loss. To train the diffusion UNet, we use the cosine variance schedule (38 with 300 timesteps. The model trained using the AdamW optimizer for 500 epochs with learning rate 2e-4. It takes about 40 hours using one Nvidia A100 GPU.
Algorithm 1: Spatial Consistency Guided Sampling at Single Scale
Data: (=",e² Juevi Ea, {n,, }}
while > 0; do
// Parallel guidance
for j = 1,2,...,JJ
// Gradient descent for multiple steps
do
2/ = Pred(x c², ce)
// Predict t0 using Eq // Patch to global layout
20 Reshape([$ā)
((0)) Te Ls (#: 28) + ((t))
// Eq.
xt + xt ((to)
end
27
+ Denoise(x;;: c², cc))))))
Parallel denoising (Eq.)
3 for each patch
end
return
{2 {
// Denoised normal prediction
Algorithm 2: Lighting Consistency Guidance
Data: {xi, cett.chhee heev
j0 = Robust infer(xk,,c))
// Infer light source direction as in Eq. 11
2 Cluster center {L1, L2), assignment 1e K-Means Clustering ({"), clusters // Assume that 11* 1| > k 2|
for u E V, k. = 2; do
(xx)x f- - (xx))x (xityy <- (x:))); // Convex/concave flip of the normals
end
return {xi) u
14

---

(c)
Figure 9: Sampled output normals and integrated depth maps when lighting consistency is not enforced. In contrast the samples in Fig. regions within the same image can undergo independent convex/concave flips.
Data Augmentation. During training, we augment the dataset with convex/concave flips of interior patches, meaning those that do not include any occluding contour and background. In Sec. A.7 compare two models trained with and without such augmentation. While the augmentation leads to more balanced multimodal output distribution and thus smaller Wasserstein distance, model trained without augmentation is already capable producing multistable outputs on test images.
Sampling. We use the DDIM sampler 48 with 50 sampling steps with guidance. Details of the multiscale sampling schedule and guidance learning rate T). can be found in Appendix A.
Visualization and Evaluation Metric. show the multistable reconstructions. We also show depth maps by integrating the normals using
We visualize the output normal map from multiple samples
the method by Frankot & Chellappa [15].
To produce the t-SNE visualizations in Figs 6 and 13 draw 100 samples from each model, and then downsize the sampled normal maps to 64 x 64 resolution for computational efficiency. Then we project the samples 2-dimensional space using t-SNE with perplexity value of 30. To compute the 1-Wasserstein distance, we set the ground truth distribution to be uniform distribution over the reference normal maps: the one used render the input image, and its convex/concave flip. When computing the 1 I-Wasserstein distance of model outputs to the ground truth distribution, all normal maps are first downsampled 64 64 resolution.
Baseline models. For testing with Wonder3D [3)] we extract the frontal view prediction and use the default setting in their online demo and code base with crop size of 256 by 256, classifier-free guidance scale 3, and 50 sampling steps.
A.3 Ablation study on lighting consistency guidance
In Fig. we show additional samples from our model where lighting consistency guidance turned off. Results show that in this case the output distribution of our model allows different regions within a surface (e.g., each dimple, ring or mouse-shape) to undergo an independent convex/concave flip. This is in contrast to Fig. of the main paper, where see only two global modes.
15

---

Samples
I-NE Visualization
Sumples
t-SNE Visualization
Image
Reforence
(a) Model trained object images
Model trained cubic images
Training set samples (before dividing smaller patches)
Figure 10: Output normal maps and their t-SNE visualizations when our model is applied to 16 16 image an exactly quadratic surface under directional lighting. When our model trained using images of spline surfaces (b), the outputs cluster around the four mathematical interpretations from (54) (convex, concave, and two saddles). When it trained using images everyday objects (a), the outputs exhibit more diversity. In both scenarios, samples are drawn independently without guidance.
A.4 Relation to the four-way conves/concavvessddlle ambiguity
Figure 10 examines the relationship between model and the mathematical results from Xiong et al. [54] which show that, in general, an image of an exactly quadratic surface under unknown directional lighting be explained by four quadratic shapes (convex, concave, and two saddles). If our model is consistent with the theory, we would expect its output shape distributions for such images to be concentrated around four distinct modes that match the four distinct possibilities. When we apply our model to images of exactly-quadrale surfaces like the one in the left of Fig. [O we find that its output distribution is concentrated However, we find that the four-mode behavior emerges when the model is retrained on a different dataset comprising random cubic splines (right panel), which by construction contain a much higher proportion of exactly-que quaadratc surface patches.
four modes (middle panel in the figure).
One potential explanation for this behavior is that everyday scenes for a vision system to usefully exploit. This may relate to the perceptual experiments in [51) that suggest humans also struggle perceive the four distinct shapes for such images.
surface patches are too rare in
A.5 Additional results perceptual stimuli
Figure 11 shows our model's output for images that were used to study human perception in [31 and [37]. Our model produces plausible shapes and multimodal output distributions, while other models sometimes fail to recover plausible shape produce only of the global concave/convex possibilities. The bottom row an image several bumps ('cobbles') from below. observe that Wonder3D [33], Marigold (27 and SIRFS 4 interpret the bumps concave, while Depth Anything [56] interprets them as convex.
A.6 Additional results on astronomical images; relation to the crater illusion
Figure 12 shows results for two satellite images of the surface of Mars. In images like these, humans often misperceive craters mountains and vice versa, perhaps due their bias toward lighting from above. (This is sometimes called the crater illusion.) We tested our model, the diffusion- models 33 27). and Depth Anything (56). Our model sees both the crater and mountain possibilities, but the other models only see one of the two.
Data acquisition is expensive in these situations, so there could be benefit to having monocular vision model that can automatically produce the multitude of explanations, thereby allowing all possibilities to be examined by gathering additional context. Similar to human perception, in astronomical imaging
is beneficial to have access to all of the possible "bottom-up" explanations, so that one can use context or "top-down" information effectively possible.
16

---

Wonder3D
DerenderAD wia masking
Marigold
Depth Anything
without masking
welo
with marsiing
with masking
Figure 11: Inferred normal map samples from our model on ridge images taken from 31 and 'cobble" test image from 37
Image
Ours
Wonder3D
Marigold
Depth Anything
Figure 12:
Shape from Martian crater images. For diffusion- models we show two samples that qualitatively represent all of the 25 samples that we generated for that model. For depth models (Marigold and Depth Anything) brighter is closer. Our model outputs both possibilities, crater and mountain, while other models only see one of the two. Outputs from Wonder3D and Marigold also include artefacts (yellow boxes), with sharp spurious variations in normals depth.
The two images are taken from:
(1) A triple crater in Elysium Planitia on Mars. Credit: NASA/JPL/University of Arizona. https //www, universeoday cwt. haamm//1181//// hwtp////amaiing---e iippit--mpctttte wherer-rrrraaeeeeeae----a-e aari-lle-rppee-ee asteroi d smashed into - maros
(2) A fresh impact crater, about kilometers wide, gouged from lava-covered plain in the Lunae
Planum region Mars. http:///ppp//ayaaaaalaoo.ooooorr
otg.::/eeeeee.aaaaea.oooooooooa- 8 s-show-craters--ee settteerr-aa-bisteerre blisters/
A.7 Ablation dataset augmentation
To explore the effect of convex/concave data augmentation during training (see the top of Section 4 we perform an ablation in which we train a model from the same set of patches but without the augmentation. Since the original patches from our training set tend to be dominated by convexity, expect this to have an affect on the model's response to ambiguous images. Figure 13 and Table [ bear this out. The figure and table show the same t-SNE visualizations and Wasserstein distances as in the main paper, but this time with without augmentation during training. The model without augmentation exhibits some multistability, especially for the last images, but it tends provide less diversity, especially for circular shapes. We hypothesize that this is caused by the existence of predominantly convex spherical shapes in the training set.
A.8 Quantitative results shape from shading benchmark
Table shows quantitative comparisons using photographs from the dataset in (54). We follow prior practice and report the median angular error of the predicted normal field, where the angular error at each pixel is AE an()) |cos (fq. ng Diffusion- baasusdoo--beedd outputs stochastic and
17

---

Table 1: Wasserstein distance on multistable perception stimuli
Model
(a) four circles
(b) nested rings
(c) star
(d) snake
Wonder3D 33
30.94
38.96
27.78
27.90
Ours (w/o data augmentation)
33.79
39.56
22.04
23.02
Ours
12.59
29.96
17.32
18.27
augmentation)
Reference
plane
(4)
(b)
(6)
(d)
Figure 13: t-SNE plots showing that our model trained without flip augmentation already exhibits multistable outputs, especially on the last two images. Training with data augmentation improves the overall diversity those test images.
potentially multi-modal (e.g., with global convex concave possibilities), so for these report the average error the top-five predictions taken from 50 independent runs. Note that the method from [54] requires the true light direction to be provided as input, whereas ours and others do not.
Overall, our model shows competitive performance on this benchmark compared with existing methods. We notice that our model trained without data augmentation via per- per-patch concave/convex flips (see the top of Section 4 performs better, while our model trained with data augmentation has slightly worse performance, likely due to its larger per-patch search space. This may be related to other muality-versus-diverr
trade-offs that have been observed in previous conditional diffusion models [1O, 21). We leave for future research questions of how to achieve better combinations of quality and diversity, and how to incorporate other cues such as occluding contours and top-down recognition cues.
A.9 Ablation on lighting distribution in training set
Figure [4 shows experiment where change the distribution of light source directions in the training set. The two models A and are trained on the same shapes without any data augmentation, but Model B has 80% of the images lit from above. Model A is trained with uniformly sampled lighting from above and from below. We test both models on an image that appears concave when 'lit from above'. From 50 random samples and their t-SNE projections, we that Model B's distribution is biased towards the concave answer while Model shows a more balanced distribution. This shows that lighting bias in the training set have effect the output distribution.
Table Shape from shading benchmark quantitative results. Errors are measured by median angular error of normals maps. Model performance for diffusion based models is averaged over the top estimates over 50 random seeds.
Model
cat
frog
hippo
lizard
pig
turtle
scholar
Xiong et. 54 (known lighting)
14.83
11.80
20.25
12.70
15.29
17.90
28.13
SIRFS Cross-Scale 4
20.02
19.86
21.00
23.26
13.17
11.96
25.80
Wonder3D [8
14.29
18.20
16.81
15.70
10.10
9.59
25.32
Ours (w/o data augmentation)
11.49
15,56
14.17
13.30
9.27
8.72
22.01
Ours
14.95
21.46
15.94
12.82
11.98
9.99
27.21
18

---

Test image
Rotated version
(e) Model
(b) Model
Figure 14: Effect of lighting bias during training. Model A is trained using synthetic images with lighting directions that are uniformly sampled within 60% cone around the view direction. Model B trained using the same shapes but with lighting that is distributed non-uniformly within the cone, where 80% of images a test image, using the same schedule and guidance hyperparameters. We project the results using 1-SNE (dots randomly colored for visual clarity) and show representative samples. Model A produces a balanced distribution across convex and concave explanations, whereas Model B produces concave predictions more often. (To humans, the image usually appears concave, and usually appears convex when rotated. These are both physically consistent with lighting from above.)
lit from above. After training,
draw 50 samples from each model for
Table Multiscale optimization schedule
Perception Stimuli
Captured Photo
Resolution
[160, 128, 64, 80, 96, 112, 128, 144, 160]
[256, 160, 96, 128, 192, 224, 240, 256]
Guidance rate
[20, 15, 10, 10, 10, 15, 15, 20, 20]
[30, 20, 12, 15, 20, 25, 28, 30] x + [T] + [F] x
Lighting guidance
[T] 2 [F] 7
& start
[300] + [232]
[300] [238]
Runtime (scconds)
105s (single Quadro RTX 8000)
125s (single Quadro RTX 8000)
A.10 Multiscale schedule specification
In Table 3 lists the scheduler hyperparameters for multiscale guided sampling. We use a notation for lists with repeating elements, where concatenation represented follows: for example, [A] x 2 + [B] 3 denotes the list [A, A, B, B, BJ.
When designing the multiscale schedule for inference, we find helpful have consecutive resolu- tions that are not integer multiples of the previous one, especially in the coarse to fine direction. This leads improved quality because pixels that adjacent to patch seams at one resolution become interior to patch at another resolution.
For the initial resolution (our experiments 160 x 160 256 x 256), guidance is applied only after the 8th DDIM denoising step since the predicted ân at very early stages are often not informative enough for guidance. In terms of lighting consistency guidance, find that it is often not necessary to apply at every resolution for perceptually consistent normal estimation.
We apply normal field fusion using the last three resolutions, after the spatial predictions and choices of per-patch convex/concave modes have stabilized.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our experiments in Sec. rrfflect the claims in abstract and introduction. We show that the model achieves multistable outputs various perception stimuli and also produce veridical shape estimates for real objects.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We include text in Sec. dedicateed for such discussions and possible future directions.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
This paper does not include new theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have included in the paper a detailed description of the model architecture, and podu for our algorithms. We also provide the hyperparameter and schedulers used in inference time in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We include in the supplemental material the implementation of the main algorithms in the paper. The data preparation instructions for training dataset also included in the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/pu
www.eimmmssioooPooooooo ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips.cc/pip peapps.cc/uubllc/gg..eeeee////// Sttp:///CoeoSSeeieieeeeeee
www.bbmmmmisioooooooocoooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included all necessary details about the experiment settings and training details in Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report statistical significance because the main observation in the paper that there are many "ground truth" shapes for this monocular shape inference problem. They are related, for example, by convex/concave flips and bas-relief transforma- tions. However, we use large numbers of samples to support the claims in our paper. Our quantitative results of the Wasserstein distance perception stimuli is computed over 100 samples, and the angular errors for shape from shading benchmark is computed from 50 samples for all diffusion-based models.
Guidelines:
22

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute, memory and runtime requirement for reproducing the experi- ments are included in Appendix A.10
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our work conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
23

---

Justification: We have discussed the potential positive impact of our model: (1) our model is compute-effiiien and does not require many GPUs for training at large scale and (2) our work can help with understanding human vision. We do not perceive immediate negative societal impact from our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The data and model used in our work do not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our dataset is based previous work Universal Photometric Stereo Network using Global Contexts (CVPR 2022) which uses GPL License and allows user to run, study, modify and share the project.
Guidelines:
24

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi snperswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not include any new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing research with human subjects.
25

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "SlwYdBaLDt", "reviewer": "Reviewer_xHhf", "summary": "This paper presents a patch-wise diffusion based Shape-from-shading strategy to recover multiple shapes satisfying concave/convex ambiguity from the images. It models shape inference as a generative process conditioned by light intensities. The generative process is governed by  small, non-overlapping patches at multiple scales. Simple measures of overall spatial consistency and global light consistency allows the disjoint patches to converge to globally coherent shapes.", "questions": "Since the method yields multiple solutions satisfying concave/convex ambiguities. I am wondering if it is possible to determine the exact number of solutions. For example, in figure 5, 3 solutions are reported. However, it is easy to see that there are only 2 solutions possible in these cases satisfying the discrete (convex/concave ambiguity are possible.  The third solution is normally a bas-relief of one of these two solution.  Therefore, it would be nice to add a mechanism to classify the solutions according to possible ambiguities.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The use of patch-wise diffusion is an  interesting for the proposed methodology. By breaking the images into small patches, allows the method to generalise beyond the shapes belonging to the training datasets. It also allows to preserve the concave/convex ambiguities.\n\n2.  Experiments show that the method is able to reconstruct shapes unto convex/concave ambiguities well, even on the datasets on which it was not trained on. \n\n3. Ablation studies are a plus.", "weaknesses": "The reconstruction (figure 8) obtained by the proposed method is quite accurate, quite similar to Wonder3D. However, since the method yields multiple solutions, how is this particular solution chosen and what is the range of variations in the different solutions obtained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses**: \n> The reconstruction (figure 8) obtained by the proposed method is quite accurate, quite similar to Wonder3D. However, since the method yields multiple solutions, how is this particular solution chosen and what is the range of variations in the different solutions obtained.\n\nThe samples shown in Fig. 8 are drawn from the 10 most accurate reconstructions from among 50 samples, for both our model and Wonder3D. Since both models are stochastic, the median angular error can vary from sample to sample. Figure 8 conveys how close each method comes to having the ground truth shape within its output distribution. \n\nThe following chart shows the diversity of the top 10 samples for each method, measured by standard deviation of error (in degrees) with respect to ground truth.\n\n|  | cat | frog | hippo | lizard | pig | turtle | scholar |\n|----------|----------|----------|----------|----------|----------|----------|----------|\n|   Wonder3D  |   $14.42\\pm0.17$  |  $18.46\\pm0.33$  |   $16.97\\pm0.18$  |   $15.81\\pm0.12$  |  $10.22\\pm0.15$  |   $9.69\\pm0.12$  |   $25.55\\pm0.31$  |\n|   Ours  |   $15.49\\pm0.68$  |   $22.41\\pm1.13$  |   $16.59\\pm0.82$  |   $13.66\\pm1.03$  |   $13.03\\pm1.15$  |   $10.53\\pm0.63$  |   $27.95\\pm0.93$  |\n\nIn terms of reconstruction error, our model performs on par with Wonder3D and has slightly larger standard deviation. This is expected since our model is designed to avoid overfitting to its training set and to correctly cover a diverse solution space instead of returning one solution. \n\nPlease also revisit Appendix Table 1, which shows the quantitative results for top-5 outputs and comparisons to additional baselines.\n\n**Questions**: \n> Since the method yields multiple solutions satisfying concave/convex ambiguities. I am wondering if it is possible to determine the exact number of solutions. For example, in figure 5, 3 solutions are reported. However, it is easy to see that there are only 2 solutions possible in these cases satisfying the discrete (convex/concave ambiguity are possible. The third solution is normally a bas-relief of one of these two solution. Therefore, it would be nice to add a mechanism to classify the solutions according to possible ambiguities.\n\nYes, one would think that the solution space usually consists of globally convex/concave reconstructions, each with their own family of GBR variations. However, the results from [29] suggest that regions enclosed by so-called “critical contours” of shading can each flip independently, giving even more solutions. An example adapted from [29] is shown in Appendix Fig.9b (nested rings), where we see that our method successfully recovers more solutions.\n\nWhen global lighting consistency is included, some solutions are eliminated by our algorithm, typically leaving two main modes that have a global convex/concave relationship. We believe it is important to maintain flexibility in how and when global lighting consistency is applied, because excluding it allows exploring a larger solution space, and because perceptual studies like [*] include compelling examples that make it clear that lighting consistency at the global level is not always strictly enforced in human perception. \n\nTo characterize the outputs of our model for a specific image, we find it helpful to inspect a lower-dimensional projection, such as from t-SNE, to assess the presence of discernible clusters. Figure 6 shows an example. \n\nWe thank the reviewer for this question, which will help clarify these points in the final revision.\n\n* [29] Benjamin Kunsberg and Steven W Zucker. From boundaries to bumps: when closed (extremal) contours are critical. Journal of Vision, 21(13):7–7, 2021.\n\n* [*] Yuri Ostrovsky, Patrick Cavanagh, and Pawan Sinha. Perceiving illumination inconsistencies in scenes. Perception, 34(11):1301–1314, 2005.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XkywvQ8H6w", "reviewer": "Reviewer_3BEx", "summary": "The authors argue that monocular shape reconstruction models ought to produce distributions of outputs rather than point estimates or tight distributions, to adequately cope with known ambiguities, and draw additional motivation from multistable perception in humans. By training a small patch-based diffusion model, the authors convincingly demonstrate multistable solutions to shape from shading through a fairly lightweight algorithm.", "questions": "Technical:\n========\n- L41: I wonder if training on common objects free of illusions, similar to what humans are most used to, is necessary for the resulting models to not be biased to any particular interpretation. That is, in comparison to pre-trained diffusion models who seem to be biased to \"lighting from above.\" Since the diffusion model employed is small, it would be nice to include such an experiment and assess the level of bias in the generated reconstructions in terms of related biases in the training data.\n- L208: why the 3D objects from [23]?\n- L207: why the 0.7 lower bound?\n- L31: it would be nice to go back and relate the proposed algorithm to the principle of least commitment.\n\nPresentation:\n==========\nSection 2\n- L103: \"learned\" Gaussian transitions\n  - The forward process, with a predetermined noise schedule, doesn't seem to entail any learning, right?\n- L107: $\\mathcal{N}(x_{t-1}, \\cdot)$ -> $\\mathcal{N}(x_t, \\cdot)$\n- L118: $\\epsilon(x_t; c)$ -> $\\epsilon(x_t, t; c)$\nSection 3\n- L128: *differentiable surface* sounds off. What's a better way to say this? A representation?\n- Eq. 6: flattening the $d \\times d$ patch is a bit counterintuitive, and was later abandoned for Eq. 8.\n- Figure 3 is quite confusing. Could use more work.\nSection 5\n- L271: We do provide -> We do not provide (?)\n\nNitpicking:\n========\n- L38: Instead we approach [..] and *are* (?) inspired\n- L134: It would have been nice to be able to refer to the equation on L109.\n- L142: Better to say something about $\\mathcal{E}$ by way of introduction or definition.", "limitations": "As the authors point out, this study inherits some of the restrictions common in theoretical work. In addition, the proposed model while powerful is not particularly fully developed or optimized. That said, none of that seems to diminish the value of the contribution in terms of either the concept or the obtained results demonstrating the capabilities and potential of such reconstruction models.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Proposes multistable perception to cope with the ambiguity of monocular shape reconstruction.\n- Presents a viable solution based on a small patch-based diffusion model and a fairly simple consistency-enforcing algorithm.\n- Demonstrates compelling results for coping with ambiguities, even when the model was trained on synthetic renderings of common 3D objects.\n- Very well written and easy to read.", "weaknesses": "Nothing major, other than the contribution being more conceptual in nature, with no immediate practical impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and the detailed suggestions!\n\n**Weaknesses & Limitations**: \n\nPlease see the global response.\n\n**Questions**:\n\n**Technical**\n> L41: I wonder if training on common objects free of illusions, similar to what humans are most used to, is necessary for the resulting models to not be biased to any particular interpretation. That is, in comparison to pre-trained diffusion models who seem to be biased to \"lighting from above.\" Since the diffusion model employed is small, it would be nice to include such an experiment and assess the level of bias in the generated reconstructions in terms of related biases in the training data. \n\nThis is a great question. We believe that diffusion models like Wonder3D and Marigold show such bias not due to the type of objects in their training sets, but due to the distribution of illumination conditions in their training sets, which are likely biased to being “from above”. In contrast, our synthetic training dataset is rendered with randomly sampled light source directions. In the attached **PDF Fig. R2**, we show an experiment where we change the distribution of light source directions in the training set. The two models A and B are trained on the same shapes without any data augmentation, and the only difference is that for model B we make 80% of the images lit from above. We then test on an image that looks concave when one holds the ‘lighting from above’ prior. From 50 random samples and their t-SNE projections, we see that model B’s distribution is biased towards the concave answer while model A shows a more balanced distribution. This shows that lighting bias in the training set can have an effect on the output distribution. \n\nThe bias in existing diffusion models could also come from their fine-tuning schemes, but more studies are required for a thorough answer. \n\nPlease also revisit Appendix Fig. 10, which shows another effect of dataset bias. The model in (b) is trained solely on cubic-spline images, and it behaves differently from our main model for certain inputs. When given a quadratic-patch image, it produces four distinct clusters that  correspond to the well-known convex/concave/saddle ambiguity. However, our main model (a), which is trained on everyday objects, produces a qualitatively different distribution. \n\n> L208: why the 3D objects from [23]?\n\nWe were inspired by the success of using the dataset of [23] for a different 3D reconstruction task, namely photometric stereo [23, #], and in particular that work’s successful generalization from synthetic to real images. Our results show analogous generalization to unseen images. \n\n* [23] Universal Photometric Stereo Network using Global Contexts. Satoshi Ikehata. CVPR 2022.\n\n* [#] Scalable, Detailed and Mask-free Universal Photometric Stereo. Satoshi Ikehata. CVPR 2023.\n\n> L207: why the 0.7 lower bound?\n\nThank you for asking. We checked our code again and found our lower bound is actually 0.5 instead of 0.7. We apologize for the error and will update this in the paper.\n\nTraining with a randomly sampled albedo instead of a fixed unit albedo helps the model generalize to images that have different exposure levels. We found that training with fixed albedo causes the model to produce overly-flattened shapes in under-exposed images.\n\nWe chose the lower bound of 0.5 because white paint often has albedo between 0.5 and 0.9, and because empirically we found it sufficient for generalization to real images. \n\n> L31: it would be nice to go back and relate the proposed algorithm to the principle of least commitment.\n\nGreat suggestion. We will add a sentence in the conclusion to connect back to the least commitment principle. \n\n**Presentation**\n\n> L103: \"learned\" Gaussian transitions. The forward process, with a predetermined noise schedule, doesn't seem to entail any learning, right?\n\nYes this word should be removed as there is no learning component. \n\n> L107: $\\mathcal{N}(x_{t-1}, \\cdot)$ -> $\\mathcal{N}(x_t, \\cdot)$\n\nHere $x_{t-1}$ denotes the random variable and will be the same on both sides of the equation. We have also verified this with the DDPM paper [19] (Ho et al., 2020).\n\n> L118: $\\epsilon (x_t; c)$ -> $\\epsilon (x_t, t; c)$\n\nYes, thank you very much.\n\n> L128: differentiable surface sounds off. What's a better way to say this? A representation?\n\nYes. We will change it to *a surface represented by a differentiable height function $h(x, y)$*.\n\n> Eq. 6: flattening the $d \\times d$ patch is a bit counterintuitive, and was later abandoned for Eq. 8.\n\nAgreed. We will change the indexing in Eq. (6) to (i, j)-notation, making it consistent with Eq. (8).\n\n> Figure 3 is quite confusing. Could use more work. \n\nWe agree that it can be improved. We have proposed a revised version in the **PDF Fig. R3** and we welcome feedback.\n\n> L271: We do provide -> We do not provide (?)\n\nYes. We do NOT provide global conditioning here.\n\n> L38: Instead we approach [..] and are (?) inspired\n\nThank you for the suggestion. We will make the sentence clearer.\n\n> L134: It would have been nice to be able to refer to the equation on L109\n\nYes. We will improve the paragraph and add the equation label. \n\n> L142: Better to say something about $\\mathcal{E}$ by way of introduction or definition.\n\nGreat advice. We will add descriptions of the edge set as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kgwiGOp4Tg", "reviewer": "Reviewer_E5e1", "summary": "This paper deals with the problem of normal map estimation from images. The authors trained a conditional diffusion model to sample a normal map output given an image input by processing 16x16 patches. The paper suggests a multiscale approach for resampling consistently at multiple scales. Due to the model’s non-deterministic property, it can cover possible outputs in ambiguous cases. The model is also integrated with a lighting consistency module, to verify that each possible output normal map is consistent with a global directional lighting.  Experiments show generalization to unseen cases, for both synthetic and real images.", "questions": "* Are there real-world applications that could be solved or improved by the suggested method?\n* Could this method be used in multiview settings, for consistently reconstructing texture-less surfaces?", "limitations": "The authors discussed limitations and societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* A novel diffusion model for consistently predicting normal maps from an input image\n* Clear presentation of the model. \n* Dealing with ambiguous cases consistently. \n* Efficient and lightweight model. \n* Strong generalization to unseen images.", "weaknesses": "Limited applicability. Most of the tested images are either synthetic or captured in a controlled setup. The compared baseline methods are not limited to textureless and shadowless Lambertian shading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses**: \n> Limited applicability. Most of the tested images are either synthetic or captured in a controlled setup. The compared baseline methods are not limited to textureless and shadowless Lambertian shading.\n\nYes, it is absolutely true that the comparison methods are not limited to Lambertian surfaces, but it is also true that they cannot produce multistable solutions, as we demonstrate. \n\nWe chose to focus on Lambertian surfaces because there are rich theoretical and perceptual characterizations of the ambiguities, as described in Section 2.1. Focusing on Lambertian surfaces allows us to compare our model’s outputs to those predicted by theory, and in Figures 5 and 6, we indeed see that our model produces both discrete modes and continuous GBR families.\n\nWe will explore expanding our model to non-Lambertian (e.g., glossy) surfaces with global light transport, including cast shadows and interreflections. These phenomena produce additional visual cues that dramatically reduce the ambiguity in many cases, but since they are not always present, one cannot rely on them entirely. It is an interesting open challenge to create models that can properly exploit these cues when they are present, and that can also avoid overfitting by correctly producing diverse local and global shapes when they are absent. \n\n**Questions**:\n> Are there real-world applications that could be solved or improved by the suggested method?\n\nPlease see the global response.\n\n> Could this method be used in multiview settings, for consistently reconstructing texture-less surfaces?\n\nThank you for the great suggestion. Since correspondences are harder to detect for texture-less surfaces (see e.g., [%]), shape-from-shading models like ours can be a useful complement. To adapt our model to a multiview setting, one could add additional terms to the guidance loss or consistency module, verifying agreement between views. We will add this point to the conclusion section.\n\n* [%] Cryer JE, Tsai PS, Shah M. Integration of shape from shading and stereo. Pattern recognition. 1995 Jul 1;28(7):1033-43.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful comments. We are encouraged that all reviewers find our research problem interesting and our lightweight patch diffusion model novel and effective. Reviewers (3BEx) and (E5e1) also find our presentation to be well-written and reviewer (xHhf) finds the ablation studies helpful. 

Here we address common questions from all reviewers: What problems can benefit from the proposed model, and how will it be useful in practice? 

Multistable perception happens much more often than one may imagine. When visual cues that help resolve the shape are missing, such as occluding boundaries, depth profiles can become ambiguous, as clearly demonstrated in our figures. One may not often notice this in their daily life because humans usually have access to additional visual cues. Yet, it shouldn’t be hard to recall an occasion when your shape perception was corrected by conscious reasoning or action—such as consciously considering the context of occlusions and lighting, or by moving your head to gather additional viewpoints—after an incorrect first glance at a scene. Models like ours are essential for understanding how human perception achieves this fine balance of efficiency and robustness, and how it can be imitated in robotics. 

As for relevance to scientific disciplines beyond human and robot perception, consider astronomical imaging. Our attached PDF includes two photographs of the Martian surface. Humans misperceive a crater as a mountain and vice versa in these photos, due to their ingrained perceptual bias. (These are instances of the so-called crater illusion.) Data acquisition is expensive in these situations, so there is benefit to having a monocular vision model that can elucidate the possibility of multiple solutions, thereby allowing all possibilities to be examined by gathering additional context. Similar to human perception, in astronomical imaging it is essential to have access to all possible “bottom-up” solutions, so that one can use context or “top-down” information as effectively as possible. 

As you can see in the attached **PDF Fig. R1**, we tested our model on two crater images, along with two other diffusion-based models and the Depth Anything model. Our model recognizes both possibilities, crater and mountain, while the other models only see one of the two. 

We will adjust the text in our paper to clarify these important elements of our model, in the broad context of visual perception.

Meta-review: This paper addresses a classical problem in computer vision: shape from shading. It demonstrates that it is possible to recover "multistable" shapes from a single image using a diffusion-like process. All reviewers have recommended acceptance, with one reviewer giving a strong acceptance. The AC concurs with the reviewers' assessments and strongly recommends acceptance.
Decision: Accept (spotlight)