IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations
Authors: Zhibing Li
Tong Wu
Jing Tan
Mengchen Zhang
Jiaqi Wang
Dahua Lin
Venue: ICLR 2025 Poster (2025)
Keywords: inverse rendering
diffusion models
intrinsic decomposition
Primary area: generative models
TL;DR: Intrinsic decomposition for arbitrary number of input images under unconstraint and varying illuminations

Abstract: Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics. Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs, while still struggling with inherent ambiguities between lighting and material. On the other hand, learning-based approaches leverage rich material priors from existing 3D object datasets but face challenges with maintaining multi-view consistency.
In this paper, we introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations. Our method achieves highly accurate and multi-view consistent estimation on surface normals and material properties. This is made possible through a novel cross-view, cross-domain attention module and an illumination-augmented, view-adaptive training strategy. Additionally, we introduce ARB-Objaverse, a new dataset that provides large-scale multi-view intrinsic data and renderings under diverse lighting conditions, supporting robust training.
Extensive experiments demonstrate that IDArb outperforms state-of-the-art methods both qualitatively and quantitatively. Moreover, our approach facilitates a range of downstream tasks, including single-image relighting, photometric stereo, and 3D reconstruction, highlighting its broad applicability in realistic 3D content creation.
Project website: https://lizb6626.github.io/IDArb/.

Published as a conference paper at ICLR 2025
IDARB: INTRINSIC DECOMPOSITION FOR ARBITRARY NUMBER OF INPUT VIEWS AND ILLUMINATIONS
Zhibing Li', Tong Wu' Jiig Tan', Mengehen Zhang Jiaqi Wang?, Dahua Lin
1 The Chinese University of Hong Kong 2 'Zhejiang University
3 Shanghai AI Laboratory CPII under InnoHK
{1z022, wt020, dhlin]@ie.
ABSTRACT
Capturing geometric and material information from images remains fundamen- tal challenge in computer vision and graphics. Traditional optimization-base methods often require hours of computational time reconstruct geometry. ma- terial properties, and environmental lighting from dense multi-view inputs, while still struggling with inherent ambiguities between lighting and material. On the other hand, learning- based approaches leverage rich material priors from exist- ing 3D object datasets but face challenges with maintaining multi-view consis- tency. In this paper, we introduce IDArb, diffussin perform intrinsic decomposition on an arbitrary number of images under varying illuminations. Our method achieves accurate and multi-view consistent estima- tion on surface normals and material properties. This is made possible through a novel cross-view, cross-domain attention module and illumination-augun -aaiinattin-augmetteeee view-adaptive training strategy. Additionally, we introduce ARB- ARB-Objaverse, a new dataset that provides large-scale multi-view intrinsic data and renderings un- der diverse lighting conditions, supporting robust training. Extensive experiments demonstrate that IDArb outperforms state-of-the-art methods both qualitatively and quantitatively. Moreover, our approach facilitates range of downstream tasks, including single-image relighting, photometric stereo, and 3D reconstruc- tion, highlighting its broad applications in realistic 3D content creation. Project website: https: /tttps://lizbbet (tte:a2666....hhtiiee github. i0/ IDArb/
model designed to
INTRODUCTION
The color we perceive from objects results from complex interaction between the incident light, the material properties, and the surface geometry of those objects. Recovering these intrinsic prop- erties from captured images is fundamental challenge in computer vision, enabling a variety of downstream applications, such as relighting (Wimbauer et al., 2022) and photo-realistic 3D content generation (Zhang al.) 2024. Siddiqui et al. 2024). This decomposition process, commonly re- ferred to as inverse rendering, is inherently ambiguous and severely under-constrained, particularly when only one or a limited number of observation views are available. For instance, black pixel could indicate black base color or is the result of lacking incident light.
Existing inverse rendering research can be broadly categorized into two approaches: optimization based methods and learning-based based methods. The former category (e.g. NeRFactor (Zhang et al.) 2021b), NVDiffRecMC (Hasselgren et al. 2022), TensolR (Jin et al. 2023)) typically requires hun- dreds of multi-view images as input and focuses optimizing intrinsic properties for each case independently. This approach involves time-consuming iterative optimization, often requiring sev- eral hours. Moreover, without incorporating strong priors material distribution or addressing the inherent ambiguity between lighting and texture, these optimization based methods frequently converge sub-optimal solutions. This can lead unrealistic decompositions, such embed- ding lighting effects into intrinsic components, as shown in Fig. [b). To address these limitations, learning-basse methods aim extract useful priors from large-scale training datasets and perform fast inference in feed-forward manner. While many of these approaches focus on single-image decomposition, they tend produce inconsistent intrinsic properties when applied across multiple
"Corresoonding authors.

---

Published as a conference paper at ICLR 2025
Intrinsic Decomposition
Multi-view under different illuminations
Single/ htall//mmltiivieww under single illumination
albedo
roughness
metallic
NNDiffRechC
Intriniiceyytbl
Durs
Ours
Outputs
Multi-view consistency (a)
Material am ambiguity (b)
Relighting & Material Editing
Photometric Stereo
3D Reconstruction
Figure 1: IDArb tackles intrinsic decomposition for an arbitrary number of views under un- constrained illumination. Our approach (a) achieves multi niew consistency compared to learning- based methods and (b) effectively disentangles intrinsic components from lighting effects compared to optimization-based methods. Our method enhances wide range of applications such as image editing, photometric stereo, and 3D reconstruction.
views, as demonstrated in Fig. Ia). Additionally, single-image models struggle to leverage comple- mentary information from multiple views, making it difficult resolve material ambiguities, which results in less accurate outcomes more complex cases.
To mitigate these challenges, propose IDArb, model capable of taking arbitrary number of images captured under unconstrained, varying lighting conditions and predicting correspond- ing intrinsic components, including albedo, normal, metallic and roughness. Our key contribu- tions are three-fold. First, we adopt the cross-view, cross-component attention module from Won- der3D (Long et al. 2023) fuse information across different views and intrinsic components. This module facilitates holistic understanding of the multi view correspondence and joint distribution of intrinsic components, enabling consistency across viewpoints and reducing decomposition un- certainty. Despite being trained on fixed number of input views, our model shows the flexibilityy to decompose an arbitrary number of input images without requiring camera poses. Second, to improve performance under complex lighting conditions, we create a custom dataset based on Obja- verse (Deitke et al. 2022), namely ABB-Objaerrse, which contains 5.7M multi-view RGB images and intrinsic components with varying illumination scenarios for effective training. Lastly, devise a novel and effective
-augintttnonugmmeeeeedde and view-adapted training strategy to achieve robust performance under varying lighting conditions and leverage both multi-view cues and general object material prior for better multi-view and single-view inverse rendering.
We evaluate our model extensively both synthetic and real data. Our approach significantly out- performs existing learning- based methods (Kocsis et al.) 2024 Zeng et al., 2024 Chen et al.) 2024) by large margin, both qualitatively and quantitatively, achieving Wateeoff-he-aae results in intrinsic decomposition. Our model offers practical benefits for range of downstream tasks, including ma- terial editing, relighting, and photometric stereo, and it can also serve strong prior improve optimization-basee methods by better disentangling lighting effects from intrinsic appearance. We believe that IDArb provides unified solution across different input regimes in inverse rendering, advancing our ability to understand and model the physical world.
RELATED WORK
2.1
OPTIMIZATION BASED INVERSE RENDERING
Optimization based inverse rendering methods aim to jointly reconstruct shape, materials, and light- ing from multi-view images. Volumetric representation methods (Boss et al. 2021a Kuang al.)
2

---

Published as a conference paper at ICLR 2025
2022
Boss et al. 2021b Zhang al. 2021b) extend NeRF (Mildenhall et al., 2020)
model in-
trinsic appearance and lighting conditions, rendering images using volume rendering techniques.
Surface-based representation methods (Zhang et al., [02 2022a bj Sun al. 2023 Wu et al.
2023) extract surfaces as signed distance functions (SDHs) [Wagg et al. 2021) or differentiable
meshes (Munkberg et al.] [2022 Hasselgren al., 2022), apply explicit material models such as
Bidirectional Reflectance Distribution Functions (BRDFs) (Nicodemus) 1965). and render images
through physics-based procedures. Recent works explore 3D Gaussian representation Kerbl et al. 2023
Gao et al. (2023) for this task, assigning intrinsic attributes to each Gaussian point.
While existing methods effectively simulate global illumination, they often require dense multi- view inputs and can be computationally expensive, especially for complex scenes. In addition, they face the inherent ambiguity between lighting and materials, which can lead to suboptimal solutions, such incorrectly baked lighting into textures. In contrast, our proposed method offers an efficient solution for inverse rendering in a forwad manner. By leveraging well-learned priors from our large-scale, multi-view, multi-lighting dataset, we can significantly mitigate the issue of ambiguity.
2.2 BARRING-BASED INVERSE RENDERING
With advances in deep neural networks, bearning-basee approaches (Barron & Malik 2020 Li al.
2019 Zhu et al. 2022 Bi ct al. 2020 Careaga & Aksoy 2023 Shi et al. (016) have demonstrated impressive performance in intrinsic decomposition. They typically take a single image as input and decompose intrinsic properties from the input view, such as albedo, specular, and surface normal.
Early learning-basee methods (Li et al.] 2018 Wu et al.) 2021] Wimbauer et al.] 2022 Sang & Chandraker 2020; Boss et al. 2020; Yi et al., 2023) handle intrinsic decomposition as determin- istic problem, often leading to over-smoothed details in ambiguous pixels. Recent works (Koesis) (et al. 2024 Chen et al. [2024 Zeng et al., (024) adopt probabilistic distribution modeling with diffusion (Ho et al.] 2020), estimating accurate intrinsic components with high-frequency details through a generative formulation. ZZng et al. (224) presents unified diffusion framework that ad- dresses both RGB +X (estimating intrinsic properties) and X~RGB (generating realistic images) by training diffusion pipelines on multiple data sources.
These learning-based approaches typically handle inverse rendering in a single-view setting, leading to inconsistent results when applied to multi-view data. Our work extends the ree-fforward diffusion pipeline to address the under-exple explorex challenge of multi-view inverse rendering, providing a unified solution for various input types and offering valuable intrinsic priors for downstream applications.
2.3 DIFFUSION MODELS FOR OTHER MODALITIES
Denoising Diffusion Probabilistic Models (DDPMs) and their variants (Ho et al. 2020 Rombach
et al. 2021 Zhang al. 2023) have gained significant attention in text-to-image generation, yield- ing promising results across various applications. Researchers have also explored adapting diffusion models to different output modalities such as normal (Fu et al. 2024). depth (Ke et al. (024) and novel view images (Liu et al. [203 2024b Kong et al. (024). To generate multiple modality simul- taneously, Wonder3D (Long et al., 2023) introduces additional cross-domain attention modules into diffusion model that generates multi-view normal maps and corresponding color images. We ex- tend this concept intrinsic decomposition by splitting the intrinsic components into three triplets and modeling their joint distribution. By leveraging pre-trained diffusion models, which capture rich structural, semantic, and material knowledge, we can overcome data limitations and ensure generalization to real-world scenarios, when the models are trained on synthetic data.
METHOD
IDArb is diffusion-hased model for intrinsic decomposition that can handle an arbitrary number of input views and varying lighting conditions. We begin by outlining the problem statement in Section 3.11 Then, in Section we describe the construction of our custom dataset tailored to this task. Finally, we discuss the model architecture and training strategy in Sec. 3.3 An overview of IDArb is provided in Fig.
3.1 PROBLEM STATEMENT
We frame intrinsic decomposition as a conditional generation problem:
X1.N p(X i I..))
(1)
3

---

Published as a conference paper at ICLR 2025
N, views
specific text prompt
"albedo" / "normai" / "metaalickrouggnnes"
IN images
U.Net
Attention Block
coopo-compo attention
cross-uiew attention
feed-forward
cross attention
albado
normal
metallica roughnees
views
view2
View3
Viewo
Ka
Attention Haps
prompt
Hew
HeW
Figure 2: Top: Overview of IDArb. Bottom:
Illustration of the attention block within the UNet. Our training batch consists of N input images, sampled from N, viewpoints and N; illuminations. The latent vector for each image concatenated with Gaussian noise for denoising. Intrinsic com- ponents are divided into three triplets (D=3): Albedo, Normal and Metallic& text prompts are used to guide the model toward different intrinsic components. For attention block inside UNet, we introduce cross-component and cross-view attention module into it, where attention applied across components and views, facilitating global information exchange.
Specific
Here, N N denotes the number of input views; I denotes input RGB images, and X represents the intrinsic components each view. We model X using the simplified Disney BRDF parameterization (Burley & Studios 2012 Karis & Games 2013), which includes albedo A R xW x3, roughness R E xx metallic E ( xI and surface normal N E IR xW x3
The number of input images N take on arbitrary value from to many, and the input images can be rendered under arbitrary unconstrained illuminations during both training and inference.
3.2 ARB-ORIAVERSE DATASET
Obtaining ground truth data for intrinsic decomposition in real-world settings is both time- consuming and technically challenging. To overcome this, we rely on synthetic data for training. Ideally, suitable dataset should feature large-scale, diverse objects rendered under multiple light- ing conditions. However, existing datasets have notable limitations. For example, G-Objaverse QQu jet al. 2024) employs a single, low-contrast lighting setup, while ABO (Collins et al. 2022 is re- stricted to household items, suffering from lack of diversity among the objects.
To address these shortcomings, we develop custom dataset, Arb-Objaverse. We select 68k 3D models from Objaverse Deitke et al. (2022). and filter low-quality and texture-less cases. For each object, we render 12 views, using the Cycles render engine from Blender) For each viewpoint, render images under different lighting conditions. Six images are illuminated by randomly sampled high-dynamic range (HDR) environment maps from Poly Haver] which offers a collection of 718 varied environment maps. The last image illuminated by two point light sources randomly positioned on a surrounding shell. Our Arb-Objaverse dataset ends up with 5.7 million rendered RGB images along with their intrinsic components. For training, further enhance the variability by combining this dataset with G-Objaverse and ABO, Fig. offfers a visualization and comparison among these datasets.
https: htt:::/wwwwwwo.. www.:blender.com /rg/ https://polyhhate com/
4

---

Published as a conference paper at ICLR 2025
HDR maps point lights
ABO
Multiple illuminations
Universal objects
A12 Objeveru
G-Objauerse
Arb-Objauerse
X Multiple Illuminations Universal objects
Multiple Illuminations
Albedo/ Normal Metallic/ Roughness
Universal objects
Figure 3: Overview of the Arb-Objaverse dataset. Our custom dataset features diverse collection of objects rendered under various lighting conditions, accompanied by their intrinsic components.
3.3 ARCHITECTURE AND TRAINING
Given an arbitrary number of views from single to multi-view images, IDArb generates multi-view consistent intrinsic maps under unconstrained illumination using a text-guided diffusion model. We base our model on the pre are-trained Stable Diffusion (SD) (Rombach et al.) (2221) model to capitalize on its robust prior knowledge from RGB domain. Different from the 3-channel RGB images, intrin- sic components possess higher channel dimensions and cannot be directly processed by original SD model. To repurpose the VAE original SD for new intrinsic modalities, we divide intrinsic com- ponents X into three triplets: albedo A, normal N and B M, R. OJ. where M is metallic, R is roughness and O is left unused. Each triplet latent is channel-concatenater with the Gaussian noise for denoising. Specific text prompts for each triplet, i.e., 'albedo', 'normal', 'metallic&&roughness', are devised to indicate denoising targets.
Cross-view Cross-component Attention. In real-world scenarios, users may capture multiple im- ages of an object, making essential for the model to handle an arbitrary number of input views and ensure consistent results across all views. is also crucial for 3D reconstruction to have these consistent decomposition results as material guidance. To address this, we propose cross-view at- tention module within the original attention block of UNet. As shown in Fig. 2 we concatenate input features from each view, enabling the attention operation to be performed across views. This allows the model to leverage multi-view information reduce ambiguity and enforce consistency across different viewpoints.
The reflected color results from the interplay between incident light, material properties, and the surface shape. For instance, a convex shape with a dark color increases the likelihood of dark albedo. To better capture these relationships, we propose to model the joint distribution of intrin- sic components rather than predicting them separately. Inspired by Wonder3D (Long et al., 2023) and Wizar (Fu et al., 2024), we adopt cross- component attention via repurposing the vanilla self-attention module to fuse global interactions between different intrinsic components. As demon- strated in Sec. [4.3, exchanging information between components effectively reduces decomposition uncertainty, especially for roughness and metallic.
and View-Adapted Training, Multi-view images captured in uncon- trolled environments often experience varying lighting conditions, making it essential for algorithms to handle such differences effectively. To address this, we propose illumination-robue data aug- mentation strategy, where multi-view images are sampled from various lighting conditions during training. These conditions include range of setups, such uniform ambient light, HDR envi- ronment maps, and point light sources. At each training step, given N, views and N, illumination variations for each instance the dataset, randomly sample N images input. This allows
Mlamination-AAugeent
5

---

Published as a conference paper at ICLR 2025
simulate complex input scenarios, including same- some-view-die diiferentt-lllmminaaae different seetrenttview-seee view-same illumination, and different-viee seterrettview-diffeeeet www---fffeen--lllmmnnee illumination, thus enhancing the diversity of the training data. As result, model learns to distinguish different lighting conditions without the need for manually crafted modules, effectively leveraging photometric cues from multi-light captures to achieve robust intrinsic decomposition. It also shows superior generalization capability handle unseen lighting conditions at inference time.
However, training with fixed N input images leads to downgraded performance when only one view is given (as shown in Sec. 4.3 We suppose that this may be because multi-view training guides the model to focus more on cross-view information to infer intrinsic information, while single- image decomposition requires the learning of general object material priors. To overcome this, we introduce view-adapted training strategy, that swaps between multi-input and single-image settings. By incorporating this approach, our model gains robust generalization capability with an arbitrary number of input views.
Noise Scheduler. The original SD model uses the scaled linear noise scheduler, which prioritizes generating high-frequency details and allocates fewer steps to low-frequency structures. However, this approach limits model's performance in intrinsic decomposition task, as the structure of intrinsic components, particularly metallic M and roughness R. differs significantly from input RGB images. Inspired by (Shi et al. (2023), shift the noise scheduler toward higher noise levels. As shown in Sec. 4.3 increasing the number of high-noise steps significantly improves the prediction of metallic and roughness components.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Implementation Details. We finetune the UNet from the pretrained Stable Diffusion with the zero
terminal SNR schedule (Lin et al. 2024). We utilize the v-prediction as training objective and the AdamW optimizer with a learning rate of x 10-4. The model is trained on downsampled 256 x 256 resolution over 80, 000 steps. During training. the number of input images N randomly set or 1 per object. The entire training procedure takes approximately 4 days a cluster of 16 Nvidia Tesla A 1100 GPUs.
Baselines. We compare our method with two recent diffusion-based approaches: HID Kocsss ct al., 2024) and RGB++X (Zeng et al., 2024]. Since RGB+X is not yet publicly available, we training dataset. Additionally, cAnything (Chen et al., 2024) for albedo comparison and GeoWizard (Fu et al.) 2024] for normal comparison. We evaluate model in settings: (1) single-view setting, where each input im- age is processed independently, and (2) multi-view setting, where intrinsic components are jointly estimated from multiple views of each object.
re-implemented it and trained the model
include Intrinsi-
Metrics. For albedo evaluation, we use Peak Signal-to-Noise Ratio (PSNR) and Structural Similar- ity Index Measure (SSIM) (Wang et al. 2004).. Since albedo is defined up scale factor, apply scale-invariant PSNR metric by rescaling the predicted albedo as A' argmin, IIA. aAll2 Á. For surface normals, we measure Cosine Similarity. Mean Squared Error (MSE) is used to evaluate metallic and roughness components.
We evaluate the effectiveness and generalization capability of our model on both synthetic and real-world datasets. For synthetic data, we sample 441 objects from Arb- Objaverse and G-Objaverse, selecting four viewpoints for each object. For real-world data, collect set of images from Pixabay] All evaluations are conducted at a resolution of 512 x 512.
Evaluation Dataset.
4.2 EXPERIMENTAL RESULTS
Results on Synthetic Data. We present quantitative results in Tab. D where our method consistently achieves the highest accuracy across all metrics. Fig. 4 displays Visual comparison of our method in single-view setting against baseline methods.
https://pixabay com/

---

Published as a conference paper at ICLR 2025
Input
IID
RGB--X -X
IntrinsicAnything
Ours
GT
(a) Albedo estimation. Our method effectively removes highlights and shadows.
Input
RGB-XX
GeoWizard
Ours
GT
Normal estimation. Our method gives shape geometry while correctly predicting flat surface.
Input
IID
RCB X
Ours
GT
(c) Metallic estimation. Our method outperforms IID and RGB with plausible results free of inter- ference from texture patterns and lighting.
Input
IID
RGB-X
Ours
GT
(d) Roughness estimation. Our method outperforms IID and RGB++X with plausible results free of interference from texture patterns and lighting.
Figure 4: Qualitative comparison on synthetic data. IDArb demonstrates superior intrinsic esti- mation compared to all other methods.
Table Quantitative evaluation of IDArb against baselines. IDArb consistently achieves the best results among all albedo, normal, metallic and roughness metrics.
Albedo
Normal
Metallic
Roughness
SSIM
PSNR
Cosine Similarity 1
MSE 1
MSE 1
IID
0.901
27.35
0.192 0.162
0.131 0.347
RGB+>X 0.902
28.09
0.834
IntrinsicAntthing
0.901
28.17
GeoWizard
0.871
Ours(single)
0.935
32.79
0.928
0.037
0.058
Ours(multi)
0.937
33.62
0.941
0.016
0.033
For albedo estimation (Fig, 4a), our method effectively removes highlights and shadows, whereas IID and RGB+XX tend to retain lighting effects the albedo, and IntrinsicAnything produces un- realistic results for metallic surfaces. In normal estimation (Fig. 4b method provides sharp and accurate geometry, while RGB++X suffers from interference of object textures, and GeoWizare shows blurred details since it evaluates a number of samples and takes their mean. For metallic and roughness estimation (Fig. 4 and Fig. 4d)), our method delivers more plausible results, eliminating interference from texture patterns and lighting. Additionally, we observe that incorporating multi- view inputs significantly enhances metallic and roughness predictions, as they provide additional information resolve material ambiguities.
Results on Real-worid Data. We present qualitative results real- data in Fig ald compare our method with IntrinsicAnything for albedo estimation. IntrinsicAnything predicts overly dark albedo for metallic objects and produces blurry details (such as the toy's mouth in the third row),

---

Published as a conference paper at ICLR 2025
Input image
IntrinsicAnything
Ours (Albedo, Normal, Metallic, Roughness)
Figure 5: Qualitative comparison on real-world data.
wolaiiggotoo buining
W/S meeecommpeee sttetien
schediier
Input
sttention
Alaeto
Homeal
Mealc
Raghtns
(a) Ablation on compsnnt attention.
(b) Ablation on training strategy
Figure 6: Ablative studies on (a) cross-component attention and (b) training strategy.
leading loss of fidelity. In contrast, model generates accurate and convincing decompositions with preserved details. Despite being trained on synthetic data, IDArb generalizes well to real-world images. Additionally, we conduct experiments on standard benchmarks, MIT-Intrinsi (Grosse al.
2009) and Stanford-ORB (Kuang et al. 2023), with results presented in Appendix. D
4.3 ANALYSIS AND ABLATIVE STUDY
Ablation on Cross-component Attention. To assess the effect of cross-component attention, we also trained our model without cross-component attention mechanism for comparison. As shown in Fig. 6a] exchanging information between different intrinsic components helps reduce material ambiguity, particularly for metallic and roughness, which are prone to uncertainty.
Ablation Training Strategy. Fig. 6bb shows ablative studies multi-single view interleaved training strategy and the noise scheduler. Training exclusively on multi-view inputs leads to perfor- mance degradation for single-image inputs, as these two settings emphasize different capabilities of the model, as discussed in Sec. 3.3 Additionally, shifting noise scheduler towards high noise level helps the model better adapt to intrissic domains.
Analysis of Viewpoints and Lighting Effects. We analysis the effeets of the number of viewpoints and lighting conditions on our custom dataset. We evaluate our model with 1, 2, 4, 8, and 12 view- points under 1, 2 and 3 lighting conditions. As shown in Fig. increasing the number of viewpoints
8

---

Published as a conference paper at ICLR 2025
viess jess more)
Views less morel
views lles more]
vews tess morc)
Lights lines momeee)
3.05
135
1227
223
3.03
260
Albedo
Normal!
Metallic
Roughtssss 1
Figure 7: Effects number of viewpoints and lighting conditions. We find increasing the number of viewpoints and the lighting conditions generally improves decomposition performance.
Table 2: Quantitative results for photometric stereo on NeRFactor We evaluate performance using 2, 4, and 8 OLAT images, and achieve the best performance among all compared methods.
# OLAT Images
2
4
8
Methods
Albedof
Normalf
Albedot
Normalf
Albedot
Normalf
1ID
22.23
22.40
22.86
RGB +*X
21.29
0.71
22.08
0.77
23.29
0.81
SDM-UmiPS
22.95
0.74
23.20
0.76
23.37
0.81
Ours
23.50
0.83
23.64
0.84
25.15
0.85
or lights generally improves prediction accuracy. For metallic and roughness predictions, multi-light captures are particularly effective in disentangling these components from lighting effects. Empiri- cally, performance gains from adding more viewpoints diminish beyond eight viewpoints. Further details provided in Appendix. B
More Results. Additional multi-view input results provided in Appendix, E and supplementary video. For more real-world data results, please refer to Appendix.
4.4 APPLICATIONS
IDArb offers valuable intrinsic priors for various downstream applications. Here, demonstrate the model's ability handling single-image relighting and material editing, and photometric stereo problems. Additionally, we show that our generated intrinsic decompositions enhance the results of optimization-basee inverse rendering.
Single-image Relighting and Material Editing. Once high-quality intrinsic components are ob- tained, method enables relighting of captured images under novel illumination. Additionally, we can optimize the lighting the original scene and perform material editing. Specifically, we represent environment lighting cube map and adopt differentiable split-sum approximation in NVDiffRec (Munkberg et al. 2022) to optimize its parameters. Fig. 8 showcases our relighting and material editing results.
(a) Captured real data
(b) Relighting
(c) Material editing
Figure 8: Relighting and material editing results. From in-the-wild wild captures (a), our model allows for relighting under novel illumination (b) and material property modifications (c).
Photometric stereo. Photometric stereo is long-standing challenge in computer vision, aiming to deducing the surface normal and albedo from images captured under varying lighting conditions with fixed camera. We evaluate our method under the harsh One-Light-Atte Aeaght-AA-a-TTiee Time (OLAT) condi- tion, where each image is illuminated by single point light source without ambient illumination, leading to hard cast shadows. We additionally include SDM-UniPS (Ieehata 2023) for comparison, which is specifically designed and trained for this task. We conduct experiments on the real-world Opennllumination dataset (Liu et al. 2024a) and the synthetic NeRFactor dataset (Zhang et al.

---

Published as a conference paper at ICLR 2025
NVDiffRecMC NVDiffRecMC w// ours
GT
NVDiffRecMC
NVDiffRecMC w// ours
GT
Relit
Figure 9: Optimization-based inverse rendering results. Our method guides NVDiffecMC gen- erate more plausible material results.
Table 3: Ablation on IDArb pseudo labels for optimization-based inverse rendering on NeR and Synthetic4Relight datasets.
Nerfactor
Synthetic4Relight Albedo (scaled)
Albedo (raw)
Albedo (scaled)
Relighting
Albedo (row)
Relighting
Roughness
NVDiIRecMC NVDinRecMC Ours
17.89
25.88
22.65
17.03
29.64
24.05
0.046
20.90
26.61
27.20
31.01
0.014
2021bb.
Quantitative results NeRFactor summarized in Tab. and qualitative results present in Appendix. C Although our model is not explicitly trained for this setting, it still delivers reasonable estimates, particularly when the number of input images is limited.
Optimization-based Inverse Rendering. Our method can be used as a prior to enhance optimization- inverse rendering techniques. Specifically, we decompose each training image into its corresponding intrinsic components and treat these components as pseudo-material labels. We adopt NVDiffRecMC (Hasselgren et al.) 2022) as the codebase for our experiments, as em- ploys the same PBR material model as method. During each iteration, tional L2 regularization term between the intrinsic components predicted by NVDiffRecMC and those predicted by method ensure physical plausibility. Tab. 3 presents material estimation and relighting results on these dataset. As illustrated in Fig. 19 our method significantly mitigates the color-shifting issue in the reconstructed albedo from NVDiffRecMC, leading improved results in relighting tasks.
introduce addi-
CONCLUSION
In this paper, we present IDArb, that solves intrinsic decomposition via a feed-forward diffusion pipeline. Our method can process arbitrary images captured under unknown and varying illumina- tions and estimate consistent intrinsic components, including albedo, normal, metallic and rough- ness. The cross-component attention module and illmination--ugmeentd training further enhance model's ability to reduce ambiguity, fostering more robust inverse rendering under complex, high-contrast lighting conditions.
Limitations and Discussions. While our method demonstrates strong generalization capabilities on real-world data, it faces challenges in accurately predicting material maps for intricate objects, such as corroded bronze statues with spatially varying metallic and roughness properties due to corrosion levels. Given that most synthetic data employ global metallic and roughness values, our method may oversimplify estimations for complex real-world objects. Future research directions could involve incorporating real data through unsupervised techniques. Moreover, the current implementation of cross-view attention concatenates all input views, leading to a complexity of ( and posing difficulties in handling dense input views with high resolutions. Future investigations could explore more efficient cross-view attention mechanism. Further discussion on failure cases can be found in Appendix F
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This project is funded in part by Shanghai Artificial Intelligence Laboratory, the National Key R&D Program of China (2022ZD0160011) the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK. Dahua Lin is a PI of CPII under the InnoHK.
REFERENCES
Jonathan T. Barron and Jitendra Malik. Shape, illumination, and reflectance from shading, 2020.
URL htttps://aaxiiiii org/ abs/2010. 03592
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi. Deep 3d capture: Geometry and reflectance from sparse multi-view images, 2020. URL URRpsttss:/aaxiv.coo
12642
Mark Boss, Varun Jampani, Kihwan Kim, Hendrik P.A. Lensch, and Jan Kautz. Two-shot spatially- varying brdf and shape estimation. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2020.
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P. A. Lensch. Nerd: Neural reflectance decomposition from image collections. In ICCV, pp. 12664-12674. IEEE, 2021a.
Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P. A. Lensch. Neural-pil pil: Neural pre-integrated lighting for reflectance decomposition. In NeurlPS, pp. 10691 10704, 2021b.
Brent Burley and Walt Disney Animation Studios. Siggraph, volume 2012, pp. 1-7. vol. 2012, 2012.
Physically-based shading at disney. In Acm
Chris Careaga and Yağız Aksoy. Intrinsic image decomposition via ordinal shading. ACM Transac- fions on Graphics, 43(1):1-24 November 2023. ISSN 1557-7368. doi: 10.1 10.1145/3630700. URL. http::/dxx doi 1ttg//0..11/333000
Xi Chen, Sida Peng. Dongchen Yang. Yuan Liu, Bowen Pan, Chengfei Lv, and Xiaowei Zhou. Intrinsicanything: Learning diffusion priors for inverse rendering under unknown illumination, 2024. https: wwwpiiwwwrrgggeee
wwg//bggg/a....... abs 17444.115999
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world 3d object understanding. CVPR, 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani. Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects, 2022. URL httpps://axxxi..
org/ abs wtw.b//222.000555
Xiao Fu, Wei Yin. Mu Hu, Kaixuan Wang. Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. In ECCV, 2024.
Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable 3d gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing. 1. 2023.
Roger Grosse, Micah K Johnson, Edward H Adelson, and William T Freeman. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In 2009 IEEE 12th International Confer- ence on Computer Vision pp. 2335-2342. IEEE, 2009.
11

---

Published as a conference paper at ICLR 2025
Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape. Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. 2xXiv:2206.033880 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 2020.
Satoshi Ikehata. Scalable, detailed and mask-free universal photometric stereo. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zex- iang Xu, and Hao Su. Tensoir: Tensorial inverse rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphies and interactive techniques, pp. 143-150, 1986.
Brian Karis and Epic Games. Real shading in unreal engine 4. Proc. Physically Based Shading
Theory Practice, 4(3):1, 2013.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-baset image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering, 2023. URL http:t://axxii..orggg./. wwwpaaiwwww..ro...aaeaaaeeeaaaae wbw./rgg/abb.///oo/
04079
Peter Kocsis, Vincent Sitzmann, and Matthias Nicßner. Intrinsic image diffussion for indoor single- view material estimation, 2024.
12274
Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew Davison. Esch- ernet: A generative model for scalable view synthesis. arXiv preprint arXiv:2402.03908, 2024.
Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov. Neroie: neural rendering of objects from online image collections. ACM Trans. Graph.,
41(4):56:1-56:12 2022.
Zhengfei Kuang, Yunzhi Zhang, Hong-Xing Yu, Samir Agarwala, Elliott Wu, Jiajun Wu, et al. Stanford-orb: a real-world 3d object inverse rendering benchmark. 2023.
Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and spatially-varying reflectance from single image. In SIG- GRAPH Asia 2018 Technical Papers, pp. 269. ACM, 2018
Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chan- draker. Inverse rendering for complex indoor scenes: Shape, -aatiialll--rriinn lighting and svbrdf from single image, 2019. htLhttps:: http://wwwww.....////o 905. 02722
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed, 2024. URL https: wwwaaiivvorrrr. www./bsg./aa.o....ooooo........a httaa/bb//33...0 2305 08891
Isabella Liu, Linghao Chen, Ziyang Fu, Liwen Wu, Haian Jin, Zhong Li, Chin Ming Ryan Wong. Yi Xu, Ravi Ramamoorthi, Zexiang Xu, and Hao Su. Openilluminatoo: A multi-illuminttion dataset for inverse rendering evaluation on real objects, 2024a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3; Zero-shot one image to 3d object, 2023.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview images from a single-view image, 2024b. URL
httpps:/arxxv.orrr////333.....
03453
12

---

Published as a conference paper at ICLR 2025
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310. 2023.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Im- ages. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8280-8290, June 2022.
Fred E Nicodemus. Directional reflectance and emissivity of an opaque surface. Applied optics, 4 (7):767-775, 1965.
Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 9914-9925, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Shen Sang and M. Chandraker. Single-shot neural relighting and svbrdf estimation. In ECCV, 2020.
Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. Learning non-lamhertian object intrinsics across shapenet categories, 2016. URL /
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++: single image to consistent multi-view diffusion base model, 2023.
Yawar Siddiqui, Tom Monnier, Filippos Kokkinos, Mahendra Kariya, Yanir Kleiman, Emilien Gar- reau, Oran Gafni, Natalia Neverova, Andrea Vedaldi, Roman Shapovalov, and David Novotny. Meta 3d assetgen: Text-to-mesh generation with high-quality geometry, texture, and phr materi- als. arXiv, 2024.
Cheng Sun, Guangyan Cai, Zhengqin Li, Kai Yan, Cheng Zhang, Carl S. Marshall, Jia-Bin Huang, Shuang Zhao, and Zhao Dong. Neural-pbir reconstruction of shape, material, and illumination. In ICCV, pp. 18000-18010. IEEE, 2023.
Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint ar arXiv:2100 10689, 2021.
Zhou Wang, Alan C Bovik. Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Felix Wimbauer, Shangzhe Wu, and Christian Rupprecht. De-rendering 3d objects in the wild, 2022. URL htt.pps://aasiv.org///aatte 2201 02279
Shangzhe Wu, Ameesh Makadia, Jiajun Wu, Noah Snavely, Richard Tucker, and Angjoo Kanazawa. De-rendering the world's revolutionary artefacts, 2021. URL. htt.hhttp::://xxi..orr///
2104.03954
Tong Wu, Jiaqi Wang, Xingang Pan, Xudong Xu, Christian Theobalt, Ziwei Liu, and Dahua Lin. Voxurf: Voxel-based efficient and accurate neural surface reconstruction, 2023, URL https:
//axxiv. org abs 12697
Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions Graphics (TOG), 2024.
13

---

Published as a conference paper at ICLR 2025
Weicai Ye, Shuo Chen, Chong Bao, Hujun Bao, Mare Pollefeys, Zhaopeng Cui, and Guofeng Zhang. IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2023.
Renjiao Yi, Chenyang Zhu, and Kai Xu. Seakly-supervised single-view image relighting, 2023.
URL htttp:://axxii.
abs/2303 13852
Zheng Zeng. Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy Yiwei Hu, Fujun Luan, Ling-Qi Qi Yan, and Miloš Hašan. RGB ++ X: Image decomposition and synthesis using material- and lighting-aware diffusion models. arXiv preprint arXiv:2405.0066 2024.
Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely, Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5453-5462, 2021a.
Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In IEEE Conf. Comput. Vis. Pattern Recog., 2022a.
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large generative model for creating high-ques high-quality 3d assets, 2024.
ab44/405..22 2200. 13897
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models, 2023.
Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. Nerfactor: neural factorization of shape and reflectance under unknown il- lumination. ACM Transactions on Graphics, 1-106)):18,,,) December 202 ISSN 1557-7368. doi: 10.1145/3478513.3443
URLhttee dx doi org. /ttp::///....o...g///1111..111.1 110.....1111447811 0101188113.3444999 3480496
Yuanqing Zhang. Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. indirect illumination for inverse rendering. In CVPR, 2022b.
Modeling
Rui Zhu, Zhengqin Li, Janarbek Matai, Fatih Porikli, and Manmohan Chandraker: Irisformer: Dense
vision transformers for image inverse rendering in indoor scenes, 2022. URL https:
/ //axxiv.oro/abs/2200 08423
A PRELIMINARY
A.1 IMAGE DIFFUSION MODEL
In Denoising Diffusion Probabilistic Models (DDPM) (Ho al. 2020), a forward diffusion process is defined, gradually introducing small amounts of Gaussian noise to the sample each timestep, represented by (x = (x; vI- -, 3,1), where represents the timestep and acts as the variance scheduler. To recover samples from the random noise, DDPM learns to model the reverse diffusion process as po (x+-1 xx) = A 1x f), Ex(,, f)) and construct Xy through iterative denoising.
2021 employ an encoder & to compress the input image X E xx into a latent vector z E R fr/8x W/8xx before performing the diffussin process in the latent space. Following denoising, the latent representation is then converted back to pixel space through a decoder i D(z)).
Stable Diffusion (SD) (Rombach et al.
For conditional generation, the training objective of Stable Diffusion (SD) is formulated as:
: W co (o
where is uniformly sampled from (1...,,T) (v) represents the encoding of the condition y and cg is implemented as UNet.
14

---

Published as a conference paper at ICLR 2025
A.2 INTRINSIC COMPONENTS FORMATION
Our image formation is based on the classic rendering equation (Kajiya 1986) to ensure physical correctness. For point with surface normal the incident light intensity this point is denoted as (,) where represents the incident light direction. The Bidirectional Reflectance Dis- tribution Function (BRDF) (Nicodemus) 1965), denoted as fr(wo, wis wx; describes the reflectance properties of the material when viewed from direction wo. The observed light intensity (wo: x) is calculated over the hemisphere f2 {w; : wa 0} as follows:
Lo(wei x) = Li(wg fr (Wos wa (x3))((;; n)dw;.
In our approach, we aim to recover the object's surface normal and BRDF material from the ob- served color on the left-hand side of Eq.3] which are independent of illumination and view direction. We adopt the Disney Basecolor-Metie
model(Buriey & Studios 2012) for BRDF parametrization, which comprises the following components: albedo, representing the base color; roughness, con- trolling the diffuse and specular response; and metallic, governing the specular reflection.
Specifically, given a single RGB image I "xxxx we aim to jointly estimate the surface normal N I 11 ( albedo A E // rroughness R E R/'xWx1 and metallic M "
B DETAILS ABOUT THE EFFECTS OF VIEWPOINTS AND LIGHTING
We present the numerical performance results across varying numbers of viewpoints (# V) and lighting conditions (# L)), as shown Tab. 4 to
Table Albedo Performance across different numbers viewpoints (# V) and lightings L).
Table 5: Normal Performance 1 across different
numbers of viewpoints V) and lightings (# L).
V
4
8
12
#L
2
4
8
12
1
29.16
28.72
30.12
30.49
30.77
0.909
0,910
0.925
0.930
0.932
2
29.96
30.26
30.96
31.13
31.26
2
0.922
0.927
0.930
0.933
0.934
3
30.25
30.73
31.16
31.33
31.40
3
0.926
0.931
0.931
0.934
0.935
Table 6:
Metallic Performance across different
Table Roughness Performance | across different
numbers viewpoints (# V) and lightings (# L).
numbers of viewpoints (# V) and lightings (# L).
V
4
8
12
4
8
12
#
I.
0.105
0.116
0.068
0.059
0.050
1
0.049
0.050
0.024
0.019
0.021
2 0.061
0.068
0.047
0.044
0.042
2
0.043
0.026
0.019
0.016
0.015
3
0.061
0.056
0.048
0.045
0.040
3
0.031
0.022
0.016
0.014
0.013
C
ADDITIONAL RESULTS ON PHOTOMETRIC STEREO
We present qualitative results of photometric stereo in Fig, I0
D ADDITIONAL RESULTS ON REAL-WORLD DATA
We evaluate our method on two real-world benchmarks: MIT-Intrinsic (Grosse et al. 2009) and Stanford-ORB (Kuang et al. 2023). For MIT-Intrinsic, we compared our albedo estimation results with IntrinsicAnything (Chen et al.) 2024), shown in Tab, 8 and Fig. [1 For Stanford-ORB, presented results for normal estimation, albedo estimation, and re-rendering, comparing our method with StableNormal (Ye et al., 2024) and IntrinsicNeRF (Ye et al. (023). as shown in Tab. p For the re-rendering evaluation, we utilized the ground truth environment maps to render our decomposition results and compared them with the original images.
Additionally, we present qualitative results on real-world data from the Internet in Fig. and Fig. 13
15

---

Published as a conference paper at ICLR 2025
OLAT images
Albedo
Normal
Openillumination
NeRFactor
Figure 10: Photometric stereo results using OLAT images in Openlllumination and NeRFactor.
Table 8:
Quantitative comparisons on MIT-Intrinsic.
SSIM1
PSNR1
LPIPS
Ours
0.876
27.98
0.117
IntrinsicAnything
0.896
25.66
0.150
Table 9: Quantitative comparisons on Stanford-ORB.
Normal
Albcdo
Re-rendering
Cosine Distance
SSIM
PSNR1
LPIPS
PSNR HH
PSNR-L*
SSIM
LPIPS 1
Ours(single) (urs(multi)
0.041
0.978
41.30
0.039
24.11 24.36
31.28
0.969
0.024
0.029
0.978
41.46
0.038
31.43
0.970
0.024
StableNormal
0.038
IntrinsicNeRI
0.981
39.31
0.048
E ADDITIONAL RESULTS ON MULTI-VIEW INPUTS
We present additional results on multi-view input in Fig. 14
F FAILURE CASES
Several failure cases are illustrated in Fig. 16 First, our model struggles with outdoor scenes, it is primarily trained on object-centree data. While the model exhibits some generalization capability, its performance degrades in these scenarios. Second, when the model is faced with text, the decompo- sition fails to recover the correct text structures. Finally, in the third row, the model produces overly simplified outputs in certain cases, failing preserve subtle material details, such the metallic
16

---

Published as a conference paper at ICLR 2025
Input Image
Intrinsic Anything
Ours
Ground Truth
Figure 11: Qualitative comparison on MIT-Intrinsie Grrose al. 2009) with Intrinsi- cAnything (Chen et al. 2024). Input image and ground truth have been contrast-adjusted for better visibility.
features of telephone. This issue arises from the synthetic training data, which often contains simpler material variations, leading the model to overly simplify fine-graing ane-graieed material properties.
G GENERALIZATION TO SCENE-LEVEL DATA
Despite not being explicitly trained on such datasets, our model demonstrates generalization ability in outdoor and indoor scenes. We provide qualitative results in Fig. 17 Fig. 18 and Fig. 19
17

---

Published as a conference paper at ICLR 2025
Input
Albedo
Normal
Metallic
Roughness
Figure 12: More results on real-world data.
18

---

Published as a conference paper at ICLR 2025
Input Image
Albedo
Normal
Metallic
Roughness
Recon
Relit 1
Relit 2
Relit 3
Recon
Relit 1
Relit 2
Relit 3
Recon
Relit
Relit
Relit
Recon
Relit 1
Relit 2
Relit 3
Figure 13: More results on real- world data. We also provide the reconstructed and relighting images.
19

---

Published as a conference paper at ICLR 2025
Input
Albedo
Normal
Metallic
Roughness
Figure 14; More results on multi-view data. 20

---

conference paper at ICLR 2025
Published as a conference
Input Image
Normal
Merallea
Pougmmess
Albedo
Figure 15: Moulkiiaa images with adtreme lighting variation. For each scene in NeRD fataser IBoss et all. 0021a1 we imput views.
21

---

Published as a conference paper at ICLR 2025
Input Image
Albedo
Normal
Metallic
Roughness
GREEN TEA
TEA
Figure 16: Failure cases.
22

---

Published as a conference paper at ICLR 2025
Input Image
Albedo
Normal
Metallic
Roughness
Figure 17: Results on Mip-NeRF 360 (Barron et al. 2022 (Part 1, outdoor). We input views for each scene.
23

---

Published as a conference paper at ICLR 2025
Input Image
Albedo
Normal
Metallic
Roughness
Figure 18: Results on Mip-NeRF 360 each scene.
Barron et al. 2022) (Part 2, indoor). We input 4 views for
24

---

Published as a conference paper at ICLR 2025
Input Image
Albedo
Normal
Metallic
Roughness
Figure 19: Results on indoor and outdoor scenes. Input images collected from the Internet.
25


Reviews:
[{"review_id": "QTJ5xp0yFt", "reviewer": "Reviewer_VXK9", "summary": "This paper introduces a diffusion-based method that uses color images as input to predict material properties, including albedo, normal, metallic, and roughness. By leveraging the predicted material information and multi-view images, the method can support various applications such as relighting, material editing, photometric stereo, and inverse rendering. However, the proposed attention mechanism shows substantial similarity to prior work (Wonder3D), and the dataset lacks sufficient novelty and contribution.", "questions": "1. The process of 3D reconstruction from the generated multi-view images with material properties needs further elaboration. Providing more details on the reconstruction workflow, including how the material data (e.g., albedo, normal, metallic, roughness) contributes to the 3D recovery, would strengthen the clarity.\n\n2. The functionality of the cross-view, cross-component attention mechanism also requires a more in-depth explanation. Figure 2 presents only a limited structural overview, so further details on the components and interactions within this attention scheme would greatly enhance understanding.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper trains a diffusion model to predict material properties from color images, demonstrating strong performance compared to previous methods.\n\n2. Extensive experiments validate the approach, showcasing its effectiveness across various applications.\n\n3. The paper is well-organized and easy to follow.", "weaknesses": "1. The primary contribution of this paper is extending the original stable diffusion model from single-view, single-domain to multi-view, multi-domain processing. To accomplish this, the authors introduce a cross-view, cross-component attention mechanism. However, this attention method bears a strong resemblance to the multi-view, multi-domain attention proposed in the CVPR 2024 paper *Wonder3D*, which also extends stable diffusion to predict multiple domains. The authors have not cited or discussed *Wonder3D*, making such discussion necessary.\n\n2. Given the similarities between the cross-view, cross-component attention in this work and *Wonder3D*, the other key contribution is a synthetic dataset designed for intrinsic image decomposition tasks. However, the dataset’s objects are sourced from Objaverse and rendered under various lighting conditions, a technique widely used in other studies. This limits the dataset’s novelty, making it challenging to consider it a significant contribution.\n\n3. The supplementary video provides only two samples, offering limited insight. It would be highly beneficial to include additional comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LjKgRGwp1k", "reviewer": "Reviewer_zfJQ", "summary": "The paper proposes a method for intrinsic decomposition of object-centric images from single view or multi-view observations under varying lighting conditions, via training a diffusion model which is adapted to predicting intrinsic modalities via text prompt as switches, as well as utilizing cross attention mechanism between noised image latents of different modalities and views. The attention mechanism allows for the method to exploit dependency and consistency between intrinsic modalities and views, to achieve improved consistency and better disambiguate. The method is trained and evaluated on a new dataset rendered from Objaverse with physically-based materials and lighting, as well as real-world instances. The paper also includes ablation on model design, and applications to downstream applications.", "questions": "[1] Please include results of re-rendering in all of the samples using estimated intrinsic modalities, as well as more samples especially in challenging conditions. \n\n[2] Failure cases are to be included as well.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "[1] Novelty of the proposed method. The method borrows from previous methods on adapting diffusion models for intrinsic decomposition, but advances on improving the multi-view consistency by incorporating additional cross-attention mechanisms into the denosing UNet. Despite previous works which use similar attention mechanisms across multi-view inputs, the method is novel in its adaptation of the mechanism in this specific task, and is able to employ the mechanism to both cross-view and multi-task image latents.\n\n[2] Impressive results on object centric intrinsic decomposition. The paper showcases impressive results in this task compared to some of the very recent works, demonstrating the effectiveness of the proposed attention mechanism. Moreover, special training strategy has been designed to avoid overfitting to multi-view inputs (VS single-view), and to improve quality by switching the noise scheduler. \n\n[3] Applications to downstream applications. The paper showcases applications in downstream applications to enable relight and material editing, and application as prior for optimization-based methods.", "weaknesses": "[1] Limited novelty. The major novelty of the paper is in its cross-attention module, which is nodel in this formulation but not entirely new due to applications in other tasks (e.g. LRMs) by previous methods. In this case, the theoretical contribution of the attention model is limited, and the pipelines offers few other contributions besides the attention module.\n\n[2] Limited evaluation samples. The paper offers limited account of samples in evaluation, which do not cover challenging situations including (1) scenes of high secularity (e.g. a model car); (2) multi-view images of extreme lighting variation; (3) scenes with few views. More importantly, re-rendering results are not provided as a criterion to determine the challenging modalities including roughness and metallic, which in some cases may look correct but a slight error will contribute to artifacts in re-rendered results.\n\n[2] Evaluation on more datasets. Indoor inverse rendering has been a particular challenging domain than object-centric settings. It would be great to see evaluations on indoor setting (e.g. RGBX), if the authors are able to access sufficient indoor inverse rendering data for training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AKfe49ItI7", "reviewer": "Reviewer_cXk1", "summary": "This paper introduces IDVI, a diffusion-based multi-view model to decompose the intrinsic properties of the scanned subjective in a view-consistent manner. It leverages the commonly used Stable-Diffusion model and upgraded it with a cross-view attention and cross-domain attention to align the latents across the view/domains. The model is trained on the also proposed dataset: ARB-Objaverse, a large scale datasets rendered from Objaverse assets using multiple lighting conditions,  and achieves impressive performance in various settings.", "questions": "Overall, I think this paper is a complete work and has good performance. While it provides limited contributions on the technical part, it is a good practice of scaling up the system in the task of object intrinsic decomposition. I’m leaning to accept this paper. In addition to that, I’m happy to raise my score to a higher one if: 1) The authors will release the dataset/model and 2) The authors can show more comparisons on other domains.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The results from the experiments cases looks impressive and thoroughly conducted. The model can produce high fidelity normal/albedo/material decomposed that outperforms the baselines by a large margin.\n\nThe model is designed in a good shape, by leveraging the power of cross-view and cross-domain attention which are proven to be very effective in previous works.\n\nThe proposed ARB-Objaverse can largely benefit to the inverse rendering community, if it will be released to the public.", "weaknesses": "Issues (Ranked by priority): \n\nThe model architecture of this work is somewhat a combination of attention modules that proposed by previous works (such as GeoWizard). \n\nReproducibility: The model is trained on the proposed dataset ARB-Objaverse, which is rendered in a large scale. It will be hard for the community to reproduce the training if the dataset is not publicity available. \n \nAll of the experiments are conducted on object-centric cases only. While the model is trained by object data, it is also important to see how it can generalizes to other domains, for example indoor scenes (from which the RGB<->X is trained) \n\nThe Attention Block part in figure 2 is a bit confusing, a few comments that might make it more intuitive: \n\t(1) Add captions to explain what the images correspond to in the cross-component attention and cross-view attention (I would assume they are RGB/normal/material, and view 1/2/3 respectively)\n\t(2) Since the model is a combination of attention modules applied in different dimensions, the figure 2. in the paper of Group Normalization might be a good example of how to illustrate the operation in a very intuitive manner. \n\t(3) It is not clear how the attentions are assembled in the block. Is it follow by the order of: cross-component attention->cross-view attention->cross attention-> feed forward? If so, please clarify it in the figure.\n\nFigure 7: Please use consistent color tones to represent good / bad performance. i.e. inverse the color map of metallic and roughness comparison.\n\nMissing Citations: \n\tA couple of previous works such as NeRD, NeROIC, Neural-PIL also focuses on intrinsic decomposition on images under various illuminations; These works are also highly related to the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o8g8RyNnEj", "reviewer": "Reviewer_GJXm", "summary": "This paper presents IDVI, a diffusion-based model for intrinsic decomposition that accurately estimates surface normals and material properties from multiple images under varying illuminations. It introduces a novel cross-view attention module and a new dataset, ARB-Objaverse, enabling robust training and demonstrating superior performance on various tasks, including single-image relighting and 3D reconstruction.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A new dataset is contributed. \nThe paper presentation is good, with figures helpful for understanding. \nThe results are pretty good.", "weaknesses": "1. The method mainly targets single objects, limiting the applications to scenes. How about performance on scenes? Could you please discuss whether the method can apply on scenes? Does it need re-training? What kind of scenes (indoor, outdoor, 360 scenes, or multi-object scenes) that it can solve, with some examples to demonstrate?\n2. The method highly relies on supervised training data, while many other prior works are unsupervised/self-supervised. Could you please discuss and compare the pros/cons against self-supervised ones, and compare with them. E.g., DyNeRFactor, IntrinsicNeRF. \n3. Methods to compare are too few. There are many diffusion models to estimate normal, which should also be compared. Standard benchmarks such as MIT-Intrinsic should be tested to check whether the model is overfitted on training data. E.g. StableNormal. \n4. Figure 4 is compared to synthetic data, likely from the same dataset with training. That would be unfair to other methods. \nI am hesitant to give a positive score mainly because with a supervised method training on a proposed synthetic dataset, it seems to have limited technical contributions. Though the results are good, experiments and evaluations could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y0sJKTB6WK", "reviewer": "Reviewer_PUK6", "summary": "This paper proposes to utilize diffusion priors for BRDF estimation from arbitrary number of images of an object. The key technical contributions including a novel transformer that uses attention mechanism to model relations between images across different views and different intrinsic components and a new training dataset that can be used for high-quality BRDF estimation. Experiments on several popular synthetic dataset show that the proposed method achieves very impressive inverse rendering results. Moreover, those results are multi-view consistent and can be used to optimize a high-quality relightable 3D model.", "questions": "Overall I like this paper and hope that it can be accepted. My major question about the paper is experiments on real data. I hope this paper can include experimental results on Stanford-ORB and Objects with lighting datasets to verify the trained model's generalization ability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This is a technically sound paper with clear technical contributions. The novel dataset is built in a reasonable and thoughtful way. The novel attention module can successfully capture multi-view information and multi-channel information and shown to achieve better performance on synthetic datasets. \n\n2. From the results shown in the paper, the proposed method achieved very impressive inverse rendering results, with shadows and highlights being fully removed from the material parameters while preserving all texture details. In addition, the roughness and metallic values are piecewise smooth. It avoids the artifacts of only low roughness values near the specular highlights, which usually can observe in the optimization-based methods. \n\n3. Given that inverse rendering is a highly ill-posed problem, it makes a lot of sense to use a generative model to sample more visually appealing and higher quality results. I believe this paper is a solid and predictable step forward compared to RGB-X paper which only considers a single image.", "weaknesses": "1. Experiments on real data.\n\nMy major concern of this paper is lacking real-world data experiments. I specifically recommend Stanford-ORB dataset and , where we can easily get all the comparisons with state-of-the-art dense-view reconstruction methods, including geometry and relighting accuracy. Without that experiment, it is very difficult to verify if the proposed method actually generalize well to real-world data. \n\n2. The method section can be improved with more necessary details. \n\nI had a little bit difficulties to under Method 3 and Figure 2. I believe I eventually understood it but felt some modifications can make it easier to read. \n\nIn Figure 2, I assume D equal to 3 and presents the number of intrinsic components. This can be explicitly explained in Figure 2. The blue, green, dark green blocks look a bit too similar and we may consider using color with stronger contrast. Also the two attention matrix has very different color and it is unclear what the colors represent. For the top half of Figure 2, it is unclear how the image latents are combined with noisy triplet latents. I assume it is a concatenation. Also for the noisy triplet latent, it may be more informative to keep some original image information so we know what it represents. \n\nLine 246: Here, it is mentioned that each triple latent is concatenated with the Gaussian noise. Do we actually mean \"added\" instead of concatenated?\n\nIllumination-augmented training: it is a bit weird to me what we sent images with different illumination to the diffusion model as during real-world data capture, we usually assume the lighting is unchanged. I wonder how this training strategy may influence the generalization on real data. \n\n3. Quality of reference should be improved. \n\nFor the optimization-based inverse rendering, authors skipped a large number of recent works solving this problem, such as Nerfactor, NeRD, Neural-PIL, nvidffrec, nvdiffrecmc, MII, Neural-PBIR  and also many recent relightable 3D Gaussian papers. For learning-based inverse rendering, there are also many more SVBRDF estimation, lighting estimation, intrinsic decomposition papers that should be cited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)