Joint Shadow Generation and Relighting via Light-Geometry Interaction Maps
Authors: Shan Wang
Peixia Li
Chenchen Xu
Ziang Cheng
Jiayu Yang
Hongdong Li
Pulak Purkait
Venue: ICLR 2026 Poster (2026)
Keywords: Shadow Generation
Relight
Bridge matching
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We propose Light–Geometry Interaction (LGI) maps, a novel representation that encodes light-aware occlusion from monocular depth. Unlike ray tracing, which requires full 3D reconstruction, LGI captures essential light–shadow interactions reliably and accurately, computed from off-the-shelf 2.5D depth map predictions. LGI explicitly ties illumination direction to geometry, providing a physics-inspired prior that constrains generative models. Without such prior,  these models often produce floating shadows, inconsistent illumination, and implausible shadow geometry. Building on this representation, we propose a unified pipeline for joint shadow generation and relighting-unlike prior methods that treat them as disjoint tasks-capturing the intrinsic coupling of illumination and shadowing essential for modeling indirect effects. By embedding LGI into a bridge-matching generative backbone, we reduce ambiguity and enforce physically consistent light–shadow reasoning. To enable effective training, we curated the first large-scale benchmark dataset for joint shadow and relighting, covering reflections, transparency, and complex interreflections. Experiments show significant gains in realism and consistency across synthetic and real images. LGI thus bridges geometry-inspired rendering with generative modeling, enabling efficient, physically consistent shadow generation and relighting.

Published as a conference paper at ICLR 2025
JOINT SHADOW GENERATION AND RELIGHTING VIA LIGHT-GEOMETR, INTERACTION MAPS
Shan Wang
Peixia Li
Chenchen Xu'
Ziang Cheng'
Jiayu Yang'
Hongdong Li
Pulak Purkait'
Amazon
Australian National University
ABSTRACT
We propose Light-Geometry Interaction (LGI) maps, novel representation that encodes light-aware occlusion from monocular depth. Unlike ray tracing, which requires full 3D reconstruction, LGI captures essential light-shadon interactions reliably and accurately, computed from off-the-shelf 2.5D depth map predic- tions. LGI explicitly ties illumination direction geometry, providing physics- inspired prior that constrains generative models. Without such prior, these mod- els often produce floating shadows, inconsistent illumination, and implausible shadow geometry. Building this representation, we propose a unified pipeline for joint shadow generation and relighting-unlin unlike prior methods that treat them disjoint tasks-capturing the intrinsic coupling of illumination and shadow- ing essential for modeling indirect effects. By embedding LGI into bridge- matching generative backbone, we reduce ambiguity and enforce physically con- sistent light-shadow reasoning. To enable effective training, we curated the first large-scale benchmark dataset for joint shadow and relighting, covering reflec- tions, transparency, and complex interreflections. Experiments show significant gains in realism and consistency bridges geometry-inspired rendering with generative modeling, enabling efficient, physically consistent shadow generation and relighting.
synthetic and real images. LGI thus
INTRODUCTION
(a) Table insertion
(b) Bag insertion
Glass insertion
Two lights
(e) Two lights
Figure I: Effectiveness of our joint shadow generation and relighting pipeline. Our method produces realistic, texture-ave aattre shadows consistent with object and scene geometry, while preserving faithful relighting across diverse materials such as wood, leather, metal, and glass. (a-c) Multiple object interactions. (d-e) Generalization to multiple light sources.
Shadow generation and relighting are important tasks wide range of visual computing applica- tions, including virtual product placement, post-capture image editing, augmented reality, and digital content creation. Realistic shadow generation and relighting require reasoning about how light in- teracts with scene geometry. Traditional physically based rendering methods achieve this through explicit 3D reconstruction and ray tracing [harr et al. (2023): Keller (199)) but they are compu- tationally expensive and impractical in single-view settings. On the other hand, recent generative approaches, such as bridge matching Tasar et al. (2024); Chadebec et al. (2025) or diffusion Liu et al. (2020); Hong al. (2022); [ al. (2024a) Winter et al. (2024), can synthesize shadows and

---

Published as a conference paper at ICLR 2025
illumination from RGB inputs, but in the absence of physical constraints they often produce floating shadows, inconsistent illumination, or implausible geometry, particularly under complex lighting.
We introduce Light-Geomert Interaction (LGI) maps, novel 2.5D representation that directly encodes occlusion relationships between light and geometry from monocular depth. LGI maps differ from traditional ray tracing, which requires full 3D geometry, by providing compact, differentiable approximation of light transport suitable for end-to-end learning. They also improve upon prior depth-based conditioning by explicitly coupling depth with light direction, serving a physics- inspired prior that constrains generative models while remaining computationally efficient.
Building on LGI maps, we design unified pipeline for joint shadow generation and object-level relighting in scene-aware settings, where a newly inserted object must cast shadows, produce re- flections, and be illuminated consistently with the background light. Prior works typically treat these tasks independently: shadow models Liu et al. (2024a); /Zao et al. (2025) Tasar et al. (0024); Chadebec et al. (2025) generate planar or template-he craelate-based shadows, while relighting methods Zhang al. (2025a): Kim et al. (2024) focus solely on object reflectance. However, light and shadows are intrinsically coupled accurate modeling requires reasoning about direct illumination, secondary reflections, and inter-reflections simultaneously. Our joint formulation enforces this coupling, yield- ing coherent shadow-light interactions that cannot be achieved when the two tasks are separated.
To support this task, construct large-scale synthetic dataset, ShadRel, for joint modeling of shadows and relighting. Unlike existing datasets that emphasize either hard shadows Liu et al. (024a); Tasar et al. (2024) object-only relighting (Helou et al. (2020); (Kim et al. (2024), Sha dataset includes soft shadows, reflective and transparent materials, and inter-reflections, providing a comprehensive resource for training and evaluation.
Through extensive evaluation, we demonstrate that our framework achieves state-of-the-art (SOTA) performance across wide range of challenging visual scenes, bridging the gap between geometry- free neural rendering and computationally intensive physically-base the single-object and single-light level, it naturally extends to multi-object editing and multiple light sources (Fig. T.. Our framework thus offers an efficient yet physically inspired alternative suitable for practical shadow-aware image editing and realistic relighting.
approaches. Though designed
Our contributions are summarized as follows:
Interaction Map: a novel light-aware occlusion representation that bridges the gap between geometry-inspired rendering and unconstrained generative models.
Joint shadow-relighting pipeline: a unified framework to couple shadow generation and relighting, enabling physically consistent reasoning about direct lighting, secondary reflec- tions, and inter-reflectioos.
ShadRel dataset for coupled light transport: a large-scale dataset designed to capture chal- lenging illumination effects, supporting rigorous training and evaluation.
2 RELATED WORK
Classical rendering techniques such as ray tracing Wald et al. (0001): Pur- cell et al. (2005) and path tracing Christensen et al. (2018): Lafortune & Willems (1996) accurately simulate light transport but require detailed 3D geometry and are computationally expensive. Neu- ral methods attempt reconstruct geometry from multi-view inputs Zhao al. (2024); Lin et al. (023). making them unsuitable for single-view settings. Alternative approaches bypass full 3D reconstruction. SSN Sheng et al.] (2021) estimates ambient occlusion from object masks, and pixel height maps approximate geometry for shadow simulation Shhnng et al. (2022) 2023). GAN-based models Zhang et al. (2019): Liu et al. (2020) use adversarial training and spatial attention to syn- thesize shadows, while diffusion liffusion-based Hong et al. 20 Liu et al. (2024a): Winter et al. (2024) and bridge-based methods Tasar et al. (2024); Chadebec et al. al. (2025) generate shadow regions with adjustable placement. Yet most of these methods are fundamentally 2D, relying templates or bounding boxes. Hong et al. (002)) decompose synthesis into shape estimation and filling, while Zhao et al. (205)) use rotated boxes and templates. Such heuristics break down in complex scenes with ambiguous occlusion. Depth-conditioned works Grifitths et al. (2022); Kocsis et al. (2024) leverage predicted depth to approximate geometry but require multi-stage pipelines and shading
Shadow Generation.
2

---

Published as a conference paper at ICLR 2025
notations. In contrast, we treat predicted depth 2.5D structural cue, embedding it directly into LGI maps to provide a differentiable prior for sight-shadow interactions in an end-to-end setting.
Image Relighting. Relighting methods based inverse rendering and reflectance models Wenger ct al. (005):: Wang et al. (2020); Zeng et al. (2024a); Zhu et al. (2022), primarily focus on regress-
ing radiance object surfaces, while largely ignoring cast shadows.
More recent deep learning
approaches Bhattad et al.] p0 Xing et al. (2024); Kim et al.]
(2024);
Liang et al. (2025): ) Zeng
al. 2024b); Zhang al. (2025b) Lin al. (003); Liang et al. (025) Zhu al. (2022) estimate
or disentangle image intrinsics (e.g.. albedo, normals, roughness, metallic) to constrain relighting, but these typically require strong supervision. ZZhang et al. (2025a) instead enforces linear con- sistency between appearances under different illuminations and their mixture, but requires training scenes with least two distinct light sources. Some recent work highlights the role of shadows in achieving consistent relighting. Baatinng-Fooierrriei Coortee-Chhuinarte et al. (2024) shows that coarse shadow modeling improves illumination quality, underscoring the need for light-shadow coherence. Still, most prior methods focus object illumination alone, overlooking the coupled effects of shadows.
Toward Joint Approaches. Although shadow synthesis and relighting are tightly coupled in real scenes, very few methods tackle them together. Most follow sequential design: for example, Griffiths et al. ) predicts shadow shape before relighting, and Shortie-CChuinan et al. (2024)
uses coarse shadows guide relighting. These pipelines remain disjoint and rely handerafted intermediate steps. Recent generative frameworks such as bridge matching Tasar et al. (024);;
Chadebec et al. (2025) and diffusion-basee harmonization Winter et al.) (2024) offer flexible image- lo-image translation, but without explicit light-geometry modeling they often produce inconsistent illumination and floating shadows. In contrast, our method introduces Light-Geometry Interaction maps as a structured prior and integrates them into a single pipeline for joint shadow generation and relighting, capturing secondary reflections and inter-reflcction that independent pipelines cannot.
BASELINE MODEL: LATENT BRIDGE MATCHING
Recent works Winter et al. (2024); Tasar et al. (2024) demonstrate that diffusion models and bridge matching techniques can effectively generate realistic shadows and enable photorealistic image re- lighting. We adopt latent bridge matching Chadebee et al. (0255) as our baseline model.
Latent bridge matching learns to transform samples from a source distribution To to a target distri- ma Tı. For efficiency, the method operates in latent space using an encoder-decoder pipeline: = E(x) and x = D(2),, following Rombach et al. (2022). The method defines a Brownian bridge Revuz & Yor (2013) between latent codes 2g and ø1 sampled from their latent distributions, respectively:
bution æ1, given paired samples (zg.
=t)) (1 111-t7 z)z0 + t2 10/((11))
c~N(O, 10,
(1)
where t denotes the time step and controls noise level. A neural network is then trained to regress the corresponding SDE drift vo along this bridge by minimizing a mean squared error loss, defined as the expectation [[|] over paired samples (20, 21) and time steps t:
(21 2())))
L E [I((((), vo (x()),, [) t()(t),t)
1-t
The framework naturally extends to conditional generation by introducing conditioning variables c, in which case the SDE drift becomes va v=(=(t), i, c). During training, paired samples are encoded to latents, timestep is sampled, and noisy samples z(l) are created based on Eq. The predicted target latent is retrieved as:
8 (1- vo (=()))), f, =tt),
(3)
which is then decoded image space d1 D(2)). The final loss combines latent matching (Eq. 2b with pixel-level loss (.):):
L L= L=((o))) E(x)) + (f1,
where the baseline model adopts LPIPS (hang al. (2018) its pixel-level loss. While effec- tive for general image-to coagge-to-imaan to-image translation tasks, this baseline framework has limitations for shadow generation and relighting applications. By solely relying on 2D image information, the method lacks
3

---

Published as a conference paper at ICLR 2025
access to crucial geometric properties such surface normals, depth, and spatial relationships. This absence of 3D geometric understanding limits its ability to accurately model the complex interac- tions between objects, lighting conditions, and shadow formation, ultimately affecting the physical realism and consistency of the generated results.
4 METHOD
Our framework enables accurate and physically informed shadow generation and relighting by in- troducing LGI information calculated only from monocular depth images. An overview of our approach is shown in Fig. 2 The system transforms shadow-free images ny E To into shadowed counterparts 21 E mi. We initialize from a pretrained diffusion model, and keep the encoder E-) and decoder (-)) frozen. Training then focuses on bridge matching conditioned on LGI maps.
e0,((
EXCOOER
x
Light-Geometry
DRCADER
Internction
Bridge Matching
LGI Maps (c*))
LIGHT (c)
Figure 2: Overview of the proposed method. Our approach uses a transform shadow-fres latent codes (2n) into shadowed counterparts (21), conditioned global light cues (e.g.. light color, radius) and image-derived light- tett-ggemeeryy interaction maps. The key novelty lies in generating these interaction maps from image (rr) and light (see 4.1 for details).
strategy to
Following Eq. [. we obtain =(t) and pass the network to predict target latent & in Eq The drift network by is conditioned on = {c', c²h }. where o' denotes the global lighting parameters and em denotes the LGI maps. The global parameters c' contain information about light color, radius, distance, intensity, and direction (azimuth and elevation). The azimuth is defined in the camera plane, while elevation measured relative it, consistent with our LGI maps construction. The LGI maps e provide image-derived lighting-geometry interactions, as detailed in Section 4.1
To focus computation on the most important regions, we replace the pixel-level loss term (x(() in Eq. with weighted LI loss in image space that emphasizes areas of brightness changes:
M Σ
(m))
(ar(m)
#1 (m)).
(m)
(Ix(())
x (1)) T) K,
(5)
m=1
where è denotes the predicted shadowed image, m indexes over the M training samples, T is a threshold used identify regions with significant brightness changes, and @/C denotes dilation operation to expand these regions.
4.1 LIGHT-GEOMETRY INTERACTION MAPS GENERATION
In this section, we describe how LGI maps e are derived from input images vo and lighting l. Light sources are modeled as point lights, and LGI maps are generated through five steps: depth estimation, 3D lifting, ray sampling, elevation difference calculation, and final map construction.
Depth Estimation. We first estimate depth D using an off-the-shelf monocular method Chadebec et al. (2025). Since monocular depth estimation typically produces only normalized depth up to an unknown scale, we rescale the predictions to match the light coordinates. This does not require consistency between light and scene geometry. In ShadRel dataset, light is given in camera coordinates with meter units, while we also provide an image- harhhhirmozdd variant (see supplementary material) with normalized depth (0, 1] and spatial extent [-0.5, 0.5).
metric

---

Published as a conference paper at ICLR 2025
Lifting 2D to 3D. Each pixel in the 2D image is lifted into 3D space by using the predicted depth map. Specifically, given the pixel coordinates and the estimated depth, we compute the 3D position in the camera coordinate frame via the inverse camera projection:
p D(,, D). K- [u V 1]
where (u,, v) is the depth value at pixel (u, v), K is the camera intrinsic matrix and p E R$ is the resulting 3D point in camera coordinates. To provide intuition, we use a simple example-a a blue ball-in Fig. 3 The diagram shows cross-sectional view of the 3D space. Because depth maps capture only 2.5D geometry, they fail to provide depth for occluded surfaces. Such ambiguous or occluded regions are marked in pink in Fig. 3
Ancioquaua umea horm depil reop
Antigguaa coad from Seetl rapp
rarquees araa foom suptimmp fuatur
PS CAMBRA
CAMERAA
DAMBRA
Boo fight foo porty??
Fomparey
See fight foo poin
Seelight hoo point y?
(a) Front lighting
(b) Back lighting
(c) Shadow
(d) Ambiguous shadow
Figure 3: Elevation difference calculation. Scene objects shown in blue. Each 2D pixel is lifted
to 3D at point p. from which a ray is cast toward the light source I. Along this ray, we uniformly sample n points S within the valid front-facing camera frustum. Each sampled point is reprojected onto the image plane to retrieve its depth from the predicted depth map, yielding a set of reprojected points S". The elevation angles e* of these reprojected points are then compared with the elevation angle e' of the light ray to compute elevation difference e". If the light is occluded when viewed from point p. that point is likely to lie in shadow.
Ray Sampling. As shown in the Fig. ray is cast from each lifted point p toward the light source I. We uniformly sample N points along this ray, forming the set S = {p + 6, p)) where ón are evenly spaced scalar distances constrained within the front-facing camera frustum. Each sampled point in S is projected back to the image plane (u/,, "/,)) to retrieve its corresponding depth from the predicted depth map, and following Eq. resulting reprojected points S (S% 1 Points with infinite depth are marked as invalid and excluded from further computation.
Elevation Difference Calculation. For each valid point in S', we compute the elevation angle e and compare it with the elevation angle of the light ray e' to determine potential light occlusion:
Vi n2 [|
e' =aresin V''
e = eñ - ee, e arcsin
|
where of S% - p the vector from point p to surface point Sy of 1 p is the vector from point to light point I, and n = [0 O 1]" is the normal vector of camera plane. Here, represents the dot product. The elevation angle is measured relative the camera plane and lies within the range (-T/2. /2. π /2) Consequently, the LGI values ed are naturally bounded within (-T, T), which is favorable for deep network input stability.
LGI Maps Generation. Since the depth map provides only 2.5D information, it cannot capture occluded regions behind foreground objects. An example of ambiguous shadows is shown in Fd If the 2.5D representation sufficiently captures the scene geometry (e.g., piece-like objects), a hard shadow mask (Fig. 4eh can be derived by checking whether surface elevation matches light elevation using the condition min ea
17. where 7) is small threshold (set to 5" in our visualization).
In cases where geometric information is insufficient, we defer occlusion reasoning to the model by embedding three-channel LGI maps derived from the elevation difference: (c"²) the minimum eleva- tion difference, indicating the potential start of occlusion; (c the maximum elevation difference, indicating the potential end of occlusion; and (cf) the value with the smallest absolute difference, representing the most likely point of direct occlusion:
cTTh = min edd
N
c²
N max n=1
C = 1' 1-= -==arg arg min [ 1Sn5N
(8)
5

---

Published as a conference paper at ICLR 2025
(a) Shadowed Image
(b) cf
(c) ca
(d) mm
(e) Hard Shadow
Figure 4: Example of LGI maps. They indicate shadows cast in the environment and also reveal self-shadowing and shading effects, which can be useful for relighting.
An example visualization of these embeddings shown in Fig. 4 They capture not only the cast shadows within the environment but also reveal self-shadowing and shading effects. We use the LGI maps (c")) as conditioning signals by concatenating them with image features guide the bridge-matching in Eq. providing light-aware occlusion cues.
4.2 EXTENSION TO IMAGE HARMONIZATION
In addition to explicit light condition setting, we also extend our method the task of image har- monization, where the light sources are implied by the image, demonstrating its robustness and generalization to outdoor and realistic images. The core architecture of our method remains un- changed. We introduce additional light estimation network to infer lighting conditions directly from the composited image, removing the need for explicit lighting input. Since our LGI maps are fully differentiable, they enable self supersupee light estimation. Specifically, mask to supervise the predicted lighting. Further details are provided in the supplementary material.
the shadow
4.3 SHADDEL DATESET
As dataset currently exists to support training for coupled light transport, construct the first large-scale dataset consisting of 817K virtual 3D objects, all crafted by professional 3D artists. The textures of these 3D assets assume the physically-accure
principled Bidirectional Scattering Distribution Function (BSDF) formulation by |urley (2012), capable simulating a variety of materials including those of glossy, metallic, or transparent appearances. This dataset not only enables effective training but also provides benchmark for evaluating future methods, and research in coupled illumination and shadow modeling.
To simulate real-world lighting and shadow effects, we render images in Blender using photoreal- istic path tracer Cycles] For each object, we generate following types of images for training:
Input images: the object lit by a random High Dynamic Range Imaging (HDRI) environ- ment map drawn from dataset of panoramas collected from the Internet.
Background images: planar floor and/or vertical wall of random RGB color, illuminated by a single target point light, where the object is to be inserted.
Target images: the composite of object and background under the same point light this is the supervision signal for output image. Additionally, corresponding depth map is also rendered for generating light-occlusion maps.
Each object is augmented by sampling 4 random camera poses and focal lengths; for every camera configurations, totaling 20 target images per object.
we further sample Point lights vary in position, color, and radius, producing shadows of diverse direction and softness.
Above images are stored in physical units of scene radiance. During training, we apply real-world camera response functions provided by colour-science package) and tone-map results to the [0, 1] standard intensity range. The reader referred to supplementary material for details about dataset composition, rendering configuration and costs.

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
We evaluate our approach on three benchmark datasets for light-controllable generation and im- age harmonization (implicit lighting), and present qualitative visualizations on real-world images to demonstrate generalization capability. We further conduct ablation studies, including component analysis, depth-estimation variants, and efficiency evaluation, validate our design choices.
Datasets and Evaluation Metries. Our method jointly addresses controllable shadow generation and image relighting. As published dataset supports this combined task, we build a new ShadRel dataset for training and primary evaluation. We also assess the controllable shadow generation component on the public benchmark of Tasar al. (2024), which targets lean-background shadow generation. For the joint task, we compare with LBM Chadebec et al.] (2025), measuring overall image quality (RMSE, SSIM). shadow quality (overall: BER/IOU, shadow: RMSS/SSSI/BBR)) and object relighting quality (object RMSE/SSIM), We further provide qualitative comparison with SwitchLight Kim et al. 2024). For controllable shadow generation, we compare against CSG Tasar et al. (2024) on their benchmark and metrics (IoU and RMSE). Finally, we extend our method to image harmonization, following [iu al. (2024b) on the DESOBAv2 dataset Liu al. (2023). reporting global/local RMSE (GR, LR), SSIM (GS, LS), and BER (GB, LB).
Implementation Details. We initialize our framework from Stable Diffusion XL. v1.0 Podell et al. 2023). All experiments use 512 x 512 images with batch size of 5. Training employs Adam W optimizer Loshchilov & Hutter (2017) with a learning rate of 3 x 10 -5 5, The threshold in Eq. is empirically set to 0.01, and dilation operation with a kernel size of 17 is applied. In the final loss function, we set the objective weighting factor = 10. The number of sample points is set to 16. All models are implemented in Py Torch and trained end-to-end.
5.1 EXPERIMENTAL RESULTS
Evaluation on joint shadow synthesis and relighting. Tab. presents quantitative results on the ShadRel dataset. For faimess,, retrain LBM with the same settings. Our method outperforms LBM in controllable shadow generation and object object relighting, demonstrating the effectiveness of our light-geomett interaction formulation. Fig- 5 shows qualitative comparisons, where method re- sponds more accurately to light and preserves object geometry, producing realistic shadows, precise relighting, and capturing complex abject-envirommen light transport.
Table 1: Joint shadow synthesis and relighting results on ShadRel dataset.
Overall
Shadow region
Object region
Method
RMSE
SSIM f
BER 4
IoU 1
RMSE
SSIM
BER +
RMSE L
SSIM f
LBM
0.0417 0.0334
0.7148
0.0847
0.7166
0.1543
0.5690
0.1549
0.0298
0.6797
Ours
0.7227
0.0588
0.8096
0.0898
0.6195
0.1103
0.0282
0.6875
Composite
LBM
Ours
GT
Composite
LBM
Ours
GT
Qualitative comparison with LBM synthetic object images. Our method produces shadows and relighting effects consistent with geometry and lighting conditions.
Figure 5:
Qualitative analysis on real images. We further provide qualitative comparisons with SwitchLight and LBM on real-word object images in Fig. 6 Switchl.ight is designed for human relighting with point light sources and does not support shadow generation. We compare their method with ours on both human relighting and other real-world objects. Our method produces realistic relighting effects, including for humans, while faithfully adhering the light direction and preserving the original object colors. In addition, it generates more accurate shadows that align with both the light direction and object geometry, avoiding floating artifacts inconsistencies in shadow orientation. Despite being trained solely on synthetic data, without any real-world or human sppcific samples, method generalizes well to to real images, including human portraits and complex objects.
7

---

Published as a conference paper at ICLR 2025
Inputs
Switch light
LBM
light
light
light
light
light
light
Figure 6: Qualitative comparison with SwitchLight (object relighting without shadow generation) and LBM on real object insertions. Despite being trained solely on synthetic object images, our method outperforms SOTA relighting approaches real-world object images.
Evaluation on clean-background shadow generation. Tab. reports quantitative comparisons with CSG Tasar et al. (2024) on their benchmark. As CSG did not release training data or model weights, we adapt our approach to their clean-background setting by training on ShadRel with pure white backgrounds. Results show that our light-geometry interaction formulation enables more accurate control of shadow shapes. Qualitative results in Fig. further confirm that our method produces shadows with accurate shapes and realistic density.
Table 2: Comparison with CSG Tasar et al.
2024 their benchmark.
Track (Softess Control)
Track 2 (Horz. D. Control)
Track 3 (Vert. D. Control)
Method
IoU 1
RMSE
IoU
RMSE
IoU 1
RMSE
CSG
0.818
0.021
0.780
0.030
0.776 0.785
0.028
Ours
0.821
0.021
0.798
0.030
0.027
Figure 7: Visual example of back-obacg shadow generation, demonstrating that our light- geometry interaction approach provides accurate control over shadow shapes.
Generalization to Harmonization. We compare method with SOTA approaches on the DES- OBAv2 Liu et al. (2023), with quantitative results Tab, and qualitative comparison in Fig. g Our method delivers overall performance comparable to the top-performing approach, SGDGP, while achieving higher accuracy shadow regions. Although SGDGP also leverages geometry priors, restricts them to 2D rotated bounding boxes and shadow templates. Qualitative comparisons reveal
8

---

Published as a conference paper at ICLR 2025
Table Comparison on Image Harmonization on DESOBAv2 Dataset.
BOS Test Images
BOS-free Test Images
Method
GR
LR
GS
L
GB
4
GR L
LR 4
GS f
LS
GB |
LB
ShadowGAN
7.511
67.464
0.961 0.197 0.446
0.890
17.325
76.508
0.901
0.060 0.425
0.842
Ma
8.997
79.418
0.951
0.180 0.500
1.000
19.338
94.327
0.906
0.044
0.500
1.000
19
7.335
58.037
0.961
0.241
0,383
0.761
16.067
63.713
0.908
0.104
0.349
0.682
IT
7.184
68.255
0.964
0.206 0.301
0.596
15.596
60.350
0.909
0.100
0.271
0.534
DMASNe
8.256
59.380
0.961
0.228 0.276
0.547
18.725
86.694
0.913
0.055 0.297
0.574
SGDiffusion
6.098
53.611
0.971
0.370 0.245
0.487
15.110
55.874
0.913
0.117 0.233
0.452
SGDGP 47
5.896
46.713
0.966
0.374
0.213
0.423
13.809
55.616
0.917
0.166
0.197 0.384
Ours
5.900
44.753
0.961
0.415
0.239 0.415
12.979
52.543
0.912 0.201 0.213 0.358
that the main challenge lies in generating shadows that align with object geometry and light direc- tion. Our method addresses this challenge more effectively, as the LGI-based framework directly embeds ligh-geomeety interactions in sauce-lligned representation, providing stronger geometric and illumination cues. This leads to improved robustness in outdoor and realistic scenarios.
SGDGB
Ohis
GT
Figure 8: Qualitative comparison with SGDGP Zhao et al. (0255) Our method achieves higher
accuracy in shadow regions by aligning more closely with object geometry and light direction.
5.2
ABLATION STUDY
Table 4: Analysis of Method Components on ShadRel Dataset
Components
Overall
Shadow region
Object region
W-L1
LGI
RMSE |
SSIM t
BER
IOU 1
RMSE 1
SSIM t
BER L
RMSE
SSIM 1
0.0408
0.7109
0.1012
0.7193
0.1236
0,5868
0.1923
0.0330
0.6679
0.0391
0.7148
0.0940
0.7353
0.1154
0.5974
0.1784
0.0317
0,6680
0.0334
0.7227
0.0588
0.8096
0.0898
0.6195
0.1103
0.0282
0.6875
-LGl+Depth
0.0388
0.7148
0.0932
0.7344
0.1166
0,5938
0.1765
0.0315
0.6719
+Depth
0.0339
0.7188
0.0719
0.7921
0.0942
0.6139
0.1364
0.0283
0.6836
LGI Ch3
0.0351
0.7179
0.0670
0.7824
0.0932
0.6095
0.1343
0.0290
0.6807
w/ DAv2
0.0334
0.7188
0.0602
0.8148
0.0901
0.6212
0.1269
0.0283
0.6875
GT depth
0.0326
0.7266
0.0558
0.8107
0.0894
0.6195
0.1070
0.0280
0.6875
We analyze component contributions in Tab. 4 using LBM with standard L1 loss as the baseline. Each added module yields consistent performance gains Comparing LGI embeddings with di- rect depth embeddings shows that depth alone (* ('-.GGIDeethh brings only marginal improvements, while combining both ('+Depth') slightly degrades performance, likely due to noise and misalign- ment in depth estimation. Using only the third channel of LGI ('LGI Ch3') also reduces perfor-
A visual comparison with and without the LGT module is provided in the Appendix G

---

Published as a conference paper at ICLR 2025
mance, confirming the necessity of proposed channels. Finally, replacing the original depth estima- tor with DepthAnything V2 Yang et al. (2024) ('w/ DAv2') and ground truth depth produces minimal variation, demonstrating robustness to the choice of depth backbone.
Our model requires 4.82 GB of parameter memory and 3.12 TFLOPs, increasing these costs over the baseline by only 0.0004% and 0.001 I %, respectively, underscoring its efficiency. The method further extends naturally to multiple objects and light sources, with additional details, visualizations, and analysis of the impact of the sample-point count N provided in the Appendix D
CONCLUSION
We presented a unified, physically inspired framework utilizing Light-Geometry Interaction maps for joint shadow generation and object-level relighting. By approximating scene geometry from pre- dicted depth maps, our method generates coherent and physically plausible shadows and relighting effects without requiring full 3D reconstruction. Additionally, introduced high-quality syn- thetic dataset specifically designed for joint shadow generation and relighting tasks, featuring diverse object categories, multiple material types, and challenging scenarios involving object-environment interreflection. Extensive experiments confirm that our approach achieves SOTA performance, ef- fectively balancing visual consistency, physical accuracy, efficiency, and generalizability.
ACKNOWLEDGMENTS
This work was conducted during an internship at Amazon. Hongdong Li is co-employed by Aus- tralian National University (full-time, primary) and Amazon IML (as part-time Amazon Scholar). Hongdong Li also funded part by an ARC Discovery Grant DP220100800.
REFERENCES
Anand Bhattad, James Soole, and David A Forsyth. Stylitgan: Image-based relighting via latent con- trol. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4231-42240 2024.
Brent Burley. Physically Based Shading at Disney. In ACM SIGGRAPH 2012 Course Notes, Los Angeles, CA, 2012. ACM.
Clément Chadebec, Onur Tasar, Sanjeev Sreetharan, and Benjamin Aubin. Lbm: Latent bridge matching for fast imaee-to-imagee translation. arXiv preprint arXiv:2503.07535, 2025.
Per Christensen, Julian Fong, Jonathan Shade, Wayne Wooten, Brenden Schubert, Andrew Kensler, Stephen Friedman, Charlie Kilpatrick, Cliff Ramshaw, Mare Bannister, et al. Renderman: An advanced path-tracing architecture for movie rendering. ACM Transactions on Graphics (TOG),
37(3):1-21, 2018.
colour-science package. Colour Science for Python. colour-science
URL https://pppi.oocc/ppojeerr
Cycles. Blender Cycles Render Engine. URL UBt.htppss/gtttabb.///////eeeeee
Paul Debevec. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. Acm siggraph 2008 classes, pp. 1-10. 2008.
Frédéric Fortier-Chouinann Chouinard, Zitian Zhang, and -ran-François Lalonde. Spotlight: Shadow-guide object relighting via diffusion. arXiv preprint 1. 2024.
Messier, Mathieu Garon, Anand Bhattad,
David Griffiths, Tobias Ritschel, and Julien Philip. Outcast: Outdoor single-image relighting with cast shadows. In Computer Graphics Forum, volume 41, pp. 179-193. Wiley Online Library, 2022.
10

---

Published as a conference paper at ICLR 2025
Majed EI Helou, Ruofan Zhou, Johan Barthas, and Sabine Süsstrunk. VIDIT: virtual image dataset for illumination transfer. CoRR, abs/2005.05466, 2020. URL https://arxiv..rrt abs/
2005. 05460)
Yan Hong, Li Niu, and Jianfu Zhang. Shadow generation for composite image in real-world scenes. In Proceedings of the AAAI conference on arlificial intelligence, volume 36, pp. 914-922, 2022.
Xiaowei Hu, Yitong Jiang, Chi-Wing Fu, and Pheng-Ann Heng. Mask-shadowgan: Learning to remove shadows from unpaired data. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2472-2481, 2019.
Alexander Keller. Instant radiosity. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pp. 49-56, 1997.
Hoon Kim, Minje Jang, Wonjun Yoon, Jisoo Lee, Donghyun Na, and Sanghyun Woo. Switchlight: Co-design of physics-driven architecture and pre-training framework for human portrait relight- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25096-25106, 2024.
Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, and Yannick Hold-Geoffroy. Lightit: Illumination modeling and control for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9359-9369, 2024.
Eric Lafortune and Yves D Willems. Rendering participating media with bidirectional path tracing. In Eurographics Workshop Rendering Techniques, pp. 91-100. Springer, 1996.
Ruofan Liang. Zan Gojeic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Chih-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, et al. Diffusion renderer: Neural inverse and forward rendering with video diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26069-260886 2025.
Zhi-Hao Lin, Bohan Liu, Yi-Ting Chen, Kuan-Sheng Chen, David Forsyth, Jia-Bin Huang, Anand Bhattad, and Shenlong Wang. Urbanir: Large-scale urban scene inverse rendering from a single video. arXiv preprint wxXiv:2306.09349 2023.
Daquan Liu, Chengjiang Long, Hongpan Zhang, Hanning Yu, Xinzhi Dong, and Chunxia Xiao. Ar-
shadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
8139 8148, 2020.
Qingyang Liu, Jianting Wang, and Li Niu. Desobav2: Towards large-scale real-world dataset for shadow generation. arXiv preprint arXiv:2308.0999 2023.
Qingyang Liu, Junqi You, Jianting Wang, Xinhao Tao, Bo Zhang, and Li Niu. Shadow generation for composite image using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8121 8130, June 2024a.
Qingyang Liu, Junqi You, Jianting Wang, Xinhao Tao, Bo Zhang, and Li Niu. Shadow generation for composite image using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8121-8130, 2024b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101. 2017.
Matt Pharr, Wenzel Jakob, and Greg Humphreys. Physically based rendering: From theory to implementation. MIT Press, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Timothy J Purcell, Ian Buck, William R Mark, and Pat Hanrahan. Ray tracing on programmable graphics hardware. In ACM SIGGRAPH 2005 Courses, pp. 268-es es. 2005.
11

---

Published as a conference paper at ICLR 2025
Daniel Revuz and Mare Yor. Continuous martingales and Brownian motion, volume 293. Springer Science & Business Media, 2013.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Yichen Sheng. Jianming Zhang, and Bedrich Benes. Ssn: Soft shadow network for image composit- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4380 4390, 2021.
Yichen Sheng, Yifan Liu, Jianming Zhang, Wei Yin, A Cengiz Oztireli, He Zhang, Zhe Lin, Eli Shechtman, and Bedrich Benes. Controllable shadow generation using pixel height maps. In European Conference on Computer Vision, pp. 140-256, Springer, 2022.
Yichen Sheng. Jianming Zhang. Julien Philip, Yannick Hold-Geoffroy, Xin Sun, He Zhang, Lu Ling, and Bedrich Benes. Pixht-lab: Pixel height based light effect generation for image compositing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16643-16653, 2023.
Xinhao Tao, Junyan Cao, Yan Hong, and Li Niu. Shadow generation with decomposed mask predic- tion and attentive shadow filling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1198-520,, 2024.
Onur Tasar, Clément Chadebec, and Benjamin Aubin. Controllable shadow generation with single- step diffusion models from synthetic data. arXiv preprint arXiv:2412.11977 2024.
Ingo Wald, Philipp Slusallek, Carsten Benthin, and Markus Wagner. Interactive rendering with co- herent ray tracing. In Computer graphics forum, volume 20, pp. 153-165. Wiley Online Library, 2001.
Zhibo Wang, Xin Yu, Ming Lu, Quan Wang, Chen Qian, and Feng Xu. Single image portrait relighting via explicit multiple reflectance channel modeling. ACM Transactions Graphics (ToG), 1-6) 2020.
Gregory Ward. The radiance lighting simulation and rendering system. In Proceedings of the 2 annual conference on Computer graphics and interactive techniques, pp. 459-472, 1994.
Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul Debevec. Performance relighting and reflectance transformation with iume-multppeeedd illumination. ACM Transactions on Graphics (TOG), 24(3):756-764, 2005.
Daniel Winter, Matan Cohen, Shlomi Fruchter, Yael Pritch, Alex Rav-Acha, and Yedid Hoshen. Objectdrop: Bootstrapping counterfactuals for photorealistic object removal and insertion. In European Conference on Computer Vision, pp. 112-129. Springer. 2024.
Xiaoyan Xing, Vincent Tao Hu, Jan Hendrik Metzen, Konrad Groh, Sezer Karaoglu, and Theo Gevers. Retinex-diffusion On controlling illumination conditions in diffusion models via retinex theory. arXiv preprint arXiv:2407.20785. 2024.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875-21111 2024.
Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained lighting control for diffusion-based image generation. In ACM SIGGRAPHI 2024 Conference Papers, pp. 1-12, 2024a.
Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Cold-Geefffroy, Yiwei Hu, Fujun Luan, Ling- Qi Yan, and Miloš Hašan. Rgbe-x: Image decomposition and synthesis using material- and lighting-awe aware diffusion models. In ACM SIGGRAPH 2024 Conference Papers, SIGGRAPH '24 New York, NY, USA, 2024b. Association for Computing Machinery. ISBN 9798400705500. 10. N0.1145/36415193366 19644444 URL wttptttts///t //ooi http://....or/0..111.1111
doi:
12

---

Published as a conference paper at ICLR 2025
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Scaling in-the-wild training for diffusion -based illumination harmonization and editing by imposing consistent light transport. In The Thirteenth International Conference on Learning Representations, 2025a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Shuyang Zhang, Runze Liang, and Miao Wang- Shadowgan: Shadow synthesis for virtual objects with conditional adversarial networks. Computational Visual Media, 5(1):105-115, 2019.
Zitian Zhang, Frédéric Fortier- Sarter-Chhuinarde Mathieu Garon, Anand Bhattad, and Jean-François Lalonde. Zerocomp: Zero-shot object compositing from image intrinsics via diffusion. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), pp. 483-494. IEEE, 2025b.
Haonan Zhao, Qingyang Liu, Xinhao Tao, Li Niu, and Guangtao Zhai. Shadow generation using diffusion model with geometry prior. In Proceedings of the Computer Vision and Pattern Recog- nition Conference, pp. 7603-7612, 2025.
Xiaoming Zhao, Pratul Srinivasan, Dor Verbin, Keunhong Park, Ricardo Martin Brualla, and Philipp Henzler. Illuminerf: 3d relighting without inverse rendering. Advances in Neural Information Processing Systems, 01442293-426111 2024.
Jingsen Zhu, Fujun Luan, Yuchi Huo, Zihao Lin, Zhihua Zhong, Dianbing Xi, Rui Wang, Hujun Bao, Jiaxiang Zheng, and Rui Tang. Learning-based inverse rendering of complex indoor scenes with differentiable monte carlo raytracing. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-8, 2022.
13

---

Published as a conference paper at ICLR 2025
A EXTEND TO SHADOW-CENTEREE IMAGE HARMONIZATION
In this section, extend our method realistic image harmonization, where foreground object is composited onto a new background. Most harmonization datasets do not provide ground-truth lighting information; however, some datasets include shadow masks. our setting, the model infers lighting conditions directly from the composited image. As shown in Fig. [ the light estimation network predicts a complete set of lighting parameters, including color, radius, distance, intensity, and direction. These parameters are integrated into our light-aware shadow generation and relighting framework to ensure consistent and realistic harmonization fully differentiable, it enables end-to-end training of the light estimation network. Furthermore, when shadow mask is available, we formulate a loss function that leverages LGI to supervise the estimation of lighting direction.
the inserted object. Since LGI is
Light Estimanion
LIGHT
LGI
LGI maps
Shadow mask
Our
Method
Figure 9: Overview of the image harmonization pipeline. The light estimation network predicts full set of lighting parameters, with the estimated lighting direction used to compute LGI maps su- pervised by ground-truth shadow masks. These estimates integrated into light-aware shadow generation and relighting framework to produce consistent and realistic harmonization of the in- serted object. Since the LGI module fully differentiable, it enables end-to-end training of the light estimation network.
The light estimation network consists of four convolutional layers with progressively increasing channel dimensions. The resulting feature representation is then processed by two-layer multilayer perceptron (MLP), producing an 8-dimensional vector encoding the full set of lighting parameters. During training, first freeze the light-aware shadow generation and relighting framework for epochs to warm up the light estimation module, and then continue with end-to-end training where all parameters are updated.
For outdoor scenarios, sunlight serves as the primary light source and can be effectively modeled a point light infinite distance, emitting parallel rays. To accommodate this scenario, we introduce modified variant of our LGI maps specifically adapted for sunlight. Instead of casting rays toward finite light position, rays are cast along the estimated light direction relative our normalized camera coordinate system, where the x-axis and y-axis lie within the range (-0.5, 0.5) we assume default
and the z-axis within (0, I]. According to this coordinate system Eq. 6
w 0 W/2
0 и H/2 where W denote the height and width of the input image. An intrinsic matrix K
o
visual example of LGI maps for sunlight is shown in Fig. 10
Building on the LGI maps, we obtain the predicted hard shadow mask My. We then define a light- direction estimation loss Li, applied when a ground trruun shadow mask M, is available. Since we only consider the inserted object's shadow, the loss computed within region restricted to the dilated ground-truth shadow mask. The overall loss combines BCE and loU terms:
(,M = (Mc + LMo Mg).
(9)
where the BCE loss is defined as
[nc Mg) = -E Mg
M, log MP + +1 M2) log(1 - (M)]1
(10)
14

---

Published as a conference paper at ICLR 2025
(a) Shadowed Image
(b) cf
(c) c²
(d) mm
(e) Hard Shadow
Figure 10: Example of sunlight LGI maps. They establish correspondence between light direction and shadow shape when scene geometry is known.
and the IoU loss is defined
Mg
Lia (Mp, =1
(11)
E[MM + Mg MpMMg
with E denoting the expectation over pixels.
B MULTIPLE OBJECTS
Our method extends naturally to scene editing via object-level insertion. We further demonstrate mujti-object insertion by applying the method sequentially, inserting objects one at a time. As shown in Fig. 11 our approach generalizes well to the multi-object setting: it casts realistic shadows onto previously inserted objects, ensures that the generated shadows align with the scene geometry, and maintains relighting consistent with the materials of each object.
(d)
(f
Figure 11: Examples of multiple object insertion. (a-c) Inserted objects. (d-f) Scenes with inserted objects after shadow generation and relighting. Our method produces realistic, texture-aware shad- ows on the table and preserves faithful relighting across wood, leather, metal, and glass materials.
15

---

Published as a conference paper at ICLR 2025
C MULTIPLE LIGHT SOURCES
While our primary focus is single-light scenarios, the method naturally extends to multiple light sources by exploiting the linearity radiance Debevee (2008); Ward (1994).
Specifically, we
accumulate per-light contributions through additive composition:
I.
(12)
where 21 denotes the relight result under the 1-th light, and the total number of light sources. This straightforward accumulation requires no additional modification and still captures complex phenomena such as overlapping shadows and varying intensities, as shown Fig. 12
Figure 12: Examples of scenes with various light sources (top- rendered using our method, and fused two-light-source scenes (bottom)
D STUDY ON SAMPLE-POINT COUNT
Table 5: Study on Sample Point Count
Overall
Shadow region
Object region
RMSE 1
SSIM
BER
IOU
RMSE 1
SSIM t
BER
RMSE
SSIM
0.0344
0.7188
0.0612
0.7954
0.0991
0.6125
0.1141
0.0283
0.6875
16
0.0334
0.7227
0.0588
0.8096
0.0898
0.6195
0.1103
0.0282
0.6875
32
0.0334
0.7231
0.0586
0.8099
0.0869
0.6198
0.1096
0.0283
0.6875
16

---

Published as a conference paper at ICLR 2025
We further study the effect of the sample-point count N. Because its computational impact is negli- gible at our reporting precision, Tab. reports accuracy for N {8, 16, 32). While N=32 provides only marginal gains, adopt 16 by default.
COMPARISON TO LBM WITH DEPTH INPUT
We additionally compare with LBM equipped with depth input in Tab. 6 Our method consistently outperforms this depth-augmented LBM across all regions and metrics, showing that the gains of LGI stem from our proposed design rather than merely from using an additional depth modality.
Table 6: Comparison to LBM with Depth Input
Overall
Shadow region
Object region
RMSE +
SSIM
BER
IOU f
RMSE +
SSIM f
BER
RMSE +
SSIM 1
LBM+t
0.0395
0.7122
0.1003
0.7284
0.1208
0.5923
0.1901
0.0321
0.6692
Ours
0,0334
0.7227
0.0588
0.8096
0.0898
0.6195
0.1103
0.0282
0.6875
F MORE VISUALIZATION UNDER DIFFERENT LIGHTS
In this section, we present additional real-world examples of shadow generation and relighting under varying light sources (Fig. [3]. Despite being trained exclusively on synthetic data, our model generalizes effectively to real images and successfully captures indirect lighting effects, such as reflections. More visualizations are included in the supplementary video.
Composite
Light
Light
Light
Light
Light
Figure 13: Examples produced by our method on real-world object images.
G VISUAL COMPARISON WITH AND WITHOUT THE LGI MODULE
Fig 14commasss results with and without the LGI module. Incorporating LGI increases sensitivity to scene geometry, yielding shadows that are more realistic and better aligned with geometric structure.
17

---

Published as a conference paper at ICLR 2025
Composile
wil L.C
Qualitative comparison with and without LGI, illustrating that LGI enables geometry- aligned shadows and relighting effects.
Figure 14:
H COMPLEX BACKGROUND EXAMPLES
Object
Background
Ours Output
Figure 15: Qualitative results on complex backgrounds. Our method remains robust on both non- planar and textured backgrounds.
In this section, we shift our focus to complex backgrounds. While the proposed ShadRel dataset primarily designed to study complex object-background inter-refloctions and therefore uses rela- tively simple backgrounds (c.g.. floor or floorwwall configuration), the objects themselves span wide range of materials, geometries, and textures. As result, complex geometry and texture
18

---

Published as a conference paper at ICLR 2025
still exercised through self-occlusion effects. Our real-image visualizations in Fig. show that the method effectively handles textured backgrounds (e.g.. desks, cluttered layouts, and object-rich scenes). Additional qualitative results are provided in Fig. [5 where observe that method remains robust on non-planar and textured backgrounds.
Furthermore, the DESOBAv2 dataset is real-world benchmark with images captured golf courses, tennis courts, beaches, gardens, and other outdoor environments, where the backgrounds non-planar and contain rich textures and patterns. The qualitative results on these challenging cases in Fig. T6 demonstrate that our method is robust to such complex backgrounds. We further demonstrate generalization capabilities on in-the-wild images Fig. 17, showing that our method generates reasonable, hightnn--oosister shadows even on non-planar and textured backgrounds.
Figure 16: Complex backgrounds in the DESOBAv2 dataset, including non-planar geometry and rich textures and patterns. Our qualitative results this dataset show that the proposed method produces reasonable predictions even under these challenging background conditions.
Object
Rack konkgronnd
Duiput
Object
Hackgroudd
Chtput
Figure 17: Visual results challenging in-the-wild backgrounds. Our model maintains lighting consistency and shadow realism across diverse surface geometries and textures.
I STUDY ON COMPLEX OBJECTS
To quantitatively evaluate our method on complex objects, we construct 200-sample subset consist- ing solely of challenging complex objects and evaluate our method this subset. The quantitative results in Tab. show our method outperforms LBM by large margin, together with the additional qualitative examples in Fig. 18 demonstrate that our approach remains robust beyond simple shapes. We further provide additional gualitative results on this subset and on real images in the supplemen- tary video. Moreover, the DESOBAv2 dataset also includes complex objects, such humans with bicycles in Fig. 8 our results on this dataset likewise show that the method preserves high-quality predictions in these cases, further supporting its robustness complex geometries.
19

---

Published as a conference paper at ICLR 2025
Table
Study on Complex Objects
Overall
Shadow region
Object region
RMSE
SSIM 1
BER +
IOU1 1
RMSE
SSIM 1
BER 4
RMSE
SSIM
LBM
0.0431
0.7112
0.0954
0.0691
0.1575
0,5609
0.1758
0.0342
0.6743
Ours
0.0341
0.7196
0.0658
0.7940
0.0929
0.61
0.1241
0.0286
0.6828
Composile
Light
Light
Prodiet Light
GT Light3
Figure 18: Qualitative results on complex objects. Our ShadRel dataset includes complex objects, and our method remains robust on these challenging cases. As the dataset provides Ground Truth (GT) for only 5 discrete lighting conditions, the Light 3 shows direct comparison against the available GT. Light and Light represent unseen illumination directions (sampled from the 24 distinct lights in the supplementary video), demonstrating the model's generalization capabilities where GT is unavailable.
J VISUALIZATION ON COMPLEX INTERACTION
In this section, we present additional qualitative results on complex object-backgroune interac- tion effects (Fig. 19), including cast shadows from transparent objects and shadows modulated by inter-reflections and secondary reflections. When inserting an object into scene, not only is the object relit by the background, but the background is also altered by the object (e.g., through shad- ows and reflections), and these changes in turn further influence the object. Such higher-order bacegt-backgoound interactions remain an open challenge. Our ShadRel dataset is specifically con- structed to capture these challenging phenomena, featuring strong reflections from materials such as glass and weaker reflections from fabrics- scenarios that are close to real applications but rarely covered by existing models and datasets. Even under these demanding conditions, method
20

---

Hubliande as a conference conference paper apprraa at RCLR 2025
prodiaenn diighti Aadelity,piii phy both Lighinng
fidehiddeeii both the the unique difluueddfffuu
consistent pred pecticiiie that alien well with the ground ao truth, hight robustness, and
of ShadRel and the sectiveenes e
of our approosch
Composil
Ous
GT
Composite
Oura
GT
Figure 19: Quaiitatte results on compinng
object- backkeen interactia Geld dataue contains diverse materials, coluding reffective ones such as class and poncolaii wilcoi introduce
Abjectt with S
mooniinoooo.. E
challenge shadows from transparent objects, with both primary and secondary vilection in these e difffiuutt scenarios, our method produces Sreessssee Sordeeliii high-
comsisteet sicallllly
predhte Predictions.
CARPOOOEEE COMPOSITION AND AND EXAMPLE
DATASET K
eeeentt example images Seeeessai from our dataser Each sampul containss 4 random cam- F Fig. 20 the dataset consists of input images of objects, backkrooud configurations As shown w in images rendered under varying and occasionally a wall), and target images where the same objects vactss and e
In this section we present
sta poses: and for each pose we generate 3 5 jint-iet
Heghtngg conditions (floor and
backgrounds illuminated with different lighting effects.
21

---

Published as a conference paper at ICLR 2025
Inpur image
Background
Target imagg
Figure 20: Examples from our dataset. Each sample includes (top) input images of objects from different camera poses, (middle) background images with floor and optionally a wall under point- light illumination, and (bottom) target images combining the objects and backgrounds under the same point light.
L LIMITATIONS
Since our approach is initialized from diffusion model, it inevitably inherits some of the well- known limitations of this class of generative models. Most notably, the generated content may not remain fully faithful to the input specification, particularly in regions requiring high-fidelity reproduction of fine-graines details. As illustrated in Fig. 21 elements such as logos can appear inconsistent, distorted, or incomplete.
fastrack
LIGHTSTRINE
EXETTY
LIGHTSINNUE
CEEEMEBEE
C GAAA!!
Figure 21: Examples illustrating the limitations our method in reproducing fine-grained details. (Top: Ground Truth; Bottom: Our results.) While our approach can handle certain structural details, tiny elements, such as Logos, may appear inconsistent, distorted, or incomplete.
22


Reviews:
[{"review_id": "WHacwsURkK", "reviewer": "Reviewer_YF9H", "summary": "This paper proposes a unified pipeline that jointly generates shadows and performs object relighting, achieving physics-based control. It also introduces the ShadRel dataset for coupled light transport. The experimental setup is comprehensive, and the visual results are excellent for both synthetic and real-world cases.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper presents the first generative method for the joint synthesis of shadows and object relighting from a single 2D RGB image. A key innovation is the introduction of a control mechanism conditioned on monocular depth, which enables the generation of continuous and coherent shadow and relighting effects in direct response to continuous variations in illumination.\n- This paper introduces ShadRel, a large-scale synthetic dataset developed for the tasks of shadow generation and object relighting. The dataset is notable for its extensive volume and the inclusion of diverse and complex lighting scenarios, rendering it a valuable asset for training and evaluating models in these domains.\n- This paper introduces a novel light-aware occlusion representation, LGI map. The methodology for acquiring this map is clearly articulated, and both theoretical analysis and experimental results compellingly demonstrate that the LGI map enables the generation of results with physically-based shadow and illumination constraints. This representation effectively encodes light-aware occlusion from monocular depth, providing a physics-inspired prior that constrains generative models to produce more realistic and consistent outputs.\n- The paper is well-supported by thorough experimentation, clear theoretical explanations, and compelling visual results.", "weaknesses": "The visual results presented in the paper are compelling. However, the experiments primarily showcase objects with relatively simple geometric structures. To more rigorously assess the robustness and generalization capabilities of the proposed method, I would encourage the authors to include results on more structurally complex objects. For instance, objects with fine-grained details, intricate parts, or significant self-occlusion (e.g., a bicycle, a detailed sculpture, or a potted plant) would serve as more challenging test cases. Demonstrating high-fidelity shadow and relighting effects on such objects would provide stronger evidence of the model's effectiveness and further strengthen the paper's contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cWn2P7aMp7", "reviewer": "Reviewer_FiRe", "summary": "This work presents a new 2.5D representation from RGB images, depth information, and lighting parameters, termed as Light-Geometry Interaction maps (LGI), to boost the joint shadow generation and object relighting. As claimed in the paper, this LGI representation provides a physics-inspired prior to reduce floating and implausible shadow effects and empowers a neat pipeline to tackle direct illumination, secondary reflections, and inter-reflections. Besides, to enhance their pipeline training and evaluation, a synthetic dataset with 800K samples is constructed. Their experimental results illustrate better quantitative and qualitative performance compared to prior works.", "questions": "1. The potential impact of inaccurate depth estimation should be justified.\n2. Enhanced ablation studies are recommended, which illustrate other ways to incorporate the depth information or inject this information into baseline methods. These studies are beneficial for validating the paper’s core design LGI. A fairer comparison would require facilitating baselines with similar depth guidance or reporting ablations isolating the effect of added depth input.\n3. To validate model generality, it is recommended to evaluate models on real-image benchmarks using identical training data or provide cross-dataset results demonstrating robustness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. LGI provides an intuitive utilization of the depth prior and suits the pipeline objective as joint shadow generation and object relighting.\n2. The proposed pipeline integrates the modern large model backbones, such as depth estimation models and latent bridge matching models.\n3. Dataset contribution: The provided dataset, ShadRel, provides a valuable contribution to this community.", "weaknesses": "1. Dependence on monocular depth information. A significant improvement of the proposed pipeline is introducing the depth information into the LGI representation, which also raises concerns. Monocular depth is an ill-posed estimation, and it frequently fails to provide accurate estimations, thus leading to malicious priors in the generation pipeline. This requires further clarification. At the same time, although LGI provides better performance compared to previous baselines, the depth information constitutes an extra input requirement not shared by several baselines (e.g., LBM, CSG), which only rely on RGB and light parameters.\n2. The experimental results are weakened by the tight relationships between the proposed method and the ShadRel dataset. Although the authors have retrained baseline methods on the ShadRel dataset, this does not fully address the domain bias, since the ShadRel is synthetic and specifically tailored for the method’s assumptions and requirements (such as a single object, known light direction, and planar contact). This raises concerns that the illustrated improvements may primarily reflect the dataset alignment instead of model effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RxgZj7WsJD", "reviewer": "Reviewer_RYL2", "summary": "This paper presents a novel shadow generation and relighting framework, which introduces LGI maps to provide lighting and shadow cue for the model. The proposed method derives the LGI maps from the estimated 2.5 D depth map and lighting conditions, then a LGI maps and lighting parameter conditioned bridge matching model generates the relit images .  Experiment results are provided to demonstrate the effectiveness of the framework, highlighting the good performance and efficiency of the proposed method.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a novel shadow generation and relighting framework. The LGI map can model the light-geometry interactions from the depth map which encodes useful shadow and illumination information, enabling the model to generate reasonable relighting results.\n2. Extensive experiments show that the method achieve better performance than baselines.", "weaknesses": "1. I have some doubts about the proposed LGI map and whether it can accurately represent shadow and lighting information. Consider the point $p$ in fig.3 (a), and for this point, $e^d_n$ is always negative, which means that the elevation of light is always higher than the horizon, so this point can see the light source and its $c^m_1$ is negative. But for the point $p$ in fig.3 (c), this point is in the shadow, but its $c^m_1$ is also negative. So why the $c^m_1$ indicates the potential start of occlusion. And why is $c^m_1$ darker in the shadow area of fig.4(b)? Logically, in areas under a light source, $c^m_1$ should always be negative, meaning that the illuminated region should have a darker $c^m_1$.\n2. The derivation of LGI maps and shadow mask does not consider the radius or the area of the light. When the light source is a single point, then we will obtain the hard shadow. But when the light source is not a single point, then we should consider soft shadow, including umbra and penumbra. It seems that the mixture of $c^m_1$, $c^m_2$ and $c^m_3$ can handle the soft shadow, but i am not very sure.\n3. The cases in the dataset may be too simple, only a floor and a wall. Could the proposed method handle background with more complex geometry and texture?\n4. The baseline used is not strong enough. Could the authors use more recent generative relighting methods, such as DiffusionRenderer and DiLightNet?\n\nLiang, Ruofan, et al. \"Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models.\" *Proceedings of the Computer Vision and Pattern Recognition Conference*. 2025.\n\nZeng, Chong, et al. \"DiLightNet: Fine-grained lighting control for diffusion-based image generation.\" *ACM SIGGRAPH 2024 Conference Papers*. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yXkmMNzHjS", "reviewer": "Reviewer_2xqw", "summary": "This paper proposes a framework to generate shadow maps according to the specified lighting condition and supports object-background composition with plausible shadows and relighting. The paper leverages latent bridge mapping to generate the results, and condition the model by introducing Light–Geometry Interaction (LGI) maps. The LGI map leverages information from the 2.5D estimated depth map and captures light-geometry interaction by the elevation angle of the reprojected points from the depth map. The LGI provides light-aware occlusion cues and controls the diffusion model to generate plausible results. Experiments show plausible object insertion results with realistic shadow effects.", "questions": "1. Can the authors clarify the difference between the proposed LGI and SSRT in the rebuttal and summarize their contributions in this design?\n2. Can the proposed method work well on object insertion into more complex background environments, rather than a simple planar background with point light sources?\n3. The example illustrated in Fig. 3 is interesting, but I don't find how the proposed method handles the ambiguous case in Fig. 3(d). How will the LGI encode the \"not sure\" information in such case?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Reasonable design. The idea of using 2.5D information from depth maps to obtain occlusion cues is reasonable. The proposed LGI method leverages ray sampling and reprojection, which constitute a physically informed algorithm.\n2. Dataset contribution. The paper proposes a large-scale dataset, ShadRel, providing synthetic data for lighting and shadow effects. The research community can benefit from the proposed dataset for training and benchmarking.", "weaknesses": "## Major\n1. Limited novelty. This paper is built upon the existing LBM framework with a new LGI representation, which is the paper's main contribution, as a control signal providing occlusion cues. However, the idea of ray sampling in LGI is not new -- similar to screen-space raytracing (SSRT) in computer graphics and existing works [1,2] already use this for object insertion, cast shadows and relighting.\n2. Simple setting. Most experiments use the setting of inserting the object into a planar background with point light sources, which is a simple setting and not very interesting. The image harmonization setting is more interesting -- but the paper only provides limited results in Fig. 7.\n3. Missing baselines. Although this paper is more focused on shadow generation, general-purpose image-based relighting/harmonization methods such as IC-Light [3] are still important baselines since they also handle shadows, especially for the image harmonization setting.\n\n## Minor\n1. Low image quality in the figures. The paper uses bitmap images for Figs. 2, 6, 7, causing sawtooth artifacts in fonts when zooming in. In addition, the image resolution in Fig. 7 is low. I may expect a higher-quality result for image harmonization.\n2. Missing citations. Lots of relevant papers, such as SSRT-related graphics papers [1,2] or image-based lighting papers [3] are not cited.\n\n## References\n- [1] Griffiths, David, Tobias Ritschel, and Julien Philip. \"OutCast: Outdoor Single‐image Relighting with Cast Shadows.\" Computer Graphics Forum. Vol. 41. No. 2. 2022.\n- [2] Zhu, Jingsen, et al. \"Learning-based inverse rendering of complex indoor scenes with differentiable monte carlo raytracing.\" SIGGRAPH Asia 2022 Conference Papers. 2022.\n- [3] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. \"Scaling in-the-wild training for diffusion-based illumination harmonization and editing by imposing consistent light transport.\" The Thirteenth International Conference on Learning Representations. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)