MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation
Authors: Kerui Ren
Jiayang Bai
Linning Xu
Lihan Jiang
Jiangmiao Pang
Mulin Yu
Bo Dai
Venue: NeurIPS 2025 poster ()
Keywords: Object Compositing; Multi-view Harmonization; Shadow Generation
Primary area: applications

Abstract: Object compositing offers significant promise for augmented reality (AR) and embodied intelligence applications. Existing approaches predominantly focus on single-image scenarios or intrinsic decomposition techniques, facing challenges with multi-view consistency, complex scenes, and diverse lighting conditions. Recent inverse rendering advancements, such as 3D Gaussian and diffusion-based methods, have enhanced consistency but are limited by scalability, heavy data requirements, or prolonged reconstruction time per scene. To broaden its applicability, we introduce MV-CoLight, a two-stage framework for illumination-consistent object compositing in both 2D images and 3D scenes. Our novel feed-forward architecture models lighting and shadows directly, avoiding the iterative biases of diffusion-based methods. We employ a Hilbert curve-based mapping to align 2D image inputs with 3D Gaussian scene representations seamlessly. To facilitate training and evaluation, we further introduce a large-scale 3D compositing dataset. Experiments demonstrate state-of-the-art harmonized results across standard benchmarks and our dataset, as well as casually captured real-world scenes demonstrate the framework's robustness and wide generalization.

MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation
Kerui Ren Jiayang Bai
Linning Xu* Lihan Jiang
Jiangmiao Pang Mulin Yu Bo Dai
Shanghai Jiao Tong University, Shannhai Artificial Intelligence Laboratory,
NNajinng University, "The Chinese University of Hong Kong. Uniiersity of Science and Technology of China, The University of Hong Kong
(a) Composited Scene with added objects
Before
Colight
After
(e)ore After MV CoLight
(b) Harmonized Scene
view
view
view 3
Figure 1: Illustration of our object compositing pipeline with harmonization and relighting using MV-CoLight. In (a), show composite scene with visually inconsistent inserted objects. Applying our MV-CoLight method in (b), we generate realistic lighting, shadows, and harmonious integration of objects into the 3D scene. Panel (c) highlights clear visual differences before and after harmonization, accompanied by consistent novel view renderings below. Explore more demos on our project page: https:/ddiyappeeegiibbbboooollhh
Abstract
Object compositing offers significant promise for augmented reality (AR) and embodied intelligence applications. Existing approaches predominantly focus on single-image scenarios or intrinsic decomposition techniques, facing challenges with multi-view consistency, complex scenes, and diverse lighting conditions. Re- cent inverse rendering advancements, such as 3D Gaussian and diffusion- methods, have enhanced consistency but limited by scalability. heavy data requirements, or prolonged reconstruction time per scene. To broaden its applicabil- ity, introduce MV-CoLighht two-stage framework for illumination-coosiitenn object compositing both 2D images and 3D scenes. Our novel feed-forward architecture models lighting and shadows directly, avoiding the iterative biases of diffusion-basse methods. We employ a Hilbert curve-based mapping to align
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2D image inputs with 3D Gaussian scene representations seamlessly. To facilitate training and evaluation, we further introduce large-scale 3D compositing dataset. Experiments demonstrate state-of-the-ar harmonized results across standard bench- marks and our dataset, as well as casually captured real-world scenes demonstrate the framework's robustness and wide generalization.
Introduction
Object compositing in 3D scenes remains a formidable challenge due to the interplay of color harmonization, shadow synthesis, light transport simulation, and multi-view consistency, all of which must be addressed to achieve photorealistic integration. This capability is fundamental to AR, robotics, and interactive media, where realism directly impacts user immersion and perception.
Early object compositing research focuses primarily on isolated subtasks like scene relighting [54 shadow generation [24 25,, and color harmonization [2), yielding promising yet fragmented solutions. However, The transition toward unified frameworks reveals intricate couplings between these components, necessitating adherence to physical principles governing light transport and occlusion phenomena. Diffusion-based pipelines such as ObjectStitch (38 and ControlCom (53) attempt single-image object insertion by synthesizing harmonious lighting and shadows within background bounding box, but their reliance on stochastic sampling and the lack of large-scale, high-quality compositing datasets limit their robustness and generalization in real-world scenarios.
In this work, we tackle the problem of seamlessly inserting novel objects into static 3D scenes captured from multiple viewpoints. Our goal is to relight each object so that its appearance, including ambient illumination, surface reflections, and cast shadows, matches the lighting of the scene, while also modeling the reciprocal effects of the object on its surroundings (e.g. secondary shadows and interreflections). We introduce MV-CoL.gght, a unified framework that preserves both geometric fidelity and photorealism across views by learning and enforcing conssitg- priors at both the image and scene levels. MV-CoLight adopts a two-stage training pipeline. In the 2D object compositing stage, train feed-forward model to capture scene-specific lighting characteristies, including background shadows and indirect illumination, from individual images. In the 3D object compositing stage, we transform these learned features into 3D Gaussian representation using 3D Gaussian splatting [9]. ordering them via a Hilbert curve to ensure spatial coherence and enforce multi-view consistency. Leveraging recent advances in video-level instance segmentation and 3D-aware object insertion, our framework effectively eliminates common 2D mask artifacts while achieving efficient inference (0.07s per frame) without compromising stability visual quality.
To support training and evaluation, we introduce a large-scale synthetic dataset of over 480k composite scenes rendered in Blender. Each scene features table from the Digital Twin Catalog [8] augmented with Poly Haven HDR environment maps and materials [30, and additional light sources for varied illumination. We render 16 uniformly sampled RGB views per scene, along with depth maps and segmentation masks. To simulate realistic compositing challenges, we mix foreground and background layers under different lighting conditions, creating deliberate lighting inconsistencies for training and evaluation. Further implementation details are provided in the supplementary material.
Our main contributions are as follows: 1) feed-forward architecture for multi-view object composit- ing that, unlike diffusion- based alternatives, offers improved computational efficiency and robustness with high visual quality; 2) a two-stage training framework that connects 2D object compositing with 3D Gaussian color fields via a Hilbert curve ordering mechanism, thereby enforcing geometrically consistent illumination priors and coherent multi-view shadows; and 3) curate large-scale bench- mark of over 480 annotated multi-view scenes under varying lighting conditions, and demonstrate that our method achieves state-of-the-art performance across several public datasets.
2 Related Works
Object compositing, the seamless integration of foreground objects into background scenes, is fundamental task in both image editing and 3D graphics. the following, we briefly discuss three principal paradigms that have guided existing solutions.

---

Multi-Task Decomposition Approach. Object compositing generally involves addressing three challenges, including color harmonization, relighting, and shadow generation. Below, we briefly review related works in these areas. Color harmonization has evolved from classical low-level
techniques using color statistics and gradient adjustments (20 32 39 44 to learning-based meth- 12
powered by large-scale datasets like iHarmony 77]. Relighting modifies an object's shading while preserving its geometry and material properties. Recent learning- based relighting techniques focused on specific image types, including outdoor scenes [10 (1),
ods (37 13
28
14
4 6
portraits [54] 31. and human subjects [6 55), achieving high-quality results. Shadow genera- tion employs diverse strategies, from using pixel height information to generate diverse lighting 45] and generative models [25) that bypass ray-tracing requirements. While recent progress in these subdomains demonstrates improved fidelity, multi-view harmonization and physically grounded shadow synthesis remain open challenges, highlighting the need for holistic frameworks that ensure cross-task and cross-view coherence.
effects 36 35 to GAN-based 42 56
End-to-End Unified Frameworks. Unified end-to-end frameworks for image compositing have emerged in recent studies 38 3 53 40) ObjectStitch (38) introduces diffussoon-bsed architecture
that concurrently addresses geometry correction, harmonization, shadow generation, and view syn- thesis. ControlCom [53] further enhances composite fidelity by incorporating a dedicated foreground refinement module. However, these approaches predominantly process single-view inputs. Building on ObjectStitch, MureObjectStitch [3] adopts a multi- reference strategy for multi pullti-pespecctve com- positing, yet it still struggles with inconsistent harmonization when applied to multi-view images from the same scene. In contrast, our work leverages 3D modeling ensure visual consistency across views, directly addressing these limitations. By integrating 3D priors, our approach simplifies the task color-mappin transformations for inserted objects. This formulation inherently obviates the need for diffusion-basee generative capabilities while necessitating precise per-pixel color trans- formations. Consequently, we employ beed-forward network rather than diffusion-based models, which prioritize pixel-level generation and often yield unstable color outputs.
Inverse Rendering Paradigm. This approach for object compositing first estimates intrinsic scene properties, such as geometry, materials, and lighting, from input images through inverse rendering [ Subsequently, traditional rendering pipelines or neural rendering pipelines
employed to render novel views of the scene with inserted objects. Recent advancements [88 9 23 have incorporated
3D scene representations like NeRF (29) and 3D Gaussian Splats [9] within neural rendering
pipelines. The emergence of large-scale image generative models 22
52 21 has recently revitalized
inverse rendering research. The RGB++X [52) framework first trains image diffusion model to estimate G-buffers from object and scene data. It then composites synthetic objects into these estimated channels and employs diffusion model to generate final images with consistent lighting and shadow effects. However, such methods demand extensive high-quality datasets with fully paired intrinsic properties to achieve robust generalization capabilities, which poses significant challenges for real-worke environment applications.
3 Methods
In this work, we focus on efficiently synthesizing consistent lighting and shadows to harmonize scenes. Formally, given a background scene and foreground object, we assume that the inserted object and the background scene are entirely aligned and our task to produce multi-view renderings that insert, harmonize, and relight the object under novel illumination while maintaining overall coherence, which presents an essential requirement for AR and embodied- sondiedd-itelligenee applications that demand real-time, view-consistent integration.
Fig. 2 illustrates our two-stage framework. (1) We begin with an inharmonious scene and convert it into pixel-aligned 3D Gaussian representation (.). (2) Each image is then processed independently by mansformer-basee network for single-view object compositing (Sec (3) To achieve multi-view consistency, we concatenate the extracted pixel-wise features with Gaussian wise features, order them along a Hilbert curve 15), and decode them with a second transformer to predict harmonious Gaussian color attributes (Sec 3.3 (4) Finally, descrrbes the loss functions that drive our two-stage training. A brief introduction to Gaussian splatting and the Hilbert curve provided in the supplementary materials.
3

---

Swin ViT Block
ekgaal
Paxh Eabhbiii
Dilundd Cazs
Pact hnmbbodiitt
Prenhzed
W-
2D Object Compositing Medel
5-3
(a) Data Preprocesinng
(b) Object Compositing
Groand
Gazoiaa wia Featus
Swia VIT Block
2D OC Mode
Plush Entolking
Dlacc Cure
Coesboling
masaan
OC Madel
2lbaosera Ireat
Randard
Lea
Lww Apercy
C
3D Object Compasting Medel
camera poss
parsnacies
GrEmese Ciise
Inhormniioss Japur
Object Compositing
Figure 2: Pipeline of MV-Colight. In (a), insert white puppy the composite object onto the table between basketballs, and render multi-view inharmonious images, background-only images, and depth maps using a camera trajectory moving from distant close-up positions. Subsequently in (b), we input a single-view data into the 2D object compositing model, which processes the data through multiple Swin Transformer blocks output the harmonized result. Finally in (c), we project the multi- view features from 2D models into Gaussian space via Φ(-), combine them with the original inharmonious Gaussian colors projected into 2D Gaussian color space through V/ (-), and then feed them into the 3D object compositing model. The model outputs harmonized Gaussian colors and computes rendering loss by incorporating Gaussian shape attributes.
3.1 Data Preprocessing
Begin with composed 3D scene, obtained via synthesis, 3D scanning, multiview reconstruction we place set of cameras orbiting the scene center to obtain multi-view composite images, background-only images, and depth maps. From each view's images, camera poses, and depth data, we build point maps including 3D positions and colors, then randomly sample fixed number M of points initialize the 3D Gaussian model g'. During optimization, we fix each Gaussian's 's opacity at 1 and adjust only its shape parameters. As illustrated in Fig. [ we organize 3D Gaussian primitives, each tied to a unique training pixel, into spatially coherent patches by mapping their centers along prace-filling Hilbert curve 15], denoted as mapping (f.).
pipeline (e.g., 34 43)
3.2 2D Object Compositing
Given an inharmonious composite image I E B its background reference G e R3xHxW and depth map / E RIxHxW, we form the input tensor {T, G, D) and feed it into our 2D object compositing network:
A MeM,
(1)
where fl the predicted harmonized image and 024 are the network parameters. Following Grounded DINO (26) and DINO-X [33], we adopt the Swin Transformer as backbone. Its shifted-window attention mechanism achieves linear computational complexity with respect to image size, while the hierarchical pyramid structure ensures high hardware efficiency. Compared to the original transformer architecture, swin transformer delivers superior performance across a variety of vision tasks, yet requires fewer FLOPS and parameters under the same input resolution.
Specifically, M2d begins with several x 3 convolutions to extract shallow features, which are then partitioned into non-overlapping P x P patches and fed into L Swin Transformer layers [27]. For each layer E (1...., I.). the input F, normalized, processed by window-based multi-head self-attention with residual connection, renormalized, passed through an MLP with second residual

---

Gever Gemobe Neppnert
Cántes
cempaited Eargi& depl saged cineu
90
Hhet
Friject Ganolaen Billier
20 ctlteesccnn
Nera
po simes
Cteinal Hilerr Care
VO
Gamizas
Cencianpronti
Figure 3: Mapping mueti-view observations into 2D Hilbert-ordered Gaussian color map. Starting from inharmonious multi-view images, depth maps, and camera poses, we compute per-view point maps and randomly sample M points to initialize 3D Gaussian primitives, which we then optimize to fit the scene. Next, we construct 3D Hilbert curve through the Gaussian centers and assign each primitive to its nearest curve point, yielding an ordered ID sequence. Finally, we fold this sequence Hilbert curve, producing a spatially coherent projection which each pixel encodes the color of its corresponding Gaussian.
into 2D grid along
skip, and output as F, as summarized by
F: W-Atten (LN(FF)) + Fi-1,
(2)
F. = MLP(LN(F.)) + F,.
At inference time, M2d is applied independently to each input. We extract the output feature maps F E x= from the final attention block, where m is the number of views and is the feature dimension. These features are then transformed into 3D Gaussians via the mapping function 2(-) for downstream 3D compositing network (Sec. 3.3 See supplementary material for more details.
3.3 3D Object Compositing
For 3D object compositing, we seck an illumination-consistee Gaussian model that preserves each primitive's 3D position (x, y, 2), scale, and rotation from the inharmonious model g', but updates only its color attributes C'. Thus, we freeze all positional and geometric parameters and learn a color-only mapping. Inspired by recent advancements [7 in processing point clouds with transformers, like Point Transformer V3, we use mapping that first linearizes the sparse 3D Gaussians into a 1D sequence via a 3D Hilbert curve and then arranges them into 2D grid by the inverse 2D Hilbert curve. We concatenate these Gaussian wise features, combining the transformed 2D colors and the Mas features via ((-), and feed them into our 3D compositing network:
l = Maa (( (F))))) (CC))); 0z4).
where AE the network parameters and C' the original inharmonious Gaussian colors. The output I& gives harmonious, view-consistent colors, which we back-projec' onto the 3D Gaussians to complete the compositing. Notably, Mad adopts the similar architectural designs M2a, differing only in its input and output dimensions. Please refer to supplementary material for more details.
3.4 Loss Design
During the two-stage training process, employ similar loss function design, utilizing mean-square error loss Lmse loss and perceptual loss L.p to optimize the object compositing models:
Led Lmse H) (H,
(3)
Laa = 3(Cmse (CC) + AC v(C))) + (1-3)
Cmse (H, H.) + (L, H.)
(4)
n
5

---

Table Single-view quantitative performance on our purposed dataset and the Objects With Lighting dataset AT. We report visual quality metrics, inference time and memory storage, highlighting the best and second-best in each category. Our and Ourj denote our method without depth input and without both depth and background input, respectively.
Dataset
Simple Synthetic Scene PSNR1
Complex Synthetic Scene PSNR1
Objects With Lighting SSIM1
Time|
Memory.
Paradigm
Method
SSIM
IPIPS]
SSIM
I.PIPS]
PSNR
1.PIPS]
Diffusion-based
LamiNet
16.94
0.614
0.287
19.94
0.671
0.274
17.15
0.781
0.222
23.82s
13.79G
Feed forward
GP
14.60
0.418
0.369
0.415
12.14
0.479
0.351
1.36m
Feed-forward
Oursf
28.35
0.957
0.031
29.61
0.947
0.029
27.48
0.945
0.051
32.89M
Feed-forward
PCT-NNe
12
22.58
0.912
0,055
0.931
25.08
0.921
18.4M
Diffasion-based
Objectstitch
X
19.14
0.770
0.193
21.82
0.788
0.170
21.15
0.831
0.176
4.54s
5.24G
Diffusion based
ContralCom
S3
18.85
0.765
0.209
19.88
0.771
0,185
19.75
0.811
0.189
4.63x
10.94G
Diffusion-based
RGB++X
52
12.28
0.428
0.368
12.91
0.507
0.296
11.28
0.503
0.422
19.71s
10.68G
Diffusion-based
IC-Light
17.66
0.659
0.217
20.87
0.679
0. 190
18.22
0.774
0.200
1.25s
1.60G
Feed forward
Ours
29.11
0.959
0.030
30.00
0.951
0.027
28. 18
0.945
0.050
0.07%
32,92M
Food-forward
Ours
29.65
0.961
0.029
30.20
0.953
0.027
28.75
0,946
0.049
0.07s
32.94M
Compuddte
LamiNer
GPT-4o
Ouret
PCTNa
ObjasSiichh
ControlCum
RGB
IC-Light
Oure
Our
Figure 4: Single-view qualitative comparison with SOTA methods 48 38 53 on our proposed dataset and public datasets [53 41] with differences highlighted via colored patches. Compared to existing baselines, our method successfully generates illumination consistent with the background and physically plausible shadows while decoupling highlights from inserted objects, demonstrating generalization capabilities on out-of-domain datasets. The method the green box does not incorporate background images as input, whereas the others do.
where H; and H, denote the ground truth images and the rendered images from the harmonized Gaussian G'. which is composed of C' and shape parameter from G. À and are the and set as 0.05 and 0.5 by default.
4 Experiments
4.1 Experimental Setup
Datasets and Metries. Our proposed dataset contains simple synthetic, complex synthetic, and real captured scenes. From the simple synthetic set, scenes are randomly held out for evaluation, while the rest are used for training. In contrast, both the complex synthetic and real captured seenes are solely used for evaluation. Besides, we evaluate our method on two public benchmarks, FOSCom (53] and Objects With Lighting (OWL) [1]. For 2D object compositing, we test on 640 scenes from FOSCom, 72 scenes from OWL, and 58 challenging scenes from our proposed dataset. For 3D object compositing, report results on 50 simple and 8 complex synthetic scenes from our dataset,

---

Table 2: Multi-view quantitative performance our purposed dataset and real captured scenes. We report visual quality metrics, inference time (Gaussian training time #Train), highlighting the best and second-bes best in each category. Our and Ourj denote our method without depth input and without both depth and background input, respectively.
Dataset
Simple Synthetic Scene Complex Synthetic Scene
Real Captured Scene
Timel (#Train)
Paradigm
Method
PSNRT
SSIM
LPIPS]
PSNRT
SSIMT
LPIPS]
PSNR1
SSIM
LPIPS]
Diffusion based
LumiNet [88
17.15
0.573
0.304
18.45
0.663
0.222
20.05
0,770
0.198
6.31m (-)
Feed-forward
GPT-4o [
14.57
0.375
0.445
15.11
0.366
0.411
14.34
.473
0.406
21.40m (-)
Feed-forward
Ouss
28.96
0.955
0.033
29.32
0.946
0.029
25.88
0,925
0.041
1.07 (1 11.88m)
Feed-lorward
PCT-Net
12
22.97
0.908
0.057
25.19
0.927
0.035
23.39
0.824
0.103
0.47s (-)
Diffusion-based
Objectstitch
19.12
0.726
0.217
21.84
0,792
0.163
18.43
0.785
0.193
1.21m (-)
Diffusion hased
ContmlCam
53
18.95
0.722
0.231
19:60
0.773
0.181
18.54
0.778
0.199
1.23m (-)
Diffusion-based
RGBX 52
12.71
0.417
0,360
12.69
0.504
0,304
13.68
0,594
0.312
5.26m (-)
Diffusion-based
IC-Light
54
17.94
0.596
0,242
20.60
0.689
0.183
20.23
0.718
0.233
19.36s (-)
Inverse Remdering
GS 23
15.56
0.742
0.134
0.249
15.92
0.265
17.62s (57. 14m)
Inverse Rendering
GL-GS 2
18.97
0.808
0.126
16.56
0.674
0.310
16.07
0.716
0.234
16.69s (1,43h)
lnverse Rendering
IRGS 11
17.79
0,688
0.237
21.04
0.702
0.291
20.19
0,744
0.215
9.72m (3,02h)
Feed-forward
Ours*
29.73
0.958
0.031
29.51
0.949
0
26. 10
0,926
0.041
1.07s (1.08m)
Feed forward
Ours
30.29
0.960
0.030
30.13
0.952
0.027
26.39
0.927
0.040
1.08s (1.08m)
Composite
ObjeatSütch
Figure 5: Multi-view qualitative comparison with SOTA methods 48 4 38 53 52 54 23
on our proposed dataset and real captured scenes, with differences highlighted via colored patches. Our method synthesizes plausible illumination and shadows while ensuring multi-view consistency. The method in the green box does not incorporate background images as input, whereas the others do.
with another two real captured scenes. All images are center-cropped and rescaled to 256 x 256 for uniform comparison. Performance is quantified using PSNR, SSIM [46), and LPIPS (55). Since each trouh image embodies only one physically plausible lighting/albedo configuration, perceptual metrics (SSIM and LPIPS) offer additional assessments of structural and visual fidelity than PSNR alone.
Baselines. For 2D object compositing evaluation, we conduct comprehensive comparisons with rep- resentative methods: PCTNet [12], ObjectStitch (38]. ControlCom [53), RGB++X (52), ICLight [54]. LumiNet (48] and GPT-4o 4. For 3D object compositing evaluation, we additionally incorporate Gaussian- inverse rendering method such as GS-IR (23), GI-GS (2] and IRGS [I]. establishing a unified benchmark comparing conventional 2D pipelines with emerging 3D-aware methods built upon differentiable rendering frameworks.
Implementation Details. Our model architecture employs a unified Swin Transformer backbone with consistent configurations for both 2D and 3D object compositing tasks. The network processes
7

---

Compasiic
Ourr (Vic D)
Ouss (Vic 21
(Vic
Compesiic
Ours Jlck
Our (Veew 21
OpI
We evaluate our method on real-world scenes captured under varying illumination with six cameras arranged in a circular array. On the left, we insert cake and a black box; on the right, we insert a toy, a mouse, and a backpack. MV-CoLight consistently harmonizes object colors, produces physically plausible lighting interactions, and accurately casts shadows across all viewpoints.
Figure 6:
256x256 resolution inputs with embedding dimension of 96, structured with 3 cascaded transformer blocks. Each block contains 6 successive Swin Transformer layers with 6 parallel attention heads.
We train the 2D object compositing model for IM iterations with batch size 128 using AdamW (base Ir=2e-3, weight decay=0.05, momentum parameters 31=0.9, 3y=0.95), 10k iteration linear warmup followed by cosine decay 1e-6, FP16 mixed precision, gradient clipping at 10.0, and EMA of 0.99. For 3D object compositing model, we reduce the learning rate to 1e-3 and batch size to 32, training for 100k iterations. For trainig time, We train the 2D model from scratch for 15 days, and the 3D model for 3 days with 16 NVIDIA A100 (80 GB) GPUs.
4.2 Performance Analysis
Below show our method delivers physically plausible lighting and shadows for inserted objects, out-performing both 2D harmonization [48
[4 and Gaussian- inverse rendering [23 [ baselines. The approach also generalizes from synthetic training to challenging
12
38
53
52
real-world captures, maintaining photorealism under diverse lighting and materials.
Single-view Harmonized Result. Current image harmonization methods exhibit notable limitations when compositing new objects into a scene, as illustrated in Fig. For example, PCT-Net [2] enforces only color consistency and omits realistic highlights and cast shadows, while RGB-X (52)
material estimation yields inaccurate albedo maps that blur illumination and misalign geometry during neural relighting. Diffusion based frameworks such ObjectStitch [38] and ControlCom (53) produce visually compelling composites but often distort object shape and texture in the generative process. ICLight [54] 's illumination estimator lacks robustness in cross-domate scenarios, resulting in pronounced appearance artifacts under complex real-world lighting, while the light transport module of LumiNet (48) generates non-physical highlight patterns and jagged shadow boundaries. Even advanced multimodal systems like GPT-4o (4) which improve local lighting coherence, introduce unintended global modifications that undermine overall scene integrity which is particularly hard to be strictly enforced via prompting.
Multi-view Harmonized Result. Multi-view object compositing compounds the inconsistencies of 2D harmonization methods, resulting in visible color shifts and misaligned shadows across viewpoints. Caussian-based inverse rendering techniques attempt remedy this by enriching each primitive with material attributes, such as estimated albedo and normals, and estimating an environment map from the background Gaussians to relight the composite. However, their reliance on imperfect decoupling causes specular highlights and shadowed regions from the original images to be treated as textures. As shown in Fig. 5 the result is a conflated relighting effect that blurs the distinction between intrinsic material properties and new environmental illumination, failing to achieve true multi-view coherence. Unlike environment mapping-bard based methods, our approach directly learns illumination and shadow priors from the multi-view composite scene and transfers them to the inserted object, guaranteeing seamless, view-consistent lighting. By encoding these learned visual cues into Gaussian feature representations and propagating them through our transformer-basee 2D-3D pipeline, we maintain spatial coherence and realistic shadowing without explicit environment map estimation. Extensive evaluations on public benchmarks [53] 41 and our own dataset demonstrate that our method outperforms both 2D harmonization and Gaussian-based inverse-rendering baselines in quantitative metrics and visual quality.
Real Scene Harmonized Result. We further assess our method on diverse real-world multi-view captures that diverge markedly from our synthetic training data in terms of lighting complexity and
8

---

Composite
Ours
Ground Truth
Figure Visual results of inserting luminous objects. Our method successfully simulates the illumination effects of luminous spheres within the scene environment.
Table 3: Ablation study conducted the simplified synthetic scenes within our proposed dataset. We report visual quality metrics, inference time and model parameters. For variable control, the cor- responding 2D models for 3D object compositing are kept bascline versions, and their parameters are not counted.
Model
2D Object Compositing Model
3D Object Compositing Model
Method
PSNR1
SSIM
LPIPS).
Timel. Params.
PSNR
SSIM
LPIPS,.
Timel.
Params..
baseline
29.65
0.961
0.029
0.07s
2.957M
30.29
0.960
0.030
1.08s
2.957M
transformer block (2)
28.34
0.955
0.032
0.06s
2.085M
28.70
0.953
0.035
0.97s
2.085M
wansformer layer (4)
28.39
0.956
0,032
0.06s
2.119M
28.83
0.957
0.034
0.98s
2.118M
embedding dim (60)
27.68
0.951
0.035
0.05s
1.779M
28.11
0.949
0.039
0.81s
1.778M
Linear transform
28.99
0.951
0.036
1.08s
2.957M
w/ PTv3 feat. extractor
30.08
0.959
0.031
1.16s
49. 16M
w/o 2D OC model
25.83
0.913
0.051
0.08s
2.957M
w/o depth input
29.11
0.959
0.030
0.07s
2.957M
29.73
0.958
0.031
1.07s
2.957M
w/o background input
28.81
0.958
0.030
0.07s
2.955M
29.44
0.956
0.032
1.07s
2.957M
material detail, as shown in Fig. These scenes feature unpredictable illumination conditions and intricate textures that typically confound traditional harmonization and inverse-rendering pipelines. Nevertheless, our approach consistently produces photorealistic composites, accurately estimat- ing lighting and casting coherent shadows across all viewpoints. This robust performance under uncontrolled, real-world conditions highlights the generalization robustness of our learning-based illumination priors.
4.3 Efficiency and Extensibility
Inference Time Comparison. For 2D object compositing, our feedforward architecture enables inference times as short as about 0.07 seconds, significantly outperforming diffusiio-baase methods like LumiNet [48] which require more than 20 seconds to complete the multi-step denoising process (default 50 steps). While GPT-4o 4 remains closed-source, source, we utilize its web interface to generate results, incurring& a latency of several minutes. For 3D object compositing, Gaussian based inverse rendering methods necessitate both environmental map extraction and material attribute optimization atop pre-trained Gaussians. In contrast, our method achieves harmonized Gaussian color attributes with about second of inference time after a few minutes of scene-specific Gaussian representation learning, demonstrating superior efficiency. Notably, our framework eliminates the need for Gaussian retraining when repositioning inserted objects. This efficiency makes our framework especially well suited for real-time AR and -ibtldied-iteelligenee applications.
Extentions on Various Illumination Priors. By design, our unified compositing pipeline can also accommodate other illumination priors whether HDR environment maps, learned light distributions, or discrete emitters, with similar training and inference pipeline. This challenging capability has been largely neglected by prior works. As a demonstrative extension in Fig. 7 we focus here on inserting new light sources dynamically relight the scene. To support emissive-object compositing with true multi-view consistency, our method estimates light propagation through the existing 3D Gaussian geometry, capturing how point or area emitters illuminate surrounding surfaces, and computes secondary shadows and interrefiections to generate physically plausible shading all objects. This extension underscores the flexibility of our approach and applicability wide range of illumination scenarios.

---

4.4 Ablation Study
We perform a series of ablations to isolate the factors driving our 2D and 3D compositing pipelines, the results are shown in Tab. E First, vary the number of swin transformer blocks, layers and the feature-embedding dimension. We find that embedding size has a greater impact on final performance than the depth of the transformer stack, with modest drop in harmonization quality when reducing the block number or layer number. Next, we examine the 2D compositing inputs, removing the background image prevented reliable object placement, while omitting the depth map eliminated essential geometric priors. We find that both scenarios degrade the model's ability to learn illumination conditions. When feed these 2D features into the 3D network, excluding them entirely still allowed plausible color matching but produced unrealistic shadows and highlights, underscoring the importance 2D illumination cues. Introducing the Hilbert curve reordering further accelerate training convergence and improve visual quality by preserving Gaussian color locality in 2D Hilbert space. Furthermore, a comparison with the use of PTv3 [47] to extract Gaussian features also indicates that our method based on the Hilbert curve is more lightweight and effective.
Limitation
While our approach achieves superior performance across diverse synthetic and real- world scenes, several limitations persist: (a) Color bias in real-world scenes. Trained predominantly synthetic data, the model occasionally exhibits color discrepancies when applied to certain real- world environments. (b) Physically inconsistent illumination. Due to the absence of strict physical constraints, deviations specular highlight positions and shadow directions may arise under complex lighting conditions. (c) Gaussian representation limitations. Errors in Gaussian parameterization, constrained by their inherent capacity to model complex scene details, can propagate to degrade harmonization quality. To address these problems, future work may focus on: (a) Integration of physical constraints. Enhancing shadow consistency by estimating light source positions and constraining shadow regions. (b) 4D scene harmonization. Extending harmonization to dynamic scenes to enable consistent movement of inserted objects within dynamic environments.
large-scale
Conclusion
In this paper, we introduce MV-CoLight, two-stage framework that seamlessly combines 2D feed- forward harmonization network with 3D Gaussian based compositing model to deliver efficient, view-consistent object insertion. In the first stage, our 2D network rapidly learns per-view color and illumination alignment; in the second, the 3D Gaussian fields enforce geometric and lighting coherence across viewpoints, producing realistic shadows and reflections with minimal runtime overhead. Extensive experiments on both synthetic and real-world benchmarks show that MV. CoLight outperforms ofte--of-thearr 2D and 3D baselines in visual fidelity and consistency. To drive further progress, we also introduce new large-scale multi-vice compositing dataset with photorealistic accurate annotations. Finally, we demonstrate that our pipeline naturally generalizes to additional lighting effects, underscoring its versatility for broader applications.
7
Acknowledgments
This work was funded in part by the National Key R&D Program of China (202ZD01660001 Shanghai Artificial Intelligence Laboratory, the National Natural Science Foundation of China (Grant No. 62502247) and the HKU Startup Fund.
References
[1] Harry Barrow, J Tenenbaum, A Hanson, and E Riseman. Recovering intrinsic scene characteris- tics. Comput. vis. syst, 2(3-26):2, 1978.
[2] Hongze Chen, Zehong Lin, and Jun Zhang. Gi-gs: Global illumination decomposition on gaussian splatting for inverse rendering, arXiv preprint arXiv:2410.02611 2024.
10

---

[3] Jiaxuan Chen, Bo Zhang, and Li Niu. Mureobjectstitch: Multi-reference image composition. arXiv preprint arXiv:2411.07460 2024.
[4] Sixiang Chen, Jinbin Bai, Zhuoran Zhao, Tian Ye, Qingyu Shi, Donghao Zhou, Wenhao Chai, Xin Lin, Jianzong Wu, Chao Tang, et al. An empirical study of gpt-4o image generation capabilities. arXiv preprint arXiv:2504. 05979, 2025.
[5] Wanli Chen, Xinge Zhu, Guojin Chen, and Bei Yu. Efficient point cloud analysis using hilbert curve. In European Conference Computer Vision, pages 730-747. Springer, 2022.
[6] Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, and Liqing Zhang. High-resolution image harmonization via collaborative dual transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18470-18479,
2022.
[7] Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and Liqing Zhang, Dovenet: Deep image harmonization via domain verification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8394-8403, 2020.
[8] Zhao Dong. Ka Chen, Zhaoyang Lv, Hong-Xing Yu, Yunzhi Zhang, Cheng Zhang, Yufeng Zhu, Stephen Tian, Zhengqin Li, Geordie Moffatt, et al. Digital twin catalog: A large-scale photorealistic 3d object digital twin dataset. arXiv preprint 2rXiv:2504.0854/ 2025.
[9] Jian Gao, Chun Gu, Youtian Lin, Zhihao Li, Hao Zhu, Xun Cao. Li Zhang, and Yao Yao. Relightable 3d gaussians: Realistic point cloud relighting with brdf decomposition and ray tracing. In European Conference on Computer Vision, pages 73-89. Springer, 2024.
[10] David Griffiths, Tobias Ritschel, and Julien Philip. Outcast: Outdoor single- image relighting with cast shadows. In Computer Graphics Forum, volume 41, pages 179-193. Wiley Online Library, 2022.
[11] Chun Gu, Xiaofei Wei, Zixuan Zeng. Yuxuan Yao, and Li Zhang. Irgs: Inter-reflective gaussian splatting with 2d gaussian ray tracing. arXiv preprint arXiv:2412.15800 2024.
[12] Julian Jorge Andrade Guerreiro, Mitsuru Nakazawa, and Bjöm Stenger. Pet-net: Full resolution image harmonization using pixel-wise color transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5917-5926, 2023.
[13] Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, and Junyu Dong. Im- age harmonization with transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14870-14879, 2021.
[14] Zonghui Guo, Haiyong Zheng. Yufeng Jiang. Zhaorui Gu, and Bing Zheng. Intrinsic image harmonization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16367-16376,, June 2021.
[15] David Hilbert. Dritter Band: Analysis- Grundlagen der Mathematik- Physik Verschiedenes: Nebst Einer Lebensgeschichte. Springer- Verlag, 2013.
[16] Chaonan Ji, Tao Yu, Kaiwen Guo, Jingxin Liu, and Yebin Liu. Geometry-aware single-image full-body human relighting. In European Conference Computer Vision, pages 388 405. Springer, 2022.
[17] Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. Advances in Neural Information Processing Systems, 37:141 17:141129-141122 2025.
[18] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. Tensoir: Tensorial inverse rendering. Proceedings of the IEEE/VF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2023.
[19] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
11

---

[20] Lalonde and Alexei A Efros. Using color compatibility for assessing image realism. In 2007 IEEE 11th International Conference on Computer Vision, pages 1-8. IEEE, 2007.
[21] Ruofan Liang, Zan Gojeie, Huan Ling. Jacob Munkberg, Jon Hasselgren, Zhi-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, et al. Diffusionrenderer: Neural inverse and forward rendering with video diffusion models. arXiv preprint arXiv:2501. 18590, 2025.
[22] Ruofan Liang, Zan Gojeic, Merlin Vimier-Davie David Acuna, Nandita Vijaykumar, Sanja Fidler, and Zian Wang. Photorealistic object insertion with diffusion-guided inverse rendering. In European Conference on Computer Vision, pages 146-465. Springer, 2024.
[23] Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21644-216535 2024.
[24] Daquan Liu, Chengjiang Long, Hongpan Zhang, Hanning Yu, Xinzhi Dong, and Chunxia Xiao. Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 81 39-8148, 2020.
[25] Qingyang Liu, Junqi You, Jianting Wang, Xinhao Tao, Bo Zhang, and Li Niu. Shadow generation for composite image using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8121-8130 2024.
[26] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang. Qing Jiang. Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pages 38-55. Springer, 2024.
[27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[28] Quanling Meng, Liu Qinglin, Zonglin Li, Xiangyuan Lan, Shengping Zhang, and Ligiang Nie. High-resolution image harmonization with adaptive-interval color transformation. Advances in Neural Information Processing Systems, 37:137699 2025.
[29] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1))99- 2021.
[30] Poly Haven. Poly haven.
[31] Pramod Rao. Gereon Fox, Abhimitra Meka, Mallikarjun BR, Fangneng Zhan, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, et al. Lite2relight: 3d-aware single image portrait relighting. ACM SIGGRAPH 2024 Conference Papers, pages 1-12, 2024.
[32] Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. Color transfer between images. IEFE Computer graphics and applications, 21(:3 2001.
[33] Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, et al. Dino-x: A unified vision model for open-world object detection and understanding. arXiv preprint arXiv:2411.14347, 2024.
[34] Johannes Lutz Schönberger and Jan-Michael Frahm. fiructre-from-mottio revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[35] Yichen Sheng, Yifan Liu, Jianming Zhang, Wei Yin, A Cengiz Oztieeli, He Zhang, Zhe Lin, Eli Shechtman, and Bedrich Benes. Controllable shadow generation using pixel height maps. In European Conference on Computer Vision, pages 240-256. Springer, 2022.
12

---

[36] Yichen Sheng, Jianming Zhang, and Bedrich Benes. Ssn: Soft shadow network for image compositing, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4380-4390, 2021.
[37] Konstantin Sofiiuk, Polina Popenova, and Anton Konushin. Foreground-aware semantic rep- resentations for image harmonization. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1620-1629, 2021.
[38] Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang. Soo Ye Kim, and Daniel Aliaga. Objectstitch: Object compositing with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18310-18319, 2023.
[39] Kalyan Sunkavalli, Micah K Johnson, Wojeiech Matusik, and Hanspeter Pfister. Multi-scale image harmonization, ACM Transactions on Graphics (TOG), 29(4):1-10, 2010.
[40] Gemma Canet Tarrés, Zhe Lin, Zhifei Zhang, Jianming Zhang. Yizhi Song, Dan Ruta, Andrew Gilbert, John Collomosse, and Soo Ye Kim. Thinking outside the bbox: Unconstrained generative object compositing. arXiv preprint arXiv:2409.04555 2024.
[41] Benjamin Ummenhofer, Sanskar Agrawal, Rene Sepulveda, Yixing Lao, Kai Zhang, Tianhang Cheng, Stephan Richter, Shenlong Wang, and German Ros. Objects with lighting: real-world dataset for evaluating reconstruction and rendering for object relighting. In 2024 International Conference on 3D Vision (3DV), pages 137-147. IEEE, 2024.
[42] Lucas Valença, Jinsong Zhang, Michaël Gharbi, Yannick Hold-Geoffroy, and Jean-François Lalonde. Shadow harmonization for realistic compositing. In SIGGRAPH Asia 2023 Conference Papers, pages 1-12, 2023.
[43] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2025.
[44] Ke Wang, Michaël Gharbi, He Zhang. Zhihao Xia, and Eli Shechtman. Semi-supervised parametric real-world image harmonization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5927-5936, 2023.
[45] Tianyu Wang. Jiamming Zhang, Haitian Zheng, Zhihong Ding, Scott Cohen, Zhe Lin, Wei Xiong, Chi-Wing Fu, Luis Figueroa, and Soo Ye Kim. Metashadow: Object-centered shadow detection, removal, and synthesis, 2024.
[46] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IECE transactions image processing, (4)):600 612, 2004.
[47] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4840-4851, 2024.
[48] Xiaoyan Xing, Konrad Groh, Sezer Karaoglu, Theo Gevers, and Anand Bhattad. Lu- minet: Latent intrinsics meets diffusion models for indoor scene relighting. arXiv preprint arXiv:2412.001177200 2024.
[49] Ben Xue, Shenghui Ran, Quan Chen, Rongfei Jia, Binqiang Zhao, and Xing Tang. Doef: Deep comprehensible color filter learning framework for high-resolution image harmonization. In European conference on computer vision, pages 300-316. Springer, 2022.
[50] Kanamori Yoshihiro. Relighting humans: occlusion-aware inverse rendering for full-body human images. ACM Trans. Graph., 37:270-1, 2018.
[51] Ye Yu, Abhimitra Meka, Mohamed Elgharib, Hans-Peter Seidel, Christian Theobalt, and William AP Smith. Self-supervised outdoor scene relighting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pages 101. Springer, 2020.
13

---

[52] Zheng Zeng, Valentin Deschaintre, Iliaan Georgiev, Yannick Hold- Geoffroy, Yiwei Hu, Fujun Luan, Ling- Qi Yan, and Miloš Hašan. Rgb-x: Image decomposition and synthesis using material- and lighting-awar diffusion models. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Pupers '24, SIGGRAPH '24, page 1-11. ACM, July 2024.
[53] Bo Zhang, Yuxuan Duan, Jun Lan, Yan Hong. Huijia Zhu, Weiqiang Wang, and Li Niu. Control- com: Controllable image composition using diffusion model. arXiv preprint arXiv:2308. 100400 2023.
[54] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Scaling in-the-wild training for diffusion- based illumination harmonization and editing by imposing consistent light transport. In The Thirteenth International Conference Learning Representations.
[55] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pages 586-595, 2018.
[56] Shuyang Zhang, Runze Liang and Miao Wang. Shadowgan: Shadow synthesis for virtual objects with conditional adversarial networks. Computational Visual Media, 5:105-115, 2019,
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We described the method and its quality.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitation in the main content.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We discussed the reason in the Exp section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We described clearly in the method and exp section.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release the code after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pbllc/guideeee http://oeeuuiiisssoooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification:
We described clearly in the Exp section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Don't need.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide them in the implementation section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: No Broader impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: No risk
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cited them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We described well the proposed dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/modde part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We provide user study the supplementary material, and describe the method of investigation, sample capacity and quantitative results in detail.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMS if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //trrr/neuippsp cc/ Coonferenncee/2225/ for what should or should be described.
21


Reviews:
[{"review_id": "2opTPFPGAe", "reviewer": "Reviewer_LrgJ", "summary": "MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation introduces a framework for image harmonization in the context of object compositing in 2D or 3D. Given an composited image or sequence of images from a 3D scene, the original background image, corresponding depth maps and camera poses, the pipeline first performs 2D adjustments on each image individually using a transformer network, before transforming the results into point clouds and subsequently Gaussians, that are projected using a 3D Hilbert curve before being processed by a second transformer. The output color information of the second network is projected back into Gaussian space (using the inverse Hilbert projection) and then rendered to apply photometric loss or to extract the final result image. For 2D input the second part of the pipeline can be omitted. The method is evaluated on the FOSCom and Objects With Lighting benchmark as well as a newly curated dataset. The proposed method achieves state-of-the-art performance and shows visual promising results even on self-emitting objects which are often not well represented in prior work.", "questions": "- How well does the method generalize to arbitrary scenes that don't feature a table with objects placed on top (e.g. landscape background and some foreground objects or even people)?\n- What is the \"iterative bias\" (abstract, l.12) of diffusion-based methods in this context? Do you mean the exposure bias? Overall it seems like diffusion models can generate very high quality results by now, especially for tasks like image manipulation and generation.\n- How can the image resolution be increased? 256x 256 is very small for today's image processing pipelines. \n- Can the training times be reduced by using more available pre-trained components (especially the 2D object network)?\n- What perceptual loss is used during training (sec. 3.4)? Is it LPIPS as during evaluation?\n- Would the method work with a single image input and monocular depth?\n- What it the user study documented in the supplementary material? How many subjects have been asked? Were they human? If so, also the NeurIPS checklist needs to be updated. I would suggest to delete the study or discuss the results properly.", "limitations": "Limitations are (only) discussed in the supplementary material.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Fast inference that proves to be effective for both 2D and 3D inputs.\n- Quantitative results look promising. Interestingly also the \n- Extensive evaluation including multiple open benchmarks and captured real-world data.\n\nWeaknesses:\n- The motivation for the individual components could be made clearer in the method section, maybe also incorporating prior works. The ablation study (sec.4.4) is the only part where some motivation is given for the component choices. \n- The Gaussian representation needs to be initialized for each scene which takes about a minute extra. This means that the fast inference only pays off if many views of a single scene are processed.\n- Long training times. The proposed pipeline seems complex, featuring two transformer networks and multiple representation changes.\n- Image resolution is small (256 x 256) limiting the applicability. \n- Text clarity could be improved in some areas. E.g. First sentence of the abstract (l.3) could introduce object compositing with some motivation, section 3.1, l.117 starts with an incomplete sentence (or typo at \"Begin\"?). Overall, it is not that clear what types of inputs are supported during inference and wether a Gaussian representation needs to be fitted to the input at all times. For the single view evaluation only the 2D network is run, isn't it? The used training dataset should be mentioned in the training section.\n- It seems like the model has only been trained on scenes featuring a table with objects placed on top. It is not clear how well this generalized to other scene content.\n\nOverall, the paper proposes a complex but effective method to harmonize composited scenes in 2D or 3D that is limited by the low processing resolution and the need for camera pose data.", "was_revised": true, "final_justification": "Given the other proposed refinements from the rebuttal, specifically the motivation for the Hilbert Curve mapping and choice of losses, the information on the user study and the revised text including a clearer overview of model inputs, I can see the work fit for acceptance to the conference. The dependence and relationship to the input 3D representation should be discussed in more detail as it seems to be the main limitation.", "rebuttal": "We thank all your valuable comments of our paper. The detailed responses are shown below.\n\n**W1: Motivation for Individual Components**\n\n**Response:** We clear the motivation for two key components below.\n\n- **Swin Transformer:** The Swin Transformer introduces **shifted-window attention**, achieving linear computational complexity while maintaining hardware efficiency through its hierarchical pyramid structure. It demonstrates superior performance over the original transformer across multiple vision tasks (**+2.7 box AP on COCO detection** and **+3.2 mIoU on ADE20K segmentation**), while also requiring **fewer FLOPS (-45.5%)** and **parameters (-35.8%)** at the same input resolution. Consequently, it serves as the backbone for several widely-used models like **Grounded DINO** and **DINO-X**. In our work, we adopt the Swin Transformer as the backbone for both our 2D and 3D object compositing models. (Quantitative comparison is from the paper \"Swin Transformer: Hierarchical Vision Transformer using Shifted Windows\")\n- **Hilbert Curve Mapping:** Inspired by recent advancements in processing point clouds with transformers, like **Point Transformer V3**, we utilize the Hilbert curve maximally preserves **spatial locality**, which means points that are close in 3D remain closing in the 1D sequence. Then, the ordered sequence is then folded into a 2D grid, allowing our transformer to effectively process the geometric structure of the scene. The ablation study (e.g. PSNR in the 3D task falls from 30.29 to 28.99) provides clear empirical evidence for the Hilbert curve's benefit.\n\n**W2: Concerns about Gaussian Initialization**\n\n**Response:** We appreciate the reviewer's concern regarding efficiency. We clarify two key points:\n\n1. The Gaussian initialization is a **one-time, per-scene preprocessing step**, not a per-inference overhead. We clarify that the one-minute training for the Gaussians was chosen to achieve the highest reconstruction quality and we can **trade off this training time for quality**. To demonstrate this, we conducte a comparison experiment below. Once this 3D scene representation is constructed, our framework can perform harmonization for any number of viewpoints or even for newly inserted objects at a rapid speed (e.g., 0.07 seconds per frame). This initial cost should be viewed as analogous to loading or preparing a 3D asset, after which **real-time interaction** becomes possible.\n2. Our method is fully compatible with the latest advancements in **feed-forward** 3D reconstruction. Recent geometry foundation models can generate 3DGS representations directly from sparse inputs in a single feed-forward pass. To demonstrate this, we conduct experiments using **AnySplat**, which generates a 3DGS scene from input frames without per-scene optimization, drastically reducing the initial setup time.\n\n|Paradigm|Method|Simple Synthetic Scene (PSNR)|Simple Synthetic Scene (SSIM)|Simple Synthetic Scene (LPIPS)|Complex Synthetic Scene (PSNR)|Complex Synthetic Scene (SSIM)|Complex Synthetic Scene (LPIPS)|Real Captured Scene (PSNR)|Real Captured Scene (SSIM)|Real Captured Scene (LPIPS)|Time(s)|\n|---|---|---|---|---|---|---|---|---|---|---|---|\n|Per-scene|3DGS(#10k iter)|30.29|0.960|0.030|30.13|0.952|0.027|26.39|0.927|0.040|64.8|\n|Per-scene|3DGS(#1k iter)|28.43|0.926|0.076|28.54|0.895|0.092|25.24|0.899|0.081|6.5|\n|Per-scene|3DGS(#250 iter)|26.50|0.892|0.115|27.16|0.869|0.126|22.03|0.853|0.148|1.6|\n|Feed-forward|AnySplat|22.28|0.705|0.211|30.03|0.935|0.050|24.46|0.864|0.095|0.8|\n\nThe results reveal a clear trade-off between reconstruction paradigm, speed, and scene complexity. AnySplat achieves the fastest reconstruction time at **under one second** and delivers quality nearly on par with the highest-quality, per-scene optimized 3DGS in complex, **forward-facing** scenes. However, its performance significantly degrades in simple, **surround-view** scenes, likely due to the inherent challenge of jointly estimating camera poses. Meanwhile, by reducing training iterations of the vanilla 3DGS to 1k, it produces comparable results in **just a few seconds**, offering a practical compromise between the instant inference of AnySplat and the lengthy optimization of a full 10k-iteration run.\n\n**W3: Training Times and Model Complexity** \n\n**Response:** We acknowledge that the training time (15+3=18 days on 16 A100 GPUs) is quite long. However, this is standard for training large-scale, feed-forward vision models from scratch on a new, large dataset. For context, comparable large-scale models like VGGT requires 9 days of training on 64 A100 GPUs and Fast3r requires 6.13 days of training on 128 A100 GPUs. In terms of total GPU-hours, our training budget is in a similar range.\n\nRegarding complexity, while our pipeline involves two stages, the core transformer architecture is nearly identical for both the 2D and 3D models, as illustrated in Figure 2 of our supplementary material. We propose a unified architecture for both 2D and 3D tasks with different data representations.\n\n**W4&Q3: Image Resolution**\n\n**Response:** Our model supports inference at a resolution of **2K**, as shown in Figure 7. Furthermore, we are also conducting training with higher resolution (**512x512**) and will release the pretrained models with different resolutions.\n\n**W5: Text Clarity and Supported Inputs**\n\n**Response:** We thank the reviewer for these actionable suggestions. We will incorporate all proposed textual edits, including adding a definition for object compositing in the abstract and correcting the typo in Section 3.1. We will also clarify the supported input modalities and the training data details.\n\n- **Supported Inputs:**\n  1. **Single-view image:** For a single view input, only the 2D Object Compositing model is used.\n  2. **Multi-view images with known poses:** This is the primary use case for our full pipeline, where the 3D model leverages the 3DGS representation to enforce multi-view consistency.\n  3. **Multi-view images without known poses:** This is also supported as an extension. As demonstrated in our supplementary material (Section A.7), camera poses can first be estimated with VGGT, after which our pipeline can proceed.\n\nThe training dataset we used is simple synthetic scenes from the DTC-MultiLight dataset we proposed. The complex synthetic scenes and real captured scenes are used for out-of-domain evaluating.\n\n**W6&Q1: Generalization to Arbitrary Scenes**\n\n**Response:** We apologize if this was unclear. While we train models on tabletop scenes (simple synthetic scenes), our evaluation is deliberately performed on a much wider variety of out-of-domain content to test generalization:\n\n- **Complex Indoor Scenes:** Figure 4 and Figure 4&5 in the supplementary material show strong performance on the Objects With Lighting (OWL) dataset and complex synthetic scenes of DTC-MultiLight dataset, which features complex indoor environments.\n- **Diverse Outdoor Scenes:** Figure 4  in the supplementary material shows qualitative results on the FOSCom dataset, which includes varied scenes such as outdoor streets.\n- **Real-World Captures:** Figure 6 demonstrates robust performance on casually captured real-world scenes that differ significantly from our synthetic training data.\n\n**Q2: Clarification of \"Iterative Bias\"**\n\n**Response:**  I'm certain it refers to **exposure bias**. While these models produce high-quality textures, their stochastic and iterative nature can lead to results that are visually plausible but physically inconsistent, especially for precise effects like shadows and highlights.\n\n**Q4: Use of Pre-trained Components**\n\n**Response:** Thank you for your suggestion. We agree that using more available pre-trained components is a method to accelerate training like VGGT. However, Our network backbone is a specific architecture that takes a **multi-channel input tensor** concatenating the composite image, background, and depth map. Standard pre-trained weights from datasets like ImageNet are trained on 3-channel RGB images and would not be directly applicable. Meanwhile, the **hyperparameters** of the network architecture of our model are also different.\n\n**Q5: Perceptual Loss Details**\n\n**Response:** The perceptual loss Lp used during training is a VGG19-based feature loss, not the LPIPS metric. While both are derived from VGG features, LPIPS involves an additional learned linear layer on top of the features to better align with human perceptual judgments. Our training loss uses a standard L2 distance in the VGG feature space, a common and effective practice for image-to-image tasks.\n\n**Q6: Monocular Depth Input**\n\n**Response:**  For all our experiments on public datasets like FOSCom and OWL where ground-truth depth was unavailable, the depth maps were generated by **Depth Anything v2**.\n\n**Q7: User Study Details**\n\n**Response:** The user study was conducted via an online questionnaire and we collected **374 valid responses from human participants**. The study presented participants with side-by-side comparisons and asked them to choose which result demonstrated more realistic and consistent lighting and shadows. The two questions corresponded to the single-view and multi-view compositing tasks, respectively, with the results summarized in the pie charts on Figure 3 in the supplementary material."}, {"review_id": "EVkMPavy7S", "reviewer": "Reviewer_Zm8y", "summary": "The paper presents a method for inserting novel objects into existing scenes such that shading, shadows, occlusions and interreflections are properly modeled. Instead of performing intensive inverse rendering, the approach trains a forward prediction network that has been trained on many (about 480k) synthetic object insertion tasks with different illumination. The authors present a 2D pipeline where insertion into a single image is possible as well as a multi-view 3D setup where the resulting color adjustment of the newly inserted object as well as on the existing parts of the scene is predicted by altering the colors of a 3DGS representation. Here, the 3D neighborhood of Gaussians is mapped into 2D by use of 2D and 3D space-filling Hilbert curves. Both in the 2D as well as in the 3D object comopsiting model a Swin ViT block is trained to predict the necessary scene alterations. Extensive comparisons with other existing approaches, including also diffusion and inverse rendering attempts, indicate vastly better output quality for both synthetic and real-world scenes at very fast inference speeds (0,07s).", "questions": "For the task of inserting a luminous objects, which is also quite convincing, are there sufficiently many cases in the original training data set are those added later on?", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The paper addresses the problem of object compositing as a pure feed-forward approach by training on a massive synthetic dataset. \n- The presented results are highly convincing as long as the depth and the background mask are given. \n- The inference speed is way faster than existing approaches.\n- The object insertion data set is interesting. \n- The proposed mapping from 3D Gaussian to 2D maps using 3D and 2D Hilbert curves to train a 2D SWIN ViT for a 3D task is highly original.\n- The paper is well written with a clear motivation and quite clear presentation of the approach\n- The authors provide a quite exhaustive comparison to existing work \n\nWeaknesses\n- The authors do not discuss failure cases. It would be good to show some of the worst-performing examples and hint at why they might be challenging.", "was_revised": true, "final_justification": "After reading the other reviews and seeing the additional discussion necessary on the initial 3D alignment, I would slightly lower my score to 5.", "rebuttal": "We thank your appreciations and suggestions for our paper. The detailed responses are shown below.\n\n**W1: Discussion of Failure Cases**\n\n**Response:** Thanks for your excellent suggestion. We agree that a transparent discussion of limitations is essential. Due to space constraints, the limitation is presented in Section A.8 of the supplementary material. We also summarize some major failure cases as follows:\n\n- **Shadow Precision:** Our model can learn the general direction of highlights and shadows from the scene, but sometimes it fails to generate **accurate and hard-edged shadows**, as shown in the last column of Figure 5. The challenge lies in the fact that the information in the picture cannot determine **the position of the light source**, so creating sharp and accurate shadows is difficult.\n- **Reflective Surfaces:** Our model, trained primarily on diffuse materials, struggles to generate physically accurate reflections of composited objects on **specular or mirrored surfaces**. The challenge lies in capturing the characteristics of the materials from the images, and reflections and shadows are often easily confused.\n\n**Q1: Training Data for Luminous Objects**\n\n**Response:** Thank you for highlighting this. The original training dataset did not contain a large number of luminous objects. To specifically probe our model's ability to generalize to complex light transport phenomena, we curated a targeted dataset (about **1/3** of the scenes in the original dataset) where some objects were **assigned emissive material properties**. The strong performance on this task, as shown in Figure 7, even with relatively limited exposure during training, demonstrates the powerful generalization capability of our framework. We will clarify this in Section 4.3."}, {"review_id": "WIfhzkGbtI", "reviewer": "Reviewer_xTvT", "summary": "This paper proposes a method for object composition and scene relighting. A large-scale synthetic 3D compositing dataset is collected to train a feed-forward model. The model adopts a two-stage framework, consisting of 2D image-level compositing and 3D scene-level compositing, which are connected via a Hilbert curve-based mapping. Extensive experiments demonstrate that the proposed method outperforms previous state-of-the-art approaches.", "questions": "1. It is suggested that the video rendering results be visualized.\n2. It is suggested that more details be provided regarding the mapping function based on the 3D Hilbert curve. From Fig. 3, it appears that the 3D Gaussians in the scene occupy only a small portion of the full space. How is it ensured that multiple 3D Gaussian primitives are not assigned to the same curve point? How to deal with curve points that have no primitives assigned?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The quality of the results appears strong and promising. The paper is well-written and easy to follow.\n2. The Hilbert curve-based mapping is a novel texture mapping method of 3D GS.\n3. The experiments are thorough and demonstrate that the method achieves state-of-the-art performance.\n4. The proposed large-scale synthetic dataset could be valuable to the community and serve as a strong benchmark for this task.\n\nWeaknesses:\nNo major weaknesses identified.", "was_revised": true, "final_justification": "After reading the author's responses to my questions and other reviewers, I can see the author's efforts to address the issues and improve the paper. Although my biggest concern is the visual effect of the video results, I believe the author can handle them well in their revision, so I am inclined to maintain the score.", "rebuttal": "We thank your recognition and suggestions for our paper. The detailed responses are shown below.\n\n**Q1: Visualization of Video Results**\n\n**Response:**  Due to time constraints, we were unable to produce the final videos for the initial submission. We have since rendered several high-quality videos showcasing multi-object composition with smooth camera motion in complex scenes. These will be contained in the camera-ready version of the paper.\n\n**Q2: Details of Hilbert Curve Mapping**\n\n**Response:** Thank you for these insightful questions. We will add a more detailed explanation of the mapping process to.\n\n1. **Space Occupancy and Collisions:** The purpose of constructing the Hilbert curve is to preserve the spatial information of sparse Gaussians and we do not explicitly prevent multiple Gaussians from mapping to **the same** discrete point on the curve. Our process is as follows: we build a **KD-Tree** on the Hilbert curve's points for efficient lookup. Each Gaussian is then mapped to its nearest neighbor on the curve. The final 1D sequence is ordered first by position along the curve. If a \"collision\" occurs (multiple Gaussians map to the same point), their relative order is resolved using a stable sort based on their original index in the input list.\n2. **Empty Curve Points:** Curve points that have no assigned Gaussians are simply ignored. The final sequence passed to the transformer consists only of the ordered Gaussian features, with no gaps."}, {"review_id": "UaTcTgEoqr", "reviewer": "Reviewer_UGzN", "summary": "The paper presents a compositing framework capable of handling both 2D single-image and 3D multi-view compositing. The authors use an image transformer in the 2D stage. For the 3D stage, they apply a Hilbert curve to convert the 3D point cloud into a 2D image representation, allowing them to reuse the feature maps from the 2D stage as input. This conversion enables them to continue using the same image transformer architecture.", "questions": "I would appreciate if authors could clarify my concerns mentioned in weaknesses section. \n\n1- motivation behind use of hilbert curve + image transformer vs standard transformer\n2- clarification on the training data, evaluation data and user study.", "limitations": "While the paper discusses some limitations, it does not address the strong assumption that both the 3D model of the scene and the composited object are available. This assumption significantly limits the practical applicability of the method — especially in the 2D case, where acquiring a correctly aligned view of the object is non-trivial. The absence of any discussion on this constraint is notable.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strenghth:\n1. The new proposed dataset can be very usefull as training data for similar task. \n2. The Hilbert curve mapping is interesting and can be used for feature mapping in other tasks.\n\nWeaknesses:\n1. The method assumes that the composited object is perfectly aligned with the scene (3D representation of both composited object and background are available). This is a significant limitation. In the 2D case, a view of the object that precisely matches the scene’s viewpoint is not available and this alignment is non-trivial in practice if the 3D model is not available which is the case with general compositing problem. In contrast, diffusion-based approaches can internally handle misalignment by generating a new view that blends naturally with the scene.This can be more clear in Supp-Fig4 results. \n\n2. In the 3D case, the quality of the generated depth map depends heavily on the number and diversity of input views. However, the paper does not discuss how varying view coverage or sparse geometry affects the final compositing quality. There is also no discussion of robustness to errors in geometry estimation or alignment.\n\n3. The paper uses a Hilbert curve to convert the 3D Gaussian representation into a 2D texture to be processed by an image transformer, but this design choice is not well motivated. A standard transformer can operate directly on token sequences, including the 3D Gaussian features, without requiring 2D spatial layout. The ablation study includes a row where the Hilbert curve is removed, but it is unclear what architectural changes are made in that variant. More clarity is needed on how this version is implemented and how a basic transformer would perform without spatial mapping.\n\n4. Using metrics like SSIM and PSNR to compare against diffusion-based methods can be unfair. These metrics rely on pixel-level alignment with ground truth, while diffusion models often modify the composited region significantly. Therefore, a lower score does not necessarily reflect lower perceptual quality. Although the authors include a user study in the supplementary material, they do not explain how it was conducted — including details such as the number of samples, the dataset used, the number of participants, or the evaluation protocol. \n\n5. The paper does not specify which datasets were used during training. It is also unclear which evaluations are zero-shot (i.e., tested on entirely unseen datasets) and which are conducted on held-out splits from the same datasets used for training.", "was_revised": true, "final_justification": "The extra provided ablations and  the clarifications authors provide during the rebuttal addresses my concerns and I am convinced to increase my rating to accept.", "rebuttal": "We thank all your valuable comments of our paper. The detailed responses are shown below.\n\n**W1: Assumption of Perfect Object Alignment**\n\n**Response:** Thank you for your valuable suggestions. We acknowledge that our method requires an initial 3D alignment of the object. For example, we can align the bottom surface of the object with the plane of the scene.\n\nHowever, we clarify that the core contribution of our paper is illumination harmonization and consistent ligthing and shadow generation, not reconstruting the background scene or aligning the composited object, as described in Section 3.1.\n\nWhile diffusion-based models can often blend a misaligned 2D object into a single image, this generative flexibility becomes a critical weakness in a multi-view 3D context. Running a diffusion model independently on each view would almost certainly produce **inconsistent object geometry, lighting, and shadows**, shattering the 3D illusion. Our method is explicitly designed to solve this multi-view consistency problem, which necessitates a **consistent** 3D position for the object as a starting point.\n\n**W2: Impact of Sparse Geometry or View Coverage**\n\n**Response:** This point relates to the scope of our work. Our method assumes a **pre-existing, high-quality** 3D representation of the background scene (e.g., a pre-trained 3DGS). Consequently, the depth maps required by our model are **rendered from this representation**, rather than being generated from sparse views. Since 3D representation is known, the input views can be picked, thus we did not analyze the varying view coverage or the sparse geometry in the main text.\n\nTo explore this boundary condition, our supplement (Section A.7) includes an experiment where we use VGGT to estimate geometry from unposed images. The results show that our adjustments to the Gaussian reconstruction process effectively mitigated the impact of noisy input geometry on the harmonization quality.\n\n**W3: Motivation for Hilbert Curve vs. Standard Transforme**\n\n**Response:** The motivation is to preserve **spatial locality**. A standard transformer requires the input sequence order to be meaningful. For an unordered set of 3D points like Gaussians, an arbitrary ordering provides no useful spatial bias for the attention mechanism. The Hilbert curve imposes a **meaningful order**, ensuring that Gaussians that are close in 3D space are also close in the 1D sequence, which aids learning.\n\nFor the \"w/o Hilbert transform\" ablation in Table 3, the architectural change is in the input preparation. Instead of Hilbert ordering, we simply take the list of N Gaussians and perform a **naive reshape** into a pseudo-image. This is equivalent to ordering them by their arbitrary initial index. The resulting performance drop (e.g., PSNR in the 3D task falls from **30.29** to **28.99**) provides clear empirical evidence for the Hilbert curve's benefit. We will add this clarification to Section 4.4.\n\n**W4: Evaluation Metrics and User Study**\n\n**Response:** We agree that pixel-wise metrics have limitations for evaluating generative models. We use them because they are the most established quantitative standards for image restoration and translation tasks. To provide a more complete picture, we supplement these metrics with two crucial elements:\n\n1. **Extensive Qualitative Results:** We provide **numerous visual comparisons** in Figure 4~7 and Figure 4~6 in the supplementary material., which allow for direct perceptual judgment of quality.\n2. **Formal User Study:**  The user study was conducted via an online questionnaire and we collected **374 valid responses from human participants**. The study presented participants with side-by-side comparisons and asked them to choose which result demonstrated more realistic and consistent lighting and shadows. The two questions corresponded to the single-view and multi-view compositing tasks, respectively, with the results summarized in the pie charts on Figure 3 in the supplementary material.\n\n**W5: Training/Test Datasets and Zero-Shot Evaluation**\n\n**Response:** Thank you for the reminder, and we will make it explicit in Section 4.1.\n\n- **Training Data:** The model was trained exclusively on the \"simple synthetic scenes\" from our proposed DTC-MultiLight dataset except 50 scenes for evaluation.\n- **In-Domain Testing:** A held-out split of 50 \"simple synthetic scenes\" from DTC-MultiLight was used for in-domain evaluation, as shown in Figure 4&5 and Figure 5 in the supplementary material.\n- **Zero-Shot Testing:** All other evaluations were conducted in a zero-shot setting on entirely unseen datasets to test generalization. This includes:\n  - our \"complex synthetic scenes\" (Figure 4&5 and Figure 5 in the supplementary material).\n  - our \"real captured scenes\" (Figure 6).\n  - FOSCom and Objects With Lighting (OWL) dataset (Figure 4 and Figure 4 in the supplementary material)."}]

Author rebuttal:

Meta-review: This paper considers the task of image harmonization for inserting objects into existing scenes (given by an RGBD image or set of RGBD images and camera poses) such that various aspects relating to lighting (including shadows, shading, interreflections, occlusions, and self-emission) are modeled realistically. The identified strengths include the quality of the results, the inference speed, and the approach for mapping unstructured 3D Gaussians into a structured 2D grid via Hilbert curves, which could potentially be used in more general settings for processing Gaussian mixtures. The identified weaknesses mirror some of these strengths. Most reviewers identified the motivation and effectiveness of the Hilbert curve approach, especially compared to standard positional encoding approaches, as a cause for potential concern. Other concerns include the inference time, after considering the time required for per-scene initialization, the lack of failure cases, the lack of video results, and the hard requirement of perfectly scene-aligned composited objects.

The initial ratings were quite divergent (two borderline rejects, an accept and a strong accept). There was a considerable amount of discussion between authors and reviewers, which ultimately led the reviewers to form a consensus to accept the paper, contingent on several recommended revisions. The authors have consented to the following revisions: 
1. Providing the motivations for the main design choices (Hilbert curve, Swin Transformer), as outlined in the author response; 
2. Discussing the failure cases and assumptions (e.g., object alignment); 
3. Clarifying the data splits;
4. Including the new Hilbert curve ablation experiment; and
5. Including video results. 

Given this commitment, the AC sees no reason to override the consensus of the reviewers and is satisfied by the technical merit of the paper and its potential to make an impact on the field.

However, in addition to the revision items raised by the authors in their final response, the AC requests that the revision acknowledge that the underlying idea of using a Hilbert curve to process unstructured data comes from prior work, such as [A, B], which should be cited. In addition, the AC concurs with reviewer UGzN that a stronger baseline would be a standard transformer with a 3D positional encoding based on the Gaussian center positions; i.e., lending an inductive bias of spatial locality during self-attention. It is not clear from the author response ("we will use a pre-trained point transformer to extract positional and color features from the Gaussian primitives") that the provided experiment does this. One would assume that the primary benefit of the proposed approach over a standard positional encoding is that the resulting 2D grid facilitates the use of efficient 2D convolutions and Swin transformer layers.

[A] Wu et al., Point Transformer v3: Simpler, Faster, Stronger, CVPR 2024

[B] Chen et al., Efficient point cloud analysis using Hilbert curve, ECCV 2022
Decision: Accept (poster)