LuminAIRe: Illumination-Aware Conditional Image Repainting for Lighting-Realistic Generation
Authors: Jiajun Tang
Haofeng Zhong
Shuchen Weng
Boxin Shi
Venue: NeurIPS 2023 poster (2023)
Keywords: Illumination
Image Generation
Conditional Image Repainting
TL;DR: integrating explicit illumination constraints into current image geration pipeline with new dataset proposed.

Abstract: We present the ilLumination-Aware conditional Image Repainting (LuminAIRe) task to address the unrealistic lighting effects in recent conditional image repainting (CIR) methods. The environment lighting and 3D geometry conditions are explicitly estimated from given background images and parsing masks using a parametric lighting representation and learning-based priors. These 3D conditions are then converted into illumination images through the proposed physically-based illumination rendering and illumination attention module. With the injection of illumination images, physically-correct lighting information is fed into the lighting-realistic generation process and repainted images with harmonized lighting effects in both foreground and background regions can be acquired, whose superiority over the results of state-of-the-art methods is confirmed through extensive experiments. For facilitating and validating the LuminAIRe task, a new dataset Car-LuminAIRe with lighting annotations and rich appearance variants is collected.

LuminAlRe: IlluiinationAwarr Conditional Image Repainting for Lighting-Realistee Generation
Jiajun Tangl Haofeng Zhong Shuchen Wengl.2 Boxin Shi
National Key Laboratory for Multimedia Information Processing
National Engineering Research Center of Visual Technology
School of Computer Science, Peking University (jiajun. tang, hfzhong, shuchenweng, (abbxxin@@pku edu. cn
Abstract
We present the IlLumination-Awwee conditional Image Repainting (LuminAIRe) task to address the unrealistic lighting effects in recent conditional image repainting (CIR) methods. The environment lighting and 3D geometry conditions are explic- itly estimated from given background images and parsing masks using a parametric lighting representation and learning-bur -baedd priors. These 3D conditions are then converted into illumination images through the proposed physically- tion rendering and illumination attention module. With the injection of illumination images, physically-en correctt lighting information fed into the highting-realistee generation process and repainted images with harmonized lighting effects in both foreground and background regions can be acquired, whose superiority over the results of state-of-the-arr methods is confirmed through extensive experiments. For facilitating and validating the LuminAlRe task, new dataset CAR-LUMINAIRE with lighting annotations and rich appearance variants is collected.
illumina-
Introduction
Advanced image editing in high demand across multitude of applications, e.g., old photo
colorization [78, 32 68), damaged image restoration 48 73
72), and artistic style transfer [22] 35
770. Recently conditional image repainting (CIR) 667
66 58 has emerged as an innovative research topic, proven effective in controllable image editing while "freeing" users from the necessity of expert proficiency and retaining the "freedom" to actualize their creative visions for image modification. By utilizing provided attributes or textual descriptions, fine-grained strokes, and Gaussian noise to separately represent colors, contours, and texture conditions, users could insert generative objects with desired appearances in specified image positions, as shown in the blue line of Fig. [
Although CIR methods have made great progress in synthesizing photo-realistic and visually-pleasing conditional images by avoiding gradient vanishing pitfall (67), adopting flexible condition representa- tion (66), and designing condition fusion modules (58). there is still crucial element missing from the CIR task: making the synthesized results harmonized with the illumination of the scene, e.g., spatially-varying dark and bright regions in accordance to the lighting condition in the background, physically-accurat highlight effects for highly-specular surfaces (shining objects), and perceptually- realistic shadow avoiding "floating objects" artifacts, as shown in the lower right example of Fig. [
Specifically, existing CIR methods handle image harmonization purely in 2D image space by esti- mating a pixel-wise color transformation of the repainted regions from the background regions. Current approaches use semantic parsing maps as "geometry" representations and do not exploit the lighting information contained in given background images, which prevents them from having
Conresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Lighting
Geometry
Repainted bg.
Illuminatonn
Lighting Sohting-Realisti Realistic generation
LuminAIRe
sedan with dark glass, dark doar glass, and sky blue clearcoat car
Illumination attention
CiR
Conditional image- to-iimae generation
paint
image space
Background
Parsing mask
Attributes
3D space
LuminAIRe
Figure 1:
Illustraioon of proposed LuminAIRe task and result. Compared with the previous CIR task [58] (blue line) which takes all condition imputs at once conducting conditional image-to-image generation purely in 2D image space, LuminAIRe (orange line) exploits 3D lighting and geometry information and repaints both foreground (fg.) and background (bg.) regions via rghting-realistte generation process. The 3D information is transformed back to 2D image space in the form of an illumination image, with the desired reflective properties obtained from illumination attention module. LuminA ARRe handles (i) surface shading, (ii) highlight effects, and (iii) realistic shadow in the repainted image (top right).
awareness of physically-basse -bsed lighting in 3D space. To introduce physically-comee 3D lighting instead of hallucinated lighting effects into CIR results, there remain some major challenges: (i) The lighting condition 3D space should be extracted from the limited field of view (limited-FoV) 2D transformed back into 2D
LDR images; (ii) the lighting condition should be image space; (iii) dataset suitable for learning- ceaniing-bssed solutions the proposed task is needed.
To achieve lighting-reallstic generation within the CIR pipeline in an illumination-aware manner, hereby propose the task of ilLumination-A ware conditional Image Repainting, denoted as LuminAiRe. We first lift geometry conditions from 2D parsing maps to 31 normal maps using learning-based shape priors and estimate lighting conditions from limited-For LDR background images by designing a proper parametric representation. Then, we use physically- based reflection models render illumination candidate images capture possible lighting effects in 2D image space. With the help of illumination attention module, surface regions with different reflective properties are learned to adopt correct lighting effects in the resulting appearance. rich geometry and lighting annotations with abundant object variants is collected to facilitate the learning- based solution of the LuminAIRe task. As far as we know, we are the first emphasize saaaeeeeesonssaeesssssss in the image editing task of conditional image repainting.
dataset containing
Our contributions can be summarized as follows:
introducing a new task of ill Luminatioon A ware conditional Image Repainting LuminAARe) by exploiting the lighting information from background images;
designing a full LuminAIRe pipeline that represents, extracts, converts, and injects lighting information to acquire more realistically repainted results; and
collecting a new dataset MAR-LUMINAIRE with rich material and lighting condition variants for facilitating and validating the LuminAIRe task.
2
Related Work
Our method aims at introducing physical lighting constraints into generative image synthesis pipelines. In this section, we briefly review relevant works first and then discuss the relationships to our task.
Controllable image synthesis. Researchers have presented numerous works synthesize images under the guidance of diverse user- provided conditions, e.g., synthesizing specific object with category
label 10 43 45 75,, transferring the texture from paintings to daily photos [2 35
restoring
I Strictly speaking, the normal maps are in 2.5D. Here we use 3D simply distinguish from 2D. In this paper, attributes shown in templated sentences for formatting, and texture is omitted for simplicity.

---

the colors of old photos [1 69 68), and directly generating images from text descriptions
51
50 [55 [6]. Recently, with the development of the condition injection mechanism [i 80
researchers explore to control synthesized images with multiple cross-modality conditions, e.g..
condition guided image inpainting 44 555, controllable person synthesis [54, 65), and inversion-
based style transfer [79]. However, few works focus on synthesizing images strictly following lighting conditions. Following DIH-GAN [6] that considers introducing illumination estimation in harmonization task that adjusts the highlight of the inserted given object, we further explore the lighting condition in synthesizing illumination-conssterr objects under the guidance of multiple cross- moss-modality conditions.
Conditional image repainting and image harmonization. Conditional image repainting (CIR) aims at synthesizing reasonable visual content on an existing image, where the generated visual content should both meet the requirement of the user-provided conditions (e.g., color; geometry, and texture) and in harmonization with the existing background image. The first CIR task is proposed in MISC 67 for person image synthesis, where the foreground person image is synthesized first and then composited with the background. Weng et al. [66] design the semantic-brigge attention mechanism which allows more freely expressed color conditions by the users in text. UniCoRN (58) breaks the two-stage dependency and proposes unified architecture that achieves more visually pleasing results. Despite recent achievements made by previous works in condition consistency, existing CIR models suffer from the issue of illumination inconsistency: although techniques such as color tone transform are applied, the lighting from the given background and on the generated visual contents often differ lot, making lighting effects in the image rather unrealistic, such incorrect shading. highlights, and shadows. In this paper, we address this issue by exploiting lighting and shape constraints in 3D space, which allows more physically-comees rendering processing for generating lighting effects. Image harmonization methods [23 24 25
59] with a similar goal of CIR to realistically composite image foreground and background regions, have focused on illumination harmonization recently [6] g However, this thread of works has poor control of visual content in foreground regions and may fail to preserve the color tone in background regions as they were.
Lighting representation and estimation. Achieving Illumination-awann synthesis/repainting re- quires appropriate lighting representation and estimation from images. Lalonde al. [37] is the first to use shadows, shading, and sky appearance variations observed in the image to infer outdoor lighting. A physics-based Hošek-Wilkie (HW) sky model (2 30 is proposed to recover HDR parameters for deep outdoor lighting estimation (28). A more weather-robust Lalonde-Matthers (LM) model [38 77 is then proposed to cover more comprehensive lighting conditions in the outdoor environment, More recently, a learning-baa based lighting representation (27) is used on a large sky panorama dataset (36] with an autoencoder network. The encoder-decoder framework is further proposed
39 to estimates lighting as a spherical HDR lighting map. HDSky 7Z and SOLD-Net [60] disentangle several physically meaningful attributes into separate learned latent spaces by hierarchical autoencoders and make the estimation editable. Parametric models such as spherical harmonic (SH) coefficients 21) and spherical Gaussian (SG) [19 40) are also widely used, especially in indoor scenes. Gardner et al. [20]. Neurlllum (57). and SOLID-Net [8]] design sophisticated networks to hallucinate the missing parts in the panoramic view and predict lighting as environment maps. 3D volumetric lighting representations are also widely used in recent works, which facilitate the lighting-reallstic scene editing for indoor 41 and outdoor 644 scenes, however heavily require computation resources. Considering the demand for lighting-realistic generation, we propose parametric lighting representation for outdoor scenes that is both easy to predict and simple to use.
3 Problem Formulation
For seff-containedness, we briefly review the CIR formulation before introducing ours.
3.1 Preliminaries about CIR
The previous CIR tasks (66 67 (58) aim generating the repainted image y" by repainting certain regions in an image x E x according to user-specified conditions in different modalities: x", xP, _x. and b for the "geometry", "texture", "color", and background conditions respectively.
In their works, the "geometry" condition x E Hx is a binary semantic parsing mask, where Ng is the number of possible parts of the visual content to be repainted and L = {0, 1]; the "texture"
3

---

condition 2 ~ (0, 1) is a Gaussian noise: the "color" condition can be represented as attributes x² E ILNoNN, or text descriptions xº = {=(] Nr, where No. Ny. and Ni represent the numbers of attributes and available choices, and the length the user-inputted sentences, respectively; the background condition x² E R is the image of background region with respect to the repainted
region as foreground region, f.e., x² = (1 - m) @ 2, where the binary mask m indicating foreground region can be directly acquired from the parsing mask 2², as shown in lower left of Fig. |
The repainted image y" can be further decomposed as blending of repainted foreground image yf and repainted background image y":
y' = m C y + (1 m) + v..
Previous works assume unchanged background region, ie.. y" = zh, leaving the key question of CIR tasks as generating realistic foreground region yf constrained by given conditions:
y' = F x", ,xxxx²),
where previous works ignore clues in 3D space and implement the generation pipeline F as a conditional image-to-imant to-image generation purely in 2D image space. To make the repainted image harmonized as a whole, previous works [58] 67 design additional harmonization modules to adjust the color tone of intermediate repainting result based on clues in xbb
3.2 Formulation of AuminAARe
However, the image-based harmonization modules have limited representation ability for complex lighting effects (e.g.. varying shading and shiny surfaces) due to a lack of 3D representation. Besides, directly using n as y Eq. [ may neglect possible light transport effects (e.g., shadows) introduced by the repainted region its corresponding behaviors in the 3D real world might be.
As illustrated by the rendering equation [33] physically-correct and -realistic appearance of an object is derived from its geometry, reflective property, and omnidirectional environment lighting in 3D space. Therefore, to make the repainted image y" more lighting-reant realistic, the repainted foreground y should also be conditioned by the lighting condition L and geometry condition G in 3D space:
yf = F (x
L,,,,,,, G).
(3)
Given L and G in 3D space, proper 2D representation x¹ containing both the information from L and G should be derived for compatibility with current image generation architectures:
R (L,
and then the lighting-realistic generation for foreground of can be rewritten as:
FF (a x., x², x aa)).
(5)
Similarly, the repainted background و should also be conditioned on a² recover lighting effects:
b= PB (T.b x²))
The limited-FoV background image b itself is partial observation of environment lighting and thus can provide clues about L. Therefore, the lighting condition can be inferred in the form of:
I. F.*")
Similarly. by finding the shape priors of certain types of objects, the 3D geometry condition G can be lifted from its "2D flattened version". i.e., parsing mask 2e:
G = Geo (z
(8)
Moreover, in our LuminAIRe formulation, we extend the attributes x beyond colors, which allows the users to describe the reflective property and have control over the lighting effects of repainting results. A sample of attributes is shown as the bold text in the lower left of Fig.
As aforementioned, both the repainted foreground yf and background yd are given by the lighting- realistic generation in our LuminAIRe formulation, which leads to more realistic and harmonized results than traditional CIR pipelines [58), as shown in Fig. [

---

HDR envmap
Aligned envmap
Enhanced materials
warping
Nc
High-res. LDR panorama
2D Insertion point
3D positioning
3D labeling
Lighting-amotated background cropping
Roalistic object insertion
Data sample
Figure 2: Data preparation process of the CAR-LUMINAIRE dataset.
4 Data Preparation
To tackle the data shortage issue, we create the first dataset suitable for the LuminAIRe task, named CAR-LUMINAIRE, with its data preparation process and data sample shown in Fig. 2
3D car models with hierarchical semantic labeling. Collecting large-scale real data for learning- based LuminAIRe methods is infeasible since the geometry and lighting capture in 3D space requires specialized equipment and extensive human labor. Therefore, here we resort to computer graphics techniques to create photo-realistic synthetic data. The cars are chosen as the foreground objects for the obviousness of lighting effects and the availability of high-quality synthetic models. We collect 198 detailed 3D car models in 17 different categories from online model stores (2] 4 and then label the parts of the models in 3D space, which allows us to get the accurate parsing mask in 2D image space from any viewpoint. Following the common structure of vehicles, we divide the car models into 35 semantic part labels. The part labels are organized sub-part of the door) to accommodate car models in different granularity. Besides 3D labeling, we manually adjust the scales of each model to fit the real-world dimensions.
hierarchical way (e.g.. the door window is
Background images with lighting annotations. Then we prepare background images with known lighting annotations. Here we use the SUN360-HDR dataset (27 (6),, which contains HDR panoramic enviromment maps (envmaps) corresponding to the LDR panoramas of outdoor scenes in the SUN360 dataset 77. Limited field-of-view (limited-FoV) background images are cropped from the LDR panoramas with virtual cameras of randomized Fs and camera poses. For each cropped background warped align with the viewing direction of the virtual camera. Background images unsuitable for realistic object insertion are manually filtered out, leaving 1,321 images of diverse scenes and lighting conditions.
image, the corresponding HDR envmap in the SUN360-HDD dataset (27 76
Enhanced data rendering with realistic placement. For each background image, we randomly select insertion points within the central region of the "placeable flat ground" marked by off- the-shelf segmentation toolbox [15]. Then, for each 2D insertion point in the image, we calculate the relative transformation from the camera coordination O, to the local coordination of the object O, from the depth d and the normal Zo estimated by depth [52] 53 and normal [5] estimation methods. With the aligned envmaps and the ray-tracing based Blender 3 Cycles rendering engine, soysically-correc lighting effects can be rendered into the composited images. In the rendering process, besides the original materials of the models, several physics- based rendering (PBR) car paint materials are randomly applied for more appearance variants, especially in lighting effects; besides, the inserted models are randomly rotated around Z, axis for more geometry variants. The rendered images are filtered to ensure reasonable pixel portions of both foreground and background regions. At last, 52,581 composited images at the resolution of 256 256 are collected, accompanied by parsing mask and normal map annotations, shown in the data sample of Figure 2
Method
To realize the LuminAIRe formulation, we first estimate 3D lighting and geometry from background images and parsing masks (Sec. Then the lighting information is injected into the lighting- realistic generation process as illumination images (Sec.15.2 By further introducing hierarchical labeling enhancement (Sec. 5.33) our method can generate reasonable results even with coarse-level parsing masks. Our pipeline is shown in Fig. [ with detailed network architectures and loss functions for network modules in supplementary materials.
5

---

NeiL
Lighting parameters
Illumination candidates (3
Feat. emb.
NetB
Repainted
Atritulles Background
Parsing mask x
NetS
map
Illumiationn
Bg.
Maek m
Marrhbaoo encooder
with normal wind glass, normal door glass, and light
Fest. emb.
Hierarchical
gray frostad
car peint
broadcast
NelF
Repainted yf
Repainting y
Illumination injection
Input conditions
Feature embedding map
+
Element-wise addition
Figure 3: Overview of LuminAlRe pipeline.
Limited-FoV image
Image
Limited-FoV Image
Figure 4: Our lighting representation can capture most of the lighting effects in all weather conditions.
5.1
Estimating 3D Information with Learning-based Priors
Our CAR-LUMINA dataset consists of outdoor scene images, where the lighting can be ap- proximately decomposed into the high-frequency sunlight and the low-frequency ambient light [60]. Accordingly, we model the lighting condition L as the addition of a directional light and 2-nd order spherical harmonics (SH) lighting, which can be specifically described as lighting parameters:
L {zvis Piat. Pang, Csun, Iunn JSH
where Zvis {0, 1} is the visibility, 2int is the intensity of sunlight, Zang describes the "size" of the (in solid angle formally), Csun IR is the normalized sun color in RGB channels, 4un R2 indicates the sun position, and dsı1 E _3x9 is the 2-nd order SH coefficients for RGB channels.
As shown in Fig. the parametric representation in Eq. (9) can well fit real-world lighting in sunny, cloudy, and low light conditions. On the other hand, the proposed parametric lighting representation is convenient for network prediction. Here we design NeiL to serve as F in Eq. 77). where 4un is estimated by classification task and other parameters are estimated by regression tasks. To apply method to other types of background scenes, specifically tailored lighting representations can be directly adopted, without modification to our underlying formulation of LuminAIRe.
For 3D geometry, we use the normal map G E 13x#xW as the representation where each pixel indicates the surface normal direction that surface point in 3D space. For certain types of objects, there exist strong shape priors (such as sedans and hatchbacks), which can be learned in supervised way. Similarly, NetS of encoder-decoder structure further proposed to serve as Geo in Eq.
5.2 Injecting Lighting Information using Illumination Images
To bridge the 3D lighting and geometry with 2D images, the rendering equation [33 is a handy tool to serve R in Eq. 4). which physically models the image formation process the light reflection:
Lo(w)) [ L(wi) (n wi)dw,
(10)
where L,(w,) is the environment lighting from direction w;, L.(w,) is the reflected lighting toward direction w, Qn is the visible hemisphere determined by surface normal n. and fi(wi, w,, describes the reflective properties of all possible combination of incoming and outgoing directions.
LLghhiig parameters converted back tone map-mapped HDR environment maps for visualization.

---

P
2
Appearance
Illumination images
Figure 5: Our illumination candidate images can cover realistic lighting effects in appearance variants.
For certain image pixel with the known camera viewing direction v, ideally. the pixel intensity can be calculated as Lo(-v) and accurate lighting effects can be calculated as illumination images.
However, with L and G' estimated from imput conditions, fi(w,, wr) still remains unknown. Therefore, in a similar spirit to Gao ef al. [18] and Pandey et al. [46]. instead of directly calculating the actual illumination image,
fr in Eq. (TO) and render corresponding illumination candidate images {Ie}. For the physics-based rendering of {Te). we the Lambertian refectance model [amrww,, w.) 1/a and normalized Blinn-Phong model [9] Sspec(es w,) = (o+t)(n-h"/888 with M different values of roughness p. where h w.tt// |w++w: the half vector. At last, we have {Te) = [Tamr U [
set of uncolored "standard materials"
As shown in Fig. 5 most lighting effects in different appearance variants can be covered by the linear combinations the pre-computed {1a}. However, it's worth noting that the correspondence of the appearance image and { may vary pixel-wisely (e.g., the tires, hood, and windshield have different reflective properties thus different lighting effects). Accordingly. we design an illumination attention module A to estimate the combination coefficient maps Cj A' A'(E) for each image pixel, where E is the feature embedding map containing information of both part labels and aast-associated attributes in a pixel-aligned way. After the illumination image / derived as / > Σ -M-1 Ci C L2 which covers lighting effects of parts with different materials, we use / as x² in Eq. ( and conduct aeghinn-realistic generations of foreground and background regions using our proposed NerF and NerB respectively. For NetF, we adopt the network backbone of F in UniCoRN (58), and the illumination image I injected in similar way as other conditions 2D image space at different resolutions. The NetB is also an encoder-decode architecture, serving as FB in Eq. (6). We adopt the same loss functions for NetF as used in UniCoRN
58
5.3 Generating Realistic Results from Coarse Parsing Masks
As mentioned in Sec. 4 the parsing masks in our CAR-LUMINAIRE dataset can be very coarse, which also reflects the possible application scenarios when the user only specifies interested parts. Previous CIR formulations may fail to generate realistic results in regions without fine-graines labels since their generation follows strictly pixel-wise semantic mapping between labels and images. We hereby introduce hierarchical labeling enhancement: randomly coarsening the input parsing mask training time (e.g., door glass label becomes door label) and encouraging the fine-grained parts (door glass) to be generated. Besides, the part-associated attributes of lower-level parts (door glass) should also associated with their upper-level parts (door) to avoid loss of condition in attributes xx which can be done by modifying the association matrix 58
A E L NNNNN accordingly.
6 Experiments
In this section, we conduct comparisons with state-of-the-art methods and validate our design with an ablation study and robustness test. Please see supplementary materials for implementation details.
6.1 Comparison with State-of-the-art Methods
Baseline methods. We conduct quantitative and qualitative comparisons with three state-of-the the-art CIR methods (UniCoRN (58), Weng al. [66), and MISC (67]) and mest-relevant conditional image generation method (Pavllo et al. 44 Among them, modifications are made for Payllo et al. 49 and Weng et al. [66] accept conditions represented as attributes.
Quantitative metrics. Following previous work [58), we adopt Fréchet inception distance (FID) 26 for assessment of perception quality, R- R-precision [66] for assessment of alignment between generated
7

---

images y' and given attributes x², and M-score [61] for assessment of authenticity. We use the latest manipulation detection model [T7 13 for calculating the M-score [6]] We also report the structural
similarity index (SSIM) 63 for comparing the major image structure with the reference image.
Table Comparison with the state-of-the-ee ahe methods and variants our proposed method. Quantitative evaluation scores and user study results are shown. t (4) means higher (lower) is better. "Real." and "Har." are abbreviations of "Realistic" and "Harmonized".
Quantitarive Evaluation
Study
Method
FID 4
R R-pren
M M-score
SSIM
Real. 1
Har. 1
MISC 67
51.84
34, 34,94%
31.25
0.6660
0.25%
0.28%
Weng
0.6697
Pavllo et ml. 4p
9.29
56.98%
36.77
0.7050
43.00%
36.72%
UniCoRN 58
11.55
62.13%
29.72
0.6940
7.78%.
9.90%
LuminAIRe (Ours)
4.62
74.13%
13.68
0.7211
48.12%
52.25%
63,27%
DuA--HA
6.31
63.94%%
13.95
0.7214
Ours-IIA
8.00
62.13%
15.83
0.7054
Conds.
MISC
Weng et al.
Pavllo et
UniCoRN
LuminAIRe
Reference
hatchback vind glass. door glass, roof glass, silver carpaint
hatchback with normal wind glass, normal door glass, normal reof sky blue diffuse carpaint
a sedan with normal wind glass. dark door glass. and sky blue metallic carpaint
Figure 6: Qualitative comparison with the state-of-the-art methods, with given conditions (conds.).
The scores in Tab. and the second and the third columns of Fig. show that results of MISC 67
appearances, since the color tone transform not applied [66]. or conducted in a two-phase manner [67]. As shown in the fourth column of Fig. 6 Pavllo al. [49] tend generate foreground regions in flat shadings with fewer texture patterns, which makes its results generally look reasonable when only focusing foreground regions in low light or cloudy scenes (as indicated by the FID and user study results), but computer vision models can easily find the disharmony due the sharp boundaries between foreground and background regions [58), as also indicated by the worst M-score. UniCoRN (58) fails to generate correct lighting effects from its unified color tone transform (the first row), therefore tends hallucinate highlights at the top of cars regardless of lighting in background regions (the second row). The hallucinated lighting effects along with the undesired texture pattern on car bodies drastically damage the perceptual preferences, as confirmed by the FID score and user study results in Tab. [ LuminAIRe generates realistic lighting effects close to the reference images in both sunny (the first and the third rows) and cloudy (the second row) scenes of specified materials and even when coarse-level parsing mask is given (the third row), with large margin in all quantitative metrics compared with baseline methods. LuminAIRe also learns to avoid the undesired texture pattern with the hints of the smoothly varied shading in the illumination images (Fig. 7
and Weng et al. [66] far from righting-realisti with gerayon-drawing

---

Conds:
Geometry
Illumination
Ours
Ours-H
Ours-HA
Ours-HAI
liftback with normal wind glass, normal door glass, normal roof glass, and blue gray metallic carpaint
liftback normal wind gless, normal door glass. dark rool glees, and light grey metallic carpeint
Figure 7: Ablation study for three variants of proposed method, with given conditions (conds.).
CUV with normal vind glass, dark door glass, dark roof glass, light gray diffuse clearceat paint
CUV with light wind glass, light door light roof glass, gray metallic point
Material editing
Lighting consistency
Figure 8: Our method can generate realistic lighting effects with given materials (left), which consistent across different geometry conditions (right). Green and blue boxes mark individual cases.
6.2 Evaluations
Ablation study. We conduct an ablation study with three variants of Ours: (i) Ours-H, (ii) Ours-HA, and (iii) Ours-HAL where "-H", "-A", and "-I" mean disabling the hicrarchical labeling enhancement, the illumination attention, and the illumination injection for the foreground, respectively.
The hierarchical labeling enhancement is confirmed helpful in generating realistic results with coarse- level parsing masks, as shown in Fig. / and the third row of Fig. [6 where Ours generates more regions with specified part labels (marked in blue purple),
consistent and better repaintings which is also demonstrated by the FID and R-pren score in Tab. The second row of Fig. shows example where the lack of illumination attention module wrongly renders a diffuse appearance, with further evidence from the drop of FID from Ours-H to Ours-HA in Tab. [ It's quite obvious from Tab. ! and Fig. 7 that the illumination injection helps foreground generation by comparaing Ours-HA and Ours-HAL From UniCoRN to Ours-HAL, the improvements in FID score and M-score validate the contribution of the lighting-rellsticaal generated background.
Besides, Ours-HA gets an unexpectedly good SSIM score. It's possibly because a slight misalignment of lighting effects (especially highlights) due errors in lighting geometry estimation would lead to a considerable drop in the SSIM score (which honestly measures the pixel-wise difference) but with very little harm the lighting-realistic perception (as indicated by the FID and M-score).
User study. We also conduct a user study with 20 volunteers on the Amazon Mechanical Turk M platform, where 200 sets of results randomly drawn from the test set shown and volunteers asked to choose one each set with (i) the most realistic foreground and (ii) the most harmonized lighting. The results of the user study in Tab. D are basically aligned with the trending of FID and SSIM scores in quantitative evaluation, showing that repainting results of our LuminAIRe are most favored subjectively, with greater lead in realistic and harmonized lighting perception.
Robustness Test. Fig- shows the robustness our method to varying materials and geometry conditions, where different materials and geometry conditions are correctly handled with realistic lighting effects accordingly and consistently generated. test the robustness of our method to varying parsing masks (e.g., aasually-drawe parsing masks), we compare in Fig. 9 the repainting

---

results of from the input parsing masks before and after the disturbing, where the boarders are randomly extended and the inner structures are coarsened. To test the robustness of our method to varying lighting conditions, we conduct an experiment where the estimated lighting conditions are rotated clockwise while all other conditions are left unchanged. The results in Fig- TO]show that our method correctly handles most of the lighting rotations in the sense of the lighting effects on the foreground objects and the shadow effects in the background regions. The repainting results in the second column with no lighting conditions given ("No light") further validate the effectiveness of our illumination injection module. To test the robustness of our method to varying background conditions, also show the results of in-the- -hll examples in the supplemental material.
Parsing mask
Normal
Illumination
Repainting
Parsing mask
Normal
Illumination
Repainting
00
with light wind glass, light door glass, dark roof glass, and light green clearcoat carpaint
a liftback with light wind glass, light door glass, light root glass, night blue clearcoat carpaint
Figure 9: Qualitative results of normal maps, illumination images and repaintings using original (first row) and disturbed (second row) parsing masks as input conditions. Backgrounds are omitted here.
Conditions
light/Normal
0%/+180"
200
+00%
20%
+200
+30%
normal glass, normal glass, normal glass, and blue grey metallic carpaint
Figure 10: Qualitative results of repaintings and illumination images the estimated lighting rotates.
7 Conclusion
In this paper, we introduce the task of LuminAIRe for the realistic generation of lighting effects. The synthetic CAR-LUMINAIRE dataset is collected for the newly proposed task. Extensive experi- ments and the user study confirm that our method achieves perceptually more harmonized repainting results compared with the state-of-the-ar methods. The effectiveness and consistency of our illumination-awart design are shown in the robustness test.
and
Limitations and future works. In this paper, only the results of cars as foreground objects shown, resulting from the inadequate fcasibility of data collection. Besides, our model can not handle complex thin structures and some translucent glass materials very well, which are not well covered by our synthetic data for now, As single-image- maage-eeea method for generic outdoor scenes, our method currently ignores the non-local inter-reflections with other objects and focuses on the shadows cast directly on the ground. Therefore, datasets of richer object categories and finer details will be helpful to boost the training of learning- baased methods. Combining the lighting constraints with the newly emerged latent diffusion models (55) would also be an interesting direction for our future work.
Acknowledgement. This work is supported by the National Natural Science Foundation of China under Grant No. 62136001, 62088102.
10

---

References
[1] Amazon Mechanical Turk.
wwtp:wattuuk.... com
[2] Best 3D models of Cans, Objects and more Hum3D store. attpp:/
hhm.dhmmmddcmm com
[3] Blender. httpp://wwk/ wwwp://wwweeed.....mm org
[4] Squir Home. htttps://squii.
[5] G. Bae, I. Budvytis, and R. Cipolla. Estimating and exploiting the aleatoric uncertainty in surface normal estimation. In Proc. of International Conference on Computer Vision, 2021.
[6] Z. Bao, C. Long, G. Fu, D. Liu, Y. Li, J. Wu, and C. Xiao. Deep image- illumination harmonization. In Proc. of Computer Vision and Pattern Recognition, 2022.
[7] Barron and J. Malik. Intrinsic scene properties from a single RGB-D image. Proc. of Computer Vision and Pattern Recognition, 2013.
[8] A. Bhattad and Forsyth. Cut-and paste object insertion by enabling deep image prior reshading. In Proc. of International Conference on 3D Vision, 2022.
[9] J. F. Blinn. Models of light reflection for computer synthesized pictures. SIGGRAPH Comput. Graph., 112)1192-198, 1977.
[10] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv: /809, 1096, 2018
[11] Z. Chang, S. Weng, Y. Li, S. Li, and B. Shi. L-CoDer: Language-based colorization with color-object decoupling transformer. In Proc. of European Conference on Computer Vision, 2022.
[12] Z. Chang, S. Weng, P. Zhang, Y. Li, S. Li, and B. Shi. L-Colns: Language-basee colorization with instance awareness, In Proc. of Computer Vision and Pattern Recognition, 2023.
[13] X. Chen, C. Dong, J. Ji, j. Cao, and X. Li. Image manipulation detection by multi-view multi-scale supervision. In Proc. of International Conference Computer Vision, 2021.
[14] W. Cong, J. Zhang, L. Niu, L. Liu, Z. Ling, W. Li, and L. Zhang. DoveNet: Deep image harmonization via domain verification. In Proc. of Computer Vision and Pattern Recognition, 2020.
[15] Contributors, MMSegmennation:: Openmmlab semantic segmentation toolbox and benchmark. https //githbb. wwtp://.coo/ppeeeonmm// mmlaennnmmla///ooeeo www.m/aaammmmmeatttiiiiioo 2020.
[16] Y. Deng, Tang, W. Dong, C. Ma, Pan, L. Wang, and Xu. StyTr2: Image style transfer with transformers. Proc. of Computer Vision and Pattern Recognition, 2022.
[17] C. Dong, X. Chen, R. Hu, Cao, and X. Li. MVSS-Net: Multi-view multi-scale supervised networks for image manipulation detection. IEFE Transactions on Pottern Analysis and Machine Intelligence, 45(3):3539-33) 3353333993333,,,0 2023.
[18] D. Gao, G. Chen, Dong, P. Peers, K. Xu, and X. Tong. Deferred neural lighting: Free-viewpoint relighting from unstructured photographs. ACM Transactions on Graphics (Proc. of ACM SIGGRAPH Asia), 1-15, 2020.
[19] M.-A. Gardner, Y. Hold-Geoffroy, K. Sunkavalli, C. Gagné, and J.-F. Lalonde, Deep parametric indoor lighting estimation. In Proc. of International Conference on Computer Vision, 2019.
[20] M.-A. Gardner, K. Sunkavalli, F. Yumer, X. Shen, F. Gambaretto, C. Gagné, and J.-F. Lalonde. Learning to predict indoor illumination from a single image. In Proc. of ACM SIGGRAPH Asia, 2017.
[21] M. Garon, Sunkavalli, S. Hadap, N. Carr, and J.-F. Lalonde. Fast spatially-varying indoor lighting estimation. In Proc. of Computer Vision and Puttern Recognition, 2019.
[22] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In Proc. of Computer Vision and Pattern Recognition, 2016.
[23] J. J. A. Guerreiro, M. Nakazawa, and B. Stenger. PCT-Net: pixel-wise color transformations. In Proc. Computer Vision and Pattern Recognition, 2023.
Full resolution image harmonization using
[24] Z. Guo, Z. Gu, B. Zheng, J. Dong, and H. Zheng. Transformer for image harmonization and beyond. JEEE Transactions on Pattern Analysis and Machine Intelligence, (3(1)):12600119999 2023.
11

---

[25] Z. Guo, D. Guo, H. Zheng, Z. Gu. B. Zheng, and J. Dong. Image harmonization with transformer. In Proc. of International Conference on Computer Vision, 2021.
[26] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter. GANs trained by two time-scale update rule converge to a nash equilibrium. In Proc. of Advances Neural Information Processing Systems, 2017.
[27] Y. Hold-Geoffroy, A. Athawale, and J.-F. Lalonde. Deep sky modeling for single image outdoor lighting estimation. In Proc. of Computer Vision and Pattern Recognition, 2019.
[28] Y. Hold-Geoffroy, K. Sunkavalli, S. Hadap, E. Gambaretto, and J.-F. Lalonde. Deep outdoor illumination estimation. In Proc. of Computer Vision and Pattern Recognition, 2017.
[29] L. Hosek and A. Wilkie. An analytic model for full spectral sky-dome radiance. ACM Trans, Graphics (TOG), 31(4), 2012.
[30] L. Hosek and A. Wilkie. Adding solar-radiance function the Hošek- Wilkie skylight model. IEEE Computer Graphics and Applications. 33(3):44 52, 2013.
[31] X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proc. of International Conference on Computer Vision, 2017.
[32] Z. Huang. N. Zhao, and J. Liao. UniColor: A unitied framework for multi-moda colorization with transformer. In Proc. of ACM SIGGRAPH Asia, 2022.
[33] J. T. Kajiya. The rendering equation. In Proc. of ACM SIGGRAPH, 1986.
[34] W. Kim, B. Son, and I. Kim. ViLT: Vision-and-languaal transformer without convolution or region supervision. In Proc. of International Conference Machine Learning, 2021.
[35] G. Kwon and J. C. Ye. CLIPstyler: Image style transfer with single text condition. In Proc. of Computer Vision and Patiern Recognition, 2022.
[36] J.-F. Lalonde, L.PP. Asselin, J. Becirovski, Y. Hold-Geoffroy M. Garon, M.-A. Gardner, and J. Zhang. The laval HDR sky database.. 2016. http://sky http///skky.rrrb....oo
hdddb. com.
[37] J.-F. Lalonde, A. A. Efros, and S. G. Narasimhan. Estimating the natural illumination conditions from a single outdoor image. International Journal of Computer Vision, 2012.
[38] J.-F. Lalonde and I. Matthews, Lighting estimation in outdoor image collections. In Proc. of International Conference on 3D Vision, 2014.
[39] C. LeGendre, W.-C Ma, G. Fyffe, J. Flynn, L. Charbonnel, J. Busch, and P. Debevec. DecpLight: Learning illumination for unconstrained mobile mixed reality. In Proc. of Computer Vision and Pattern Recognition, 2019.
[40] Z. Li, M. Shafiei, R. Ramamoorthi, K. Sunkavalli, and M. Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and SVBRDF from single image. In Proc. of Computer Vision and Pattern Recognition, 2020.
[41] Z. Li, L. Yu, M. Okunev, M. Chandraker, and Z. Dong, Spatiotemporally consistent hdr indoor lighting estimation. ACM Trans. on Graphics (TOG), 42(3):1-15, 2023.
[42] J. Ling, H. Xue, L. Song, R. Xie, and X. Gu. Region- harmonization. In Proc. of Computer Vision and Pattern Recognition, 2021.
Region-aware aware adaptive instance normalization for image
[43] T. Miyato and M. Koyama. eGANs with projection discriminator. arXiv preprint arXiv:/802.055: 2018.
[44] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Megrew, I. Sutskever, and M. Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proc. of International Conference on Machine Learning, 2022.
[45] A. Odena, C. Olah, and Shlens. Conditional image synthesis with auxiliary classifier GANs. Proc. of International Conference on Machine Learning, 2017.
[46] R. Pandey, S. O. Escolano, C. Legendre, C. Haene, S. Bouaziz, C. Rhemann, P. Debevec, and S. Fanello. Total relighting: learning to relight portraits for background replacement. ACM Transactions on Graphics (Proc, of ACM SIGGRAPH), 10(4):1-21, 2021.
12

---

[47] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu. Semantic image synthesis with apatially-adaptive normal- ization. In Proc. of Computer Vision and Pattern Recognition, 2019.
[48] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. Proc. of Computer Vision and Pattern Recognition, 2016.
[49] D. Pavllo, A. Lucchi, and T. Hofmann. Controlling style and semantics in weakly-supervisee image gencration. In Proc. of European Conference on Computer Vision, 2020.
[50] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204006125, 2022.
[51] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In Proc. of International Conference on Machine Leurning, 2021.
[52] R. Ranftl, A. Bochkovskiy, and V. Koltun. Vision transformers for dense prediction. In Proc. of Interna- tional Conference on Computer Vision, 2021.
[53] R. Ranftl, K. Lasinger. D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(3):1623-163 2020,
[54] Y. Ren, X. Fan, G. Li, S. Liu, and T. H. Li. Neural texture extraction and distribution for controllable person image synthesis. In Proc. of Computer Vision and Pattern Recognition, 2022.
[55] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proc. of Computer Vision and Pattern Recognition, 2022.
[56] C. Saharia, W. Chan, S. Saxena, I.. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Proc. of Advances in Neural Information Processing Systems, 2022.
[57] S. Song and Funkhouser. Neural Illumiatioo: Lighting prediction for indoor environments. Proc. of Computer Vision and Pattern Recognition, 2019.
[58] J. Sun, S. Weng, Z. Chang, S. Li, and B. Shi. UniCoRN: A unified conditional image repainting network. In Proc. of Computer Vision and Pattern Recognition, 2022.
[59] K. Sunkavalli, M. K. Johnson, W. Matusik, and H. Pfister. Multi-scale image harmonization. ACM Transactions on Graphics (Proc. of ACM SIGGRAPH), 29(4):1-10, 2010.
[60] J. Tang, Y. Zhu, H. Wang, J. Chan, S. Li, and B. Shi. Estimating spatially-warying lighting in urban scenes with disentangled representation. In Proc. of European Conference on Computer Vision, 2022.
[61] S. Tripathi, S. Chandra, A. Agrawal, A. Tyagi. J. M. Rehg, and V. Chari. Learning generate synthetic data via compositing. In Proc. of Computer Vision and Pattern Recognition, 2019.
[62] Y.-H. Tsai, X. Shen, Z. Lin, K. Sunkavalli, X. Lu, and M.-H. Yang. Deep image harmonization. Proc. of Computer Vision and Pattern Recognition, 2017.
[63] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE Transactions on Image Processing, 13(4) 612, 2004.
[64] Z. Wang, W. Chen, D. Acuna, J. Kautz, and S. Fidler. Neural light field estimation for street scenes with differentiable virtual object insertion. In Proc. of European Conference Computer Vision, 2022.
[65] Z. Wang, X. Qi, K. Yuan, and M. Sun. Self-supervised correlation mining network for person image generation. In Proc. of Computer Vision and Pattern Recognition, 2022.
[66] S. Weng, W. Li, D. Li, H. Jin, and B. Shi. Conditional image repainting via semantic bridge and piecewise value function. In Proc. of European Conference on Computer Vision, 2020.
[67] S. Weng, W. Li. D. Li, H. Jin, and Shi. MISC: Multi-condition injection and spatially-adaptice compositing for conditional person image synthesis. In Proc. of Computer Vision and Pattern Recognition, 2020.
[68] S. Weng, J. Sun, Y. Li, S. Li, and B. Shi. CT²: Colorization transformer via color tokens. In Proc. of European Conference on Computer Vision, 2022.
13

---

[69] S. Weng, H. Wu, Z. Chang. J. Tang, S. Li, and B. Shi. L-code: Language-bise colorization using color-object decoupled conditions. In Proc. the AAAI Conference Artificial Intelligence, 2022.
[70] S. Weng, P. Zhang, Z. Chang, X. Wang, S. Li, and B. Shi. Affective image filter: Reflecting emotions from text to images. In Proc. of International Conference Computer Vision, 2023.
[71] J. Xiao, K. A. Ehinger, A. Oliva, and A. Torralba. Recognizing scene viewpoint using panoramic place representation. In Proc. Computer Vision and Pattern Recognition, 2012.
[72] W. Xiong, J. Yu, 7. Lin, J. Yang, X. Lu, C. Barnes, and Luo. Foreground-aware image inpainting. In Proc. of Computer Vision and Puttern Recognition, 2019.
[73] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Free-form image inpainting with gated convolution.
In Proc. of International Conference Computer Vision, 2019.
[74] P. Yu, J. Guo, F. Huang, C. Zhou, H. Che, X. Ling, and Y. Guo. Hierarchical disentangled representation learning for outdoor illumination estimation and editing. In Proc. International Conference on Computer Vision, 2021.
[75] H. Zhang, I. Gondfellow, D. Metaxas, and Odena. Self-attention generative adversarial networks. In Proc. of International Conference on Machine Learning, 2019.
[76] J. Zhang and J.-E. Lalonde. Learning high dynamic range from outdoor panoramas. In Proc. of International Conference on Computer Vision, 2017.
[77] Zhang, K. Sunkavalli, Y. Hold-Gcoffroyy Hadap, Eisenmann, and J.-F. Lalonde. All-weather deep outdoor lighting estimation. Proc. of Computer Vision and Puttern Recognition, 2019.
[78] R. Zhang, Isola, and A. A. Efros. Colorful image colorization. Proc. of European Conference on Computer Vision, 2016.
[79] Y. Zhang, N. Huang, Tang, H. Huang, C. Ma, W. Dong, and C. Xu. Inversion- based creativity transfer with diffuusion models. In Proc. of Computer Vision and Pattern Recognition, 2023.
[80] P. Zhu, R. Ahdal, Qin, and P. Wonka. SEAN: Image synthesis with semantic region-adantive normaliza- tion. Proc. of Computer Vision and Pattern Recognition, 2020.
[81] Y. Zhu, Y. Zhang. S. and B. Shi. Spatially varying outdoor lighting estimation from intrinsics. In Proc. of Computer Vision and Pattern Recognition, 2021.
14


Reviews:
[{"review_id": "tCTSyZ9X61", "reviewer": "Reviewer_U8wa", "summary": "The paper proposed a learning-based method for conditional image inpainting. The method \ntakes an image with a masked region and the conditioned attributes as input, and \nsynthesizes a new image by filling the masked region. Previous works for this task \nusually fail to generate images with realistic shading effects such as specularity\nand shadows. The paper tackles this problem by involving geometry and lighting cues  in the inpainting process. The paper estimates lighting information from the input \nimage and renders shading images of the inpainting content using the estimated lighting,\nnormal and a set of predefined roughness values. Conditioned on the shading images \nand the attributes, additional networks are trained to inpaint the foreground \nand background to generate photorealistic shading effects. The paper shows results \non inpainting cars onto outdoor images and demonstrates better results than previous \nmethods in terms of both FID score and user preference.  \n ", "questions": "See weakness.", "limitations": "The limitations look good to me.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. By explicitly predicting lighting, normal and materials, and rendering the shading \nimages, the proposed method can generate inpainting results that have realistic shading \neffects such as shadows and specularity, and outperforms basline methods.     \n\n2. The paper performs a thorough ablation study to validate the effectiveness of \ndifferent design chocies. \n", "weaknesses": "1. The paper only demonstrates results on a very limited scenario where cars \nare inpainted onto outdoor images. The designed lighting representation is also \ntailored for outdoor scenes. It's not clear whether the proposed method is scalable \nto other cases such as indoor images and more diverse kinds of inpainting content.\n\n2. In terms of shadows, the paper only shows cast shadows on a flat surface. How does \nit perform when the cast shadows are on other objects such as walls? In the meantime,\nthe method can only generate shadows cast by the inpainted content, and cannot \nproduce shadows on the inpainted content cast by the background objects. For example,\nin Figure 13 Row 7 of the supplemental material, there are no shadows on the inpainted car.\n\n3. In the results shown in the paper, all input masks follow a perfect car silhouette.\nI am wondering whether this is a requirement of the method. What if a rough mask that \nmay extend to regions outside of the car is given? What would the normal/shading prediction \nlook like? \n\n4. The paper should make it clear that it is following previous works on the lighting \nrepresentation and add clear references to them. For the illumination image, such a \ntechnique has also been used in previous works such as:\n* Deferred Neural Lighting: Free-viewpoint Relighting from Unstructured Photographs\n* Total Relighting: Learning to Relight Portraits for Background Replacement\n\nIn addition, the paper is also related to inverse rendering tasks in computer graphics,\nand it would be good to add discussions on them:\n* Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image\n \n5. What does the normal mean in the text prompt? \n\nOverall, I think the proposed method is technically sound and the results are convincingly \nbetter than the baseline methods. I refrain from giving a higher rating due to concerns \non the scalability of the method considering that it only shows results on car inpainting\nand requires a lot of manually annotated 3D data which may not be easily available for other\nsubjects such as animals and other general objects. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your very detailed review and suggestions, and here we give responses to the mentioned concerns.  We are looking forward to discussing with you during the author-reviewer discussion period.\n\n### Imperfect parsing masks as input\n\n>  What if a rough mask that may extend to regions outside of the car is given? What would the normal/shading prediction look like?\n\nWe conduct an experiment of using disturbed parsing masks (the silhouette extends outside the car, and the internal structure is coarsened with random noise) as input, with the results shown in **Fig. I** in the attached pdf.\n\nIn each case, the first row shows results for the accurate parsing mask, while the second row is for the disturbed mask. The other input conditions are kept the same. The results show that the normal and shading predictions for disturbed masks are still reasonable, as well as the final repainting results, which show the robustness of our proposed method to imperfect parsing masks.\n\n Our repainting formulation follows a pixel-wise correspondence between the semantics in the parsing masks and the repainting results. For example, in the right case, the door handles in the repainted image shrink following the disturbed mask. Therefore, better parsing masks generally lead to better repainting results. On the other hand, the repainting results may be negatively affected given severe mask degradation.\n\n### Concerns on shadow generation\n\n> The paper only shows cast shadows on a flat surface.\n\nAs stated in **Fig. 2** in the main paper and **Sec. 8.1** in the supplementary material of the original submission, the cars are inserted as foreground objects onto a flat “insertable” ground (*e.g.*, roads, grass, and dirt). The shadows of the inserted cars are rendered using a flat shadow catcher fitting the ground. More realistic shadows can be rendered by setting more sophisticated shadow catchers fitting all geometry in the background images,  which is infeasible for a large amount of data with great diversities. Therefore, our shadow net (NetS) only learns to cast shadows on a flat surface with imperfectly rendered shadow labels. Nevertheless, a flat shadow should be reasonably good for open outdoor scenes.\n\n> The method can only generate shadows cast by the inpainted content, and cannot produce shadows on the inpainted content cast by the background objects.\n\nUnlike the cast shadow on the ground, the shadow on the object cast by another object is a highly challenging non-local secondary lighting effect and is very difficult for networks to directly predict by purely operating in the image space. For general outdoor scenes without additional priors, a full 3D reconstruction of the whole scene is needed to recover this type of non-local shadows [V], which usually need multiple view inputs. We will clearly mention such limitations about shadows in the final version.\n\n[V] Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes. Wang *et al*. CVPR 2023.\n\n### Scalability of the proposed method\n\nPlease refer to **Sec. II** of the global response.\n\n### More citations needed\n\n> The paper should make it clear that it is following previous works on the lighting representation and add clear references to them.\n\nAs correctly pointed out, our idea of illumination candidate images is indeed similar to what has been proposed in the two mentioned papers. Thanks for suggesting these relevant works, and we will add clear references to those papers as suggested.\n\n> The paper is also related to inverse rendering tasks in computer graphics, and it would be good to add discussions on them.\n\nWe will add more citations and discussions of inverse rendering works as suggested. \n\nAs for the specific paper mentioned, Li *et al*.'s paper proposes a single-view method of lighting and BRDF estimation utilizing the learning-based priors on indoor scenes (which might capture scene properties such as Manhattan rooms, local smoothness and global sparsity of the materials, and spatially-consistency of local lighting) from a synthetic rendered dataset (which also require lots of computation, especially for the densely-labeled indoor spatially-varying lighting annotations). \n\nThey use 12 SG lobes to represent indoor local lighting near the object surface, which generally works fine for indoor scenes. However, for outdoor scenes with extreme sun intensity, optimizing the ground truth SG parameters may be numerically unstable. Nevertheless, finding a proper lighting representation that works fine in both indoor and outdoor scenes with high efficiency to use is still an open problem.\n\nThe commonly used framework of “decomposition-->rendering-->reconstruction loss back-propagation” is more challenging in general outdoor scenes at a single-view setting as the possible priors mentioned may not stand anymore, and it becomes less reliable for the monocular depth and normal estimation, which prohibits precise reconstruction of target image via rendering. In our image repainting setting, we have no accurate “target image” for minimizing reconstruction loss. Therefore, their method cannot directly apply to our task. In our proposed method, we only conduct a forward rendering process where the normal is estimated from the semantics of object categories (cars in our work specifically), and the need for depth is circumvented by the shadow network (NetS).\n\nDue to the incomplete 3D observation of the scene, Li *et al*.'s paper does not handle secondary lighting effects in forward rendering, which is also the limitation of our proposed method.\n\n### Clarifications\n\n> What does the normal mean in the text prompt?\n\nSorry for the ambiguity in wording. The “normal” in the text prompt is not related to the concept of “surface normal.” It is closer to the concept of “default values” (for instance, the middle darkness of the glass between light and dark).\n"}, {"review_id": "POzCTt17gt", "reviewer": "Reviewer_Ex3g", "summary": "\nThis paper tackles the task of illumination-aware conditional image repainting. Given an input image and a set of conditions, the proposed method aims to inpaint / re-generate a certain region based on the input conditions. This can be used to achieve functionalities such as object insertion and image composition. Compared to prior works, this paper is with the goal of injecting physics-based illumination information into the image generation process. \n\nIn a high-level, instead of formulating this task as a simple image-to-image translation in 2D image space, this work aims to introduce explicit physics-based rendering in 3D into a 2D neural renderer. This can be achieved by incorporating physics-based rendering buffers. To enable training and evaluation of the method, the authors also curate a photorealistic synthetic dataset with material and lighting conditions. \n\nThe results of the proposed method is qualitative visualized and quantitatively evaluated. A user study is included to compare the photorealism of the edited results. The proposed method can significantly outperform baselines wrt lighting effects. \n", "questions": "Please see weaknesses section above. ", "limitations": "The limitations and failure cases are discussed in paper and supp. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "In general I find this paper with a sufficient amount of workload and technically solid. \n\nOriginality: \n\n- The task definition is well motivated. The analysis on why we need 3D information in conditional generation is generally informative and convincing.\n- The proposed method is sensible and novel. Despite a complicated pipeline, it presents a smart approach to inject physics-based rendering process into a 2D neural renderer. \n\nQuality: \n\n- The qualitative and quantitative results outperform baselines and achieves SOTA. \n\n\nClarity: \n\n- This paper is well written and easy to follow. \n- The descriptions on method details in main paper and supp are thorough. \n\n\nSignificance: \n\n- This paper proposes a carefully designed approach for illumination-aware image generation, which has not been extensively explored in recent generative models.\n- The proposed dataset can be beneficial for future research works. \n\n", "weaknesses": "In general I do not find critical concerns of this paper but have some questions to further elicit insights: \n\n- The proposed lighting representation is a slightly modified version of prior works. How does the parametric light representation (in Eq.9) compare to prior sky models [22, 23, 32, 63]? \n- The model is trained on synthetic data, which can be a concern when the ultimate goal is to apply on real-world imagery. How well does it work on real-world images, and how to measure the domain gap? \n- What is the core advantage of generative repainting compared to fully physics-based lighting estimation methods such as SOLID-Net? \n\nThe motivation of conditional image repainting is still a relatively small scope. The authors could consider including discussion of these works in related works. For explicit lighting estimation, a line of work estimates 3D lighting volume: \n- Wang et al. Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion \n- Li et al. Spatiotemporally Consistent HDR Indoor Lighting Estimation \n\nIn a similar spirit to this paper, many works in relighting and neural rendering also combine neural modules with PBR. For example, \n- Philip et al. Multi-view Relighting using a Geometry-Aware Network \n- Pandey et al. Total Relighting: Learning to Relight Portraits for Background Replacement \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your very detailed review and suggestions, and here we give responses to the mentioned concerns.  We are looking forward to discussing with you during the author-reviewer discussion period.\n\n### Comparison of different lighting representations\n\nThe parameters in **Eq. 9** share many common variables with prior works since these variables are bound with the same physical meanings ($l_\\text{sun}$ for sun direction and $c_\\text{sun}$ for sun mean color). However, the underlying modeling is differently designed with reasons. \n\nAs a common point, the HW sky model [22, 23], LM sky model [32, 63], and our model all adopt the representations that separate the contribution of sun light and sky light in the environment lighting. The HW sky model is derived by modeling light from the sun going through the Earth's atmosphere, which is spectrum-related and somewhat complicated by taking atmospheric scattering into consideration. The LM sky model is derived by directly fitting the environment map of the sky dome (captured on the ground, that is to say, at the bottom of the atmosphere). The sun light part of the LM model is simplified as a double exponential fallout, while the sky light part is modeled using the analytic Preetham sky model [VI]. \n\nIn our task, we need to represent environment lighting from a complete panoramic view ($360\\degree\\times180\\degree$) and the sky might be occluded. However, the LM and HW models are used to model only the sky dome ($360\\degree\\times90\\degree$) and therefore are not suitable. Since the ambient light may also be from the grounding, where the color variations are much more significant than the sky, we resort to spherical harmonic (SH) for low-frequency ambient light. For representing sun light, we have tried and found the double exponential falloff a little overkill for our lighting data and task requirement. Therefore, we use a simpler spherical Gaussian (SG) modeling for the high-frequency sun light (which is further approximated as directional lighting). The choice of our lighting representation allows us to render the illumination of the object surface with the Blinn-Phong model on the fly efficiently without conducting numerical integral on the surface hemisphere (**Eq. 15** and **Eq. 16** of the supplemental materials) and shallower gradient chain for possible end-to-end training in future work.\n\n[VI]  A Practical Analytic Model for Daylight. Preetham *et al*. SIGGRAPH 1999.\n\n### Real-world evaluations\n\n> How well does it work on real-world images, and how to measure the domain gap?\n\nWe have shown a set of real-world examples in the supplementary material of the original submission, please refer to **Sec. I** of the global response.\n\nFor domain gap measurement, the current evaluation protocol cannot be directly used on real-world data due to the lack of ground truth labels, one possibility is to collect a small real-world test set at an affordable cost and then perform the same quantitative evaluation as on the synthetic dataset.\n\n### Comparison with SOLID-Net\n\nSOLID-Net is a physics-based lighting estimation method that combines the idea of intrinsic image decomposition and differentiable rendering. Their final output is spatially-varying lighting represented as panoramic environment maps. \n\nThe repainting results of our methods can be achieved by SOLID-Net using virtual object insertion (VOI). However, even with known lighting, for a realistic VOI, a 3D model along with a properly set shadow catcher is needed, as done in the data collection pipeline of our work. Whereas for realistic repainting, our method only takes a parsing mask along with user-given attributes as input, which is more convenient and has flexibility allowing editing using different attributes.\n\nTo sum up, our generative repainting pipeline relieves the demand for 3D models and expert skills in the application of virtual object insertion and gives the users more freedom to control generated content while keeping consistently realistic lighting effects.\n\n### More citations needed\n\nAs suggested, we will add more citations and discussions of works on 3D volumetric lighting estimation and neural PBR relighting.\n\nAs for the mentioned papers on  3D volumetric lighting estimation, both papers use the additional depth maps as input, which help the learning of 3D volumetric lighting representation. Wang *et al*.'s paper also utilizes a learned sky dome lighting for long-distance global lighting in outdoor scenes and uses differentiable rendering and adversarial learning techniques to facilitate better lighting estimation for VOI. Li *et al*.'s paper utilizes the spatiotemporal consistent constraint in the video clips by using an RNN to blend the lighting volume estimation results from individual video frames.\n\nAs for the mentioned papers on neural PBR relighting, Philip *et al*.'s paper uses a 3D proxy of the outdoor scene reconstructed by multi-view stereo to compute shadow masks and RGB shadow images at the source and target lighting conditions, then obtains refined shadow masks by the refining network, which are used to relighting new scene appearances in a neural rendering module along with illumination buffers from the 3D proxy as other inputs. Pandey *et al*.'s paper aims at relighting portraits given the target environment lighting by conducting intrinsic decomposition to get normal and albedo predictions and then using the given HDR environment map to render a diffuse light map and a set of specular light maps (which has a similar idea as our illumination candidate images, and we'll add a clear reference in the final version), at last, a shading network decides how to combine specular light maps and generates self-shadowing and specularities effects in the relit result."}, {"review_id": "1RtyoxAovn", "reviewer": "Reviewer_Xnyj", "summary": "This paper proposes a new method called LuminAIRe to address unrealistic lighting effects in image repainting analogous to cut-and-paste object insertion. This method estimates 3D geometry and environment lighting conditions from background images and parsing masks and uses physically-based illumination rendering and attention modules to inject physically-correct lighting information into the image generation process. The result is repainted images with harmonized lighting effects in both foreground and background regions. To facilitate and validate this task, a new dataset called CAR-LUMINAIRE with lighting annotations and appearance variants has been collected.", "questions": "The rebuttal must address the following points raised in weakness: \n\n- Clear comparisons with the methods listed in the missing references \n- Recommended ablations \n- Real-world examples \n- Foreground background separate evaluations\n- Correcting claimed contributions on \"illumination-aware image editing tasks\"\n- results demonstrating the change in appearance when environment maps are rotated for the same scene and obects \n", "limitations": "- Missing citations, references, and comparison\n- Missing comprehensive ablation \n- Missing real-world examples \n- overstating claimed contributions and undermining several related work contributions \n\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Unrealistic lighting effects in image repainting is an open task, especially getting the background shadows right. This paper attempts to have a solution to address it\n- The use of physically-based illumination rendering and attention modules is interesting\n- Results in repainted images with harmonized lighting effects in both foreground and background regions appear plausible to some extent\n- CAR-LUMINAIRE dataset with lighting annotations and appearance variants though not realistic as real images but potentially could be used for prototyping experiments", "weaknesses": "- The paper does not compare or cite several relevant previous works [1, 2, 3, 4, 5]\n- L48-49 \"As far as we know, the illumination-awareness in image editing tasks has not been emphasized yet\" is untrue. For instance see [2, 3, 4] \n- LuminAIRe pipeline has several components similar to and the current work has completely overlooked previous works and their contributions [see 3]\n- The current ablation study does not help figure out which components are adding to improved results. A leave-one-out ablation is crucial to highlight the differences. That is an ablation (a) without Hierarchical Labeling (b) without illumination attention and (c) without illumination injection is necessary. \n- There should also be an evaluation comparing foreground and background consistency separately to indicate the difference in the effectiveness for both these regions. A comparison with Image Harmonization methods for the foreground objects and perhaps also with [4] would help clearly understand how these methods compare with image harmonization and reshading methods\n- Many of the results look unrealistic and we do not how they compare to other works like [3 or 4] that show examples of the real world. \n- Missing real-world examples. Cars have complex material properties with paints and glitter in them. Simple parametric model representation of lighting cannot capture those complex lighting effects. Most of the evaluation is shown in synthetic datasets where the object appearance is mostly diffuse without strong specularities or reflections in them. There is no clear evidence in the current paper that the method would translate well to real scenes as well. \n- Results demonstrating the change in appearance (both foreground and background) when the environment's maps are rotated by fixing the camera and scene geometry would also help in understanding how good the results are with changes in lighting conditions. \n\n[1] People as Scene Probes. Wang et al. ECCV 2020 \\\n[2] Repopulating Street Scenes. Wang et al. CVPR 2021 \\\n[3] GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving. Chen et al. CVPR 2021 \\\n[4] Cut-and-Paste Object Insertion by Enabling Deep Image Prior for Reshading. Bhattad & Forsyth. 3DV 2022 \\\n[5] CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation. Wang et al. CoRL 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your very detailed review and suggestions, and here we give responses to the mentioned concerns.  We are looking forward to discussing with you during the author-reviewer discussion period. Due to the length limit, please refer to the global response for detailed reference items.\n### More citations needed\nWe will add citations and discussions of relevant works mentioned [VII-XI].\n\n[VII] and [VIII] leverage **timelapse image sequences** of street scenes for scene decomposition and extraction of pedestrians and cars (2D image), which are further combined with lighting-based object retrieval,  shadow network, and sun position estimation to composite street view images with illumination-harmonized objects inserted.\n\n[IX] and [XI] use **registered video and LiDAR sensor data** as input to create 3D mesh assets of cars and then insert the 3D car models into the scene with geometry constraints and shadow generation in the video clips for autonomous driving data enhancement.\n\n[X] takes the **background image, foreground image, and mask** of the object to be inserted as input. The reshaded image is lighting-harmonized by forcing the consistent image decomposition using deep image priors and reconstruction supervision.\n\nWe respectfully believe that it is unfair to compare our method with [VII], [VIII], [IX], and [XI] due to the **inconsistent inputs**. The only method that shares similar input conditions to us is [X]. \n\nWe find the starter code of the suggested method [X] is released. However, a time-consuming per-image optimization process (~10 min) is needed for each input image. Therefore we are unable to compare with this work given the limited time during the author response period. We will consider a comparison in the final version.\n### More ablation results\nAs suggested, we test two additional ablation variants: **Ours-A** and **Ours-AI**. The results are shown in **Tab. I** and **Fig. II** in the attached pdf. Please note that there is no “**Ours-I**” (and “**Ours-HI**”) variant since the illumination attention module cannot be enabled alone without illumination injection. The qualitative results show that **Ours-A** gives results of inconsistent lighting effects and wrongly loses/adds highlight effects in the first/second rows, while **Ours-AI** has no awareness about environment lighting and therefore wrongly gives a diffuse appearance/hallucinates lighting effects in the first/second rows. The trends of the quantitative scores also confirm the visual perceptions. \n\n So far, we have tested all possible ablation variants: **Ours**, **Ours-H**, **Ours-A**, **Ours-AI**, **Ours-HA**, and **Ours-HAI**.\n### Comparison with image harmonization methods\nAs stated above, we cannot compare with [X] during rebuttal time. Instead, we compare with two latest image harmonization methods: **DHT+** [I] and **PCT-Net** [II]. Since there are **no foreground objects to be harmonized in the background images** and we care about to what extent image harmonization methods can recover realistic lighting effects, we use the generated results of **Ours-AI** (without illumination injection in foreground generation) as inputs of  **DHT+** [I] and **PCT-Net** [II].\n\nThe results are shown in **Tab. I** and **Fig. II** in the attached pdf. There are reasonable leads in the M-score of **DHT+** [I] and **PCT-Net** [II], indicating better integrity of the harmonized results. The **PCT-Net** [II] also improves the perception of realism compared with **Ours-AI**, as indicated by the decrease in the FID score. However, the degraded the R-prcn and SSIM scores indicate the harmonized images may deviate from the user-specified conditions, as also shown in the qualitative results (**Fig. II**), where the harmonized images do not show better lighting effects and may have severe color shifting issues. We will add such comparisons in the final version.\n### Real-world examples\nPlease refer to **Sec. I** of the global response.\n### Separate evaluations for foreground and background regions\nWe conduct separate quantitative evaluations as suggested. When evaluating the foreground/background regions in the full images, we mask the background/foreground regions with zeros. The quantitative results are reported in **Tab. I** in the attached pdf (the R-prcn and M-score metrics are only meaningful for full images and thus not computed), which shows that our method also outperforms baseline methods wrt. foreground generation and the trends are consistent with the **Realistic** preference of the user study results shown in **Tab. 1** of the main paper. Besides, **Ours bg.** also performs better than **Original bg.** used by baseline methods, showing the effectiveness of our background repainting network (NetB).\n### Correcting the overclaim\nWe are sorry about some overclaim and agree that several prior works on image/video editing have considered illumination, so we will carefully revise our contribution claims in the final version.\n### Illustrations of rotating estimated lighting\nAs suggested, we conduct the experiment of rotating estimated lighting while keeping other conditions untouched. The results are shown in **Fig. III** in the attached pdf. “No light” denotes the repainted results with no illumination information (which is not **Ours-AI**, but **Ours** with illumination disabled at test time), and the degrees (*e.g.* $0\\degree/180\\degree$) mark the clockwise azimuth rotation angles (in the camera coordinates) of the estimated lighting in the first/second row. The corresponding illumination images are shown as insets. Without the illumination information, the generated foreground becomes flat and has no lighting effects, validating the effectiveness of the illumination injection. As the estimated lighting rotates, the illuminations/appearances of foreground regions correctly reflect the changes in the lighting effects while the repainted backgrounds show reasonable shadows and demonstrate visually realistic perceptions."}, {"review_id": "rBwm2I3US3", "reviewer": "Reviewer_oBS5", "summary": "This paper presents the ilLumination-Aware conditional Image Repainting (LuminAIRe) task to address the unrealistic lighting effects based on recent conditional image repainting (CIR) methods. The main contributions include : 1) introducing a new task of ilLumination-Aware conditional Image Repainting (LuminAIRe), 2) designing a full LuminAIRe pipeline to acquire more realistically repainted results, and 3) collecting a new dataset CAR-LUMINAIRE with rich material and lighting condition variants. This paper is generally written and has a clear layout. ", "questions": "see the Weakness", "limitations": "see the Weakness", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper main presents the ilLumination-Aware conditional Image Repainting (LuminAIRe) task to address the unrealistic lighting effects based on recent conditional image repainting (CIR) methods. To this end, the authors introduce a new task of ilLumination-Aware conditional Image Repainting (LuminAIRe), 2) designs a full LuminAIRe pipeline to acquire more realistically repainted results, and 3) collects a new dataset CAR-LUMINAIRE. This task is interesting, and the constructed dataset is useful for illumination community. ", "weaknesses": "The work collected a new dataset CAR-LUMINAIRE, which is interesting and useful for the illumination community. However, in the process of collecting the dataset, how to perform the \"warping\" to achieve aligned envmap? How to consider the scale of foreground objects? I think it is necessary to conduct a statistical analysis of the synthesized dataset.\n\nWhen estimating background illumination information, how to address the impact of the content occluded by foreground objects on the overall illumination information estimation? Especially for larger foreground objects?\n\nCompared to the image harmonization task, this paper considers the shadow generation of foreground objects. How do the authors evaluate the quality of generated shadows?\n\nThe variety of foreground objects in the dataset is extremely limited, which greatly limits the application of the method. More types of foreground objects may be more convincing and improve the robustness of the method.\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your very detailed review and suggestions, and here we give responses to the mentioned concerns.  We are looking forward to discussing with you during the author-reviewer discussion period.\n\n### More details on dataset creation\n\n> How to perform the \"warping\" to achieve aligned envmap?\n\nSince we adopt the spatially-uniform global lighting assumption for the outdoor scenes, we do not consider the depth when warping the environment map, and the warping is essentially a rotation that makes camera coordination of the virtual camera aligned with the coordination of the environment map (*i.e.*, making the virtual camera points toward the center of the aligned environment map and the up direction of the virtual camera is also the up direction of the aligned environment map). The rotation can be derived from the camera pitch and yaw (we use no camera roll to keep the ground in the cropped images level).\n\n> How to consider the scale of foreground objects?\n\nAs stated in **L. 158-159** of the main paper,  all car models are resized to fit their real-world dimensions. Therefore, the scales of foreground objects appearing in the image are decided by the camera FoV used,  the 2D insertion point, and the accuracy of the off-the-shelf normal and depth estimation results (influence the conversion from the 2D insertion point to 3D relative position calculation). \n\nBy converting the depth estimation results into real-world units, ideally (if all estimations are perfect), the rendered cars would be precisely the same size as a real car at the corresponding 3D point in the scene. However, due to estimation errors, the scale of the rendered objects may not always be reasonable, and therefore filtering of rendered data is conducted.\n\n> I think it is necessary to conduct a statistical analysis of the synthesized dataset.\n\nThanks for the helpful suggestion. We follow it to provide more statistical analysis results of our Car-LuminAIRe dataset:\n\n1. Portions of pixels of foreground region.\n\n   | 10%~15% | 15%~20% | 20%~25% | 25%~30% | 30%~35% | 35%~40% | 40%~45% | 45%~50% |\n   | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n   |  27.9%  |  21.2%  |  16.4%  |  12.0%  |  8.4%   |  5.6%   |  3.7%   |  4.8%   |\n\n2. Distributions of the car types.\n\n   | hatchback | sedan | race | liftback |  CUV  | pickup | micro | roadster |\n   | :-------: | :---: | :--: | :------: | :---: | :----: | :---: | :------: |\n   |   14.6%   | 22.0% | 6.0% |   8.8%   | 14.0% |  5.3%  | 3.6%  |   2.7%   |\n\n   | SUV  | MPV  | minivan | sports | coupe | universal | minibus | super | van  |\n   | :--: | :--: | :-----: | :----: | :---: | :-------: | :-----: | :---: | :--: |\n   | 6.7% | 1.8% |  0.9%   |  3.8%  | 3.5%  |   2.3%    |  0.2%   | 2.5%  | 1.3% |\n\n3. Distributions of the types of car paints.\n\n   | metallic | clearcoat | frosted | flake | diffuse |\n   | :------: | :-------: | :-----: | :---: | :-----: |\n   |  18.2%   |   26.8%   |  22.9%  | 19.8% |  12.3%  |\n\n4. Distributions of the used camera FoVs.\n\n   | $27\\degree\\sim31\\degree$ | $31\\degree\\sim36\\degree$ | $36\\degree\\sim41\\degree$ | $41\\degree\\sim46\\degree$ | $46\\degree\\sim51\\degree$ | $51\\degree\\sim56\\degree$ | $57\\degree\\sim61\\degree$ | $61\\degree\\sim66\\degree$ |\n   | :----------------------: | :----------------------: | :----------------------: | :----------------------: | :----------------------: | :----------------------: | :----------------------: | :----------------------: |\n   |           3.1%           |           4.3%           |           8.9%           |          21.2%           |          23.0%           |          22.1%           |          11.9%           |           5.5%           |\n\n### Concerns about lighting estimation\n\nAs shown in **Fig. 3** of the main paper, our lighting estimation network (NetL) only takes the background image as input. The foreground region of the background image is masked with zeros. Thus there are no foreground objects in the background images distracting the lighting estimation process. Of course, having fewer informative pixels imposes a challenge,  we address this issue through three designs:\n\n1. The NetL is trained with randomly chosen foreground masks applied on contact background images, forcing the network to have more robustness to occlusion (sorry for omitting the detail in the supplemental material).\n2. The lighting representation we adopt is relatively less complex, which lowers the difficulty of getting a reasonable lighting estimation from limited pixel observations.\n3. The synthetic dataset is filtered wrt. the portion of foreground pixels (**Sec. 8.1** of the supplemental material), avoiding extreme circumstances where the background is heavily occluded and the lighting can be barely identified.\n\n### Evaluations of generated shadows\n\n> How do the authors evaluate the quality of generated shadows?\n\nAs suggested by Reviewer Xnyj, we conduct separate evaluations of foreground and background regions, and the results are shown in **Tab. I** in the attached pdf, where **Ours bg.** performs better than **Original bg.** by a large margin, indicating that the generated shadow makes the repainted background region closer to the reference background. As the low FID score tells us, our background network (NetB) effectively adds a realistic perception to the overall repainting results. \n\n### Scalability of the proposed method\n\nPlease refer to **Sec. II** of the global response.\n\n### Clarifications\n\nIn our experiment setting, no ready foreground objects are awaiting to be harmonized with the background image. The foreground objects are generated **from scratch** following the user-specified conditions. Therefore, strictly speaking, our method is not directly comparable with image harmonization methods. Nevertheless, **Fig. II** in the attached pdf gives an impression of how image harmonization methods handle challenging lighting effects.\n"}, {"review_id": "TbRaBJfzoS", "reviewer": "Reviewer_Kvqo", "summary": "This paper proposes a method for illumination-aware conditional image repainting (LumiAIRe). \nDifferent from conventional conditional image repainting (CIE), LumiAIRe combines environmental lighting estimation, 3D normal estimation, and illumination injection for achieving harmonized lighting effects in both foreground and background regions. \nTo validate the effectiveness of the proposed illumination-aware repainting method, a new dataset named CAR-LumiAIRe with 52581 composited images using 198 detailed 3D car models and 1321 background images was proposed. \nThe overall performance of LumiAIRe is convincing especially on the generated shadows in the background images and the illumination of the car in the foreground images. \nThe effectiveness of the proposed method was well validated by the experiments and ablation studies. ", "questions": "- Why not also estimate the 3D shape and position of the car for direct relighting without using so many neural networks? \n- In Fig.7, the authors claim that \"the illumination injection helps foreground generation by comparing ours-HA and ours-HAI\". However, although missing some sort of lighting effects, in my opinion, the results of ours-HAI look better than ours-HA, please clarify this in the response. \n- How about other types of objects? Adding a type of object with more significant shape variation than cars will significantly improve the quality of the paper, but will clearly take much more effort, so this is only an option but not serious advice. ", "limitations": "Yes. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ Combining physical information into vision tasks is a promising and practical direction. The idea of incorporating lighting and 3D geometric information for illumination-aware conditional image repainting is interesting and sound. \n+ The repainted car images are with more plausible lighting effects than previous methods. \n+ The problem formulation is clear and also keeps consistent with the equations in Sec.5, which is easy to follow and understand. \n+ For dataset creation, the insertion points are carefully specified by segmenting the \"placeable flat ground\", which is more physically plausible than previous datasets. ", "weaknesses": "- The variation of the shape of the car in the results and experiments is too small. This means that the shape of the repainted cars is too similar to each other, which may limit the shape generalization capacity of the proposed method. \n- CAR-LumiAIRe was rendered at the resolution of 256x256, which is not enough to support high-resolution CIE tasks. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your very detailed review and suggestions, and here we give responses to the mentioned concerns.  We are looking forward to discussing with you during the author-reviewer discussion period.\n\n### Concerns about data diversity\n\nWe have tried our best to collect car models with sufficient diversities. However, most available car models are biased in shape (see dataset statistics in the response for **reviewer oBS5**). This issue could be solved by introducing other categories of data which is less biased in shape distribution.\n\n### Concerns about the resolution of the proposed dataset\n\nWe use the low-resolution version ($256\\times 256$) of CAR-LuminAIRe dataset in all of our experiments mainly to save computation resources and accommodate the memory limit of the GPU. We have also rendered a high-resolution version ($512\\times512$) of the CAR-LuminAIRe dataset, which would be released upon acceptance to facilitate high-resolution tasks better.\n\n### Discussion about potential full 3D solutions\n\n> Why not also estimate the 3D shape and position of the car for direct relighting without using so many neural networks?\n\nThe motivation for our work starts from the image-level repainting tasks, and the finally desired output is in image space, which is user-friendly for people without expert skills. Therefore, the full 3D pipeline is unnecessary in our task setting. If we adopt the full 3D generation and pose estimation for the car, then we still need explicit lighting estimation for relighting. Besides, to generate reasonable shadows from 3D foreground objects, a (partial) 3D reconstruction of background scenes may also be needed (otherwise, we are back to using neural networks to generate shadows, which might be more complicated since the inputs are in 3D), which would be rather challenging considering the single image setting of our task. Our pipeline breaks down the LuminAIRe task into multiple easier sub-problems using different networks (all have simple architectures). However, a full 3D solution may be more challenging for networks to handle (*e.g.*, how to map the user-specified conditions onto the 3D relightable models), with more expensive computations for 3D processing. \n\n### Discussion on visual qualities in the ablation study\n\n> However, although missing some sort of lighting effects, in my opinion, the results of ours-HAI look better than ours-HA.\n\nThe main reason hurting the visual quality of **Ours-HA** may be the part of the car body where the color is not correctly generated, which corresponds to the “unspecified car part” semantic, as the parsing mask shows. Without hierarchical labeling enhancement in training, the trained models sometimes may wrongly apply/not apply a “strange” color to this part. For the specific case (the top row of **Fig. 7** of the main paper), **Ours-HAI** happens to apply a color close to (darker) the car paint color and may look better than **Ours-HA** taken when ignoring lighting effects (a similar effect is also observed for **Ours-H**). What we want to emphasize by that case is, without illumination injection, the lighting effects are rather unrealistic, leading to a much darker blue color and window glass. Statistically speaking, **Ours-HA** should perform better than **Ours-HAI** in terms of faithfully following the user-specified conditions (R-prcn and SSIM) and realism (FID and M-score), as indicated by the scores in **Tab. 1** of the main paper. A similar trend can also be observed from the first row of **Fig. II** in the attached pdf, where **Ours-A** demonstrates better lighting effects than **Ours-AI**, showing the effectiveness of the illumination injection.\n\n### Scalability of the proposed method\n\nPlease refer to **Sec. II** of the global response.\n"}]

Author rebuttal:
We thank the reviewers for their valuable and constructive feedback on our paper, and we are looking forward to a more comprehensive discussion during the author-reviewer discussion period.

We are glad and encouraged to see that the reviewers’ comments that our paper is “well-written and easy to follow” (**Reviewer Ex3g**) and “performs a thorough ablation study” (**Reviewer U8wa**) with “excellent” presentation (**Reviewers U8wa, Ex3g, and Kvqo**) and “a sufficient amount of workload” (**Reviewer Ex3g**); the idea is “interesting” (**Reviewers Xnyj, oBS5, and Kvqo**) and “sound” (**Reviewer Kvqo**); the proposed dataset is “more physically plausible than previous datasets” (**Reviewer Kvqo**), “beneficial for future research work” (**Reviewer Ex3g**) and “useful for illumination community” (**Reviewer oBS5**); and the proposed method is “technically sound” (**Reviewers U8wa and Ex3g**) and “well validated by the experiments and ablation studies” (**Reviewer Kvqo**). 

We are also aware that the reviewers have raised many concerns about our paper. Here we post responses for overlapped concerns in global response and individual concerns in dedicated responses for each reviewer, respectively.

In the **attached pdf file**, we provide more experimental results as suggested by the reviewers:

1. Repainting results with rough masks (in **Fig. I**, as suggested by reviewer U8wa).
2. More ablation variants (in **Tab. I** and **Fig. II**, as suggested by reviewer Xnyj).
3. Separated evaluations for foreground and background regions (in **Tab. I**, as suggested by reviewer Xnyj).
4. Comparisons with image harmonization methods (in **Tab. I** and **Fig. II**, as suggested by reviewer Xnyj).
5. Illustrations of appearances as the lighting changes (in **Fig. III**, as suggested by reviewer Xnyj).

In the following text, we address the common issues the reviewers raised:

### I. Real-world examples

We have provided results of real-world examples in **Fig. 13** of the originally submitted supplemental material, where the background images are collected from other datasets [III, IV] and never seen in the whole training process. For other input conditions used in the in-the-wild test, the attributes are randomly generated, and the parsing masks are manually generated for each test image so that the repainted cars are in a suitable place with a proper size. The parsing masks are not hand-drawn but rendered from the 3D annotated car models. To further investigate how the masks influence our results, we provide the results of using rough masks in **Fig. I** of the attached pdf to show how casually-drawn masks would work for our method.

### II. Scalability of the proposed method

In our paper, we only demonstrate the results of car repainting on outdoor scenes so far. This is not limited by our proposed method but by the feasibility of data collection. 

The core of our proposed method is injecting extracted lighting information as the illumination images, which would not be affected by changing scenes or object types. On the other hand, our method firmly relies on the learned relationships between semantics and object properties (such as shape, materials, and colors). Therefore, it cannot directly repaint objects with unseen semantics (for example, the semantic of “hairs” for persons does not correspond to any semantic of cars). If enough data can be collected for other object categories, the proposed method should work well. Despite that, we cannot rush a dataset with a qualified amount of data and semantic labeling for other object categories, given the limited period during the author's response.

Since indoor and outdoor scenes have drastically different lightings, prior works often use different representations for indoor [15] and outdoor [21] scenes. Therefore, our proposed method trained on outdoor scenes basically cannot be directly used for inference on indoor scenes (there are only a few works on lighting estimation that work fine on both indoor scenes and outdoor scenes **simultaneously**, and all-scene lighting representation and estimation is still an open problem). For indoor scenes, our current lighting representation could work by only using the SH lighting part and fix $z_\text{vis}=0$. Other representations tailored for indoor scenes (such as SVSG) could be better and also remain compatible with our proposed method. 

### References

**In the pdf attachment:**

[I] Transformer for Image Harmonization and Beyond. Guo *et al*. TPAMI 2022.

[II] PCT-Net: Full Resolution Image Harmonization Using Pixel-Wise Color Transformations. Guerreiro *et al*. CVPR 2023.

**In the response texts:**

[III] Scalability in Perception for Autonomous Driving: Waymo Open Dataset. Sun *et al*. CVPR 2020.

[IV] UASOL, A Large-Scale High-Resolution Outdoor Stereo Dataset. Bauer *et al*. Scientific Data 2019.

[V] Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes. Wang *et al*. CVPR 2023.

[VI]  A Practical Analytic Model for Daylight. Preetham *et al*. SIGGRAPH 1999.

[VII] People as Scene Probes. Wang *et al*. ECCV 2020.

[VIII] Repopulating Street Scenes. Wang *et al*. CVPR 2021.

[IX] GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving. Chen *et al*. CVPR 2021.

[X] Cut-and-Paste Object Insertion by Enabling Deep Image Prior for Reshading. Bhattad & Forsyth. 3DV 2022.

[XI] CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation. Wang *et al*. CoRL 2022.


Meta-review: Reviewers and the AC read the rebuttal and took that into consideration for their final recommendation. Reviewers find this work original and potentially impactful. However, the manuscript needs a clearer discussion of the limitations to provide more accurate position of this work in the literature; the addition of missing citations; and an explicit discussion of similarities and differences to cut-and-paste reshading as well as image harmonization.
Decision: Accept (poster)