Title: LuminAIRe: Illumination-Aware Conditional Image Repainting for Lighting-Realistic Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Illumination

Image Generation

Conditional Image Repainting
Abstract: We present the ilLumination-Aware conditional Image Repainting (LuminAIRe) task to address the unrealistic lighting effects in recent conditional image repainting (CIR) methods. The environment lighting and 3D geometry conditions are explicitly estimated from given background images and parsing masks using a parametric lighting representation and learning-based priors. These 3D conditions are then converted into illumination images through the proposed physically-based illumination rendering and illumination attention module. With the injection of illumination images, physically-correct lighting information is fed into the lighting-realistic generation process and repainted images with harmonized lighting effects in both foreground and background regions can be acquired, whose superiority over the results of state-of-the-art methods is confirmed through extensive experiments. For facilitating and validating the LuminAIRe task, a new dataset Car-LuminAIRe with lighting annotations and rich appearance variants is collected.

LuminAIRe: Illumination-Aware Conditional Image Repainting for Lighting-Realistic Generation

Abstract

We present the il Lumin ationA ware conditional I mage Re painting (LuminAIRe) task to address the unrealistic lighting effects in recent conditional image repainting (CIR) methods. The environment lighting and 3D geometry conditions are explicitly estimated from given background images and parsing masks using a parametric lighting representation and learning-based priors. These 3D conditions are then converted into illumination images through the proposed physically-based illumination rendering and illumination attention module. With the injection of illumination images, physically-correct lighting information is fed into the lighting-realistic generation process and repainted images with harmonized lighting effects in both foreground and background regions can be acquired, whose superiority over the results of state-of-the-art methods is confirmed through extensive experiments. For facilitating and validating the LuminAIRe task, a new dataset CAR-LUMINAIRE with lighting annotations and rich appearance variants is collected.

1 Introduction

Advanced image editing is in high demand across a multitude of applications, e.g ., old photo colorization [78, 32, 68], damaged image restoration [48, 73, 72], and artistic style transfer [22, 35, 70]. Recently, conditional image repainting (CIR) [67, 66, 58] has emerged as an innovative research topic, proven effective in controllable image editing while 'freeing' users from the necessity of expert proficiency and retaining the 'freedom' to actualize their creative visions for image modification. By utilizing provided attributes or textual descriptions, fine-grained strokes, and Gaussian noise to separately represent colors, contours, and texture conditions, users could insert generative objects with desired appearances in specified image positions, as shown in the blue line of Fig. 1.

Although CIR methods have made great progress in synthesizing photo-realistic and visually-pleasing conditional images by avoiding gradient vanishing pitfall [67], adopting flexible condition representation [66], and designing condition fusion modules [58], there is still a crucial element missing from the CIR task: making the synthesized results harmonized with the illumination of the scene, e.g ., spatially-varying dark and bright regions in accordance to the lighting condition in the background, physically-accurate highlight effects for highly-specular surfaces (shining objects), and perceptuallyrealistic shadow avoiding 'floating objects' artifacts, as shown in the lower right example of Fig. 1.

Specifically, existing CIR methods handle image harmonization purely in 2D image space by estimating a pixel-wise color tone transformation of the repainted regions from the background regions. Current approaches use semantic parsing maps as 'geometry' representations and do not exploit the lighting information contained in given background images, which prevents them from having

CIR

LuminAIRe

Geometry

Lighting

Repainted bg.

Illumination

Conditional image-to-image generation

Attributes

a

sedan

with

dark

wind glass,

dark

door glass,

and

sky blue

clearcoat

car

paint

Illumination attention

Lighting-Realistic

generation

2D image space

3D space

CIR

LuminAIRe

Background

Parsing mask

（

awareness of physically-based lighting in 3D space. To introduce physically-correct 3D lighting instead of hallucinated lighting effects into CIR results, there remain some major challenges: (i) The lighting condition in 3D space should be extracted from the limited field of view (limited-FoV) 2D LDR images; (ii) the lighting condition should be physically-correctly transformed back into 2D image space; (iii) a dataset suitable for learning-based solutions to the proposed task is needed.

To achieve lighting-realistic generation within the CIR pipeline in an illumination-aware manner, we hereby propose the task of il Lumin ationA ware conditional I mage Re painting, denoted as LuminAIRe . We first lift geometry conditions from 2D parsing maps to 3D 1 normal maps using learning-based shape priors and estimate lighting conditions from limited-FoV LDR background images by designing a proper parametric representation. Then, we use physically-based reflection models to render illumination candidate images to capture possible lighting effects in 2D image space. With the help of illumination attention module , surface regions with different reflective properties are learned to adopt correct lighting effects in the resulting appearance. A dataset containing rich geometry and lighting annotations with abundant object variants is collected to facilitate the learning-based solution of the LuminAIRe task. As far as we know, we are the first to emphasize illumination-awareness in the image editing task of conditional image repainting.

Our contributions can be summarized as follows:

introducing a new task of il Lumin ationA ware conditional I mage Re painting ( LuminAIRe ) by exploiting the lighting information from background images;

designing a full LuminAIRe pipeline that represents, extracts, converts, and injects lighting information to acquire more realistically repainted results; and

collecting a new dataset CAR-LUMINAIRE with rich material and lighting condition variants for facilitating and validating the LuminAIRe task.

2 Related Work

Our method aims at introducing physical lighting constraints into generative image synthesis pipelines. In this section, we briefly review relevant works first and then discuss the relationships to our task.

Controllable image synthesis. Researchers have presented numerous works to synthesize images under the guidance of diverse user-provided conditions, e.g ., synthesizing specific object with category label [10, 43, 45, 75], transferring the texture from paintings to daily photos [22, 16, 35, 70], restoring

the colors of old photos [11, 12, 69, 68], and directly generating images from text descriptions [51, 50, 55, 56]. Recently, with the development of the condition injection mechanism [31, 47, 80, 34], researchers explore to control synthesized images with multiple cross-modality conditions, e.g ., condition guided image inpainting [44, 55], controllable person synthesis [54, 65], and inversionbased style transfer [79]. However, few works focus on synthesizing images strictly following lighting conditions. Following DIH-GAN [6] that considers introducing illumination estimation in harmonization task that adjusts the highlight of the inserted given object, we further explore the lighting condition in synthesizing illumination-consistent objects under the guidance of multiple cross-modality conditions.

Conditional image repainting and image harmonization. Conditional image repainting (CIR) aims at synthesizing reasonable visual content on an existing image, where the generated visual content should both meet the requirement of the user-provided conditions ( e.g ., color, geometry, and texture) and in harmonization with the existing background image. The first CIR task is proposed in MISC [67] for person image synthesis, where the foreground person image is synthesized first and then composited with the background. Weng et al . [66] design the semantic-bridge attention mechanism which allows more freely expressed color conditions by the users in text. UniCoRN [58] breaks the two-stage dependency and proposes a unified architecture that achieves more visually pleasing results. Despite recent achievements made by previous works in condition consistency, existing CIR models suffer from the issue of illumination inconsistency: although techniques such as color tone transform are applied, the lighting from the given background and on the generated visual contents often differ a lot, making lighting effects in the image rather unrealistic, such as incorrect shading, highlights, and shadows. In this paper, we address this issue by exploiting lighting and shape constraints in 3D space, which allows a more physically-correct rendering processing for generating lighting effects. Image harmonization methods [23, 24, 25, 14, 42, 62, 59], with a similar goal of CIR to realistically composite image foreground and background regions, have focused on illumination harmonization recently [6, 8]. However, this thread of works has poor control of visual content in foreground regions and may fail to preserve the color tone in background regions as they were.

Lighting representation and estimation. Achieving illumination-aware synthesis/repainting requires appropriate lighting representation and estimation from images. Lalonde et al . [37] is the first to use shadows, shading, and sky appearance variations observed in the image to infer outdoor lighting. A physics-based Hošek-Wilkie (HW) sky model [29, 30] is proposed to recover HDR parameters for deep outdoor lighting estimation [28]. A more weather-robust Lalonde-Matthews (LM) model [38, 77] is then proposed to cover more comprehensive lighting conditions in the outdoor environment. More recently, a learning-based lighting representation [27] is used on a large sky panorama dataset [36] with an autoencoder network. The encoder-decoder framework is further proposed [39] to estimates lighting as a spherical HDR lighting map. HDSky [74] and SOLD-Net [60] disentangle several physically meaningful attributes into separate learned latent spaces by hierarchical autoencoders and make the estimation editable. Parametric models such as spherical harmonic (SH) coefficients [7, 21] and spherical Gaussian (SG) [19, 40] are also widely used, especially in indoor scenes. Gardner et al . [20], NeurIllum [57], and SOLID-Net [81] design sophisticated networks to hallucinate the missing parts in the panoramic view and predict lighting as environment maps. 3D volumetric lighting representations are also widely used in recent works, which facilitate the lighting-realistic scene editing for indoor [41] and outdoor [64] scenes, however heavily require computation resources. Considering the demand for lighting-realistic generation, we propose a parametric lighting representation for outdoor scenes that is both easy to predict and simple to use.

3 Problem Formulation

For self-containedness, we briefly review the CIR formulation before introducing ours.

3.1 Preliminaries about CIR

The previous CIR tasks [66, 67, 58] aim at generating the repainted image y r by repainting certain regions in an image x ∈ R 3 × H × W according to user-specified conditions in different modalities: x g , x p , x c , and x b for the 'geometry', 'texture', 'color', and background conditions respectively.

In their works, the 'geometry' condition x g ∈ L N g × H × W is a binary semantic parsing mask, where N g is the number of possible parts of the visual content to be repainted and L = { 0 , 1 } ; the 'texture'

condition x p ∼ N (0 , 1) is a Gaussian noise; the 'color' condition can be represented as attributes x c ∈ L N c × N v or text descriptions x c = { x c t } N L t =1 , where N c , N v, and N L represent the numbers of attributes and available choices, and the length of the user-inputted sentences, respectively; the background condition x b ∈ R 3 × H × W is the image of background region with respect to the repainted region as foreground region, i.e ., x b = (1 -m ) ⊙ x , where the binary mask m indicating foreground region can be directly acquired from the parsing mask x g , as shown in lower left of Fig. 1.

The repainted image y r can be further decomposed as a blending of repainted foreground image y f and repainted background image y b :

Previous works assume unchanged background region, i.e ., y b = x b , leaving the key question of CIR tasks as generating realistic foreground region y f constrained by given conditions:

where previous works ignore clues in 3D space and implement the generation pipeline F G as a conditional image-to-image generation purely in 2D image space. To make the repainted image harmonized as a whole, previous works [58, 67] design additional harmonization modules to adjust the color tone of intermediate repainting result based on clues in x b .

3.2 Formulation of LuminAIRe

However, the image-based harmonization modules have limited representation ability for complex lighting effects ( e.g ., varying shading and shiny surfaces) due to a lack of 3D representation. Besides, directly using x b as y b in Eq. (1) may neglect possible light transport effects ( e.g ., shadows) introduced by the repainted region as its corresponding behaviors in the 3D real world might be.

As illustrated by the rendering equation [33], a physically-correct and -realistic appearance of an object is derived from its geometry , reflective property , and omnidirectional environment lighting in 3D space. Therefore, to make the repainted image y r more lighting-realistic , the repainted foreground y f should also be conditioned by the lighting condition L and geometry condition G in 3D space:

Given L and G in 3D space, a proper 2D representation x i containing both the information from L and G should be derived for compatibility with current image generation architectures:

and then the lighting-realistic generation for foreground y f can be rewritten as:

Similarly, the repainted background y b should also be conditioned on x i to recover lighting effects:

The limited-FoV background image x b itself is a partial observation of environment lighting and thus can provide clues about L . Therefore, the lighting condition L can be inferred in the form of:

Similarly, by finding the shape priors of certain types of objects, the 3D geometry condition G can be lifted from its '2D flattened version', i.e ., parsing mask x g :

Moreover, in our LuminAIRe formulation, we extend the attributes x c beyond colors, which allows the users to describe the reflective property and have control over the lighting effects of repainting results. A sample of attributes is shown as the bold text in the lower left of Fig. 1.

As aforementioned, both the repainted foreground y f and background y b are given by the lightingrealistic generation in our LuminAIRe formulation, which leads to more realistic and harmonized results than traditional CIR pipelines [58], as shown in Fig. 1.

3D labeling

Data sample

HDR envmap

Lighting-annotated background cropping

High-res. LDR panorama

Aligned envmap

warping

Enhanced materials

Realistic object insertion

𝑂

!

𝑂

"

𝑋

#

𝑌

#

3D positioning

𝑌

#

𝑋

#

-𝑍

#

𝑂

"

𝑂

!

𝑍

!

𝑋

!

𝑑

𝑌

!

2D insertion point

7

4 Data Preparation

To tackle the data shortage issue, we create the first dataset suitable for the LuminAIRe task, named CAR-LUMINAIRE, with its data preparation process and data sample shown in Fig. 2.

3D car models with hierarchical semantic labeling. Collecting large-scale real data for learningbased LuminAIRe methods is infeasible since the geometry and lighting capture in 3D space requires specialized equipment and extensive human labor. Therefore, here we resort to computer graphics techniques to create photo-realistic synthetic data. The cars are chosen as the foreground objects for the obviousness of lighting effects and the availability of high-quality synthetic models. We collect 198 detailed 3D car models in 17 different categories from online model stores [2, 4] and then label the parts of the models in 3D space, which allows us to get the accurate parsing mask in 2D image space from any viewpoint. Following the common structure of vehicles, we divide the car models into 35 semantic part labels. The part labels are organized in a hierarchical way ( e.g ., the door window is a sub-part of the door ) to accommodate car models in different granularity. Besides 3D labeling, we manually adjust the scales of each model to fit the real-world dimensions.

Background images with lighting annotations. Then we prepare background images with known lighting annotations. Here we use the SUN360-HDR dataset [27, 76], which contains HDR panoramic environment maps (envmaps) corresponding to the LDR panoramas of outdoor scenes in the SUN360 dataset [71]. Limited field-of-view (limited-FoV) background images are cropped from the LDR panoramas with virtual cameras of randomized FoVs and camera poses. For each cropped background image, the corresponding HDR envmap in the SUN360-HDR dataset [27, 76] is warped to align with the viewing direction of the virtual camera. Background images unsuitable for realistic object insertion are manually filtered out, leaving 1,321 images of diverse scenes and lighting conditions.

Enhanced data rendering with realistic placement. For each background image, we randomly select insertion points within the central region of the 'placeable flat ground' marked by an offthe-shelf segmentation toolbox [15]. Then, for each 2D insertion point in the image, we calculate the relative transformation from the camera coordination O c to the local coordination of the object O o from the depth d and the normal Z o estimated by depth [52, 53] and normal [5] estimation methods. With the aligned envmaps and the ray-tracing based Blender [3] Cycles rendering engine, physically-correct lighting effects can be rendered into the composited images. In the rendering process, besides the original materials of the models, several physics-based rendering (PBR) car paint materials are randomly applied for more appearance variants, especially in lighting effects; besides, the inserted models are randomly rotated around Z o axis for more geometry variants. The rendered images are filtered to ensure reasonable pixel portions of both foreground and background regions. At last, 52,581 composited images at the resolution of 256 × 256 are collected, accompanied by parsing mask and normal map annotations, as shown in the data sample of Figure 2.

5 Method

To realize the LuminAIRe formulation, we first estimate 3D lighting and geometry from background images and parsing masks (Sec. 5.1). Then the lighting information is injected into the lightingrealistic generation process as illumination images (Sec. 5.2). By further introducing hierarchical labeling enhancement (Sec. 5.3), our method can generate reasonable results even with coarse-level parsing masks. Our pipeline is shown in Fig. 3, with detailed network architectures and loss functions for network modules in supplementary materials.

NetF

a

hatchback

with

normal

wind glass,

normal

door

glass, and

light

grey frosted

car paint

Pixel-wise

broadcast

Attribute encoder

Repainted fg.

𝑦𝑦

f

Repainted bg.

𝑦𝑦

b

Repainting

𝑦𝑦

r

NetS

Attributes

𝑥𝑥

c

Parsing mask

𝑥𝑥

𝑔𝑔

Background

𝑥𝑥

b

Illumination

rendering

Hierarchical

enhancement

Lighting

parameters

𝐿𝐿

Illumination

attention

Illumination

𝐼𝐼

Normal map

𝐺𝐺

Illumination injection

Input conditions

Mask

𝑚𝑚

Feature embedding map

Bg.

𝑥𝑥

b

Element-wise addition

Feat. emb.

𝐸𝐸

NetL

Feat. emb.

𝐸𝐸

NetB

Illumination

candidates

{

𝐼𝐼

c

}

Limited-FoV image

GT

Ours

Limited-FoV image

GT

Ours

Limited-FoV image

GT

Ours

5.1 Estimating 3D Information with Learning-based Priors

Our CAR-LUMINAIRE dataset consists of outdoor scene images, where the lighting can be approximately decomposed into the high-frequency sunlight and the low-frequency ambient light [60]. Accordingly, we model the lighting condition L as the addition of a directional light and a 2-nd order spherical harmonics (SH) lighting, which can be specifically described as lighting parameters:

where z vis ∈ { 0 , 1 } is the sun visibility, z int is the intensity of sunlight, z ang describes the 'size' of the sun (in solid angle formally), c sun ∈ R 3 is the normalized sun color in RGB channels, l sun ∈ R 2 indicates the sun position, and σ SH ∈ R 3 × 9 is the 2-nd order SH coefficients for RGB channels.

As shown in Fig. 4, the parametric representation 3 in Eq. (9) can well fit real-world lighting in sunny, cloudy, and low light conditions. On the other hand, the proposed parametric lighting representation is convenient for network prediction. Here we design a NetL to serve as F L in Eq. (7), where l sun is estimated by a classification task and other parameters are estimated by regression tasks. To apply our method to other types of background scenes, specifically tailored lighting representations can be directly adopted, without modification to our underlying formulation of LuminAIRe.

For 3D geometry, we use the normal map G ∈ R 3 × H × W as the representation where each pixel indicates the surface normal direction n at that surface point in 3D space. For certain types of objects, there exist strong shape priors (such as sedans and hatchbacks), which can be learned in a supervised way. Similarly, a NetS of encoder-decoder structure is further proposed to serve as F Geo in Eq. (8).

5.2 Injecting Lighting Information using Illumination Images

To bridge the 3D lighting and geometry with 2D images, the rendering equation [33] is a handy tool to serve as R i in Eq. (4), which physically models the image formation process as the light reflection:

where L i ( ω i ) is the environment lighting from direction ω i , L o ( ω r ) is the reflected lighting toward direction ω r , Ω n is the visible hemisphere determined by surface normal n , and f r ( ω i , ω r ) describes the reflective properties of all possible combination of incoming and outgoing directions.

Diffuse

𝜌𝜌

= 4

𝜌𝜌

= 16

𝜌𝜌

= 32

𝜌𝜌

= 64

𝜌𝜌

= 128

Appearance variants

Illumination candidate images

For a certain image pixel with the known camera viewing direction v , ideally, the pixel intensity can be calculated as L o ( -v ) , and accurate lighting effects can be calculated as illumination images .

However, with L and G estimated from input conditions, f r ( ω i , ω r ) still remains unknown. Therefore, in a similar spirit to Gao et al . [18] and Pandey et al . [46], instead of directly calculating the actual illumination image, we use a set of uncolored 'standard materials' as f r in Eq. (10) and render corresponding illumination candidate images { I c } . For the physics-based rendering of { I c } , we use the Lambertian refectance model f diff ( ω i , ω r ) = 1 / π and normalized Blinn-Phong model [9] f spec ( ω i , ω r ) = ( ρ +4)( n · h ) ρ / 8 π with M different values of roughness ρ , where h = ω i + ω r / || ω i + ω r || is the half vector. At last, we have { I c } = { I diff } ∪ { I ρ i spec } i M =1 .

As shown in Fig. 5, most lighting effects in different appearance variants can be covered by the linear combinations of the pre-computed { I c } . However, it's worth noting that the correspondence of the appearance image and { I c } may vary pixel-wisely ( e.g ., the tires, hood, and windshield have different reflective properties thus different lighting effects). Accordingly, we design an illumination attention module A I to estimate the combination coefficient maps C I = A I ( E ) for each image pixel, where E is the feature embedding map containing information of both part labels and part-associated attributes in a pixel-aligned way. After the illumination image I derived as I = ∑ M +1 i =1 C i I ⊙ I i c , which covers lighting effects of parts with different materials , we use I as x i in Eq. (5) and conduct lighting-realistic generations of foreground and background regions using our proposed NetF and NetB respectively. For NetF , we adopt the network backbone of F G in UniCoRN [58], and the illumination image I is injected in a similar way as other conditions in 2D image space at different resolutions. The NetB is also an encoder-decoder architecture, serving as F B in Eq. (6). We adopt the same loss functions for NetF as used in UniCoRN [58].

5.3 Generating Realistic Results from Coarse Parsing Masks

As mentioned in Sec. 4, the parsing masks in our CAR-LUMINAIRE dataset can be very coarse, which also reflects the possible application scenarios when the user only specifies interested parts. Previous CIR formulations may fail to generate realistic results in regions without fine-grained labels since their generation follows a strictly pixel-wise semantic mapping between labels and images. We hereby introduce a hierarchical labeling enhancement : randomly coarsening the input parsing mask at training time ( e.g ., door glass label becomes door label) and encouraging the fine-grained parts ( door glass ) to be generated. Besides, the part-associated attributes of lower-level parts ( door glass ) should be also associated with their upper-level parts ( door ) to avoid loss of condition in attributes x c , which can be done by modifying the association matrix [58] A ∈ L N c × N g accordingly.

6 Experiments

In this section, we conduct comparisons with state-of-the-art methods and validate our design with an ablation study and a robustness test. Please see supplementary materials for implementation details.

6.1 Comparison with State-of-the-art Methods

Baseline methods. We conduct quantitative and qualitative comparisons with three state-of-the-art CIR methods ( UniCoRN [58], Weng et al . [66], and MISC [67]) and a most-relevant conditional image generation method ( Pavllo et al . [49]). Among them, modifications are made for Pavllo et al . [49] and Weng et al . [66] to accept conditions represented as attributes.

Quantitative metrics. Following previous work [58], we adopt Fréchet inception distance (FID) [26] for assessment of perception quality, R-precision [66] for assessment of alignment between generated

images y r and given attributes x c , and M-score [61] for assessment of authenticity. We use the latest manipulation detection model [17, 13] for calculating the M-score [61]. We also report the structural similarity index (SSIM) [63] for comparing the major image structure with the reference image.

Reference

LuminAIRe

UniCoRN

Pavllo

Weng

MISC

a

hatchback

with

normal

wind glass,

normal

door glass,

normal

sky blue diffuse

carpaint

a

sedan

with

normal

wind glass,

dark

door glass, and

sky blue metallic

carpaint

a

hatchback

with

normal

wind glass,

normal

door glass,

normal

silver metallic

carpaint

Conds.

The scores in Tab. 1 and the second and the third columns of Fig. 6 show that results of MISC [67] and Weng et al . [66] are far from lighting-realistic with 'crayon-drawing-like' appearances, since the color tone transform is not applied [66], or conducted in a two-phase manner [67]. As shown in the fourth column of Fig. 6, Pavllo et al . [49] tend to generate foreground regions in flat shadings with fewer texture patterns, which makes its results generally look reasonable when only focusing on foreground regions or in low light or cloudy scenes (as indicated by the FID and user study results), but computer vision models can easily find the disharmony due the sharp boundaries between foreground and background regions [58], as also indicated by the worst M-score. UniCoRN [58] fails to generate correct lighting effects from its unified color tone transform (the first row), therefore tends to hallucinate highlights at the top of cars regardless of lighting in background regions (the second row). The hallucinated lighting effects along with the undesired texture pattern on car bodies drastically damage the perceptual preferences, as confirmed by the FID score and user study results in Tab. 1. LuminAIRe generates realistic lighting effects close to the reference images in both sunny (the first and the third rows) and cloudy (the second row) scenes of specified materials and even when a coarse-level parsing mask is given (the third row), with a large margin in all quantitative metrics compared with baseline methods. LuminAIRe also learns to avoid the undesired texture pattern with the hints of the smoothly varied shading in the illumination images (Fig. 7).

Ours-HAI

Ours-HA

Ours-H

Ours

Illumination

Geometry

a

liftback

with

normal

wind glass,

normal

door glass,

normal

blue grey metallic

carpaint

a

liftback

with

normal

wind glass,

normal

door glass,

dark

carpaint

Conds.

Material editing

a

CUV

with

normal

wind glass,

dark

door glass,

dark

light grey diffuse / brown clearcoat

car paint

Lighting consistency

a

CUV

with

light

wind glass,

light

door glass,

light

grey metallic

car paint

6.2 Evaluations

Ablation study. We conduct an ablation study with three variants of Ours : (i) Ours-H , (ii) Ours-HA , and (iii) Ours-HAI , where '-H', '-A', and '-I' mean disabling the hierarchical labeling enhancement, the illumination attention, and the illumination injection for the foreground, respectively.

The hierarchical labeling enhancement is confirmed helpful in generating realistic results with coarselevel parsing masks, as shown in Fig. 7 and the third row of Fig. 6, where Ours generates more consistent and better repaintings at regions with no specified part labels (marked in blue purple), which is also demonstrated by the FID and R-prcn score in Tab. 1. The second row of Fig. 7 shows an example where the lack of illumination attention module wrongly renders a diffuse appearance, with further evidence from the drop of FID from Ours-H to Ours-HA in Tab. 1. It's quite obvious from Tab. 1 and Fig. 7 that the illumination injection helps foreground generation by comparaing Ours-HA and Ours-HAI . From UniCoRN to Ours-HAI , the improvements in FID score and M-score validate the contribution of the lighting-realistically generated background.

Besides, Ours-HA gets an unexpectedly good SSIM score. It's possibly because a slight misalignment of lighting effects (especially highlights) due to errors in lighting or geometry estimation would lead to a considerable drop in the SSIM score (which honestly measures the pixel-wise difference) but with very little harm to the lighting-realistic perception (as indicated by the FID and M-score).

User study. We also conduct a user study with 20 volunteers on the Amazon Mechanical Turk [1] platform, where 200 sets of results randomly drawn from the test set are shown and volunteers are asked to choose one in each set with (i) the most realistic foreground and (ii) the most harmonized lighting. The results of the user study in Tab. 1 are basically aligned with the trending of FID and SSIM scores in quantitative evaluation, showing that repainting results of our LuminAIRe are most favored subjectively, with a greater lead in realistic and harmonized lighting perception.

Robustness Test. Fig. 8 shows the robustness of our method to varying materials and geometry conditions, where different materials and geometry conditions are correctly handled with realistic lighting effects accordingly and consistently generated. To test the robustness of our method to varying parsing masks ( e.g ., casually-drawn parsing masks), we compare in Fig. 9 the repainting

Parsing mask

Illumination

Repainting

Normal

Illumination

Parsing mask

Repainting

Normal results of from the input parsing masks before and after the disturbing, where the boarders are randomly extended and the inner structures are coarsened. To test the robustness of our method to varying lighting conditions, we conduct an experiment where the estimated lighting conditions are rotated clockwise while all other conditions are left unchanged. The results in Fig. 10 show that our method correctly handles most of the lighting rotations in the sense of the lighting effects on the foreground objects and the shadow effects in the background regions. The repainting results in the second column with no lighting conditions given ('No light') further validate the effectiveness of our illumination injection module. To test the robustness of our method to varying background conditions, we also show the results of in-the-wild examples in the supplemental material. a liftback with light wind glass, light door glass, dark roof glass, and light green clearcoat carpaint a liftback with light wind glass, light door glass, light roof glass, and night blue clearcoat carpaint

Illumination

Parsing mask

Repainting

a

liftback

with

light

wind glass,

light

door glass,

dark

light green clearcoat

carpaint

Normal

Illumination

Parsing mask

Repainting

a

liftback

with

light

wind glass,

light

door glass,

light

night blue clearcoat

carpaint

Normal

PCT-Net [II]

DHT+ [I]

Ours-AI

Ours-A

Ours

Reference

a

liftback

with

normal

wind glass,

normal

door glass,

normal

blue grey metallic

carpaint

Conds.

a

liftback

with

normal

wind glass,

normal

door glass,

normal

blue grey metallic

carpaint

0º/+180º

Conditions

+30º/+210º

+60º/+240º

+90º/+270º

+120º/+300º

+150º/+330º

a

liftback

with

normal

wind glass,

normal

door glass,

normal

blue grey metallic

carpaint

No light/Normal

a

liftback with

normal wind glass,

normal door glass,

dark roof glass, and

Conditions

No light/Normal

7 Conclusion

In this paper, we introduce the task of LuminAIRe for the realistic generation of lighting effects. The synthetic CAR-LUMINAIRE dataset is collected for the newly proposed task. Extensive experiments and the user study confirm that our method achieves perceptually more lighting-realistic and harmonized repainting results compared with the state-of-the-art methods. The effectiveness and consistency of our illumination-aware design are shown in the robustness test.

a liftback with normal wind glass, normal door glass, normal roof glass, and blue grey metallic carpaint Limitations and future works. In this paper, only the results of cars as foreground objects are shown, resulting from the inadequate feasibility of data collection. Besides, our model can not handle complex thin structures and some translucent glass materials very well, which are not well covered by our synthetic data for now. As a single-image-based method for generic outdoor scenes, our method currently ignores the non-local inter-reflections with other objects and focuses on the shadows cast directly on the ground. Therefore, datasets of richer object categories and finer details will be helpful to boost the training of learning-based methods. Combining the lighting constraints with the newly emerged latent diffusion models [55] would also be an interesting direction for our future work.

Acknowledgement. This work is supported by the National Natural Science Foundation of China under Grant No. 62136001, 62088102.

0º/+180º

+30º/+210º

+60º/+240º

+90º/+270º

+120º/+300º

+150º/+330º

References

Amazon Mechanical Turk. https://www.mturk.com .

Best 3D models of Cars, Objects and more - Hum3D store. https://hum3d.com .

Blender. https://www.blender.org .

Squir Home. https://squir.com .

G. Bae, I. Budvytis, and R. Cipolla. Estimating and exploiting the aleatoric uncertainty in surface normal estimation. In Proc. of International Conference on Computer Vision , 2021.

Z. Bao, C. Long, G. Fu, D. Liu, Y. Li, J. Wu, and C. Xiao. Deep image-based illumination harmonization. In Proc. of Computer Vision and Pattern Recognition , 2022.

J. T. Barron and J. Malik. Intrinsic scene properties from a single RGB-D image. In Proc. of Computer Vision and Pattern Recognition , 2013.

A. Bhattad and D. Forsyth. Cut-and-paste object insertion by enabling deep image prior for reshading. In Proc. of International Conference on 3D Vision , 2022.

J. F. Blinn. Models of light reflection for computer synthesized pictures. SIGGRAPH Comput. Graph. , 11(2):192-198, 1977.

A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018.

Z. Chang, S. Weng, Y. Li, S. Li, and B. Shi. L-CoDer: Language-based colorization with color-object decoupling transformer. In Proc. of European Conference on Computer Vision , 2022.

Z. Chang, S. Weng, P. Zhang, Y. Li, S. Li, and B. Shi. L-CoIns: Language-based colorization with instance awareness. In Proc. of Computer Vision and Pattern Recognition , 2023.

X. Chen, C. Dong, J. Ji, j. Cao, and X. Li. Image manipulation detection by multi-view multi-scale supervision. In Proc. of International Conference on Computer Vision , 2021.

W. Cong, J. Zhang, L. Niu, L. Liu, Z. Ling, W. Li, and L. Zhang. DoveNet: Deep image harmonization via domain verification. In Proc. of Computer Vision and Pattern Recognition , 2020.

M. Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https: //github.com/open-mmlab/mmsegmentation , 2020.

Y. Deng, F. Tang, W. Dong, C. Ma, X. Pan, L. Wang, and C. Xu. StyTr2: Image style transfer with transformers. In Proc. of Computer Vision and Pattern Recognition , 2022.

C. Dong, X. Chen, R. Hu, J. Cao, and X. Li. MVSS-Net: Multi-view multi-scale supervised networks for image manipulation detection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3539-3553, 2023.

D. Gao, G. Chen, Y. Dong, P. Peers, K. Xu, and X. Tong. Deferred neural lighting: Free-viewpoint relighting from unstructured photographs. ACM Transactions on Graphics (Proc. of ACM SIGGRAPH Asia) , 39(6):1-15, 2020.

M.-A. Gardner, Y. Hold-Geoffroy, K. Sunkavalli, C. Gagné, and J.-F. Lalonde. Deep parametric indoor lighting estimation. In Proc. of International Conference on Computer Vision , 2019.

M.-A. Gardner, K. Sunkavalli, E. Yumer, X. Shen, E. Gambaretto, C. Gagné, and J.-F. Lalonde. Learning to predict indoor illumination from a single image. In Proc. of ACM SIGGRAPH Asia , 2017.

M. Garon, K. Sunkavalli, S. Hadap, N. Carr, and J.-F. Lalonde. Fast spatially-varying indoor lighting estimation. In Proc. of Computer Vision and Pattern Recognition , 2019.

L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In Proc. of Computer Vision and Pattern Recognition , 2016.

J. J. A. Guerreiro, M. Nakazawa, and B. Stenger. PCT-Net: Full resolution image harmonization using pixel-wise color transformations. In Proc. of Computer Vision and Pattern Recognition , 2023.

Z. Guo, Z. Gu, B. Zheng, J. Dong, and H. Zheng. Transformer for image harmonization and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(11):12960-12977, 2023.

Z. Guo, D. Guo, H. Zheng, Z. Gu, B. Zheng, and J. Dong. Image harmonization with transformer. In Proc. of International Conference on Computer Vision , 2021.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter. GANs trained by a two time-scale update rule converge to a nash equilibrium. In Proc. of Advances in Neural Information Processing Systems , 2017.

Y. Hold-Geoffroy, A. Athawale, and J.-F. Lalonde. Deep sky modeling for single image outdoor lighting estimation. In Proc. of Computer Vision and Pattern Recognition , 2019.

Y. Hold-Geoffroy, K. Sunkavalli, S. Hadap, E. Gambaretto, and J.-F. Lalonde. Deep outdoor illumination estimation. In Proc. of Computer Vision and Pattern Recognition , 2017.

L. Hosek and A. Wilkie. An analytic model for full spectral sky-dome radiance. ACM Trans. on Graphics (TOG) , 31(4), 2012.

L. Hosek and A. Wilkie. Adding a solar-radiance function to the Hošek-Wilkie skylight model. IEEE Computer Graphics and Applications , 33(3):44-52, 2013.

X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proc. of International Conference on Computer Vision , 2017.

Z. Huang, N. Zhao, and J. Liao. UniColor: A unified framework for multi-modal colorization with transformer. In Proc. of ACM SIGGRAPH Asia , 2022.

J. T. Kajiya. The rendering equation. In Proc. of ACM SIGGRAPH , 1986.

W. Kim, B. Son, and I. Kim. ViLT: Vision-and-language transformer without convolution or region supervision. In Proc. of International Conference on Machine Learning , 2021.

G. Kwon and J. C. Ye. CLIPstyler: Image style transfer with a single text condition. In Proc. of Computer Vision and Pattern Recognition , 2022.

J.-F. Lalonde, L.-P. Asselin, J. Becirovski, Y . Hold-Geoffroy, M. Garon, M.-A. Gardner, and J. Zhang. The laval HDR sky database., 2016. http://sky.hdrdb.com .

J.-F. Lalonde, A. A. Efros, and S. G. Narasimhan. Estimating the natural illumination conditions from a single outdoor image. International Journal of Computer Vision , 2012.

J.-F. Lalonde and I. Matthews. Lighting estimation in outdoor image collections. In Proc. of International Conference on 3D Vision , 2014.

C. LeGendre, W.-C. Ma, G. Fyffe, J. Flynn, L. Charbonnel, J. Busch, and P. Debevec. DeepLight: Learning illumination for unconstrained mobile mixed reality. In Proc. of Computer Vision and Pattern Recognition , 2019.

Z. Li, M. Shafiei, R. Ramamoorthi, K. Sunkavalli, and M. Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and SVBRDF from a single image. In Proc. of Computer Vision and Pattern Recognition , 2020.

Z. Li, L. Yu, M. Okunev, M. Chandraker, and Z. Dong. Spatiotemporally consistent hdr indoor lighting estimation. ACM Trans. on Graphics (TOG) , 42(3):1-15, 2023.

J. Ling, H. Xue, L. Song, R. Xie, and X. Gu. Region-aware adaptive instance normalization for image harmonization. In Proc. of Computer Vision and Pattern Recognition , 2021.

T. Miyato and M. Koyama. cGANs with projection discriminator. arXiv preprint arXiv:1802.05637 , 2018.

A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Mcgrew, I. Sutskever, and M. Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proc. of International Conference on Machine Learning , 2022.

A. Odena, C. Olah, and J. Shlens. Conditional image synthesis with auxiliary classifier GANs. In Proc. of International Conference on Machine Learning , 2017.

R. Pandey, S. O. Escolano, C. Legendre, C. Haene, S. Bouaziz, C. Rhemann, P. Debevec, and S. Fanello. Total relighting: learning to relight portraits for background replacement. ACM Transactions on Graphics (Proc. of ACM SIGGRAPH) , 40(4):1-21, 2021.

T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proc. of Computer Vision and Pattern Recognition , 2019.

D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proc. of Computer Vision and Pattern Recognition , 2016.

D. Pavllo, A. Lucchi, and T. Hofmann. Controlling style and semantics in weakly-supervised image generation. In Proc. of European Conference on Computer Vision , 2020.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 , 2022.

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In Proc. of International Conference on Machine Learning , 2021.

R. Ranftl, A. Bochkovskiy, and V. Koltun. Vision transformers for dense prediction. In Proc. of International Conference on Computer Vision , 2021.

R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V . Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(3):1623-1637, 2020.

Y. Ren, X. Fan, G. Li, S. Liu, and T. H. Li. Neural texture extraction and distribution for controllable person image synthesis. In Proc. of Computer Vision and Pattern Recognition , 2022.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proc. of Computer Vision and Pattern Recognition , 2022.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Proc. of Advances in Neural Information Processing Systems , 2022.

S. Song and T. Funkhouser. Neural Illumination: Lighting prediction for indoor environments. In Proc. of Computer Vision and Pattern Recognition , 2019.

J. Sun, S. Weng, Z. Chang, S. Li, and B. Shi. UniCoRN: A unified conditional image repainting network. In Proc. of Computer Vision and Pattern Recognition , 2022.

K. Sunkavalli, M. K. Johnson, W. Matusik, and H. Pfister. Multi-scale image harmonization. ACM Transactions on Graphics (Proc. of ACM SIGGRAPH) , 29(4):1-10, 2010.

J. Tang, Y. Zhu, H. Wang, J.-H. Chan, S. Li, and B. Shi. Estimating spatially-varying lighting in urban scenes with disentangled representation. In Proc. of European Conference on Computer Vision , 2022.

S. Tripathi, S. Chandra, A. Agrawal, A. Tyagi, J. M. Rehg, and V. Chari. Learning to generate synthetic data via compositing. In Proc. of Computer Vision and Pattern Recognition , 2019.

Y.-H. Tsai, X. Shen, Z. Lin, K. Sunkavalli, X. Lu, and M.-H. Yang. Deep image harmonization. In Proc. of Computer Vision and Pattern Recognition , 2017.

Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4):600-612, 2004.

Z. Wang, W. Chen, D. Acuna, J. Kautz, and S. Fidler. Neural light field estimation for street scenes with differentiable virtual object insertion. In Proc. of European Conference on Computer Vision , 2022.

Z. Wang, X. Qi, K. Yuan, and M. Sun. Self-supervised correlation mining network for person image generation. In Proc. of Computer Vision and Pattern Recognition , 2022.

S. Weng, W. Li, D. Li, H. Jin, and B. Shi. Conditional image repainting via semantic bridge and piecewise value function. In Proc. of European Conference on Computer Vision , 2020.

S. Weng, W. Li, D. Li, H. Jin, and B. Shi. MISC: Multi-condition injection and spatially-adaptive compositing for conditional person image synthesis. In Proc. of Computer Vision and Pattern Recognition , 2020.

S. Weng, J. Sun, Y. Li, S. Li, and B. Shi. CT 2 : Colorization transformer via color tokens. In Proc. of European Conference on Computer Vision , 2022.

S. Weng, H. Wu, Z. Chang, J. Tang, S. Li, and B. Shi. L-code: Language-based colorization using color-object decoupled conditions. In Proc. of the AAAI Conference on Artificial Intelligence , 2022.

S. Weng, P. Zhang, Z. Chang, X. Wang, S. Li, and B. Shi. Affective image filter: Reflecting emotions from text to images. In Proc. of International Conference on Computer Vision , 2023.

J. Xiao, K. A. Ehinger, A. Oliva, and A. Torralba. Recognizing scene viewpoint using panoramic place representation. In Proc. of Computer Vision and Pattern Recognition , 2012.

W. Xiong, J. Yu, Z. Lin, J. Yang, X. Lu, C. Barnes, and J. Luo. Foreground-aware image inpainting. In Proc. of Computer Vision and Pattern Recognition , 2019.

J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Free-form image inpainting with gated convolution. In Proc. of International Conference on Computer Vision , 2019.

P. Yu, J. Guo, F. Huang, C. Zhou, H. Che, X. Ling, and Y. Guo. Hierarchical disentangled representation learning for outdoor illumination estimation and editing. In Proc. of International Conference on Computer Vision , 2021.

H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial networks. In Proc. of International Conference on Machine Learning , 2019.

J. Zhang and J.-F. Lalonde. Learning high dynamic range from outdoor panoramas. In Proc. of International Conference on Computer Vision , 2017.

J. Zhang, K. Sunkavalli, Y. Hold-Geoffroy, S. Hadap, J. Eisenmann, and J.-F. Lalonde. All-weather deep outdoor lighting estimation. In Proc. of Computer Vision and Pattern Recognition , 2019.

R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In Proc. of European Conference on Computer Vision , 2016.

Y. Zhang, N. Huang, F. Tang, H. Huang, C. Ma, W. Dong, and C. Xu. Inversion-based creativity transfer with diffusion models. In Proc. of Computer Vision and Pattern Recognition , 2023.

P. Zhu, R. Abdal, Y. Qin, and P. Wonka. SEAN: Image synthesis with semantic region-adaptive normalization. In Proc. of Computer Vision and Pattern Recognition , 2020.

Y. Zhu, Y. Zhang, S. Li, and B. Shi. Spatially-varying outdoor lighting estimation from intrinsics. In Proc. of Computer Vision and Pattern Recognition , 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for illumination-aware conditional image repainting (LumiAIRe). \nDifferent from conventional conditional image repainting (CIE), LumiAIRe combines environmental lighting estimation, 3D normal estimation, and illumination injection for achieving harmonized lighting effects in both foreground and background regions. \nTo validate the effectiveness of the proposed illumination-aware repainting method, a new dataset named CAR-LumiAIRe with 52581 composited images using 198 detailed 3D car models and 1321 background images was proposed. \nThe overall performance of LumiAIRe is convincing especially on the generated shadows in the background images and the illumination of the car in the foreground images. \nThe effectiveness of the proposed method was well validated by the experiments and ablation studies.\n\nSTRENGTHS:\n+ Combining physical information into vision tasks is a promising and practical direction. The idea of incorporating lighting and 3D geometric information for illumination-aware conditional image repainting is interesting and sound. \n+ The repainted car images are with more plausible lighting effects than previous methods. \n+ The problem formulation is clear and also keeps consistent with the equations in Sec.5, which is easy to follow and understand. \n+ For dataset creation, the insertion points are carefully specified by segmenting the \"placeable flat ground\", which is more physically plausible than previous datasets.\n\nWEAKNESSES:\n- The variation of the shape of the car in the results and experiments is too small. This means that the shape of the repainted cars is too similar to each other, which may limit the shape generalization capacity of the proposed method. \n- CAR-LumiAIRe was rendered at the resolution of 256x256, which is not enough to support high-resolution CIE tasks.\n\nQUESTIONS:\n- Why not also estimate the 3D shape and position of the car for direct relighting without using so many neural networks? \n- In Fig.7, the authors claim that \"the illumination injection helps foreground generation by comparing ours-HA and ours-HAI\". However, although missing some sort of lighting effects, in my opinion, the results of ours-HAI look better than ours-HA, please clarify this in the response. \n- How about other types of objects? Adding a type of object with more significant shape variation than cars will significantly improve the quality of the paper, but will clearly take much more effort, so this is only an option but not serious advice.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the ilLumination-Aware conditional Image Repainting (LuminAIRe) task to address the unrealistic lighting effects based on recent conditional image repainting (CIR) methods. The main contributions include : 1) introducing a new task of ilLumination-Aware conditional Image Repainting (LuminAIRe), 2) designing a full LuminAIRe pipeline to acquire more realistically repainted results, and 3) collecting a new dataset CAR-LUMINAIRE with rich material and lighting condition variants. This paper is generally written and has a clear layout.\n\nSTRENGTHS:\nThis paper main presents the ilLumination-Aware conditional Image Repainting (LuminAIRe) task to address the unrealistic lighting effects based on recent conditional image repainting (CIR) methods. To this end, the authors introduce a new task of ilLumination-Aware conditional Image Repainting (LuminAIRe), 2) designs a full LuminAIRe pipeline to acquire more realistically repainted results, and 3) collects a new dataset CAR-LUMINAIRE. This task is interesting, and the constructed dataset is useful for illumination community.\n\nWEAKNESSES:\nThe work collected a new dataset CAR-LUMINAIRE, which is interesting and useful for the illumination community. However, in the process of collecting the dataset, how to perform the \"warping\" to achieve aligned envmap? How to consider the scale of foreground objects? I think it is necessary to conduct a statistical analysis of the synthesized dataset.\n\nWhen estimating background illumination information, how to address the impact of the content occluded by foreground objects on the overall illumination information estimation? Especially for larger foreground objects?\n\nCompared to the image harmonization task, this paper considers the shadow generation of foreground objects. How do the authors evaluate the quality of generated shadows?\n\nThe variety of foreground objects in the dataset is extremely limited, which greatly limits the application of the method. More types of foreground objects may be more convincing and improve the robustness of the method.\n\nQUESTIONS:\nsee the Weakness\n\nLIMITATIONS:\nsee the Weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method called LuminAIRe to address unrealistic lighting effects in image repainting analogous to cut-and-paste object insertion. This method estimates 3D geometry and environment lighting conditions from background images and parsing masks and uses physically-based illumination rendering and attention modules to inject physically-correct lighting information into the image generation process. The result is repainted images with harmonized lighting effects in both foreground and background regions. To facilitate and validate this task, a new dataset called CAR-LUMINAIRE with lighting annotations and appearance variants has been collected.\n\nSTRENGTHS:\n- Unrealistic lighting effects in image repainting is an open task, especially getting the background shadows right. This paper attempts to have a solution to address it\n- The use of physically-based illumination rendering and attention modules is interesting\n- Results in repainted images with harmonized lighting effects in both foreground and background regions appear plausible to some extent\n- CAR-LUMINAIRE dataset with lighting annotations and appearance variants though not realistic as real images but potentially could be used for prototyping experiments\n\nWEAKNESSES:\n- The paper does not compare or cite several relevant previous works [1, 2, 3, 4, 5]\n- L48-49 \"As far as we know, the illumination-awareness in image editing tasks has not been emphasized yet\" is untrue. For instance see [2, 3, 4] \n- LuminAIRe pipeline has several components similar to and the current work has completely overlooked previous works and their contributions [see 3]\n- The current ablation study does not help figure out which components are adding to improved results. A leave-one-out ablation is crucial to highlight the differences. That is an ablation (a) without Hierarchical Labeling (b) without illumination attention and (c) without illumination injection is necessary. \n- There should also be an evaluation comparing foreground and background consistency separately to indicate the difference in the effectiveness for both these regions. A comparison with Image Harmonization methods for the foreground objects and perhaps also with [4] would help clearly understand how these methods compare with image harmonization and reshading methods\n- Many of the results look unrealistic and we do not how they compare to other works like [3 or 4] that show examples of the real world. \n- Missing real-world examples. Cars have complex material properties with paints and glitter in them. Simple parametric model representation of lighting cannot capture those complex lighting effects. Most of the evaluation is shown in synthetic datasets where the object appearance is mostly diffuse without strong specularities or reflections in them. There is no clear evidence in the current paper that the method would translate well to real scenes as well. \n- Results demonstrating the change in appearance (both foreground and background) when the environment's maps are rotated by fixing the camera and scene geometry would also help in understanding how good the results are with changes in lighting conditions. \n\n[1] People as Scene Probes. Wang et al. ECCV 2020 \\\n[2] Repopulating Street Scenes. Wang et al. CVPR 2021 \\\n[3] GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving. Chen et al. CVPR 2021 \\\n[4] Cut-and-Paste Object Insertion by Enabling Deep Image Prior for Reshading. Bhattad & Forsyth. 3DV 2022 \\\n[5] CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation. Wang et al. CoRL 2022\n\nQUESTIONS:\nThe rebuttal must address the following points raised in weakness: \n\n- Clear comparisons with the methods listed in the missing references \n- Recommended ablations \n- Real-world examples \n- Foreground background separate evaluations\n- Correcting claimed contributions on \"illumination-aware image editing tasks\"\n- results demonstrating the change in appearance when environment maps are rotated for the same scene and obects\n\nLIMITATIONS:\n- Missing citations, references, and comparison\n- Missing comprehensive ablation \n- Missing real-world examples \n- overstating claimed contributions and undermining several related work contributions", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the task of illumination-aware conditional image repainting. Given an input image and a set of conditions, the proposed method aims to inpaint / re-generate a certain region based on the input conditions. This can be used to achieve functionalities such as object insertion and image composition. Compared to prior works, this paper is with the goal of injecting physics-based illumination information into the image generation process. \n\nIn a high-level, instead of formulating this task as a simple image-to-image translation in 2D image space, this work aims to introduce explicit physics-based rendering in 3D into a 2D neural renderer. This can be achieved by incorporating physics-based rendering buffers. To enable training and evaluation of the method, the authors also curate a photorealistic synthetic dataset with material and lighting conditions. \n\nThe results of the proposed method is qualitative visualized and quantitatively evaluated. A user study is included to compare the photorealism of the edited results. The proposed method can significantly outperform baselines wrt lighting effects.\n\nSTRENGTHS:\nIn general I find this paper with a sufficient amount of workload and technically solid. \n\nOriginality: \n\n- The task definition is well motivated. The analysis on why we need 3D information in conditional generation is generally informative and convincing.\n- The proposed method is sensible and novel. Despite a complicated pipeline, it presents a smart approach to inject physics-based rendering process into a 2D neural renderer. \n\nQuality: \n\n- The qualitative and quantitative results outperform baselines and achieves SOTA. \n\n\nClarity: \n\n- This paper is well written and easy to follow. \n- The descriptions on method details in main paper and supp are thorough. \n\n\nSignificance: \n\n- This paper proposes a carefully designed approach for illumination-aware image generation, which has not been extensively explored in recent generative models.\n- The proposed dataset can be beneficial for future research works.\n\nWEAKNESSES:\nIn general I do not find critical concerns of this paper but have some questions to further elicit insights: \n\n- The proposed lighting representation is a slightly modified version of prior works. How does the parametric light representation (in Eq.9) compare to prior sky models [22, 23, 32, 63]? \n- The model is trained on synthetic data, which can be a concern when the ultimate goal is to apply on real-world imagery. How well does it work on real-world images, and how to measure the domain gap? \n- What is the core advantage of generative repainting compared to fully physics-based lighting estimation methods such as SOLID-Net? \n\nThe motivation of conditional image repainting is still a relatively small scope. The authors could consider including discussion of these works in related works. For explicit lighting estimation, a line of work estimates 3D lighting volume: \n- Wang et al. Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion \n- Li et al. Spatiotemporally Consistent HDR Indoor Lighting Estimation \n\nIn a similar spirit to this paper, many works in relighting and neural rendering also combine neural modules with PBR. For example, \n- Philip et al. Multi-view Relighting using a Geometry-Aware Network \n- Pandey et al. Total Relighting: Learning to Relight Portraits for Background Replacement\n\nQUESTIONS:\nPlease see weaknesses section above.\n\nLIMITATIONS:\nThe limitations and failure cases are discussed in paper and supp.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a learning-based method for conditional image inpainting. The method \ntakes an image with a masked region and the conditioned attributes as input, and \nsynthesizes a new image by filling the masked region. Previous works for this task \nusually fail to generate images with realistic shading effects such as specularity\nand shadows. The paper tackles this problem by involving geometry and lighting cues  in the inpainting process. The paper estimates lighting information from the input \nimage and renders shading images of the inpainting content using the estimated lighting,\nnormal and a set of predefined roughness values. Conditioned on the shading images \nand the attributes, additional networks are trained to inpaint the foreground \nand background to generate photorealistic shading effects. The paper shows results \non inpainting cars onto outdoor images and demonstrates better results than previous \nmethods in terms of both FID score and user preference.\n\nSTRENGTHS:\n1. By explicitly predicting lighting, normal and materials, and rendering the shading \nimages, the proposed method can generate inpainting results that have realistic shading \neffects such as shadows and specularity, and outperforms basline methods.     \n\n2. The paper performs a thorough ablation study to validate the effectiveness of \ndifferent design chocies.\n\nWEAKNESSES:\n1. The paper only demonstrates results on a very limited scenario where cars \nare inpainted onto outdoor images. The designed lighting representation is also \ntailored for outdoor scenes. It's not clear whether the proposed method is scalable \nto other cases such as indoor images and more diverse kinds of inpainting content.\n\n2. In terms of shadows, the paper only shows cast shadows on a flat surface. How does \nit perform when the cast shadows are on other objects such as walls? In the meantime,\nthe method can only generate shadows cast by the inpainted content, and cannot \nproduce shadows on the inpainted content cast by the background objects. For example,\nin Figure 13 Row 7 of the supplemental material, there are no shadows on the inpainted car.\n\n3. In the results shown in the paper, all input masks follow a perfect car silhouette.\nI am wondering whether this is a requirement of the method. What if a rough mask that \nmay extend to regions outside of the car is given? What would the normal/shading prediction \nlook like? \n\n4. The paper should make it clear that it is following previous works on the lighting \nrepresentation and add clear references to them. For the illumination image, such a \ntechnique has also been used in previous works such as:\n* Deferred Neural Lighting: Free-viewpoint Relighting from Unstructured Photographs\n* Total Relighting: Learning to Relight Portraits for Background Replacement\n\nIn addition, the paper is also related to inverse rendering tasks in computer graphics,\nand it would be good to add discussions on them:\n* Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image\n \n5. What does the normal mean in the text prompt? \n\nOverall, I think the proposed method is technically sound and the results are convincingly \nbetter than the baseline methods. I refrain from giving a higher rating due to concerns \non the scalability of the method considering that it only shows results on car inpainting\nand requires a lot of manually annotated 3D data which may not be easily available for other\nsubjects such as animals and other general objects.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe limitations look good to me.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}