SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiajun Tang $^{1,2}$  Haofeng Zhong $^{1,2}$  Shuchen Weng $^{1,2}$  Boxin Shi $^{1,2*}$ $^{1}$ National Key Laboratory for Multimedia Information Processing  
 $^{2}$ National Engineering Research Center of Visual Technology  
School of Computer Science, Peking University  
{jiajun.tang, hfzhong, shuchenweng, shiboxin}@pku.edu.cn
We present the iLumination-Aware conditional Image Repainting (LuminAIRe) task to address the unrealistic lighting effects in recent conditional image inpainting (CIR) methods. The environment lighting and 3D geometry conditions are explicitly estimated from given background images and parsing masks using a parametric lighting representation and learning-based priors. These 3D conditions are then converted into illumination images through the proposed physically-based illumination rendering and illumination attention module. With the injection of illumination images, physically-correct lighting information is fed into the lighting-realistic generation process and repainted images with harmonized lighting effects in both foreground and background regions can be acquired, whose superiority over the results of state-of-the-art methods is confirmed through extensive experiments. For facilitating and validating the LuminAIRe task, a new dataset CAR-LUMINAIRE with lighting annotations and rich appearance variants is collected.
Advanced image editing is in high demand across a multitude of applications, e.g., old photo colorization [78, 32, 68], damaged image restoration [48, 73, 72], and artistic style transfer [22, 35, 70]. Recently, conditional image repainting (CIR) [67, 66, 58] has emerged as an innovative research topic, proven effective in controllable image editing while "freeing" users from the necessity of expert proficiency and retaining the "freedom" to actualize their creative visions for image modification. By utilizing provided attributes or textual descriptions, fine-grained strokes, and Gaussian noise to separately represent colors, contours, and texture conditions, users could insert generative objects with desired appearances in specified image positions, as shown in the blue line of Fig. 1.
Although CIR methods have made great progress in synthesizing photo-realistic and visually-pleasing conditional images by avoiding gradient vanishing pitfall [67], adopting flexible condition representation [66], and designing condition fusion modules [58], there is still a crucial element missing from the CIR task: making the synthesized results harmonized with the illumination of the scene, e.g., spatially-varying dark and bright regions in accordance to the lighting condition in the background, physically-accurate highlight effects for highly-specular surfaces (shining objects), and perceptually-realistic shadow avoiding "floating objects" artifacts, as shown in the lower right example of Fig. 1.
Specifically, existing CIR methods handle image harmonization purely in 2D image space by estimating a pixel-wise color tone transformation of the repainted regions from the background regions. Current approaches use semantic parsing maps as "geometry" representations and do not exploit the lighting information contained in given background images, which prevents them from having
Figure 1: Illustration of proposed LuminAIRe task and result. Compared with the previous CIR task [58] (blue line) which takes all condition inputs² at once conducting a conditional image-to-image generation purely in 2D image space, LuminAIRe (orange line) exploits 3D lighting and geometry information and repairs both foreground (fg.) and background (bg.) regions via a lighting-realistic generation process. The 3D information is transformed back to 2D image space in the form of an illumination image, with the desired reflective properties obtained from an illumination attention module. LuminAIRe handles (i) surface shading, (ii) highlight effects, and (iii) realistic shadow in the repaired image (top right).
awareness of physically-based lighting in 3D space. To introduce physically-correct 3D lighting instead of hallucinated lighting effects into CIR results, there remain some major challenges: (i) The lighting condition in 3D space should be extracted from the limited field of view (limited-FoV) 2D LDR images; (ii) the lighting condition should be physically-correctly transformed back into 2D image space; (iii) a dataset suitable for learning-based solutions to the proposed task is needed.
To achieve lighting-realistic generation within the CIR pipeline in an illumination-aware manner, we hereby propose the task of ilLumination-Aware conditional Image Repainting, denoted as LuminAIRe. We first lift geometry conditions from 2D parsing maps to  $3\mathrm{D}^1$  normal maps using learning-based shape priors and estimate lighting conditions from limited-FoV LDR background images by designing a proper parametric representation. Then, we use physically-based reflection models to render illumination candidate images to capture possible lighting effects in 2D image space. With the help of illumination attention module, surface regions with different reflective properties are learned to adopt correct lighting effects in the resulting appearance. A dataset containing rich geometry and lighting annotations with abundant object variants is collected to facilitate the learning-based solution of the LuminAIRe task. As far as we know, we are the first to emphasize illumination-awareness in the image editing task of conditional image inpainting.
- introducing a new task of iLumination-Aware conditional Image Repainting (LuminAIRe) by exploiting the lighting information from background images;
- designing a full LuminAIRe pipeline that represents, extracts, converts, and injects lighting information to acquire more realistically repainted results; and
- collecting a new dataset CAR-LUMINAIRE with rich material and lighting condition variants for facilitating and validating the LuminAIRE task.
Our method aims at introducing physical lighting constraints into generative image synthesis pipelines. In this section, we briefly review relevant works first and then discuss the relationships to our task.
Controllable image synthesis. Researchers have presented numerous works to synthesize images under the guidance of diverse user-provided conditions, e.g., synthesizing specific object with category label [10, 43, 45, 75], transferring the texture from paintings to daily photos [22, 16, 35, 70], restoring
Strictly speaking, the normal maps are in 2.5D. Here we use 3D to simply distinguish it from 2D.
In this paper, attributes are shown in the templated sentences for formatting, and texture is omitted for simplicity.
the colors of old photos [11, 12, 69, 68], and directly generating images from text descriptions [51, 50, 55, 56]. Recently, with the development of the condition injection mechanism [31, 47, 80, 34], researchers explore to control synthesized images with multiple cross-modality conditions, e.g., condition guided image inpainting [44, 55], controllable person synthesis [54, 65], and inversion-based style transfer [79]. However, few works focus on synthesizing images strictly following lighting conditions. Following DIH-GAN [6] that considers introducing illumination estimation in harmonization task that adjusts the highlight of the inserted given object, we further explore the lighting condition in synthesizing illumination-consistent objects under the guidance of multiple cross-modality conditions.
Conditional image repainting and image harmonization. Conditional image inpainting (CIR) aims at synthesizing reasonable visual content on an existing image, where the generated visual content should both meet the requirement of the user-provided conditions (e.g., color, geometry, and texture) and in harmonization with the existing background image. The first CIR task is proposed in MISC [67] for person image synthesis, where the foreground person image is synthesized first and then composited with the background. Weng et al. [66] design the semantic-bridge attention mechanism which allows more freely expressed color conditions by the users in text. UniCoRN [58] breaks the two-stage dependency and proposes a unified architecture that achieves more visually pleasing results. Despite recent achievements made by previous works in condition consistency, existing CIR models suffer from the issue of illumination inconsistency: although techniques such as color tone transform are applied, the lighting from the given background and on the generated visual contents often differ a lot, making lighting effects in the image rather unrealistic, such as incorrect shading, highlights, and shadows. In this paper, we address this issue by exploiting lighting and shape constraints in 3D space, which allows a more physically-correct rendering processing for generating lighting effects. Image harmonization methods [23, 24, 25, 14, 42, 62, 59], with a similar goal of CIR to realistically composite image foreground and background regions, have focused on illumination harmonization recently [6, 8]. However, this thread of works has poor control of visual content in foreground regions and may fail to preserve the color tone in background regions as they were.
Lighting representation and estimation. Achieving illumination-aware synthesis/repainting requires appropriate lighting representation and estimation from images. Lalonde et al. [37] is the first to use shadows, shading, and sky appearance variations observed in the image to infer outdoor lighting. A physics-based Hošek-Wilkie (HW) sky model [29, 30] is proposed to recover HDR parameters for deep outdoor lighting estimation [28]. A more weather-robust Lalonde-Matthews (LM) model [38, 77] is then proposed to cover more comprehensive lighting conditions in the outdoor environment. More recently, a learning-based lighting representation [27] is used on a large sky panorama dataset [36] with an autoencoder network. The encoder-decoder framework is further proposed [39] to estimates lighting as a spherical HDR lighting map. HDSky [74] and SOLD-Net [60] disentangle several physically meaningful attributes into separate learned latent spaces by hierarchical autoencoders and make the estimation editable. Parametric models such as spherical harmonic (SH) coefficients [7, 21] and spherical Gaussian (SG) [19, 40] are also widely used, especially in indoor scenes. Gardner et al. [20], Neurlllum [57], and SOLID-Net [81] design sophisticated networks to hallucinate the missing parts in the panoramic view and predict lighting as environment maps. 3D volumetric lighting representations are also widely used in recent works, which facilitate the lighting-realistic scene editing for indoor [41] and outdoor [64] scenes, however heavily require computation resources. Considering the demand for lighting-realistic generation, we propose a parametric lighting representation for outdoor scenes that is both easy to predict and simple to use.
For self-containedness, we briefly review the CIR formulation before introducing ours.
3.1 Preliminaries about CIR
The previous CIR tasks [66, 67, 58] aim at generating the repaired image  $y^{\mathrm{r}}$  by repairing certain regions in an image  $x \in \mathbb{R}^{3 \times H \times W}$  according to user-specified conditions in different modalities:  $x^{\mathrm{g}}$ ,  $x^{\mathrm{p}}$ ,  $x^{\mathrm{c}}$ , and  $x^{\mathrm{b}}$  for the "geometry", "texture", "color", and background conditions respectively.
In their works, the "geometry" condition  $x^{\mathrm{g}} \in \mathbb{L}^{N_{\mathrm{g}} \times H \times W}$  is a binary semantic parsing mask, where  $N_{\mathrm{g}}$  is the number of possible parts of the visual content to be repainted and  $\mathbb{L} = \{0,1\}$ ; the "texture"
condition  $x^{\mathrm{p}} \sim \mathcal{N}(0,1)$  is a Gaussian noise; the "color" condition can be represented as attributes  $x^{\mathrm{c}} \in \mathbb{L}^{N_{\mathrm{c}} \times N_{\mathrm{v}}}$  or text descriptions  $x^{\mathrm{c}} = \{x_{t}^{\mathrm{c}}\}_{t=1}^{N_{\mathrm{L}}}$ , where  $N_{\mathrm{c}}, N_{\mathrm{v}}$ , and  $N_{\mathrm{L}}$  represent the numbers of attributes and available choices, and the length of the user-inputted sentences, respectively; the background condition  $x^{\mathrm{b}} \in \mathbb{R}^{3 \times H \times W}$  is the image of background region with respect to the repaired region as foreground region, i.e.,  $x^{\mathrm{b}} = (1 - m) \odot x$ , where the binary mask  $m$  indicating foreground region can be directly acquired from the parsing mask  $x^{\mathrm{g}}$ , as shown in lower left of Fig. 1.
The repaired image  $y^{\mathrm{r}}$  can be further decomposed as a blending of repaired foreground image  $y^{\mathrm{f}}$  and repaired background image  $y^{\mathrm{b}}$ :
$$
y ^ {\mathrm {r}} = m \odot y ^ {\mathrm {f}} + (1 - m) \odot y ^ {\mathrm {b}}. \tag {1}
$$
Previous works assume unchanged background region, i.e.,  $y^{\mathrm{b}} = x^{\mathrm{b}}$ , leaving the key question of CIR tasks as generating realistic foreground region  $y^{\mathrm{f}}$  constrained by given conditions:
$$
y ^ {\mathrm {f}} = \mathrm {F} ^ {\mathrm {G}} \left(x ^ {\mathrm {g}}, x ^ {\mathrm {p}}, x ^ {\mathrm {c}}, x ^ {\mathrm {b}}\right), \tag {2}
$$
where previous works ignore clues in 3D space and implement the generation pipeline  $\mathrm{F}^{\mathrm{G}}$  as a conditional image-to-image generation purely in 2D image space. To make the repaired image harmonized as a whole, previous works [58, 67] design additional harmonization modules to adjust the color tone of intermediate repairing result based on clues in  $x^{\mathrm{b}}$ .
3.2 Formulation of LuminAIRe
However, the image-based harmonization modules have limited representation ability for complex lighting effects (e.g., varying shading and shiny surfaces) due to a lack of 3D representation. Besides, directly using  $x^{\mathrm{b}}$  as  $y^{\mathrm{b}}$  in Eq. (1) may neglect possible light transport effects (e.g., shadows) introduced by the repainted region as its corresponding behaviors in the 3D real world might be.
As illustrated by the rendering equation [33], a physically-correct and -realistic appearance of an object is derived from its geometry, reflective property, and omnidirectional environment lighting in 3D space. Therefore, to make the repaired image  $y^{\mathrm{r}}$  more lighting-realistic, the repaired foreground  $y^{\mathrm{f}}$  should also be conditioned by the lighting condition  $L$  and geometry condition  $G$  in 3D space:
$$
y ^ {\mathrm {f}} = \mathrm {F} ^ {\mathrm {F}} \left(x ^ {\mathrm {g}}, x ^ {\mathrm {p}}, x ^ {\mathrm {c}}, x ^ {\mathrm {b}}, L, G\right). \tag {3}
$$
Given  $L$  and  $G$  in 3D space, a proper 2D representation  $x^{\mathrm{i}}$  containing both the information from  $L$  and  $G$  should be derived for compatibility with current image generation architectures:
$$
x ^ {\mathrm {i}} = \mathrm {R} ^ {\mathrm {i}} (L, G), \tag {4}
$$
and then the lighting-realistic generation for foreground  $y^{\mathrm{f}}$  can be rewritten as:
$$
y ^ {\mathrm {f}} = \mathrm {F} ^ {\mathrm {F}} \left(x ^ {\mathrm {g}}, x ^ {\mathrm {p}}, x ^ {\mathrm {c}}, x ^ {\mathrm {b}}, x ^ {\mathrm {i}}\right). \tag {5}
$$
Similarly, the repainted background  $y^{\mathrm{b}}$  should also be conditioned on  $x^{\mathrm{i}}$  to recover lighting effects:
$$
y ^ {\mathrm {b}} = \mathrm {F} ^ {\mathrm {B}} \left(x ^ {\mathrm {b}}, x ^ {\mathrm {i}}\right). \tag {6}
$$
The limited-FoV background image  $x^{\mathrm{b}}$  itself is a partial observation of environment lighting and thus can provide clues about  $L$ . Therefore, the lighting condition  $L$  can be inferred in the form of:
$$
L = \mathrm {F} ^ {\mathrm {L}} \left(x ^ {\mathrm {b}}\right). \tag {7}
$$
Similarly, by finding the shape priors of certain types of objects, the 3D geometry condition  $G$  can be lifted from its "2D flattened version", i.e., parsing mask  $x^{\mathfrak{g}}$ :
$$
G = \mathrm {F} ^ {\operatorname {G e o}} \left(x ^ {\mathrm {g}}\right). \tag {8}
$$
Moreover, in our LuminAIRe formulation, we extend the attributes  $x^{\mathrm{c}}$  beyond colors, which allows the users to describe the reflective property and have control over the lighting effects of repainting results. A sample of attributes is shown as the bold text in the lower left of Fig. 1.
As aforementioned, both the repainted foreground  $y^{\mathrm{f}}$  and background  $y^{\mathrm{b}}$  are given by the lightingrealistic generation in our LuminAIRe formulation, which leads to more realistic and harmonized results than traditional CIR pipelines [58], as shown in Fig. 1.
Figure 2: Data preparation process of the CAR-LUMINAIRE dataset.
4 Data Preparation
To tackle the data shortage issue, we create the first dataset suitable for the LuminAIRe task, named CAR-LUMINAIRE, with its data preparation process and data sample shown in Fig. 2.
3D car models with hierarchical semantic labeling. Collecting large-scale real data for learning-based LuminAIRe methods is infeasible since the geometry and lighting capture in 3D space requires specialized equipment and extensive human labor. Therefore, here we resort to computer graphics techniques to create photo-realistic synthetic data. The cars are chosen as the foreground objects for the obviousness of lighting effects and the availability of high-quality synthetic models. We collect 198 detailed 3D car models in 17 different categories from online model stores [2, 4] and then label the parts of the models in 3D space, which allows us to get the accurate parsing mask in 2D image space from any viewpoint. Following the common structure of vehicles, we divide the car models into 35 semantic part labels. The part labels are organized in a hierarchical way (e.g., the door window is a sub-part of the door) to accommodate car models in different granularity. Besides 3D labeling, we manually adjust the scales of each model to fit the real-world dimensions.
Background images with lighting annotations. Then we prepare background images with known lighting annotations. Here we use the SUN360-HDR dataset [27, 76], which contains HDR panoramic environment maps (envmaps) corresponding to the LDR panoramas of outdoor scenes in the SUN360 dataset [71]. Limited field-of-view (limited-FoV) background images are cropped from the LDR panoramas with virtual cameras of randomized FoVs and camera poses. For each cropped background image, the corresponding HDR envmap in the SUN360-HDR dataset [27, 76] is warped to align with the viewing direction of the virtual camera. Background images unsuitable for realistic object insertion are manually filtered out, leaving 1,321 images of diverse scenes and lighting conditions.
Enhanced data rendering with realistic placement. For each background image, we randomly select insertion points within the central region of the "placeable flat ground" marked by an off-the-shelf segmentation toolbox [15]. Then, for each 2D insertion point in the image, we calculate the relative transformation from the camera coordination  $O_{\mathrm{c}}$  to the local coordination of the object  $O_{\mathrm{o}}$  from the depth  $d$  and the normal  $Z_{\mathrm{o}}$  estimated by depth [52, 53] and normal [5] estimation methods. With the aligned envmaps and the ray-tracing based Blender [3] Cycles rendering engine, physically-correct lighting effects can be rendered into the composited images. In the rendering process, besides the original materials of the models, several physics-based rendering (PBR) car paint materials are randomly applied for more appearance variants, especially in lighting effects; besides, the inserted models are randomly rotated around  $Z_{\mathrm{o}}$  axis for more geometry variants. The rendered images are filtered to ensure reasonable pixel portions of both foreground and background regions. At last, 52,581 composited images at the resolution of  $256 \times 256$  are collected, accompanied by parsing mask and normal map annotations, as shown in the data sample of Figure 2.
To realize the LuminAIRe formulation, we first estimate 3D lighting and geometry from background images and parsing masks (Sec. 5.1). Then the lighting information is injected into the lighting-realistic generation process as illumination images (Sec. 5.2). By further introducing hierarchical labeling enhancement (Sec. 5.3), our method can generate reasonable results even with coarse-level parsing masks. Our pipeline is shown in Fig. 3, with detailed network architectures and loss functions for network modules in supplementary materials.
Figure 3: Overview of LuminAIRe pipeline.
Figure 4: Our lighting representation can capture most of the lighting effects in all weather conditions.
5.1 Estimating 3D Information with Learning-based Priors
Our CAR-LUMINAIRE dataset consists of outdoor scene images, where the lighting can be approximately decomposed into the high-frequency sunlight and the low-frequency ambient light [60]. Accordingly, we model the lighting condition  $L$  as the addition of a directional light and a 2-nd order spherical harmonics (SH) lighting, which can be specifically described as lighting parameters:
$$
L = \left\{z _ {\text {v i s}}, z _ {\text {i n t}}, z _ {\text {a n g}}, c _ {\text {s u n}}, l _ {\text {s u n}}, \sigma_ {\mathrm {S H}} \right\}, \tag {9}
$$
where  $z_{\mathrm{vis}} \in \{0,1\}$  is the sun visibility,  $z_{\mathrm{int}}$  is the intensity of sunlight,  $z_{\mathrm{ang}}$  describes the "size" of the sun (in solid angle formally),  $c_{\mathrm{sun}} \in \mathbb{R}^3$  is the normalized sun color in RGB channels,  $l_{\mathrm{sun}} \in \mathbb{R}^2$  indicates the sun position, and  $\sigma_{\mathrm{SH}} \in \mathbb{R}^{3 \times 9}$  is the 2-nd order SH coefficients for RGB channels.
As shown in Fig. 4, the parametric representation<sup>3</sup> in Eq. (9) can well fit real-world lighting in sunny, cloudy, and low light conditions. On the other hand, the proposed parametric lighting representation is convenient for network prediction. Here we design a NetL to serve as  $\mathrm{F}^{\mathrm{L}}$  in Eq. (7), where  $l_{\mathrm{sun}}$  is estimated by a classification task and other parameters are estimated by regression tasks. To apply our method to other types of background scenes, specifically tailored lighting representations can be directly adopted, without modification to our underlying formulation of LuminAIRe.
For 3D geometry, we use the normal map  $G \in \mathbb{R}^{3 \times H \times W}$  as the representation where each pixel indicates the surface normal direction  $n$  at that surface point in 3D space. For certain types of objects, there exist strong shape priors (such as sedans and hatchbacks), which can be learned in a supervised way. Similarly, a NetS of encoder-decoder structure is further proposed to serve as  $\mathrm{F^{\mathrm{Geo}}}$  in Eq. (8).
5.2 Injecting Lighting Information using Illumination Images
To bridge the 3D lighting and geometry with 2D images, the rendering equation [33] is a handy tool to serve as  $\mathbf{R}^{\mathrm{i}}$  in Eq. (4), which physically models the image formation process as the light reflection:
$$
L _ {\mathrm {o}} \left(\omega_ {\mathrm {r}}\right) = \int_ {\Omega_ {n}} L _ {\mathrm {i}} \left(\omega_ {\mathrm {i}}\right) f _ {\mathrm {r}} \left(\omega_ {\mathrm {i}}, \omega_ {\mathrm {r}}\right) \left(\boldsymbol {n} \cdot \omega_ {\mathrm {i}}\right) \mathrm {d} \omega_ {\mathrm {i}}, \tag {10}
$$
where  $L_{\mathrm{i}}(\omega_{\mathrm{i}})$  is the environment lighting from direction  $\omega_{\mathrm{i}}$ ,  $L_{\mathrm{o}}(\omega_{\mathrm{r}})$  is the reflected lighting toward direction  $\omega_{\mathrm{r}}$ ,  $\Omega_{n}$  is the visible hemisphere determined by surface normal  $n$ , and  $f_{\mathrm{r}}(\omega_{\mathrm{i}}, \omega_{\mathrm{r}})$  describes the reflective properties of all possible combination of incoming and outgoing directions.
3 Lighting parameters are converted back to tone-mapped HDR environment maps for visualization.
Appearance variants
Illumination candidate images
Figure 5: Our illumination candidate images can cover realistic lighting effects in appearance variants.