SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For a certain image pixel with the known camera viewing direction  $\pmb{v}$ , ideally, the pixel intensity can be calculated as  $L_{\mathrm{o}}(-\pmb{v})$ , and accurate lighting effects can be calculated as illumination images.
However, with  $L$  and  $G$  estimated from input conditions,  $f_{\mathrm{r}}(\omega_{\mathrm{i}},\omega_{\mathrm{r}})$  still remains unknown. Therefore, in a similar spirit to Gao et al. [18] and Pandey et al. [46], instead of directly calculating the actual illumination image, we use a set of uncolored "standard materials" as  $f_{\mathrm{r}}$  in Eq. (10) and render corresponding illumination candidate images  $\{I_{\mathrm{c}}\}$ . For the physics-based rendering of  $\{I_{\mathrm{c}}\}$ , we use the Lambertian reflectance model  $f_{\mathrm{diff}}(\omega_{\mathrm{i}},\omega_{\mathrm{r}}) = 1 / \pi$  and normalized Blinn-Phong model [9]  $f_{\mathrm{spec}}(\omega_{\mathrm{i}},\omega_{\mathrm{r}}) = (\rho +4)(n\cdot h)^{\rho} / 8\pi$  with  $M$  different values of roughness  $\rho$ , where  $h = \omega_{\mathrm{i}} + \omega_{\mathrm{r}} / ||\omega_{\mathrm{i}} + \omega_{\mathrm{r}}||$  is the half vector. At last, we have  $\{I_{\mathrm{c}}\} = \{I_{\mathrm{diff}}\} \cup \{I_{\mathrm{spec}}^{\rho_i}\}_{i = 1}^{M}$ .
As shown in Fig. 5, most lighting effects in different appearance variants can be covered by the linear combinations of the pre-computed  $\{I_{\mathrm{c}}\}$ . However, it's worth noting that the correspondence of the appearance image and  $\{I_{\mathrm{c}}\}$  may vary pixel-wisely (e.g., the tires, hood, and windshield have different reflective properties thus different lighting effects). Accordingly, we design an illumination attention module  $\mathrm{A}^{\mathrm{I}}$  to estimate the combination coefficient maps  $C_{\mathrm{I}} = \mathrm{A}^{\mathrm{I}}(E)$  for each image pixel, where  $E$  is the feature embedding map containing information of both part labels and part-associated attributes in a pixel-aligned way. After the illumination image  $I$  derived as  $I = \sum_{i=1}^{M+1} C_{\mathrm{I}}^{i} \odot I_{\mathrm{c}}^{i}$ , which covers lighting effects of parts with different materials, we use  $I$  as  $x^{\mathrm{i}}$  in Eq. (5) and conduct lighting-realistic generations of foreground and background regions using our proposed NetF and NetB respectively. For NetF, we adopt the network backbone of  $\mathrm{F}^{\mathrm{G}}$  in UniCoRN [58], and the illumination image  $I$  is injected in a similar way as other conditions in 2D image space at different resolutions. The NetB is also an encoder-decoder architecture, serving as  $\mathrm{F}^{\mathrm{B}}$  in Eq. (6). We adopt the same loss functions for NetF as used in UniCoRN [58].
5.3 Generating Realistic Results from Coarse Parsing Masks
As mentioned in Sec. 4, the parsing masks in our CAR-LUMINAIRE dataset can be very coarse, which also reflects the possible application scenarios when the user only specifies interested parts. Previous CIR formulations may fail to generate realistic results in regions without fine-grained labels since their generation follows a strictly pixel-wise semantic mapping between labels and images. We hereby introduce a hierarchical labeling enhancement: randomly coarsening the input parsing mask at training time (e.g., door glass label becomes door label) and encouraging the fine-grained parts (door glass) to be generated. Besides, the part-associated attributes of lower-level parts (door glass) should be also associated with their upper-level parts (door) to avoid loss of condition in attributes  $x^{\mathrm{c}}$ , which can be done by modifying the association matrix [58]  $A \in \mathbb{L}^{N_{\mathrm{c}} \times N_{\mathrm{g}}}$  accordingly.
In this section, we conduct comparisons with state-of-the-art methods and validate our design with an ablation study and a robustness test. Please see supplementary materials for implementation details.
6.1 Comparison with State-of-the-art Methods
Baseline methods. We conduct quantitative and qualitative comparisons with three state-of-the-art CIR methods (UniCoRN [58], Weng et al. [66], and MISC [67]) and a most-relevant conditional image generation method (Pavllo et al. [49]). Among them, modifications are made for Pavllo et al. [49] and Weng et al. [66] to accept conditions represented as attributes.
Quantitative metrics. Following previous work [58], we adopt Fréchet inception distance (FID) [26] for assessment of perception quality, R-precision [66] for assessment of alignment between generated
images  $y^{\mathrm{r}}$  and given attributes  $x^{\mathrm{c}}$ , and M-score [61] for assessment of authenticity. We use the latest manipulation detection model [17, 13] for calculating the M-score [61]. We also report the structural similarity index (SSIM) [63] for comparing the major image structure with the reference image.
Table 1: Comparison with the state-of-the-art methods and variants of our proposed method. Quantitative evaluation scores and user study results are shown.  $\uparrow (\downarrow)$  means higher (lower) is better. "Real." and "Har." are abbreviations of "Realistic" and "Harmonized".
Method	Quantitative Evaluation	User Study
FID ↓	R-prcn ↑	M-score ↓	SSIM ↑	Real. ↑	Har. ↑
MISC [67]	53.84	34.94%	31.23	0.6660	0.25%	0.28%
Weng et al. [66]	38.12	46.66%	30.84	0.6697	0.85%	0.85%
Pavlo et al. [49]	9.29	56.98%	36.77	0.7050	43.00%	36.72%
UniCoRN [58]	11.55	62.13%	29.72	0.6940	7.78%	9.90%
LuminAIRe (Ours)	4.62	74.13%	13.68	0.7211	48.12%	52.25%
Ours-H	5.83	63.27%	13.97	0.7163	—	—
Ours-HA	6.31	63.94%	13.95	0.7214	—	—
Ours-HAI	8.00	62.13%	15.83	0.7054	—	—
Figure 6: Qualitative comparison with the state-of-the-art methods, with given conditions (conds.).
The scores in Tab. 1 and the second and the third columns of Fig. 6 show that results of MISC [67] and Weng et al. [66] are far from lighting-realistic with "crayon-drawing-like" appearances, since the color tone transform is not applied [66], or conducted in a two-phase manner [67]. As shown in the fourth column of Fig. 6, Pavllo et al. [49] tend to generate foreground regions in flat shadings with fewer texture patterns, which makes its results generally look reasonable when only focusing on foreground regions or in low light or cloudy scenes (as indicated by the FID and user study results), but computer vision models can easily find the disharmony due the sharp boundaries between foreground and background regions [58], as also indicated by the worst M-score. UniCoRN [58] fails to generate correct lighting effects from its unified color tone transform (the first row), therefore tends to hallucinate highlights at the top of cars regardless of lighting in background regions (the second row). The hallucinated lighting effects along with the undesired texture pattern on car bodies drastically damage the perceptual preferences, as confirmed by the FID score and user study results in Tab. 1. LuminAIRe generates realistic lighting effects close to the reference images in both sunny (the first and the third rows) and cloudy (the second row) scenes of specified materials and even when a coarse-level parsing mask is given (the third row), with a large margin in all quantitative metrics compared with baseline methods. LuminAIRe also learns to avoid the undesired texture pattern with the hints of the smoothly varied shading in the illumination images (Fig. 7).
Figure 7: Ablation study for three variants of our proposed method, with given conditions (conds.).
Figure 8: Our method can generate realistic lighting effects with given materials (left), which are consistent across different geometry conditions (right). Green and blue boxes mark individual cases.
6.2 Evaluations
Ablation study. We conduct an ablation study with three variants of Ours: (i) Ours-H, (ii) Ours-HA, and (iii) Ours-HAI, where “-H”, “-A”, and “-I” mean disabling the hierarchical labeling enhancement, the illumination attention, and the illumination injection for the foreground, respectively.
The hierarchical labeling enhancement is confirmed helpful in generating realistic results with coarse-level parsing masks, as shown in Fig. 7 and the third row of Fig. 6, where Ours generates more consistent and better repaintings at regions with no specified part labels (marked in blue purple), which is also demonstrated by the FID and R-prcn score in Tab. 1. The second row of Fig. 7 shows an example where the lack of illumination attention module wrongly renders a diffuse appearance, with further evidence from the drop of FID from Ours-H to Ours-HA in Tab. 1. It's quite obvious from Tab. 1 and Fig. 7 that the illumination injection helps foreground generation by comparing Ours-HA and Ours-HAI. From UniCoRN to Ours-HAI, the improvements in FID score and M-score validate the contribution of the lighting-realistically generated background.
Besides, Ours-HA gets an unexpectedly good SSIM score. It's possibly because a slight misalignment of lighting effects (especially highlights) due to errors in lighting or geometry estimation would lead to a considerable drop in the SSIM score (which honestly measures the pixel-wise difference) but with very little harm to the lighting-realistic perception (as indicated by the FID and M-score).
User study. We also conduct a user study with 20 volunteers on the Amazon Mechanical Turk [1] platform, where 200 sets of results randomly drawn from the test set are shown and volunteers are asked to choose one in each set with (i) the most realistic foreground and (ii) the most harmonized lighting. The results of the user study in Tab. 1 are basically aligned with the trending of FID and SSIM scores in quantitative evaluation, showing that repainting results of our LuminAIRe are most favored subjectively, with a greater lead in realistic and harmonized lighting perception.
Robustness Test. Fig. 8 shows the robustness of our method to varying materials and geometry conditions, where different materials and geometry conditions are correctly handled with realistic lighting effects accordingly and consistently generated. To test the robustness of our method to varying parsing masks (e.g., casually-drawn parsing masks), we compare in Fig. 9 the repainting
results of from the input parsing masks before and after the disturbing, where the boarders are randomly extended and the inner structures are coarsened. To test the robustness of our method to varying lighting conditions, we conduct an experiment where the estimated lighting conditions are rotated clockwise while all other conditions are left unchanged. The results in Fig. 10 show that our method correctly handles most of the lighting rotations in the sense of the lighting effects on the foreground objects and the shadow effects in the background regions. The repainting results in the second column with no lighting conditions given ("No light") further validate the effectiveness of our illumination injection module. To test the robustness of our method to varying background conditions, we also show the results of in-the-wild examples in the supplemental material.
Figure 9: Qualitative results of normal maps, illumination images and repaintings using original (first row) and disturbed (second row) parsing masks as input conditions. Backgrounds are omitted here.
Figure 10: Qualitative results of repaintings and illumination images as the estimated lighting rotates.
In this paper, we introduce the task of LuminAIRe for the realistic generation of lighting effects. The synthetic CAR-LUMINAIRE dataset is collected for the newly proposed task. Extensive experiments and the user study confirm that our method achieves perceptually more lighting-realistic and harmonized repainting results compared with the state-of-the-art methods. The effectiveness and consistency of our illumination-aware design are shown in the robustness test.
Limitations and future works. In this paper, only the results of cars as foreground objects are shown, resulting from the inadequate feasibility of data collection. Besides, our model can not handle complex thin structures and some translucent glass materials very well, which are not well covered by our synthetic data for now. As a single-image-based method for generic outdoor scenes, our method currently ignores the non-local inter-reflections with other objects and focuses on the shadows cast directly on the ground. Therefore, datasets of richer object categories and finer details will be helpful to boost the training of learning-based methods. Combining the lighting constraints with the newly emerged latent diffusion models [55] would also be an interesting direction for our future work.
Acknowledgement. This work is supported by the National Natural Science Foundation of China under Grant No. 62136001, 62088102.
[69] S. Weng, H. Wu, Z. Chang, J. Tang, S. Li, and B. Shi. L-code: Language-based colorization using color-object decoupled conditions. In Proc. of the AAAI Conference on Artificial Intelligence, 2022.
[70] S. Weng, P. Zhang, Z. Chang, X. Wang, S. Li, and B. Shi. Affective image filter: Reflecting emotions from text to images. In Proc. of International Conference on Computer Vision, 2023.
[71] J. Xiao, K. A. Ehinger, A. Oliva, and A. Torralba. Recognizing scene viewpoint using panoramic place representation. In Proc. of Computer Vision and Pattern Recognition, 2012.
[72] W. Xiong, J. Yu, Z. Lin, J. Yang, X. Lu, C. Barnes, and J. Luo. Foreground-aware image inpainting. In Proc. of Computer Vision and Pattern Recognition, 2019.
[73] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Free-form image inpainting with gated convolution. In Proc. of International Conference on Computer Vision, 2019.
[74] P. Yu, J. Guo, F. Huang, C. Zhou, H. Che, X. Ling, and Y. Guo. Hierarchical disentangled representation learning for outdoor illumination estimation and editing. In Proc. of International Conference on Computer Vision, 2021.
[75] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial networks. In Proc. of International Conference on Machine Learning, 2019.
[76] J. Zhang and J.-F. Lalonde. Learning high dynamic range from outdoor panoramas. In Proc. of International Conference on Computer Vision, 2017.
[77] J. Zhang, K. Sunkavalli, Y. Hold-Geoffroy, S. Hadap, J. Eisenmann, and J.-F. Lalonge. All-weather deep outdoor lighting estimation. In Proc. of Computer Vision and Pattern Recognition, 2019.
[78] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In Proc. of European Conference on Computer Vision, 2016.
[79] Y. Zhang, N. Huang, F. Tang, H. Huang, C. Ma, W. Dong, and C. Xu. Inversion-based creativity transfer with diffusion models. In Proc. of Computer Vision and Pattern Recognition, 2023.
[80] P. Zhu, R. Abdal, Y. Qin, and P. Wonka. SEAN: Image synthesis with semantic region-adaptive normalization. In Proc. of Computer Vision and Pattern Recognition, 2020.
[81] Y. Zhu, Y. Zhang, S. Li, and B. Shi. Spatially-varying outdoor lighting estimation from intrinsics. In Proc. of Computer Vision and Pattern Recognition, 2021.