Image relighting aims to recalibrate the illumination set- ting in an image. In this paper, we propose a deep learning- based method called multi-modal bifurcated network (MB- Net) for depth guided image relighting. That is, given an image and the corresponding depth maps, a new image with the given illuminant angle and color temperature is gener- ated by our network. This model extracts the image and the depth features by the bifurcated network in the encoder. To use the two features effectively, we adopt the dynamic dilated pyramid modules in the decoder. Moreover, to in- crease the variety of training data, we propose a novel data process pipeline to increase the number of the training data. Experiments conducted on the VIDIT dataset show that the proposed solution obtains the 1stplace in terms of SSIM and PMS in the NTIRE 2021 Depth Guide One-to-one Re- lighting Challenge. 1. Introduction Given an image, image relighting aims to relight this im- age into another image with different ambient conditions. In the NTIRE 2021 Depth Guide One-to-one Relighting Chal- lenge, the depth maps are provided. We plot examples in- cluding the original image, the corresponding depth map, the relighted image by the proposed method, and the ground truth in Fig. 1. As shown in Fig. 1, there are two inherent challenges for image relighting. First, to generate the image with different ambient conditions, it is necessary to gener- ate shadows into the relighted image. Second, similarly, the shadow from the original image needs to be removed. For example, in Fig. 1 (c), although the region shadows are removed, the texture of the grass cannot be recovered ap- *Equally-contributed ﬁrst authors. (a) (b) (c) (d) Figure 1: Example of depth guided image relighting. (a): Original input image. (b): Corresponding depth map. (c): Relighted image by our method. (d): Ground truth. propriately. Previously, many methods [1, 2, 3] based on develop- ing visual priors or capture properties of relighted images have achieved impressive performance. Recently, some deep learning-based methods [4, 5, 6, 7] are proposed with- out explicit inverse rendering steps for estimating scene properties. However, these methods do not consider com- plex scenes and various ambient conditions. Moreover, in NTIRE 2021 Depth Guide One-to-one Relighting Chal- 1 lenge, there are extra difﬁculties needed to be addressed. First, the number of training data given in the VIDIT [8] dataset is 300, which is not enough to train the network. To address these issues, our network is based on the backbone pre-trained from ImageNet. Furthermore, we leverage extra data from Track II: Any-to-Any relighting [7] and develop a strategy that can generate new image pairs to increase the number of the training data and the robustness of the net- work. Second, depth and image features should be effectively extracted and fused. Depth maps and images contain the different attributes of features. Depth maps present spatial information while the images provide texture, light cues, and dark cues. Thus, we refer to the methods proposed in the RGB-D salient object detection [9, 10, 11] and propose a multi-modal bifurcated network to deal with this issue. This network contains encoder and decoder parts. Our encoder applies two branches without sharing weight to extract fea- tures, respectively. Additionally, we apply the dynamic di- lated pyramid module to effectively integrate two features. In the decoder parts, we gradually magnify the feature maps and recover the image. Motivated by the U-Net [12, 13], we apply skip connection to connect the feature maps with identical size from the encoder and decoder parts to obtain better relighted images. We make the following contributions in this paper: 1. The multi-modal bifurcated network is proposed for depth guided image relighting. This structure can ex- tract image and depth features by two branches. Then, these two features are fused by dynamic dilated pyra- mid modules effectively. 2. We propose a new strategy to leverage additional im- ages from Track II and construct more input-output pairs as the training data. 3. Several experiments performed on the VIDIT [8] dataset demonstrate that our solution achieves the 1st place in terms of MPS and SSIM in the NTIRE 2021 Depth Guided One-to-one Relighting Challenge. 2. Related Works Two tasks are very similar to depth guided image relight- ing: RGB-D saliency object detection and image relighting. In this section, we brieﬂy describe several works related to these tasks. 2.1. RGB-D Salient Object Detection Salient Object Detection (SOD) aims to imitate the hu- man visual system and detect certain regions or objects that attract human attention. RGB-D SOD is known as combin- ing the extra depth maps to fulﬁll salient detection. Some models [10, 11] extract features from images and depthmaps independently, and conducts feature maps fusion of the two modalities in the decoder. In [11], Asymmetric Two-Stream Architecture (ATST) is proposed which con- siders the inherent differences between the RGB and the depth data for the salient detection. In [10], Guided Resid- ual (GR) blocks are proposed to feed the RGB image and the depth image alternately to reduce the mutual degrada- tion. They also address progressive guidance in the stacked GR blocks within each side-output to remedy the false de- tection and the missing parts. On the other hand, in [14], the single stream network is designed that concatenates the RGB image and depth image across the channel dimension and directly applies the depth map to guide both the early fusion and the middle fusion between the RGB information and the depth information, which saves the feature encoder of the depth stream. Additionally, in [15], authors propose the RD3D that 3D convolutional neural networks are intro- duced to address the RGB-D SOD. This network adopts the progressive fusion involving both the encoder and the de- coder stages. Since the multi-modal fusion methods from RGB-D SOD can be leveraged for the depth-guided image relighting, in this paper, the proposed MBNet is based on the HDFNet [9]. 2.2. Image Relighting The algorithms for image relighting can be categorized as the physical-based method and the deep learning-based method. Traditional methods [1, 2, 3] depend on the phys- ical observation to further estimate the ambient conditions, reﬂectance, and lighting of the scene in the image and then re-render this scene by another illumination setting. In [2], an algorithm treating the complex scene as a linear system that transforms the original light into the reﬂected light is proposed. This algorithm progressively reﬁnes the approxi- mation of the reﬂectance ﬁeld until the required precision is reached. In [3], authors prove that the light transport of diffuse scenes under the spatially varying illumination can be decomposed into the direct, near-range and far-range transports. They separate these three components in the fre- quency space. On the other hand, many deep learning-based methods for image relighting [4, 5, 7] are proposed. Im- age relighting can be seen as an image-to-image translation problem. Several low-level image processing problems like haze/smoke removal [16, 17, 18], underwater enhancement [19], reﬂection removal [20], image deraining [21] and im- age desnowing [22] are very similar to image relighting. Generally speaking, the encoder-decoder structure like U- Net [12, 13] can deal with these tasks. Some methods that deal with the ambient conditions especially for the image re- lighting are developed. Gray loss described in [5] can drive the network to learn the illumination gradient in target do- main images. Xu et.al propose a CNN-based method [4] to relight a scene under a new illumination based on ﬁve im- 2 Figure 2: The architecture of the proposed multi-modal bifurcated network. The network consists of two streams as encoder parts: depth stream and RGB-image stream. We use the dense architecture ,DDPM and skip connection for better feature extraction as decoder parts. ages captured under a pre-deﬁned illumination setting. This method tries to estimate a non-linear function that general- izes the estimation from the optimal sparse samples. 3. Proposed methods 3.1. Overall Neural Network As shown in Fig. 2, to leverage the depth information for depth guided image relighting task effectively, we refer the dual-stream bifurcated architecture in [9] as the back- bone. Our network consists of two streams to extract the depth and the image features. We apply the two ResNet 50 [23] pre-trained from the ImageNet as the backbones. In the network design, the depth and image features from three intermediate layers are fused to achieve the representative features. We fuse the features from the conv3, the conv4 and the conv5 layers to balance the effectiveness and the ef- ﬁciency of the network. Speciﬁcally, for the features in the shallower layers, they are generally noisier and the high- resolution of these features may increase the computational burden. However, the features in the conv3 to the conv5 may still preserve the valid information [9] and with lower resolution. To fuse these two features with multiple recep- tive ﬁelds, we leverage the densely connected architecture to generate the combined features with the fruitful texture and the structural information. Then, these features are fed to the Dynamic Dilated Pyramid Module (DDPM) [9] that can generate a more discriminative result. We will describe this module in the following section. The out-put of the DDPM is combined with the output of the de- coder by convolving with the multi-scale convolution ker- nels [24, 22, 25]. In the decoder part, similar to the U-net [13, 12], we gradually magnify the feature maps and imple- ment the skip connection to concatenate the identical size feature maps. Furthermore, we make our network learn the residual [23] instead of the whole images. That is, the ﬁnal output is the difference between the original image and the relighted image. 3.2. Dynamic Dilated Pyramid Module In this part, we illustrate the detail of the DDPM. As shown in Fig. 3, the input of DDPM is the fused feature fEand the feature (fRGB)Dfrom encoders, respectively. First, the (fRGB)Dpasses through the convolution kernel to reduce the dimension of the feature which is termed as (fRGB)D. Second, the kernel generation units (KGUs) [9] are adopted on the fused feature fEto generate different weight tensors (i.e, f1 KGU ,f2 KGU ,f3 KGU ) which can cover three square neighborhoods (i.e., 33,77, and1111) . It is noted that KGUs are with four densely connected layers [26] which can improve the feature propagation and feature reuse effectively. Then, we leverage the kernel transforma- tion units (KTUs) to yield regular convolution kernels with various dilation rate (i.e., 1, 3, 5) by reorganizing kernel ten- sors and inserting various numbers of zeros. Then, the three parallel output features combine with the(fRGB)Dby the convolution kernels, respectively. We term these combined features asf1 K;f2 K;f3 K. Finally, we combine f1 K;f2 K;f3 K 3 Figure 3: The architecture of the dynamic dilated pyramid module (DDPM). There are two modules in the network, that is, the kernel generation units (KGUs) and the kernel transformation units (KTUs). and(fRGB)Dto generate the output of the DDPM fC. 3.3. Extra Data Usage In this section, we propose two strategies to increase the training data so that our model can learn the mapping func- tion of the depth guided image relighting robustly. In this challenge, the input image (6500-N) is with 6500K color temperature and the north illumination angle while the out- put image (4500-E) is with 4500K color temperature and with the east illumination angle. In order to improve the ro- bustness of our network, we leverage the images in Depth Guided Image Relighting: Track II Any-to-Any relighting [7]. This track provides the images with various illumina- tion temperature and different illumination angles. Specif- ically, we apply the images with the different illumination angle (6500-NE) but the identical color temperature as the input and the ground truths are the same ones (4500-E). As shown in Fig. 4, the additional image (6500-NE) is very similar to the original one (6500-N). With this additional data, the model can understand the direction information comprehensively. Moreover, We adopt additional images which contain the same scene but with the different illuminating angle (4500- W). We can ﬂip horizontally the image with the west illu- mination angle to achieve the new image with the east illu- mination angle. Thus, we develop a new strategy to further increase training data and illustrate it in Fig. 5. As shown in Fig. 5 (d) and Fig. 5 (e), we ﬂip the original RGB (6500- N) images and the corresponding depth map horizontally. The horizontally ﬂipped image of (4500-W) is the output of the ﬂipped input (6500-N). With this operation, the training data can be increased.3.4. Loss Functions In this paper, we leverage three loss functions to mea- sure the differences between the relighted images and the ground truth. The three loss functions are the Charbonnier loss [27], the SSIM loss [28], and the perceptual loss [29]. The Charbonnier loss can be expressed as: LCha(x;^x) =1 TTX ip (xi^xi)2+2 (1) wherexand^xare the ground truth and relighted images, respectively. eis a tiny constant for the stable and the robust convergence. The second loss function is the SSIM loss function. SSIM loss is expressed as: LSSIM (x;^x) =SSIM (x;^x) (2) SSIM loss is beneﬁcial to reconstruct local structures and details. Finally, the perceptual loss is written as: LPer(x;^x) =j(VGG (x)VGG (^x)j (3) whereVGG means the VGG19 network [30]. In our work, we use the features from conv3-3 layer. The overall loss function containing three terms is expressed as: LTotal =1Lcha+2LSSIM +3LPer (4) where1,2and3are weights to control the ﬁnal objec- tive functions. These three weights are empirically adjusted as hyper-parameters. 4 (a) (b) (c) (d) Figure 4: Additional images are used in the training phase. (a) Original image (6500-N). (b) Different illuminating angle image (6500-NE). (c) Guided depth map. (d) Output image. (a) (b) (c) (d) (e) (f) Figure 5: Using (4500-W) images to increase the diversity of the training data. (a): Original input image. (b): Original depth map. (c): Output image. (d): Flipped image of (a). (e): Flipped depth map of (b). (f): Flipped image of (4500-W). 4. Experiments 4.1. Experimental Setting In the NTIRE 2021 Depth Guide One-to-one Relighting Challenge, the novel Virtual Image Dataset for Illumination Transfer (VIDIT) [8] is provided as the training and the val- idation data. This dataset consists of 390 various scenes that are captured at 40 different illumination conditions in- cluding 8 different azimuthal angles and ﬁve color temper- atures such as 2500K, 4500K, etc. Furthermore, the corre- sponding depth maps are provided. In track I - depth guided one-to-one relighting, the input images are depth map and a pre-deﬁned illumination condition = North, temperature = 6500K (e.g., (6500-N)) and the output image is set at a dif- ferent illumination setting = East, temperature = 4500K. Though in track I, only two conditions of images are used as the input and the output pairs, it is allowed to utilize the extra data to improve the accuracy of the model. During the training and the evaluation phases, the image size is 1024  1024, and we do not use any data augmentation like random ﬂip and random crop. The Adam optimizer [31] is utilized with a batch size of 3 to train the network. We train the network for 200 epochs with the momentum 1= 0.5, 2= 0.999. The learning rate is initialed as 104and divided byten after 50 epochs. The 1,2and3in (4) are set as 1, 1.1 and 0.1, respectively. We perform our experiments on a single Nvidia V100 graphic card and the PyTorch platform. We spend about 11 hours ﬁnishing the model training. In the testing phase, we take 2.8867 seconds to predict a sin- gle image. The source code will be available in our project page. Table 1: The ablation experiment of applying the different data and the residual learning. Description PSNR SSIM Baseline 18.0215 0.6834 + Extra data 18.9677 0.7103 + Extra data + Residual learning 19.3558 0.7175 4.2. Ablation Experiments We conduct the ablation experiment to verify that each module applied in this paper can beneﬁt the proposed re- lighting network. In all experiments, the image size is set as 10241024. We test each module and report the results un- der the validation set. We select the peak signal-to-noise ra- tio (PSNR) and the structural similarity (SSIM) as objective metrics for the quantitative evaluation. Overall, the ablation 5 Figure 6: Visual comparison for the relighted results recovered by the MBNet and other solutions. studies consist of three different experimental scenarios: 1) We apply the original training data to train the MBNet as the baseline. 2) We apply the method described in Section 3 to increase the training data to train our MBNet. 3) We increase the training data and also apply the residual learn- ing [23] strategy. We summary the ablation experiments inTable 1. One can see that both PSNR and SSIM scores of setting 2 are increased compared with setting 1. It can show that increasing training data is beneﬁcial for better perfor- mance and robustness. Additionally, compared with setting 2, the performance of setting 3 is improved effectively. It demonstrates that the residual learning can further improve 6 Table 2: Relighted results by some state-of-the-art RGB-D salient object methods. Methods PSNR SSIM ATST [11] 18.2678 0.665 DANET [14] 18.3341 0.6805 RD3D [15] 17.7763 0.6668 PGAR [10] 17.6436 0.6748 Ours 19.3558 0.7175 the accuracy of the relighting. 4.3. Comparison with State-of-the-art Methods First, we compare the MBNet with four state-of-the-art RGB-D SOD methods including ATST [11], DANET [14], RD3D [15] and PGAR [10] as described in the Section 2. Note that, we use the same training set to train these meth- ods. We replace the ﬁnal convolutional layers of RGB-D SOD networks, so their output is three-channel tensors as relighted images. As shown in Table 3, the MBNet out- performs other methods with a large margin. Our method achieves the best performance on both PSNR and SSIM, which surpasses the second place 1.02 dB and 0.037 in SSIM. Furthermore, we report the performances of some sub- missions in the NTIRE 2021 Depth Guide One-to-one Re- lighting Challenge [32]. The performance is evaluated on the validation and the test dataset and the results are shown in Table 3. Additionally, The Mean Perceptual Score (MPS) is used for ﬁnal evaulation. The MPS is deﬁned as the av- erage of the normalized SSIM and LPIPS [33] scores. The MBNet produces moderate quality outputs and the 1stplace performance in both MPS and SSIM metrics. We also plot some relighted images generated by our method and other participants in Fig. 6. Compared to other methods, our im- ages can remove more shadows and present clear outlines of objects, though some results are not satisfactory. 5. Conclusion In this paper, to address depth guided image relighting, we develop the multi-modal bifurcated network. This net- work extracts both depth and image features by a dual bi- furcated backbone. To fuse multi-modal features, the dy- namic dilated pyramid module is introduced. This module contains densely connected layers and multi-scale kernels to fuse and reﬁne features from a dual bifurcated backbone. Furthermore, to improve the robustness and performance, we propose a new strategy to increase the training image pairs by leveraging extra images. Several experiments im- plemented on the novel VIDIT [8] dataset proves that our solution achieves the 1stplace in terms of MPS and SSIM in the NTIRE 2021 Depth Guided One-to-one RelightingChallenge. 6. Acknowledgement We thank to National Center for High-performance Computing (NCHC) for providing computational and stor- age resources.