Signiﬁcant picture relighting grants photo change by illumination-speciﬁc altering without human effort and so it is getting much interested of late. Most of the existing predominant procedures open for relighting are run-time genuinely and memory inefﬁcient. Keeping these issues in judgment skills, we propose the utilize of Stacked Signiﬁcant Multi-Scale Dynamic Organize, which sums highlights from each picture at different scales. Our course of action is differentiable and solid for decoding picture brightening setting from input picture to target picture. Besides, we have besides showed up that utilizing a multi-step planning approach to this issue with two unmistakable hardship capacities can inside and out boost execution and can ﬁnish a tall quality diversion of a relighted picture. 1 Introduction Most of the professional tools have limited ability to manipulate light for small ﬂuctuations in intensity or color. Designers are required to spend endless hours to get the appropriate settings for their images with limited options of enhancements being possible. In contrast, this can be solved on a large scale by using deep learning-based methods with the availability of higher variability. This has massive applicability in research as well as in practice. There are multiple instances where relighting based methods would be helpful for artistic enhancement. Some of them include its usage in applications like editing of photos, manipulation of shadows, and variation of the color temperature of the image. Image relighting can also be used for data augmentation to increase the amount of training data, which will make the model robust to changes in color temperature or position of light source. It can also be used for domain adaptation, in which input images will be transformed from different illumination settings into a unique set of illumination settings that the particular network was trained upon. Here, the problem description is to transfer one ﬁxed set of illumination settings from an input image taken at a scene to another ﬁxed set of illumination settings as target image at same scene. In this paper, we propose the use of both Deep Multi-Scale Hierarchical Network and Stacked Deep Multi-Scale Hierarchical Network for robust translation of image color temperature from input image to target image. These models have used a “coarse-to-ﬁne” approach by exploiting multi-scale input images at different processing levels. Our end to end method is differentiable unlike other two step non-differentiable methods where color temperature transfer and shadow generation is taking place in two separate sub-networks with intermediate steps. Our model is also very lightweight and have a lower average inference time than other comparable multi-scale models. We have also demonstrated how our weighted combined loss function performs much better than differentiable sobel loss function which are used for getting reconstructed relighted images with more prominent edges. In addition, we have also shown that with two distinct loss functions, using a multi-step training approach to this problem can signiﬁcantly increase performance and can achieve a high-quality reconstruction of a relighted image. Workshop on Differentiable Vision, Graphics, and Physics in Machine Learning at NeurIPS 2020. 2 Related Work Deep learning based methods have played a vital role in relighting real-life and synthetic scenarios with great efﬁciency. Gafton et al. [2] proposed a GAN based Image translation methods like pix2pix [ 5]. There is no additional knowledge requirement of the view geometry and parameters to generate proper lighting effects in this type of approach. Yang et al. [7] proposed the use of Spherical Harmonics lighting parameters [ 10] in the bottleneck layer of the U-Net hourglass network to achieve state-of-the-art results over the portrait relighting task. Ren et al. [9] introduces a regression-based neural network for relighting real-world scenes from a small number of images. They described relighting as the product of two matrices, a light transport matrix generated from input images, and new lighting vectors. Xu et al. [14] proposed a deep learning-based framework to generate images under novel illumination using only ﬁve images captured under predeﬁned directional lights. Their framework included a fully convolution neural network to predict the relighting function and input light direction. 3 Proposed Approach Base Network: We use a multi-scale network which works on a three-level image pyramid. In each level of the network, we have an encoder and a decoder. Decoder output of each level is upscaled and added to input image in next level and passed to encoder of next level. To capture global contextual information better, encoder output of each level is added with encoder output of previous level before feeding it to decoder. We call this base network Deep Multi-Scale Hierarchical Network (DMSHN) [1]. We can not improve performance of our model by adding additional lower level at DMSHN model as discussed in [ 15]. Thus, we cascade the same network twice to increase the performance of our network. We refer our ﬁnal model as Stacked DMSHN. The model architecture diagram is shown in Fig. 1. Figure 1: Architecture Diagram of Stacked DMSHN. Loss function: We have done two stage training to enhance our results. First, we are using only L2 loss as a reconstruction loss to train our model. Then, we used a linearly weighted loss function as objective loss function in second phase of training to get our ﬁnal trained model. The combined loss function is given by, LCL=1L1+2LSSIM +3Lp+4Ltv (1) where,L1is Mean Absolute Error (MAE) loss, LSSIM is SSIM loss [ 13] ,Lpis Perceptual loss [ 3] andLtvis TV loss [ 8]. During training, values of 1,2,3and4are chosen to be 1,5e3, 0:006and2e8respectively. 4 Experiments 4.1 System Description We have implemented our code using Pytorch on a machine with AMD 1950X, 64GB RAM and 11 GB NVIDIA GTX 1080 Ti GPU. We have used Adam optimizer to train our networks. During 2 training, the Batch Size is 2 and we are resizing input image from 1024 1024 to512512. Initial learning rate is chosen to be 2e3and the learning rate is gradually decreased to 5e5. We have trained our model with these settings for 2500 epochs. During testing, we feed the full resolution image to the network to get our output relighted image. 4.2 Dataset Details We have used VIDIT dataset [ 4], a multi-scene synthetic dataset where the light source varies in 8 different equally-spaced azimuthal angles directions with a ﬁxed polar angle, and ﬁve different colour temperatures. The dataset includes 390 different scenes that are captured with a total of 15,600 images at 40 different illumination settings for each illumination direction and colour temperature per scene. The dataset comprises of 390 samples with 300 images that have been used as a training dataset and 45 images being used as validation and testing each. In this paper, we have considered input illumination setting as light direction from North with color temperature 6500K and output illumination setting as light direction from East with color temperature 4500K in our experiments. 4.3 Results Comparison with other methods: As ground truth for test set is not made publicly available, we are using validation set to show the qualitative and quantitative results for our model. Quantitative results on PSNR, SSIM [ 13] and LPIPS [ 16] metrics are shown in the Table 2. We have also compared our performance of the model with other two similar Multi-Scale models, Scale-Recurrent Network (SRN) [ 11] and Dense GridNet [ 6]. We also compared our work with DRN[ 12] as they had worked on the same problem with same dataset. For qualitative comparison against other networks, example of output images are shown in Fig. 2. From the both qualitative and quantitative results, it can be observed that two stage training of both DMSHN and Stacked DMSHN network signiﬁcantly improved the performance of the model and in three out of four metrics, our method is achieving better performance than others. From the Fig. 2, it can be inferred that predicted images from Dense-GridNet is dis-conﬁgured and have artifacts in spite of having the lowest LPIPS score but output images from our network have less artifacts and are perceptually better than the later one. Due to unavailability of depth information, our model has limited success in light gradient generation. Figure 2: Qualitative Comparison of proposed method against Other Similar Multi-Scale Models. From left: (a) SRN [ 11] , (b) Dense-GridNet [ 6], (c) DRN[ 12], (d) DMSHN (Ours) [ 1], (e) Stacked DMSHN (Ours), (f) Ground Truth. Ablation Study: We have experimented with different loss functions such as (a) A linear combination ofL2loss and Advanced Sobel loss [ 17](LSobel )as a Combined Sobel Loss ( LCSL) (b) OnlyL2 loss (LMSE )and (c) Combined Loss (LCL)to train our networks. In case (a) and (b), same loss function was used throughout the training, whereas in case (c) training was done in two stages. For DMSHN,LCSL produces best SSIM score, whereas LCLyields best scores for PSNR and LPIPS metrics as shown in Table 2. LCLperforms better than LMSE for Stacked DMSHN. Qualitative comparison for different model and loss functions are shown in Fig. 3. 3 Method PSNR SSIM LPIPS Runtime(s) Dense-GridNet[6] 16.67 0.2811 0.3691 0.9326 SRN[11] 16.94 0.5660 0.4319 0.87 DRN[12] 17.59 0.596 0.440 0.5 DMSHN[1] (Our Method) 17.20 0.5696 0.3712 0.0058 Stacked DMSHN (Our Method) 17.89 0.5899 0.4088 0.0116 Table 1: Quantitative Comparison of our method against Other Similar Multi-Scale Models on validation data. Figure 3: Single View Approach Results for both stage of training. From left: (a) Input Image, (b) DMSHN with Combined Sobel Loss, (c) DMSHN with L2Loss, (d) DMSHN with Combined Loss, (e) Stacked DMSHN with L2Loss, (f) Stacked DMSHN with Combined Loss, (g) Ground Truth. 5 Conclusion In this paper, we have proposed a fast and efﬁcient stacked network for one to one image relighting. It is a network with provisions for Multi-Scale training. Our model has the best runtime compared to currently available models to process a High resolution image. Our method is fairly generalized and can be utilized for the tasks mentioned above. Though occluded dark parts of images cannot be generated by the network as there is no information present in that region to be recovered by the network. This problem can be mitigated to some extent if we have two input images illuminated from opposite directions, which can be a research direction to be explored in future. The network is fairly good for translating image color temperature from input image to target image and also performs moderately for light gradient generation with respect to target image.