# Image Shadow Removal Using End-To-End Deep Convolutional Neural Networks
## 3 End-To-End Convolutional Neural Network
### Training  
The relationship between the shadow image \(I_{s}\) and its shadow mask \(\alpha\) is given by Equation (1). In training, the actual shadow mask \(\alpha\) was calculated according to a given pair of shadows and shadow-free images. Then, to establish the relationship between the shadow image \(I_{s}\) and the shadow mask \(\alpha\), the mapping function was as follows:  
\[\alpha=F(I_{s},\beta). \tag{3}\]  
Among them, \(\beta\) was the learning parameter of the convolution network. During training, a linear fusion of two loss functions was used to adaptively train the RSnet model. Figure 1: The architecture of the RSnet. RSnet consists of two cooperative sub-networks. The left side of the dotted line shows the encoding–decoding network, and the right side shows the refinement network. \(\alpha\) prediction loss: The first loss was called \(\alpha\) prediction loss, which was the difference between the absolute ground actual \(\alpha\) value and the predicted \(\alpha^{\prime}\) value of each pixel. The mean square error (MSE) is used as a loss function to approximate it:  
\[L(\beta)=\frac{1}{N}\frac{N}{i}\Big{\|}F(I_{s}^{i},\beta)-\alpha_{i}\Big{\|}, \tag{4}\]  
where \(N\) is the total number of training samples in the batch, \(F(I_{s}^{i},\beta)\) is the shadow mask of the \(ith\) image pair output by the prediction layer through the training network, and \(\alpha_{i}\) is the true shadow mask of the \(ith\) image pair. Composition loss: The second loss was called the composition loss, which was the discrepancy between the true shadow image RGB color and the predicted shadow image RGB color. The predicted shadow image RGB color was synthesized by the shadow-free image and the shadow layer factor \(\alpha^{\prime}\) outputted in the prediction layer. The loss function approximation is as follows:  
\[L(c)=\sqrt{(c^{\prime}-c)^{2}+\epsilon^{2}}, \tag{5}\]  
where \(c\) represents the RGB channel of the real shadow image, \(c^{\prime}\) represents the RGB channel of the predicted shadow image synthesized by the predicted shadow mask factor \(\alpha^{\prime}\), \(\epsilon\) represents a small value, and \(10^{-5}\) is taken in this training. The derivative \(\frac{\partial L}{\partial c^{\prime}}\) is easy to calculate, its formula is defined as follows:  
\[\frac{\partial L}{\partial c^{\prime}}=\frac{c^{\prime}-c}{\sqrt{\left(c^{ \prime}-c\right)^{2}+\epsilon^{2}}}. \tag{6}\]  
Overall loss: It is a linear fusion of two losses, namely:  
\[L_{overall}=\lambda L(\beta)+(1-\lambda)L(c)\quad\lambda\in[0,1]. \tag{7}\]  
Here, \(\lambda\) was set to 0.5. Training method: Although the performance increased significantly with the increase of network depth, it was an extraordinary task to train a deeper network due to the problem of vanishing gradient and exploding gradient. In this paper, the following three methods were used to achieve fast convergence and avoid over-fitting:  
1. Staged training. Since the RSnet network structure was a combination of two networks, each network was trained separately in the training process. When the network reached a certain precision, the two were combined and trained together, and finally the combined optimization of the two was realized. 2. Multi-level training. According to the size of the shadow scale factor, different levels of images were set for training, for instance, training hard shadow image databases before soft shadow image databases, and combining the two to form a database for training in the last step. At the same time, considering the difference in pixel size of the user input image, images of different pixel sizes can also be divided into multiple levels for training. All of those methods are more robust in this paper, and help the network to learn context and semantics information better, thus, multi-level training results were finally achieved. 3. Data augmentation. Hundreds of individual scene images cannot achieve good results for the purpose of training a deeper network.