## 3 Proposed Method
### Loss Function  
Our entire network is based on GAN. In an image-to-image task, the objective function of the GAN is:  
\[ \begin{split}\min_{\theta}\,\max_{\eta}\,L\Big{(}F_{\text{G}}^{ \theta},& F_{\text{D}}^{\eta}\Big{)}=E_{\text{X, Y \sim p(x,y)}}\Big{[}\log F_{\text{D}}^{\eta}(\text{X, Y})\Big{]}\\ +& E_{\text{X}\sim p(x)}\Big{[}\log\Big{(}1-F_{\text{D }}^{\eta}\big{(}F_{\text{G}}^{\theta}(\text{X})\big{)}\Big{)}\Big{]}\end{split} \,, \tag{11}\]  
where \(\theta\) represents the parameters of the rain streaks removal network, and \(\eta\) represents parameters related to the H-G discriminator. The loss of the generator part is as follows:  
\[L_{\text{gen}}=\log\Big{(}1-F_{\text{D}}^{\eta}\Big{(}F_{\text{G}}^{\theta}( \text{X})\Big{)}\Big{)}. \tag{12}\]  
Additionally, the loss of the discriminator part as follows:  
\[L_{\text{dis}}=-\Big{[}\log\Big{(}F_{\text{D}}^{\eta}(\text{Y})\Big{)}+\log \Big{(}1-F_{\text{D}}^{\eta}\big{(}\overline{\text{Y}}\big{)}\Big{)}\Big{]}. \tag{13}\]  
Training the network to remove rain streaks from the image that relied only on adversarial loss was often unable to achieve a satisfactory effect, and the output image was fuzzy. Therefore, we used an additional loss to further constrain the output image to improve the effect of the removal of the streaks. Given X, the difference in pixels between the predicted image \(\overline{\text{Y}}\) and the truth image Y is minimal and is recorded as pixel-pixel loss \(L_{\text{pix}}\). It can be expressed by the Frobenius norm of the matrix:  
\[L_{\text{pix}}=\frac{1}{\text{N}}\sum_{i=1}^{\text{N}}\|\text{Y}^{(i)}- \overline{\text{Y}}^{(i)}\|_{\text{F}}, \tag{14}\]  
where N represents the batch size used for training. The SSIM indicates the degree of similarity between images, and it can evaluate the quality of the image obtained after the removal of rain streaks. Its value is in the interval \([0,1]\). The larger the value the closer the generated image is to the ground truth image. It is expressed as \(F_{\text{SSIM}}\). As our network first generates the residual image and then generates the final derained image, we combined the SSIM of the residual image and the de-rained image as the total loss of SSIM, i.e.,  
\[L_{\text{SSIM}}= \begin{bmatrix}1-F_{\text{SSIM}}\big{(}\overline{\text{Y}}, \text{Y}\big{)}\end{bmatrix} + \begin{bmatrix}1-F_{\text{SSIM}}\big{(}\overline {\text{S}},\text{S}\big{)}\end{bmatrix} \\ =2-F_{\text{SSIM}}\big{(}\overline{\text{Y}},\text{Y}\big{)}-F_{\text{SSIM}} \big{(}\overline{\text{S}},\text{S}\big{)}\end{bmatrix}\,. \tag{15}\] \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**No.** & **Layer** & **K 1** & **ST 2** & **In-Out Channel** & **BN** & **A 3** & **F 4** & **Output** \\ \hline
1-1 & Conv & \(5\times 5\) & \(1\) & 3-16 & Yes & ReLU & I 5  & \(320\times 480\times 16\) \\
1-2 & Conv 6  & \(3\times 3\) & \(1\) & 3-16 & Yes & ReLU & I & \(320\times 480\times 16\) \\
2-1 & Conv & \(3\times 3\) & \(1\) & 16-16 & Yes & ReLU & I & \(320\times 480\times 16\) \\
2-2 & Max-Pool 7  & \(2\times 2\) & 2 & 16-16 & No & / & 1/4 & \(160\times 240\times 16\) \\
3-1 & Conv & \(3\times 3\) & \(1\) & 16-32 & Yes & ReLU & 1/4 & \(160\times 240\times 32\) \\
3-2 & Max-Pool & \(2\times 2\) & 2 & 32-32 & No & / & 1/16 & \(80\times 120\times 32\) \\
4-1 & Conv & \(3\times 3\) & \(1\) & 32-64 & Yes & ReLU & 1/16 & \(80\times 120\times 64\) \\
4-2 & Max-Pool & \(2\times 2\) & 2 & 64-64 & No & / & 1/64 & \(40\times 60\times 64\) \\
5-1 & Conv & \(3\times 3\) & \(1\) & 64-128 & Yes & ReLU & 1/64 & \(40\times 60\times 28\) \\
5-2 & Max-Pool & \(2\times 2\) & 2 & 128-128 & No & / & 1/256 & \(20\times 30\times 64\) \\
6 & FC & / & / & 128-1 & No & Sigmoid & 1 & \\ \hline \hline \end{tabular}  
* K = kernel_size, 2 ST = stride, 3 A = activation function, 4 F = feature map, 5 I = input image, 6 Conv = convolutional layer, 7 Max-Pool = Maximum Pooling.  
\end{table} 
Table 2: The specific structure of H-G Discriminator.