# Transmission Removal from a Single Glass Scene and Its Application in Photographer Identification
## 3 Proposed Method
### Loss Function
#### 3.3.3 Patch-Level Loss
The Gram matrix function \(\operatorname{Gram}(\mathbf{X},l)\) is defined as a \(C_{l}\times C_{l}\) matrix to characterize feature correlations of \(F^{l}(\mathbf{X})\), and each element in this matrix, denoted as \(\operatorname{Gram}(\mathbf{X},l)_{i,j}\), can be determined as  
\[\operatorname{Gram}(\mathbf{X},l)_{i,j}=\sum_{k=1}^{H_{l}\times W_{l}}F^{l}_{ ik}(\mathbf{X})F^{l}_{jk}(\mathbf{X}),i,j=1,2,\dots,C_{l}, \tag{9}\]  
where \(i\) and \(j\) represents the two channels of the feature map, \(k\) represents the element number in the feature matrix under each channel, and \(F^{l}_{jk}(\mathbf{X})\) denotes the \(k\)-th element of  
Figure 4: The framework of using contextual loss. The VGG19 network is used to optimize the content and style of the resulting images. the \(i\)-th channel of the feature map \(F^{l}(\mathbf{X})\). Then calculate the relevance distance with MSE as follows:  
\[E_{l}(f(\mathbf{R}),\hat{f}(\mathbf{R}))=\frac{\sum_{B}\|\operatorname{Gram}( \hat{f}(\mathbf{R}),l)-\operatorname{Gram}(f(\mathbf{R}),l)\|_{2}}{BH_{l}^{2} W_{l}^{2}}, \tag{10}\]  
where \(B\) represents the batch size reserved in the feature map. \(E_{l}(f(\mathbf{R}),\hat{f}(\mathbf{R}))\) is calculated by the average scoring distance of each element in the Gram matrix, which minimizes \(\operatorname{Gram}(f(\mathbf{R}),l)\) and the estimated \(\operatorname{Gram}(\hat{f}(\mathbf{R}),l)\). Compared with the original method [39], we add the concept of batch calculation, so we only focus on the style transfer and speed up the generator's learning of color domains. The specific style loss is defined as follows:  
\[L_{\text{style}}(f(\mathbf{R}),\hat{f}(\mathbf{R}))=\sum_{l}w_{l}E_{l}(f( \mathbf{R}),\hat{f}(\mathbf{R})),l=1,2,\ldots,5, \tag{11}\]  
where \(w_{l}\) represents the weight factor in each convolution layer, and we select \"Conv1_2\", \"Conv2_2\", \"Conv3_4\", \"Conv4_4\", and \"Conv5_4\" as features in our network, as shown in Figure 4.  
After transmission, the estimated image that only satisfies the visual effect on color is not enough; it should obtain more contextual information about the actual reflection. In the previous processing of content information, perceptual loss [10] is used to constrain the high-level content information and low-level details of the generated image to satisfy the human perception. However, low-level features are enough to express the image's texture, and too many features waste resources [25]. Therefore, instead of perceptual loss, we add content loss based on the pre-trained model VGG19, which is a complete feature extraction network trained under the ImageNet dataset. We use it to constrain the content, and the content loss function is defined as follows:  
\[ \begin{split} L_{\text{content}}(f(\mathbf{R}),\hat{f}(\mathbf{R }))=\sum_{l^{\prime}}w_{l^{\prime}}\Big{\|}F^{l^{\prime}}(f(\mathbf{R}),F^{l^ {\prime}}(\hat{f}(\mathbf{R}))\Big{\|}_{2^{\prime}}\\ l^{\prime}=1,2,3,\end{split}  \tag{12}\]  
where \(w_{l^{\prime}}\) denotes the weight factor, and we use the middle-frequency features with richer textural information \"Conv3_4\", \"Conv4_4\", and high-frequency feature \"Conv5_4\" in our experiments and generate the output through \(L_{2}\) distance. The patch-level loss function is combined as:  
\[L_{\text{patch}}=\lambda_{1}^{\prime}L_{\text{excl}}+\lambda_{2}^{\prime}L_{ \text{style}}+\lambda_{3}^{\prime}L_{\text{content}}, \tag{13}\]  
where \(\lambda_{1}^{\prime}\) to \(\lambda_{3}^{\prime}\) are the weights to balance different terms and set as 9, 3, and 2 in our experiments, respectively. By considering the influence of image gradient, content, and style, the final loss function for the transmission removal network can be expressed as follow:  
\[L_{\text{total}}=\lambda_{1}^{\prime\prime}L_{\text{adv}}+\lambda_{2}^{\prime \prime}L_{\text{global}}+\lambda_{3}^{\prime\prime}L_{\text{patch}}, \tag{14}\]  
where \(\lambda_{1}^{\prime\prime}\) to \(\lambda_{3}^{\prime\prime}\) are the weights to balance different terms and set as 1, 3, and 1 in our experiments, respectively.