# Patch-Wise Infrared and Visible Image Fusion Using Spatial Adaptive Weights
## 3 Proposed Method
#### Loss Function  
Loss function plays a principle role in the learning of any model. In usual GAN, the discriminator is trained on the real and the generated images but the generator is trained indirectly via discriminator. In infrared and visible image fusion methods, the dual purposed generator is not only tasked to fool the discriminator but it should also keep the correspondence between the generated image and the source images. This is supervised by the loss function of \(G\). Each discriminator is first trained with the input patches (i.e.,  
Figure 1: The overall architecture of our proposed method.  
the visible image patches in \(D_{v}\) and the infrared image patches in \(D_{i}\)) and then the fused patches. The total loss of each discriminator is the sum of both the losses which aims to discriminate between the fused image by \(G\) and the source images.  
For the discriminators, the patches from the source images are real data while the patches from the generated/fused images are fake data. The discriminator \(D_{i}\) is first trained on the loss between the patch from input infrared image and the real label vector with each value 1. Then, this \(D_{i}\) is trained on the loss between the patch from fused image and the fake label vector with each value 0. Similarly, the discriminator \(D_{v}\) learns the loss between the patch from input visible image and the real label vector. After this, this \(D_{v}\) is trained on the loss between the patch from fused image and the fake label vector. We are using the mean square error loss for the discriminators.  
Our purpose is to train the network in such a way that it gives more weight to the high activation region of infrared images which contains the significant information. For this, we define a new loss function for each patch by infusing the weights in the loss. We propose two methods of doing this; one is to use this new weighted loss only in the discriminator \(D_{i}\) as \(L_{D_{i}}^{*}\), and the other method is to use it in the discriminator \(D_{i}\) as \(L_{D_{i}}^{*}\) and generator as \(L_{D_{i}}^{adv*}\) simultaneously. Generally, the loss of the discriminator is defined as follows:  
\[L_{D}=\min_{D}V(D)=\frac{1}{2}\mathbb{E}[(D(x)-a)^{2}]+\frac{1}{2}\mathbb{E}[ (D(G(x))-b)^{2}] \tag{1}\]  
where \(D(x)\) and \(D(G(x))\) is the output of discriminator given the image \(x\) and the generated image \(G(x)\), respectively. '\(a\)' is the target label vector which is 1 in the case of source images and '\(b\)' is the target label vector which is 0 in the case of a fused image.  
Using the new loss in \(D_{i}\), the losses of both the discriminators become  
\[L_{D_{i}}^{*}=\min_{D}V(D_{i})=\frac{1}{2}\mathbb{E}[W*(D_{i}(i)-a)^{2}]+\frac {1}{2}\mathbb{E}[W*(D_{i}(f)-b)^{2}] \tag{2}\]  
\[L_{D_{v}}=\min_{D}V(D_{v})=\frac{1}{2}\mathbb{E}[(D_{v}(v)-a)^{2}]+\frac{1}{2} \mathbb{E}[(D_{v}(f)-b)^{2}] \tag{3}\]  
The loss function of the generator consists of the content loss and the adversarial loss:  
\[L_{G}=L_{con}+L_{adv} \tag{4}\]  
\(L_{ssim}\) and \(L_{mse}\) are the structural similarity loss and the mean square error loss, respectively, between the input images and the generated/fused image. They are used as the content loss.  
\[L_{con}=L_{ssim}+L_{mse} \tag{5}\]  
\(L_{D_{i}}^{adv}\) and \(L_{D_{v}}^{adv}\) are the adversarial losses provided by the discriminators \(D_{i}\) and \(D_{v}\). Here, mean square error loss is used as an adversarial loss. So the total loss of \(G\) becomes  
\[L_{Total}=L_{ssim}+L_{mse}+\gamma L_{D_{i}}^{adv}+\lambda L_{D_{v}}^{adv} \tag{6}\]  
In the case of using the new loss in the generator, we replace the mean square error loss \(L_{D_{i}}^{adv}\) with the new loss \(L_{D_{i}}^{adv*}\). Here, the output of the discriminator is compared with real label only.  
\[L_{D_{i}}^{adv*}=\min_{G}V(G)=\frac{1}{2}\mathbb{E}[W*(D_{i}-a)^{2}] \tag{7}\]  
\[L_{D_{v}}^{adv}=\min_{G}V(G)=\frac{1}{2}\mathbb{E}[(D_{v}-a)^{2}] \tag{8}\]