## 3 Improve Image Colorization Generation Network
### U-Net Generation Network  
The U-Net network is a model used for semantic segmentation [26]. It is called U-Net because of its U-shaped network structure. The U-Net is symmetrical on the left and right sides, with the left side performing downsampling for feature extraction, and the right side performing upsampling for feature reconstruction. Since infrared images usually contain some weak edge information, the U-Net network is better able to preserve these details in image colorization, resulting in higher image accuracy. Compared to VGG-16 and ResNet, the U-Net network has fewer parameters and trains faster. Moreover, in some infrared image colorization tasks, there may be a large imbalance in the number of pixels of different colors, but the structure of the U-Net network allows it to better handle such unbalanced datasets.  
We compared the performance of U-Net with two other networks for colorizing near-infrared images. As shown in Figure 2, VGG-16 produces images with low saturation and color distortion after colorization. Although ResNet improves upon the colorization results of VGG-16, it still cannot fully restore the colors of the original image. Moreover, the images colorized by U-Net are closest to the real images, with clearer object edges. Therefore, we chose U-Net as the generator and made improvements to it.  
As the number of convolutions increases, each convolutional layer loses some information from the previous layer's feature map. This loss of information can have a significant impact on the quality of the reconstructed image during upsampling. However, the U-Net  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Convolution Layer** & **Convolution Kernel** & **Step Length** & **Number of Channels** \\ \hline Conv1 & 4 \(\times\) 4 & 2 & 64 \\ Conv2 & 4 \(\times\) 4 & 2 & 128 \\ Conv3 & 4 \(\times\) 4 & 2 & 256 \\ Conv4 & 4 \(\times\) 4 & 2 & 512 \\ Conv5 & 4 \(\times\) 4 & 2 & 512 \\ Conv6 & 4 \(\times\) 4 & 2 & 512 \\ Conv7 & 4 \(\times\) 4 & 2 & 512 \\ Conv8 & 4 \(\times\) 4 & 2 & 512 \\ ConvTranspose1 & 4 \(\times\) 4 & 2 & 512 \\ ConvTranspose2 & 4 \(\times\) 4 & 2 & 512 \\ ConvTranspose3 & 4 \(\times\) 4 & 2 & 512 \\ ConvTranspose4 & 4 \(\times\) 4 & 2 & 512 \\ ConvTranspose5 & 4 \(\times\) 4 & 2 & 256 \\ ConvTranspose6 & 4 \(\times\) 4 & 2 & 128 \\ ConvTranspose7 & 4 \(\times\) 4 & 2 & 64 \\ ConvTranspose8 & 4 \(\times\) 4 & 2 & 2 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Generator structure.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Convolution Layer** & **Convolution Kernel** & **Step Length** & **Number of Channels** \\ \hline Conv1 & 4 \(\times\) 4 & 2 & 64 \\ Conv2 & 4 \(\times\) 4 & 2 & 128 \\ Conv3 & 4 \(\times\) 4 & 2 & 256 \\ Conv4 & 4 \(\times\) 4 & 1 & 512 \\ Conv5 & 4 \(\times\) 4 & 1 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Discriminator network structure.  
network utilizes skip connections to concatenate and merge low-level and high-level feature maps, resulting in feature maps that contain both low-level and high-level information. For U-Net, information from each scale is important as it can capture multi-scale feature information and thus improve feature extraction capability. The U-Net structure is shown in Figure 3.