# ColorMedGAN: A Semantic Colorization Framework for Medical Images
## 2 Methodology
### ColorMedGAN Architecture
#### 2.1.2 Attention Guided Module  
Inspired by [44], we introduce an attention-guided generator that leverages edge semantic information to enhance the feature representation in the hierarchy. The attentional fusion structure is depicted in Figure 3a. We explore two feature fusion approaches (Figure 3) to transfer more efficient semantic information from the edge feature map \(F_{e}\) to the image feature map \(F_{i}\). One is to concatenate the edge features with the colorization network features and input \(\text{concat}(F_{e},F_{i})\) to the encoder at the next layer; the other is to use the attention-guided approach to input the edge features to the activation layer and then multiply them per wise with the feature map from the generator. The latter approach is experimentally proven to be superior as shown in Table 1. The edge feature map is first processed by the sigmoid activation function to generate the corresponding attention map \(F_{a}=\sigma(F_{e})\). Then, we multiply the generated attention map with the corresponding image feature map to obtain a refined map containing local structure and details, which is further inputted into the next convolutional layer. The feature maps of the two fusion modes are presented in Figure 4. The right figure shows that our attention mode outperforms the concatenate mode as it preserves more information from the segmentation and reduces noise. The generated images (comparison of Figure 5c and Figure 5d) also demonstrate the superiority of the sigmoid mode  
\[F_{i}^{j}=\text{Sigmoid}(F_{e}^{j})\times F_{i}^{j} \tag{1}\]  
where \(F_{e}^{j}\) denotes the feature map of the segmentation network, \(F_{i}^{j}\) denotes the feature map of the generator.  
Then, we perform another convolution operation on \(F_{0}\) to obtain a feature map \(F_{0}\in R(C+N+3+3)\times H\times W\) to enhance the representativeness of this feature. In addition, \(F_{0}\) is identical to the original input \(F\). This makes the module flexible and inserted into other architectural structures without modifying other parts to refine the output. Finally, we perform a module with the same output as the module.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **SSMI \(\uparrow\)** & **PSNR \(\uparrow\)** & **FID \(\downarrow\)** \\ \hline Concat mode & 0.896 & **33.325** & 94.59 \\
**Sigmoid mode** & **0.901** & 33.224 & **84.81** \\ \hline \hline \end{tabular} Bold fonts indicate better results.  
\end{table}
Table 1: Comparison of Concatenate and Sigmoid mode.  
Figure 4: The feature map of two fusion modes. The left one presents the concatenate mode, the right one presents the sigmoid mode.  
Figure 5: Comparison of the influence of semantic feature. (**a**) origin image; (**b**) without semantic information; (**c**) with semantic information in concatenate mode; (**d**) with semantic information in sigmoid mode.  
Figure 3: Two modes of feature fusion: (**a**) element-wise multiplication and Sigmoid activation function (**b**) channel-wise concatenate.  
the feature map \(F_{0}\) is fed into the convolutional layer followed by the \(\tanh()\) nonlinear activation layer to obtain the final result, allowing the generator to converge faster. The proposed semantic preservation module enhances the representational power of the model by adaptively recalibrating the semantic class-dependent feature maps and is in a similar spirit to style transfer and the recent works SENet (Squeeze-and-Excitation Networks) [45] and EncNet (Context Encoding Network) [46]. An intuitive example of this module's utility is in generating small object classes, which are easily lost in the generated results due to the loss of spatial resolution. Our scaling factor can emphasize small objects and help preserve them.