# A Realistic Hand Image Composition Method for Palmprint ROI Embedding Attack
## 3 Methodology
### ROI Harmonization  
Given potential hue, illumination, and skin color disparities between \(R_{Atk}\) and carrier hand image \(C\), ROI harmonization becomes imperative to mitigate these differences. Moreover, for computational efficiency in matching and aligning with the grayscale nature of ROI images, \(R_{Atk}\) is converted to a grayscale image. Specifically, we leverage the pretrained Contrastive Arbitrary Style Transfer (CAST) proposed in [4] to achieve ROI harmonization. As shown in Figure 5, CAST employs cycle generation to train encoders and decoders, adeptly separating and merging style and content. However, training a model that accommodates the significant diversity within hand images across various datasets proves challenging. We devise an iterative style transfer method based on pretrained CAST to initialize and optimize image quality for individual palms. This iterative process resembles the training process of CAST but focuses on optimizing the image quality rather than the model ability, which enables the synthesis of realistic images through limited iterations. Figure 4: The pipeline of ROI embedding. Specifically, let \(A\) and \(B\) symbolize \(R_{Atk}\) and \(R_{Ref}\), respectively. The notation \(A|B\) and \(B|A\) denote \(A\) or \(B\) and \(B\) or \(A\), respectively. The ROI harmonization process commences by employing the encoder \(E(\cdot,\cdot)\) to amalgamate the content of \(A|B\) and the style of \(B|A\) from the two input images, \(A\) and \(B\). Subsequently, the decoder operates on \(E(A|B,B|A)\) to generate the harmonized image \((A|B)_{f}\) based on the content of \(A|B\) and the style of \(B|A\). Once \((A|B)_{f}\) is obtained, it is combined with \(A|B\) to yield a reconstructed image \((A|B)_{r}\). The final harmonized image is refined by minimizing the difference between \(A\) and \(A_{r}\) and \(B\) and \(B_{r}\). Within our proposed ROI harmonization framework, the loss function is essential in enhancing both image quality and harmonization effectiveness. Notably, the loss function employed in CAST solely incorporates the \(\mathcal{L}_{1}\) loss and assigns equal weights to \(\mathcal{L}_{cycA}\) and \(\mathcal{L}_{cycB}\). However, this approach proves insufficient for attaining optimal image quality and harmonization effects. Furthermore, preserving texture information in \(R_{Atk}\) is critical, necessitating measures to minimize interference from \(R_{Ref}\). Consequently, essential and imperative modifications must be made to the loss function to address these considerations. Building upon the existing \(\mathcal{L}_{1}\) loss, the cycle loss is improved by introducing two loss functions, namely \(\mathcal{L}_{2}\) and \(\mathcal{L}_{lpips}\)[28], to enhance the preservation and separation of content and style information, in which the overall image visual quality can be improved. To further preserve texture information in synthetic images, a texture loss \(\mathcal{L}_{T}\) is devised. This involves utilizing a separate texture feature extractor, \(f_{t}()\) which is composed of two Gabor convolution layers and a leakyReLU function for feature extraction. For a comprehensive feature extraction, a six-direction Gabor kernel is employed. Following the acquisition of texture features, \(\mathcal{L}_{1}\) is employed to calculate the differences between input images and reconstructed images. The \(\mathcal{L}_{T}\) is the average of the six Gabor feature losses, as depicted in Equation (2). \[f_{t}\left(x,g_{i}\right)=\varphi_{LR}\left(Conv(Conv(x,g_{i}),g_{i})\right) \tag{1}\]  
\[\mathcal{L}_{T}(x,y)=\frac{\sum_{i=1}^{6}\mathcal{L}_{1}(f_{t}(x,g_{i}),\ f_{t}(y,g_{i}))}{6} \tag{2}\]  
where \(Conv\) denotes convolution, \(\varphi_{LR}\) is the LeakyReLU function, and \(g_{i}\) is the Gabor convolution kernel toward \((i-1)\times 30\) degrees. The final modified cycle loss is Equation (4), which constitutes \(\mathcal{L}_{cycA}\) and \(\mathcal{L}_{cycB}\).