# A Photo Identification Framework to Prevent Copyright Infringement with Manipulations
## 3 Copyright Photo Identification Framework
### Image Verification
#### 3.4.2 Image Similarity Siamese Network  
As mentioned in the previous section, the input image pair is assumed to have a similar geometric structure. Though the images have a similar structure, directly computing the image similarity by using a general pixel-wise metric would result in an inappropriate similarity score due to differences in color or other noise factors. Furthermore, because of the possible noise from misalignment, measuring similarity between images is not accurate. For instance in Figure 8, both the mean structure similarity (SSIM-S) which is structure term of SSIM of different images as in (b) and the same images as in (c) have the same high value (=0.96). To address this problem, we propose a robust image similarity metric based on a deep Siamese CNN model. In addition, to make the proposed model catch the small structural difference between images, we design the input to our CNN model as not simply using image pair but the patches that are sampled around the local minimum of the  
Figure 8: Image alignment between the matched image from the database and its corresponding query image. Images may be taken in slightly different viewpoints (**a**) or pose differences of some objects in the image (**b**). These are considered as different images. On the other hand, the image processed from matched image (**c**) needs to be recognized as the same image. The overlapped region is marked in green. We additionally plot the structure similarity and boxes centered at a local minimum. Note that mean value for (**a**–**c**) are 0.33, 0.96, and 0.96, respectively. SSIM-S. The process outline of the proposed patch-based Siamese CNN model is shown in Figure 9.  
From the geometrically aligned image pair, the image similarity Siamese network predicts their image similarity score \(P_{image}\). If \(P_{image}\) is less than the pre-defined threshold \(\tau_{v}\), it returns a decision that the two input images are different to the framework. Let the patch pair extracted from two images be \((\mathbf{x},\mathbf{y})\). Each patch pair is extracted on the local minimum of the SSIM-S map, which is defined as structure part of SSIM [40]. The loss function is defined as sum of patch similarity loss \(L_{patch}\) on patch pairs \((\mathbf{x},\mathbf{y})\) and image similarity loss on aggregated features \(\mathbf{F}\) (i.e., \(L=L_{\text{P}}+L_{\text{I}}\)). The patch similarity loss is expressed as:  
\[L_{patch}=\sum_{(x,y)\in\text{Patches}}\|P_{patch}(\mathbf{x},\mathbf{y})-P_{ label}(\mathbf{x},\mathbf{y})\|_{2}^{2}, \tag{11}\]  
\[P_{patch}(\mathbf{x},\mathbf{y})=\mathbf{FC_{P}}(|\mathbf{CONV}(\mathbf{x})- \mathbf{CONV}(\mathbf{y})|), \tag{12}\]  
where \(P_{label}(\mathbf{x},\mathbf{y})\), \(\mathbf{CONV}\), and \(\mathbf{FC_{P}}\) are the ground truth label, which is 1 for patches extracted from identical images and 0 otherwise, convolutional neural network for computing patch-wise features, and patch similarity network composed of fully-connected layers, respectively. The last layer of the patch similarity network \(\mathbf{FC_{P}}\) is sigmoid to make output value in \([0,1]\). This loss term is introduced for the network to capture significant local dissimilarity within patch pairs. Meanwhile, the image similarity loss measures the loss in terms of entire image with aggregated feature \(\mathbf{F}\) from intermediate feature vectors of \(\mathbf{FC_{P}}\):  
\[L_{image}=P_{label}(\mathbf{X},\mathbf{Y})\log{(P_{image})}-(1-P_{label}( \mathbf{X},\mathbf{Y}))\log{(1-(P_{image}))}, \tag{13}\]  
\[P_{image}=\mathbf{FC_{A}}(\mathbf{F}), \tag{14}\]  
where \(P_{label}(\mathbf{X},\mathbf{Y})\) and \(\mathbf{FC_{A}}\) are the ground truth label, which is 1 for same images, and image similarity patch aggregated similarity network with fully-connected layers, respectively. To make the image similarity score have value in \([0,1]\), the patch aggregated similarity network \(\mathbf{FC_{A}}\) has a sigmoid layer as an output. In Figure 8a,b, the measured similarity scores using the proposed model are 0.0025 and 0.69.