# Deep Feature Learning with Manifold Embedding for Robust Image Retrieval
## 5 Siamese Feature Learning
### Objective and Optimization  
The objective function consists of two parts, i.e., the similarity embedding part and the feature consistency preserving part. The similarity embedding part constrains the network from needing  
Figure 2: The network architecture of the proposed network. A pair of images are fed into two fully-convolutional networks (FCNs) separately to generate MAC representations. These two FCNs are identical in architecture and share parameters. The generated MACs and their corresponding pre-extracted MACs (with original network parameters) are then sent to calculate the loss. The loss consists of a similarity embedding part to embed manifold information into network parameters and a consistency preserving part to prevent dramatic modification of model parameters.  
to embed similarity information carried by input image pairs so that the learned MAC feature distance of the same pair should be small in Euclidean space. With this part, manifold information is transferred into fine-tuned feature embedding. The feature consistency preserving part constrains the learned feature embedding from changing the original feature embedding dramatically, preventing the similarity context of unpaired images from corruption. Without this part, the fine-tuning process will be devoted to fitting those similar image pairs. In addition, this is one of the reasons that we do not strictly need negative pairs in this work, which will be explained in detail later.  
The loss function for input image pair (\(\mathbf{I}^{(1)}\), \(\mathbf{I}^{(2)}\)) is defined as follows:  
\[\mathcal{L}(\mathbf{I}^{(1)},\mathbf{I}^{(2)})=\parallel\mathbf{f}^{(1)}- \mathbf{f}^{(2)}\parallel^{2}+\beta\sum_{i=1}^{2}\parallel\mathbf{f}^{(i)}- \widetilde{\mathbf{f}}^{(i)}\parallel^{2}, \tag{3}\]  
where \(\beta\) is a balance factor, \(\mathbf{f}^{(1)}\) and \(\mathbf{f}^{(2)}\) are MACs generated by those two FCNs and \(\widetilde{\mathbf{f}}^{(1)}\) and \(\widetilde{\mathbf{f}}^{(2)}\) are pre-extracted MACs for image \(\mathbf{I}^{(1)}\) and \(\mathbf{I}^{(2)}\), respectively. Minimizing the first term means smaller Euclidean distance in the newly learned embedding space for selected image pairs. However, this will make the second term, i.e., the Euclidean distance of features between the newly learned embedding and the original embedding space enlarged. A proper balance factor can ensure efficient similarity embedding, while the correct similarity relationship of the original feature embedding is maximally kept.  
Note that, in training, we do not involve negative image pairs. In this work, our goal is to embed similarity information, especially nearest neighbor information learned by manifold learning algorithms, into a new feature embedding. Since such information is carried by similar pairs or positive pairs, there is no need to train our model with additional negative pairs. In addition, the feature consistency preserving part of the learning objective ensures that the fine-tuning process does not modify the original model dramatically. As a consequence, the similarity relationships of negative pairs are not corrupted after fine-tuning because they are embedded in the model as parameters. For the above two reasons, negative pairs are not strictly needed in our proposed method.