# Revisiting Low-Resolution Images Retrieval with Attention Mechanism and Contrastive Learning
## 2 Materials and Methods
### Contrastive Learning Framework  
We train projection heads on top of the ViT model in a contrastive way to maximize the agreement of positive pairs in an embedding space. The contrastive learning framework can be decomposed into four modules: data augmentation, backbone network, projection head, and contrastive objective function. **Data augmentation**: This study concentrates on low-resolution image retrieval, and thus the augmentation method we applied here only varies the input's resolution. A stochastic data augmentation module transforms the input data randomly, resulting in two correlated views of the same sample but with different resolutions. In this work, we apply random cropping followed by a low-pass filter. The low-pass filter used here is the Gaussian blur with a fixed size of the kernel and random kernel standard deviation. **Base encoder**: The input to the base encoder network is augmented data, and output is a representation vector. We mainly use ViT as our base encoder model; however, we also experiment with other encoder networks, such as BiT [20] and EfficientNet [21]. In the case of BiT and EfficientNet encoders, we extract the final convolutional layer before fully connected layers and then apply adaptive average pooling to obtain a 1-D representation vector. This study uses \(\mathbf{z}=Encoder(\mathbf{x})\) as a notation for the representation vector. As a result, the dimension of the representation vector varies based on encoder architectures. The base encoder is visualized as two \"Encoder\" blocks as in Figure 2.  
**Projection head**: The projection head consists of a non-linear mapping that maps representation vectors into an embedded space where the similarity between samples can be measured. As suggested in [16], we use an MLP with one hidden layer to formulate the projection head. This study uses \(\mathbf{e}=Project(\mathbf{z})\) as a notation for the embedded vector. The projection head is depicted in Figure 2 as \"Projector\" blocks with a hidden layer. The size of the hidden layer is the same as the number of dimensions of embedded vectors. **Contrastive objective function**: A contrastive function is defined so that minimizing it results in maximizing the agreement between positive pairs; in other words, it is meant to pull similar samples closer to each other and push dissimilar samples in the opposite direction. We measure the similarity within a minibatch of \(N\) random samples. Following  
Figure 4: Extracting attention representation from Visual Transformer model: The output of the final encoder module of the ViT model includes \(N\times N\) patch’s embeddings and \(N^{2}\times N^{2}\) attention weights. The attention representation of an image is calculated by summation of the patches’ embeddings weighted by the attention scores. the setting of the data augmentation module, two different resolution versions of the original input are generated inside a minibatch, resulting in \(2N\) data points. Within a multiview minibatch, let \(i\in I\equiv\{1,2,\ldots,2N\}\) be the index of an arbitrary augmented sample, and let \(j(i)\in I\) be the index of the other augmented sample obtained from the same source sample. In self-supervised contrastive learning [16], the loss function for a positive pair of examples \((i,j)\) is defined as  
\[L^{self}=\sum_{i\in I}L^{self}_{i}=-\sum_{i\in I}\log\frac{\exp\Bigl{(}sim \left(e_{i},e_{i(i)}\right)/\tau\Bigr{)}}{\sum_{k=1}^{2N}\mathbb{I}_{\left[k \
eq i\right]}\exp\bigl{(}sim(e_{i},e_{k})/\tau\bigr{)}}, \tag{6}\]  
where \(e_{i}\) is embedding obtained from the projection head, and \(\tau\) is the temperature scaling factor. This loss is infoNCE loss that maximizes a lower bound on mutual information of two observations. Regarding the presence of labels, supervised contrastive losses [37] can be used and also be generalized to an arbitrary number of positive pairs.