# Revisiting Low-Resolution Images Retrieval with Attention Mechanism and Contrastive Learning
## 1 Introduction
First row: 1 st image is query image from class “mourning warbler”, 2 nd image from class “magnolia warbler”, 3 rd image from class “yellow-throated viroe”, 4 th image from class “white-crowned sparrow”, 5 th image from class “blue-headed viroe”, 6 th image from class “yellow-throated viroe”. Second row: 1 st image is query image from class “Clark nutcracker”, 2 nd image from class “Western Kingbird”, 3 rd image from class “song sparrow”, 4 th image from class “American pipits”, 5 th image from class “blue-headed viroe”, 6 th image from class “scissor-tailed flycatcher”. Third row: 1 st image is query image from class “stapler”, 2 nd image from class “lamp”, 3 rd image from class “cabinet”, 4 th and 5 th images from class “stapler”, 6 th image from class “kattle”. Fourth row: 1 st image is query image from class “lamp”, 2 nd and 3 rd image from class “cabinet”, 4 th image from class “lamp”, 5 th and 6 th images from class “chair”. irrelevant details. Approaches tackling the image retrieval problem include using attention structures introduced in [12; 13; 14; 15]. This study employed a simple contrastive learning framework [16] for image retrieval but investigated an effective architecture for the feature extractor. As a result, we found that the Visual Transformer [11] model is especially effective for visual representation learning. Inspired by Transformer architecture [10] from the natural language processing domain, ViT is proposed as a promising model that naturally integrates a self-attention mechanism to solve computer vision problems. Self-attention is introduced to visual tasks to interpret the correlation among pixels where a high attention score between two visual patches indicates their strong relation and vice versa. Although ViT is not the first method to implement self-attention for the visual task [17; 18; 19], it is remarkable thanks to its overwhelming results and efficiency in hardware accelerators, in addition to its simplified implementation. When pre-trained using large-scale datasets and transferred to multiple recognition benchmarks, ViT outperforms state-of-the-art convolution-based neural networks [11; 20; 21]. However, Transformers lack some inductive biases compared to CNNs, such as translation equivariance, and thus training on sufficient amounts of data is recommended. Otherwise, the self-attention mechanism inherits receptive field properties from CNNs and considers wide regions even from low layers. **Contrastive learning**: Learning visual representation is mostly a label-driven task, where learnable feature extractors are trained to optimize objective functions that involve the label of samples, such as categories and pairs of negative and positive samples. The success of such tasks requires large amounts of labeled data [22; 23; 24], which is not always available and is often very expensive to acquire. However, unsupervised visual representation learning remains an unexploited area in computer vision research. Recently, a considerable research effort has been put into methods to enhance vision systems without providing a large amount of full supervision. In particular, this effort is characterized by advances in self-supervised learning with a contrastive loss function [25; 26; 27]. Self-supervised learning frameworks formulate pretext learning tasks that leverage unlabeled data to learn high-level semantic visual representations useful for the downstream task of interest. For example, pretext tasks such as predicting orientation of rotated image [28], filling in a missing patch [29], or jigsaw re-ordering [30] are beneficial for downstream tasks such as recognition and semantic segmentation because high-level concepts of objects (e.g., shape and texture) are encoded when solving the pretext task. Precisely, self-supervised representation learning's underlying concept is maximization of the mutual information between different views of the data [31; 32; 33]. The main theme of contrastive learning is an instance discrimination task.