# Revisiting Low-Resolution Images Retrieval with Attention Mechanism and Contrastive Learning
## 2 Materials and Methods
### Feature Extractor  
We manipulate the Visual Transformer [11] model to extract a visual representation of given images. The main component of the ViT model is the self-attention encoder module, which implements Transformer architecture [10] in the most standard way. According to the original version of ViT, our feature extractor involves three main steps. **Patch embedding**: We split an image into a sequence of patches and map each patch to a \(D\) dimensions embedding space. Precisely, we put an image \(x\in\mathbb{R}^{C\times W\times H}\) through \(D\) 2d convolutions with the kernel size of \(P\times P\) and stride of \(P\), resulting in a feature map with the size \(D\times N\times N\), then flattening the feature map into a sequence of \(N^{2}\) latent vectors with a constant size of \(D\). In the above configuration, \(N^{2}=H\times W/P^{2}\) is the number of embedded patches, where \((H,\,W)\) represents the resolution of the original image, and \((P,\,P)\) is the resolution of an image patch. Apart from the embedded patches, the ViT model adds an extra learnable class embedding for classification tasks, and the results obtained from using this class token are referred to as \"ViT-class\" in our study. To maintain the position of the patches after flattening, we follow the standard way by adding a learnable 1D positional embedding into each patch, and this positional embedding does not share weights across patches. **Encoder**. Encoder is a computational block consisting of a multi-head attention module [10] and an MLP with two consecutive linear layers. Input and output of encoder module are both embedded vectors of batches. LayerNorm is applied before feeding embedded vectors into the attention module and MLP. The multi-head attention module expands the model's ability to jointly focus on different positions, thus providing different representation subspaces of pair \((key\,\,\textbf{\emph{K}},\,query\,\,\textbf{\emph{Q}},value\,\,\textbf{ \emph{V}})\) from different attention heads:  
\[multihead=Concat(head_{1},\,\,\ldots,head_{h}), \tag{1}\]  
where each head is a context vector from scale dot-product attention. \[head_{i}=softmax\left(\frac{\textbf{\emph{Q}}_{i}\textbf{\emph{K}}_{i}^{T}} {\sqrt{d}}\right)\textbf{\emph{V}}_{i}, \tag{2}\]  
_Q, K_, and V represent a query, key, and value, respectively, calculated inside the transformer architecture that encodes information from the image's patches with the self-attention mechanism to mutually attend to each other. \(d\) is the dimensions of patch embeddings; Equation (2) employs \(d\) to scale the attention scores. The encoder module is illustrated in Figure 3.  
Figure 3: Transformer encoder: the module is combined with three key processes: normalization, multi-head attention, and non-linear mapping (MLP). **Visual descriptor**: We develop a novel visual descriptor for the corresponding image based on the attention mechanism. Our technique naturally arises from attention maps of the ViT model. We generate visual descriptors by combining the final embeddings of informative regions selected by ranking attention weights. Given an image \(\mathbf{x}\in\mathbb{R}^{C\times W\times H}\), let \(\mathbf{z}^{(i)}=\left[z_{1}^{(i)},z_{2}^{(i)},\dots,z_{N^{2}}^{(i)}\right]\) be the list of embedded patches resulting from the \(i^{th}\) encoder. Obviously, the first list of embedded patches \(\mathbf{z}^{(0)}\) is spawned from the summation of linear projections and positional embedding as described above, \(\mathbf{z}_{0}=\left[\textit{embed}(x_{1})+\textit{position\_embed}(x_{1}),\ \dots,\textit{embed}(x_{N^{2}})+\textit{ position\_embed}(x_{N^{2}})\right]\). The \(i\)th embedding layer is the output from the \(i\)th encoder module, while input is the \((i-1)\)th embedding layer \(\mathbf{z}^{i}=\textit{encoder}\big{(}\mathbf{z}^{i-1}\big{)}\).