# Revisiting Low-Resolution Images Retrieval with Attention Mechanism and Contrastive Learning
## 2 Materials and Methods
### Feature Extractor
Let \(a_{p}\) be a joint self-attention weight of a local patch \(x_{p}\), \(p=1,\dots,N^{2}\), then our visual descriptor for an image \(\mathbf{x}\) is the ranking weighted sum of its patch embeddings,  
\[\mathbf{z}(K)=\sum_{i}^{K}a_{i}\mathbf{z}^{i}, \tag{3}\]  
where \(K\) is the desired rank, \(\mathbf{a}=\sigma([a_{1},\dots,a_{N^{2}}])\) is a permutation of the list of joint attention weights, and \(\mathbf{z}=\sigma([z_{1},\dots,z_{N^{2}}])\) is the corresponding list of embedded patches' output from the final encoder and sorted with the same order. When \(\sigma\) is the arrangement from greatest to least, our visual descriptor is the weighted sum of the most \(K\) attentive regions. According to ViT architecture, each encoder block has its own self-attention maps from the multi-head attention. The self-attention mechanism allows ViT to interpret information across the entire image, even in the early layers. In the early layers, some heads consistently focus on small areas, while others attend to most parts of the image, indicating that the ability to unite information globally is already in use inside the early layers. Meanwhile, the attention regions from all heads tend to be wider when going through higher layers, showing that the model aims to capture global information at higher layers. This ability is analogous to the receptive field concept in CNNs, which is the strength of convolutional layers capable of integrating both local and global information. Instead of considering the correlation among patches of an image, we investigate parts of the image that should be attended to and extract their corresponding embedded patches. As highlighted in the equation, the embedded patches are output from the final encoder block, but the attention weights are jointly measured across multi-head attention modules. Let \(\mathbf{A}^{(i)}\in[0,\ 1]^{h\times(N^{2}\times N^{2})}\) be the attention map calculated inside the multi-head attention module of the \(i\)th encoder, where \(h\) is the number of attention heads, and \(N^{2}\) is the number of patches as earlier mentioned. Then, \(\mathbf{A}^{(i)}\) is a collection of attention maps, which are symmetric matrices of size \(N^{2}\times N^{2}\) whose coefficients estimate the degree of attention between two patches. To derive the joint attention map, we first average all attention maps for different heads to obtain an attention map responsible for a layer. To account for residual connections, we add an identity matrix to the attention map and re-normalize the weights,  
\[\mathbf{Ana}^{(i)}=\textit{normalize}\Bigg{(}\frac{1}{h}\sum_{j}^{h}A_{j}^{(i)}+I _{N^{2}\times N^{2}}\Bigg{)}, \tag{4}\]  
where \(\mathbf{Ana}^{(i)}\in[0,\ 1]^{N^{2}\times N^{2}}\) is the normalized average attention map from the \(i\)th layer, \(I_{N^{2}\times N^{2}}\) is an identity matrix of size \(N^{2}\times N^{2}\), and \(\textit{normalize}(\mathbf{X})=\mathbf{X}/\sum_{i,j}X_{ij}\). Finally, the joint attention map is obtained by multiplying the attention maps across all layers. \[\mathbf{Ajoint}=\prod_{i}^{L}\mathbf{Ana}^{(i)}, \tag{5}\]  
where \(\mathbf{Ajoint}\in[0,\ 1]^{N^{2}\times N^{2}}\) and \(L\) is the number of encoder blocks. To generate the visual descriptors, we require only the attention weights assigned to image patches; theseattention weights attend to themselves and can be attained by extracting diagonal of the joint attention matrix \(\mathbf{a}= \begin{bmatrix}a_{1},\dots,a_{N^{2}\times N^{2}}\end{bmatrix} =diagonal(\mathbf{ Ajoint})\). The attention weights derived here are exactly the weights involved in Equation (3). The aforementioned method is referred to as attention rollout and was introduced in [11]. Figure 4 illustrates our approach to extract visual representations from the ViT model.