# Transformer-Based Distillation Hash Learning for Image Retrieval
## 3 Method
### Transformer Teacher Module and Student Module  
**Query update:** The visual features \(\digamma_{0}\) of the image obtained from the above backbone are then input into the decoder of Transformer as key, value, and query. The image's spatial features are calculated using a multi-layer Transformer for self-attention. The input image is divided into \(d\) embedding vectors, and each embedding vector obtains a one-dimensional position encoding token corresponding to its position and is used as input to the decoder regularly. The prior position is merged by adding a learnable one-dimensional position encoding code to the input embedding vectors. An extra learnable CLS token is added to the input embedding vector to represent its correspondence to the output token as a global concept. The Transformer consists of \(L\) layers, each layer consisting of two main modules: a multi-headed self-attentive (MSA) layer, which applies the self-attentive operation to different embedding vectors of the inputs, and a feed-forward network (FFN). Both the MSA and FFN layers are preceded by a normalization layer, and followed by a skip connection layer. The query of the \((i)\)-th decoder layer \(Q_{i}\) is updated based on the output of its previous layer \(Q_{i-1}\) as follows:  
\[Self-Attention:Q_{i}^{{}^{\prime}}=Q_{i-1}^{{}^{\prime}}+MultiHead(\widetilde {Q}_{i-1},\widetilde{Q}_{i-1},Q_{i-1}) \tag{2}\]  
\[FFN:Q_{i}=FFN(Q_{i}^{{}^{\prime}}) \tag{3}\]  
where \(\widetilde{Q}_{i-1}\) denotes the embedding vectors with the position encoding of outputs at layer \((i-1)\)-th, \(Q_{i}^{{}^{\prime}}\) is the intermediate variable, \(MultiHead(query,key,value)\) and \(FNN(x)\) are the multi-headed attention mechanism and feed-forward network, respectively.  
**Pooling:** We need to extract a compact code that globally describes the image. In our reference pooling approach, we directly treat the output of the CLS embedding as a global image descriptor.  
**Reduction and binarization:** After obtaining the global image descriptors, we further perform dimensionality reduction and binarization to improve the retrieval speed. Specifically, we use a combination of a linear projection layer and a Sigmoid function to map the global image descriptors through the fully connected layer to a smaller size \(b\) bit codes and project the logit values to \([0,1]\) using the Sigmoid function:  
\[\mathbf{\hat{h}}=Sigmoid(\mathbf{W}_{k}^{T}\mathbf{z}_{cls}+\mathbf{b}_{k}) \tag{4}\]  
where \(\mathbf{W}_{k}^{T}\) and \(\mathbf{b}_{k}\) are the parameters of the linear projection layer and \(\mathbf{\hat{h}}\) is the pseudo hash code. It is worth mentioning that since the symbolic function sgn() is not derivable, the pseudo hash codes generated in the training phase are real-valued, while in the inference phase, the binary hash codes are generated by the following formula:  
\[\mathbf{h}=\frac{1}{2}(sgn(\mathbf{\hat{h}}-0.5)+1) \tag{5}\]  
Figure 2: A brief framework diagram of ResNet-50.  
where \(\mathbf{h}\) denotes the binary hash code.  
**Student module:** The designed student model is straightforward. We directly remove the Transformer part of the teacher model. The feature map generated from the backbone will go through two fully-connected layers and output the same size as the teacher model.