# UAV Image Multi-Labeling with Data-Efficient Transformers
## 3 Methodology
### Encoder Module  
The encoder was adopted from DeiT architecture. It is an improved version of ViT but with the advantage of requiring less data for training. To feed the images into the model, the data are first converted into a sequence of patches. An image \(X\) of dimension \(h\times w\times c\) is divided into small patches, where \(h\), \(w\), and \(c\) are the height, the width, and the number of channels of that image, respectively. The patches \(X=\left\{x_{p}^{1},\ x_{p}^{2},\ \cdots,\ x_{p}^{m}\right\}\) form a sequence of length \(m\), where each patch \(x_{p}\) has the dimension of \(p\times p\) and \(m=h\times w\ /\ p^{2}\). These patches are analogous to word tokens in the original transformer. The flattened image patches are converted into embeddings by feeding them into a linear embedding layer \(E\) to match their dimension to the model dimension \(d_{model}\). Figure 1: Proposed multi-labeling method. Flattening causes the loss of positional information, which is crucial for understanding the image content. To retain this information, each patch embedding is added to its corresponding positional information. The resultant position-aware embeddings are appended with a learnable class token \(x_{class}\). The DeiT architecture also introduces another distillation token \(x_{distil}\) that is appended along with the class token \(x_{class}\) to the patch embeddings (Equation (1)). The two tokens and the patch embeddings interact with each other via the self-attention mechanism. \[z_{0}=\Big{[}x_{class};\ x_{distil};\ x_{p}^{1}E;\ x_{p}^{2}E;\dots;\ x_{p}^{m}E \Big{]}+E_{pos},\ \ E\in\mathbb{R}^{(p^{2}.\ \ c)\times d_{model}},\ E_{pos}\in \mathbb{R}^{(m+2)\times d_{model}} \tag{1}\]  
The encoder consists of a stack of \(L\) identical layers; each one is composed of two main blocks: A multi-head self-attention (MSA) block, which is the key component of the model, and a feed-forward network block (FFN). The MSA utilizes the self-attention mechanism to derive long-range dependencies between different patches in the given image. Equation (2). Shows the details of the computations that take place in one self-attention head (SA). First, the input sequence is transformed into three different matrices which are the key \(K\), the query \(Q\), and the value \(V\) using three linear layers \(W_{i}^{Q}\in\mathbb{R}^{d_{model}\times d_{Q}}\), \(W_{i}^{K}\in\mathbb{R}^{d_{model}\times d_{K}}\) and \(W_{i}^{V}\in\mathbb{R}^{d_{model}\times d_{V}}\) for \(i=1,\ 2,\ \dots h\) where \(h\) is the number of heads. The attention map is computed by matching the query matrix against the key matrix using the scaled-dot-product. The output is scaled by the dimension of the key \(d_{K}\) and then converted into probabilities by a softmax layer. Finally, the result is multiplied with the value \(V\) to get a filtered value matrix which assigns high focus to more important elements. \[Attention(Q,\ K,V)=softmax\Big{(}\frac{QK^{T}}{\sqrt{d_{K}}}\Big{)}.V \tag{2}\]  
The multi-head self-attention applies this process in parallel using multiple self-attention heads, as shown in Figure 2. Each head has the role to focus on one relationship among the image patches. The outputs of all heads are then concatenated together and passed to a linear layer to project it to the desired dimension, as shown in the following equation:  
\[ \begin{split} MSA(Q,\ K,V)=Concat(SA_{1};\ SA_{2};\dots\ SA_{h})W ^{O},W^{O}\in\mathbb{R}^{h.d_{K}\times d_{model}}\\ SA_{i}=Attention\Big{(}QW_{i}^{Q},\ KW_{i}^{K},\ WW_{i}^{V}\Big{)} \end{split}  \tag{3}\]  
where \(W^{O}\) represents the final projection layer and \(W^{O},\ W_{i}^{Q},W_{i}^{K}\), and \(W_{i}^{V}\) are all learnable weights. The second main block in the encoder is the feed-forward network (FFN) that is applied after the MSA block. It consists of two fully connected layers with a GELU activation function [33] within them.