# Contrasting Dual Transformer Architectures for Multi-Modal Remote Sensing Image Retrieval
## 2 Materials and Methods
### Language and Vision Representation Encoders  
The language transformer was first introduced for machine translation in 2017 [11]. Since then, more advanced variants have been proposed for text modeling such as Generative Pre-trained Transformer (GTP) [12; 13] and Bidirectional Encoder Representations from Transformers (BERT) [14]. Typically, the transformer encoder is based on the so-called self-attention mechanism, which allows it to capture long-range dependencies. This mechanism makes it a better alternative to recurrent models in modeling long sequences. In our work, we rely on these models in generating the textual feature representations. To this end, each text \(tx_{i}\) is first tokenized into words \(tx_{i}=(w_{1},\ w_{2},\ \dots\ w_{m})\), where \(m\) is the maximum number of tokens. Then, the word embedding layer \(E_{tx}\) converts the words into sequence of features of dimension \(d_{tx}\). A learnable positional embedding is added to the sequence before feeding it into the encoder to provide knowledge of the word sequence. To denote the beginning and end of the sequence, two additional special tokens, CLS and SEP, are added to the input tokens. The final matrix, \(Z_{tx0}\), has the following representation:  
\[Z_{tx0}=[w_{class};\ w_{1}E_{tx};w_{2}E_{tx};\dots;\ w_{m}E_{tx}]+E_{pos} \tag{1}\]  
where \(E_{pos}\in\mathbb{R}^{(m+1)\times d_{tx}}\) and \(w_{class}\) is a special classification token. The final representation \(Z_{txL}\) is created at layer L by feeding the original representation \(Z_{tx0}\) through several identical layers. These encoder layers each start with a tiny multilayer perceptron (MLP) block and a multi-head self-attention (MSA) block. Both blocks are joined together using LayerNorm and skip connections (LN). \[Z^{\prime}_{t\ell}=\text{MSA}(\text{LN}(Z_{t\ell-1}))+Z_{t\ell-1},\,\ell=1\dots L \tag{2}\]  
\[Z_{t\ell}=\text{MLP}\big{(}\text{LN}\big{(}Z^{\prime}_{t\ell}\big{)}\big{)}+Z^{ \prime}_{t\ell},\,\ell=1\dots L \tag{3}\]  
It is worth-recalling, that each input matrix \(Z_{t\ell-1}\) for layer \(\ell\) of the encoder is projected into queries, keys, and values using the weight matrices \(W^{\mathcal{Q}}\), \(W^{\mathcal{K}}\), \(W^{\mathcal{V}}\in\mathbb{R}^{d_{\ell}\times d_{K}}\), such that \(Q=W^{\mathcal{Q}}Z_{t\ell-1}\), \(K=W^{\mathcal{K}}Z_{t\ell-1}\), and \(V=W^{\mathcal{V}}Z_{t\ell-1}\) where \(d_{K}\) is the dimension of the key. The queries, keys, and values are used by the MSA block to generate a weighted sum of token features. In particular, MSA comprises \(h\) multiple independent self-attention heads operating in parallel, each head computes a different attention score using the scaled dot product similarity between the queries, keys, and values expressed by this equation:  
\[Attention(Q,K,V)=softmax\Big{(}\frac{QK^{T}}{\sqrt{d_{K}}}\Big{)}\cdot V \tag{4}\]  
All heads' outputs are combined, and a learnable weights matrix is used to project them to the appropriate dimension \(d_{t}\). The MLP block copmposed of two linear layers with a GELU [15] activation in between. In the vision encoder, the image \(X_{\text{im}}\) is reshaped into a sequence of \(n\) non-overlapping patches \(X_{\text{im}}=(x_{1},x_{2},\dots\ x_{n})\). Each patch in the resulting sequence has the size of \(c\times p\times p\) where \(c\) is the number of channels in the image and \(p\) is the patch size (e.g., \(p=16\) or \(32\)). Figure 2: Overview of the proposed text-image aligning model. To produce L2-normlized visual \(\{f_{qi}\}_{i=1}^{b}\) and textual \(\{f_{ri}\}_{i=1}^{b}\) features, we sample a mini-batch of b images-text pairings for training the network and feed them to the vision and language transformer encoders. Then, we generate a similarity matrix of size \(b\times b\) by calculating the similarity between all potential visual and textual pairs in the mini-batch. These patches are then flattened and mapped to a sequence of embeddings of the model dimension \(d_{v}\) with a linear embedding layer \(E_{v}\in\mathbb{R}^{(p^{x}_{c})\times d_{v}}\).