# Contrasting Dual Transformer Architectures for Multi-Modal Remote Sensing Image Retrieval
## 3 Experimental Results and Discussion  
### Dataset Description  
In the experiments, we evaluated the proposed retrieval method on four benchmark datasets, TextRS [21], Merced [22], Sydney [22], and RSICD [23]. In the following, we provide more details about these RS text-image datasets.  
TextRS caption dataset: This dataset was introduced recently by collecting images randomly from four different scene classification datasets with different image sizes and spatial resolutions. Namely, AID [24] which is composed of 30 classes, PatternNet [25] which contains 38 classes, UC-Merced [26] which has 21 classes, and finally NWPU [27] dataset which has 45 different classes. From each dataset, 16 random images were extracted from every class. This creates a new dataset with a total number of 2144 of images, each image is annotated with five different experts to ensure diversity. Figure 3a shows some images with their textual annotations from TextRS dataset.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Config. 1** & **Config. 2** & **Config. 3** \\ \hline \multirow{4}{*}{**Image encoder**} & ViT base & ViT base & Deit base distilled \\  & (layers = 12, hidden = 768, parameters = 86M & (layers = 12, hidden = 768, parameters = 86M, input image size: 224 \(\times\) 224 pixels, patch size=16 \(\times\) 16 pixels). & (layers = 12, hidden = 768, parameters = 110M). \\ \hline \multirow{4}{*}{**Pre-training mode**} & BERT base & RoBERTa base & BERT base \\  & (layers = 12, hidden = 512, parameters = 63M). & (layers = 12, hidden = 768, parameters = 110M). & (layers = 12, hidden = 768, parameters = 110M). \\ \hline \multirow{4}{*}{**Pre-training mode**} & \multirow{4}{*}{CLIP model for image text} & Models were learned & Models were learned \\  & & independently in a & \\ \cline{1-1}  & & self-supervised mode on image & independently in a standard \\ \cline{1-1}  & & and language tasks (DINO for image and SimCSE for text). & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Pre-trained configurations investigated for transferring knowledge to the cross-modal RS image–text retrieval task.  
Merced caption dataset: This dataset comprises 2100 images, each size of 256 \(\times\) 256 pixels with a spatial resolution of 30 cm. It was obtained from the Merced Land-use dataset [26], in which the images were manually labeled to one of 21 land-use classes. Five captions are used to describe each image in this collection, totaling 10,500 descriptions. Yet many sentences are highly correlated. Some examples of this dataset are illustrated in Figure 3b.  
Sydney caption dataset: This dataset was built based on the Sydney scene classification dataset, which includes RS images belonging to seven land-use classes including residential, airport, grassland, river, ocean, industrial area, and runway. It has 613 images, with a spatial resolution equal to 0.5 m.  
RSICD caption dataset: This dataset is the biggest of the earlier datasets. It was gathered from a number of sources, including Tianditu, Baidu Map, MapABC, and Google Earth. The dataset contains 224 \(\times\) 224 pixels of images with various spatial resolutions. There are 10,921 total images in the dataset, and each image has a unique text that describes it. RSICD has a total of 24,333 captions, which suggests that not every image has five phrases. For the sake of consistency, captions for photographs with fewer than five sentences have been duplicated. Table 2 summarizes image retrieval RS datasets  
Figure 3: Example of images with their corresponding captions (**a**) TextRS, (**b**) Merced.