# Multi-Subject Image Retrieval by Fusing Object and Scene-Level Feature Embeddings
## 4 Experiments  
Various experiments were conducted to compare the performance of the previous image retrieval methods and the proposed method on single-subject and multi-subject datasets. As a single-subject dataset, the MS COCO [8], aerial image dataset (AID) [50], and Paris6K and Oxford5K [2] datasets, which predominantly include object, scene, and landmark images, respectively, were used. MS COCO is typically used for the training and evaluation of object recognition models, and consists of 80 object classes. Many images in the COCO dataset include multiple classes of objects. However, image retrieval performance is mainly measured by a single class match of a query image with the retrieved images. Images with multiple classes were unsuitable for evaluation. Therefore, images in which only objects of a single class exist in COCO were selected, where 1025 images composed of a single class from 5000 verification images were used as query images, and 24,186 images out of 118,287 images were reorganized as a database for search. AID consists of 30 aerial scene classes, and 50 images per class were used as query images. A database consisting of 170 images per class was used for the search. Paris6K and Oxford5K are each composed of 12 landmark classes. Five images per class were used as query images, and a database of 100 images per class was used for the search. Our implementation follows the practice in [16; 51]. The image is resized to 800 \(\times\) 800 resolution, which was the shortest size that DETR used for the object recognition process. The image for scene-level feature embedding extracts multiple random patches of 224 \(\times\) 224 resolution, which follows the practice in [51]. The backbone for feature extraction is with the ImgeNet-pretrained ResNet model from TORCHVISION with frozen batchnorm layers. Specifically, the ResNet-101 model is used for all the experiments. For scene-level feature embeddings, we applied a pooling layer to the feature map of ResNet. The corresponding models with average pooling, regional max-pooling [10], and GeM pooling [9] are called ResNet, R-MAC, and R-GeM, respectively. In all tables, the CNN-based global feature has 2048 dimensions, which follow, in practice, [51]; BoVW- and Mix BoVW-based feature  
Figure 9: The overview of feature fusion methods. We combine local feature, codebook, and histogram representing different aspects. We state these as feature-, codebook- and histogram-level. embeddings have 1000 dimensions. The number of local features for object-level feature embeddings used for generating codebook and histogram is 100, which follow the practice in [16]. The number of local features for scene-level feature embeddings is the same as the object-level feature embeddings for comparison. All hyperparameters are empirically set in consideration of a trade-off between complexity and performance. We fine-tune revised DETR which is an added linear layer with AdamW [52] for class embedding. The clustering method for codebook is a k-means clustering algorithm from the KMeans of SKLEARN library. We extract global feature, object, and scene-level feature embeddings using PYTORCH, OPENCV, and PYTHON, versions 1.12.1, 4.6.0, and 3.8.8, respectively, on an Intel I7 CPU with an Nvidia RTX 3090 GPU. We calculated the similarity between image embeddings extracted from a given query image and database. The retrieved images were then sorted in descending order by similarity. At this time, the cosine similarity was used for calculating the similarity, which is defined as follows:  
\[cosine\ similarity=\frac{q\cdot x}{\|q\|\|x\|} \tag{2}\]  
where \(q\) is an image embedding of a given query image, \(x\) are the image embeddings of the database. Image retrieval performance was measured using the evaluation metric, which is defined as follows:  
\[accuracy=\frac{100}{N}\sum_{i=1}^{N}\omega_{i}\cdot\mathbb{I}(X_{class}^{i}=T_{ class}) \tag{3}\]  
where \(N\) is the number of retrieved images. \(i\) means the rank of the retrieved image. \(\mathbb{I}(\cdot)\) is the indicator function.