# Multi-Subject Image Retrieval by Fusing Object and Scene-Level Feature Embeddings
## 4 Experiments
### Multi-Subject Image Retrieval  
We applied the proposed method, which combines object- and scene-level features based on the BoVW framework to MSIR, and compared it with other image retrieval methods. In Table 2, C, A, P, and O indicate the COCO, AID, Paris6K, and Oxford5K datasets, respectively. The multi-subject dataset comprises a combination of these single-subject datasets. To avoid data imbalance, five images per class were selected as query images, and 100 images were used as a search database. The feature, codebook, and histogram indicate the fusion of the feature vector, codebook, and histogram levels, respectively. In the MSIR, R-MAC and R-GeM performed worse in contrast to single-subject datasets. Images with similar backgrounds but different object classes often appear in multi-subject datasets. As the conventional retrieval methods cannot completely discriminate a background and objects, images with similar backgrounds behave as distractors, which results in poor performance. DETR-based BoVW has low performance, even if all the datasets used in the experiment contain object images. DETR has excellent object representation ability, but low scene representation ability. Therefore, the local features of the DETR generated for the scene image interfere with the creation of an intact codebook and cause a decrease in the overall performance. The proposed method showed better performance for all combinations of multi-subject datasets. In particular, histogram-based fusion shows the best performance in all cases, because the image embedding of histogram level fusion can effectively reflect both object-level and scene-level information. The codebook and feature vector-based fusion methods also showed significantly better performances compared to other methods in the C+P and C+P+O datasets. We divided the BoVW method into three stages to generate different image embeddings. As shown in Table 2, the image embedding of histogram-level performs well in all multi-subject databases. Image embeddings of codebook- and feature-levels performed relatively poorly, as object and scene-information can be imbalanced during the fusion process. However, in most cases, the proposed methods performed better than the traditional methods. The image embedding of the histogram-level has distinct and balanced scene and object information, and presents the best results compared to the other descriptors. In addition, since the two histograms are simply concatenated, the fusion process requires \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **COCO** & **AID** & **Paris** & **Oxford** & **Base** \\ \hline DETR & **85.35** & 15.86 & 18.12 & 13.05 & BoVW \\ Random & 60.79 & 36.1 & 69.73 & 21.41 & BoVW \\ SIFT & 13.41 & 22.18 & 30.58 & 15.46 & BoVW \\ ResNet & 56.46 & 54.98 & 65.43 & 23.32 & CNN \\ R-MAC & 64.57 & 51.08 & 67.11 & 24.88 & CNN \\ R-GeM & 64.39 & **58.99** & **72.15** & **25.27** & CNN \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of accuracy in a single-subject dataset. The bold and underline mean the best and second accuracy for each dataset, respectively. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **C+A** & **C+P** & **C+P+O** & **C+A+P+O** & **Base** \\ \hline DETR & 33.77 & 45.59 & 38.89 & 28.78 & BoVW \\ Random & 27.62 & 33.39 & 28.95 & 26.54 & BoVW \\ SIFT & 5.21 & 6.66 & 6.71 & 6.23 & BoVW \\ ResNet & 35.14 & 34.76 & 31.73 & 35.92 & CNN \\ R-MAC & 33.88 & 36.42 & 33.56 & 36.26 & CNN \\ R-GeM & 36.39 & 36.74 & 33.77 & 38.36 & CNN \\ Feature & 37.16 & 50.08 & 43.79 & 30.32 & Mix BoVW \\ Codebook & 40.24 & 53.16 & 47.22 & 31.31 & Mix BoVW \\ Histogram & **43.54** & **55.42** & **48.49** & **39.55** & Mix BoVW \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison of accuracy in a multi-subject dataset. The bold and underline mean the best and second accuracy for each dataset, respectively. little resources. Figure 11 shows an example of the image retrieval results of the global feature and fusion embeddings proposed in this study.