# Multi-Subject Image Retrieval by Fusing Object and Scene-Level Feature Embeddings
## 1 Introduction  
Image retrieval is a problem of searching for images related to a given query image in a large database. The existing image retrieval studies mainly dealt with content-based image retrieval (CBIR) [1, 2, 3] or instance retrieval, which is based on the similarity of visual content between images. Recently, object-based image retrieval (OBIR) problems [4, 5, 6] have been introduced for splice detection, as shown in Figure 1.  
Deep learning networks based on convolutional neural networks (CNNs) are used for computer vision tasks such as image classification and object recognition. They are pre-trained on large datasets such as ImageNet [7] or Microsoft Common Objects in Context (MS COCO) [8]. These pre-trained networks demonstrate excellent image representation capabilities and are used as backbone networks in various computer vision tasks, such as image retrieval and instance segmentation. In CBIR, a feature map extracted from a pre-trained network is used as a global feature to calculate the similarity between the images. In instance retrieval, refs. [2, 9, 10] images are represented by embeddings based on a variation of global features. However, in OBIR, image retrieval is performed based on local features extracted from object detection models such as Faster-RCNN [11] or HoG [12]. This is because the image is represented based on object information excluding the background of the image. Existing image retrieval methods are mostly optimized for single-subject datasets. In the case of a database consisting of multi-subject images, or if the purpose of retrieval is complicated, both the scene and object information are required. While the conventional image retrieval methods have only optimized information for thespecific target subject, we propose multi-subject image retrieval (MSIR) utilizing image representation by combining object and scene information. Several image retrieval studies propose image embedding combining features representing different aspects. The feature fusion methods generate distinctive descriptors to incorporate various information and to improve the performance of image retrieval. However, previous fusion methods only used local, global, and deep-learning features. We propose fusion strategy utilizing object and scene-level feature embeddings to represent image information that is appropriate for MSIR tasks. To generate image embeddings for MSIR, we used two 'bag of visual words' (BoVW) models [13; 14; 15] representing object- and scene-level information. The first BoVW, for extracting object-level local feature vectors, used the Detection Transformer (DETR) [16] model. DETR can generate object embeddings corresponding to target objects in an image using the attention mechanism. Object embedding is generated based on the COCO dataset [8], which was used for the training and evaluation of object recognition models. Class embedding, which is extracted from the output vector of DETR via a multi-layer perceptron (MLP), is used for a better representation of object embeddings. The second BoVW, for scene-level features, was generated using a random patch generator [17] and  
Figure 1: The example of single-subject and multi-subject image retrieval. The image on the left is the query image and the image on the right side of the arrow is the retrieved image. The green box means that it is the right image for the purpose of the search. The red box means that the image is not suitable for search purposes. Contrary to single-subject image retrieval that searches a single topic such as content, instance, and object at once, multi-subject image retrieval deals with multiple topics in the same framework. image classification network pre-trained on ImageNet. As the random patches have local information including objects or a background, scene-level local features are obtained by inputting each patch into the image classification network. In conclusion, we propose an image embedding that combines object- and scene-level feature embeddings to improve the performance in MSIR.