# Multi-Subject Image Retrieval by Fusing Object and Scene-Level Feature Embeddings
## 3 Methods
In particular, because bounding box regression mainly utilizes the extremit information of objects, such as legs and heads, only limited object information was used for classification compared to class embedding, which focuses on the overall information of objects. Therefore, class embedding, which represents object information well, was used as the object embedding for object-level features. The weighted histogram in Figure 7 reflects the object class confidence of object embedding. Similar to the bounding box shown in the query image, each object embedding may correspond to an object or a non-object region. However, in the existing histogram, all of the object embeddings have the same weight, as shown in Figure 7a. Because object embeddings with low confidence in generating object-level features can be viewed as outliers, confidence should be reflected in the histogram generation process, such that the object embedding corresponding to the actual object can have a higher weight. As shown in Figure 7b, the distribution of the weighted histogram was more suitable as an object-level feature because the codewords representing the objects are emphasized more than other non-object codewords. The weighted histogram can generate more significant object-level features by reflecting the confidence of object embedding. Figure 5: The visualization of weights of decoder. Each object query points to a different instance. Figure 6: The overview of class embedding. This is generated after the transformer’s output vector has passed through the MLP. A scene-level feature was generated using the feature vectors obtained by generating \(M\) patches from an image through a random patch generator [17], and passing each patch through a CNN, as shown in Figure 8. As each patch can have either object or background information, in contrast to DETR, it can be viewed as a content-level feature that represents the overall information of the image. The fusion of image embeddings for MSIR was generated by the integration of object- and scene-level features, as introduced above. We propose a fusion strategy in three stages: (i) feature vector-level fusion (ii) codebook fusion, and (iii) histogram-level fusion between the DETR-based BoVW and random patch-based BoVW, as shown in Figure 9. In feature vector-level fusion, scene and object-level feature vectors are combined and used to generate a codebook by clustering the combined feature vectors. As clustering is performed based on the distance between feature vectors without labeling, codewords, which are representative values of K clusters, can simultaneously include scene- and object-level characteristics. In codebook-level fusion, codebooks from DETR and random patch-based BoVWs are combined. In feature vector-level fusion, an imbalance of object- and scene-level codewords may occur depending on the ratio of images in the database and extracted local feature vectors, whereas in codebook level fusion, both codewords have the same proportion. Histogram-level fusion fuses histograms of the occurrences of visual words. This fusion strategy has the advantage of a better representation of image content, such as scenes or objects if the codebook is generated from a larger and more diverse database. Figure 4.0 The pipeline of scene-level feature extraction. Figure 4.1 The overview of weighted histogram. Weights are applied when computing the histogram according to the confidence of each region.