# Evaluation of Multi-Stream Fusion for Multi-View Image Set Comparison
## 1 Introduction  
By \"visual similarity\", we understand that the visual content of two images, i.e., pictured objects, is the same or somehow similar. This type of comparison is not highly challenging if photographs of the same object are taken from the same place, so that they differ by only a few details. A typical solution to such a task is to extract standard image features or descriptions (embedding vectors) generated by deep neural networks, and to classify them or to apply a hand-crafted decision by using vector-distance metrics (i.e., the Euclidean distance in vector space) [1, 2]. The same image similarity problem becomes more difficult if the image content is a 3D object and the compared images present it from different viewpoints (Figure 1). Direct comparison or classification of entire images (i.e., a front-to-end approach) by deep neural networks is a recent approach to image analysis [3, 4, 5, 6] that has produced outstanding results. However, consider the problem of defining the \"similarity\" of two image sets and combine it with the problem of identifying the same \"semantic\" content,i.e., the same 3D object visible from different viewpoints. This complex problem appears in various real-world business scenarios, i.e., fraud detection, visual navigation, or product search. It is also useful in visual search or data duplication avoidance--for example, in a car insurance application process. A customer would have to take a set of photos of the car in order to for the insurance policy to start, and after an insurance event happens, the customer would have to take another set of photos of the same car. An automatic image set comparison system can improve the quality and speed of the claim handling process. The growing demand for visual similarity solutions is clear and comes from companies which seek effective answers for the aforementioned business problems. In recent years, some significant papers have been published about this topic. The first one mentioned here gives a review of a few methods developed for finding similar images in datasets [7]. Another important paper proposes a deep learning technique for the image similarity task called \"DeepRanking\" [8]. In this paper, we propose a solution to the image set comparison problem (Figure 2). This problem is analyzed in Section 2, while the proposed solution is described in Section 3. Our approach uses both image (more exactly: foreground object) classification and object-pair similarity evaluation procedures, and places them in a general-purpose framework system for image-set comparison. The solution requires not only the use of some category-level similarity, but also an instance-to-instance metric. Thus, the similarity score combines the evaluation of the visual content of image pairs with the comparison of image features computed from object ROIs (region of interest; the regions in the image which provide the needed information). We expect that the proposed solution might be deployed for comparisons of sets of images showing cars, cats, buildings, etc. An earlier developed base solution to the same problem is referred to in Section 4. The experimental verification of our approach is summarized in Section 5. The presentation of limitations and conclusions as well as a final discussion complete this paper. The main objective is to provide answers to questions such as \"is my set of images presenting the same object from the same viewpoint as the other set of images\". The object is application-dependent. The exact recognition of views and objects in particular images is not the question--the ultimate challenge is how to define a similarity measure for sets of object-centered images. During the search for a solution to this problem, we discovered that this case is nearly identical to the evaluation of multi-channel blind signal separation results [9], where the objective is to specify whether the set of M signals extracted from N  
Figure 1: The “image-to-image (object-to-object)” comparison problem in images.