Large-Scale Unsupervised Object Discovery

1 Introduction . This paper addresses the problem of identifying prominent objects in large image collections without manual annotations , a process known as unsupervised object discovery ( UOD ) . Early approaches to UOD focused mostly on finding clusters of images featuring objects of the same category [ 22 , 58 , 61 , 62 , 64 , 69 ] . Some of them [ 58 , 61 , 62 ] also output object locations in images , but their evaluations are limited to small datasets with distinctive object classes . More recent techniques [ 8 , 66 , 67 ] focus on the discovery of image links and individual object locations within much more diverse image collections . They typically rely on combinatorial optimization to select objects , or rather , object bounding boxes , among thousands of candidate region proposals [ 46 , 65 , 67 , 83 ] given similarity scores computed for pairs of proposals associated with different images . Although these techniques achieve promising results , their computational cost and inherently sequential nature limit the size of the dataset they can be applied to . Attempts to scale up the state-of-the-art approach [ 67 ] by reducing the search space size have revealed that this compromises its ability to discover multiple objects in 1Our code is publicly available at https : //github.com/huyvvo/LOD . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . each image . Other approaches to UOD focus on learning image representations by decomposing images into objects [ 5 , 12 , 23 , 43 , 47 ] . These techniques do not scale up ( yet ) to large natural image collections , and focus mostly on small datasets containing simple shapes in constrained environments . It is natural to cast unsupervised object discovery ( UOD ) as the task of finding repetitive visual patterns in image collections . Recent approaches ( e.g. , [ 66 , 67 ] ) to UOD formulate it as a combinatorial optimization problem in a graph of images , selecting simultaneously image pairs that contain similar objects and region proposals that correspond to objects , with the corresponding computational limitations . The motivation behind our work is to formulate UOD as a simpler graph-theoretical problem with a more efficient solution , where objects correspond to well-connected nodes in a graph whose nodes are region proposals ( instead of images in [ 66 , 67 ] ) , and edges are weighted by region similarity and objectness . In this scenario , finding object-proposal nodes is now a ranking problem where the goal is to rank the nodes based on how well they are connected in the graph . From another perspective , ranking is rather a natural modelization choice for UOD as in our context , discovering objects means finding the most object-like regions in a set of initial region proposals which naturally amounts to ranking them according to their “ objectness ” . As a result , a large array of methods available for eigenvalue problems and link analysis [ 48 ] can be applied to solve UOD on much larger datasets than previously possible ( Fig . 1 ) . We consider three variants of this approach : the first one re-defines the UOD objective [ 66 , 67 ] as an eigenvalue problem on the graph of region proposals , the second variant explores the applicability of PageRank [ 4 , 48 ] for UOD , and the final variant combines the other two into a hybrid algorithm , dubbed LOD ( for large-scale object discovery ) , which uses the solution of the eigenvalue problem to personalize PageRank . LOD offers a fast , distributed solution to object discovery on very large datasets . We show in Sec . 4.1 and Table 1 that its performance is comparable or better than the state of the art in the single object discovery setting for datasets of up to 120K images , and over 37 % better than the only algorithms we are aware of that can handle up to 1.7M images . In the multi-object discovery setting , LOD significantly outperforms all existing techniques on datasets from 20K to 1.7M images . While LOD does not explicitly address discovering relationships between images ( e.g. , grouping images into classes ) , we demonstrate that categories can be discovered as a post-processing step ( see Sec . 4.2 ) . The best performing approaches to UOD so far all use supervised region proposals and/or features . We also demonstrate for the first time in Sec . 4.1 that self-supervised features can give good UOD performance . Our main contributions can be summarized as follows : • We propose a new formulation of UOD as a ranking problem , allowing the application of parallel and distributed link analysis methods [ 4 , 48 ] . • We scale UOD up to datasets 87 times larger than those considered in the previous state of the art [ 67 ] . Our novel LOD algorithm outperforms others on medium-size datasets by up to 32 % . • We propose to use self-supervised features for UOD and show that LOD , combined with these features , offers a viable UOD pipeline without any supervision whatsoever . • We conduct extensive experiments on the COCO [ 42 ] and OpenImages [ 35 ] datasets to empirically validate our method . We also demonstrate applications of our approach to object category discovery and retrieval , outperforming other existing unsupervised baselines on both tasks by a large margin . 2 Problem statement and related work . 2.1 Problem statement . Consider a collection of n images , each equipped with a set region proposals [ 66 , 67 ] . For the sake of simplicity , we assume in this presentation that all images have exactly r region proposals . We wish to find which ones of these correspond to objects , and link images that contain similar objects , without any information other than how similar pairs of proposals are . This problem is known as unsupervised object discovery ( UOD ) and can be formulated as an optimization problem [ 67 ] over a graph where images are represented as nodes . Let epq ∈ { 0 , 1 } for p , q = 1 , 2 , . . . , n be a set of binary variables indicating if two images are connected in the graph , with epq = 1 when images p and q share similar visual content . Similarly , let xkp ∈ { 0 , 1 } for p = 1 , 2 , . . . , n and k = 1 , 2 , . . . , r be indicator variables such that xkp = 1 when region proposal k of image p is an object-like region in image p that is similar to an object-like region in one of the neighbors of image p. Let also xp be ( x1p , . . . , x r p ) T , x be the n× r matrix whose rows are xp for p = 1 , 2 , . . . , n and e be the binary adjacency matrix of the image graph . Then , the object discovery problem can be formulated as a combinatorial maximization problem : max x , e n∑ p=1 ∑ q∈N ( p ) epqx T p Spqxq s.t . r∑ k=1 xkp ≤ ν and ∑ q 6=p epq ≤ τ ∀ 1 ≤ p ≤ n , ( C ) where Spq ∈ Rr×r is a matrix whose entry Sk ` pq ≥ 0 measures the similarity between region k of image p and region ` of image q as well as the saliency of the respective regions , N ( p ) is a set of potential high-similarity neighbors of image p , and ν and τ are predefined constants corresponding to the maximum number of objects in an image and the maximum number of its neighbors , respectively . Previous approaches [ 66 , 67 ] to UOD solve a convex relaxation of ( C ) in the dual domain and/or use block-coordinate ascent on its variables x and e. The similarity scores Sk ` pq are typically computed using the Probabilistic Hough Matching ( PHM ) algorithm from [ 8 ] , which combines local appearance and global geometric consistency constraints to compare pairs of regions . A high PHM score between a pair of proposals is an indicator of whether the corresponding two proposals may correspond to a common foreground object . We follow this tradition and also use PHM scores ( Sec . 4 ) . The objective of UOD as formulated in ( C ) is to find both the objects ( variables xkp ) and the edges linking the images that contain them ( variables epq ) . Its combinatorial nature makes it hard to scale up to large values of n and r. [ 67 ] uses a block-coordinate ascent algorithm to ( C ) , updating variables x and e alternatively to optimize the objective . It attempts to scale up ( C ) with a drastic approximation , running on parts of the image collection to reduce r to only 50 before running on the entire dataset . However , using significantly reduced sets of region proposals hinders its ability to discover multiple objects ( Table 1 ) . Moreover , this algorithm is inherently sequential . According to [ 66 ] , in each iteration of optimizing x , an index i is chosen and xi is updated while e and all xj with j 6= i are kept fixed . The update of xi depends on the updated values of other xj if xj is updated before xi . This is crucial to guarantee that the objective always increases . If all xi are updated in parallel , there is no guarantee that the objective would increase . Consequently , this process is not parallelizable , preventing the algorithm from scaling up to datasets with millions of images . We therefore drop the second objective of UOD , and rely only on a fully connected , weighted graph of proposals where edge weights encode proposals ’ similarity ( edge weights can be zeros , see Sec . 3 ) . In turn , we can reformulate UOD as a ranking problem [ 4 , 30 , 34 , 36 , 51 ] , amenable to the panoply of large-scale distributed tools available for eigenvalue problems and link analysis . We consider two different ranking formulations : the first ( Q ) tackles a quadratic optimization problem , and the second ( P ) is based on the well-known PageRank algorithm [ 4 , 48 ] . We combine these two approaches into a joint formulation ( LOD ) that gives the best results on large-scale datasets . See Sections 3 and 4 for details . 2.2 Related work . Unsupervised object discovery . Early works on unsupervised object discovery focused on finding groups of images depicting objects of the same categories , employing probabilistic models [ 58 , 61 , 69 ] , non-negative matrix factorization ( NMF ) [ 62 ] or clustering techniques [ 22 ] , see [ 64 ] for a survey . In addition to finding image groups , some of these approaches , e.g. , multiple instance learning ( MIL ) [ 82 ] , graph mining [ 75 ] , contour matching [ 39 ] and topic modeling [ 58 , 61 ] , also output object locations , but focus on smaller datasets with only a handful of distinctive object classes . Saliency detection [ 80 ] is related to UOD , but seeks to generate a per-pixel map of saliency scores , while UOD attempts to find bounding boxes around objects in each image without supervision . Object discovery in large real-world image collections remains challenging due to a high degree of intra-class variation , occlusion , background clutter and appearance of multiple object categories in one image . For this challenging setting , [ 8 ] proposes an iterative algorithm which alternates between retrieving image neighbors and localizing salient regions . Based on this approach , [ 66 ] is the first to formulate UOD as the optimization problem ( C ) , finding first an approximate solution to a continuous relaxation , then applying block-coordinate ascent to find the solution of the original problem . The first step involves solving a max-flow problem [ 1 ] exactly , which is too costly for medium- to large-scale datasets . The datasets have scaled up with successive approaches , from about 3,500 images for [ 8 , 66 ] to 20,000 for [ 67 ] , an extension of [ 66 ] using a two-stage approach to UOD and a new approach to region proposal design . However , these works are inherently sequential and difficult to scale further . Additionally , [ 67 ] operates on reduced sets of region proposals containing only tens of regions , compromising its ability to discover multiple objects ( Table 1 ) . In contrast , our proposed approach considers all region proposals , is parallelizable and can be implemented in a distributed way . Thus , it scales well to very large datasets without compromising performance . Indeed , we demonstrate effective UOD in challenging datasets with up to 1.7 million images ( Fig . 1 ) . Weakly-supervised object localization and image co-localization . These problems are related to UOD but take advantage of image-level labels . Weakly-supervised object localization ( WSOL ) considers scenarios where the input dataset contains image-level labels [ 9 ] . Most recent WSOL methods localize objects using saliency maps obtained from convolutional features in a neural network [ 7 , 59 , 81 ] . Since networks tend to learn category-discriminating features , various strategies for improving the quality of features for localization have been proposed [ 2 , 10 , 76–79 ] . Colocalization is another line of work where all input images are assumed to contain at least one instance of a single object category . [ 63 ] uses discriminative clustering [ 28 ] for co-localization in noisy image sets and [ 29 ] extends this work to the video setting . [ 41 ] learns to co-localize objects by learning sparse confidence distributions , mimicking behaviors of supervised detectors . [ 70 ] and [ 71 ] observe that activation maps generated from CNN features contain information about object locations . They propose to cluster locations in the images into background and foreground using PCA and return the tight bounding box around the foreground pixels as an object . Since [ 71 ] can deal with large-scale datasets and can be easily adapted to UOD , we use it as a baseline in our experiments . Ranking applications in computer vision . The goal of ranking is to assign a global importance rating to each item in a set according to some criterion [ 48 ] . Many computer vision problems admit a ranking formulation , including image retrieval [ 6 ] , object tracking [ 3 ] , person re-identification [ 44 ] , video summarization [ 72 ] , co-segmentation [ 52 ] and saliency detection [ 40 ] . Several techniques specifically designed for large-scale ranking problems [ 34 , 48 ] have been used to explore large datasets of images [ 27 ] and shapes [ 16 ] . PageRank-based approaches in particular have been popular [ 27 , 49 , 56 ] due to their scalability . [ 31 ] proposed an algorithm for object discovery that combines appearance and geometric consistency with PageRank-based link analysis for category discovery . However , it does not scale beyond 600 images . A more scalable follow-up work by [ 32 ] discovers regions of interest ( RoIs ) from images with successive applications of PageRank . This algorithm includes two main steps . The first one attempts to find object representatives ( hubs ) from the current RoIs of all images using PageRank . PageRank is then utilized again in the second step to analyze the links between regions in each image and the hubs , this time to update the RoIs of the images . Finally , good RoIs are found by repeating these two steps until convergence . We compare our method to this technique in Sec . 4.1 .

Summary: This paper introduces an efficient approach for Unsupaervised Object Discovery (UOD). This efficiency was achieved by considering the exising UOD formulation as a ranking problem by modifying the binary importance vector as having a floating value. This modification allows to apply this approach to a very large-scale dataset (up to 1.7M images) without compromising accuracy.