Large-Scale Unsupervised Object Discovery

1 Introduction . This paper addresses the problem of identifying prominent objects in large image collections without manual annotations , a process known as unsupervised object discovery ( UOD ) . Early approaches to UOD focused mostly on finding clusters of images featuring objects of the same category [ 22 , 58 , 61 , 62 , 64 , 69 ] . Some of them [ 58 , 61 , 62 ] also output object locations in images , but their evaluations are limited to small datasets with distinctive object classes . More recent techniques [ 8 , 66 , 67 ] focus on the discovery of image links and individual object locations within much more diverse image collections . They typically rely on combinatorial optimization to select objects , or rather , object bounding boxes , among thousands of candidate region proposals [ 46 , 65 , 67 , 83 ] given similarity scores computed for pairs of proposals associated with different images . Although these techniques achieve promising results , their computational cost and inherently sequential nature limit the size of the dataset they can be applied to . Attempts to scale up the state-of-the-art approach [ 67 ] by reducing the search space size have revealed that this compromises its ability to discover multiple objects in 1Our code is publicly available at https : //github.com/huyvvo/LOD . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . each image . Other approaches to UOD focus on learning image representations by decomposing images into objects [ 5 , 12 , 23 , 43 , 47 ] . These techniques do not scale up ( yet ) to large natural image collections , and focus mostly on small datasets containing simple shapes in constrained environments . It is natural to cast unsupervised object discovery ( UOD ) as the task of finding repetitive visual patterns in image collections . Recent approaches ( e.g. , [ 66 , 67 ] ) to UOD formulate it as a combinatorial optimization problem in a graph of images , selecting simultaneously image pairs that contain similar objects and region proposals that correspond to objects , with the corresponding computational limitations . The motivation behind our work is to formulate UOD as a simpler graph-theoretical problem with a more efficient solution , where objects correspond to well-connected nodes in a graph whose nodes are region proposals ( instead of images in [ 66 , 67 ] ) , and edges are weighted by region similarity and objectness . In this scenario , finding object-proposal nodes is now a ranking problem where the goal is to rank the nodes based on how well they are connected in the graph . From another perspective , ranking is rather a natural modelization choice for UOD as in our context , discovering objects means finding the most object-like regions in a set of initial region proposals which naturally amounts to ranking them according to their “ objectness ” . As a result , a large array of methods available for eigenvalue problems and link analysis [ 48 ] can be applied to solve UOD on much larger datasets than previously possible ( Fig . 1 ) . We consider three variants of this approach : the first one re-defines the UOD objective [ 66 , 67 ] as an eigenvalue problem on the graph of region proposals , the second variant explores the applicability of PageRank [ 4 , 48 ] for UOD , and the final variant combines the other two into a hybrid algorithm , dubbed LOD ( for large-scale object discovery ) , which uses the solution of the eigenvalue problem to personalize PageRank . LOD offers a fast , distributed solution to object discovery on very large datasets . We show in Sec . 4.1 and Table 1 that its performance is comparable or better than the state of the art in the single object discovery setting for datasets of up to 120K images , and over 37 % better than the only algorithms we are aware of that can handle up to 1.7M images . In the multi-object discovery setting , LOD significantly outperforms all existing techniques on datasets from 20K to 1.7M images . While LOD does not explicitly address discovering relationships between images ( e.g. , grouping images into classes ) , we demonstrate that categories can be discovered as a post-processing step ( see Sec . 4.2 ) . The best performing approaches to UOD so far all use supervised region proposals and/or features . We also demonstrate for the first time in Sec . 4.1 that self-supervised features can give good UOD performance . Our main contributions can be summarized as follows : • We propose a new formulation of UOD as a ranking problem , allowing the application of parallel and distributed link analysis methods [ 4 , 48 ] . • We scale UOD up to datasets 87 times larger than those considered in the previous state of the art [ 67 ] . Our novel LOD algorithm outperforms others on medium-size datasets by up to 32 % . • We propose to use self-supervised features for UOD and show that LOD , combined with these features , offers a viable UOD pipeline without any supervision whatsoever . • We conduct extensive experiments on the COCO [ 42 ] and OpenImages [ 35 ] datasets to empirically validate our method . We also demonstrate applications of our approach to object category discovery and retrieval , outperforming other existing unsupervised baselines on both tasks by a large margin . 2 Problem statement and related work . 2.1 Problem statement . Consider a collection of n images , each equipped with a set region proposals [ 66 , 67 ] . For the sake of simplicity , we assume in this presentation that all images have exactly r region proposals . We wish to find which ones of these correspond to objects , and link images that contain similar objects , without any information other than how similar pairs of proposals are . This problem is known as unsupervised object discovery ( UOD ) and can be formulated as an optimization problem [ 67 ] over a graph where images are represented as nodes . Let epq ∈ { 0 , 1 } for p , q = 1 , 2 , . . . , n be a set of binary variables indicating if two images are connected in the graph , with epq = 1 when images p and q share similar visual content . Similarly , let xkp ∈ { 0 , 1 } for p = 1 , 2 , . . . , n and k = 1 , 2 , . . . , r be indicator variables such that xkp = 1 when region proposal k of image p is an object-like region in image p that is similar to an object-like region in one of the neighbors of image p. Let also xp be ( x1p , . . . , x r p ) T , x be the n× r matrix whose rows are xp for p = 1 , 2 , . . . , n and e be the binary adjacency matrix of the image graph . Then , the object discovery problem can be formulated as a combinatorial maximization problem : max x , e n∑ p=1 ∑ q∈N ( p ) epqx T p Spqxq s.t . r∑ k=1 xkp ≤ ν and ∑ q 6=p epq ≤ τ ∀ 1 ≤ p ≤ n , ( C ) where Spq ∈ Rr×r is a matrix whose entry Sk ` pq ≥ 0 measures the similarity between region k of image p and region ` of image q as well as the saliency of the respective regions , N ( p ) is a set of potential high-similarity neighbors of image p , and ν and τ are predefined constants corresponding to the maximum number of objects in an image and the maximum number of its neighbors , respectively . Previous approaches [ 66 , 67 ] to UOD solve a convex relaxation of ( C ) in the dual domain and/or use block-coordinate ascent on its variables x and e. The similarity scores Sk ` pq are typically computed using the Probabilistic Hough Matching ( PHM ) algorithm from [ 8 ] , which combines local appearance and global geometric consistency constraints to compare pairs of regions . A high PHM score between a pair of proposals is an indicator of whether the corresponding two proposals may correspond to a common foreground object . We follow this tradition and also use PHM scores ( Sec . 4 ) . The objective of UOD as formulated in ( C ) is to find both the objects ( variables xkp ) and the edges linking the images that contain them ( variables epq ) . Its combinatorial nature makes it hard to scale up to large values of n and r. [ 67 ] uses a block-coordinate ascent algorithm to ( C ) , updating variables x and e alternatively to optimize the objective . It attempts to scale up ( C ) with a drastic approximation , running on parts of the image collection to reduce r to only 50 before running on the entire dataset . However , using significantly reduced sets of region proposals hinders its ability to discover multiple objects ( Table 1 ) . Moreover , this algorithm is inherently sequential . According to [ 66 ] , in each iteration of optimizing x , an index i is chosen and xi is updated while e and all xj with j 6= i are kept fixed . The update of xi depends on the updated values of other xj if xj is updated before xi . This is crucial to guarantee that the objective always increases . If all xi are updated in parallel , there is no guarantee that the objective would increase . Consequently , this process is not parallelizable , preventing the algorithm from scaling up to datasets with millions of images . We therefore drop the second objective of UOD , and rely only on a fully connected , weighted graph of proposals where edge weights encode proposals ’ similarity ( edge weights can be zeros , see Sec . 3 ) . In turn , we can reformulate UOD as a ranking problem [ 4 , 30 , 34 , 36 , 51 ] , amenable to the panoply of large-scale distributed tools available for eigenvalue problems and link analysis . We consider two different ranking formulations : the first ( Q ) tackles a quadratic optimization problem , and the second ( P ) is based on the well-known PageRank algorithm [ 4 , 48 ] . We combine these two approaches into a joint formulation ( LOD ) that gives the best results on large-scale datasets . See Sections 3 and 4 for details . 2.2 Related work . Unsupervised object discovery . Early works on unsupervised object discovery focused on finding groups of images depicting objects of the same categories , employing probabilistic models [ 58 , 61 , 69 ] , non-negative matrix factorization ( NMF ) [ 62 ] or clustering techniques [ 22 ] , see [ 64 ] for a survey . In addition to finding image groups , some of these approaches , e.g. , multiple instance learning ( MIL ) [ 82 ] , graph mining [ 75 ] , contour matching [ 39 ] and topic modeling [ 58 , 61 ] , also output object locations , but focus on smaller datasets with only a handful of distinctive object classes . Saliency detection [ 80 ] is related to UOD , but seeks to generate a per-pixel map of saliency scores , while UOD attempts to find bounding boxes around objects in each image without supervision . Object discovery in large real-world image collections remains challenging due to a high degree of intra-class variation , occlusion , background clutter and appearance of multiple object categories in one image . For this challenging setting , [ 8 ] proposes an iterative algorithm which alternates between retrieving image neighbors and localizing salient regions . Based on this approach , [ 66 ] is the first to formulate UOD as the optimization problem ( C ) , finding first an approximate solution to a continuous relaxation , then applying block-coordinate ascent to find the solution of the original problem . The first step involves solving a max-flow problem [ 1 ] exactly , which is too costly for medium- to large-scale datasets . The datasets have scaled up with successive approaches , from about 3,500 images for [ 8 , 66 ] to 20,000 for [ 67 ] , an extension of [ 66 ] using a two-stage approach to UOD and a new approach to region proposal design . However , these works are inherently sequential and difficult to scale further . Additionally , [ 67 ] operates on reduced sets of region proposals containing only tens of regions , compromising its ability to discover multiple objects ( Table 1 ) . In contrast , our proposed approach considers all region proposals , is parallelizable and can be implemented in a distributed way . Thus , it scales well to very large datasets without compromising performance . Indeed , we demonstrate effective UOD in challenging datasets with up to 1.7 million images ( Fig . 1 ) . Weakly-supervised object localization and image co-localization . These problems are related to UOD but take advantage of image-level labels . Weakly-supervised object localization ( WSOL ) considers scenarios where the input dataset contains image-level labels [ 9 ] . Most recent WSOL methods localize objects using saliency maps obtained from convolutional features in a neural network [ 7 , 59 , 81 ] . Since networks tend to learn category-discriminating features , various strategies for improving the quality of features for localization have been proposed [ 2 , 10 , 76–79 ] . Colocalization is another line of work where all input images are assumed to contain at least one instance of a single object category . [ 63 ] uses discriminative clustering [ 28 ] for co-localization in noisy image sets and [ 29 ] extends this work to the video setting . [ 41 ] learns to co-localize objects by learning sparse confidence distributions , mimicking behaviors of supervised detectors . [ 70 ] and [ 71 ] observe that activation maps generated from CNN features contain information about object locations . They propose to cluster locations in the images into background and foreground using PCA and return the tight bounding box around the foreground pixels as an object . Since [ 71 ] can deal with large-scale datasets and can be easily adapted to UOD , we use it as a baseline in our experiments . Ranking applications in computer vision . The goal of ranking is to assign a global importance rating to each item in a set according to some criterion [ 48 ] . Many computer vision problems admit a ranking formulation , including image retrieval [ 6 ] , object tracking [ 3 ] , person re-identification [ 44 ] , video summarization [ 72 ] , co-segmentation [ 52 ] and saliency detection [ 40 ] . Several techniques specifically designed for large-scale ranking problems [ 34 , 48 ] have been used to explore large datasets of images [ 27 ] and shapes [ 16 ] . PageRank-based approaches in particular have been popular [ 27 , 49 , 56 ] due to their scalability . [ 31 ] proposed an algorithm for object discovery that combines appearance and geometric consistency with PageRank-based link analysis for category discovery . However , it does not scale beyond 600 images . A more scalable follow-up work by [ 32 ] discovers regions of interest ( RoIs ) from images with successive applications of PageRank . This algorithm includes two main steps . The first one attempts to find object representatives ( hubs ) from the current RoIs of all images using PageRank . PageRank is then utilized again in the second step to analyze the links between regions in each image and the hubs , this time to update the RoIs of the images . Finally , good RoIs are found by repeating these two steps until convergence . We compare our method to this technique in Sec . 4.1 .

Summary: This is a very interesting paper. It builds upon object discovery formulation by [Vo et al., CVPR’19; ECCV’20], which formulates the discovery as (co-)detection of frequently occurring patterns within an image collection. This approach assumes input in the form of object proposals (together with a feature extractor or related mechanism that will be used to determine pairwise similarities) and formalizes object co-detection as a combinatorial optimization problem that seeks object proposals with high connectivity across the dataset. It has been established by [Vo et al., CVPR’19; ECCV’20] that this approach performs notably better compared to sliding window-based approaches that rank proposals in images independently [e.g., Zitnick et al., ECCV’14]. The paper makes a case that optimization methods proposed in prior work, do not scale to very large datasets, largely because they are not parallelizable. To tackle the scalability issue, this paper draws parallels between the discovery objective [Vo et al., CVPR’19] and object ranking, cast as a quadratic optimization problem, and well-known PageRank algorithm, as well as a combination of the two. Under this formulation, optimal ranking of proposals can be obtained via Eigenvalue decomposition of the adjacency matrix using the power iteration algorithm. While single-core implementation does not present a significant advantage over solvers proposed to tackle the original optimization formulation [Vo et al., CVPR’19; ECCV’20], this approach is highly parallelizable and, in turn, allows to apply this method to very large datasets, such as OpenImages, containing over 1.7M images and is on-par or slightly better in terms of the object discovery performance.