Learning-to-Count by Learning-to-Rank: Weakly Supervised Object Counting & Localization Using Only Pairwise Image Rankings

1 INTRODUCTION . Object counting is a popular computer vision problem that involves localizing and quantifying the number of objects within an image , which has broad applicability across several domains . For example , counting problems have been explored throughout plant analysis ( David et al. , 2020 ; 2021 ; Häni et al. , 2020 ; Bargoti & Underwood , 2017 ; Minervini et al. , 2015 ; Teimouri et al. , 2018 ) , wildlife population monitoring ( Arteta et al. , 2016 ) , crowd surveillance ( Change Loy et al. , 2013 ; Chen et al. , 2013 ; Loy et al. , 2013 ; Chen et al. , 2012 ; Zhang et al. , 2016 ; 2015 ) , tissue sample analysis ( Marsden et al. , 2018 ; Paul Cohen et al. , 2017 ; Xie et al. , 2018 ; Kainz et al. , 2015 ) , and vehicle surveillance ( Guerrero-Gómez-Olmedo et al. , 2015 ; De Almeida et al. , 2015 ; Wang et al. , 2008 ) . However , we highlight two major problems that prevent counting problems from being widely applicable . First , there is no consensus on how to structure the optimization target . Different methods have approached the problem using a variety of annotations , which include bounding boxes ( Ren et al. , 2015 ; Redmon et al. , 2016 ) , global object counts ( Chattopadhyay et al. , 2017 ) , point maps ( Laradji et al. , 2018 ) , and density maps ( Lempitsky & Zisserman , 2010 ) . Bounding boxes are a popular optimization target for detection problems . However , these annotations tend to under-perform on counting problems , in particular , when scenes contain many highly occluded objects at a variety of scales ( Chattopadhyay et al. , 2017 ) . Density map annotations often perform better , but suffer from a lack of information related to object characteristics such as scale ( Arteta et al. , 2016 ) . Second , as the quantity and density of objects in an image increase , all of the above annotation types become labour intensive to collect and prone to significant annotator noise ( Arteta et al. , 2016 ) . We circumvent the burden of object labelling by introducing a simple form of annotation that can be rapidly collected , and design a method to propose density maps from these annotations . We propose using pairwise image ranking , a binary valued annotation that orders image pairs based on per-image object counts . Previous work has demonstrated these annotations to be an effective training signal for semi-supervised counting problems ( Liu et al. , 2018 ) . Whereas this work relied on leveraging automatically collected intra-image relationships between the whole image and subimage crops , we extend this idea further by proposing a method for learning to count and localize objects exclusively from inter-image pairwise image rankings . Our inter-image annotations carry a weak but information rich training signal despite being quick to collect . Humans are adept at rapidly assessing which of two images has more objects of interest , and untrained annotators are capable of discriminating between two images at a ratio of 10:11 objects in under 0.75 seconds ( Pica et al. , 2004 ; Halberda & Feigenson , 2008 ) . While previous optimization targets require an annotator to identify and count out each object , pairwise image rankings only require a single click from an annotator . In Tables 2 , 3 , and 4 we evaluate the annotation burden for recent object counting strategies and demonstrate that our annotation strategy carries only a fraction of the burden . Given these savings , there is significant value in developing methods that learn to extract counts and locations from this type of training data . The major technical challenge in exploiting this type of annotation is finding a way to extract counts and locations given only pairwise rankings . In this work , we propose an adversarial strategy for regularizing the penultimate representation of a ranking network to have the properties of a density map by comparing it to a pseudo-density map distribution . This places a soft integer and localization constraint on the representation . We argue that this strategy forces the model to solve the ranking sub-task by detecting re-occurring objects that can satisfy all of the ranking constraints . Combined with the weak quantity signal provided by pairwise image ranking annotations , this strategy produces a counting and localization model that is competitive with architectures trained on images with their density maps and global counts annotations . In summary , our work makes the following contributions : 1 . We propose object-based pairwise inter-image ranking as a novel , low-cost annotation strategy for weakly supervised counting and localization , and demonstrate that it performs comparably with fully supervised counting methods . 2 . We propose adversarial density map regularization , a method for enforcing that the network output has the properties of a density map . 2 PREVIOUS WORK . Object Counting with Limited Data . Object counting methods perform best when learning from density maps ( Lempitsky & Zisserman , 2010 ; Chattopadhyay et al. , 2017 ) , which carry a high annotation burden . Several methods have emerged that attempt to eliminate this burden . These methods can be loosely split into 4 categories – semi-supervised , knowledge transfer , sample selection , and weakly supervised methods . Semi-supervised counting methods alleviate the annotation burden by including additional unlabelled data ; a recent approach ( Liu et al. , 2019b ; 2018 ) introduced an unsupervised ranking loss that exploits the fact that any image has as many or more objects than any cropped sub-portion of that image . Other methods ( Sam et al. , 2019 ) have proposed more general feature learning strategies . Knowledge transfer methods explore transferring features between counting problems ; one method ( Zhang et al. , 2015 ) proposed learning from a multi-modal dataset which contained a mix of density maps and global object counts . A more recent approach ( Ranjan et al. , 2021 ) proposed a few-shot learning strategy where the model was trained using exemplar and density map pairs such that it could be extended to novel object classes . Active learning methods ( Ranjan et al. , 2020 ; Zhao et al. , 2020 ) approach the problem by finding ways to only label the most important examples in a new domain . Weakly supervised methods have explored strategies for better utilizing global counts as annotations . Recent methods ( Yang et al. , 2020 ; Lei et al. , 2021 ) proposed various regularization terms . One such method introduced a soft-sorting multi-task loss ( Yang et al. , 2020 ) , which involved learning from global object count annotations directly and indirectly through a soft-sorting task . Another recent work ( Liang et al. , 2021 ) has suggested that self-attention can improve learning from global counts . A different approach ( Cholakkal et al. , 2019 ) involved using object recognition combined with global object counts to create class-specific density maps . Our method differs from the above weakly-supervised methods by learning exclusively from a much weaker signal than global object counts . Weakly-Supervised Object Localization . Related to weakly-supervised object counting is weakly-supervised object localization ( WSOL ) ( Zhang et al. , 2021 ) . This task involves localizing objects using only some weak image-level label . Arguably , the most important WSOL methods are class activation maps ( CAM ) ( Zhou et al. , 2016 ) and Grad-CAM ( Selvaraju et al. , 2017 ) . The authors of CAM discovered that using global average pooling in a classification network allowed them to localize parts of the object being classified . Grad-CAM extends this work by re-weighting the activation map using the gradients of some target class . Separately , image region erasing has become a popular strategy for learning localized features from global class labels . Some works ( Singh & Lee , 2017 ; Choe & Shim , 2019 ) randomly erase image regions , forcing the network to rely on the whole object to classify the image . Other works ( Choe & Shim , 2019 ; Zhang et al. , 2018 ; Kim et al. , 2017 ; Wei et al. , 2017 ) proposed adaptive erasing strategies for removing the image regions that contribute most to classification , forcing the network to explore other regions in an attempt to accurately classify the image . Our problem shares similarities with WSOL in that we are using a global image label , albeit a pairwise image ranking label and not an image class label , to detect objects . Counting , Detection , and Output Representations . Localizing and counting target objects in images is an important computer vision problem . However , there is no consensus on the appropriate optimization target for this task . Bounding boxes ( Ren et al. , 2015 ; Redmon et al. , 2016 ) are a popular approach , but regressing over box coordinates is a difficult problem that underperforms on counting tasks ( Chattopadhyay et al. , 2017 ) . Density maps ( Lempitsky & Zisserman , 2010 ) are frequently used as an alternative , but , in their basic form , suffer from a lack of information about object scale and scene geometry . To remedy this , different strategies for producing adaptive density maps have been explored . Self-attention ( Wan & Chan , 2019 ) for selecting the best density map scale has worked for crowd counting problems . Similarly , ad-hoc geometric priors ( Zhang et al. , 2016 ) have been used to fuse scene geometry and density maps . Other methods have explored finding object blobs from dot maps ( Laradji et al. , 2018 ) ; multi-scale and multi-resolution density maps ( Idrees et al. , 2018 ) ; and learning directly from dot maps by treating density map estimation as a perspective-guided unbalanced optimal transport problem ( Wan et al. , 2021 ) . Ranking as Supervision The use of ranking as a training signal , sometimes known as ordinal regression , originates within information retrieval research . RankNet ( Burges et al. , 2005 ) , a document retrieval network , emerged as the first deep learning approach to ranking . However , this approach has been extended into several computer vision applications . Facial age estimation ( Chen et al. , 2018 ; Lim et al. , 2020 ) has benefited from pairwise image rankings , which learn the ‘ amount ’ of age in an image . Pairwise image ranking has been used to localize facial attributes ( Singh & Lee , 2016 ) , compare agent skills in videos ( Doughty et al. , 2018 ) , and detect video highlights ( Yao et al. , 2016 ) . Ranking has also been used for ranking object proposals and class label proposals ( Li et al. , 2017 ; Liu et al. , 2021 ) . These applications highlight that ranking as an optimization signal plays a significant role in computer vision applications . 3 PAIRWISE RANKING AS WEAK SUPERVISION . Pairwise image ranking is a task which requires an annotator to provide an ordering for image pairs based on the quantity of objects present in the image . Given some dataset Drank = { ( xi , xj ) , rij ≡ ci ≥ cj } N , we would like to extract the underlying object counts ci and cj given only the image ranking constraints rij for N pairs of images ( xi , xj ) . The goal of this paper is to develop a strategy to use the implicit weak object quantity signal present in Drank to solve the counting task . However , we first provide a motivation for this strategy by exploring the limitations of other annotation strategies and the strengths of pairwise image ranking . Density maps are the most popular annotation strategy for solving counting problems . Here , a density map is defined as a heatmap where a Gaussian kernel is convolved with point-wise annotations , with the property that the density map integrates to the global count . This annotation formulation is useful because it provides a location-based target for the object counting problem but has less complexity than bounding-box annotations . However , collecting point-wise annotations presents significant challenges . Strategies that learn directly from global object counts are able to reduce the annotation burden associated with density maps , but these methods still carry their own significant annotation burden . Global counting annotations and point-wise annotations require an annotator to visually localize all objects and record those labels . This can be time consuming and error prone , especially as object quantity , occlusion , and density increase . In fact , it has been demonstrated ( Arteta et al. , 2016 ) that when multiple annotators label an image with point-wise annotations , the annotators often suffer from fatigue or make inconsistent decisions when faced with ambiguity and make regular errors that lead to significant signal noise . Given the annotation burden that exists with all of the above methods , it would be valuable to establish a weak signal that could be rapidly collected by annotators while still leading to similar performance on the counting problem . Researchers in human psychology have found that humans can rapidly assess which of two groups of objects has the most objects if the difference in object count is greater than a particular ratio . This ratio is approximated by Weber-Fechner law , which is a law that describes the change in a stimulus necessary for some human to perceive the difference relative to the existing stimulus . For the task of pairwise image ranking , researchers ( Pica et al. , 2004 ; Halberda & Feigenson , 2008 ) have found that within 0.75 seconds , untrained adults are capable of determining which of two images has the most objects if the object count ratio is smaller than approximately 9:10 and 10:11 ( independent of the absolute count ) . This suggests that , for untrained adults , the task of rapidly annotating image pairs with their respective rank should be relatively simple as long as the ratio between object counts within the image pairs remains smaller than the given ratio . Thus , given a weak signal that can be rapidly collected , we are left to answer the difficult question of how to extract the counts .

Summary: This paper proposed an object counting and localization method which uses pairwise image ranking information for training. A count-based ranking objective is proposed for model optimization and an adversarial density map generation strategy is introduced to regularize the features of the network. The method was evaluated with multiple datasets.