Learning-to-Count by Learning-to-Rank: Weakly Supervised Object Counting & Localization Using Only Pairwise Image Rankings
Authors: Adriano C. D'Alessandro
Ali Mahdavi Amiri
Ghassan Hamarneh
Venue: ICLR 2022 Submitted (2022)
Keywords: Object Counting
Weak Supervision
Ranking

Abstract: Object counting and localization in dense scenes is a challenging class of image analysis problems that typically requires labour intensive annotations to learn to solve. We propose a form of weak supervision that only requires object-based pairwise image rankings. These annotations can be collected rapidly with a single click per image pair and supply a weak signal for object quantity. However, the problem of actually extracting object counts and locations from rankings is challenging. Thus, we introduce adversarial density map generation, a strategy for regularizing the features of a ranking network such that the features correspond to an object proposal map where each proposal must be a Gaussian blob that integrates to 1. This places a soft integer and soft localization constraint on the representation, which encourages the network to satisfy the provided ranking constraints by detecting objects. We then demonstrate the effectiveness of our method for exploiting pairwise image rankings as a weakly supervised signal for object counting and localization on several datasets, and show results with a performance that approaches that of fully supervised methods on many counting benchmark datasets while relying on data that can be collected with a fraction of the annotation burden.

Under review as a conference paper at ICLR 2022
LEARNING-TO-COUMT BY LEARNING-TO-RANG WEAKLY SUPERVISED OBJECT COUNTING & LOCAL- IZATION USING ONLY PAIRWISE IMAGE RANKINGS
Anonymous authors Paper under double-blind review
ABSTRACT
Object counting and localization in dense scenes is challenging class of image analysis problems that typically requires labour intensive annotations to learn to solve. We propose form of weak supervision that only requires object-based pairwise image rankings. These annotations can be collected rapidly with a sin- gle click per image pair and supply weak signal for object quantity. However, the problem of actually extracting object counts and locations from rankings is challenging. Thus, we introduce adversarial density map generation, strategy for regularizing the features of a ranking network such that the features corre- spond to an object proposal map where each proposal must be Gaussian blob that integrates to 1. This places a soft integer and soft localization constraint on the representation, which encourages the network to satisfy the provided rank- ing constraints by detecting objects. We then demonstrate the effectiveness of our method for exploiting pairwise image rankings as a weakly supervised signal for object counting and localization on several datasets, and show results with a performance that approaches that of fully supervised methods on many counting benchmark datasets while relying on data that can be collected with fraction of the annotation burden.
INTRODUCTION
Object counting is a popular computer vision problem that involves localizing and quantifying the number of objects within an image, which has broad applicability across several domains. For example, counting problems have been explored throughout plant analysis (David et al. 2020 [021]]
Häni et al. 2020 Bargoti & |nderwood[[01 2017 MMinervini et al. [2015]Teimour et al.] [2018), wildlife
population monitoring (Arteta et 2016), crowd surveillance (Change Loy al. [013 Chen et al.)
2013 Loy et al. 2013 Chen et al.] 2012, et al.
2016 2015). tissue sample analysis (Marsden al. 2018, Paul Cohen et al. 2017 Xie et al. 2018 Kainz et al. 2015), and vehicle surveillance
Stuerrero-Gómer Gómez-Ommezooo et al. 2015] De Almeida et al. 2015 Wang et al.] 2008). However, highlight two major problems that prevent counting problems from being widely applicable. First, there is no consensus on how to structure the optimization target. Different methods have approached the problem using variety of annotations, which include bounding boxes (Ren al. 2015 Redmon et al. 2016), global object counts (Chattopadhyar et al., (017). point maps (Laradji 2018), and density maps (Lempitsky & Zisserman 2010). Bounding boxes are popular optimization target for detection problems. However, these annotations tend to under-perform on counting problems, in particular, when scenes contain many highly occluded objects variety of scales (Chattopadhyay et al.) 2017). Density map annotations often perform better, but suffer from lack of information related object characteristics such as scale (Arteta et al. 2016). Second, as the quantity and density of objects in an image increase, all of the above annotation types become labour intensive to collect and prone to significant annotator noise (Arteta et al.) 2016). We circumvent the burden of object labelling by introducing a simple form of annotation that can be rapidly collected, and design a method to propose density maps from these annotations.
We propose using pairwise image ranking, binary valued annotation that orders image pairs based on per-image object counts. Previous work has demonstrated these annotations to be an effective training signal for semi-supervised counting problems (Liu et al. 2018). Whereas this work relied

---

Under review as a conference paper at ICLR 2022
on leveraging automatically collected intra-image relationships between the whole image and sub- image crops, we extend this idea further by proposing a method for learning to count and localize objects exclusively from inter-image pairwise image rankings. Our inter-image annotations carry weak but information rich training signal despite being quick to collect. Humans are adept at rapidly assessing which of two images has more objects of interest, and untrained annotators are capable of discriminating between two images at a ratio of 10:11 objects in under 0.75 seconds (Pica et al.) 2004 Halberda & Feigenson 2008). While previous optimization targets require an annotator to identity and count out each object, pairwise image rankings only require a single click from an an- notator. Tables 3 and I we evaluate the annotation burden for recent object counting strategies and demonstrate that our annotation strategy carries only fraction of the burden. Given these sav- ings, there is significant value developing methods that learn to extract counts and locations from this type of training data.
The major technical challenge in exploiting this type of annotation finding a way to extract counts and locations given only pairwise rankings. In this work, we propose adversarial strategy for regularizing the penultimate representation of a ranking network to have the properties of density map by comparing it to pseudo-densin map distribution. This places soft integer and localization constraint on the representation. We argue that this strategy forces the model to solve the ranking sub-task by detecting re-occurring objects that satisfy all of the ranking constraints. Combined with the weak quantity signal provided by pairwise image ranking annotations, this strategy pro- duces a counting and localization model that is competitive with architectures trained on images with their density maps and global counts annotations. In summary, our work makes the following contributions:
1. We propose object-based pairwise inter-image ranking a novel, low-cost annotation strat- egy for weakly supervised counting and localization, and demonstrate that it performs com- parably with fully supervised counting methods.
2. We propose adversarial density map regularization, method for enforeing that the network output has the properties of a density map.
PREVIOUS WORK
Object Counting with Limited Data. Object counting methods perform best when learning from
density maps (Lempitsky & Zisserman 2010 Chattopadhyay et al. (017)). which carry a high anno- tation burden. Several methods have emerged that attempt to eliminate this burden. These methods can be loosely split into 4 categories semi- cuniisuuervised, knowledge transfer, sample selection, and weakly supervised methods.
Semi-supervised counting methods alleviate the annotation burden by including additional unla- belled data; a recent approach (L.iu et al. 2019b) 2018) introduced an unsupervised ranking loss that exploits the fact that any image has as many or more objects than any cropped sub-portion of that image. Other methods (Sam et al. 2019) have proposed more general feature learning strategies. Knowledge transfer methods explore transferring features between counting problems; one method (Zhang et al. 2013) proposed learning from a multi-modal dataset which contained mix of density maps and global object counts. A more recent approach (Ranjan et al. (021)) proposed few-shot learning strategy where the model was trained using exemplar and density map pairs such that it could be extended to novel object classes. Active learning methods (Ranjan et al., 2020 Zhao et al.) 2020) approach the problem by finding ways to only label the most important examples in new domain. Weakly supervised methods have explored strategies for better utilizing global counts as annotations. Recent methods (Yang et al., 2020 Lei al. 2021) proposed various regularization terms. One such method introduced soft-sorting multi-task loss (Yang et al.] 2000), which involved learning from global object count annotations directly and indirectly through soft-sorting task. An- other recent work (Liang et al.) 2021) has suggested that self-attention can improve learning from global counts. A different approach (Cholakkal et al. 2019) involved using object recognition com- bined with global object counts create class-specifie density maps. Our method differs from the above weakly-supervisee methods by learning exclusively from a much weaker signal than global object counts.
2

---

Under review as a conference paper at ICLR 2022
Peakkl-Superviser Object Localization. Related to weakly object counting is object localization (WSOL) (Zhang et al., 2021). This task involves localiz- ing objects using only some weak image-level label. Arguably, the most important WSOL methods are class activation maps (CAM) (Zhou et al. 2016) and Grad-CAM (Selvaraju et al. 2017). The authors of CAM discovered that using global average pooling in classification network allowed them to localize parts of the object being classified. Grad-CAM extends this work by re- re-weighting the activation map using the gradients of some target class. Separately, image region erasing has be- come a popular strategy for learning localized features from global class labels. Some works (Singh & Leej 2017: Choe & Shim 2019) randomly erase image regions, forcing the network to rely on the whole object to classify the image. Other works (Choe & Shim 2019; Zhang et al. 2018 8 Kim ett al., 2017 Wei et al. 2017) proposed adaptive erasing strategies for removing the image regions that contribute most to classification, forcing the network to explore other regions in an attempt to accurately classify the image. Our problem shares similarities with WSOL in that we are using a global image label, albeit a pairwise image ranking label and not image class label, to detect objects.
Counting, Detection, and Output Representations. Localizing and counting target objects in images is an important computer vision problem. However, there is no consensus on the appropriate optimization target for this task. Bounding boxes (Ren al., 2015: Redmmnn et al. (016) are a popular approach, but regressing over box coordinates is difficult problem that underperforms on counting tasks (Chattopadhyay et al. 2017). Density maps (Lempitsky & Zisserman 2010) are frequently used as alternative, but, in their basic form, suffer from lack of information about object scale and scene geometry. To remedy this, different strategies for producing adaptive density maps have been explored. Self-attention (Wan & Chan 2019) for selecting the best density map scale has worked for crowd counting problems. Similarly, ad-hoc geometric priors {Zhang et al., 2016) have been used to fuse scene geometry and density maps. Other methods have explored finding object blobs from dot maps (Laradji et al. [2018); multi-scale and multi-resolution density maps (Idrees et al., 2018); and learning directly from dot maps by treating density map estimation as perspective-guie guudeed unbalanced optimal transport problem (Wan et al. 2021
Ranking as Supervision The use of ranking as a training signal, sometimes known as ordinal regression, originates within information retrieval research. RankNet (Burges et al.] 2005). a doc- ument retrieval network, emerged as the first deep learning approach to ranking. However, this approach has been extended into several computer vision applications. Facial age estimation (Chen) et al. 2018 Lim et al. 2020) has benefited from pairwise image rankings, which learn the 'amount' of age an image. Pairwise image ranking has been used to localize facial attributes (Singh & Leep compare agent skills in videos (Doughty et al. [018). and detect video highlights (Yao et al.,
2016),
20161. Ranking has also been used for ranking object proposals and class label proposals (Li et al.,
2017
Liu et al. 2021). These applications highlight that ranking as an optimization signal plays significant role in computer vision applications.
3 PAIRWISE RANKING AS WEAK SUPERVISION
Pairwise image ranking is a task which requires an annotator to provide an ordering for image pairs based on the quantity of objects present in the image. Given some dataset
Drank = {( 2z), ru = c 2 C
we would like to extract the underlying object counts C2 and ej given only the image ranking con- straints ray for N pairs of images (,,,,r,). The goal of this paper is to develop strategy to use the implicit weak object quantity signal present in Drank to solve the counting task. However, we first provide motivation for this strategy by exploring the limitations of other annotation strategies and the strengths of pairwise image ranking.
Density maps are the most popular annotation strategy for solving counting problems. Here, a den- sity map is defined as a heatmap where a Gaussian kernel is convolved with point-wise annotations, with the property that the density map integrates to the global count. This annotation formulation is useful because it provides a location- based target for the object counting problem but has less complexity than bounding-box annotations. However, collecting point-wise annotations presents
3

---

Under review as a conference paper at ICLR 2022
Pseudo Density Map Distribttin
Generator, Shared Weights
Estimated Density Maps
ResNer50
Upsample
Discriminator
Lygan
conce
x
ResNet50
Upsample
Arank
Figure 1: Method overview. Pairs of images with object count ranking labels used train neural network to count and localize objects. Each image X; and Tÿ are passed to the generator model and their respective outputs 2; and zj are used calculate Lrank, which provides the weak object quantity training signal. In addition to this, we include an adversarial density map generation loss. Lyan. which encourages the output of the generator model have the properties density map.
significant challenges. Strategies that learn directly from global object counts are able to reduce the annotation burden associated with density maps, but these methods still carry their own significant annotation burden. Global counting annotations and point-wise annotations require an annotator to visually localize all objects and record those labels. This be time consuming and error prone, pecially as object quantity, occlusion, and density increase. In fact, it has been demonstrated (Arteta al. 2016) that when multiple annotators label an image with point-wise annotations, the annota- tors often suffer from fatigue or make inconsistent decisions when faced with ambiguity and make regular errors that lead significant signal noise. Given the annotation burden that exists with all of the above methods, it would be valuable to establish a weak signal that could be rapidly collected by annotators while still leading to similar performance on the counting problem.
Researchers in human psychology have found that humans can rapidly assess which of two groups of objects has the most objects the difference object count is greater than particular ratio. This ratio is approximated by Weber-Fechner law, which is law that describes the change in stimulus necessary for some human to perceive the difference relative to the existing stimulus. For the task of pairwise image ranking, researchers (Pica et al.) 2004 Halberda & Feigenson 2008) have found that within 0.75 seconds, untrained adults are capable of determining which of two images has the most objects if the object count ratio is smaller than approximately 9:10 and 10:11 (independent of the absolute count). This suggests that, for untrained adults, the task of rapidly annotating image pairs with their respective rank should be relatively simple as long the ratio between object counts within the image pairs remains smaller than the given ratio. Thus, given a weak signal that can be rapidly collected, we are left answer the difficult question of how extract the counts.
4
METHOD
The goal of our method is to develop model which can extract object counts given only pairwise image rankings. These annotations contain weak signal for object quantity, which we use to train neural network, leamn representation from which we can extract global object counts. We solve this problem by proposing adversarial density map estimation, strategy which structures the output of to have the properties of a density map. With this strategy, our model learns to count objects and even localize those objects within image.
fzz; 0), outlined in Figure [ However, this target alone is not enough to
4

---

Under review as a conference paper at ICLR 2022
4.1 RANKING NETWORK
The purpose of the network f( 0) is to extract the underlying weak object quantity signal from the pairwise image ranking annotations. As outlined in Figure [ our base counting model So receives two images as input, 24 and 2,, and outputs representations 2, E [0, 1] 1]56x66. We use these representations model the probability that & Cj by approximating the true distribution as follows:
Prank P(rsslasz of E (zt) (z))
where a is the sigmoid operation and (k. 1) are the indices for the representations and 2j. Here, benefit from the fact that when the difference between 2; and 2j is positive, the sigmoid operation outputs value greater than 0.50. Whereas when the difference is negative, the sigmoid operation outputs a value less than 0.50. This allows us to model which of two images in pair has more objects by inspecting the magnitudes of the sum over 2, and 2j. Thus, by optimizing O using the following loss function:
Crank Epaaca Dog(Pran))]
(2)
the model must learn to minimize the number of pairwise inversions (from the ground truth distri- bution Pdota) among all ranking examples in the training dataset, which creates a partial ordering of all the images by object count. However, this ordering is decoupled from any notion of object identity and location. In the next section, we propose an approach to explicitly connect the output representation to object locations.
4.2 ADVERSARIAL. DENSITY MAP GENERATION
Previous empirical results have demonstrated the value of density maps location-based annota- tion for counting problems. Density maps are structured such that they place Gaussian density where objects occur and integrate the global count. These properties are useful because they explicitly connect the detection and counting task. While we do not have access to density maps, we argue that allowing the counting network fo to propose density maps when solving the pairwise image ranking problem captures some of the useful properties of density maps. We explore a strategy for structuring the output representation 2; have these properties.
We start by establishing pscudo point map distribution from which we can randomly sample point maps Zpeint E (0, 1] 166x66 We convolve 2point with K3, 2D kernel with width 6, to generate a pseudo density map:
Zamap
Kg Zgoint.
(3)
an appropriate sampling distribution is chosen, then we can establish an adversarial training objec- tive that penalizes the network output 2; when it deviates from the properties of a density map. We first describe the adversarial training objective, and then describe methods of selecting appropriate pseudo density map distributions.
Adversarial training (Goodfellow et al. 2014) a widely adopted technique for modeling the un- derlying generating distribution that explains dataset. This training strategy involves optimizing two neural networks, our counting network f (termed the generator) and a discriminator D. The generator is tasked with generating samples from an underlying distribution and the discriminator is tasked with evaluating whether a sample came from the pseudo point map distribution or the generator's 's distribution. The generator is optimized using feedback from the discriminator, and the discriminator is optimized using the samples from the generator. Under this training paradigm, the discriminator D 56x56 -> IR predicts whether its input a sample from the pseudo-density map distribution. We use the LS-GAN objective function (Mao et al. 2017), which is given as:
CJ cg -Eer [D(So(xs) 1)?],
(4)
for the generator, and:
gan Eza.n [(((smmpp) 1)2]
(5)
+ E, [x(fo(x))
5

---

Under review as a conference paper at ICLR 2022
Selecting point map distribution is challenging problem, as it is known that convolution neural networks can encode spatial information (Islam et al. 2020). Given this, discriminator can penalize the underlying spatial distribution of objects learned by the generator. Given this situation, choose to sample point maps uniformly. To produce the pseudo point map distribution, first, we uniformly sample total count, Cpscudos for the number of Gaussian blobs in a particular density map:
Cpseudo ~ N.).N.):
(6)
where Ne the estimated maximum object count for the dataset. Then, we uniformly sample Cpeeudo co-ordinates:
i,j U |0.56] x x(0,66),
which gives us pooint- This uninformative prior makes more difficult for the discriminator to exploit the underlying spatial distribution, as any point location is equally likely.
4.3 IMPLEMENTATION DETAILS
Architecture. We use ResNet50 (He al. 2016) as the default architecture for the underlying counting model fe. The ResNet50 model shares parameters across both branches of the network in Figure We generate a density map 2; in a similar fashion to previous work on segmentation that use fully convolutional networks (Long et al.) (013). First, we pass an image through ResNet50 and collect features at different resolutions throughout the network. Then, we up-sample the features using transposed convolutions and combine these features to form To construct the ranking network, we pass two images through fe and then we take the sum of the difference between the respective density map estimates before passing the final number through sigmoid activation whose output is used as the predicted image ranking.
56x 56 density map estimate.
Adversarial Setup. The density map estimates, 2; and Ej. are passed to the discriminator: The discriminator predicts whether not the resulting density map estimates look like samples of the pseudo density map distribution. The discriminator itself is designed to have a relatively simple architecture, comprised of 5 convolutional layers with 4 filters and LeakyReLU activations that downsample the density map. Then, these features are passed through a final fully connected layer.
Training. Each model for each experiment is trained for 200 epochs using the Adam optimizer with batch size 32. The learning rate is randomly selected between 10 and 10 with 5x 10-5 being reliable choice. Similarly, the 31 parameter of the Adam optimizer from a normal distribution, with 0.50 being the most reliable choice. The optimizer hyperparameters shared by both the ranking architecture and the discriminator. Each image is augmented using random combination of horizontal flipping, rotation, colour jitter, and noise such that no objects are lost and such that the object identity preserved. Due the instability of GANs, we multiple experiments and select the best performing model using the validation set.
randomly selected
EXPERIMENTS
5.1 DATASETS
We benchmark our results four object counting datasets, which we describe here. TRANCOS Cluerrero-Gōmez-OImme
et al. 2013) is a 2015 vehicle counting benchmark dataset containing .244 images with point map annotations for 46,796 highly occluded vehicles in traffic. Penguins Arteta et al., 2016) is a 2016 animal counting benchmark dataset containing around 82,000 images of penguin colonies taken under a diverse set of environmental conditions across 40 different loca- tions. Each image contains several dot maps from different annotators. The UCSD dataset (Chan) & Vasconcelos 2008 Chan et al.) 2008) is a 2008 crowd counting dataset containing 2,000 video
frames of pedestrian traffic across campus. The Mall dataset (Change Loy et al.) 2013 Chen al.)
2013 Loy et al. 2013, Chen et al. 2012) is a 2013 crowd counting data containing 2,000 video frames of pedestrian traffic in shopping malls, with over 60,000 human heads annotated across the dataset. All of these datasets are challenging benchmarks as they contain highly occluded objects in dense scenes with a variety of environmental conditions and scales.

---

Under review as a conference paper at ICLR 2022
Figure 2:
Qualitative examples of density maps predicted by the baseline model at inference time.
Top: Trancos. Middle: Penguins. Bottom: MALL
Ablation of results for the baseline model modified by removing different components to reveal their underlying contribution to the model
Table I:
Method
Trancos
Mall
Penguins
UCSD
MAE
R
MAE
R²
MAE
Ra
MAE
R²
Drank + Laaan 5,89
0.76
3.62
0.41
7.43
0.52
5.04
0.47
Lgan 13.19
-0.15
4.80
0.00
13.70
-0.08
7.70
-0.45
Crank 9.47
0.51
5.46
-0.14
7.72
0.58
7.04
-0.15
5.1.1 SAMPLING IMAGE RANKING DATA
There are presently no well-established image ranking datasets available for weakly supervised ob- ject counting and localization benchmarking. Given this, all image ranking datasets used for evaluat- ing our experiments must be curated. We experiment with the object counting and detection datasets outlined above, and reformulate all of the available datasets as ranking datasets as follows. Given counting dataset
D count {2r IR... ,c E NJ
where the number of channels and e; the object count image He, we randomly sample N 2, 000 image pairs (z,,,,) and calculate their pairwise ranking as ri 2 cz- This
provides us with a curated ranking dataset:
Drank ((r,, (riz)n (c: 2g))
For our experiments, we impose no constraints on the sampling procedure and generate the training ranking dataset by simply uniformly sampling examples from the training dataset. We benchmark against the original counting test data provided with each dataset.
5.2 RESULTS
Ablation of Model Components. In Table explore the contribution of each model compo- nent to the test error. We evaluate each experiment using MAE and R². MAE is the mean absolute emor and measures the difference between the predicted count and ground truth count, with lower

---

Under review as a conference paper at ICLR 2022
corresponds to better performing model. R is the coefficient of determination, which de- scribes how well the model fits the data with a higher score corresponding to a better fit. When only find that the model fails to detect and count objects. Lrank is included during model training, to the model trained with both Crank and aan This result demonstrates that Cgan contributes an important training signal when solving the object counting problem using pairwise image rankings.
Lgan is included during model training, This result is intuitive, as the weak quantity signal is only provided by Lrank. However, when only find that the model under-performs when compared
Evaluating the Annotation Burden vs. Error Trade-Off. We compare our results with state- of-the-art object counting methods and provide an estimate of the annotation burden for each
Table 2: Comparison of the test error and annotation time for state-of-the- the--rte counting methods on the TRANCOS crowd counting dataset. When methods use the standard dot-map training set, their annotation times are equivalent.
Method
Supervision
Est. Annotation Time
MAE
Hydra CCNN (oñro-RRbio & López-Sastre
2016
Dot-map
9 hr 11 min
10.99
FCN-MT
Zhang et al.
2017b
Dot-map
9 hr 11 min
5.31
FCN-HA Zhang et al. 2017a)
Dot-map
9 hr 11 min
4.20
LC-PSPNet (Laradji et al.
2018)
Dot-map
9 hr 11 min
3.57
CSRNet (Li et al.
2018)
Dot-map
9 hr 11 min
3.56
SPN (Chen et al.
2019)
Dot-map
9 hr 11 min
3.35
ADSCNet Bai al.
2020
Dot-map
9 hr 11 min
2.60
Glance Chhttoopadyya et al.
2017)
Counts
2 hr min
7.00
7 hr 54 min
Adv. Dmap (Ours)
Pairwise Rank
25 min
5.89
Table 3: Comparison of the test error and annotation time for state-of-the-art counting methods on the MALL crowd counting dataset.
Method
Supervision
Est. Annotation Time
MAE
CNN-Boosting (Walach
Wolf 2016)
Dot-map
7 hr 13 min
2.01
LC-PSPNet (Laradji et al. 2018)
Dot-map
7 hr 13 min
2.01
AL-AC (Zhao et al.
2020
10% w/ Dot-map
43 min
3.80
Adv. Dmap (Ours)
Pairwise Rank
25 min
3.62
Table 4: Comparison of the test error and annotation time for state-of-the-at counting methods on the USCD crowd counting dataset.
Method
Supervision
Est. Annotation Time
MAE
ADCrowdNet (Liu al. 2019a
Dot-map
hr 59 min
0.98
Sorting Yang et al. 2020
Counts
hr 00 min 5 hr 54 min
1.80
Adv. Dmap (Ours)
Pairwise Rank
25 min
5.04
8

---

Under review as a conference paper at ICLR 2022
method. Here, we establish annotation burden estimate for dot-maps and global object counts calculated over the training and validation set and compare it to the estimated annotation burden of our method. To estimate the annotation burden for map-m,
use the per-object annotation time of 1.1 s established by Cholakkal et al. (020)) and multiply this by the number of objects in the dataset. To estimate the annotation time for global object counts, we use a slightly more complex formula which includes the human ability to rapidly count objects within the range of to 4, often referred as the subitizing range. /Saltzman & Garner (1948) established counting speed of 0.1 s for each object within the subitizing range and 0.35 for each additional object outside of the range. However, the participants this experiment were only asked to count simple shapes placed on a white background. Cholakkal et al. (2020) looked more closely at human counting of complex object classes realistic scenes and established counting speed of 0.5 within the subitizing range and 1.0 s for objects outside of this range. We use these two measures to create a range for our estimate of object counting speed. We calculate the annotation speed by analyzing the per-image count in each dataset and assigning a lower and upper bounds for the counting time of [0.1, 0.5] seconds for each object within the subitizing range and [0.35, 1.0] seconds for objects outside of the range. This provides our lower bound and upper bound for global object counting. To calculate pairwise image ranking speed, we use the per image-pair ranking time of 0,75 established
by Pica et al. (2004); Haaberda & Feigenson (2008) and multiply it by 2,000, which is the number of Baonge-pairss in our sampled ranking dataset.
Table comparess our method to previous state-of-the-art counting methods evaluated on the TRAN- COS dataset, where we find that our method performs similarly to the method proposed by Zhang al. (2017b), despite their method being supervised by dot maps requiring x22 the annotation time. More recent methods, such as the method proposed by Bai et al. (2020), outperform ours by a mean error of 3.29 vehicles per images, where each image contains an average of 38 vehicles. However, we find that our method requires 4.54% of the annotation time as the best performing fully supervised methods. We also find that our method outperforms Glance (Chattopadhyay et al.) 2017). which learns from image-level object counts, while also requiring smaller annotation bur- den (by a factor of 15,5% to 5.27%). Likewise, Table presents the same comparison evaluated on the MALL dataset. We find that our method performs comparably to current state-of-the-ae count- ing methods (our method's s MAE differs by -0.18 to 1.61 object count), while requiring a fraction (5.77% to 5.81%) of the annotation time. Further, find that our method outperforms the method proposed Zhao et al. (2020), which was specifically developed to deal with the annotation burden, while still only requiring a fraction (5.81%) of the annotation time. In Table 4 evaluate our method on the USCD crowd dataset. We find that our method performs modestly when compared to state-of -h-the- method (our MAE differs by 3.24 4.06 objects). but still requires a significantly smaller annotation burden (by factor of 5.97% to 20.8% We argue that these results demonstrate the value of pairwise image-ranking as a weak object counting signal and the value of our method for extracting object counts while minimizing the annotation burden.
CONCLUSION
In this paper, we present a weakly supervised strategy for extracting object counts and locations from pairwise image rankings, which can be easily and rapidly collected by annotators. We demon- strate that our method performs well on various benchmarks and approaches fully supervised base- lines. This work highlights the value in exploring novel weak annotation formulations and provides direction forward for solving counting problems in domains where annotation collection would otherwise be impermissible.
7 ETHICS STATEMENT
Object counting methods are broadly useful computer vision methods that are considered highly valued across several industries (see Section I). However, object counting is also frequently used in human surveillance and thus lowering the burden necessary to develop new surveillance applications poses serious ethical concerns. In particular, tracking human movement can directly aid in military and carceral system applications, which can potentially empower unethical actors in the violation of fundamental human rights. While this should not inherently dissuade researchers from developing
9

---

Under review as a conference paper at ICLR 2022
new techniques for this application, it does call on the community remain vigilant about which end-users benefit from the technology.
REPRODUCIBILITY STATEMENT
we make an effort to detail network architecture, hyperparametters and training pa- rameters necessary to achieve the provided results. In sections [.1 and 5.1.1 we detail the bench- mark datasets used within our experiment and the strategy used to generate the ranking dataset. Additionally, we anonymously provide the source code necessary for reproducing the results.
In section 4.3
REFERENCES
C. Arteta, V. Lempitsky, and A. Zisserman. Counting in the wild. In European Conference on Computer Vision, 2016.
Shuai Bai, Zhiqun He, Yu Qiao, Hanzhe Hu, Wei Wu, and Junjie Yan. Adaptive dilated network with self-correction supervision for counting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Suchet Bargoti and James Underwood. Deep fruit detection in orchards. In 2017 IEEE International Conference on Robotics and Automation (ICRA, pp. 3626-3633. IEEE, 2017.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning to rank using gradient descent. In Proceedings of the 22nd International Con- ference on Machine learning, pp. 89-96. 2005.
Antoni B Chan and Nuno Vasconcelos. Modeling, clustering, and segmenting video with mixtures of dynamic textures. IEEE transactions on pattern analysis and machine intelligence, 30(5): 909-926, 2008.
Antoni B Chan, Zhang-Shemt ShenggShengaaae John Liang, and Nuno Vasconcelos. Privacy preserving crowd mon- Counting people without people models or tracking. In 2008 IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 1-7. IEEE, 2008
itoring:
Chen Change Loy, Shaogang Gong, and Tao Xiang. From semi-supervised to transfer counting of crowds. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2256- 2263, 2013.
Prithvijit Chattopadhyay, Ramakrishna Vedantam, Ramprasaath R Selvaraju, Dhruv Batra, and Devi Parikh. Counting everyday objects in everyday scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1135-1144, 2017.
Ke Chen, Chen Change Loy, Shaogang Gong, and Tony Xiang. Feature mining for localised crowd counting. In Bmvc, volume 1, pp. 2012.
Ke Chen, Shaogang Gong, Tao Xiang, and Chen Change Loy. Cumulative attribute space for age and crowd density estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2467-2474, 2013.
Shixing Chen, Caojin Zhang, and Ming Dong. Deep age estimation: From classification to ranking. IEEE Transactions on Multimedia, 20(8):2009-222.. 2018 doi: 10. 10.1109/TM..2217.777666
Xinya Chen, Yanrui Bin, Nong Sang, and Changxin Gao. Scale pyramid network for crowd count- ing. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV, pp. 1941 1950. IEEE, 2019.
Junsuk Choe and Hyunjung Shim. Attention-based dropout layer for weakly supervised object lo- calization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 2219-2228, 2019.
Hisham Cholakkal, Guolei Sun. Fahad Shahbaz Khan, and Ling Shao. Object counting and instance segmentation with image-level supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12397-12405, 2019.
10

---

Under review as a conference paper at ICLR 2022
Hisham Cholakkal, Guolei Sun, Salman Khan, Fahad Shahbaz Khan, Ling Shao, and Lue Van Gool. Towards partial supervision for generic object counting in natural scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Etienne David, Simon Madec, Pouria Sadeghi-Tehran, Helge Aasen, Bangyou Zheng, Shouyang Liu, Norbert Kirchgessner, Goro Ishikawa, Koichi Nagasawa, Minhajul Badhon, Curtis Poz- niak, Benoit de Solan, Andreas Hund, Scott C. Chapman, Frederic Baret, lan Stavness, and Wei Guo. Global wheat head detection (gwhd) dataset: a large and diverse dataset high-resolution rgb-labelled images to develop and benchmark wheat head detection methods. Plant Phenomics, 2020, 2020.
Etienne David, Mario Serouart, Daniel Smith, Simon Madec, Kaaviya Velumani, Shouyang Liu, Xu Wang, Francisco Pinto Espinosa, Shahameh Shafiee, Izzat S. A. Tahir, Hisashi Tsujimoto, Shuhei Nasuda, Bangyou Zheng, Norbert Kichgessner, Helge Aasen, Andreas Hund, Pouria Sadhegi-Tehran, Koichi Nagasawa, Goro Ishikawa, Sebastien Dandrifosse, Alexis Carlier, Benoit Mercatoris, Ken Kuroki, Haozhou Wang, Masanori Ishii, Minhajul A. Badhon, Curtis Pozniak, David Shaner LeBauer, Morten Lilimo, Jesse Poland, Scott Chapman, Benoit de Solan, Frederic Baret, Ian Stavness, and Wei Guo. Global wheat head dataset 2021: an update to improve the benchmarking wheat head localization with more diversity, 2021.
Paulo RL De Almeida, Luiz S Oliveira, Alceu S Britto Jr, Eunelson J Silva Jr, and Alessandro L Koerich. Pklot-a robust dataset for parking lot classification. Expert Systems with Applications, 42(11):4937-4949 2015.
Hazel Doughty, Dima Damen, and Walterio Mayol-Cuevas. Who's better? who's best? pairwise deep ranking for skill determination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ricardo Guerrero-Gómee Gómez-Olmedo, Beatriz Torre-Jiméne Jiméé-iméénz, Roberto López-Sastre, Saturnino Maldonado--asseot and Daniel Onoro-Rubio. Extremely overlapping vehicle counting. In Iberian Conference on Pattern Recognition and Image Analysis, pp. 423-431. Springer, 2015.
Justin Halberda and Lisa Feigenson. Developmental change the acuity of the" number sense": The approximate number system in 3-, 4-, 5-, and 6-year-olds and adults. Developmental psychology, 44(5):1457, 2008
Nicolai Häni, Pravakar Roy, and Volkan Isler. Minneapple: a benchmark dataset for apple detection and segmentation. IEEE Robotics and Automation Letters, 5(2):852-858, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Haroon Idrees, Muhmmad Tayyab, Kishan Athrey, Dong Zhang, Somaya Al-Maadeed, Nasir Ra- jpoot, and Mubarak Shah. Composition loss for counting, density map estimation and localization in dense crowds. In Proceedings of the European Conference on Computer Vision, pp. 532-546, 2018.
Md. Amirul Islam, Sen Jia, and Neil D. B. Bruce How much position information do convo- lutional neural networks encode? In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreww
Philipp Kainz, Martin Usschler, Samuel Schulter, Paul Wohlhart, and Vincent Lepetit. You should use regression to detect cells. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pp. 276-283. Springer, 2015.
Dahun Kim, Donghyeon Cho. Donggeun Yoo, and In So Kweon. Two-phase learning for weakly supervised object localization. In Proceedings of the IEEE international Conference on Computer Vision, pp. 2017.
11

---

Under review as a conference paper at ICLR 2022
Issam H. Laradji, Negar Rostamzadeh, Pedro O Pinheiro, David Vazquez, and Mark Schmidt. Where are the blobs: Counting by localization with point supervision. In Proceedings of the European Conference on Computer Vision, pp. 547-562, 2018.
Yinjie Lei, Yan Liu, Pingping Zhang, and Lingqiao Liu. Towards using count-level weak supervision for crowd counting. Pattern Recognition, 109:107616, 2021.
Victor Lempitsky and Andrew Zisserman. Learning to count objects images. Advances in neural information processing systems, 23:1324-1332 2010.
Yuhong Li, Xiaofan Zhang, and Deming Chen. Csmnet: Dilated convolutional neural networks for understanding the highly congested scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1091-1100, 2018.
Yuncheng Li, Yale Song, and Jiebo Luo. Improving pairwise ranking for multi-label image classi- fication. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, July 2017.
Dingkang Liang, Xiwu Chen, Wei Xu, Yu Zhou, and Xiang Bai. Transcrowd: Weakly-supervised crowd counting with transformer. arXiv preprint arXiv:2104.09116, 2021.
Kyungsun Lim, Nyeong-Ho Shin, Young-Yoon Lee, and Chang-Su Kim. Order learning and its
application to age estimation. In International Conference on Learning Representations, 2020.
net forum? Hid.uummddd-ggyaauaaaauaaa URL https::///// pettps::/openreviiet wtwprevwweeeeeetooumniiiiee...dd
Ji Liu, Dong Li, Rongzhang Zheng, Lu Tian, and Yi Shan. Rankdetnet: Delving into ranking constraints for object detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 264-273, June 2021.
Ning Liu, Yongchao Long, Changqing Zou, Qun Niu, Li Pan, and Hefeng Wu. Aderowdnet: attention-ijjectivee deformable convolutional network for crowd understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3225-3234, 2019a.
Xialei Liu, Joost Van De Weijer, and Andrew D) Bagdanov. Leveraging unlabeled data for crowd counting by learning to rank. In Proceedings of the IEEE Conference on Computer Vision and Pastern Recognition, pp. 7661-7669, 2018
Xialei Liu, Joost Van De Weijer, and Andrew Bagdanov. Exploiting unlabeled data in enns by self-supervised learning to rank. IEEE transactions on pattern analysis and machine intelligence, 4168):1822-1878, 2019b.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic seg- mentation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3131-3333444 2015.
Chen Change Loy, Ke Chen, Shaogang Gong, and Tao Xiang. Crowd counting and profiling: Methodology and evaluation. In Modeling, simulation and visual analysis of crowds, pp. 347- 382. Springer, 2013.
Xudong Mao, Qing Li. Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Confer- on Computer Vision, pp.
2017.
Mark Marsden, Kevin McGuinness, Suzanne Little, Ciara Keogh, and Noel E O'Cnnor.. People, penguins and petri dishes: Adapting object counting models to new visual domains and object types without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
Massimo Minervini, Andreas Fischbach, Hanno Scharr, and Sotirios A. Tsaftaris. Finely-graines annotated datasets for image-based plant phenotyping. Pattern Recognition Letters, pp. -, 2015. ISSN 0167-8655. doi: http:/ukk.bbooo/.o.ooooopccpooop
URL http: //www. www.
sci encedi rect http:/cco.cc.ceeeeeeeeeeec httpa//tttttttteeeeeeoeaaaaaaaaa
12

---

Under review as a conference paper at ICLR 2022
Daniel Oñoro-Rubio and Roberto J. López-Sastre. Towards perspective-free object counting with deep learning. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 615-629, Cham, 2016. Springer International Publishing. ISBN 978-3-319- 46478-7.
Joseph Paul Cohen, Genevieve Boucher, Craig A Glastonbury, Henry Z Lo, and Yoshua Bengio. Count-ception: Counting by fully convolutional redundant counting. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 18-26, 2017.
Pierre Pica, Cathy Lemer, Véronique Izard, and Stanislas Dehaene. Exact and approximate arith- metic in an amazonian indigene group. Science, 306(5695):499-555 503,, 2004.
Viresh Ranjan, Boyu Wang, Mubarak Shah, and Minh Hoai. Uncertainty estimation and sample selection for crowd counting. In Proceedings of the Asian Conference on Computer Vision, 2020.
Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning to count everything. In Pro- ceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 3394 3403, 2021.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pp. 779-788. 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Saltzman and WR Garner. Reaction time as measure of span of attention. The Journal of psychology, 25(2):227-241, 1948.
Deepak Babu Sam, Neeraj N Sajjan, Himanshu Maurya, and R Venkatesh Babu. Almost unsuper- vised learning for dense crowd counting. In Proceedings of the AAAI Conference on Artificial 2019.
Intelligence, volume 33, pp.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference Computer Vision, pp. 618-626, 2017.
Krishna Kumar Singh and Yong Jae Lee. End-to-end localization and ranking for relative attributes. In European Conference Computer Vision, pp. 753-769. Springer, 2016.
Krishna Kumar Singh and Yong Jae Lee. Hide-and-seek: Forcing network to be meticulous for weakly-supervised object and action localization. In 2017 IEEE international Conference on Computer Vision (ICCV, pp. 3544-3553. IEEE, 2017.
Nima Teimouri, Mads Dyrmann, Per Rydahl Nielsen, Solvejg Kopp Mathiassen, Gayle J. Somerville, and Rasmus Nyholm Jørgensen. Weed growth stage estimator using deep convo- lutional neural networks. Sensors, 18(5), 2018. ISSN 1424-8220. doi: 10.3390/s18051580 URL https: www. mdpi. com/ 1424 82224422222//2155500
Elad Walach and Lior Wolf. Learning to count with enn boosting. In European Conference on Computer Vision, pp. 660-676. Springer, 2016.
Jia Wan and Antoni Chan. Adaptive density map generation for crowd counting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1130-1139, 2019.
Jia Wan, Ziquan Liu, and Antoni B Chan. A generalized loss function for crowd counting and lo- calization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 1974-1983, 2021.
Xiaogang Wang, Xiaoxu Ma. and W Eric L Grimson. Unsupervised activity perception in crowded and complicated scenes using hierarchical bayesian models. IEEE Transactions on pattern anal- ysis and machine intelligence, 31(3):539-555 -555,, 2008
13

---

Under review as a conference paper at ICLR 2022
Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. OB- ject region mining with adversarial erasing: A simple classification to semantic segmentation approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1568-1576, 2017.
Weidi Xie, J Alison Noble, and Andrew Zisserman. Microscopy cell counting and detection with fully convolutional regression networks. Computer methods in biomechanics and biomedical engineering: Imaging & Visualization, 6(3):283-29) 292, 2018.
Yifan Yang. Guorong Li, Zhe Wu, Li Su, Qingming Huang, and Nicu Sebe. Weakly-supervised crowd counting learns from sorting rather than locations. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIIII 16, pp. 1-17. Springer, 2020.
Ting Yao, Tao Mei, and Yong Rui. Highlight detection with pairwise deep ranking for first-person video summarization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2016.
Cong Zhang, Hongsheng Li, Xiaogang Wang, and Xiaokang Yang. Cross-scent crowd counting via deep convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 833-841, 2015.
Dingwen Zhang, Junwei Han, Gong Cheng, and Ming- Hing--suann Yang. Weakly supervised object local- ization and detection: survey. IEEE transactions on pattern analysis and machine intelligence, 2021.
Shanghang Zhang, Guanhang Wu, Joao P Costeira, and José MF Moura. Fen-rlstm: Deep spatio- temporal neural networks for vehicle counting in city cameras. In Proceedings of the IEEE Inter- national Conference on Computer Vision, pp. 3667-3676, 2017a.
Shanghang Zhang, Guanhang Wu, Joao Costeira, and Jose MF Moura. Understanding traffic density from large-scale web camera data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5898-5907, 2017b.
Xiaolin Zhang, Yunchao Wei, Jiashi Feng. Yi Yang, and Thomas S Huang. Adversarial complemen- tary learning for weakly supervised object localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1325-1334, 2018.
Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd count- ing via multi-column convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 589-597, 2016.
Zhen Zhao, Miaojing Shi, Xiaoxiao Zhao, and Li Li. Active crowd counting with limited supervi- sion. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pp. 565-581. Springer, 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921-2929, 2016.
14


Reviews:
[{"review_id": "s3OkcFcWcZI", "reviewer": "Reviewer_yR54", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The weakly supervised setting could reduce the annotation cost and still achieve acceptable performance in several object counting benchmarks under specific conditions .\n\n2. The proposed adversarial density map regularization technique could further improve the performance.\n\nWeakness:\n1. Novelty: The main contribution of this work is to use ranking information in object counting, as explained by the authors, ranking information is already used in previous object counting work[1,2], this work only extends  it to inter-image setting, which requires extra annotation while the original intra-image ranking is purely self-supervised[1,2] without extra annotation information ( in [1,2] the supervised loss is also used but the ranking loss itself is purely self-supervised ). Therefore, the novelty of this paper seems to be trivial. \n\n2. Supervision: The pairwise ranking weak supervision can be highly unreliable in real world applications. For example for two images with both people and trees, and the first image contains both more people and trees than the other one, therefore, the ranking information can be vague as the model may not know which object to be counted, people or trees?\n\n3. Adversarial Density Map: the point maps are sampled uniformly thus does not really capture the location information, I am a bit surprised that such pseudo point map could really help as it does not bring extra information to the network.\n\n4. Annotation Cost: This pairwise ranking supervision has less annotation cost if the number of pair used is similar as the number of images in the dataset. However, if we have 500 images in the training set and there are can be up to 124,750 different pairs, then the most efficient way to obtain the ranking information is to annotate object number of all the images. Besides, if the object number is similar, it is hard to get the ranking information. Therefore, I think this setting does not generalize well in a broader setting.\n\n5. Experiments: Missing evaluation with dense crowd counting benchmarks, for example ShanghaiTech PartA, UCF_QNRF, NWPU and UCF_CC_50 that are often followed by previous work and could be a good example to show the generalization of the proposed approach. Without such experiments, we cannot judge the generalization of the proposed approach.\n\n\n\n\n\n[1]Xialei Liu, Joost Van De Weijer, and Andrew D Bagdanov. Leveraging unlabeled data for crowd counting by learning to rank. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7661–7669, 2018.\n\n[2]Xialei Liu, Joost Van De Weijer, and Andrew D Bagdanov. Exploiting unlabeled data in cnns by self-supervised learning to rank. IEEE transactions on pattern analysis and machine intelligence, 41(8):1862–1878, 2019.\n\n\n\n\n\n\n\n\n\n------------------------------Post Rebuttal Comments--------------------------------------------------------------------------------------\n\n\n\nI would like to thank the authors for providing the rebuttals, however, as explained in the post rebuttal comments, it does not solve my concerns and actually shows the limitation of this work. I therefore choose to keep my original ratings.", "summary_of_the_paper": "In this work, authors propose to learn to count objects in weakly supervised setting where only pairwise ranking information is used as supervision. The annotation cost is expected to be much lower than the widely-used dot annotation. Besides, an adversarial density map regularization method is proposed to enforce the output remain the properties of a density map.", "summary_of_the_review": "This paper propose an interesting setting of pairwise ranking information as weak supervision to count objects. However, as I mentioned in the main review part, there are several limitations in both methodology and experiments. I hope the authors could address my concerns listed above and I think we could not accept this paper at current stage.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MHRFYf5T1CX", "reviewer": "Reviewer_GhpV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contributions of the paper are important and interesting, the paper is well written and easy to follow, and the experiments seem to show the advantages of using the adversarial and rank loss. I still have few concerns that needs to be addressed:\n\n1) Using GAN to learn structuring the output is interesting and important aspect of the paper. My question is if this approach has been used before in any previous work? Specifically, using GAN to structure the predictions like heatmap or segmentation mask where the \"real\" examples for training the GAN are synthetically generated.\n\n2) I do not understand why only 2K pairs are sampled from each dateset? This is very small for all the datasets but specifically for Penguins with 82K images. If this is a saturation point for the accuracy it should be clearly mentioned in the paper otherwise the authors should provide the effect of increasing N in the final performance for some of these datasets.\n\n3) Related the point (2), I also think it is important to show the effect of reducing the training set size for the state-of-the art methods and see how much their performance drops if fewer images are labeled. At the minimum, author can compare their method to the state-of-the-art when both method uses datasets with equal *annotation time*. Only then one can judge if using weak ranking labels is beneficial for learning to count.\n\n4) What is the ratio of object counts in the sampled 2K pairs for each dataset (mean and std)? It is important to know this ratio is in the range in which human can easily do the ranking.", "summary_of_the_paper": "The paper propose a learning to rank surrogate approach as surrogate for the problem of counting and localization in dense senses. It is claimed that this labeling approach significantly reduces the labeling time and it sufficient to learn a dense map which can be used for counting and localization. In addition to the pairwise ranking loss, the paper use an adversarial training strategy to make the predictions look more like dense maps.", "summary_of_the_review": "Reviewer believes the paper has some interesting novelties but more ablation studies are necessary to support the points made in the paper. I am looking forward to reading the rebuttal and will adjust my score accordingly. I am willing to update my score to accept if the rebuttal addresses my concerns.\n\n\n----------------- Post Rebuttal -------------------------\n\nThanks for addressing most of my concerns. I increased my score to 6 with regard to the rebuttal. I think the paper could be stronger with more experiments including dataset size vs. accuracy plots which show where the accuracy plots saturate. \n\nMost of the points raised by the reviewers are just and important. However, I disagree with reviewer yR54 on the importance of using \"Adversarial Density Map\". As the reviewer mentioned, the problem of weakly supervised learning is typically ambiguous as the network might find to detect non-relevant features that still can decrease the final ranking loss. As this is also the case for other weakly supervised methods including object detection, I don't think the authors should be penalized for this. Instead, we need to focus on the novelties that can address some of these ambiguities. Adversarial density map loss put a meaningful constraint on the predictions to make them look like a mixture of gaussian among all the possible distributions that the network can generates. It is unlikely that it can solve all the ambiguities in the task but it should resolve some.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G--nV4nHjM3", "reviewer": "Reviewer_MGT7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe idea of using count-based ranking to train a weakly supervised object counting and localization method is interesting. The authors attempted to analyze the model performance vs the annotation burden. The overall writing is clear.\n\n\nWeaknesses:\n\nThe comparison of the annotation burden is not convincing. Experiments were conducted under different settings. For example, annotation costs are referred from different works under different settings. The ranking cost should be relevant to the difference between two images. While, the experiment was conducted using randomly sampled image pairs. We don't know the count differences. Overall, the cost comparison needs more evidence.\n\nThe technical contribution is also limited.\n\n\n------------post rebuttal--------------------\nAfter reading the authors's rebuttal, I decide to keep my original score as 3. This work is interesting, but needs to be improved.\n\nFirst, reducing annotation cost and improving model accuracy are the key motivations for weakly supervised localization. It is important to have a fair comparison between the proposed pair-wise ranking cost and other existing annotation costs to justify the reason of using the pair-wise ranking. The current version does provide some comparison, but it is not convincing. Different experimental settings may lead to significantly different cost estimations. I would suggest the authors do some annotation cost study in a fair setting.\n\nSecond, the performance comparison with existing methods is not convincing. As mentioned earlier, two factors (annotation cost and model accuracy) are used to evaluate weakly supervised methods. But, the annotation costs are all estimated using different sources. Besides, not only the ranking cost, but also the model optimization might be different given different pairs of ranked images. This paper only provides one set of randomly sampled image pairs, but does not analyzes how the model performance will be affected by the randomness. The authors mentioned that \"Weber-Fechner law, which states that a human's ability to perceive a difference between the magnitude of two stimuli is related to the ratio between those stimuli\". This is for the annotation cost, but empirical experiments are needed to verify if this is the case for model accuracy as well.", "summary_of_the_paper": "This paper proposed an object counting and localization method which uses pairwise image ranking information for training. A count-based ranking objective is proposed for model optimization and an adversarial density map generation strategy is introduced to regularize the features of the network. The method was evaluated with multiple datasets.", "summary_of_the_review": "The idea of using count ranking is interesting. The annotation cost comparison is not convincing and the work is lack of technical contribution. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Even though reviewers found some responses by the authors satisfactory, several concerns regarding the paper still remain. The authors are strongly encouraged to:

1) Explore how dataset size impacts accuracy.
2) Reason about annotation costs via empirical experiments.
3) Including benchmark datasets in experimental evaluations.
Decision: Reject