Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose a learning to rank surrogate approach as surrogate for the problem of counting and localization in dense senses. It is claimed that this labeling approach significantly reduces the labeling time and it sufficient to learn a dense map which can be used for counting and localization. In addition to the pairwise ranking loss, the paper use an adversarial training strategy to make the predictions look more like dense maps.
main_review: The main contributions of the paper are important and interesting, the paper is well written and easy to follow, and the experiments seem to show the advantages of using the adversarial and rank loss. I still have few concerns that needs to be addressed:

1) Using GAN to learn structuring the output is interesting and important aspect of the paper. My question is if this approach has been used before in any previous work? Specifically, using GAN to structure the predictions like heatmap or segmentation mask where the "real" examples for training the GAN are synthetically generated.

2) I do not understand why only 2K pairs are sampled from each dateset? This is very small for all the datasets but specifically for Penguins with 82K images. If this is a saturation point for the accuracy it should be clearly mentioned in the paper otherwise the authors should provide the effect of increasing N in the final performance for some of these datasets.

3) Related the point (2), I also think it is important to show the effect of reducing the training set size for the state-of-the art methods and see how much their performance drops if fewer images are labeled. At the minimum, author can compare their method to the state-of-the-art when both method uses datasets with equal *annotation time*. Only then one can judge if using weak ranking labels is beneficial for learning to count.

4) What is the ratio of object counts in the sampled 2K pairs for each dataset (mean and std)? It is important to know this ratio is in the range in which human can easily do the ranking.
summary_of_the_review: Reviewer believes the paper has some interesting novelties but more ablation studies are necessary to support the points made in the paper. I am looking forward to reading the rebuttal and will adjust my score accordingly. I am willing to update my score to accept if the rebuttal addresses my concerns.


----------------- Post Rebuttal -------------------------

Thanks for addressing most of my concerns. I increased my score to 6 with regard to the rebuttal. I think the paper could be stronger with more experiments including dataset size vs. accuracy plots which show where the accuracy plots saturate. 

Most of the points raised by the reviewers are just and important. However, I disagree with reviewer yR54 on the importance of using "Adversarial Density Map". As the reviewer mentioned, the problem of weakly supervised learning is typically ambiguous as the network might find to detect non-relevant features that still can decrease the final ranking loss. As this is also the case for other weakly supervised methods including object detection, I don't think the authors should be penalized for this. Instead, we need to focus on the novelties that can address some of these ambiguities. Adversarial density map loss put a meaningful constraint on the predictions to make them look like a mixture of gaussian among all the possible distributions that the network can generates. It is unlikely that it can solve all the ambiguities in the task but it should resolve some.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed an object counting and localization method which uses pairwise image ranking information for training. A count-based ranking objective is proposed for model optimization and an adversarial density map generation strategy is introduced to regularize the features of the network. The method was evaluated with multiple datasets.
main_review: Strengths:

The idea of using count-based ranking to train a weakly supervised object counting and localization method is interesting. The authors attempted to analyze the model performance vs the annotation burden. The overall writing is clear.


Weaknesses:

The comparison of the annotation burden is not convincing. Experiments were conducted under different settings. For example, annotation costs are referred from different works under different settings. The ranking cost should be relevant to the difference between two images. While, the experiment was conducted using randomly sampled image pairs. We don't know the count differences. Overall, the cost comparison needs more evidence.

The technical contribution is also limited.


------------post rebuttal--------------------
After reading the authors's rebuttal, I decide to keep my original score as 3. This work is interesting, but needs to be improved.

First, reducing annotation cost and improving model accuracy are the key motivations for weakly supervised localization. It is important to have a fair comparison between the proposed pair-wise ranking cost and other existing annotation costs to justify the reason of using the pair-wise ranking. The current version does provide some comparison, but it is not convincing. Different experimental settings may lead to significantly different cost estimations. I would suggest the authors do some annotation cost study in a fair setting.

Second, the performance comparison with existing methods is not convincing. As mentioned earlier, two factors (annotation cost and model accuracy) are used to evaluate weakly supervised methods. But, the annotation costs are all estimated using different sources. Besides, not only the ranking cost, but also the model optimization might be different given different pairs of ranked images. This paper only provides one set of randomly sampled image pairs, but does not analyzes how the model performance will be affected by the randomness. The authors mentioned that "Weber-Fechner law, which states that a human's ability to perceive a difference between the magnitude of two stimuli is related to the ratio between those stimuli". This is for the annotation cost, but empirical experiments are needed to verify if this is the case for model accuracy as well.
summary_of_the_review: The idea of using count ranking is interesting. The annotation cost comparison is not convincing and the work is lack of technical contribution.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, authors propose to learn to count objects in weakly supervised setting where only pairwise ranking information is used as supervision. The annotation cost is expected to be much lower than the widely-used dot annotation. Besides, an adversarial density map regularization method is proposed to enforce the output remain the properties of a density map.
main_review: Strength:
1. The weakly supervised setting could reduce the annotation cost and still achieve acceptable performance in several object counting benchmarks under specific conditions .

2. The proposed adversarial density map regularization technique could further improve the performance.

Weakness:
1. Novelty: The main contribution of this work is to use ranking information in object counting, as explained by the authors, ranking information is already used in previous object counting work[1,2], this work only extends  it to inter-image setting, which requires extra annotation while the original intra-image ranking is purely self-supervised[1,2] without extra annotation information ( in [1,2] the supervised loss is also used but the ranking loss itself is purely self-supervised ). Therefore, the novelty of this paper seems to be trivial. 

2. Supervision: The pairwise ranking weak supervision can be highly unreliable in real world applications. For example for two images with both people and trees, and the first image contains both more people and trees than the other one, therefore, the ranking information can be vague as the model may not know which object to be counted, people or trees?

3. Adversarial Density Map: the point maps are sampled uniformly thus does not really capture the location information, I am a bit surprised that such pseudo point map could really help as it does not bring extra information to the network.

4. Annotation Cost: This pairwise ranking supervision has less annotation cost if the number of pair used is similar as the number of images in the dataset. However, if we have 500 images in the training set and there are can be up to 124,750 different pairs, then the most efficient way to obtain the ranking information is to annotate object number of all the images. Besides, if the object number is similar, it is hard to get the ranking information. Therefore, I think this setting does not generalize well in a broader setting.

5. Experiments: Missing evaluation with dense crowd counting benchmarks, for example ShanghaiTech PartA, UCF_QNRF, NWPU and UCF_CC_50 that are often followed by previous work and could be a good example to show the generalization of the proposed approach. Without such experiments, we cannot judge the generalization of the proposed approach.





[1]Xialei Liu, Joost Van De Weijer, and Andrew D Bagdanov. Leveraging unlabeled data for crowd counting by learning to rank. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7661–7669, 2018.

[2]Xialei Liu, Joost Van De Weijer, and Andrew D Bagdanov. Exploiting unlabeled data in cnns by self-supervised learning to rank. IEEE transactions on pattern analysis and machine intelligence, 41(8):1862–1878, 2019.









------------------------------Post Rebuttal Comments--------------------------------------------------------------------------------------



I would like to thank the authors for providing the rebuttals, however, as explained in the post rebuttal comments, it does not solve my concerns and actually shows the limitation of this work. I therefore choose to keep my original ratings.
summary_of_the_review: This paper propose an interesting setting of pairwise ranking information as weak supervision to count objects. However, as I mentioned in the main review part, there are several limitations in both methodology and experiments. I hope the authors could address my concerns listed above and I think we could not accept this paper at current stage.
review_title: null

Meta-review: Even though reviewers found some responses by the authors satisfactory, several concerns regarding the paper still remain. The authors are strongly encouraged to:

1) Explore how dataset size impacts accuracy.
2) Reason about annotation costs via empirical experiments.
3) Including benchmark datasets in experimental evaluations.
Decision: Reject