Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of predicting density maps for counting given semi-supervised datasets.
It proposes a novel transformer-based architecture, a distribution-matching loss function for quantized density prediction, the use of two overlapping discretisations and a consistency loss for the two discretisations that is only applied for confident predictions in unsupervised images.
The method achieves strong results in both fully and partially supervised settings, achieving state-of-the-art results.
Some aspects of the design are demonstrated to be important through ablative experiments.
strength_and_weaknesses: **Strengths**

1. The distribution matching loss is nice.
1. It's neat to use the uncertainty of the predicted discrete distribution to obtain the mask. I like that the non-confident examples are ignored by the ECR loss.
1. The supervised loss gives a significant improvement over the baseline methods in the fully-supervised setting (Table 5, appendix).
1. To me, it was surprising that the ECR loss, despite being simple, gave a large improvement in the semi-supervised setting (Table 2).
1. The attention map visualizations look great (appendix).

**Weaknesses**

1. The authors argue that predicting a distribution is more credible and less noisy than predicting a single value. However, the targets are always one-hot and the distribution is simply used to parametrize the prediction via its expectation, so it seems like simply an alternative parametrization and loss for predicting a single value. Another plausible explanation is that this parametrization and loss simply improve learnability and/or generalization for training a deep network.
1. The loss in eq. 4 seems more general than the loss which is actually used, since $\mathbf{y}$ is always one-hot in practice. Can the loss be simplified for one-hot targets?
1. The transformer architecture was not well justified, either by arguments or by empirical evidence. It's not clear why it's necessary. The number of categories is fixed, so couldn't we just predict logits for the categories using a conv-net? The appendix includes some variations of decoders. What about the effect of completely removing the decoders and simply comparing the learnt density tokens to the patch features?
1. The first self-attention within the decoder seems like it could be removed completely? It always takes the same set of tokens as input. Why not just learn the output tokens directly?
1. Several other ablative experiments are missing. What is the effect of using ECR loss without the mask (or varying the threshold $\xi$)? What is the effect of setting $\omega = 0.5$ instead of using the max-norm?
1. To compare the supervised loss alone to its baselines (Table 3), it seems like it would make more sense to consider the fully-supervised setting, rather than the 5% supervised setting. The fully-supervised comparison in Table 5 (appendix) does not include the CE baseline.
1. In the comparison of supervised losses (Table 3), Bayesian loss (BL) and DM loss are worse than cross entropy (CE). What could explain this?

**Minor issues**

1. It should be more clear in the main text that "semi-supervised" means that the training set is a union of fully-supervised and unsupervised sets.
1. The description of "query initialisation" seemed to refer to the fixed semantic meaning of the tokens, rather than the initial values of the query vectors, which I assume are randomly initialised and trained? This is unclear. Or if I have misunderstood, then the initialisation procedure was unclear.
1. I felt that absolute error (MAE) alone would be sufficient in the main text, and would make the tables easier to interpret. The squared error (MSE) makes the tables more cluttered and doesn't add much.
1. It wasn't clear how the 2D Gaussian smoothing was applied. Does this replace the ground-truth point annotations with a 2D Gaussian? How is the sigma chosen?
1. Why not also use the PDM loss to encourage the two discretizations to have similar outputs? (accounting for the shift)
1. The fact that the multi-head attention modules have multiple layers (4 layers) was not mentioned until the experimental details.

**Nitpicks**

* "rationale for" or "motivation of" is more appropriate than "rationality of"
* Several uses of "forward" and "forwards" as a noun. It's unclear what this means. Model evaluation?
* Some typos and grammar errors.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is quite clear, although some of the choices needs to be better justified, particularly the transformer architecture.

Quality: The method is well-designed and the empirical evaluation seems rigorous.

Novelty: The method is sufficiently novel.

Reproducibility: I believe there are enough details to reproduce the experiments. Code will be released.
summary_of_the_review: The paper describes an effective loss and parametrization for predicting continuous densities, as well as a simple consistency loss for unsupervised images. The transformer architecture is not demonstrated to be necessary and I wonder whether a simple conv-net would work as well using these loss functions. The ablative experiments should also be expanded, and it would be better to evaluate the supervised loss in the fully-supervised setting. I'm leaning towards accept. I may increase my rating if these issues are addressed or decrease it if they are not.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method of semi-supervised crowd counting which only leverages a subset of labeled data and learn from the unlabeled data. This method relies on pixel-wise distribution matching and leverage optimal transport in the optimzation process.
strength_and_weaknesses: Strength:
 Based on Table 1, it seems that this approach achieve good performance in certain benchmarks.

Weakness:
1. The idea of leveraging pixel-wise uncertainty/distribution for crowd counting is already proposed in previous work[1], which is not discussed at all.

2. The intervals mentioned in Eq.2 is highly depends on Gaussian kernel value in density generation step and the pixel-wise value of crowd density is generally very small, which makes the chosen of interval values less reliable in different settings.

3. Eq.4 is not Wasserstein distance at all, the real Wsserstein distance is an optimization process that with respect to specific constraint(transport map), thus requires iteration-based solver to get the result, while Eq.4 is a normal L2 norm. Actually, it is higly unefficient for pixel-wise Wasserstein distance as it takes extremely long time for a single forward pass.

4. How to use unlabeled images is not very clear to me. 


[1]Liu, Weizhe, Nikita Durasov, and Pascal Fua. "Leveraging Self-Supervision for Cross-Domain Crowd Counting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.


------------------------------------------------------------------------------
Post-Rebuttal Comments:

I would like to thank the austhors provided such detailed rebuttal information, here is my comments after reading all of them:

The Wasserstein distance does have closed-form solution for 1-d formulation, but I still do not think  Eq.(4) with one-hot vector formulation is the correct formulation of 1-d Wasserstein distance as you mentioned in the reference paper. Besides, I even doubt if this formulation has any advantage over simply soft-max loss. As the code is not available, I'm not able to judge it.

I raised my recomendation considering the rebuttal did address some of my concerns.
clarity,_quality,_novelty_and_reproducibility: No code available.
summary_of_the_review: This paper proposes a method to leverage unlabeled image by pixel-wise distribution matching, as I mentioned, I think the methodology is not sound especially the formulation of optimal transport. Therefore, I think this paper is not qualified for ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this manuscript the authors propose a semi-supervised crowd counting model, which include a pixel-wise distribution matching loss to measure the difference in pixel-wise density distribution between predicted and the ground-truth, density tokens to augment Transformer decoder, and an interleaving consistency self-supervised learning mechanism to efficiently learn from unmarked data.
strength_and_weaknesses: Strengths:
(1) Transforming the density map regression task into a density-level classification task in semi-supervised crowd counting is reasonably efficient and alleviates the noise problem in the traditional semi-supervised learning of directly predicting density maps.
(2) The proposed method achieves good results on several datasets in semi-supervised crowd counting.


Weakness:
(1)In line 127 of page 4, [b1, b2] duplication occurred, which should be a clerical error.
(2)In line 215 of page 5, it is mentioned that the initialization method of density tokens in this manuscript is compared with randomly initialized queries. Ablation study about the initialization method should add to prove this assertion.
(3)In Table 1, “The results of other methods under the 40% labeled setting are referred to (Meng et al., 2021)”. However, according to (Meng et al., 2021), the quantitative results they show for the semi-supervised learning method use 50% of the labeled data. Please check to verify it.
(4)In Table 1, the methods of comparison were all proposed before 2022. If there are new articles on semi-supervised crowd counting in 2022, please cite them and compare.
(5)A visual presentation of the predicted density map compared to SOTA's semi-supervised population counting method seems to be missing in this manuscript, please add if space allows. 
clarity,_quality,_novelty_and_reproducibility: The overall expression of the manuscript is clear and of high quality, and with beautiful figures, but some mathematical expressions are not described clearly enough. The idea of introducing the idea of discrete density classification into semi-supervised population counting is also relatively novel.
summary_of_the_review: Although there are still some flaws in this manuscript, the ideas are novel and the arguments are basically sufficient. I think it is a high quality manuscript.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper is mostly clearly written. The transformer decoder and the pixel-wise distribution matching loss are adopted in the paper. The results are far better than the-state-of-art methods on four datasets which is quite good.The codes might be released for validation.
strength_and_weaknesses: Strength: The performance is dramaticly improved.

Weakness:
1) The N stands for the pixel number in line 189 while it means N regions in 200. It is contradictionary and with no further explanation. v1 and v2 in equation (7) is not explained.
2) How does the interleaving dual branch structure work in semi-supervied learning. 
3) What's the relation between (a) and (b) in Figure1 . What's the difference in Branch 1 and 2 in Figure 1.
4) How are the density levels generated from the annotations?
5)cross attention lacks referred paper before equation (5).
clarity,_quality,_novelty_and_reproducibility: Most of the work is innovations based on the previous work.Most of the innovations are reasonable.
summary_of_the_review: It's recommended as an acception if the problems mentioned above are clearly explained.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** This paper considers the problem of predicting density maps for counting given semi-supervised datasets. It proposes a novel transformer-based architecture, a distribution-matching loss function for quantized density prediction, the use of two overlapping discretisations and a consistency loss for the two discretisations that is only applied for confident predictions in unsupervised images. The method achieves strong results in both fully and partially supervised settings, achieving state-of-the-art results. Some aspects of the design are demonstrated to be important through ablative experiments.

**Strengths:** transforming the semi-supervised problem from regression to density classification is reasonable for removing the noise. The masking strategy using uncertainty is interesting. Good performance on benchmarks. 

**Weaknesses:** The following weaknesses were identified:

1. alternative explanation that the one-hot parametrization improves learnability. [yGdj]
2. can the loss in (4) be simplified since one-hot encodings are used? [yGdj]
3. needs justification for using transformer architecture. [yGdj]
4. can the first self-attention in the decoder be removed? [yGdj]
5. missing ablation studies (ECR loss w/o mask, effect of w=0.5) [yGdj]
6. should include fully supervised results. [yGdj]
7. why are BL and DM loss worse than CE? [yGdj]
8. the 1D closed from solution for Wasserstein distance used in the paper (with CDF instead of inverse CDF) requires p=1 (L1 norm). However, the paper also uses p=2, which achieves better results. [yGdj]
9. Previous work on using pixel-wise uncertainty for crowd counting. [BsbD]
10. the intervals used depends on the Gaussian kernel, which makes it less reliable for other settings. [BsbD]
11. Eq 4 is not the Wasserstein distance. [BsbD]
12. How to use the unlabeled images? [BsbD]
13. missing ablation study on the initialization method [ccra]
14. results using 40% labeled data, but Meng et al used 50% labeled data. [ccra]
15. any new semi-supervised counting in 2022? [ccra]
16. How does the interleaving dual branch structure work in semi-supervised learning? [FHRP]
17. What's the difference between the two branches? more clarifications of Fig 1 [FHRP]
18. How are the density levels generated from the annotations? [FHRP]

The authors wrote a response to address these issues. 
justification_for_why_not_higher_score: While the results look good and framework is reasonable, the major issue is the formulation of the loss in (4) and its underlying principle is not 1-D Wasserstein as claimed.  Further investigation is needed to explain the loss from a theoretical aspect.
justification_for_why_not_lower_score: n/a
Decision: Reject