Semi-supervised Counting via Pixel-by-pixel Density Distribution Modelling
Authors: Hui LIN
Zhiheng Ma
Rongrong Ji
Yaowei Wang
su zhou
Xiaopeng Hong
Venue: Submitted to ICLR 2023 (2023)
Keywords: Computer Vision
Crowd Counting
Semi-Supervised Learning

Abstract: This paper focuses on semi-supervised crowd counting, where only a small portion of the training data are labeled. We formulate the pixel-wise density value to regress as a probability distribution, instead of a single deterministic value, and utilize a dual-branch structure to model the corresponding discrete form of the distribution function. On the basis, we propose a semi-supervised crowd counting model. Firstly, we enhance the transformer decoder by usingdensity tokens to specialize the forwards of decoders w.r.t. different density intervals; Secondly, we design a pixel-wise distribution matching loss to measure the differences in the pixel-wise density distributions between the prediction and the ground-truth; Thirdly, we propose an interleaving consistency regularization term to align the prediction of two branches and make them consistent. Extensive experiments on four datasets are performed to show that our method clearly outperforms the competitors by a large margin under various labeled ratio settings.

Under review as a conference paper at ICLR 2023
SEMI-SUPERVISED COUNTING VIA PIXEL-BY-PIXEL DENSITY DISTRIBUTION MODELLING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper focuses on semi-super crowd counting, where only small por- tion the training data are labeled. We formulate the pixel-wise density value regress
2
probability distribution, instead of single deterministic value. On this basis, we propose a semi-supervised crowd counting model. Firstly, we design a pixel-wise distribution matching loss to measure the differences in the pixel-wise density distributions between the prediction and the ground-truth; Sec- ondly, we enhance the transformer decoder by using density tokens to specialize the forward propagations of decoders w.r.t. different density intervals: Thirdly, we design the interleaving consistency self-supervised learning mechanism to learn from unlabeled data efficiently. Extensive experiments on four datasets are per- formed to show that our method clearly outperforms the competitors by large margin under various labeled ratio settings. Code will be released.
11
INTRODUCTION
13
Crowd counting (Zhang et al. 2016 Cao et al. 2018 Ma et al., 2019) is becoming increasingly important in computer vision. It has wide applications such congestion estimation and crowd management. A lot of fully-supervised crowd counting models have been proposed, which require a large number of labeled data to train an accurate and stable model. However, considering the density of the crowd, it is laborious and time-consuming to annotate the center of each person's 's head in dataset of all dense crowd images. To alleviate the requirement for large amounts of labeled data, this paper focuses on semi runissuper counting where only small portion of training data are labeled (Liu et al., 20186).
14
15
16
17
18
19
Traditional semi-supervised counting methods target density regression and then leverage self- supervised criteria (L.iu et al.] 2008b) (009b) or pseudo-label generation (Sindagi et al. 2020h Meng et al. (2021) to exploit supervision signals under unlabeled data. These methods are designed to directly generate density maps, where each pixel is associated with a definite value. However, is still extremely difficult to learn good model due to the uncertainty of pixel labels. Firstly, there
23
25
are commonly erroneous head locations in the annotations (Wan & Chan 2020: Bai et al. 2020); Secondly, the pseudo labels for unlabeled training data assigned by the models are pervasively noisy.
27
To address these challenges, we propose a new semi-supervised counting model, termed by the Pixel-by-Pixel Probability distribution modelling Network (et Unlike traditional methods which generate deterministic pixel density value, we model the targeted density value of a pixel as probability distribution. On this premise, we contribute to sumi-supervised counting in four ways.
29
31
We propose a Pixel-wise probabilistic Distribution (PDM) loss to match the distributions of the predicted density values and the targeted ones pixel by pixel. The PDM loss, designed in line with the discrete form of the ID Wasserstein distance, measures the cumulative gap between the predicted distribution and the ground-truth one along the density (interval) dimension. By 36
33
35
modeling the density intervals probabilistically, our method responds well to the uncertainty in the labels. It thus surpasses traditional methods that regards the density values deterministic.
38
We incorporate the transformer decoder structure with a density-toke token scheme to modulate the features and generate high-quality density maps. A density token encodes the semantic infor-
39
mation of a specific density interval. In prediction, these density-speciff tokens specialize the forward propagations of the decoder with respect to the corresponding density intervals.
41

---

Under review as a conference paper at ICLR 2023
We create two discrete representations of the pixel-wise density probability function and shift one to be interleaved, which are modelled by a dral-branch network structure. Then we propose an inter-branch Expectation Consistency Regularization term to reconcile the expectation of the predictions made by the two branches.
44
46
47
We set up new state-of-the-art performance for semi-supervised crowd counting on four chal- lenging crowd counting datasets, i.e. UCF-QNRE (Idrees et al.] 2018), et al.] (020a), ShanghaiTech A and B (Zhang et al. (016). Our method outperforms previous state-of-the-art methods by a wide margin under all three settings of labeled ratio. Especially, on the QNRF dataset, our method achieves remarkable enror reduction by over 44 in mean absolute error and 79 in mean square error under the challenging 5% label setting.
48
(Sindagi
50
51
52
2 RELATED WORKS
53
Fully-supervised Crowd Counting, Early methods tackle the crowd counting problem by exhaus- tively detecting every individual in the image (Liu et al. 2019c) (Liu et al. 2018a). However, these methods are sensitive to occlusion and require additional annotations like bounding boxes. With the introduction of density map (Lempitsky & Zisserman 2010), numerous CNN-based approaches are proposed to treat crowd counting as a regression problem. MCNN (Zhang et al.) 2016) em- ploys multi-column network with adaptive Gaussian kernels to extract multi-scale features. Switch- CNN (Babu Sam et al., 2017) handles the variation of crowd density by training a switch classifier to relay a patch to particular regressor. SANet (Cao et al. al.] (2018) proposes local pattern consis- tency loss with scale aggregation modules and transposed convolutions. CSRnet (Li et al. 2018) uses dilated kernels to enlarge receptive fields and perform accurate count estimation of highly con- gested scenes. BL (Ma et al. 2019) introduces the loss under Bayesian assumption to calculate the expected count of pixels. Furthermore, methods based on multi-scale mechanisms (Zeng al, 2017 (Sindagi & Patel 2019b Ma et al. 2020), perspective estimation (Shi et al. [019: Yan et al,,
55
56
57
58
59
60
61
62
63
64
66
67
2019)
and optimal transport (Wang et al. 2020a Ma et al. 2021 Lin et al. 2021
proposed to
68
overcome the problem caused by large scale variations in crowd images.
Recently, to alleviate the problem of inaccurate annotations in crowd counting, few studies begin to find solutions from quantizing the count values within each local patch into set of intervals and learning to classify. S-DCNet proposes a classifier and division decider to decide which sub-region should be divided and transform the open-set counting into closed-set problem (Xiong et al., 2211). A block-wise count level classification framework is introduced to address the problems of inaceu- rately generated regression targets and serious sample imbalances (Liu et al.) 2019a).. The work (Liu et al. 2020a) proposes an adaptive mixture regression framework and leverages on local counting
70
71
72
73
74
75
map to reduce the inconsistency between training targets and evaluation criteria. UEPNet (Wang) 77 ct al. 202 uses two criteria to minimize the prediction risk and discretization errors of classifica-
76
78
tion model. Our method is distinct to most existing approaches. We revisit the paradigm of density classification from the perspective of supi-super learning and reveal that the interleaving quan- tization interval has natural consistency self self-supervision mechanism.
79
81 Semi and studies gradually focus on semi- and weakly-supervvied crowd counting. For semi-supervised count- ing, 1.2 (Liu et al., 2018b) introduces an auxiliary sorting task by learning containment relation- ships to exploit unlabeled images. A learning mechanism based on Gaussian Process-based is pro- posed to generate pseudo-labels for unlabeled data in (Sindagi et al., 2020bb. Zhao et al. (2020) propose an active learning framework 2020b) leverages a set of surrogate binary segmentation tasks to exploit the underlying constraints of unlabeled data. (Meng et al., 202T) proposes spatial uncertainty aware teacher-student frame- work to alleviate uncertainty from labels. (Lin et al. 2022a) proposes a density agency to construct correlations among unlabeled images. constrast, we consider semi-supervised crowd counting as quantitative censity-interve distribution matching problem and provide a self-supervised scheme via a cewiissennc-constrannn dual- branch structure. Moreover, there are also relevant studies about weakly supervised counting (Yang al., 2020 Lei et al. 2021 Sindagi & Patel 2019a), which pay more attention to learning from coarse annotation such as image-level labels or total counts.
Crowd Counting. As labeling crowd images is expensive, recent
82
89
84
85
86
minimize the expensive label work. IRAST (Liu et al.,
87
86
89
90
91
92
98
94
Vision Transformer. Vision Transformer (ViT) (Dosovitskiy al. 2020) introduces the Trans- 96 former networks (Vaswani et al.) (017)) to image recognition. Transformers further advances various tasks, such object detection (Carion et al. 2020 Zhu et al. 2020 Zheng et al.) 2020) (un et al.)
87
2

---

Under review as a conference paper at ICLR 2023
Density Token
Densty Tokers Branch
Festure
Dust rottonn Map.
Predicted
(b) Supersised Learning
Predicted Puseiility
Paoudo Closuiicction
Decodor1
S
Pixel -wise Distribution Matching Loes
VSG-19
Refined Tokers
Softmax
Decoder2
Clasufication Map!
Tonal Structore
Deraily Tobera 2
Consstency Regularosnin
head Alwetion
Narm
head Attertion
Tayer
Nomn
Fined Ioweard
Novri
Densky
Distribsiion
Feature
Map
Decader
Expectation
(4) Unsupervised learning
Figure I: The structure of Net. (a) The interleaving dual-branch structure with density tokens to predict density category. Different token colors represent the specified different density intervals. The softmax operation targets on the category while each predicted distribution map represents the segmentation map learned by this specific token and the corresponding density category.
(b) The structure of the decoder. (c) The inter-branch Expectation Consistency Regularization for
self-supervised learning. (d) The horizontal axis stands for the density values, with the attached squares for the discrete density intervals corresponding to the tokens. The vertical axis is the nor- malized distribution value of each category for that pixel.
2021
instance or semantic segmentation (Zheng et al. 2021 Wang al.) 2021c Strudel et al.,
2021
Cheng et al. (021). and object tracking (Chen et al. |021 Wang et al. 2021b) lb; Sun et al.,
89
2020
Lately, the works (Lin et al.) 2022bb Wei al.) 2021 Liang et al. (021) use the trans-
100
former encoder with self-attention to refine the image feature for crowd counting, whilst our method leverages the decoder with cross- noss-attention learn the density classification tokens.
101
102
3 COUNTING VIA PIXEL-BY-PIXELI PROBABILISTIC DISTRIBUTION MODELLING
103
104
In this section, we first describe the setting of semi-supervised crowd counting and then explain the rationale for adopting the probability distribution to represent the crowd density.
106
Formally, have labeled dataset X' consisting of images with point annotated ground truth and an unlabeled dataset LI consisting of only crowd images. In semi-supervised crowd counting, the training set includes both X and U. Usually, 24 contains much more images than X' for training counting model, i.e., [2] >>>>>> | For crowd counting, the popular proportion settings are that the labeled dataset occupies 5 5%, 10% and 40% of the total training set respectively.
107
108
109
110
111
Previous methods have utilized self-supervised criteria (Liu et al. 2018b; 2019b) or pseudo- generation (Sindagi et al. 2020b Meng et al., (022)) to exploit supervision signals under unlabeled data. These methods rely on the density prediction pipeline as the most traditional supervised meth-
112
113
114
ods (Zhang et al.) 2016 Ma al. 2019: Lin et al. 20226). However, when only partial labels are available for training the model, the obtained density maps are likely to be noisy. It becomes increasingly challenging to predict deterministic, accurate density value for each single pixel or small patches. To solve this problem, we model the targeted density value of a pixel as probability distribution, instead of a deterministic single value. The predicted density value d is then given by the expectation as follows.
115
116
117
119
d
p(x)
121
where x is the probable density value ranged in [0. +0o). The conventional prediction way. which can be represented as the Dirac delta, p(z) 8(x d), is special case of Eq. [ This approach is fragile when there is uncertainty and noise. Instead, by Eq. [ we revert to a general distribution function p(x) without introducing any prior about the distribution such as Dirac.
122
123
124
3

---

Under review as a conference paper at ICLR 2023
126
To find a numerical form, to which deep learning models can be applied, we further discretize Eq.
d = p.
127
128
The set V f21, F22, .C) are the discrete representations of the density intervals, which are obtained by quantizing the continuous density range [0, +00) into C' mutually exclusive discretized intervals [0, b1), [b, b2). .... (bc-1.+0)), where 61 .... bc- are the ascending interval borders. P(x)) is the discrete distribution function, which can be easily implemented through a softmax func- tion and is consistent with the convolutional neural network. As a result, in this work, we transform the regression problem into a density interval classification problem, i.e. from predicting an exact count to choosing a pre-defined density interval, in order to build more reliable prediction signals for supi-supee counting.
129
130
132
134
135
136
On this basis, we propose our Pixel-by Exxel-by-Pixel Probability distribution modelling Network (P'Net) for superviidd learning. NeNet is composed of three modules to enhance the classification paradigm to semi-supervised crowd counting. First, we propose a Pixel-wise Distribution Matching (PDM) loss meet the needs of effectively matching the distributions between the prediction and the label. After that, we introduce a transformer decoder with proposed density tokens to learn and preserve density information from different density intervals. And finally, we design dual-branch structure and propose a corresponding self-supervision mechanism for semi-supervised learning.
138
139
140
141
142
148
3.1
PIXEL-WISE DISTRIBUTION MATCHING Loss
144
In this section, we detail the proposed PDM loss and the corresponding supervision between pre- dicted distribution and the ground-truth.
146
To punish the difference between predicted distributions and ground truth, we first generate the train- ing label Y E {0, 1 for the dual-branch from annotated points. We perform a 2-D Gaussian smoothing on these points, and then calculate the expected density value of each pixel. Each row in the label y {0, 1,et is in the form of one hot distribution and the category where the value equals to represents the specific interval that the density of this certain pixel falls into.
148
150
We match the predicted distribution to the ground-truth distribution by minimizing the divergence between them. On this basis, we adopt the Wasserstein distance to act as the measuring function. It represents the least cost of pushing one distribution q towards another q and is defined as:
152
153
W(q, q) = min /. c(qu, (r)dd v). u,v
154
155
(u, v) is the transport map from qu to qu while e is the moving cost function. Typically, we adopt the square of Euclidean distance as c. We discretize the calculation of the Wasserstein distance and define the Pixel-wise Distribution Matching (PDM) loss. When both distributions are one- dimensional distribution vectors, the matching loss will have a closed-form solution (Kolouri al., 2018). Given p and as the prediction and ground-trutt truth label for a certain pixel respectively. and
156
157
158
159
Gly.j) E'= y/s the cumulative distribution function, the loss can be calculated by
161
Cp Σ W) Y.D
Σ ((b.:))((pp.)))))- Y.D 1=1
(4)
162
The PDM loss measures the cumulative gap between the predicted distribution and the ground truth along the density dimension. It penalizes the distributions that are deviated.
163
164
The Rationale for PDM Loss. We provide an example to illustrate the advantages of our loss function. Suppose there are four intervals and we have an instance with the label of [0, 1,,0,) Given two predicted outputs A: [0.2,0.3.0.5,00 and B: [0.2,0.3.0.0.3 single-mode output which shall be considered better than B. However, the loss values of A and B are the same in terms of the Cross Entropy (0.36) and Mean Square Error (0.78), they can not be distinguished. In contrast, in terms of our PDM loss, the cumulative forms to calculate Eq. 4 for A respectively and the corresponding loss values are 0.29 and 0.54. As result, the two can be well differentiated.
185
186
clearly A gets a more compact,
168
170
and B are 10.2,0.5,1.0.11
and 10.2,0.5.0.5,11
4

---

Under review as a conference paper at ICLR 2023
Differences from DM-Count. DM-Count (Wang et al. 2020a) is an insightful optimal transport
based counting approach to match the probability distributions of occurrence over the spatial do- main. In contrast, the proposed PDM loss matches the pixel-wise probability distributions over the density intervals. Hence, the domains where optimal transport performs by the two methods are distinctly different.
173
175
3.2 TRANSFORMER SPECIALIZATION
177
Next, we introduce set of density tokens specialize the forward propagations of the transformer decoder with respect to the corresponding density intervals. The density tokens are learnable embed- dings with different density information, which are fed interact with the input extracted feature
179
180
vectors to instruct the model prediction. Each token is endowed with unique semantic information and acts as an indicator of a density interval. In other words, the density tokens are prototypes cor- responding to different density intervals. Specifically, we set b to small value and treat the token assigned to the first interval [0, b1) as the background token. tures in areas without crowd in the image. We denote T E RCx2 as a matrix capsuling all C' tokens
181
182
responsible for learning the fea-
184
where is the dimension of both the features and tokens.
186
Then we use the transformer decoder (Vaswani et al. 2017) to break the limitation of local convo- lutional kernels, correlating similar density information from various regions inside an image. The decoder is composed of stack of mutiple identical layers. In each decoder layer, the tokens are firstly processed by a multi-head attentonn module and a normalization layer. The relationships between tokens and the whole feature map are computed through cross attention:
188
189
190
191
C)T S( (TW)) (FWV)).
(5)
182
VZ
F E R is the matrix of the input features, where N is the pixel or patch number. S is the softmax function, and WQ, W WV E 2x are weight matrices for projections. Afterwards, we get the refined tokens T, after processing further by a layer normalization and a feed-forward network, as illustrated in Figure (c).
194
195
Note that in Equation 5 through the inner product of the two vectors, the cross attention learns which regions in the feature map that each category token should focus on. Inspired by this idea, in the forward pass, we leverage the modulate the input patch features for predicting the final probabilities:
197
interval-spccialized token through a softmax activation to
199
200
O O=S(T-F)
201
where the softmax operation is performed along the category dimension. The predicted matrix O RCXN denotes the C' predicted distribution maps, each of which represents the region distribution of the corresponding density interval the whole image of N patches. By Eq. we measure the similarity between the region features and refined density tokens, modulate the regional features and output the predicted density-interval distribution.
203
204
205
206
As the idea of proposed density tokens is natural extension in semantics of that of the query tokens in the transformer, can be optimized through the training pipeline of transformer using back propagation. Note that, only the original density tokens are restored, while the tokens refined
207
208
209
adapted the input regional features are not retained. As a result, the final density tokens are the papee-parramters shared by all inputs in the reference stage.
210
211
The Rationale for Tokens arises from the observation that similar regions with same density inter-
212
vals can be mined within an image. An example is shown in Figure for specific density interval
213
like (al), we can easily find similar regions (a2/a3) all over the image. The density tokens (a/b/c)
214
play role of grouping different regions with the same density levels. During learning, the tokens are connected to the discrete representation of density probability distribution and finally with clear semantic associations. During inference, by using the tokens traversally, specialize each forward propagation of our decoder module with respect to a particular density interval distribution in turn.
216
217
218
Differences from handomly-initialize tialized queries as the input in each forward pass. Here, we use DETR (Carion et al.) (020) as an example. In DETR, there is no clear distinction among the representative semantics of different
queries. Transformer decoder usually uses randomly ini-
220
221
5

---

Under review as a conference paper at ICLR 2023
Taken
Token
Token c
(e
(rt)
(b2)
(c2)
(o3)
(b
Figure 2: Similar Regions of the same density levels exists within an image. We use a density token specify density interval and group the regions of that level.
queries. Thus in the training stage, association methods like Hungarian algorithm are required in every iteration to match queries with objects. Instead, we explicitly associate an exclusive density interval to each query throughout the model's lifetime. Thus generate tailored tokens with clear semantics.
223
225
226
3.3
INTER- BRANCH EXPECTATION CONSISTENCY BASED ON DUAL-BRANCH INTERLEAVING STRUCTURE
228
Although modelling the predicted density value as discrete probabilistic distribution leads to more credible and less noisy prediction generally, when the value falls near the boundary, noise in the output density values can easily lead to incorrect quantization, and further corrupt the classification results. Meanwhile, when converting the predicted interval category into density, the pre-defined discrete representation x will inevitably have a quantization gap.
229
230
231
233
To alleviate these limitations, we use an interleaving dual branch structure, which consists of two parallel classification tasks with overlapping count intervals. The density which falls near the inter- val border of the first branch is more likely to be classified easily another branch and meanwhile reducing the conversion gap without increasing the number of classification intervals.
235
237
The interleaving dual branch structure has been used to address the inaccurate ground truth (Wang et al. 2021a). To further accommodate it with semi-supervised counting, we make step forward in this direction from two perspectives. Firstly, we associate the output of the network with the pixel-by-pine probabilistic distribution and introduce weighted, soft quantization level assignment mechanism. More specifically, during the inference stage, the work (Wang et al.] (001a) selects the
239
241
242
category with maximum predicted value for each pixel or patch, and directly converts to the corresponding representation value. Instead, we keep the distributions of predicted possibilities and
243
244
leverage on the expectation to alleviate the conversion error. As a result, rather than simply averaging the predicted densities of two branches, we give each a certainty weight, which can be represented by the maximum classified possibility. When branch predicts a large possibility for certain category rather than similar values for multi categories, the branch has higher confidence about that prediction, thus we increase the proportion of it in the final prediction. Secondly, the interleaving two-branch structure provides natural self-supervising mechanism that allows for imposing the consistency constraint between the two branches. On top of this constraint, we design an interleaving consistency regularization term which penalizes the deviation between the output expectations of the two-branches, to provide rich supervised signals in the absence of labels.
245
246
248
250
252
Specifically, we denote by two C-dimensional vector p. q E RC the predicted classified possibilities of the dual branches for a certain pixel or patch. The vectors satisfy that ||p|| = ||q|]] = and their elements in the range of (0, 1]. Thus the final density can be expressed by
254
255
256
d = w v + (1 (1-6a w) q V.J
257
where the weight w = ||pllsc (IIpll + ll9) and |1- is the vector maximum norm. V1 and V2 denote the represented quantized value for each branch. We extend the proposed network to fit the dral-branch structure, where two different decoders are adopted and the density tokens are split into two interleaved sets, as shown in Figure
259
281 On the basis, a self-supervised learning scheme is designed to leverage the unlabeled data for refining the model, where the expectations of classified probability distribution two branches tend to

---

Under review as a conference paper at ICLR 2023
be consistent. We based this constraint to construct the inter-branch Expectation Consistency Regularization (ECR) term. Moreover, to prevent the regularization term from being negatively affected by the wrongly predicted probability distribution, we impose selection mechanism to only consider the patches which are predicted with high certainty. The mechanism is based on a dynamic pixel-wish wiie mask TR which elements are in the range of [0, 1] select weigh the regions for supervision. Given O1, O2 as the predicted probability matrices by the two branches, the self-supervised ECR is defined
264
266
268
269
LE = 3 c R112-
270
where v1 V202 is vector reflecting the inconsistency between the density expectations by the two branches and is the element-wise multiplication.
271
Similar with Eq. 7 we regard the maximum possibility |pll x each distribution as the confidence. If the distribution is even, the confidence will be low, indicating that the model cannot predict certain class for that patch with high certainty. In this case, we shall reduce its importance or exclude
273
274
276
this patch in back-propagation dynamically. For efficient computation, we binarize & E {0, 1)
Only when the both confidences of two branches are sufficiently high, regularization on that pixel is activated. Given the confidence threshold & E 1) and the boolean function r(cond) which outputs when the condition is true and 0 otherwise, the supervision mask is defined as:
277
279
= > & r(02 E),
where on and 02 are N-dimensional vectors takeing the maximum values of O₁ and O₂ along the interval dimension respectively. Finally, the overall training loss is the combination of density aware loss using in labeled data and consistency regularization with the parameter À using unlabeled data.
261
263
L = Lp + ALE-
(10)
285
The Rationale for the Regularization. A common issue in self-supervised consistency regulariza- tion is the confirmation bias (Tarvainen & Valpola 2017). which indicates that the mistakes of the model will probably be accumulated during semi-supervised learning. We utilize the regularization term to alleviate this bias from the following aspects. First, we select the most reliable instances for self-supervision by using the mask in Eq. 8] Second, our network adopts two independent decoders and respective density tokens. As shown by (Ke et al. (2019), learning independent models helps to address the performance bottleneck caused by model coupling. Thus the proposed regularization term is plausible. We also provide a detailed study in the appendix.
287
268
289
290
291
292
253
4 EXPERIMENTS
294
We conduct extensive experiments on five crowd counting benchmarks to verify the effectiveness of proposed Net. Experiments and descriptions of NWPU-Crowd (Wang al. 2020b can be referred to the appendix. The datasets are described as follows:
285
296
297
UCF-QNRF (Idrees et al. 2018) The dataset contains congested crowd images, which are crawled from Flickr. Web Search, and Hajj footage. It includes 1,535 high-resolution images with 1.25
298
299
million annotated points. There are 1,201 and 334 images the training and testing sets respectively.
300
301
JHU-Crowd++ (Sindagi al.) 2020a) The dataset includes 4.372 images with 1.51 million anno-
302
tated points. There are 2,272 images used for training. 500 images for validation, and the rest 1,600
303
images used for testing. The crowd images are collected from several sources on the Internet using
304
different keywords and typically chosen under various conditions and geographical locations.
ShanghaiTech A (Zhang et al., 2016) The dataset contains 482 crowd images with 244, anno-
306
tated points. The images are randomly chosen from the Internet where the number of annotations
307
in an image ranges from 33 to 3,139. The training set has 300 images, and the testing set has the remaining 182 images.
308
309
ShanghaiTech B (Zhang et al., 2016) The dataset contains 716 crowd images, which are taken in the crowded street of Shanghai. The number of annotations in an image ranges from to 578. The training set has 316 images, and the testing set has the remaining 400 images.
310
311
312
The network strucutre and the trianing datails are summarized follows.
313

---

Under review as a conference paper at ICLR 2023
Labeled Percentage
UCF-QNRF
JHU++
ShanghaiTTcch A ShanghaiTech
Methods
MAE
MSE
MAE MSE
MAE
MSB
MAE
MSE
MT TTavaiinnn & Valpola
2017)
5%
172.4 284.9
101.5 363.5
104.7
156.9
19.3
33.2
1.2R (Lau al. 2018b)
5%
160.1 272.3
101.4 338.8
103.0
155.4
20.3
27.6
GP (indagi et
2020b
5%
160.0
275.0
102.0
172.0
15.77
27.9
Net (Ours)
5%
115.3
195.2
80.8
306.1
85.5
131.0
12.0
22.0
MT (Tarvainen Valpola
2017 7)
10%
156.1
145.5
250.3
90.2
319.3
94.5
15.6
24.5
L2R (Lu et al. 2018b
10%
148.9
249.8
87.5
315.3
90.3
153.5
15.6
24.4
AL-AC Zhao et al 2020
10%
87.9
139.5
13.9
26.2
IRAST
(Lu et al. 2020b
10%
86.9
148.9
14.7
22.9
IRAST+SPN (Liu cf al. N'Net (Ours)
2020b
10%
83.9
140.
10%
103.4
179.0
71.8
294.4
72.1
116.4
10.1
18.2
MT
(Tarvainen & Valpola
2017
40%
147.2
249.6
121.5
388.9 9
88.2
151.1
15.9
25.7
L2R (Liu ct al. 2018b)
40%
145.1 256.1
123.6
376.1
86.5
148.2
16.8
25.1
GP (iidaai et al 2020bb
40%
136.0
89.0
IRAST
LLu al. 20206
40%
138.9
SUA Meng el al. Net (Ours)
202T
40%
130.3 226.3
80.7
290.8
68.5
121.9
14.1
20.6
40%
90.0 155,4 58.9 251.9
63.0
100.9
7.1
12.0
Table 1: Comparisons with the state of the arts semi-supervised counting methods on four datasets. The best performance is shown in bold. The results of other methods under the 40% labeled setting are referred to (Meng et al. 2021) and all other results are from the original papers.
314
Network Details. VGG-19, which is pre-trained on ImageNet, is adopted as our CNN backbone to extract features. We use Adam algorithm (Kingma & Bal 2014) to optimize the model with the learning rate 10-5 The number of decoder layers is set as 4. We set C = 25 and follow (Wagg
316
al. 2021a) to calculate the reasonable density intervals. For the loss parameters, set À 0.01
318
and { = 0.5.
319
Training Details. We adopt horizontal flipping and random scaling of [0.7, 1.3] for each training
320
image. The random crop with a size of 512 x 512 is implemented, and as some images in Shang- each A contain smaller resolution, the crop size for this dataset reduces 256 256. We limit the shorter side of each image within 2048 pixels in all datasets. The experiments are held on one
322
323
GPU of RTX3080.
4.1
RESULTS AND DISCUSSIONS
325
Comparisons to the State of the Arts. We evaluate Net on these datasets and compare it with state-of-the-art semi-supervised methods, as shown in Table [ Since in the 50% labeled setting of paper (Meng et al. 202T). 10% of the labeled data will be used as the validation set, we consider it as the 40% labeled setting. NeNet outperforms other methods by a large margin. Compared to the second best methods on the most challenging setting of 5%, method reduces the MAE by 44.7, 20.6, 16.5 and 3.7 points on QNRF, JHU++, ShanghaiTech A and B. Specifically, on QNRF dataset, Net achieves significant reductions which are over about 27.9% in mean absolute error and 28.3% in mean square error under three different settings of labeled ratio. The excellent results demonstrate the effectiveness of our method in superiisee crowd counting.
326
328
330
332
333
334
The impact of PDM and ECR loss. We conduct experiments to study the impact of two proposed loss functions. Specifically, Lp represents the PDM loss without ECR, and the combination of Lp and Lg forms the proposed Net. The comparison result is shown in Table 2 With the help of unlabeled data and the corresponding ECR, NeNee improves the counting accuracy of 'supervisions from only labeled data' over 7.8 and 12.2 in terms of MAE and MSE respectively. The experimental results validate that through the self-supervision of ECR from unlabeled data, the prediction capa- bility and accuracy of the model is enhanced. The improvement is the sense of semi-supervised learning.
335
336
337
339
341
The impact of Pixel-wise Distribution Matching loss. We study the proposed PDM loss by comparing it with the Cross Entropy (CE) loss and MSE loss, and more noteworthy, the counting loss including the Bayesian loss (Ma et al. 2019) and DM loss (Wang et al. 2020a) which achieve best results in the sully-supervised domain. The experimental result is shown in Table 3 which is held on UCF-QNRF dataset with labeled ratio of 5%. Our loss outperforms all four losses by large
343
345
346
8

---

Under review as a conference paper at ICLR 2023
Labeled Percentage
Loss
MAE
MSE
Loss
MAE
MSE
5%
Cp
129.5
212.8
Cp ALE
115.3
195.2
10%
Cp
117.4
211.8
Lp + ACE
103.4
179.0
40%
Cp
97.8
167.6
Cp + ALE
90.0
155.4
100%
Cp
78.5
135.8
Table The impact of ECR loss. Experiments are conducted on UCF-QNRE. With the help of ECR to exploit supervisions from unlabeled data, get further improvement counting accuracy.
margins. The result suggests that the awareness of the semantic information is helpful in matching the distribution between prediction and ground truth. Moreover and surprisingly, the CE loss and specialized for classification originally, surpass the counting loss in this
348
MSE loss, which case. The reason probably lies in that when only a small number of ground-truth labels is available, regarding the single-value density probability distribution provides better way for improving the robustness and accuracy of the counting model.
350
351
352
CE
MSE
PDM
Cp
MAE
MSE
MAE MSE
125,4
132.8
115.3
5%
134.5
240.6
211.6
223.2
195.2
100%
85.8
142.7
BL
DM
PDM
Cr
MAE
MSE
MAE MSE
136.5
133.4
115.3
5%
129.5
212.8
234.7
225.3
195.2
100%
78,5
135,8
Table 3:
Comparisons of using different Table 4: The influence of probabilistic distribution mod- losses to get supervisions from ground- elling. Lp denotes the conventional prediction way, truth. Experiments are held UCF- which selects the value of maximum predicted category and then a simple average is made between dual branch.
QNRF with 5% labeled ratio.
The influence of probabilistic distribution modelling. Table 4 reeports the influence of modelling each pixel by probability distribution. For comparison, we denote the conventional prediction way as Cp. which first selects the category with maximum predicted score for each pixel in cach branch and converts it to the corresponding predefined representation value. Then we make a simple average UCF-QNRF with the
364
356
instead of using Eq. 7 between dual branch. We study its performance settings 5% and 100% labeled ratios. Compared with the proposed model, the counting accuracy of Cp has an obvious decrease. This indicates that probabilistic distribution modelling and the use of expectation of different branches effectively improves the performance.
358
350
360
5 DISCUSSION AND CONCLUSION
We propose a dual-branch semi-supervised counting approach based on interleaved modelling of pixel-wise density probability distributions. The PDM loss matches the pixel-by exxleel-pixel density prob- ability distribution to the ground truth. shows good generalization capability, even when only a small amount of labeled data is available. Moreover, a set of tokens with clear semantic associations to the density intervals customizes the transformer decoder for the counting task. Furthermore, the inter-branch ECR term reconciles the expectations of two predicted distributions, which provides rich supervised signals for learning from unlabeled data. Our method compasses other methods by an average relative MAE reduction of over 22.0%, 23.5%, and 28.9% with the label ratios of 5%, 10%, and 40% respectively. As result, a new strong state of the art for semi semi-supervised crowd counting set up.
362
364
366
367
368
369
370
371
We also evaluate our approach under the fully-supervised setting. The detailed experimental results are reported in the appendix. Our approach achieves 78.5 MAE on QNRF and thus works remark- ably well under the fully supervised setting. This consistent performance boost implies that optimal
372
373
374
semi-supervised counting built on both the ability to learn from labeled data and unlabeled data.
375
Compared with those methods focusing more on learning from unlabeled data, P'Nett reaches a bet- ter balance of learning from both labeled and unlabeled data. The limitation of our method that
377
ECR alleviate, but cannot eliminate the bias in self- sulf-supervision. When the image background is too complex or the image is too crowded, it may lead to poor results. Nonetheless, this study still brings much inspiration for future studies about semi-supervised learning.
379
360

---

Under review as a conference paper at ICLR 2023
REFERENCES
382
Deepak Babu Sam, Shiv Surya, and R Venkatesh Babu. Switching convolutional neural network for crowd counting. In CVPR, 2017.
383
384
Shuai Bai, Zhiqun He, Yu Qiao, Hanzhe Hu, Wei Wu, and Junjie Yan. Adaptive dilated network with self-correction supervision for counting. In CVPR, 2020.
Xinkun Cao, Zhipeng Wang, Yanyun Zhao, and Fei Su. Scale aggregation network for accurate and efficient crowd counting. In ECCV, 2018.
387
388
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
389
380
Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer track- ing. In CVPR, 2021.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NIPS, 2021.
393
394
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
395
396
Haroon Idrees, Muhmmad Tayyab, Kishan Athrey, Dong Zhang, Somaya Al-Maadeed, Nasir Ra- jpoot, and Mubarak Shah. Composition loss for counting, density map estimation and localization in dense crowds. In ECCV, 2018.
396
400
Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy Ren, and Rynson WH Lau. Dual student: Breaking the limits of the teacher in semi-supervised learning. In ICCV, 2019.
401
402
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint, 2014.
403
404
Sohcil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. wasserseasnn autoen- coder: An embarrassingly simple generative model. arXiv preprint, 2018.
Yinjie Lei, Yan Liu, Pingping Zhang, and Lingqiao Liu. Towards using count-level weak supervision for crowd counting. Pattern Recognition, 2021.
407
408
Victor Lempitsky and Andrew Zisserman. Learning to count objects in images. NIPS, 2010.
Yuhong Li, Xiaofan Zhang, and Deming Chen. Csmnet: Dilated convolutional neural networks for understanding the highly congested scenes. In CVPR, 2018.
410
Dingkang Liang, Xiwu Chen, Wei Xu, Yu Zhou, and Xiang Bai. Transcrowd: Weakly-supervised crowd counting with transfommr.. arXiv preprint, 2021.
411
413
Hui Lin, Xiaopeng Hong, Zhiheng Ma, Xing Wei, Yunfeng Qiu, Yaowei Wang, and Yihong Gong. Direct measure matching for crowd counting. IJCAI, 2021.
414
Hui Lin, Zhiheng Ma, Xiaopeng Hong, Yaowei Wang, and Zhou Su. Semi-supervised crowd count- ing via density agency. ACM MM, 2022a.
416
417
Hui Lin, Zhiheng Ma, Rongrong Ji, Yaowei Wang, and Xiaopeng Hong. Boosting crowd counting via multifaceted attention. 2022b.
418
419
Jiang Liu, Chenqiang Gao, Deyu Meng, and Alexander G. Hauptmann. Decidenet: Counting varying density crowds through attention guided detection and density estimation. In CVPR, 201 8a.
420
I. Liang Liu, Hao Lu, Haipeng Xiong, Ke Xian, Zhiguo Cao, and Chunhua Shen. Counting objects by blockwise classification. TCSVT, 2019a
422
423 Xialei Liu, Joost Van De Weijer, and Andrew Bagdanov. Leveraging unlabeled data for crowd
counting by learning rank. In CVPR, 2018h.
10

---

Under review as a conference paper at ICLR 2023
Xialei Liu, Joost Van De Weijer, and Andrew D Bagdanov. Exploiting unlabeled data in cmns by self-supervised learning to rank. IEEE TPAMI, 2019b.
426
Xiyang Liu, Jie Yang, Wenrui Ding, Tieqiang Wang, Zhijin Wang, and Junjun Xiong. Adaptive
mixture regression network with local counting map for crowd counting. ECCV, 2020a.
428
Yan Liu, Lingqiao Liu, Peng Wang, Pingping Zhang, and Yinjie Lei. Semi-supervised crowd count- ing via self-training on surrogate tasks. ECCV, 2020b.
430
Yuting Liu, Miaojing Shi, Qijun Zhao, and Xiaofang Wang. Point in, box out: Beyond counting persons crowds. In CVPR, 2019c.
431
452
Zhiheng Ma, Xing Wei, Xiaopeng Hong. and Yihong Gong. Bayesian loss for crowd count estima- tion with point supervision. ICCV, 2019.
433
434
Zhiheng Ma, Xing Wei, Xiaopeng Hong, and Yihong Gong. Learning scales from points: A scale- aware probabilistic model for crowd counting. ACM Multimedia, 2020.
435
436
Zhiheng Ma, Xing Wei, Xiaopeng Hong, Hui Lin, Yunfeng Qiu, and Yihong Gong. Learning to count via unbalanced optimal transport. In AAAI, 2021.
437
438
Yanda Meng, Hongrun Zhang, Yitian Zhao, Xiaoyun Yang, Xuesheng Qian, Xiaowei Huang, and Yalin Zheng. Spatial uncertainty-aware semi-supervised crowd counting. In ICCV, 2021.
439
440
Miaojing Shi, Zhaohui Yang. Chao Xu, and Qijun Chen. Revisiting perspective information for efficient crowd counting. In CVPR, 2019.
441
442
Vishwanath Sindagi, Rajeev Yasarla, and Vishal MM Patel. Jhu-crowd++: Large-scale crowd count- ing dataset and benchmark method. PAMI, 2020a.
443
Vishwanath A Sindagi and Vishal M Patel. Ha-cen: Hicrarchical attention- based crowd counting network. IEEE Transactions on Image Processing, 2019a.
455
Vishwanath A. Sindagi and Vishal M. Patel. Multi-level bottom-top and top-bottom feature fusion for crowd counting. In ICCV, 2019b.
447
Vishwanath Sindagi, Rajeev Yasarla, Deepak Sam Babu, Venkatesh Babu, and Vishal M Patel. Learning count in the crowd from limited labeled data. In ECCV, 2020b.
449
Robin Strudel, Ricardo Gareia, Ivan Laptev, and Cordelia Schmid. semantic segmentation. arXiv preprint, 2021.
Segmenter: Transformer for
451
452
Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple-object tracking with transformer. arXiv preprint, 2020.
453
454
455
Zhiqing Sun, Shengcao Cao, Yiming Yang. and Kris M Kitani. Rethinking mansformer-based set prediction for object detection. In ICCV, 2021.
456
457
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. NIPS, 2017.
458
459
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
460
461
Jia Wan and Antoni Chan. Modeling noisy annotations for crowd counting. NIPS, 2020.
462
Boyu Wang, Huidong Liu, Dimitris Samaras, and Minh Hoai Nguyen. Distribution matching for crowd counting. NIPS, 2020a.
464
Changan Wang, Qingyu Song. Boshen Zhang, Yabiao Wang, Ying Tai, Xuyi Hu, Chengjie Wang, Jilin Li, Jiayi Ma, and Yang Wu. Uniformity in heterogeneity: Diving deep into count interval partition for crowd counting. In ICCV, 2021a.
466

---

Under review as a conference paper at ICLR 2023
Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li. Transformer meets tracker: Exploiting temporal context for robust visual tracking. In CVPR, 2021b.
489
470
Qi Wang, Junyu Gao, Wei Lin, and Xuelong Li. Nwpu-crowd: A large-scale benchmark for crowd counting and localization. PAMI, 2020b.
472
Yuqing Wang, Zhaoliang Xu, Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR, 2021c.
473
Xing Wei, Yuanrui Kang, Jihao Yang. Yunfeng Qiu, Dahu Shi, Wenming Tan, and Yihong Gong. Scene-adaptive attention network for crowd counting. arXiv preprint, 2021.
475
476
Haipeng Xiong, Hao Lu, Chengxin Liu, Liang Liu, Zhiguo Cao, and Chunhua Shen. From open set to closed set: Counting objects by spatial divide-andd-onquur. In ICCV, 2019.
478
Zhaoyi Yan, Yuchen Yuan, Wangmeng Zuo, Xiao Tan, Yezhen Wang. Shilei Wen, and Errii Ding. Perspective-guided convolution networks for crowd counting. In ICCV, 2019.
479
Yifan Yang, Guorong Li, Zhe Wu, Li Su, Qingming Huang, and Nicu Sebe. Weakly-supervised crowd counting learns from sorting rather than locations. In ECCV, 2020.
481
482
I Lingke Zeng, Xiangmin Xu, Bolun Cai, Suo Qiu, and Tong Zhang. Multi-scale convolutional neural networks for crowd counting. In ICIP, 2017
484
Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting via multi-column convolutional neural network. In CVPR, 2016.
485
Zhen Zhao, Miaojing Shi, Xiaoxiao Zhao, and Li Li. Active crowd counting with limited supervi- sion. In ECCV, 2020.
487
488
Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprint, 2020.
490
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from perspective with transformers. CVPR, 2021.
481
492
493
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020.
12


Reviews:
[{"review_id": "tBxx7RnWCI", "reviewer": "Reviewer_ccra", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this manuscript the authors propose a semi-supervised crowd counting model, which include a pixel-wise distribution matching loss to measure the difference in pixel-wise density distribution between predicted and the ground-truth, density tokens to augment Transformer decoder, and an interleaving consistency self-supervised learning mechanism to efficiently learn from unmarked data.", "summary_of_the_review": "Although there are still some flaws in this manuscript, the ideas are novel and the arguments are basically sufficient. I think it is a high quality manuscript.", "strength_and_weaknesses": "Strengths:\n(1) Transforming the density map regression task into a density-level classification task in semi-supervised crowd counting is reasonably efficient and alleviates the noise problem in the traditional semi-supervised learning of directly predicting density maps.\n(2) The proposed method achieves good results on several datasets in semi-supervised crowd counting.\n\n\nWeakness:\n(1)In line 127 of page 4, [b1, b2] duplication occurred, which should be a clerical error.\n(2)In line 215 of page 5, it is mentioned that the initialization method of density tokens in this manuscript is compared with randomly initialized queries. Ablation study about the initialization method should add to prove this assertion.\n(3)In Table 1, “The results of other methods under the 40% labeled setting are referred to (Meng et al., 2021)”. However, according to (Meng et al., 2021), the quantitative results they show for the semi-supervised learning method use 50% of the labeled data. Please check to verify it.\n(4)In Table 1, the methods of comparison were all proposed before 2022. If there are new articles on semi-supervised crowd counting in 2022, please cite them and compare.\n(5)A visual presentation of the predicted density map compared to SOTA's semi-supervised population counting method seems to be missing in this manuscript, please add if space allows. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PBGj9VmkPs", "reviewer": "Reviewer_BsbD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method of semi-supervised crowd counting which only leverages a subset of labeled data and learn from the unlabeled data. This method relies on pixel-wise distribution matching and leverage optimal transport in the optimzation process.", "summary_of_the_review": "This paper proposes a method to leverage unlabeled image by pixel-wise distribution matching, as I mentioned, I think the methodology is not sound especially the formulation of optimal transport. Therefore, I think this paper is not qualified for ICLR.", "strength_and_weaknesses": "Strength:\n Based on Table 1, it seems that this approach achieve good performance in certain benchmarks.\n\nWeakness:\n1. The idea of leveraging pixel-wise uncertainty/distribution for crowd counting is already proposed in previous work[1], which is not discussed at all.\n\n2. The intervals mentioned in Eq.2 is highly depends on Gaussian kernel value in density generation step and the pixel-wise value of crowd density is generally very small, which makes the chosen of interval values less reliable in different settings.\n\n3. Eq.4 is not Wasserstein distance at all, the real Wsserstein distance is an optimization process that with respect to specific constraint(transport map), thus requires iteration-based solver to get the result, while Eq.4 is a normal L2 norm. Actually, it is higly unefficient for pixel-wise Wasserstein distance as it takes extremely long time for a single forward pass.\n\n4. How to use unlabeled images is not very clear to me. \n\n\n[1]Liu, Weizhe, Nikita Durasov, and Pascal Fua. \"Leveraging Self-Supervision for Cross-Domain Crowd Counting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n\n------------------------------------------------------------------------------\nPost-Rebuttal Comments:\n\nI would like to thank the austhors provided such detailed rebuttal information, here is my comments after reading all of them:\n\nThe Wasserstein distance does have closed-form solution for 1-d formulation, but I still do not think  Eq.(4) with one-hot vector formulation is the correct formulation of 1-d Wasserstein distance as you mentioned in the reference paper. Besides, I even doubt if this formulation has any advantage over simply soft-max loss. As the code is not available, I'm not able to judge it.\n\nI raised my recomendation considering the rebuttal did address some of my concerns.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OO0uz4E_kOl", "reviewer": "Reviewer_FHRP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is mostly clearly written. The transformer decoder and the pixel-wise distribution matching loss are adopted in the paper. The results are far better than the-state-of-art methods on four datasets which is quite good.The codes might be released for validation.", "summary_of_the_review": "It's recommended as an acception if the problems mentioned above are clearly explained.", "strength_and_weaknesses": "Strength: The performance is dramaticly improved.\n\nWeakness:\n1) The N stands for the pixel number in line 189 while it means N regions in 200. It is contradictionary and with no further explanation. v1 and v2 in equation (7) is not explained.\n2) How does the interleaving dual branch structure work in semi-supervied learning. \n3) What's the relation between (a) and (b) in Figure1 . What's the difference in Branch 1 and 2 in Figure 1.\n4) How are the density levels generated from the annotations?\n5)cross attention lacks referred paper before equation (5).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1Y5PNGT0up", "reviewer": "Reviewer_yGdj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of predicting density maps for counting given semi-supervised datasets.\nIt proposes a novel transformer-based architecture, a distribution-matching loss function for quantized density prediction, the use of two overlapping discretisations and a consistency loss for the two discretisations that is only applied for confident predictions in unsupervised images.\nThe method achieves strong results in both fully and partially supervised settings, achieving state-of-the-art results.\nSome aspects of the design are demonstrated to be important through ablative experiments.", "summary_of_the_review": "The paper describes an effective loss and parametrization for predicting continuous densities, as well as a simple consistency loss for unsupervised images. The transformer architecture is not demonstrated to be necessary and I wonder whether a simple conv-net would work as well using these loss functions. The ablative experiments should also be expanded, and it would be better to evaluate the supervised loss in the fully-supervised setting. I'm leaning towards accept. I may increase my rating if these issues are addressed or decrease it if they are not.", "strength_and_weaknesses": "**Strengths**\n\n1. The distribution matching loss is nice.\n1. It's neat to use the uncertainty of the predicted discrete distribution to obtain the mask. I like that the non-confident examples are ignored by the ECR loss.\n1. The supervised loss gives a significant improvement over the baseline methods in the fully-supervised setting (Table 5, appendix).\n1. To me, it was surprising that the ECR loss, despite being simple, gave a large improvement in the semi-supervised setting (Table 2).\n1. The attention map visualizations look great (appendix).\n\n**Weaknesses**\n\n1. The authors argue that predicting a distribution is more credible and less noisy than predicting a single value. However, the targets are always one-hot and the distribution is simply used to parametrize the prediction via its expectation, so it seems like simply an alternative parametrization and loss for predicting a single value. Another plausible explanation is that this parametrization and loss simply improve learnability and/or generalization for training a deep network.\n1. The loss in eq. 4 seems more general than the loss which is actually used, since $\\mathbf{y}$ is always one-hot in practice. Can the loss be simplified for one-hot targets?\n1. The transformer architecture was not well justified, either by arguments or by empirical evidence. It's not clear why it's necessary. The number of categories is fixed, so couldn't we just predict logits for the categories using a conv-net? The appendix includes some variations of decoders. What about the effect of completely removing the decoders and simply comparing the learnt density tokens to the patch features?\n1. The first self-attention within the decoder seems like it could be removed completely? It always takes the same set of tokens as input. Why not just learn the output tokens directly?\n1. Several other ablative experiments are missing. What is the effect of using ECR loss without the mask (or varying the threshold $\\xi$)? What is the effect of setting $\\omega = 0.5$ instead of using the max-norm?\n1. To compare the supervised loss alone to its baselines (Table 3), it seems like it would make more sense to consider the fully-supervised setting, rather than the 5% supervised setting. The fully-supervised comparison in Table 5 (appendix) does not include the CE baseline.\n1. In the comparison of supervised losses (Table 3), Bayesian loss (BL) and DM loss are worse than cross entropy (CE). What could explain this?\n\n**Minor issues**\n\n1. It should be more clear in the main text that \"semi-supervised\" means that the training set is a union of fully-supervised and unsupervised sets.\n1. The description of \"query initialisation\" seemed to refer to the fixed semantic meaning of the tokens, rather than the initial values of the query vectors, which I assume are randomly initialised and trained? This is unclear. Or if I have misunderstood, then the initialisation procedure was unclear.\n1. I felt that absolute error (MAE) alone would be sufficient in the main text, and would make the tables easier to interpret. The squared error (MSE) makes the tables more cluttered and doesn't add much.\n1. It wasn't clear how the 2D Gaussian smoothing was applied. Does this replace the ground-truth point annotations with a 2D Gaussian? How is the sigma chosen?\n1. Why not also use the PDM loss to encourage the two discretizations to have similar outputs? (accounting for the shift)\n1. The fact that the multi-head attention modules have multiple layers (4 layers) was not mentioned until the experimental details.\n\n**Nitpicks**\n\n* \"rationale for\" or \"motivation of\" is more appropriate than \"rationality of\"\n* Several uses of \"forward\" and \"forwards\" as a noun. It's unclear what this means. Model evaluation?\n* Some typos and grammar errors.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject