Title: Semi-Supervised Semantic Segmentation via Marginal Contextual Information
Venue: NeurIPS 2023
Accepted: no
Keywords: Semantic segmentation

semi-supervised learning

contextual information

semi-supervised segmentation
Abstract: We present a novel confidence refinement scheme that enhances pseudo-labels in semi-supervised semantic segmentation. Unlike current leading methods, which filter pixels with low-confidence teacher predictions in isolation, our approach leverages the spatial correlation of labels in segmentation maps by grouping neighboring pixels and considering their pseudo-labels collectively. With this contextual information, our method, named S4MC, increases the amount of unlabeled data used during training while maintaining the quality of the pseudo-labels, all with negligible computational overhead. Through extensive experiments on standard benchmarks, we demonstrate that S4MC outperforms existing state-of-the-art semi-supervised learning approaches, offering a promising solution for reducing the cost of acquiring dense annotations. For example, S4MC achieves a substantial 6.34 mIoU improvement over the prior state-of-the-art method on PASCAL VOC 12 with 92 annotated images. The code to reproduce our experiments is available at https://s4mcontext.github.io/

Abstract

We present a novel confidence refinement scheme that enhances pseudo-labels in semi-supervised semantic segmentation. Unlike current leading methods, which filter pixels with low-confidence teacher predictions in isolation, our approach leverages the spatial correlation of labels in segmentation maps by grouping neighboring pixels and considering their pseudo-labels collectively. With this contextual information, our method, named S4MC, increases the amount of unlabeled data used during training while maintaining the quality of the pseudolabels, all with negligible computational overhead. Through extensive experiments on standard benchmarks, we demonstrate that S4MC outperforms existing stateof-the-art semi-supervised learning approaches, offering a promising solution for reducing the cost of acquiring dense annotations. For example, S4MC achieves a substantial 6.34 mIoU improvement over the prior state-of-the-art method on PASCAL VOC 12 with 92 annotated images. The code to reproduce our experiments is available at https://s4mcontext.github.io/ .

1 Introduction

Supervised learning has been the driving force behind advancements in modern computer vision, including classification (Krizhevsky et al., 2012; Dai et al., 2021), object detection (Girshick, 2015; Zong et al., 2022), and segmentation (Zagoruyko et al., 2016; Chen et al., 2018a; Li et al., 2022; Kirillov et al., 2023). However, it requires extensive amounts of labeled data, which can be costly and time-consuming to obtain. In many practical scenarios, while there is no shortage of available data, only a fraction can be labeled due to resource constraints. This challenge has led to the development of semi-supervised learning (SSL) (Rasmus et al., 2015; Berthelot et al., 2019b; Sohn et al., 2020a; Yang et al., 2022a), a methodology that leverages both labeled and unlabeled data for model training.

This paper focuses on applying SSL to semantic segmentation, which has applications in various areas such as perception for autonomous vehicles (Bartolomei et al., 2020), mapping (Van Etten et al., 2018) and agriculture (Milioto et al., 2018). SSL is particularly appealing for segmentation tasks, as manual labeling can be prohibitively expensive.

A widely adopted approach for SSL is pseudo-labeling (Lee, 2013; Arazo et al., 2020). This technique dynamically assigns supervision targets to unlabeled data during training based on the model's predictions. To generate a meaningful training signal, it is essential to adapt the predictions

before integrating them into the learning process. Several techniques have been proposed for that, 31

such as using a teacher network to generate supervision to a student network (Hinton et al., 2015). 32

The teacher network can be made more powerful during training by applying a moving average to 33

the student network's weights (Tarvainen and Valpola, 2017). Additionally, the teacher may undergo 34

weaker augmentations than the student (Berthelot et al., 2019b), simplifying the teacher's task. 35

Beforerefinement

0.6

0.4

0.2

0.0

Cat

Bg

0.6

0.4

0.2

0.0

Cat

Bg

However, pseudo-labeling is intrinsically susceptible to confirmation bias, which tends to reinforce 36 the model predictions instead of improving the student model. Mitigating confirmation bias becomes 37 particularly important when dealing with erroneous predictions made by the teacher network. 38

One popular technique to address this issue is confidence-based filtering (Sohn et al., 2020a). This 39 approach assigns pseudo-labels only when the model's confidence surpasses a specified threshold, 40 thereby reducing the number of incorrect pseudo-labels. Though simple, this strategy was proven 41 effective and inspired multiple improvements in semi-supervised classification (Zhang et al., 2021; 42 Rizve et al., 2021), segmentation (Wang et al., 2022), and object detection in images (Sohn et al., 43 2020b; Liu et al., 2021) and 3D scenes (Zhao et al., 2020; Wang et al., 2021). However, the strict 44 filtering of the supervision signal leads to extended training periods and, potentially, to overfitting 45 when the labeled instances used are insufficient to represent the entire sample distribution. Lowering 46 the threshold would allow for higher training volumes at the cost of reduced quality, further hindering 47 the performance (Sohn et al., 2020a). 48

49

50

51

52

53

54

55

56

57

58

In response to these challenges, we introduce a novel confidence refinement scheme for the teacher network predictions in segmentation tasks, designed to increase the availability of pseudo-labels without sacrificing their accuracy. Drawing on the observation that labels in segmentation maps exhibit strong spatial correlation, we propose to group neighboring pixels and collectively consider their pseudo-labels. When considering pixels in spatial groups, we asses the event-union probability, which is the probability that at least one pixel belongs to a given class. We assign a pseudo-label if this probability is sufficiently larger that the event-union probability of any other class. By taking context into account, our approach Semi-Supervised Semantic Segmentation via Marginal Contextual Information (S4MC), enables a relaxed filtering criterion which increases the number of unlabeled pixels utilized for learning while maintaining high-quality labeling, as demonstrated in Fig. 1.

59

60

61

62

63

64

65

66

We evaluated S4MC on multiple semi-supervised segmentation benchmarks. S4MC achieves significant improvements in performance over previous state-of-the-art methods. In particular, we observed a remarkable increase of +6.34 mIoU on PASCAL VOC 12 (Everingham et al., 2010) using only 92 annotated images and an increase of +1.85 mIoU on Cityscapes (Cordts et al., 2016) using only 186 annotated images. These findings highlight the effectiveness of S4MC in producing high-quality segmentation results with minimal labeled data.

2 Related Work

2.1 Semi-Supervised Learning

Pseudo-labeling (Lee, 2013) is a popular and effective technique in SSL, where labels are assigned to 67 unlabeled data based on model predictions. To make the most of these labels during training, it is 68 essential to refine them (Laine and Aila, 2016; Berthelot et al., 2019b,a; Xie et al., 2020). One way to 69 achieve this is through consistency regularization (Laine and Aila, 2016; Tarvainen and Valpola, 2017; 70

data. Alternatively, a teacher model can be used to obtain pseudo-labels, which are then used to train 72 a student model. To ensure that the pseudo-labels are useful, the temperature of the prediction (soft 73 pseudo-labels; Berthelot et al., 2019b) can be increased, or the label can be assigned to samples with 74 high confidence (hard pseudo-labels; Xie et al., 2020; Sohn et al., 2020a; Zhang et al., 2021). In this 75 work we follow the hard pseudo-label assignment approach and improve upon previous methods by 76 proposing a confidence refinement scheme. 77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

2.2 Semi-Supervised Semantic Segmentation

In semantic segmentation, most SSL methods rely on a combination of consistency regularization and the development of augmentation strategies compatible with segmentation tasks. Given the uneven distribution of labels typically encountered in segmentation maps, techniques such as adaptive sampling, augmentation, and loss re-weighting are commonly employed (Hu et al., 2021). Feature perturbations on unlabeled data (Ouali et al., 2020; Zou et al., 2021; Liu et al., 2022b) are also used to enhance consistency, along with the application of virtual adversarial training (Liu et al., 2022b). Curriculum learning strategies that incrementally increase the proportion of data used over time are beneficial in exploiting more unlabeled data (Yang et al., 2022b; Wang et al., 2022). A recent approach introduced by (Wang et al., 2022) cleverly utilizes unreliable predictions by employing contrastive loss with the least confident classes predicted by the model. However, most existing works primarily focus on individual pixel label predictions. In contrast, we delve into the contextual information offered by spatial predictions on unlabeled data.

2.3 Contextual Information

Contextual information encompasses environmental cues that assist in interpreting and extracting 92 meaningful insights from visual perception (Toussaint, 1978; Elliman and Lancaster, 1990). 93 Incorporating spatial context explicitly has been proven beneficial in segmentation tasks, for example, 94 by encouraging smoothness like in the Conditional Random Fields (CRF) method (Chen et al., 95 2018a) and attention mechanisms (Vaswani et al., 2017; Dosovitskiy et al., 2021; Wang et al., 2020). 96 Combating dependence on context has shown to be useful by Nekrasov et al. (2021). In this work, 97 we leverage the context from neighboring pixel predictions to enhance pseudo-label propagation. 98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

3 Method

3.1 Overview

In semi-supervised semantic segmentation, we are given a labeled training set of images D ℓ = { ( x ℓ i , y i ) } N ℓ i =1 , and an unlabeled set D u = { x u i } N u i =1 sampled from the same distribution, i.e., { x ℓ i , x u i } ∼ D x . Here, y are 2D tensors of shape H × W , assigning a semantic label to each pixel of x . We aim to train a neural network f θ to predict the semantic segmentation of unseen images sampled from D x .

We follow a teacher-student approach (Tarvainen and Valpola, 2017) and train two networks f θ s and f θ t that share the same architecture but update their parameters separately. The student network f θ s is trained using supervision from the labeled samples and pseudo-labels created by the teacher's predictions for unlabeled ones. The teacher model f θ t is updated as an exponential moving average (EMA) of the student weights. f θ s ( x i ) and f θ t ( x i ) denote the predictions of the student and teacher models for the x i sample, respectively. At each training step, a batch of B ℓ and B u images is sampled from D ℓ and D u , respectively. The optimization objective can be written as the following loss:

where L s and L u are the losses over the labeled and unlabeled data correspondingly, λ is a 113 hyperparameter controlling their relative weight, and ˆ y i is the pseudo-label for the i -th unlabeled 114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

Unlabeled data

EMA

Labeled data

Student network

Teacher network

DPA

Pseudo Label

Refinement

Pseudo Label Refinement

>

<

image. Not every pixel of x i has a corresponding label or pseudo-label, and M l and M u denote the number of pixels with label and assigned pseudo-label in the image batch, respectively.

3.1.1 Pseudo-label Propagation

For a given image x i , we denote by x i j,k the pixel in the j -th row and k -th column. We adopt a thresholding-based criterion inspired by (Sohn et al., 2020a). By establishing a score, denoted as κ , which is based on the class distribution predicted by the teacher network, we assign a pseudo-label to a pixel if its score exceeds a threshold γ t :

where p c ( x i j,k ) is the pixel probability of class c . A commonly used score is given by κ ( x i j,k ; θ t ) = max c { p c ( x i j,k ) } . However, we found that using a pixel-wise margin, inspired by the work of Scheffer et al. (2001) and Shin et al. (2021), produces more stable results. This approach calculates the margin as the difference between the highest and the second-highest values of the probability vector:

3.1.2 Dynamic Partition Adjustment (DPA)

Following U 2 PL (Wang et al., 2022), we use a decaying threshold γ t . DPA replaces the fixed threshold with a quantile-based threshold that decreases with time. At each iteration, we set γ t as the α t -th quantile of κ margin over all pixels of all images in the batch. α t is defined as follows:

As the model predictions improve with each iteration, gradually lowering the threshold increases the number of propagated pseudo-labels without compromising their quality.

3.2 Marginal Contextual Information

Utilizing contextual information (Section 2.3), we look at surrounding predictions (predictions on neighboring pixels) to refine the semantic map at each pixel. We introduce the concept of 'Marginal

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

Contextual Information,' which involves integrating additional information to enhance predictions across all classes. At the same time, reliability-based pseudo-label methods focus on the dominant class only (Sohn et al., 2020a; Wang et al., 2023). Section 3.2.1 describes our confidence refinement, followed by our thresholding strategy and a description of S4MC methodology.

3.2.1 Confidence Margin Refinement

We refine the predicted pseudo-label of each pixel by considering the predictions of its neighboring pixels. Given a pixel x i j,k with a corresponding per-class prediction p c ( x i j,k ) , we examine neighboring pixels x i ℓ,m within an N × N pixel neighborhood surrounding it. We then calculate the probability that at least one of the two pixels belongs to class c :

where p c ( x i j,k , x i ℓ,m ) denote the joint probability of both x i j,k and x i ℓ,m belonging to the same class c .

While the model does not predict joint probabilities, it is reasonable to assume a non-negative correlation between the probabilities of neighboring pixels. This is largely due to the nature of segmentation maps, which are typically piecewise constant. Consequently, any information regarding the model's prediction of neighboring pixels belonging to a specific class should not lead to a reduction in the posterior probability of the given pixel also falling into that class. The joint probability can thus be bounded from below by assuming independence: p c ( x i j,k , x i ℓ,m ) ⩾ p c ( x i j,k ) · p c ( x i ℓ,m ) . By substituting this into Eq. (7), we obtain an upper bound for the event union probability:

This formulation allows us to filter out confidence margins that do not exceed the threshold.

For each class c , we select the neighbor with the maximal information gain using Eq. (8):

Computing the event union over all classes employs neighboring predictions to amplify differences in ambiguous cases. Consider, for instance, an uncertain pixel prediction with a 0 . 5 probability of belonging to one of two classes. If a neighboring pixel has a 0 . 7 probability of belonging to the first class and only a 0 . 3 probability of belonging to the second, this results in a significant event union probabilities margin of 0 . 2 . Similarly, this prediction refinement prevents the creation of over-confident predictions that is not supported by additional spatial evidence and helps in reducing confirmation bias. The refinement is visualized in Fig. 1. In our experiments, we used a neighborhood size of 3 × 3 . To determine whether the incorporation of contextual information could be enhanced with larger neighborhoods, we conducted an ablation study focusing on the neighborhood size and the neighbor selection criterion, as detailed in Table 4a. For larger neighborhoods, we decrease the probability contribution of the neighboring pixels with a distance-dependent factor:

where β ℓ,m = exp ( -1 2 ( | ℓ -j | + | m -k | ) ) is a spatial weighting function. Empirically, contextual information refinement affects mainly the most probable one or two classes. This aligns well with our choice to use the margin confidence (5).

Considering more than two events (more than one neighbor), one can use the formulation for three or four event-union. In practice, we find two event-union using Eq. (10), assign it as p c ( x i j,k ) , find the next desired event using Eq. (9) with the remaining neighbors, and repeat the process.

3.2.2 Threshold Setting

Setting a high threshold can mitigate confirmation bias from the teacher model's 'beliefs' transferring to the student model. However, this comes at the expense of learning from fewer examples, potentially resulting in a less comprehensive model. Dynamic Partition Adjustment (DPA; Wang et al., 2022) attempt to address this issue by setting a threshold that decreases over time. We adopt this method in determining the threshold from the teacher predictions pre-refinement p c ( x i j,k ) , but we filter values based on ˜ p c ( x i j,k ) . Consequently, more pixels pass the threshold that remains unaffected. We set α 0 = 0 . 4 , i.e., 60% of raw predictions pass the threshold at t = 0 , as this value demonstrated superior performance in our experiments. An ablation study for α 0 is provided in Table 4b.

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

baseline

ours

baseuncertaintyouruncertainty

3.3 Putting it All Together

We perform semi-supervised for semantic segmentation by pseudo-labeling pixels using their neighbors' contextual information. Labeled images are only fed into the student model, producing the supervised loss (Eq. (2)). Unlabeled images are fed into the student and teacher models. We sort the margin based κ margin (Eq. (5)) values of teacher predictions and set γ t as described in Section 3.2.2. The per-class teacher predictions are refined using the weighted union event relaxation, as defined in Eq. (10). Pixels with higher margin values than γ t are assigned with pseudo-labels as described in Eq. (4), producing the unsupervised loss (Eq. (3)). A visualization of the entire pipeline is depicted in Fig. 2.

The impact of S4MC is demonstrated in Fig. 4, which compares the fraction of pixels that pass the threshold with and without refinement. (a) Our method makes greater use of unlabeled data during most of the training process, (b) while the refinement ensures high-quality pseudo-labels. Qualitative results are presented in Fig. 3, where one can see both the confidence heatmap and the pseudo-labels with and without the impact of S4MC.

4 Experiments

This section presents our experimental results. The setup for the different datasets and partition protocols is detailed in Section 4.1. Section 4.2 compares our method against existing approaches and Section 4.3 provides the ablation study. Further implementation details are given in the Appendix.

4.1 Setup

Datasets In our experiments, we use PASCAL VOC 2012 (Everingham et al., 2010) and Cityscapes (Cordts et al., 2016) datasets.

The PASCAL VOC dataset comprises 20 object classes (plus background). The dataset includes 2,913 annotated images, divided into a training set of 1,464 images and a validation set of 1,449 images. In addition, the dataset includes 9,118 coarsely annotated training images (Hariharan et al., 2011), in which only a subset of the pixels are labeled. Following previous research, we conduct two sets of experiments. The 'classic' experiment utilizes only the original training set (Wang et al., 2022; Zou et al., 2021), while the 'coarse' experiment uses all available data (Wang et al., 2022; Chen et al., 2021; Hu et al., 2021).

The Cityscapes (Cordts et al., 2016) dataset includes urban scenes from 50 different cities with 30 classes, of which only 19 are typically used for evaluation (Chen et al., 2018a,b). Similarly to PASCAL, in addition to 2,975 training and 500 validation images, the dataset includes 19,998 coarsely annotated images, which we do not use in our experiment.

0

20

40

60

80

Epoch

0.6

0.7

0.8

0.9

1.0

% Data passes threshold

After refinement

Before refinement

0

20

40

60

80

Epoch

0.90

0.92

0.94

0.96

0.98

1.00

% Correct pseudo-labels

Before refinement

After refinement

Implementation details We implement S4MC on top of two framework variants: CutMix-Seg 212 (French et al., 2020) and FixMatch (Sohn et al., 2020a). Both use DeepLabv3+ (Chen et al., 2018b) 213 with a Imagenet-pre-trained (Russakovsky et al., 2015) ResNet-101 (He et al., 2016). The teacher 214 parameters θ t are updated via an exponential moving average (EMA) of the student parameters 215 Tarvainen and Valpola (2017): θ η t = τθ η -1 t +(1 -τ ) θ η s , where 0 ≤ τ ≤ 1 defines how close the 216 teacher is to the student and η denotes the training iteration. We used τ = 0 . 99 . Additional details 217 are provided in Appendix D. 218

219

220

221

222

223

224

Evaluation We compare S4MC with baselines under the common partition protocols - using 1 / 2 , 1 / 4 , 1 / 8 , and 1 / 16 of the training data as labeled data. For the 'classic' setting of the PASCAL experiment, we additionally compare using all the finely annotated images. We follow standard protocols and use mean Intersection over Union (mIoU) as our evaluation metric. We use the data split published by Wang et al. (2022) when available to ensure a fair comparisons. For the ablation studies, we use PASCAL VOC 2012 val with 1 / 4 partition.

225

226

227

Methods in comparison We compare against popular SSL segmentation methods: CutMix-Seg (French et al., 2020), CCT (Ouali et al., 2020), GCT (Ke et al., 2020), PseudoSeg (Zou et al., 2021), CPS (Chen et al., 2021), PC 2 Seg (Zhong et al., 2021), AEL (Hu et al., 2021), U 2 PL (Wang et al.,

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

2022), PS-MT (Liu et al., 2022b), and ST++ (Yang et al., 2022b). 'Supervised Only' stands for supervised training without using any unlabeled data. As a baseline, we use CutMix-Seg (French et al., 2020).

4.2 Results

PASCAL VOC 2012. Table 1 compares our method with state-of-the-art baselines on the PASCAL VOC 2012 dataset. While Table 2 shows the comparison results on the PASCAL VOC 2012 dataset with additional coarsely annotated data from SBD (Hariharan et al., 2011). In both setups, S4MC outperform all the compared methods in standard partition protocols, both when using labels only for the original PASCAL VOC 12 dataset and when using SBD annotations as well. Qualitative results are shown in Fig. 3. As can be seen our refinement procedure aids in both adding falsely filtered pseduo-labels as well as removing erroneous ones.

Cityscapes. Table 3 Presents the comparison results on the Cityscapes val dataset. Table 3 compares our method with other state-of-the-art methods on the Cityscapes (Cordts et al., 2016) dataset under various partition protocols. S4MC outperforms the compared methods in most partitions, except for the 1 / 2 setting, and combined with Fixmatch scheme, S4MC outperforms compared approaches across all partitions.

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

(a) Neighborhood choice.

(b) α 0 in Eq. (6), which controls the initial proportion of confidence pixels

Contextual information at inference. Given that our margin refinement scheme operates through prediction adjustments, we explored whether it could be employed at inference time to further enhance performance. The results reveal a negligible improvement in the DeepLab-V3-plus model, from an 85 . 7 mIOU to 85 . 71 . This underlines that the performance advantage of S4MC primarily derives from the adjusted margin, as the most confident class is rarely swapped. A heatmap of the prediction over several samples is presented in Fig. 3 and Appendix E.

4.3 Ablation Study

Weablate different components of our method using the CutMix-Seg framework variant, and evaluated using the Pascal VOC 12 dataset with a partition protocol of 1 / 4 labeled images.

Neighborhood size and neighbor selection criterion. Our prediction refinement scheme employs event-union probability with neighboring pixels, which depends on the chosen neighbor to pair with the current pixel. To assess this, we tested varying neighborhood sizes ( N = 3 , 5 , 7 ) and criteria for selecting the neighboring pixel: (a) random, (b) maximal class probability, (c) minimal class probability, and (d) two neighbors, as described in Section 3.2.1. As shown in Table 4a, a small 3 × 3 neighborhood with one neighboring pixel of the highest class probability proved most efficient in our experiments.

Threshold parameter tuning As outlined in Section 3.1.2, we utilize a dynamic threshold that depends on an initial value, α 0 . In Table 4b, we examine the effect of different initial quantiles to establish this threshold. A smaller α 0 would propagate too many errors, leading to significant confirmation bias. In contrast, a larger α 0 would mask most of the data, resulting in insufficient label propagation, rendering the semi-supervised learning process lengthy and inefficient. We found that an α 0 of 40% yields the best performance.

5 Conclusion

In this paper, we introduce S4MC, a novel approach for incorporating spatial contextual information in semi-supervised segmentation. This strategy refines confidence levels and enables us to leverage a larger portion of unlabeled data. S4MC outperforms existing approaches and achieves state-of-the-art results on multiple popular benchmarks under various data partition protocols, such as Cityscapes and Pascal VOC 12. While we believe S4MC offers a good solution to lowering the annotation requirement, it has several limitations. First, the event-union relaxation is relevant in problems where spatial coherency is expected. The generalization of our framework to other dense prediction tasks would necessitate an assessment of whether this relaxation is applicable. Furthermore, our method employs a fixed shape neighborhood without considering the structure of objects. It would be intriguing to investigate the use of segmented regions to define new neighborhoods, and this is a direction we plan to explore in the future.

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

References

Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In International Joint Conference on Neural Networks , pages 1-8, 2020. doi: 10.1109/IJCNN48605.2020.9207304. URL https://arxiv.org/abs/1908.02983 . (cited on p. 1)

Luca Bartolomei, Lucas Teixeira, and Margarita Chli. Perception-aware path planning for UAVs using semantic segmentation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages

5808-5815, 2020. doi: 10.1109/IROS45743.2020.9341347. (cited on p. 1) David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin A. Raffel. ReMixMatch: semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint , November 2019a. URL https://arxiv.org/abs/1911.09785 . (cited on p. 2) David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A. Raffel. MixMatch: a holistic approach to semi-supervised learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019b. URL https://proceedings.neurips.cc/paper/2019/ hash/1cd138d0499a68f4bb72bee04bbec2d7-Abstract.html . (cited on pp. 1, 2, and 3) Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(4):834-848, 2018a. doi: 10.1109/ TPAMI.2017.2699184. URL https://arxiv.org/abs/1412.7062 . (cited on pp. 1, 3, and 6) Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In European Conference on Computer Vision (ECCV) , September 2018b. URL https://openaccess.thecvf.com/content_ECCV_2018/ html/Liang-Chieh_Chen_Encoder-Decoder_with_Atrous_ECCV_2018_paper.html . (cited on pp. 6 and 7) Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2613-2622, June 2021. URL https://openaccess.thecvf.com/content/CVPR2021/html/ Chen_Semi-Supervised_Semantic_Segmentation_With_Cross_Pseudo_Supervision_CVPR_ 2021_paper.html . (cited on pp. 6, 7, and 8) Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. URL https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Cordts_ The_Cityscapes_Dataset_CVPR_2016_paper.html . (cited on pp. 2, 6, and 8) Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: marrying convolution and attention for all data sizes. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 39653977. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc//paper/2021/hash/ 20568692db622456cc42a2e853ca21f8-Abstract.html . (cited on p. 1) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. (cited on p. 3) Dave G. Elliman and Ian T. Lancaster. A review of segmentation and contextual analysis techniques for text recognition. Pattern Recognition , 23(3):337-346, 1990. ISSN 0031-3203. doi: https://doi.org/ 10.1016/0031-3203(90)90021-C. URL https://www.sciencedirect.com/science/article/pii/ 003132039090021C . (cited on p. 3) Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes (VOC) challenge. International Journal of Computer Vision , 88(2):303-338, June 2010. doi: 10.1007/s11263-009-0275-4. URL https://doi.org/10.1007/s11263-009-0275-4 . (cited on pp. 2, 6, 7, 15, and 16)

Geoffrey French, Samuli Laine, Timo Aila, Michal Mackiewicz, and Graham D. Finlayson. Semi-supervised semantic segmentation needs strong, varied perturbations. In British Machine Vision Conference . BMVA Press, 2020. URL https://www.bmvc2020-conference.com/assets/papers/0680.pdf . (cited on pp. 7 and 8)

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

Ross Girshick. Fast R-CNN. In IEEE International Conference on Computer Vision (ICCV) , December 2015. URL https://openaccess.thecvf.com/content_iccv_2015/html/Girshick_Fast_R-CNN_ ICCV_2015_paper.html . (cited on p. 1)

Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In International Conference on Computer Vision , pages 991-998, 2011. doi: 10.1109/ICCV.2011.6126343. (cited on pp. 6 and 8)

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. URL https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/He_Deep_ Residual_Learning_CVPR_2016_paper.html . (cited on p. 7)

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. (cited on p. 1)

Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 22106-22118. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/ file/b98249b38337c5088bbc660d8f872d6a-Paper.pdf . (cited on pp. 3, 6, 7, and 8)

Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson W. H. Lau. Guided collaborative training for pixelwise semi-supervised learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision , pages 429-445, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58601-0. URL https://www.ecva.net/papers/eccv_2020/papers_ ECCV/html/1932_ECCV_2020_paper.php . (cited on pp. 7 and 8)

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv preprint arXiv:2304.02643 , 2023. (cited on p. 1)

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012. URL https://papers.nips. cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html . (cited on p. 1)

Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations , 2016. URL https://openreview.net/forum?id=BJ6oOfqge . (cited on p. 2)

Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop: Challenges in Representation Learning (WREPL) , July 2013. URL http://deeplearning.net/wp-content/uploads/2013/03/pseudo_label_final.pdf . (cited on pp. 1 and 2)

Feng Li, Hao Zhang, Huaizhe xu, Shilong Liu, Lei Zhang, Lionel M. Ni, and Heung-Yeung Shum. Mask DINO: towards a unified transformer-based framework for object detection and segmentation. arXiv preprint , June 2022. URL https://arxiv.org/abs/2206.02777 . (cited on p. 1)

Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew J. Davison. Bootstrapping semantic segmentation with regional contrast. In International Conference on Learning Representations , 2022a. URL https: //openreview.net/forum?id=6u6N8WWwYSM . (cited on p. 7)

Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=MJIve1zgR_ . (cited on p. 2)

Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, and Gustavo Carneiro. Perturbed and strict mean teachers for semi-supervised semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4258-4267, June 2022b. URL https://openaccess.thecvf.com/content/CVPR2022/html/Liu_Perturbed_and_Strict_ Mean_Teachers_for_Semi-Supervised_Semantic_Segmentation_CVPR_2022_paper.html . (cited on pp. 3, 7, and 8)

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

Andres Milioto, Philipp Lottes, and Cyrill Stachniss. Real-time semantic segmentation of crop and weed for precision agriculture robots leveraging background knowledge in CNNs. In IEEE International Conference on Robotics and Automation (ICRA) , pages 2229-2235, 2018. doi: 10.1109/ICRA.2018.8460962. URL https://arxiv.org/abs/1709.06764 . (cited on p. 1)

Takeru Miyato, Shin-Ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41(8):1979-1993, 2018. doi: 10.1109/TPAMI.2018.2858821. URL https://ieeexplore. ieee.org/abstract/document/8417973 . (cited on p. 2)

Alexey Nekrasov, Jonas Schult, Or Litany, Bastian Leibe, and Francis Engelmann. Mix3d: Out-of-context data augmentation for 3d scenes. 3DV 2021 , 2021. (cited on p. 3)

Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020. URL https://openaccess.thecvf.com/content_CVPR_2020/html/Ouali_ Semi-Supervised_Semantic_Segmentation_With_Cross-Consistency_Training_CVPR_2020_ paper.html . (cited on pp. 3, 7, and 8)

Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://papers. nips.cc/paper/2015/hash/378a063b8fdb1db941e34f4bde584c7d-Abstract.html . (cited on p. 1)

Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S. Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=-ODN6SbiUU . (cited on p. 2)

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision , 115(3):211-252, 2015. doi: 10.1007/ s11263-015-0816-y. (cited on p. 7)

Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden Markov models for information extraction. In Frank Hoffmann, David J. Hand, Niall Adams, Douglas Fisher, and Gabriela Guimaraes, editors, Advances in Intelligent Data Analysis , pages 309-318, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg. ISBN 978-3-540-44816-7. URL https://link.springer.com/chapter/10.1007/3-540-44816-0_ 31 . (cited on pp. 4 and 15)

Gyungin Shin, Weidi Xie, and Samuel Albanie. All you need are a few pixels: Semantic segmentation with pixelpick. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops , pages 1687-1697, October 2021. URL https://openaccess.thecvf.com/content/ICCV2021W/ ILDAV/html/Shin_All_You_Need_Are_a_Few_Pixels_Semantic_Segmentation_With_ICCVW_ 2021_paper.html . (cited on pp. 4 and 15)

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A. Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: simplifying semi-supervised learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 596608. Curran Associates, Inc., 2020a. URL https://proceedings.neurips.cc/paper/2020/hash/ 06964dce9addb1c5cb5d6e3d9838f733-Abstract.html . (cited on pp. 1, 2, 3, 4, 5, and 7)

Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister. A simple semisupervised learning framework for object detection. arXiv preprint arXiv:2005.04757 , 2020b. (cited on p. 2)

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/ hash/68053af2923e00204c3ca7c6a3150cf7-Abstract.html . (cited on pp. 1, 2, 3, and 7)

Godfried T. Toussaint. The use of context in pattern recognition. Pattern Recognition , 10(3):189-204, 1978. ISSN 0031-3203. doi: https://doi.org/10.1016/0031-3203(78)90027-4. URL https://www.sciencedirect. com/science/article/pii/0031320378900274 . The Proceedings of the IEEE Computer Society Conference. (cited on p. 3)

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

Adam Van Etten, Dave Lindenbaum, and Todd M. Bacastow. SpaceNet: a remote sensing dataset and challenge series. arXiv preprint , June 2018. URL https://arxiv.org/abs/1807.01232 . (cited on p. 1)

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser,

and Illia Polosukhin. Attention is all you need, 2017. (cited on p. 3) He Wang, Yezhen Cong, Or Litany, Yue Gao, and Leonidas J. Guibas. 3DIoUMatch: leveraging IoU prediction for semi-supervised 3D object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14615-14624, June 2021. URL https://openaccess.thecvf.com/content/CVPR2021/html/Wang_3DIoUMatch_Leveraging_ IoU_Prediction_for_Semi-Supervised_3D_Object_Detection_CVPR_2021_paper.html . (cited on p. 2) Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Eca-net: Efficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11534-11542, 2020. (cited on p. 3) Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. FreeMatch: self-adaptive thresholding for semi-supervised learning. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=PDrUPTXJI_A . (cited on p. 5) Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo labels. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. URL https://openaccess.thecvf.com/content/CVPR2022/html/Wang_Semi-Supervised_ Semantic_Segmentation_Using_Unreliable_Pseudo-Labels_CVPR_2022_paper.html . (cited on pp. 2, 3, 4, 5, 6, 7, and 8) Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 62566268. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 44feb0096faa8326192570788b38c1d1-Abstract.html . (cited on pp. 2 and 3) Fan Yang, Kai Wu, Shuyi Zhang, Guannan Jiang, Yong Liu, Feng Zheng, Wei Zhang, Chengjie Wang, and Long Zeng. Class-aware contrastive semi-supervised learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14421-14430, June 2022a. URL https://openaccess.thecvf.com/content/CVPR2022/html/Yang_Class-Aware_ Contrastive_Semi-Supervised_Learning_CVPR_2022_paper.html . (cited on p. 1) Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. ST++: make self-training work better for semi-supervised semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4268-4277, June 2022b. URL https://openaccess.thecvf.com/ content/CVPR2022/html/Yang_ST_Make_Self-Training_Work_Better_for_Semi-Supervised_ Semantic_Segmentation_CVPR_2022_paper.html . (cited on pp. 3, 7, and 8) Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: regularization strategy to train strong classifiers with localizable features. In IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019. URL https: //openaccess.thecvf.com/content_ICCV_2019/html/Yun_CutMix_Regularization_Strategy_ to_Train_Strong_Classifiers_With_Localizable_Features_ICCV_2019_paper.html . (cited on p. 16)

Sergey Zagoruyko, Adam Lerer, Tsung-Yi Lin, Pedro O. Pinheiro, Sam Gross, Soumith Chintala, and Piotr Dollár. A MultiPath network for object detection. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC) , pages 15.1-15.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C.30.15. URL https://dx.doi.org/10. 5244/C.30.15 . (cited on p. 1)

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 18408-18419. Curran Associates, Inc., 2021. URL https://proceedings. neurips.cc/paper/2021/hash/995693c15f439e3d189b06e89d145dd5-Abstract.html . (cited on pp. 2 and 3)

Na Zhao, Tat-Seng Chua, and Gim Hee Lee. SESS: self-ensembling semi-supervised 3D object detection. 495 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 496 June 2020. URL https://openaccess.thecvf.com/content_CVPR_2020/html/Zhao_SESS_ 497 Self-Ensembling_Semi-Supervised_3D_Object_Detection_CVPR_2020_paper.html . (cited on p. 498 2) 499

Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive-consistent 500 semi-supervised semantic segmentation. In IEEE/CVF International Conference on Computer Vision (ICCV) , 501 pages 7273-7282, October 2021. URL https://openaccess.thecvf.com/content/ICCV2021/html/ 502 Zhong_Pixel_Contrastive-Consistent_Semi-Supervised_Semantic_Segmentation_ICCV_ 503 2021_paper.html . (cited on p. 7) 504 Zhuofan Zong, Guanglu Song, and Yu Liu. DETRs with collaborative hybrid assignments training. arXiv 505 preprint , November 2022. URL https://arxiv.org/abs/2211.12860 . (cited on p. 1) 506 Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. 507 PseudoSeg: designing pseudo labels for semantic segmentation. In International Conference on Learning 508 Representations , 2021. URL https://openreview.net/forum?id=-TwO99rbVRu . (cited on pp. 3, 6, 509 and 7) 510

Peer reviews:
comments: SUMMARY:
The submission proposes a method to improve the quality of pseudo labels for semi-supervised semantic segmentation. The proposed approach leverages the spatial correlation of labels in segmentation maps by grouping neighboring pixels and considering their pseudo-labels, rather than generating pseudo labels at each pixel independently. Experiments show improvements over state-of-the-art semi-supervised semantic segmentation methods.

STRENGTHS:
The proposed approach is conceptually simple and experimental results indicate it is effective.

WEAKNESSES:
1. It is not clear to me what is the improvement from considering neighbors when generating pseudo labels. It would be good if Table 4 (a) has an extra column with experiment using neighbor size 1x1, meaning no neighbor is used. Right now, the only comparison is with SOTA models that do not consider context, but the comparison is not convincing as there might be implementation differences. Also, why using a larger neighbor size hurts performance in Table 4 (a)?
2. The experiment setting is still a bit unclear to me. It would be great to state what images are used as labeled images and what images are unlabeled. Also, I think there misses an upper bounding experiment with the same model trained with all annotated images. Right now the SSL model is outperforming fully-supervised model, which does not make much sense to me.
3. Since Pascal VOC is a relatively small dataset, can we use more unlabeled images (e.g., COCO) to further improve performance?

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
Please see weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
In this paper, the authors propose a new confidence refinement scheme called S4MC for improving the pseudo-labels in semi-supervised semantic segmentation. Unlike existing methods, S4MC considers the spatial correlation of labels in segmentation maps by grouping neighboring pixels and collectively analyzing their pseudo-labels. This approach allows for the utilization of more unlabeled data during training while maintaining the quality of the pseudo-labels, all without significant computational overhead. Through extensive experiments on standard benchmarks, the authors demonstrate that S4MC surpasses current state-of-the-art methods in semi-supervised learning, offering a promising solution to reduce the cost of acquiring dense annotations.

STRENGTHS:
+ The paper is written in a clear and concise manner, effectively presenting the motivation and ideas behind the proposed approach. 
+ The proposed approach is rigorously evaluated through extensive experiments performed on multiple benchmarks.

WEAKNESSES:
- The contribution of this paper is somewhat limited, as the idea of leveraging contextual information from neighboring pixel predictions to enhance segmentation has been extensively explored in the literature.
- In Tables 1-3, it is evident that S4MC + FixMatch (Ours) demonstrates superior performance compared to S4MC + CutMix-Seg (Ours). However, the performance of FixMatch, U2PL+FixMatch, and PS-MT+FixMatch  is not explicitly mentioned in the given context.

QUESTIONS:
Please refer to the Weaknesses.

LIMITATIONS:
The authors have adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on semi-supervised semantic segmentation. To generate more accurate pseudo-labels, this work attempts to leverage the spatial correlation of labels in segmentation maps and proposes a confidence margin refinement module to model the contextual information. Extensive experiments have been conducted to validate the effectiveness of this method. The proposed method achieves promising performance on different datasets and settings.\n\nSTRENGTHS:\nStrengths:\nThe idea is interesting and the writing is clear.\nThe effectiveness of the proposed method has been validated on several standard benchmarks.\n\nWEAKNESSES:\nWeakness:\nThe time complexity of the proposed confidence margin refinement (CMR) module is necessary to be analyzed. According to L153, Eq9, and L168-170, the neighbor pixel selection will be performed for each class at every pixel location, that is the time complexity is proportional to the number of pixels, categories, and the number of unioned events. Since semantic segmentation is a dense prediction task, such a computation cost may not be neglected. \nThe example presented in L155-158 is not convincing to support the argument in L154. In this example, the author only considers one specific neighboring pixel which has a higher probability for the first class, but for the second class, it should search again with the Eq9 and that is not necessary to select the previous neighboring pixel but to select a pixel whose probability for the second class can most contribute to the interested uncertain pixel in a predefined neighborhood. Especially at edge pixels where the category changes, this kind of argument is hardly guaranteed. \nL158-160 argues that the proposed CMR module can prevent the creation of over-confident predictions. However, the refined prediction \\tilde{p}_c((x_{j,k})) = p_c(x_{j,k}) + p_c(x_{l,m}) (1 - p_c(x_{j,k})) based on Eq8 will definitely be greater than or at least equal to the original one p_c(x_{j,k}), that is the refined prediction will become more confident. It seems the argument is not reasonable.\nThe difference between S4MC+CutMix-seg and S4MC-FixMatch requires more explanation. Since the student-teacher modeling framework seems to be the default setting in this work and the CutMix operation is normally taken as a strong data augmentation in a semi-supervised semantic segmentation task[1][2], does that mean the latter method uses a stronger data augmentation to achieve a better performance? If it is, then what is the data augmentation? \nThe latest comparing methods is published in CVPR2022. Due to the rapid development of the semi-supervised semantic segmentation area, it is necessary to compare the proposed approach with more up-to-date methods, such as [1][2]. It seems that the proposed method is comparable to [1] and inferior to [2] in the CutMix setting.\n[1] Semi-supervised Semantic Segmentation with Prototype-based Consistency Regularization, NeurIPS2022\n[2] Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation, CVPR2023\n\nQUESTIONS:\nPlease see weakness above.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo incorporate spatial contextual information in Semi-supervised segementation, the authors proposed S4MC, which refines confidence with the help of neighbors. This interesting refinement scheme aids in both adding falsely filtered pseudo-lables as well as removing erroeous ones. Additionally, a Dynamic Partition Adjustment module is employed to enable more pseduo-labeled pixels to be used. Results on several settings verified the effectiveness of S4MC.\n\nSTRENGTHS:\n* the confidence refinement scheme is novel and effective\n* the evaluation is comprehensive and the results achieve new state-of-art\n* the paper is well-written and the sturcture is clear\n* easy to be followed\n\nWEAKNESSES:\nplease refer to the questions listed below.\n\nQUESTIONS:\n* Some quantitative results should be provided to support the statement \"refinement procedure aids in both adding falsely filtere pseduo-labels as well as removing erroneous ones.\", instead of only qualitative analysis in Figure 3.\n* Is the confidence refinement scheme independent of specific frameworks and can it be combined with other pseudo-label-based semi-supervised semantic segmentation methods (i.e., is it orthogonal to other methods)? No need for further experiments, just a small discussion here.\n* What are the relationships between S4MC, CutMix-Seg, FixMatch, S4MC+CutMix-Seg and S4MC+FixMatch?\n* It seems that the *DPA* is also an important component of the overall framework. However, since DPA is proposed by a previously published work. And the improvements brought by S4MC may come from both DPA and confidence refinement. Is it primarily driven by confidence refinement? Since confidence refinement is the most novel aspect of this paper. The authors should provide experimental results to demonstrate that the improvement brought by confidence refinement is the most significant part.\n\nLIMITATIONS:\nThe authors should further discuss potential negative societal impact of this work and also should discuss the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}