Semi-Supervised Semantic Segmentation via Marginal Contextual Information
Authors: Moshe Kimhi
Shai Kimhi
Evgenii Zheltonozhskii
Or Litany
Chaim Baskin
Venue: Accepted by TMLR ()

Abstract: We present a novel confidence refinement scheme that enhances pseudo-labels in semi-supervised semantic segmentation. Unlike existing methods, which filter pixels with low-confidence predictions in isolation, our approach leverages the spatial correlation of labels in segmentation maps by grouping neighboring pixels and considering their pseudo-labels collectively. With this contextual information, our method, named S4MC, increases the amount of unlabeled data used during training while maintaining the quality of the pseudo-labels, all with negligible computational overhead. Through extensive experiments on standard benchmarks, we demonstrate that S4MC outperforms existing state-of-the-art semi-supervised learning approaches, offering a promising solution for reducing the cost of acquiring dense annotations. For example, S4MC achieves a 1.39 mIoU improvement over the prior art on PASCAL VOC 12 with 366 annotated images. The code to reproduce our experiments is available at https://s4mcontext.github.io/

Published in Transactions on Machine Learning Research (06/2024)
Semi-Supervised Semantic Segmentation via Marginal Con- textual Information
Moshe Kimhi Computer Science Department, Technion
Shai Kimhi
shaikimhi221 19nokimmh222119000all..cmm
Computer Science Department, Technion
Evgenii Zheltonozhskii
Physics Department, Technion
Or Litany
orlihanny&techhcion..o
Computer Science Department, Technion NVIDIA
Chaim Baskin
Aucinbbaskin&tteccniiin...
Computer Science Department, Technion
Reviewed on OpenReview: https: openreview. net/ forun? id= 65yKWipmjW
Abstract
We present a novel confidence refinement scheme that enhances pscudo labels in semi- supervised semantic segmentation. Unlike existing methods, which filter pixels with low- confidence predictions in isolation, our approach leverages the spatial correlation of labels in segmentation maps by grouping neighboring pixels and considering their pseudo labels collectively. With this contextual information, our method, named S4MC, increases the amount of unlabeled data used during training while maintaining the quality of the pseudo labels, all with negligible computational overhead. Through extensive experiments standard benchmarks, semi-superiised learning approaches, offering promising solution for reducing the cost of acquiring dense annotations. For example, S4MC achieves a 1.39 mIoU improvement over the prior art on PASCAL VOC 12 with 366 annotated images. The code to reproduce our experiments available at
demonstrate that S4MC outperforms existing state-of-the-art
gi github.ii
1 Introduction
Supervised learning has been the driving force behind advancements in modern computer vision, including classification (Krizhevsky et al., 2012; Dai al., 2021), object detection (Girshick, 2015; Zong et al., 2022), and segmentation (Zagoruyko al., 2016; Chen et al., 2018a; Li et al., 2022; Kirillov et al., 2023). However, it requires extensive amounts of labeled data, which can be costly and time-consuming to obtain. In many practical scenarios, there is no shortage of available data, but only fraction can be labeled due to resource constraints. This challenge has led to the development of semi-supervised learning (SSL; Rasmus et al., 2015; Berthelot et al., 2019; Sohn et. al., 2020a; Yang et al., 2022a), a methodology that leverages both labeled and unlabeled data for model training.
This paper focuses on applying SSL to semantic segmentation, which has applications in various areas such perception for autonomous vehicles (Bartolomei al., 2020), mapping (Van Etten et al., 2018) and

---

Published in Transactions on Machine Learning Research (06/2024)
Before refinement
After refinement
7/
0.4
0.2
0.0
Cat Bg
7
0.4
0.2
0.0
Cat Bg
Figure 1: Confidence refinement observation over one class (Cat). Left: pseudo labels generated without refinement. Middle: pseudo labels obtained from the same model after refinement with marginal contextual information. Right Top: prodicted probabilities of the top two classes of the pixel highlighted by the red square before, and Bottom: after refinement. S4MC allows additional correct pseudo labels to propagate.
agriculture (Milioto et al., 2018). SSL is particularly appealing for segmentation tasks, as manual labeling can be prohibitively expensive.
A widely adopted approach for SSL psoudo labeling (Loe, 2013). This technique dynamically assigns supervision targets to unlabeled data during training based on the model's predictions. To generate a meaningful training signal, it is essential adapt the predictions before integrating them into the learning process. Several techniques have been proposed, such as using teacher network to generate supervision to a student network (Tarvainen & Valpola, 2017; Ke et al., 2019; Cai et al., 2022). Additionally, the teacher may undergo weaker augmentations than the student (Berthelot et al., 2019), simplifying the teacher's lask.
However, pseudo labeling is intrinsically susceptible to confirmation bias, which tends to reinforce the model predictions instead of improving the student model. Mitigating confirmation bias becomes particularly important when dealing with erroneous predictions made by the teacher network.
Confidence- basedd filtering is a popular technique to address this issue (Sohn et al., 2020a). This approach assigns pseudo labels only when the model's confidence surpasses specified threshold, reducing the nunber of incorrect pseudo labels. Though simple, this strategy was proven effective and inspired multiple improvements in semi-supervisod classification (Zhang et al., 2021; Rizve et al., 2021), segmentation (Wang et al., 2022), and object detection (Sohn et al., 2020b; Liu et al., 2021; Zhao et al., 2020; Wang et al., 2021). However, the strict filtering of the supervision signal leads to extended training periods and, potentially, to overfitting when the labeled instances are insufficient to represent the entire sample distribution. Lowering the threshold would allow for higher training volumes at the cost of reduced quality, further hindering the performance (Sohn et al., 2020a).
In response to these challenges, we introduce a novel confidence refinement scheme for the teacher network predictions in segmentation tasks designed to increase the availability of pseudo labels without sacrificing their accuracy. Drawing on the observation that labels in segmentation maps exhibit strong spatial correlation, we propose to group neighboring pixels and collectively consider their pscudo labels. When considering pixels in spatial groups, we asses the event-union probability, which is the probability that at least one pixel belongs to given class. We assign pseudo label if this probability is sufficiently larger than the event-union probability of any other class. By taking context into account, our approach Semi-Superuised Semantic Segmentation via Marginal Contertual Information (S4MC), enables a relaxed filtering criterion which increases the number of unlabeled pixels utilized for learning while maintaining high-quality labeling, as demonstrated in Fig. 1.
We evaluated S4MC on multiple benchmarks. S4MC achieves significant improvements performance over previous state-of-the-ee methods. In particular, we observed an increase of +1.39 mIoU on PASCAL VOC 12 (Everingham et al., 2010) using 366 annotated images, +1.01 mloU on Cityscapes (Cordts et al., 2016)
2

---

Published in Transactions on Machine Learning Research (06/2024)
using only 186 annotated images, and increase +1.5 mloU COCO (Lin et al., 2014) using 463 annotated images. These findings highlight the effectiveness of S4MC in producing high-quality segmentation results with minimal labeled data.
2 Related Work
2.1 Semi-Supprrised Learning
Pseudo labeling (Tee, 2013) is an effective technique SSL, where labels are assigned to unlabeled data based on model predictions. To make the most of these labels during training, it is essential to refine them (Laine & Aila, 2016; Berthelot et al., 2019; 2020; Xie al., 2020). This be done through consistency regularization (Laine & Aila, 2016; Tarvainen & Valpola, 2017; Miyato et al., 2018), which ensures consistent predictions between different views or different models' prediction of the unlabeled data. To ensure that the pseudo labels are helpful, the temperature of the prediction (soft pseudo labels; Berthelot et al., 2019) can be increased, or the label can be assigned to samples with high confidence (hard pseudo labels; Xie et al., 2020; Sohn et al., 2020a; Zhang et al., 2021).
2.2 Semi-Superviset Semantic Segmentation
In semantic segmentation, most SSL methods rely consistency regularization and developing augmentation strategies compatible with segmentation tasks (French et al., 2020; Ke et al., 2020; Chen et al., 2021; Zhong et al., 2021; Xu et al., 2022). Given the uneven distribution of labels typically encountered in segmentation maps, techniques such as adaptive sampling, augmentation, and loss re- weiieight are commonly employed (Hu et al., 2021). Feature perturbations (FP) on unlabeled data (Ouali et al., 2020; Zou et al., 2021; Liu et al., 2022b; Yang et al., 2023) are also used to enhance consistency and the virtual adversarial training (Liu et al., 2022b). Curriculum learning strategics that incrementally increase the proportion of data used over time are beneficial in exploiting more unlabeled data (Yang et al., 2022b; Wang et al., 2022). A recent approach introduced by Wang et al. (2022) included unreliable pseudo labels into training by employing contrastive loss with the least confident classes predicted by the model. Unimatch (Yang et al., 2023) combined SSL (Sohn et al., 2020a) with several self-supervision signals, i.e., two strong augmentations and one more with FP, obtained good results without complex losses or class-level heuristics. However, most existing works primarily focus on individual pixel label predictions. In contrast, we delve into the contextual information offered by spatial predictions on unlabcled data.
2.3 Contextual Information
Contextual information encompasses environmental cues that assist in interpreting and extracting meaningful insights from visual perception (Toussaint, 1978; Elliman & Lancaster, 1990). Incorporating spatial context explicitly has been proven beneficial in segmentation tasks, for example, by encouraging smoothness like in the Conditional Random Fields method (Chen et al., 2018a) and attention mechanisms (Vaswani et al., 2017; Dosovitskiy et al., 2021; Wang et al., 2020). Combating dependence on context has shown to be helpful by Nekrasov al. (2021). This work uses the context from neighboring pixel predictions to enhance pseudo label propagation.
3 Method
This section describes the proposed method using the student paradigm with teacher averaging (Tarvainen & Valpola, 2017). Adjustments for image-level consistency are described in Appendix F.
3.1 Overview
In semi-supervised semantic segmentation, we are given a labeled training set Dj {({ y))) and an unlabeled set D. (xp) sampled from the same distribution, i.e., {xf, x++] Dz- Here, y are 2D tensors
3

---

Published in Transactions on Machine Learning Research (06/2024)
Pseudo Label Refinement
Stucient network
DPA
Unlabeled data
Teacher
Refinement
Figure 2: Left: S4MC employs a teacher-student paradigm for semi-supervised segmentation. Labeled images are used to supervise the student network directly; both networks process unlabeled images. Teacher predictions are refined and used to evaluate the margin value, which is then thresholded to produce pseudo labels that guide the student network. The threshold, denoted as Tt dynamically adjusted based on the teacher network's predictions. Right: Our confidence refinement module exploits neighboring pixels adjust per-class predictions, as detailed in Section 3.2.1. The class distribution of the pixel marked by the yellow circle the left is changed. Before refinement, the margin surpasses the threshold and erroneously assigns the blue class (dog) as pseudo label. After refinement, the margin reduces, thereby preventing error propagation.
of shape x W, assigning a semantic label to each pixel of x. We aim to train a neural network So to predict the semantic segmentation of unseen images sampled from Dz.
We follow a gacerrr-v approach and train two networks fo, and fe, that share the same architecture but update their parameters separately. The student network Se, is trained using supervision from the labeled samples and pseudo labels created by the teacher's predictions for unlabeled ones. The teacher model fo, is updated as exponential moving average (EMA) of the student weights. fe, (x,) and fa, (x,) denote the predictions of the student and teacher models for the sample, respectively. At each training step, batch of B3 and B, images sampled from D, and D.,, respectively. The optimization objective can be written as the following loss:
L = L XL
(1)
C. Mi
Σ (x)())((x())))
(2)
x(avE8
C. M x E.
1
Σ
lo (x+)
(3)
where L. and L., are the losses over the labeled and unlabeled data correspondingly, À is a hyperparameter controlling their relative weight, and ÿ, is the pseudo label for the i-th unlabeled image. Not every pixel of x; has corresponding label or pseudo label, and Mi and M, denote the number of pixels with label and assigned pscudo label in the image batch, respectively.
3.1.1 Pseudo Label Propagation
For a given image x,, we denote by 2),a the pixel the j-th row and k-th column. We adopt thresholding- based criterion inspired by FixMatch (Sohn al., 2020a). By establishing a score, denoted as K, which is based the class distribution predicted by the teacher network, assign pseudo label to pixel if its
4

---

Published in Transactions on Machine Learning Research (06/2024)
Figure 3: Qualitative results. Segmentation mask from left right: Ground Truth, CutMix-Seg and CutMix-Seg+ Heat map left is CutMix-Seg and right C SSMC, represents the uncertainty of the model (x- showing more confident predictions in certain areas and smoother segmentation maps (marked by the red boxes). Additional examples are shown in Appendix A.
exceeds threshold yx
8/ arg max, (po(s),)) if (())) > n
(4)
ignore
otherwise,
where po(x))e) the pixel probability of class c. A commonly used score given by x(a),
(ax However, using a pixel-wise margin (Scheffer et al., 2001; Shin et al., 2021), produces more stable results. Denoting by max2 the second-highest value, the margin is given by the difference between the highest and the second-hise highest values of the probability vector:
Kmargia (z),a) (ax mpx)(p((1),)))
(5)
3.1.2 Dynamic Partition Adjustment (DPA)
Following Wang et al. (2022), we use a decaying threshold 7t- DPA replaces the fixed threshold with a quantile-based threshold that decreases with time. At each iteration, we set Yt as the a,-th quantile of Emargin over all pixels of all images in the batch. a linearly decreases from ag zero during the training:
ao 1/steaation))
As the model predictions improve with each iteration, gradually lowering the threshold increases the number of propagated pscudo labels without compromising quality.
3.2 Marginal Contextual Information
Utilizing contextual information (Section 2.3), we look at surrounding predictions (predictions on neighboring pixels) to refine the semantic map at each pixel. We introduce the concept "Marginal Contextual Information," " which involves integrating additional information to enhance predictions across all classes. the same time, reliability- Wang et al., 2023). Section 3.2.1 describes our confidence refinement, followed by our thresholding strategy and description of S4MC methodology.
pseudo label methods focus on the dominant class only (Sohn et al., 2020a;
3.2.1 Confidence Margin Refinement
We refine each pixel's predicted pscudo label by considering its neighboring pixels' predictions. Given pixel 2) with corresponding per-class prediction (=())),)) examine neighboring pixels zi.,. within N x N
5

---

Published in Transactions on Machine Learning Research (06/2024)
pixel neighborhood surrounding it. As example for using one neighbor, we then calculate the probability that at least one of the two pixels belongs to class
Pelaiju U (l.m)) = pe(a),n) + Pe(ximm) Pelaraas
(6)
where po(afj,= mmmm) denote the joint probability of both of, and si.m. belonging to the same class c.
While the model does not predict joint probabilities, assuming non-negative correlation between the probabilities of neighboring pixels is reasonable. This is mainly due to the nature of segmentation maps, which are typically piecewise constant.The joint probability can thus be bounded from below by assuming independence: ((s)),:=m,m 2 p.(r),,) pa( By substituting this into Eq. (6), we obtain an upper bound for the event union probability:
Po(a), U (l S po(af,)) + po(xk,)) - po(a),)) po(xk.m).
(7)
For each class e, we select the neighbor with the maximal information utilization using Eq. (7):
Pe De(x)),) max paaxpe U x2m))
(8)
f.m.
Computing the event union over all classes employs neighboring predictions amplify differences in ambiguous cases. Similarly, this prediction refinement prevents the creation of over-confident predictions not supported by additional spatial evidence and helps reduce confirmation bias. The refinement is visualized in Fig. 1. In our experiments, we used a neighborhood size of 3 x 3. To determine whether the incorporation of contextual information could be enhanced with larger neighborhoods, we conducted an ablation study focusing on the neighborhood size and the neighbor selection critcrion, as detailed in Table For larger noighborhoods, we decrease the probability contribution of the neighboring pixels with distance-dependent factor:
Fe(x)),) pe(x),,) Be.m [po(s).m) Po(a) =/ mm,,
where Be.m = exp(-2 jl + Im - k)) is a spatial weighting function. Empirically, contextual information
refinement alfects mainly the most probable one or two classes. This aligns well with our choice to use the margin confidence (5).
We explored alternatives at the pixel level, such the maximum class probability (Kmax) and entropy (Nent)- Table F.1 in the Appendix studies the impact of different confidence functions on pseudo label refinement.
Considering multiple neighbors, we can use the formulation for three or more events. In practice, we use the associative law of set theory calculate it iteratively by grouping two events at each iteration, and exclude the chosen one from the remaining neighborhood.
3.2.2 Threshold Setting
A high threshold can prevent transferring the teacher model's wrong "belicfs" to the student model. However, this comes at the expense of learning from fewer examples, resulting in less comprehensive model. To determine the DPA threshold, we use the teacher predictions pre-refinement pa(a),)). but we filter values based Fe(w Consequently, more pixels pass the (unchanged) threshold. We tuned og value in Table 6 and set ag = 0.4, i.e., 60% of raw predictions pass the threshold at t = O.
3.3 Putting it All Together
We perform semi-supervised learning for semantic segmentation by pseudo labeling pixels using their neighbors' contextual information. Labeled images and student model prediction used for the supervised loss (2). Unlabeled images are processed by both student and leacher models. We use Kmargin (5) of teacher predictions, sort them, and set the threshold 7t Section 3.2.2). The per-class teacher predictions are refined using the weighted union event relaxation, as defined in Eq. (9). Pixels with top class matching original label and margin values higher than " are assigned pseudo labels as described in Eq. (4), for the unsupervised loss (3). The entire pipeline is visualized in Fig. 2.

---

Published in Transactions on Machine Learning Research (06/2024)
Data passes threshold
1.0
After refinement
Correct 1seudo-labels
1.00
Before refinement
0.98
0.9
0.96
0.8
0.94
m
0.7
0.92
Before refinement After refinement
0.90
0.6
20
60
80
20
Epoch
Epoch
(a) Data fraction that passes the threshold. Our
(b) Accuracy of the pseudo labels. S4MC produces method increases the number of pseudo labeled pixels, more quality pseudo labels during the training process, mostly in the early stage of the training.
most notably the early stages.
Figure 4: pseudo label quantity and quality on PASCAL VOC 12 (Everingham el. al., 2010) with 366 labeled images using our margin (5) confidence function. The training was performed using S4MC; metrics with and without S4MC were calculated.
Table 1: Comparison between our method and prior art on the PASCAL VOC 12 val (1,464 original annotated images out of 10,582 total) under different partition protocols using ResNet-101 backbone. The caption describes the share of the training set used as labeled data and the actual number of labeled images. denotes reproduced results using official implementation. + denotes the standard deviation over three runs.
Method
1/16 (92)
1/8 (183)
1/4 (366)
1/2 (732)
Full (1464)
CutMix-Seg (French al., 2020)
52.16
63.47
69.46
73.73
76.54
ReCo (Liu et al., 2022a)
64.80
72.0
73.10
74.70
ST++ (Yang et al., 2022b)
65.2
71.0
74.6
77.3
79.1
UPPL (Wang et al., 2022)
67.98
69.15
73.66
76.16
79.49
PS-MT (Liu et al., 2022b)
65.8
69.6
76.6
78.4
80.0
PCR (Xu al., 2022)
70.06
74.71
77.16
78.49
80.65
FixMatch* (Yang et al., 2023)
68.07
73.72
76.38
77.97
79.97
UniMatch (Yang et al., 2023)
73.75
75.05
77.7
79.9
80.43
CutMix-Seg + S4MC
70.96
71.69
75.41
77.73
80.58
FixMatch S4MC
73.13
74.72
77.27
79.07
79.6
UniMatch" S4MC
74.72
0.283
75.21
79.09
0.183
80.12
0.120
81.56
0.103
The impact of S4MC is shown in Fig. 4, comparing the fraction of pixels that pass the threshold with and without refinement. S4MC uses more unlabeled data during most of the training (a), while the refinement ensures high-quality pseudo labels (b). We further study true positive (TP) and false positive (FP) rates, as shown in Fig. E.2a in the Appendix. We show qualitative results in Fig. 3, including both the confidence healmap and the pseudo labels with and without the impact of S4MC.
4 Experiments
This section presents our experimental results. The setup for the different datasets and partition protocols is detailed in Section 4.1. Section 4.2 compares method against existing approaches and Section 4.3 provides the ablation study. Implementation details are given in Appendix C.
4.1 Setup
Datasets In our experiments, we use PASCAL VOC 12 (Everingham al., 2010), Cityscapes (Cordts al., 2016), and MS COCO (Lin et al., 2014) datasets.

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: Comparison between our method and prior art on the PASCAL VOC 12 val (1,464 original annotated
images out of 10,582 in total) under different partition protocols using ResNet-50 backbone. The caption describes the share of the training set used as labeled data.
Method
1 1/16 (92)
1/8 (183)
1/4 (366)
1/2 (732)
Full (1464)
Supervised Baseline
44.0
52.3
61.7
66.7
72.9
PseudoSeg (Zou et al., 2021)
54.89
61.88
64.85
70.42
71.00
PC (Zhong et al., 2021)
56.9
64.6
67.6
70.9
72.3
UniMatch (Yang et al., 2023)
71.9
72.5
76.0
77.4
78.7
UniMatch" + S4MC
72.62
72.83
76.44
77.83
79.41
Table 3: Comparison between our method and prior art the augmented PASCAL VOC 12 val dataset under different partitions, utilizing additional unlabeled data from Hariharau et al. (2011) (Lotal of 10,582 training images, 9,118 weakly annotated) and using ResNet-101 backbone. We included the number of labeled images in parentheses for each partition ratio. * denotes reproduced results using official implementation.
Method
1/16 (662)
1/8 (1323)
1/4 (2646)
1/2 (5291)
CutMix-Sog (French et al., 2020)
71.66
75.51
77.33
78.21
AEL (Hu al., 2021)
77.20
77.57
78.06
80.29
PS-MT (Liu et al., 2022b)
75.5
78.2
78.7
U (Wang et al., 2022)
77.21
79.01
79.3
80.50
PCR (Xu et al., 2022)
78.6
80.71
80.78
80.91
FixMatch (Yang et al., 2023)
74.35
76.33
76.87
77.46
UniMatch (Yang et al., 2023)
76.6
77.0
77.32
77.9
CutMix-Seg + S4MC
78.84
79.67
79.85
81.11
FixMatch + S4MC
75.19
76.56
77.11
78.07
UniMatch + S4MC
76.95
77.54
77.62
78.08
Table 4: Comparison between our method and prior art on the Cityscapes val dataset (total of 2,976 training images) under diffcrent partition protocols using ResNet-101 backbone. Labeled and unlabeled images are selected from the Cityscapes training dataset. For each partition protocol, the caption gives the share of the training used as labeled data and the number of labeled images. * denotes reproduced results using official implementation.
Method
1/16 (186)
1/8 (372)
1/4 (744)
1/2 (1488)
CutMix-Seg (French et al., 2020)
69.03
72.06
74.20
78.15
AEL (Hu et al., 2021)
74.45
75.55
77.48
79.01
UPPL (Wang et al., 2022)
70.30
74.37
76.47
79.05
PS-MT (Liu et al., 2022b)
76.89
77.6
79.09
PCR (Xu et al., 2022)
73.41
76.31
78.4
79.11
FixMatch* (Yang al., 2023)
74.17
76.2
77.14
78.43
UniMatch (Yang et al., 2023)
75.99
77.55
78.54
79.22
CutMix-Seg + S4MC
75.03
77.02
78.78
78.86
FixMatch + S4MC
75.2
77.61
79.04
79.74
UniMatch S4MC
77.0
77.78
79.52
79.76
PASCAL comprises 20 object classes (+ background). 2,913 annotated images are divided into training
and validation sets of 1,464 and 1,449 images, respectively. Zoph et al. (2020) shown that joint training of PASCAL with training images with augmented annotations (Hariharan et al., 2011) outperforms joint.
8

---

Published in Transactions on Machine Learning Research (06/2024)
training with COCO (Lin et al., 2014) ImageNet (Russakovsky et al., 2015). Based on this finding, we use extended PASCAL VOC 12 (Hariharan et al., 2011), which includes 9,118 augmented training images, wherein only a subset of pixels are labeled. Following prior art, we conducted two sets of experiments: in the first, we used only subset of the original training data as annotated, while in the second, "angmented" sotup, we also used the weakly annotated data and randomly sample images from both.
Cityscapes dataset includes urban scenes from 50 cities with 30 classes, of which only 19 are typically used for evaluation (Chen et al., 2018a;b).
MS COCO dataset is a challenging segmentation benchmark with 80 object classes (+ background). 123k images are split into 118k and 5k for training and validation.
Implementation details We implement S4MC with teacher student paradigm of consistency regularization, both with teacher averaging (Tarvainen & Valpola, 2017; French et al., 2020) and augmentation variation (Sohn et al., 2020a; Yang et al., 2023) frameworks. All variations use DeepLabv3+ (Chen et al., 2018b), while for feature extraction, we use ResNet-101 (He al., 2016) for PASCAL. VOC and Cityscapes, and Xception-65 (Chollet, 2016) for MS COCO. For the teacher averaging setup, the teacher parameters are updated via an exponential moving average (EMA) of the student parameters: 67 =0p (1 r))7,, where 0 I defines how close the teacher is to the student and denotes the training iteration. We used 0.99. In the augmentation variation approach, pscudo labels are generated through weak augmentations, and optimization is performed using strong augmentations. Additional details are provided in Appendix C.
Evaluation We compare S4MC with state-of-the-art methods and baselines under the standard partition protocols using 1/2, 1/4, 1/8, and 1/16 of the training set as labeled data. For the "classic" setting of the PASCAL experiment, we additionally use all the finely annotated images. We follow standard protocols and use mean Interscction over Union (mIoU) as our evaluation metric. We use the data split published by Wang et al. (2022) when available to ensure fair comparison. For the ablation studies, use PASCAL VOC 12 val with 1/4 partition.
4.2 Results
Results denoted by * reproduced from Yang et al. (2023) original implementation, resulting in minor difference in performance. vp denotes using UniMatch without feature perturbation branch, ie., only two instances of each image.
PASCAL VOC 12. Tables and compares our method with state-of-the-ard baselines on the PASCAL VOC 12 dataset using ResNet-50 and ResNet-101, respectively. Table 3 shows the comparison results PASCAL with additional unlabeled data from SBD (Hariharan et al., 2011) using ResNet-101. S4MC outperforms all compared methods in standard partition protocols using the PASCAL VOC 12 dataset only, while utilizing SBD, we observe comparable results to the state-of-the-art PCR (Xu et al., 2022). More significant improvement can be observed for partitions of extremely low annotated data, where other methods suffer from starvation due to poor teacher generalization. We observed that using supervised data only yield mIoU of 80.42, we conducted another experiment, where we used all available annotations from 10, 582 images, and besides, we used the 9, 118 weakly annotated images unlabeled. This experimental setting utilize the data the most, pushing S4MC to a mloU of 82.33. Qualitative results are shown in Fig. 3. Our refinement procedure aids in adding falsely filtered pseudo labels and removing erroneous ones.
Cityscapes. Table 4 presents the comparison with state-of-the-art methods on the Cityscapes val (Cordts et al., 2016) dataset under various partition protocols. S4MC outperforms the compared methods in most. partitions, and combined with the UniMatch scheme, S4MC outperforms compared approaches across all partitions.
MS COCO. Table 5 presents the comparison with stato-of-the-art methods on the MS COCO val (Lin et al., 2014) dataset. S4MC outperforms the compared data splits published in (Yang et al., 2023). In this experimental setting, the model small fraction of
methods in most regimes, using the
9

---

Published in Transactions on Machine Learning Research (06/2024)
Table 5: Comparison between our method and prior art on COCO (Lin et al., 2014) val (total of 118,336 training images) on different partition protocols using Xception-63 backbone. For each partition protocol, the caption gives the share of the training set used as labeled data and the number of labeled images. denotes reproduced results using official implementation.
Method
1/512 (232)
1/256 (463)
1/128 (925)
1/64
(1849)
1/32 (3697)
Supervised Baseline
22.9
28.0
33.6
37.8
42.2
PseudoSeg (Zou et al., 2021)
29.8
37.1
39.1
41.8
43.6
PC2Seg (Zhong et al., 2021)
29.9
37.5
40.1
43.7
46.1
UniMatch* (Yang et al., 2023)
31.9
38.9
43.86
47.8
49.8
UniMatch S4MC
32.9
40.4
43.78
47.98
50.58
Table 6:
The effect of cuu the initial proportion of confidence pixels for the Pascal VOC 12 with 1/4 labeled data and ResNet-101 backbone.
00
20%
30%
40%
50%
60%
mloU
78.13
77.53
79.1
78.24
77.99
the data, which could be hard to generalize over all classes. Yet, the mutual information using neighboring predictions seems to compensate somewhat more supervision signals propagate from the unlabeled data.
Contextual information at inference. Given that margin refinement scheme operates through prediction adjustments, explored whether it could be employed inference time to enhance performance. The results reveal no improvement, underlines that the performance advantage of S4MC primarily derives from the adjusted margin, as the most confident class is rarely swapped.
4.3 Ablation Study
Neighborhood size and neighbor selection criterion. Our prediction refinement scheme employs event-union probability with neighboring pixels. We examine varying neighborhood sizes (N = 3,5, 7), number of neighbors (k=11 = 1, 2), and selection criteria for neighbors. We compare the following methods for choosing the neighboring predictions: (a) Random selection, (b) Cosine similarity, (c) Max probability, and (d) Minimal probability from a complete neighborhood. Note that for the cosine similarity, we choose single most similar prediction vector for all classes, thus mostly enhancing confidence and not overturning predictions. We also compare with N = 1 neighborhood, corresponding to not using $4MC. As seen from Table 7, N = neighborhood with one neighboring pixel of the highest class probability proved most elficient in our experiments. Aggregating the probability of a randomly selected neighbor has negligible influence on model performance. Using multiple neighbors demonstrates improved performance as the neighborhood expands. Notably, the inclusion of minimal class probability pixels adversely affects model performance, primarily altributed to the contribution of neighbors that exhibit. high certainly in belonging to distinct. class.
We also examine the contribution of the proposed pscudo label refinement (PLR) and DPA. Results in Table 8 show that the PLR improves the mask mIoU by 1.09%, while DPA alone harms the performance. This indicates that PLR helps semi-supervised learning mainly because it enforces more spatial dependence on the pseudo labels.
Threshold parameter tuning We utilize a dynamic threshold that depends on an initial value, ag. Table 6, we examine the effect of different initial values to establish this threshold. A smaller ag propagates too many errors, leading significant confirmation bias. In contrast, a larger ag would mask most of the data, rendering the semi-supervised learning process lengthy and inefficient.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Table 7:
The effect of neighborhood size and neighbor selection criterion on the Pascal VOC 12 with 1/4 labeled data and ResNet-101 backbone. We denote the number of neighbors one neighbor at random, the one with the highest cosine similarity to the pixel embedding, max probable neighbor and min probable neighbor. The idea of similar neighboring pixel is explained in the paper, while comparing to minimum probable neighbor try to see if the spatial information contradict the prediction, reducing the likelihood to assign pseudo label to the predicted class.
We compared choosing
Neighborhood size N
Selection criterion
1x1 1
3 x
x 5
Random neighbor (k=1)
77.03
76.85
76.87
Cosine similarity (k=1)
78.02
78.05
77.99
Max-prob (k=1)
77.7
79.09
78.23
77.76
Max-prob 2k=2)
77.77
77.82
78.03
Min-prob (k=1)
75.62
75.11
73.95
Table 8: Ablation study on the different components of S4MC on lop of UniMatch for the augmeented Pascal VOC 12 with 1/2 labeled data and ResNet-101 backbone. PLR is the pseudo label refinement module and DPA is dynamic partition adjustment.
PLR
DPA
mIoU
79.51
79.94
78.25
80.13
Table Evaluation of Boundary IoU (Cheng et al., 2021) comparing models trained with S44M and with FixMatch using 183 (1/1024) annotated images on COCO, both uses Xception-65 backbone as in Table 5.
FixMatch
Fi4Match+
31.1
29.9
Mask boundarics Table domonstrates the limitation of our mothod in terms of boundary IoU (Cheng et al., 2021). Contrary the improvement S4MC provides to loU, the boundary IoU is reduced. That aligns with the qualitative results, as our model predictions masks are smoother in regions far from the boundaries and less confident around the boundaries.
5 Conclusion
In this paper, we introduce S4MC, novel approach for incorporating spatial contextual information in semi-supervised segmentation. This strategy refines confidence levels and enables to leverage more unlabeled data. S4MC outperforms existing approaches and achieves state-of-the-art results multiple popular benchmarks under various data partition protocols, such as MS COCO, Cityscapes, and Pascal VOC 12. Despite its effectiveness in lowering the annotation requirement, there are several limitations to using S4MC. First, its reliance on event-union relaxation is applicable only in cases involving spatial coherency. As a result, using our framework for other dense prediction tasks would require an examination of this relaxation's 's applicability. Furthermore, our method uses a sixed-shape neighborhood without considering the object's structure. It would be interesting to investigate the of segmented regions to define new neighborhoods; this is a future direction we plan to explore.
11

---

Published in Transactions on Machine Learning Research (06/2024)
References
Luca Bartolomei, Lucas Teixcira, and Margarita Chli. Perception-aware path planning for UAVs using semantic segmentation. In JEEE/RS) International Conference on Intelligent Robols and Systems (IROS), pp. 5815, 2020. doi: N0.1109/IROS4A 28000055578222200884144444 (cited on p.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A. Raffel. MixMatch: a holistic approach to semi-supervised Iearning. In H. Wallach, H. Larochelle, A. Beygelzimer, d' d'Alché-Bue, E. Fox, and R. Garnelt (eds.), Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL attps://prooceediga nwwrips. cta.lps.ccppppp/221111 / Bcd1880049a66144/77eeeebbabaaaa -bbttbbrrrcaaa..a.a.c..... .tml.. (cited on pp- 1, 2, and 3)
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin A. Raffel. ReMixMatch: semi-supervised learning with distribution matching and augmentation anchoring. In International Conference on Learning Representations, 2020. URL petps://openreee wwwparevveeeeeeeeee/eee net/forum?ic HkkkaR4KPB. (cited on p. 3)
Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale, 2022. (cited on p. 2)
Liang-Chieh Chieh Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 40(4):834 848, 2018a. doi: 2692299184.. URL https://arxiv.or arg/abs//112..... mates//111...0000 7062. (cited on pp. 1, 3, and 9)
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In European Conference on Computer Vision (ECCV), September 2018b. URL attps://openaccess: thecvf. com/ content, Beantetn__Ccv____ _ECCV_ _ECCV_20CC _el_22201hhhll Attouustecvc _Eccs Bttpssseeeeeee_eeeeeeee_ _012__apee__ peaapr.htmll..... htmll. (cited on p. 9)
Chang-Chiehh_ Sewehhheehhheee__eeeee Cheenehhoeeerreeeeeeeeeeeeeeeeee Encoddr-Decoodrrr htt__tittthttouuuseeeeeeees
Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), thecvf. cwm/ comm/onttent/////2//////ttt hhtee//hooo//tmmm/hheenn
pp. 2613-2622, June 2021. URL https:///penaee Semi-Supervised_ _ html. (cited on p. 3)
Segmentation With Cross_ Pseudo Supervision CVPR 2021 20001_ppp0 _apper.
Bowen Cheng, Ross Girshick, Piotr Dollar, Alexander C. Berg, and Alexander Kirillov. Bound- ary IoU: Improving object-centric image segmentation evaluation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pollern Recognition (CVPR), pp- 15334-15344 15342, June 2021. URL /ttps://openaccess: thecvf. com http:/oonne/////22/1/////// (11P22021/htm//// Saall//hhng__aa htt///hhennnnnnrrrrreeeeee Bowndaounnd____e __U httpl///ttmm//eeeeeeeeeee
Saproovig__bject-CCo Cecteeetttrr_itnneeeeeee_ee Iww.Inaaaaennneeeeeeeee Segmentation http:riiiin_aalaationnnooooo Cet.eaattiiioiaae__aeeee _aaperpp..... html. (cited p. 11)
François Chollet. Xception: Deep learning with depthwise separable convolutions. 2017 TEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1800 1807, 2016. (cited on p. 9)
Marius Cordts, Mohamod Omran, Scbastian Ramos, Timo Rchfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. Tn IEFF Conference on Computer Vision and Paltern Recognition (CVPR), June 2016. URL https: //tt:://www... werti:.v-ffoumm so-foundation. wwwiiionnnrg//eeeeeeeeeee wwtp://paacceee///ee/////////e httpe//coteeetttteeeeeee catpttcvvrp___11111ee wwte////////oooosseeooooo The_ Cityscapes_Dataace __tpaaet_taaaaaaae_e____a Sar___vV_______ __tp__papr..___ html. (cited on pp. 2, 7, and 9)
Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNel: marrying convolution and attention for all data sizes. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Neural Information Processing Systems, volume 34, pp. 3965 3977. Curran Associates, Inc., 2021. URL https://proceeding 20666622bb2246ccc222eee33332 Abstract html. (cited on p. 1)
J. Wortman Vaughan (ods.), Advances
cuuripss.co/paper///00 20/h/hash//////
12

---

Published in Transactions on Machine Learning Research (06/2024)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Usækoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL attps://openretee not//forummddm1/ccaannyy (cited on p.
3)
Dave G. Elliman and Ian T. Lancaster. A review of sogmentation and contextual analysis techniques for lext, recognition. Pallern Recognilion, 23(3):337 1990. ISSN 0031-3203. doi: hutps/ /oi. (00110066000001-2203999999000000 003132039090021C. (cited on p. 3)
URL http://wwww.scieneereeeeete. waam///cienee/raaiin
Mark Everingham, Luc Van Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303-33) -38, June 2010. doi: 10. 00-10077/11223--00-077--- URL https://doi org/10.1007/ 00000100000/111333-000077------0 (ciled on pp. 2 and 7)
Geolfrey French, Samuli Laine, Timo Aila, Michal Mackiewicz, and Graham D. Finlayson. Semi-supervised semantic segmentation needs strong, varied perturbations. In British Machine Vision Conference. BMVA Press, 2020. URL attps://wke wwwc:/20wm22000m conference. coom//sssetsspaper//ooe pdf. (cited on pp. 3, 7, 8, and 9)
bwwc2020-conferencceee
Ross Girshick. Fast R-CNN. In IEEE International Conference on Computer Vision (ICCV) December 2015. URL https::/openaccesss thecvf. 2015_paper html. (cited on p. 1)
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In International Conference on Computer Vision, pp. 991-998, 2011. doi: 01110//TCCV.2011.61222
(cited on pp. 8 and 9)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. URL https:/ /ttp:///wwwww/ cv- petprdatiiooong/pppeeeeeee wttg///pnaccess/cottetttocmmmo httpaaeccccpp..///////eeeeeeeeee Residual Lea___ne_rnnnnnn_.ca. (0VPPR20066__000
html. (cited on p. 9)
Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp- 22106 22118. Curran Associates, Inc., 2021. URL https:/ //tp://///ocengss wwweeneeeeiiiooeeeeeeee /cppaaee Papera.--ppppp..p.... (cited on pp. and 8)
Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy Ren, and Rynson Lau. Dual student: Breaking the limits of the teacher in semi-supervised learning. In 2019 IEEE/CV International Conference on Computer Vision (ICCV). TEEE, October 2019. doi: 10. http:/iccv.229..00000 2019.00683. URL http://dx.dof: doi. 2019. 00683. (cited on p. 2)
www.ddii..ooooooooo www.oo.../........ 1109/ICVV
Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson W. H. Lau. Guided collaborative training for pixel-wise semi-supervised learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), European Conference on Computer Vision, pp. 429 445, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58601-0 URL https: http://wwwwweceee ecva. www.://ww..ee/ppppss//ee/o//o/ee nwt//epers/eccoo/o weeeeecccvevpppeeeeeeee 2220/papers_BC00 htm1/1932_ ECCV_2020 20220__pprr___ php. (cited on p. 3)
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tele Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. Xrxiv preprint (cXx::2304.02623 2023. (cited on 1)
Alex Krizhevsky, Ilya. Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural networks. In F. Percira, C.J. Burges, L. Bottou, and K.Q. Weinberger (ods.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https: : http///aperr.///// wwp.rrsniippp.. .niss. Abstract...tt...e... html. (cited on p. 1)
13

---

Published in Transactions on Machine Learning Research (06/2024)
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Represenlations, 2016. URL https://openrevien http:///ppaaapreeeeeeeee//eroaap
(cited on p. 3)
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop: Challenges in Representation Learning (WREPL), July 2013. URL http:///deplearning http://pconttttttpploo///////DL http:///200//////uuu//oo sltl__lalll__aeee and 3)
(cited on pp. 2
Feng Li, Hao Zhang, Huaizhe xu, Shilong Liu, Lei Zhang, Lionel M. Ni, and Heung- Yeung Shum. Mask DINO: lowards a unified transformer-based framework for object detection and segmentation. arXiv preprinl, wwg.obs/ halpss//22000000 02777. (cited on p. 1)
June 2022. URL https: http://///.wxw..//a////////sss//
Tsung- Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva. Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), European Conference on Computer Vision, pp. 740-755, Cham, 2014. Springer International Publishing, ISBN 978-3-319- 178-3--19-10602-1. URL https://link: wewink..pringe..... com/ 1hapter/10.1007/9775 3-10------11---- 1000--1002-11111 1_48. (cited pp. 3, 7, 9, and 10)
Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew J. Davison. Bootstrapping semantic segmentation with regional contrast. In International Conference on Learning Representations, 2022a. URL https: // wtwp://wwww..eettttoorrodrddeeee
(cited p.
Yen-Cheng Liu, Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. In International Conference (cited on p. 2)
Learning Representations, 2021. URL https://openreww http:://pppproooeeeeeooooo//a/aa
Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, and Gustavo Carneiro.
Perturbed and strict mean teachers for semi-supervised semantic segmentation. In JEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 4258 4267, June 2022b. URL https://openaccoess thecvf. comevf.comcooll Teachers_for for Semi Supervisee Semantic S wetpp_cce_____________2__ ptppp..tml...........p.......... html. (cited on pp. 3, 7, and 8)
http///otttett/////////aaaaaaaaa http://///aaaiiiaaeeeuurreeedee _ttrrrrtedddd_Sttiiicteeeeee Surrdtttictt.t __aa__Me_____
Andres Milioto, Philipp Lottes, and Cyrill Stachuiss. Real-time semantic segmentation of crop and weed for precision agriculture robots leveraging background knowledge in CNNs. In IEEE International Conference Robotics and Automation (ICRA), pp. bttps://arxiv.. porg/abs////e 0071//1999.06...... (cited on p. 2)
2018. doi: 10.110//TCRAA.00..66600
URL
Takeru Miyato, Shin-Ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):1979 1993, 2018. doi: 10.1109, 10.1109TTPAML, 2AAAM1.2018.288888 URL https://ieeexplorss ieee. /ttpaaaact//cmmaa/////TTT
(cited on p. 3)
Alexey Nekrasov, Jonas Schult, Or Litany, Bastian Leibe, and Francis Engelmann. Mix3d: Out-of-context data augmentation for 3d scenes. 3DV 2021, 2021. (cited on p. 3)
Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross- consistency training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. URL https: https:// //ppaacces//cccsss.// thecvf. compc/.fccomcccoooooo/////ccc content, CVPR 2020/htm1/Qual1 Semi-Supervised, .tml. (cited on p. 3)
Semantic_ Ser___eeeeetttoonniiiiie With Saapha-CosssCooo Consistency hetpisstteoy___raiinnn_oooo hwtp:/ainnnnnPR_______aaaapaaa
Antti Rasmuu, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.),
Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https: Abstract html. (cited on
/ httpp///.ttps..ccpppr///11.15 l/
p. 1)
14

---

Published in Transactions on Machine Learning Research (06/2024)
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S. Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-superriseed learning. In International Conference on Learning Representations, 2021. URL https://openrevier wtt.ppperrrrreeeeeeoooorrmieeiie (cited on p. 2)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211 (111)22111255) 225, 2015. doi:
(cited on p. 9)
Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden Markov models for information extraction. In Frank Hoffmann, David J. Hand, Niall Adams, Douglas Fisher, and Gabriela Guimaraes (eds.), Advances in Intelligent Data Analysis, pp. 309-318, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg. TSBN 978-3-540-44816-7. URL https://link springer. wwt.r.ccccccci /ttp://ttee//1110///-/////- (cited on pp. and 22)
Gyungin Shin, Weidi Xie, and Samuel Albanie. All you need are a few pixels: Semantic segmentation with PixelPick. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 1687-1697, October 2021. URL https: //tppace///encccss.s.. thecvf wtt.coomcccc//c http///etee/////2///////// ILDAV/htmI/Shin All_ You Need Are_a_eee_ a_ae________ Few_Pixels, Semantic Serrrattiiggmmeetttiionnooooooo With ICCVW_20V2 paper.html (cited on pp. 5 and 22)
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A. Raffel, Fkknn Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: simplifying semi-supervised learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neurol Information Processing Systems, volume 33, pp. 596- 608. Curran Associates, Inc., 2020a. URL https://proceeding wewprddinnnnnniiisn...e.m wwt..ess..cppppeee/e//.... cccppaper/220///ase (654deeeaabb.6666dd44493977777 Abstrrrattabttttt.........a..... html. (cited on pp. 1, 2, 3, 4, 5, 9, 19, 22, and 23)
Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister. A simple semi-supervised learning framework for object delection. arXiv preprint ar Xix: 2ao:200550477,, 2020b. URL https://arxiv httpe:///axxiv.og//bbbba arg/abs//000 04757.. (cited on p. 2)
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: hatpa:////occeeinngee inttreeddinnnnuuuipps. neuripa. cctpwppr//011////////ppcp hatrr//sh//hhh////////// html. (cited on pp. 2, 3, and 9)
Godfried T. Toussaint. The use of context in pattern recognition. Pattern Recognition, 10(3):189- 204, 1978. ISSN 0031-3203. doi: sciencedirect. com/sciencel http////eao/eaaaeea///////////// /arraacleepp///0112220000222222/ Society Conference. (cited on p. 3)
URL https: //www. The Procoedings of the IEEE Computer
Adam Van Etten, Dave Lindenbaum, and Todd M. Bacastow. SpaceNet: a remote sensing dataset and challenge series. arXiv preprint, June 2018. URL htttps://arsee httppp::///axvvooo///b////////// wwwg//as//////ooo ht/p///0011070002...... (cited on 1)
Ashish Vaswani, Noan Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpss://penacccess. thectf cthecvf.com/conteeeee CVPR 2020/htmee/aeeeeee /Wang. ECA-Net, Mil-NNee_Effiiietttt. Channel CoaueelAttentionn forr Seee_oeere___eeeeeeeeeeeeeeeeeee D Networks Cew___222___________22 _ap20rrppprr.__ .tml. (cited on p. 3)
He Wang, Yezhen Cong, Or Litany, Yue Gao, and Lconidas Guibas. 3DIoUMatch: leverag- ing IoU prediction for semi-supervised 3D object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 14615-14624, June 2021.
15

---

IoU_ Published in Transactions on Machine Learning Research (06/2024)
URL httpe::/openaaceess thecvf. cwm/ httpa//aaaaaaao///////////////aa Prediction_fors for SarSStefffr-__iar--_e Suuriiisperiiiede 3D_ Debject_Detectingg Lttp:iio____PP_221_____1_1 2021_ _appep_pppr____ html. (cited on 2)
Qilong Wang, Banggu Wu, Penglei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. FCA- Net: Efficient channel attention for doep convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. URL https://openaccees: thecvf wwwoo///otttttneeeeeeeeeeeooe Secpnnt_______e 2020/htm1 Baamll/Waa______ ECA-Net jest__fffiieet__ AaammelAttention, for corapoooooolltinlll..... Neural Wetworks_CVRn CWPR_2020__________2_222 _020_paper _ppp_ppppr... html. (cited on p. 3)
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. FreeMatch: self-ddaptive thresholding for semi-supervised learning. In International Conference on Learning Representations, 2023. URL http:///pppeeeeeeeeeeeeeoorrraap aeed/forrurid-PDUPPXJIILI (cited on pp. 5 and 22)
Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fci, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo labels. IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2022. URL httpps://openaccess. thecvf. http://eeee.eeeeeee.ecoco/etttee ccm///nnteeett///////////aaalal Segmentation
Sarraang_Smmmiee Semi Hemi-Supervisedt _erandtimiiicc
Unreliable Laapriaaeeeeaeu-aaaaballllllel Catpeddo-aable__cPRR______ __pp__cV2_____________e html. (cited on pp. 2, 3, 5, 7,
8, and 9)
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc V. Le. Unsupervised data aug- mentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Н. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6256 6268. Curran Associates, Inc., 2020. URL attps://proceedings. www..nneriiscooo http://....rrrppppppaa/////////a httppp/c.cc/aappe////////a// 24410881a112261117778888888 Abstrrrttt-bbtttt...aaaaaa tatstract..tml.... html. (cited on p. 3)
Haiming Xu, Lingqiao Liu, Qiuchen Bian, and Zhen Yang. Semi-supervised semantic segmentation with prototype-based consistency regularization. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 26007 26020. Curran Associates, Tnc., 2022. URL https::/ppoceeddngs.eeer www.eeeww.....apappepee lamperrfill////eeeeeepeeeeereerr sallee/ppper/202/2 htmm.. (cited on pp. 3, 7, 8, and 9)
Fan Yang, Kai Wu, Shuyi Zhang, Guannan Jiang, Yong Liu, Feng Zheng, Wei Zhang, Chengjie Wang, and Long Zeng- Class-aware contrastive semi-supervised learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14421-14430,
June 2022a. URL https::/openaccess. thecvf. hatm///oneent/////22////// Baaptttmm//annn_aa_e Alase-Awrre html. (cited p. 1)
Sontrastive_Semis Seeee-Seem-miueeee Stweeeseeeiieerrieeeeeeeeeeee Learreeng_aannneeeeeee SPPPP_2022__P_ 2022_paper
Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. ST++: make self-training In IEEE/CVF Conference on
work better for semi-supervised semantic segmentation. Computer Vision and Pattern Recognition (CVPR), pp. 4268-48 188-4277, June 2022b. URL
https://openaccess thecvf. coom///onneee//////22////t/t /Yang_anggSSSS _T__S____ Si__STSMMS Mar_Maee_aae Maw.aaaaeeeeeraaiiiiieae Self-Traaning_Wonl Work_ Better for and 7)
fer__eri----_ Semi-
CVPR 2022
Bal__22222_______ paper html. (cited pp. 3
Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting weak-to- strong consistency semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7236 7246, June 2023. URL https::/ thecvf. cont//cotteet////P22///ttaa htw.://aaa//aaaneeeiiiitiiinggn www.siiiiiiiin...e wettin__eaattttttttooe Consistency BeaiiccyynnSSSiiiie Semi-Smmmimmme Supervise,, htterivieeddeemaiiieeeeeee Swwaatticmnneeeeeeeeeeeeeeee.eee Semantic__egnentatioon CVPR 2012__pperr... html. (cited on pp. 3, 7, 8, 9, 10, 19, and 23)
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. CutMix: regularization strategy to train strong classifiers with localizable features. Tn TEFE/CVF International Conference Computer Vision (ICCV), October 2019. URL
16

---

Published in Transactions on Machine Learning Research (06/2024)
httrss://openaccess.
Strategy_ to_ cotte///ttttttccoootttotttttt httpa_ooooooo/oal/////oo Classifiers With Localizable eeeeeeereesee_eee ceaetures_IC______ 2019_paperr_ ptaper.hmll. htmll.
Ltt.oo//.///ttaa///tttttttttteat Sett___ttiix___ RetMMix_Rggllarizatiio
(cited on 18)
Sergey Zagoruyko, Adam Lerer, Tsung-Yr Yi Lin, Pedro O. Pinheiro, Sam Gross, Soumith Chintala, and Piotr Dollár. A MultiPath network for object detection. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the Brilish Muchine Vision Conference (BMVC), pp. 15.12. BMVA Press, September 2016. ISBN 1-901725-59-6 doi: 15. URL https: doi otg/i..rgg..... wtgg/0..044//..30000 15. (cited on p. 1)
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp- 18408- 18419. Curran Associates, Inc., 2021. URL 99593c115449ee 19066699119555 Abbtssttttt....-tttt.t.t........ html. (cited on pp. 2, 3, and 22)
neurips. cc/papper/222//aae
Na. Zhao, Tat-Seng Chua, and Gim Hee Lee. SESS: self-ensembling semi-supervised 3D object detection. In Proceedings of the JEFE/CVE Conference on Computer Vision and Pollern Recognition (CVPR), June 2020. URL Self-Ensembling. Semi Sumnii-upevviedd. Baaeeedd_____e ab__bjecct SeeettDDeetttoccFF__ee
thecvf. com/content httm//nteettcPR.....C. CVVP__2000htmm// _esS___S_______
(cited on p. 2)
Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive- consistent semi-supervised semantic segmentation. In IEEE/CVP International Conference on Compuler Vision (ICCV), pp. 2273-7282, October 2021. URL attps:// https://oppnaccesss thecvf com/ content/ IcCV2021/htm1/2hong, Pixel_ Pixel_Contraaaa Seetraative-Cossistten Semi-Smmiiiiise Sumeiiiuppeviiedd Sewaettccimmimieeeeeeaaee Segrreeeeteaaaattieeeeeeettttto ICCV (CCV_2021_paper html. (cited on pp. 3, 8, and 10)
Zhuofan Zong, Guanglu Song, and Yu Liu. DETRs with collaborative hybrid assignments training, arXiv preprint, November 2022. URL https: wwweiii.orr//eeo aaa///bb/21..... 12860. (cited on p. 1)
Barret Zoph, Golnaz Ghiasi, Tsung-Y. Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1833-3845. Curran Associates, Inc., 2020. URL https: https://prrceeiings .ewrrinnurpppp. /c/pap/e//200///iiee 77e9661e033a/3aaaaBeee adea .aarr.pd.. pdf. (cited on p. 8)
Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. PseudoSeg: designing pseudo labels for semantic segmentation. In International Conference on Learning Representations, 2021. URL htRt::/t:///ppnnee http:///pppreeeeeee/e/e//e httpparrrviee..ee//frurrfff TV099..VRRR.. (cited on pp. 3, 8, and 10)
17

---

Published in Transactions on Machine Learning Research (06/2024)
Algorithm 1: Pseudocode: Pseudo label refinement of SIMC, PyTorch-like style.
X: predict prob of unlabeled data H W
k: number of neigbors
#create neighborhood tensor neighorhood-11
X. unsqueezee
torch. nn. fww.iiiiinal.......mmmmmmmmm.... (X, (1, 1, 1, 1, 0, 0, 0, 0))
for i,j ((None,-2) (1,,,),,1))1)1111 (1,-1), (2)),(2,N0)))) (2,None)]
x,1 in [(None,-2) (1,-1), -1), (2, :
if i==k and i==1:
continue neighborhood.apee append ;,i k:1])
neighborhood = torch.stack (neighborhood)
#pick k neighbors for union event
ktop_ cuop_neighbors neigbor idx=torch htt.:rtttttt.....oooaaaaaaaooooo k=k, axis=0)
for nbr in ktop_neight neighbors: torch.expl. exp((-1/2) neigbor_idx)
I = beta+nbr (Xenbr<beta)
A Visual Results
We present in Figs. A.1 and A.2 an extension of Fig. 3, showing more instances from the unlabeled data and the corresponding pseudo labeled with the baseline model and S4MC. In Fig. A.2 we can see that our method can eliminated undesired entities, while sometimes it also eliminate good predictions, it the process of psoudo-labeling.
Our method can achieve more accurate predictions during the inference phase without refinements. This results more seamless and continuous predictions, which accurately depict objects spatial configuration.
B Computational Cost
Let us denote the image size by H x W and the number of classes by C.
First, the predicted map dimension H W C is stacked with the padded-shifee shiffe- versions, creating a tensor of shape [n,,WW,C), K top neighbors are picked via top-k operation and calculate the union event presented in Eq. (9). (The pseudo label refinement pytorch-like pseudo-code can be obtained in Algorithm for N 4 and & max neighbors.)
The overall space (memory) complexity of the calculation is O(n x H x W x C),, which is negligible considering all parameters and gradients of the model. Time complexity adds thrce tensor operations (stack, topk, and multiplication) over the H W x C tensor, where the multiplication operates k times, which means O(k H W C.). This again negligible for any reasonable number of classes compared tens of convolutional layers with hundreds of channels.
To verify that, we conducted training time analysis comparing FixMatch and FixMatch + S4MC over PASCAL with 366 labeled examples, using distributed training with 8 Nvidia RTX 3090 GPUs. FixMatch average epoch takes 28:15 minutes, and FixMatch + S4MC average epoch takes 28:18 minutes, an increase of about 0.2% in runtime.
C Implementation Details
All experiments were conducted for 80 training epochs with the stochastic ic gradient descent (SGD) optimizer power
with a momentum of 0.9 and learning rate policy Ir Irbase (1 total iter
Iter
For the teacher averaging consistency, we apply resize, crop, horizontal flip, GaussianBlur, and with a probability of 0.5, use Cutmix (Yun et al., 2019) the unlabeled data.
18

---

Published in Transactions on Machine Learning Research (06/2024)
93.16
93.99
98.85
97.82
97.82
98.85
91.97
92.81
94.17
94.52
94.54
93.67
93.67
Figure A.1: Example of refined pseudo labels, structure of the figure as Fig, 3 and the numbers under the predictions show the pixel-wise accuracy of the prediction map.
For the augmentation variation consistency (Sohn et al., 2020a: Yang et al., 2023), we apply resize, crop, and horizontal flip for weak and strong augmentations as well ColorJitter, RandomGrayscale,, and Cutmix for strong augmentations.
19

---

Published in Transactions on Machine Learning Research (06/2024)
Figure A.2: Qualative results of our method comparison to UniMatch baseline over COCO with 1/32 of the labelod examples. The segmentation map Left right: Ground Truth, UniMatch prediction, S4MC Prediction
For PASCAL VOC 12 Iraase 0.001 and the decoder only Iraase 0.01, the weight decay is set to 0.0001 and all images are cropped to 513 513 and B, = B,, = 3.
For Cityscapes, all parameters use Iraase 0.01, and the weight decay set 0.0005. The learning rate decay parameter is set power 0.9. Due to memory constraints, all images are cropped to 769 x 769 and B: = B, 2. All experiments conducted machine with 8 Nvidia RTX A5000 GPUs.
20

---

Published in Transactions on Machine Learning Research (06/2024)
Image
Epoch 10
Epoch 30
Epoch 50
Epoch 70
Figure E.1: Qualitative evolution of the pseudo labeling process S4MC. The figure shows the progress over time the pixels that assign as pseudo labels w.r.t time.
Limitations and Potential Negative Social Impacts
Limitations. The constraint imposed by the spatial coherence assumption also restricts the applicability of this work to dense prediction tasks. Improving pseudo labels' quality for overarching tasks such classification might necessitate reliance on data distribution and the exploitation of inter-sample relationships. We are currently exploring this avenue of research.
Societal impact. Similar to most semi-supervised models, we utilize small subset of annotated data, which can potentially introduce biases from the data into the model. Further, our PLR module assumes spatial coherence. While that holds for natural images, it may yield adverse effects in other domains, such as medical imaging. It is important to consider these potential impacts before choosing to use our proposed method.
E
Pseudo Labels Quality Analysis
The quality improvement and the quantity increase of pseudo labels are shown in Fig. 4. Further analysis of the quality improvement of our method demonstrated in Fig. E.2a by separating the true positive and false positive, and Fig. E.1 shows qualitative evolution of the pscudo labeling process.
Within the initial phase of the learning process, the euhancement in the quality of pseudo labels can be primarily attributed to the advancement in true positive labels. In method, the refinement not only facilitates the inclusion of larger number of pixels surpassing the threshold but also ensures that significant majority of these pixels are high quality.
As the learning process progresses, most improvements are obtained from a decrease in false positives pseudo labols. This analysis shows that our method effectively minimizes the occurrence of incorrect pseudo labeled, particularly when the threshold is set a lower value. In other words, our approach reduces confirmation bias from decaying the threshold the learning process progresses.
21

---

Published in Transactions on Machine Learning Research (06/2024)
0.10
positive enlargement False positive reduction
Labels
20
Excluded Pseudo Labels
0.08
% Pseudo labels
% Pseudo-label
0.06
15
0.04
10
0.02
0.00
0
20
40
60
80
0
20
40
60
80
Epoch
Fnoch
(a) Quality of pseudo labels from Fig. 4 separated True posilive and False positive analysis. True posilice explain the major part of improvement at early stage, while reducing false positive explain the enhancement later on.
(b) Added and excluded pseudo labels. The pixel-
wise pseudo labels S4MC added and excluded over time (L.. the sum the graphs total pixels that change
because of S4MC).
Figure F.2: Analysis of the results of DeepLab V3+ on PASCAL VOC 12
In Fig, E.1 we see that at late stages of the training process, the bounderies of objects are the most ambignate and thus the hardest to assign with pscudo labels with S4MC.
F
Weak-Strong Consistency
We need to redefine the supervision branch to adjust the method to augmentation level consistency framework (Sohn et al., 2020a; Zhang al., 2021; Wang el al., 2023). Recall that within the teacher averaging framework, we denote fo, (x.) and fo, (x;) as the predictions made by the student and teacher models for input x,, where the teacher serves as the source for generating bonfidence-based pseudo labels. In the context of image-level consistency, both branches differ by augmented versions x,", x and share identical weights fo. Here, x." and x represent the weak and strong augmented renditions of the input x,, respectively. Following the framework above, the branch associated with weak augmentation generates the pseudo labels.
F.1 Confidence Function Alternatives
In this paper, we introduce a confidence function to determine pscudo label propagation. We introduced Kmargin and mentioned other alternatives have been examined.
Here, we define several options for the confidence function.
The simplest option is look the probability of the dominant class,
Kmax (I),.) = max p)
(F.1)
which is commonly used to generate pscudo labols.
The second alternative is negative entropy, defined as
EPolohA log (pg(p(())).)).
(F.2)
cEC
Note that this is indoed confidence function since high entropy corresponds high uncortainty, and low entropy corresponds to high confidence.
The third option is for us to define the margin function (Scheffer et al., 2001; Shin et al., 2021) the difference between the first and second maximal values of the probability vector and also described in the main paper:
Eiargin (x) = max (pa(
max2. (1.(5,)))
(F.3)
22

---

Published in Transactions on Machine Learning Research (06/2024)
Table F.1: Ablation study on the confidence function K, over Pascal VOC with partition protocols using ResNet-101 backbone.
Function
1/4 (366)
1/2 (732)
Full (1464)
Kmax
74.29
76.16
79.49
Kent
75.18
77.55
79.89
Emargin
75.41
77.73
80.58
1.0
0.350
Pixal arrangemen
Agreement
0.325
0.300
0.4
0.275
Fixmatch
0,250
Unimatch
SAMC
0.0
10
20
30
40
50
60
70
80
Fixmatch
DusPerb
Unimatch
S4MC
Epoch
(a) The spatial agreement as define in in 9 compared between different. variations of Unimatch and S4MC.
(b) The spatial agreement, compared between different variations of (Yang et al., 2023) and S4MC over time.
Figure F.1: Spatial agreement analysis off diffrent methods on PASCAL VOC 12 using ResNet-101 backbone.
where max2 denotes the vector's second maximum value. All alternatives are compared in Table F.1.
F.2 Decomposition and Analysis of Unimatch
Unimatch (Yang et al., 2023) investigating the consistency and suggest using FixMatch (Sohn et al., 2020a) and a strong baseline for semi-supervised semantic segmentation. Moreover, they provide analysis that shows that combining three students for each supervision signal (one feature level augmentation, Channel Dropout, denoted by CD, and two strong augmentations, denoted by S1 and S2) can enhance performance further more, since each student branch learn slightly different features. Fusing Unimatch and our method did not provide significant improvements, and examined the contribution of different components of Unimatch. We measured the pixel agreement as described in Eq. (9) and showed that the feature perturbation branch has the same effect on pixel agreement as S4MC. Fig. F.1 present the distribution of agreement using FixMatch (S1), DusPerb (S1,S2), Unimatch (S1, S2, CD) and S4MC (S1, S2).
G Bounding the Joint Probability
In this paper, we had the union event estimation with the independence assumption, defined as Pl sl =o po(a..)) (o(
(G.1)
In addition to the independence approximation, it is possible to estimate the unconditional expectation of two neighboring pixels belonging to the same class based on labeled data:
( ))
Σ Σ E
1LUi vi.m)
(G.2)
Mi H. IN
RENI x. EmmEN
To avoid overestimating that could lead to overconfidence, we set
pe (m = max fmmm) h.m.mm.p22.
(G.3)
23

---

Published in Transactions on Machine Learning Research (06/2024)
0.80
0.75
0.70
moo
0.65
0.60
unimatch
s4mc
0
20
40 Epoch
60
80
Figure H.1: Training curve of the mloU of S4MC and Unimatch using Resnet-101 on PASCAL VOC 12 with 366 annotated examples
That upper bound of joint probability ensures that the independence assumption does not underestimate the joint probability, preventing overestimating the union event probability. Using Eq. (G.3) increase the mloU by 0.22 on average, compared to non use of S4MC refinement, using 366 annotated images from PASCAL VOC 12 Using only Eq. (G.2) reduced the mloU by -14.11 compared to the non-use of S4MC refinement and harmed the model capabilitics to produce quality pscudo labels.
Convergence time
In this section we show that not only S4MC achieve competitive results compared with stale-of-the-ard methods, but also that PLR helps the model converge faster. Fig. H.1 we show that S4MC achieve the same mIoU Unimatch after only 10 epochs, providing faster convergence in terms of training time using the exact same optimization process.
24


Reviews:
[{"review_id": "eebk6EkjtV", "reviewer": "Reviewer_KSiJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\nThis paper is well written, the idea is well motivated, and the key ideas are simple and clear. \n\nThe coverage of related work looks fair.\n\nThe figures are informative and helpful. \n\nThe consistent improvement in results is convincing. \n\n\n\n*Weaknesses* \n\nThe main technical weakness that jumps out at me is the threshold parameter tuning. I believe that prior work is weak in this respect as well, so this is not a critical problem, but still it makes me nervous. I would love to see some assurance that this sort of tuning is worked out on a validation set (never looking at test results), and maybe even done so on a single dataset, and then the choice is transferred to other datasets with good effect.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a technique for semi-supervised learning of semantic segmentation, based on pseudo-labelling. The setup is fairly standard: student-teacher, with teacher being EMA of the student, and pseudo-labels are obtained by checking if a pixel's relative confidence exceeds an adaptive threshold. The key idea in this work is to expand the set of estimates considered \"confident\" by checking if an estimate has support from its spatial neighbors. By replacing single-pixel confidence with a max-pooled confidence (I am approximating some details of Eqs. 6-8 here), more \"good\" pixels are let through the thresholding. This leads to a ~1 point improvement in accuracy, which is small, but this is a consistent improvement across different datasets and task settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I also have several small gripes about the notation and about the method description, which I expect can be resolved easily. \n\nAn image is \\textbf{x}_i, and a pixel is \\textbf{x}_{j,k}^i,  but then later pixels are sometimes {x}^i_{j,k} (not bold). Two issues here: (1) whether pixels are bold or not, this should be consistent; (2) superscript/subscript for the image index should be consistent, so making images \\textbf{x}^i would probably be best. \n\nAbove equation 6, the discussion about \"two pixels\" is confusing and I'm not sure it makes sense. The idea of a neighborhood is introduced, and sounds promising, and then suddenly the text is talking about 2 pixels, and there is no clear connection. \n\nThe max in equation 8 does not seem to be maxing over the right things -- it is a max over $l$ and $m$, but why? The surrounding text suggests it should be a max over the neighborhood. Also, the superscript \\textbf{N} is never properly introduced. \n\nIn section 3.3, it says \"Labeled images are only fed into the student model, producing the supervised loss(2). Unlabeled images are fed into the student and teacher models.\" I am quite sure this is not true. My understanding is that labelled and unlabelled images are fed into the student, and only unlabelled are fed into the teacher. The teacher is, after all, only an EMA of the student. \n\nThe word \"Epoch\" in Fig4a is cut off.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "94d78qZirH", "reviewer": "Reviewer_94pr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written, while the proposed confidence refinement is easy to implement as clearly mentioned in Sec3.2 and Algorithm1.\n2. The authors have discussed the computational cost in Appendix B.\n3. Convincing results are shown in Fig.4. Good quantitative experimental results.\n\nWeakness:\n1. The main concern is the contradictory presentation of the confidence refinement discussed in the paper. As shown in Eq(3) and Eq(4), only the argmax class of pseudo label influence the $\\mathcal{L}_u$. So it is confusing why S4MC still works while \"S4MC primarily derives from the adjusted margin, as the most confident class is rarely swapped\" as mentioned in Sec4.2?\n2. No qualitative results are compared in the paper, while only some confidence refinement examples are shown here.\n3. More details should be discussed about Fig.4. For example, why the fraction is consistent with/without refinement when the data passing threshold is high (>0.9).\n4. Some minor problems: FP was defined twice for Feature perturbations and False positive. The data range should be [0,100] rather than [0,1.0] with symbol \"%\" in Fig.4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a confidence margin refinement manner to enhance the semi-supervised semantic segmentation with a teacher-student pipeline. Specifically, for the pseudo-labels produced by the teacher network, the neighboring-based correlation probability is updated before the Dynamic Partition Adjustment (DPA). Thus the proposed S4MC could achieve superior confidence filtering and semi-supervised segmentation performance. Experiments on PASCAL VOC12, COCO, and Cityscapes show the effectiveness of S4MC.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Weaknesses 1 and 2 should be further clarified, which are both important to be considered for acceptance.", "broader_impact_concerns": "The authors have discussed the broader impact in Appendix D.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WacPfklfJC", "reviewer": "Reviewer_P2oV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- authors propose a novel and interesting idea, also having a hypothesis of correlation between neighbor pseudo-labels or at least the need of their alignment to have less noisy pseudo-labels.\n- extensive experiments showing that the method is working on top of very different prior works and can improve every prior work\n- interesting experiment showing the limitation of the method for boundary IoU\n- interesting finding what is going on with unimatch: kind of showing that it is doing similar thing as proposed method thus improvements on top are marginal\n- diversity of datasets and benchmarks\n\n**Weaknesses**\n- The paper is not clearly written in some places, Table captions do not reflect the whole description, inconsistency in the text.\n- It is not clear / obvious to me on the choice that at least one pixel in the neighborhood should belong to the class. Why not at least two (or more as a hyper-parameter) should belong to the same class following continuity of the object? This is very not a straightforward choice of the method what authors did while describing the intuition behind, with discussion around neighborhood and taking context into account.\n- Authors do not show if their method helps to improve not only quality but also speed of convergence as they are mainly trying to highlight the need of having more reliant data during training and that filtering of many data leads to worse training. Then we really should see (I believe) faster convergence.\n- There is absence of fully supervised baseline in all Tables (or it is not clear where it is) to have the upper bound on the performance. Also the dependence of improvements based on the amount of labeled data is kind of strange (it is not monotonic)\n- Based on Table 8 it seems that main improvement is coming from the fact that confidence threshold is changing with decay over the training and not because of using refinement of the probabilities. Refinement of the probabilities even make it worse alone and seems adapting the threshold over training is more crucial for it rather than for prior works.\n\nFor all weaknesses check details in the requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a modification of pseudo-labeling method for image segmentation problem. Authors propose to revisit the confidence filtering of generated pseudo-labels by refining the probability distribution based on neighbors probability distribution. Doing this allows to have improved quality of pseudo-labels selected and also leads to more data selected every training steps. This results in improvements over the previous methods where the paper’s idea is applied on top. Moreover, authors provide analysis that unimatch prior work actually doing kind of similar thing resulting in only marginal improvement on top of unimatch. Authors also show limitation case of the method like the boundary IoU. The code is available to reproduce / follow the paper.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Improvements of text and clarity:**\n- be consistent with \"pseudo-label\" or \"pseudo label\", Voc 2012 or Voc 12\n- be consistent with the sections camel case or lowercase writing (especially in Appendix)\n- all Tables need to be fixed for the captions: it is not clear which dataset (2012 or 12 e.g. notation), which model (resnet-51, resnet-100, or other arch as it is different per dataset, better to include), which amount of labeled and unlabeled data are used. Tables 2,3,4,5 also do not have notation of number of labeled examples as stated in the caption.\n- link to Hinton - it was not really unlabeled data in the paper, so I would change the reference to something more relevant.\n- why is there a link on Arazo 2020 along with Lee 2014? Any reasons to cite Arazo in this context and not other papers?\n- Figure 1 - classes which have the max prob is cat then why on the figure it is predicted as background on the left plot?\n- Eq. (8) what is $N$?\n- \"In practice we calculate it iteratively ....\" - I don't understand how exactly you compute it\n- Sec. 3.2.2 what is $\\alpha_0$\n- \"To determine the DPA threshold, we use ...\" - I don't understand this sentence too. How do you determine the threshold itself? Is it just a hyper parameter?\n- It is not clear the difference between Table 1 and 2 based on captions.\n- Why is there no (full) column in Tables 3,4,5?\n- \"While Table 2 and 3 ... with unlabeled data ...\" - seems Table 2 doesn't use extra unlabeled data.\n- Discussion of results that you didn't outperform all methods for all partitions - I suggest that you do an apple-to-apple comparison which is fair to do and you are good to go by saying and for all baselines if you add on the baseline proposed method then it always improves the baseline.\n- Sec. 4.2 Cityscapes - it is combined with unimatch not fixmatch!\n- Sec. 4.2 \"Contextual information at inference\" - I don't understand what exactly you do. Also 85.7 and 85.71 are not different, it really seems. What is std?\n- Table 6 any caption description to understand what the model, what the data, what the baseline you used? Also max-prob (k=2) it is not clear how it is calculated.\n- \"Using multiple neighbors demonstrates improved ...\" - this is actually opposite from Table 6.\n- Change Table 7 and 8 order for better readability and paper flow.\n- Table 9 - how much unlabeled data?\n- Code in Algorithm 1 (Appendix) - 3 last rows I think are incorrect\n- Figure F1 (a) \"in in\" typo, caption should explain what is going on\n- Figure A1 - what is baseline - overall in all places captions and descriptions should be clear.\n\n**Empirical suggestions / questions**\n- Figure 4 will be great to extend to the analysis of how many passes that before didn't pass and how many didn't pass that before passed - so the dynamics of how refinement changed the selection itself (not just total number of passes). Also in this graph (b) I think it is missing to show what is happening with PLs accuracy for the baseline. Also in the same (b) figure is it accuracy of the passed PLs or all PLs?\n- Table 1 - what is (full)? Is it fully supervised model? How many unlabeled data are used if it is not? why all methods are different with accuracy on in this (full) column if it is fully supervised baseline (all data are labeled)? Why proposed method is worse than the baseline fixmatch if the proposed method is applied to fixmatch in column (full)?\n- Improvements on top of unimatch in Table 4 between 1/8 and 1/4 are really very different which is not expecting and surprising for me. Is it std big or any other problem?\n- Table 5 - there is no monotonic improvement over the labeled data size growth - why is that?\n- Table 6 what happens if 2x2 or 4x4 is used as there is not exact max/min found yet.\n- I found it to be strange that having more neighbor pixels of highest class probability is actually hurts a lot (or improvements are marginal compared to one neighbor pixel).\n- \"PLR improves the task ... while DPA alone harms ...\" - this is not true based on Table 8. What is actually correct: text or Table? Also why Table 8 is done on 1/2 and not 1/4 as other ablations?\n- There should be at least for one dataset comparison that dynamic confidence threshold over training doesn't give similar value as if authors adds also probability refinement.", "broader_impact_concerns": "I am good with the discussion of limitations and societal risks in appendix D as well as limitation shown in the main body in section 4.3.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers are positive about the paper, citing extensive empirical evaluation and quality of the presentation. All the major reviewers' comments were already incorporated into the manuscript. Thus, I concur and recommend the paper for acceptance without extra revisions.
Decision: Accept (poster)