Learning from Future: A Novel Self-Training Framework for Semantic Segmentation
Authors: Ye Du
Yujun Shen
Haochen Wang
Jingjing Fei
Wei Li
Liwei Wu
Rui Zhao
Zehua Fu
Qingjie LIU
Venue: NeurIPS 2022 Accept (2022)
Keywords: unsupervised domain adaptive semantic segmentation
self-training

Abstract: Self-training has shown great potential in semi-supervised learning. Its core idea is to use the model learned on labeled data to generate pseudo-labels for unlabeled samples, and in turn teach itself. To obtain valid supervision, active attempts typically employ a momentum teacher for pseudo-label prediction yet observe the confirmation bias issue, where the incorrect predictions may provide wrong supervision signals and get accumulated in the training process. The primary cause of such a drawback is that the prevailing self-training framework acts as guiding the current state with previous knowledge because the teacher is updated with the past student only. To alleviate this problem, we propose a novel self-training strategy, which allows the model to learn from the future. Concretely, at each training step, we first virtually optimize the student (i.e., caching the gradients without applying them to the model weights), then update the teacher with the virtual future student, and finally ask the teacher to produce pseudo-labels for the current student as the guidance. In this way, we manage to improve the quality of pseudo-labels and thus boost the performance. We also develop two variants of our future-self-training (FST) framework through peeping at the future both deeply (FST-D) and widely (FST-W). Taking the tasks of unsupervised domain adaptive semantic segmentation and semi-supervised semantic segmentation as the instances, we experimentally demonstrate the effectiveness and superiority of our approach under a wide range of settings. Code is available at https://github.com/usr922/FST.

Learning from Future: A Novel Self- Self-Training Framework for Semantic Segmentation
Ye Dul.2 Yujun Shen
Haochen Wang*
Jingjing Fei
Wei Li
Liwei Wu
Rui Zhao Zehua Fu
Qingjie Liu
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University 2 Hangzhou Innovation Institute, Beihang University
3 The Chinese University of Hong Kong
4 Institute of Automation, Chinese Academy of Sciences
SenseTTime Research
Qing Yuan Research Institute, Shanghai Jiao Tong University, Shanghai, China
{duyee, ze, qingjie liub @buan.edu.cn shenys.ju00302@gmi11 com {feij ingjing, liveil, wuliwei, (haorui)@sensettilll
wanghaaochen222201a.ccmm ac.
Abstract
Self-training has shown great potential in semi-supervised learning. Its core idea is the model learned labeled data to generate pseudo-label. for unlabeled samples, and in turn teach itself. To obtain valid supervision, active attempts typically employ momentum teacher for pseudo-label prediction yet observe the confirmation bias issue, where the incorrect predictions may provide wrong supervision signals and get accumulated in the training process. The primary cause of such drawback is that the prevailing self-training framework acts as guiding the current state with previous knowledge, because the teacher is updated with the past student only. To alleviate this problem, we propose a novel self-training strategy, which allows the model learn from the future. Concretely, each training step, we first virtually optimize the student (i.e., caching the gradients without applying them the model weights), then update the teacher with the virtual future student, and finally ask the teacher to produce pseudo-labels for the current student as the guidance. this way, we manage to improve the quality of pseudo-labels and thus boost the performance. We also develop two variants of our future-self-trainee (FST) framework through peeping at the future both deeply (FST-D) and widely (FST-W). Taking the tasks of unsupervised domain adaptive semantic segmentation and semi-supervised semantic segmentation as the instances, we experimentally demonstrate the effectiveness and superiority of our approach under a wide range settings. Code is available at https:/giitubb.oo//us/92211
Introduction
Improving the labeling efficiency of deep learning algorithms is vital in practice since acquiring high-quality annotations could consume great effort. Self-training (ST) offers a promising solution to alleviate this issue by learning with limited laheled data and large-scale unlabeled data [47, 24]. The key thought is to learn model on labeled samples and use it to generate pseudo-labels for unlabeled samples teach the model itself. In general, teacher network that maintains an exponential moving average (EMA) of the student (i.e., the model to learn) weights is used for pseudo-label prediction, as shown Fig. la. Intuitively, such a training strategy relies on the previous student states supervise
"Coreepponding Author.
This work is done when Ye Du and Haochen Wang interns SenseTime Research.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1
0
Aye
Eye
BMA
EMA
EMA
1
9.43
%-1
2, 1,41
9--1
Toucher
Teachur
(a) Self-training
(b) Enture-self-trriinng
Figure 1: Concept comparison between self-training (ST) and our self--riin-g (FST). (a) ST employs a teacher, which collects information from the past states, to supervise the student. (b) Our FST derives teacher at the future moment and utilizes it to guide the current student.
the current state, which amounts to using a poor model to guide good one given the fact that a model tends to perform better along with the training process. As result, the confirmation bias issue [4, 9] emerges from existing ST approaches, where the wrong supervision signals caused by those incorrect pseudo-labels get accumulated during training.
To break through the predicament of seeking supervision only from the past states, we propose neture-self-trainne (FST), which allows the model to learn from its future self. Fig. 1b illustrates the concept diagram of our FST. Compared to the conventional ST framework in Fig. la, which employs the t-step teacher (i.e., updated with the student at moments 1, 2, ... , t-step student, FST presents a new training manner by urging the 1-step student to learn from the (t + -step teacher. However, at the start of the training step t,, the (t + 1) 1s-ttep teacher is not available yet since is dependent on the to-be-optimized t-step student. To tackle this obstacle, we come up with a virtual updating strategy. Concretely, we first optimize the current student just like that in the traditional ST. Differently, we do not actually update the student weights but cache the gradients instead. Such stashed gradients can be treated as the "virtual future" and help derive the (I 1)-step teacher. Finally, the training of step borrows the pseudo-labels predicted by the latest teacher, and this time we apply the gradients the student weights for real.
1) to guide the
Recall that our motivation of encouraging the model to learn from the future is to help it acquire knowledge from advanced teacher. To this end, put forward two variants based on our FST framework to make the teacher more capable. On the one hand, we propose FST-D to investigate the future deeply. For this case, we ask the teacher move forward for K steps via virtual updating. thus the 1-step student can be better supervised by the (4- - K) -step teacher. On the other hand, FST-W originates from the idea of model soups [53], which reveals that the averaging weights of multiple fine-tuned models can improve the performance. We hence propose to explore the future widely with teachers developed from different training samples and expect the student learn from all these (t + 1)-step teachers simultaneously.
Pscsidoslabel Quality on Trahning Set
Performance Validation
We evaluate proposed FST the tasks of both unsupervised domain adaptive (UDA) semantic segmentation and semi- supervised semantic segmentation. The superiority of FST over the prevailing ST framework is summarized in Fig. 2, where our teacher model is capable of producing pseudo-labels with much higher quality and hence assists the student with a better performance. This is because, along with the training process, the future states usually outperform the past states and thus can provide more accurate supervision, reducing the damage of confirmation bias. Such a comparison validates our primary motive of learning from the future. Furthermore, we observe consistent performance gain under a broad range of experimental settings (e-g., network architectures and datasets), demonstrating the effecctiveness and generalizability of our approach.
St
Mean Interection cor baten
nST (sas)
Eiror Hate
5.7 %
FST (ounl)
Training Bontioo
Tralting Becetion
Figure 2: Performance comparison between self- training (ST) and our future-self-traine (FST), in- cluding the pseudo-label quality on unlabeled training samples (left) and the evaluation performance (right). The comparison is conducted under the same number of updates of the student, which is the final model used for evaluation.

---

2 Related work
Domain adaptive semantic segmentation. UDA semantic segmentation aims at transferring the knowledge from a labeled source domain to an unlabeled target domain, which is often viewed as a special semi-supervised learning problem. Early methods for UDA segmentation focus on diminishing the distribution shift between the source and target domain at the input level [23, 40, 18], the feature level [46, 10, 8, 31J, or the output level [46, 48, 36]. Over the years, adversarial learning [19, 17] has been the dominant approach to aligning the distributions. However, the alignment- based methods may destroy the discrimination ability of features and cannot guarantee a small expected error on the target domain [58]. In contrast, self-training [2], which originated from semi-supervised learning (SSL) [29], is introduced to directly minimize a proxy cross-entropy (CE) loss on the target domain. By leveraging the model itself to generate pseudo-labels on unlabeled data, self-training together with tailored strategies such as consistency regularization [61, 3], cross-domain mixup [45, 62]. contrastive learning [26, 34, 63, 31], pseudo refine [51, 58, 60], auxiliary tasks [49, 50] and class balanced training [30] achieves excellent performance. Recently, Hoyer et al. [24] empirically proved that the transformer architecture [55] is more robust domain shift than CNN. They propose a transformer-based framework with three efficient training strategies in pursuit of milestone performance.
Semi-supervised semantic segmentation. Self-training is widely studied in SSL literature [47]. To facilitate the usage of unlabeled samples, Tarvainen al. [44] propose a mean teacher framework for consistency learning between a student and a momentum updating teacher. This idea is extended later to supervupeee semantic segmentation, which trains the student model with high-confident hard pseudo-labell predicted by the teacher. On this basis, extensive attempts improve semi-supervised semantic segmentation by CutMix augmentation [16], class- balanced training [64, 25, 21] and contrastive learning [64, 1, 33, 52]. A closely relevant topic to self- self-training in SSL is consistency regularization, which believes that enforcing semantic or distribution consistency between various perturbations, such as image augmentation [27] and network perturbation [57], can improve the robustness and generalization of the model. general, consistent regularization methods are used together with a ST framework. We focus improving the basic ST in this work.
Nesterov's accelerated gradient descent. A related idea to our work Nesterov's accelerated gradient descent (NAG), Originally proposed in [37] for solving convex programming problem, NAG a first-order optimization method with a better convergence rate than gradient descent. With the rise of deep learning, NAG is adopted as an alternative to momentum stochastic gradient descent (SGD) to optimize neural networks [43, 14]. It is intuitively considered to perform a look ahead gradient evaluation and then make correction [7]. Due to its solid theoretical explanations [65, 5] and remarkable performance, many works incorporate NAG with various tasks. In [32], Lin et al. adopt NAG into the area of adversarial attack, where they propose a Nesterov's 's iterative fast gradient sign method to improve the transfer ability of adversarial examples. In [56], Yang et al. explore the utilization of NAG in federal learning. Different from NAG that pursues accelerated convergence, our work aims at building a stronger pseudo-label generator and improving the performance of traditional self-training.
Method
3.1 Background
Consider such real-world scenario where we have access to a labeled segmentation dataset D,, {z), M from distribution P and an unlabelled one Du = {Ix ""= from unknown distribution Q. We are required to build a semantic segmentation model using the combination of Dr and Dy. A general case is when P # Q, the problem falls into the category of UDA semantic segmentation. Otherwise, it is usually treated as regular SSL task.
Self-training provides unified solution and achieves sfate-of-the-ee performance on both settings [24, 52]. One of the most common and widely used forms of self self-training in semantic segmentation a variant of mean teacher, which is shown in Fig. 3. Denote by go the segmentation model required to be trained, and 0 its parameters. The mean teacher framework trains the student go on unlabeled data with pseudo-label, predicted by momentum teacher go. which has the same architecture to the student but with different parameters ø. Specifically, the training progresses, the teacher
3

---

evolves with the student by maintaining an EMA of student weights on each training iteration. This ensembling enables generating high quality predictions on unlabeled samples, and using them as training targets improves performance. Formally, at each training step, the teacher is first updated and then predict pseudo-labes train the student.
Φ++1 = p&d: + (1 - 11)t,
(I)
0,+1 0, Vo (zi), vi) + XC(go, (#w), Du (1)++)
where ju the momentum coefficient, 7 the learning rate, and À is the dynamic re-weighting parameter to weigh the training of labeled and unlabeled data. Un. denotes the pseudo-labels predicted by 0r+1, i.e,, Du = arg max P6.41 (In). L is the pixel-wise cross-entropy training objective, which can be written as
HxW Σ
((x,y)
Σ
log
(2)
==1
where x W is the imput image size and the total number of classes.
Limitation of self-training. Despite the remarkable performance, self-training suffers from the problem of confirmation bias. To be specific, the inherent noise in pseudo-labels could undesirably mislead the student training, which in return affects the pseudo- label prediction, and thereby results in noise accu- mulation. Though a momentum updating strategy in the teacher framework improves tolerance with inaccurate pscudo-labels, this issue is still a bottleneck since the student still relies on learning from its own past training states.
gh
Le
an
(b,)
%
Acce
(x,,,)
V.
%
96( x.))
9c
Figure 3: Illustration of the ST framework
with teacher 9a- "sg" means stop-gradient.
3.2 Learning from future self
An intuitive observation shown in Fig. 2 is that the performance the student model generally improves during training. despite the noise in supervision. From this perspective, reasonable conjecture is, can we model information from future moments to guide the current training iteration? Motivated by this, we propose snture-self-traiinng for facilitating the utilization of unlabeled data in semantic segmentation. Concretely, each training step, we propose to directly update the teacher model by the student weights from the next training moment. To this end, simple modification Eq. (I) made follows.
Φ++1 = póc + (1) (6, V [C(g,, (21). 91) + XL(go, (.ll)))).. 0.+1 = 0, Vo [[go, (.1), vi) + XC(go, (Ir), Vul0e+1)]
(3)
Furthermore, it can be seen that Eq. (3) only uses virtual future state to update the teacher and ignores the current student weights 0,. Our mission here to establish a reliable pseudo-label generator (i.e. stronger teacher). In terms of the ensembling effect of EMA, it is not necessary to discard fg. Therefore, an improved version of FST is proposed as follows.
*'+1 =.0: + +11 p)0e.
0 pl n' + (1 11_1111 (0) (o(C(o v)) + XC(go, (zw), Bu 16441))) 8,+1 = 8, ve [C(go, (21): y) + AC(ge, (Ia): &u Buld:41)],
where momentum parameter p' is introduced to distinguish the contribution of current and future model weights to teacher updates. We provide pseudo-codes to further illustrate how we implement Eq. (4) in Supplementary Material.
3.3 Exploring a deeper future
We reiterate that the key insight of FST is to look ahead during training, which allows mine more accurate supervision from future model states. In experiments, we found that Eq. (4) exhibits only slight improvement in performance (Tab. 2), showing that this one-step future exploration strategy insufficient.

---

Therefore, we further propose looking ahead deeper strategy to peek into deeper future student states. To be specific, each training step, we update the teacher not only with the student weights from the next moment, but also with those from deeper steps. Formally, denote by Φt = =uo + (1 - p2)& and 0. 0, two agent variables firstly. Then, we can use the co-evolving de and 0r for virtual updating as follows.
01+k+1 Petk 7 Voll(on,, (2i), ui) AC(05, (zu). Iu bul@++))
(5)
@c+++1 n'ō1+2 + (1 (11-~p))00 (0e++++)1),
where == {0,... K K- indexes the serial virtual steps for current training and K' is the total number of exploration steps. Finally, we use the future information aware teacher 0t+K as the pseudo-label generator to supervise the current training. A simple reassignment and gradient descent update applied to form the deeper version of FST, which is so called FST-D as shown below.
Φ1+1 Øt+K.
(6)
0.+1 = 0g
n)) + AC(ge, (zu), Du (++1)).
3.4 Exploring a wider future
On the other hand, looking ahead wider instead of deeper is another intuitive way to enhance future exploration. Inspired by the recent progress [53] that an ensemble of different model weights often shows excellent performance, we propose to first explore the next moment in different optimization directions and then use the average of them to update the teacher. Concretely, we obtain different optimization directions by feeding different data batches to the student model at each training moment. Thus, a wider version FST, i.e. FST-W, W. is presented as follows.
N Σ
0c+1 = p'(por + (1 p2)0} + (1 (()
(el + AC(go, (xt),
N
0++1 = Voliggl (zr), w) + AC(go, (Ix). .. Du 1)]l
where indexes different samples and N is parallel virtual exploration steps.
Eq. (7) holds due to the fact that averaging the model weights is equivalent to averaging the gradients first and then updating the parameters by gradient descent. worth noting that FST-D and FST-W are complementary that can be utilized together. However, this is beyond the scope of our work, and we leave this exploration the future.
Experiment
The experiment section is organized follows. First, illustrate the experimental setup and implementation details in Sec. 4.1 and See. 4.2. Then, we evaluate the proposed FST and analyze the two variants in Sec. 4.3. After that, we conduct extensive ablation studies to dissect method in Sec. 4.4. Finally, we compare our FST with existing state-of-the- alternatives on both UDA and semi-supervised benchmarks in Sec. 4.5.
4.1 Setup
Datasets and tasks. We evaluated our method on UDA and semi-supervised semantic segmentation. In UDA segmentation, we use synthetic labeled images from GTAV [38] and SYNTHIA [39] the source domain and use real images from Cityscapes [13] as the target domain. addition, PASCAL VOC2012 [15] is used for standard semi-supervised evaluation. To simulate a semi-supervised setting, we randomly sample a portion (i.e., 1/4, 1/8, and 1/16) of images together with corresponding segmentation masks from the training set the labeled data and treat the rest as the unlabeled samples.
Evaluation metric. Mean Intersection over Union (mloU) is reported for evaluation. In SYNTHIA - Cityscapes UDA benchmark, and 13 the 19 classes of Cityscapes are used to calculate mloU, following the common practice [3, 24].
5

---

Table 1:
Comparison between ST and our FST, where we explore the future with either (a) the same data batch as the current or (b) different data batch from the current. "SourceOnly" means training the model with labeled data only, whose result borrowed from [24] as the reference. 4
means using quadruple samples per mini-batch. All results are averaged over 3 random seeds.
Method
mloU
A
Method
Batch
mloU
A
SourceOnly
34.3 2.2
SourceOnly
1x
34.3
2.2
56.3 0.4
ST
1x
56.3
0.4
ST
4x
55.5
0.4
4 0.8
Naive-FST
56.4 4 0.4
0.1
Haive-FST Improved-FST
1x
58.7
1 2.3
2.3
Improved-FST
57.7 + 0.6
1.4
1x
58.7
03.7+00 0.7
2.4
FST-W
56.8 0.1
+ 0.5
FST-W
1x
59.3
+ 0.5
3.0
FST-D
59.8 + 0.1
3.5
FST-D
1x
59.6
- 1.4
3.3
(a) Future exploration with the same data batch.
(b) Future exploration with a different data batch.
Baselines. We first build strong baselines of the classical ST framework. For UDA segmentation. we adopt the basic framework from [45]. which contains a ClassMix augmentation. Standard cross-entropy loss is calculated on both labeled and unlabeled data. We use the efficient Encoder- Decoder structure for all semantic segmentation models, where the networks various in the structure of encoders and decoders. In the eemi-supervised benchmark, we use the classical ST without other tricks as the baseline, because it has been proved to achieve competitive performance while maintaining simplicity [27].
4.2 Implementation details
Image augmentation. The proposed FST and its baselines use the same image augmentation for fair comparison. In UDA semantic segmentation, color jitter, Gaussian blur and ClassMix [45] are used as the strong data augmentation for the unlabeled target domain, which follows the practice in [24]. In supervise semantic segmentation, we use random flip and random crop, and the images are resized to 513 513 for both teacher and student.
Network architecture. We use the DeepLabV2 [11] as the basic segmentation architecture for UDA segmentation, where the ASPP decoder only uses the dilation rates 6 and 12 following [46]. For Transformer-basee networks, we adopt from [24] and [54] as the decoders. In semi-supervised segmentation, we evaluate our method on the commonly used DeepLabV? [22]. DeepLabbV3 V3+ [12]
and PSPNet [59] with ResNet-101 [22] as the backbone.
Optimization. In UDA segmentation, the model is trained with an AdamW [28] optimizer, a learning rate of 6 x 10-5 for the encoder and 6 10-4 for the decoder, weight decay of 0.01, linear learning rate warmup with 1.5k iterations and linear decay afterwards. We train the model on a batch of two 512 512 random crops for total of 40k iterations. The momentum is set 0.999. In semi-supervised segmentation, the model is trained with SGD optimizer, a learning rate of 0.0001 for the encoder and 0.001 for the decoder, weight decay of 0.0001. We train the model with 16 labeled and 16 unlabeled images per-batch for total of 40 epochs.
4.3 Comparison with self-training
We first comprehensively compare our FST with classical ST to evaluate the effectiveness. The results are shown in Tab. 1. To simplify, we use GTAV as the labeled data and Cityscapes as the unlabeled data for evaluation. All methods use the same experimental settings for faimesss
Quantitative analyses. We illustrate the improvements of Naive-FST (Eq. (3)), Improved-FST (Eq. (4)), FST-D (Eqs. (5) and (6)) and FST-W (Eq. (7)) compared with classical ST (Eq. (1)) in Tab. 1a. These methods use the same batch of data for virtual forward at each step of future exploration. As presented, Naive-FST only shows negligible boost because the current student state is discarded without contributing to the teacher. By revising it, the improved FST in Eq. (6), which a special case FST-D when K = 1, achieves an improvement of 1.4% mloU. Further, FST-D (with K 3) clearly outperforms ST by margin of 3.5% mloU, which benefits from the higher-quality pseudo-labels generated by more reliable teacher as shown in Fig. In contrast, FST-W W shows a slight improvement of only 0.5% mloU under the same data batch setting. Thus,

---

Mean Intemnttonn Union
dlyy
petsoo
Eiight
like
milaFale
(K+2]
FST (K-5)
FBT (Ko4)
Training Iteration
(b)
Image
Ground Truth
ST FST (ouns)
Figure 4: (a) Performance curves for ST and FST with various K values. The comparison is conducted under the same number of updates of the student, which is the final model used for evaluation. (b) Qualitative comparison on Cityscapes [13], where dashed white boxes highlight visual improvements.
Table 2: Generalization of FST across architectures. All results are averaged over 3 random seeds.
Method
K
mloU
Mothod
K
mloU
Methed
K
mloU
55.0 0.9
ST
56
ST
56.3
0.8
EST
2
56.3 1.0
1.3
EST
2
57.8 + 1.3
1.5
FST
2
58.1 + 3.1
1 1.8
FST
56.9
= 0.5
1.9
3
3.5
FST
3
58.5 0.7
1,2
FST
56.4 0.9
1.4
EST
4
59.7
3.4
FST
58.8
:: 1.0
2.5
(a) DeepLabV2 (1I) ResNet- 50 (22).
(b) V2 (1I) w/ ResNet-101 [22].
(c) PSPNet (59) ResNet-101 [22).
Method
mloU
A
Method
K
mloU
A
Method
K
mlnU
A
ST
61.3 + 0.7
ST
59.9 2.0
ST
68.3 :: 0.5
FST
63.7 2.0
t 2.4
FST
2
62.5 + 1.2
2.6
FST
2
69.1
0.8
EST
64.3 2.3
13.0
FST
62.5
1.6.6
FST
60.3
1.0
FST
6..
3.1
FST
6.
2.7
FST
4
68.8 0.9
0.5
(d) UPerNet [54] w/ Swin-B [35].
(e) UPerNet [54] w/ BEiT-B [6].
DAFormer [24] w/ MiT-B5 [55].
prefer the deeper variant and adopt it as the basic approach in this paper, i.e., FST stands for FST-D unless specified. We also analyse the effect of exploration steps (i.e. K) on the training process. As suggested in Fig. 4a, FST spends only about 1/3 of the total training time reach the performance level of ST. Besides, we find that larger can achieve higher mloU at the beginning of the training process. When K 4, however, the performance in later training iterations drops and gets worse than = We speculate that this because the deeper exploration becomes unnecessary in the later training stage. This interesting phenomenon indicates that adaptive exploration mechanism may bring better results.
Qualitative analyses. Fig. 4b provides some qualitative comparisons, where FST can correct some mistakes made by ST. Taking the presented sample in the second row as an instance, ST struggles to distinguish between bicycle and motorcycle, while our FST successfully predicts it. More visualization results and analyses can be found in Supplementary Material.
Data batches for future exploration. In Sec. 3.4, we derive FST-W, which uses different samples for future exploration parallel. Tab. and Tab. 1b compare the performance of using the same and different data batches. Note that FST-W in Tab. Ia could produce slightly different mixed images for virtual forward, since we use ClassMix augmentation. is obvious that parallel exploration with different samples performs better because the differences between models are important for ensembling.
Generalization of popular architectures. To verify the generality under various advanced semantic segmentation models, we evaluate FST (the deeper variant) on two mainstream backbones (i.e., CNN and Transformer) with four commonly used segmentation decoders. As presented in Tab. 2, FST shows consistent performance improvement over classical ST, including DeepLab [11], PSPNet [59] and UPerNet [54]. Besides, FST shows significant improvements not only on supervised pretrained CNN [22] and Transformer backbones [35, 55] but also on unsupervised pretrained BEiT [6]. Note that, the established ST baselines are strong, which even surpass many complex multi-stage methods
(e.g., [58]) proposed recently. FST achieves 59. 59.8% mloU using Deepl.ab Lb and ResNet-101. More
comparisons between FST and existing CNN-based methods are provided in Supplementary Material.
7

---

Table Analyses on the two variants of FST, including FST-D (Sec. 3.3) and FST-W (Sec. 3.4).
All results are averaged over 3 random seeds.
Method
Backbone
mloU
Method
Backbone
mlnU
ST
ResNet-101
56.3 + 0.4
ST
ResNet-101
56.3 +: 0.4
FST-D
ResNet-101
2
58.6 1 0.4
2.3
FST-W
ResNet-101
58.5 1: 1.6
2.2
FST-D
ResNet-101
3
59.6 + 1.4
1 3.3
FST-W
ResNet-101
59.3 0.5
3.0
FST-D
ResNet-101
59.8 + 2.0
1 3.55
FST-W
ResNet-101
58.6 + 2.0
1 2.3
(a) Effect of K in FST-D.
(b) Effect of N in FST-W.
Table 4: (a) Ablation study on the hyper-parameter p' (Sec. 3.2). (b) Comparison with longer- training baselines. All results are averaged over 3 random seeds.
Method
n¹
mloU
Method
Backbone
Schedule
mloU
A
ST
56.3 + 0.4
ResNet-101
56.3 0.4
ResNet- 101
59.3 0.6
3.0
FST
ResNet-101
1x
59.8 + 0.1
3.5
EST
0.99
58.8 + 1.6
1 2.5
EST
0.999
59.8 + 0.1
3.5
ST
MiT-B5
1x
68.3 + 0.5
EST 0.9999
58.7 1 0.6
2.4
ST
MiT-B5
3x
68.3 + 1.1
f 0.0
FST
0.99999
59.9 = 0.9
1 3.6
FST
MiT-B5
1x
69.1 1 0.3
1 0.8
(a) Effect of p² in Eq. (5).
(b) Comparison with longer training schedules.
4.4
Ablation studies
Deeper wider. It can be seen from Tab. that FST-D performs better than FST-W matter using the same or different data batches for future exploration. is worth noting that under the setting of Tab. lb, the teacher model in FST sees more data per-iteration compared the original ST, which has the effect of expanding the batch size in disguise. To make a fair comparison, we also build a ST baseline with a larger batch size. The results show that the performance gain of FST-W comes from the method itself instead of utilizing more data in each iteration. In addition, we conduct ablations to compare FST-D and FST-W with each step (i.e., and N), which are shown in Tab. 3a and Tab. 3b. Both variants use different data batches for future exploration since FST-W performs well only under this setting. It can be concluded that FST-D performs better than FST-W, which is consistent with the conclusion in the above. Besides, comparing Tab. 2b and Tab. 3a, we observe that using different data batches amplifies the performance jitter of each run. We guess that this may be due the diversity of data sampled for future exploration.
Effect of the serial exploration steps K. K controls the number of virtual exploration steps in FST-D. In Tab. 2, we ablate K on six settings, each evaluated over 3 runs. As presented, K = 3 shows steady improvements, while increasing it further brings negligible impact. Thus, we recommend using K = 3 as the basic practice. We also ablate K when using different data batches for exploration, which is presented in Tab. 3a.
Effect of the momentum ja. It is a common practice to set the momentum of EMA to a large value such as 0.999 in self-training. A separate momentum μ' that controls the contribution of future student states to the teacher is set in our FST. We conduct ablation experiments to observe the effect of p². As shown in Tab. 4a, FST shows robustness against the change of μr. In experiments, set n' = 0.999 as the default setting unless specifically stated, which equals the value of μ.
Effect of the parallel exploration steps N. In FST-W, N controls the parallel exploration steps of the next training moment. We conduct ablation experiments in Tab. 3b to verify the influence of N. As can be seen, N 3 performs well among the evaluated values, which is similar observation to K and implics that is an acceptable choice in practice.
Longer training schedules. We perform forward and backward propagation to obtain weights as the estimation of future student states. This simple implementation linearly increases the training time over the number of exploration steps. i.e., K. Note that in our method, the student is trained with the same training iterations as classical ST and does not see more samples per-iteration, thereby the comparisons in Tabs. and 2 are totally fair. Even though, we establish stronger baselines with longer training schedules and compare them with our method. We find that the performance of the longer scheduled ST baseline decreases heavily in the later training stages the model is fitting the
8

---

Table Evaluation on the semi-supervised learning (SSL) setting on PASCAL VOC 2012 [15],
where 1/16, 1/8, and 1/4 stand for using 664, 1323, and 2646 samples as the labeled set, respectively.
PSPNet [59]
DeepLabV2 [11]
Deep [12]
Method
1/16
1/8
1/4
1/16
1/8
1/4
1/16
1/8
1/4
ST
65.47
72:24
76.47
68.45
72.54
76.21
73,31
74.20
77.78
FST (ours)
68.35
72.77
75.90
69.43
73.18
76.32
73.88
76.07
78.10
A
2.88 f
0.53
0.43 f
0.98
0.64
0.11
0.57
1.87
0.32
Table 6: Evaluation on the unsupervised domain adaptation (UDA) setting two benchmarks.
Our results are averaged over random seeds.
Road
S.walk
Build.
Wall
Pole
T.iggtt
Sign
Ternin
Sky
Pesso
Rider
Car
Trock
Bus
Tainn
Bike
Method
mloU
GTAV [38] -> Cityscapes [13]
SourceOlly
76.1
29.8
34.5
2.9
29.7
46.1
ProDA [58]
87.8
56.0
79.7
16.3
14.8
15.6
54.5
53.5
88.6
13.2
82
70.7
39.2
88.8
15.5
59.4
1.
18.9
56.4
57.5
CPSL
92.3
59.9
81.9
45.7
29.7
41.5
85.0
35.5
90.4
48.7
26.3
53.9
60.8
DAFormer [24]
95.7
70.2
89.4
53.5
48.
49.6
55.8
59.4
89.9
47.9
92.5
72.2
44.7
92.3
74.5
78.2
65.1
55.9
61.8
68.3
FST (ours)
963
07.
55.5
50.1
58.6
92.5
74.4
57.7
62.6
69.3
SYNTHIA
[39] - Cityscaes
[13]
SourceOnly
56.5
23.3
81.3
16.0
41.0
24.1
82.5
62.3
23.8
77.7
15.0
23.7
42.4
ProDA [58]
87.8
45.7
84.6
37.1
0.6
44.0
54.6
37.0
88.1
84.4
74.2
24.3
88.2
51.1
40.5
45.6
55.5
CPSL. [30]
43.9
85.5
33.6
47.7
37.2
88.5
79.0
32.0
59.8
57.9
DAFormer [24]
84.5
40.7
88.4
41.5
6.5
50.0
55.0
54.6
86.0
89.8
73.2
48.2
87.2
53.2
53.9
61.7
60.9
(ours)
88.0
41.7
32.5
23.9
48.1
noise in pseudo-labels. Besides, as shown in Tab. 4b, the performance of longer training baselines still performs worse than FST, which further proves the effectiveness of our method.
4.5 Comparison with state-of-the-art alternatives
In this subsection, we evaluate our FST with state-of-the-art approaches on the tasks of semi- supervised semantic segmentation and unsupervised domain adaptive semantic segmentation.
Evaluation semi-supervised segmentation. We first evaluate the proposed FST traditional semi-supervised semantic segmentation. As shown in Tab. 5, we compare FST (with K = 3) with ST on three partition protocols. Equipped with three commonly used semantic segmentation networks, ie., PSPNet [59]. DeepLabV2 [11], and DeepLab V3+ [12], FST consistently improves classical ST by considerable margins. For instance, on the 1/8 partition protocol, FST with DeepLab outperforms ST by 1.87% mIoU, showing substantial improvement. In short, FST demonstrates remarkable performance the -uperviid benchmark. More comprehensive comparisons between our FST against state-of-the-as art alternatives can be found in Supplementary Material.
Evaluation unsupervised domain adaptive semantic segmentation. We then compare FST with previous self-training based competitors on UDA benchmark. The results are presented Tab. Our FST is build upon DAFormer [24], which the state-of-the-art method currently. On GTAV Cityscapes benchmark, our FST exceeds DAFormer by a considerable margin of 1.0% mloU and shows dominant performance in most categories. Compared the source only trained model, FST surpasses it by 23.2% mloU, indicating further step toward practical applications. The bottom part of Tab. 6 shows the comparisons on SYNTHIA - Cityscapes benchmark. FST exceeds DAFormer by 1.0% mIoU on 16 classes. Besides, our FST achieves 68.5% mloU on 13 classes, exceeding DAFormer by 1.1%. In summary, our FST achieves new state-of to-the-rtttt performance on the UDA benchmark.
Conclusion
this paper, we present a future-self-traiinng framework semantic segmentation. As an alternative to classical self-training, our approach mitigates the confirmation bias problem and achieves better performance on both UDA and semi-supervised benchmarks. The key insight of our method is to mine model's own future states as supervision for current training. To this end, propose two

---

variants, namely FST-D and FST-W, to explore the future states deeply and widely. Experiments on wide range of settings demonstrate the effectiveness and generalizability of our methods.
Discussion. The major drawback of this work is that our approach is time-consuming, since we need to forward temporary model to acquire virtual future model states. Although the number of updates to the student does not increase, it is better to optimize the process of obtaining future model parameters. To this end, an acceptable way is to maintain ahead student model provide information from future moments, which trades space for time. Besides, our approach is general that can be applied to other supervised image recognition [44]. object detection [35]. few-shot learning [42], and unsupervised representation learning [20]. We will conduct further studies on these issues in the future.
frameworks such as FixMatch [41] and other tasks such as semi-
Acknowledgements. This work was supported by the National Natural Science Foundation of China under Grants 62176011 76017, 4187 1283 and U20B2069, and the Fundamental Research Funds for the Central Universities.
References
[1] I. Alonso, A. Sabater, D. Ferstl, L. Montesano, and A. C. Murillo. Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank. In Int. Conf. Comput. Vis., 2021.
[2] M.-R. Amini, V. Feofanov, L. Pauletto, E. Devijver, and Y. Maximov. Self-training: A survey, arXiv preprint Xro: 12040, 2022.
[3] N. Araslanov and S. Roth. Self-supervised augmentation consistency for adapting semantic segmentation.
In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
E. Arazo, D. Ortego, P. Albert, N. O'Connor, and K. McGuinness. Pseudo-labeling and confirmation bias deep semi-supervised learning. Int. Joins Conf. Neural Networks, 2020.
[5] M. Assran and M. Rabbat. On the convergence of nesterov's 's accelerated gradient method in stochastic settings. In Int. Conf. Mach. Learn., 2020.
[6] H. Bao, L.. Dong, and F. Wei. Beit: Bert pre-training of image transformers. In Int. Conf. Learn. Represent., 2022.
[7] A. Botev, G. Lever, and Barber. Nesterov's s accelerated gradient and momentum as approximations to regularised update descent, In Int. Joint Conf. Neural Networks, 2017.
[8] W.-L. Chang, H.-P. Wang, W.-H. Peng, and W.-C. Chiu, All about structure: Adapting structural information across domains for boosting semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[9] B. Chen, J. Jiang, X. Wang. J. Wang, and M. Long. Debiased pseudo labeling in self-training. arXiv preprint 202.:2200..07.36. 2022.
[10] C. Chen, W. Xie, W. Huang, Y. Rong, Ding, Y. Huang, T. Xu, and Huang. Progressive feature alignment for unsupervised domain adaptation. In JEEE Conf. Comput. Vis. Pattern Recog,, 2019.
[11] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEFF Trans. Pattern Anal. Mach. Intell., 2017.
[12] L.-C. Chen, Y. Zhu, G. Papandreou, Schroff, and Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Eur. Conf. Comput. Vis., 2018.
[13] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE Conf. Comput. Vis. Pattern Recog,, 2016.
[14] T. Dozat. Incorporating nesterov momentum into adam. In Int. Conf. Learn, Represent. Worksh., 2016.
[15] Everingham,
Gool, C. K. Williams, J. Winn, and Zisserman. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis., 2010.
[16] G. French, S. Laine, T. Aila, M. Mackiewicz, and G. D. Finlayson. Semi-supervian semantic segmentation needs strong, varied perturbations. In Brit. Mach. Vis. Confi, 2020.
10

---

[17] Y. Ganin. E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F, Laviolette, M. Marchand, and V. Lempitsky. Domain-adversarial training of neural networks. JMLR, 2016.
[18] R. Gong, W. Li, Y. Chen, and L. V. Gool. Dlow: Domain flow for adaptation and generalization. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[19] 1. Goodfellow, J. Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Adv. Neural Inform. Process. Syst. 2014.
[20] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Docrsch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Adv. Neural Inform. Process. Syst., 2020.
[21] D. Guan, J. Huang, A. Xiao, and S. Lu. Unbiased subclass regularization for semi- supervupe semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog. 2022.
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In /EEE Conf. Comput. Vis. Pattern Recog., 2016.
[23] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell. Cycada: Cycle- consistent adversarial domain adaptation. In Int. Conf. Much. Leurn., 2018.
[24] L. Hoyer, D. Dai, and L. Van Gool. Daformer: Improving network architectures and training strategies for Nomain-adaptivy semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[25] H. Hu, F. Wei, H. Hu, Q. Ye, J. Cui, and L. Wang. Sumi-super semantic segmentation via adaptive cqualization learning. Adv. Neural Inform. Process. Syst. 2021.
[26] G. Kang, Y. Wei, Y. Yang, Y. Zhuang, and A. Hauptmann. Pixel-level eycle association: new perspective for domain adaptive semantic segmentation. In Adv. Neural Inform. Process. Syst., 2020.
[27] Z. Ke, D. Qiu, K. Li, Q. Yan, and R. Lau, Guided collaborative training for pixel-wise semi-supervised learning. In Eur. Conf. Comput. Vis., 2020.
[28] D. Kingma and Ba. Adam: A method for stochastic optimization. In Int. Conf. Learn. Represent.. 2015.
[29] D.-.. Lee al. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Int. Conf. Mach. Learn 2013.
[30] R. Li, S. Li, C. He, Y. Zhang, X. Jia, and L.. Zhang. Class-balanced pixel-level self-laheling for domain adaptive semantic segmentation. In IEFE Conf. Comput. Vis. Pattern Recog., 2022.
[31] S. Li, B. Xie, B. Zang, C. H. Liu, X. Cheng, R. Yang, and G. Wang. Semantic distribution-aware contrastive adaptation for semantic segmentation. arXiv preprint arXiv:210S.05013, 2021.
[32] J. Lin, C. Song, K. He, L. Wang, and J. E. Hoperoft. Nesterov accelerated gradient and scale invariance for adversarial attacks. In Int. Conf. Learn. Represent., 2020.
[33] S. Liu, S. 7hi, F. Johns, and A. J. Davison. Bootstrapping semantic segmentation with regional contrast. In Int. Conf. Learn. Represent., 2022.
[34] W. Liu, D. Ferstl, S. Schulter, L. Zebedin, P. Fua, and C. Leistner. Domain adaptation for semantic segmentation via patch-wise contrastive learning. arXiv preprint uxXiv:2104.100:: 2021.
[35] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Int. Conf. Comput. Vis.,02.1 2021.
[36] L. Melas Kyriazi and A. K. Manrai. Pixmatch: Unsupervised domain adaptation via pixelwise consistency training. In IEFF Conf. Comput. Vis. Pattern Recog., 2021.
[37] Y. Nesterov. A method of solving a convex programming problem with convergence rate of ( Soviet Mathematics Doklady, 1983.
[38] S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games. In Eur. Conf. Comput. Vis., 2016.
[39] G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes, In IEEE Conf. Comput. Vis. Pattern Recog.. 2016.

---

[40] S. Sankaranarayanan, Y. Balaji. A. Jain, S. N. Lim, and R. Chellappa. Learning from synthetic data: Addressing domain shift for semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog,, 2018.
[41] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi semi-superrised learning with consistency and confidence. Adv. Neural Inform. Process. Syst. 2020.
[42] J.-C. Su, S. Maji, and B. Hariharan. When does self-supervision improve few-shot learning? In Eur. Conf. Comput. Vis., 2020.
[43] I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In Int. Conf. Mach. Learn., 2013.
[44] A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Adv. Neural Inform. Process. Syst. 2017.
[45] W. Tranheden, V. Olsson, J. Pinto, and L. Svensson. Dacs: Domain adaptation via cross- doms-n mixed sampling. In IEEE Winter Conf. Appl. Comput. Vis., 2021.
[46] Y.HH. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker. Learning to adapt structured oulput space for semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
[47] J. E. Van Engelen and H. H. Hoos. A survey on semi-supervised learning. Machine Learning, 2020.
[48] T.HH. Vu, Jain, M. Bucher, M. Cord, and Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[49] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Dada: Depth-aware domain adaptation in semantic segmentation. In Int, Conf. Compur. Vis., 2019.
[50] Q Wang, D. Dai, L. Hoyer, I. Van Gool, and O. Fink. Domain adaptive semantic segmentation with self-supervised depth estimation. In Int. Conf. Comput. Vis., 2021.
[51] Y. Wang, J. Peng, and Z. Zhang. Uncertaint--aware pseudo label refinery for domain adaptive semantic segmentation. In Int. Conf. Comput. Vis., 2021.
[52] Y. Wang, H. Wang. Y. Shen, Fei, W. Li, G. Jin, L. Wu, R. Zhao, and X. Le. Semi-supervised semantic segmentation using unreliable pscudo-labels. IEEE Conf. Comput. Vis. Pattern Recog,, 2022.
[53] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong. A. Farhadi, Y. Carmon, S. Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprint arXiv:2233.0545) 2022.
[54] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for scene understanding. In Eur. Conf. Comput. Vis,, 2018
[55] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Adv. Neural Inform. Process. Syst., 2021.
[56] Z. Yang, W. Bao, D. Yuan, N. H. Tran, and A. Y. Zomaya. Federated learning with nesterov accelerated gradient momentum method. arXiv preprint arXiv:2009.08716, 2020.
[57] L. Zhang and G.-J. Qi. Wep: Worst-case perturbations for semi-supervised deep learning. In IELE Conf. Comput. Vis. Pattern Recog., 2020.
[58] P. Zhang, B. Zhang, T. Zhang, D. Chen, Y. Wang, and F. Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[59] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In IEFE Conf. Comput. Vis. Pattern Recog,, 2017.
[60] Z. Zheng and Y. Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. Int. J. Comput. Vis., 2021.
[61] Q Zhou, Z. Feng, Q Gu, G. Cheng, X. Lu, J. Shi, and L. Ma. Uncertainty-aware consistency regularization for cross-domain semantic segmentation. arXiv preprint wrXiv:2004.08878, 2020.
[62] Q. Zhou, Z. Feng. Q Gu, J. Pang. G. Cheng, X. Lu, J. Shi, and L. Ma. Context-awar mixup for domain adaptive semantic segmentation. arXiv preprint arXiv:2108.033: 03557, 2021.
12

---

[63] Q. Zhou, C. Zhuang, X. Lu, and L. Ma. Domain adaptive semantic segmentation with regional contrastive consistency regularization. In Int. Conf. Multimedia and Expo, 2022.
[64] Y. Zhou, H. Xu, W. Zhang. B. Gao, and P.-.. Heng. C3-semiseg: Contrastive via cross-set learning and dynamic class-balancing. In Int. Conf. Comput. Vis., 2021.
segmentation
[65] Z. A. Zhu and L. Orecchia. Linear coupling: An ultimate unification of gradient and mirror descent. In &th Innovations Theoretical Computer Science Conference, 2017.
13


Reviews:
[{"review_id": "p8Z_5mRVZnS", "reviewer": "Reviewer_7vtH", "summary": "This paper introduces a novel extension of the mean-teacher method for semi-supervised learning. Rather than only using previous knowledge, this framework looks ahead to involve the future information to update the teacher model, which improves the quality of pseudo labels to mitigate the confirmation bias issue. Authors also propose different variants, like improved-FST, FST-D and FST-W. Comprehensive experiments demonstrate the effectiveness of the proposed pipeline. \n", "questions": "- The improved-FST is proposed since Eq(3) discards the contribution of the student weight at time t (line 153). If we let the teacher model equal to the student model at the very beginning (time t=0) and evolve the training following the EMA in equation(1), I wonder if, in this case, it could already involve the current student weights information? It may simply the computation a little. Correct me if I am wrong. \n- I am not very clear about the implementation of FST-D. Do we update the student weights when looking ahead K steps? For instance, let’s say K=3, we are looking 3 steps ahead (t+1, t+2, t+3). When we acquire the gradients at time=t+1, do we update the student model with these gradients, and then update the teacher model? If we update it, as shown in equation(5), it actually increases the number of updates to the student model(maybe another student model, but the cost of the update increases). However, in line 334, it says the number of updates remains the same. Also, in line 336, it says that it trades space for time). Even if we maintain only one ahead student model, for each time t, the original student model is updated, and we need to run this ahead student model over the next 3 steps to provide future information. In this case, I am curious about why it saves time? I am not sure whether I understand it accurately, it would be better if the authors clarify these questions. \n\nTypo: \n- Line 296: ‘of’ appears twice\n", "limitations": "Limitations and potential social impacts have been discussed in this work.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- This paper presents a novel framework that extends prevailing mean-teacher method by learning from the future. The paper is technically sound. Different variants are also designed. \n- Authors conducted extensive experiments for various tasks, like semi-supervised semantic segmentation and domain adaptation. The results on different datasets show the effectiveness of the introduced pipeline.\n- The paper is well-written and well-organized. It provides enough information for reproduction. \n- For the implementation, some parts are not clear. There are some questions posed in the questions part below. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "I2MzVapZqfm", "reviewer": "Reviewer_Sb76", "summary": "This paper presents a new self-training framework for semantic segmentation, by proposing the idea of \"learning from future\". Specifically, instead of using the current step to generate pseudo-labels and train the student, the teacher was updated with a virtual future student and supervised the following training accordingly. The proposed future-self-training (FST) framework was validated on the semantic segmentation task with extensive experimental analysis. The main contributions are the proposed FST framework and the experimental analysis.\n\n\n**Post-rebuttal**\n\nThanks to the authors for their response and the additional experiments, which addressed most of my concerns. There are some remaining concerns I would suggest the authors address in their final version:\n\n1) the motivation still sounds not that convincing to me, the \"roughly better performance\" itself as an observation does not sounds to be a convincing motivation; \n2) in the additional experiment of the newly introduced hyper-parameter, the result showed that the \"learnable\" setting performs much worse (58.87 vs. 59.81) than the \"fixed\" setting, which is not that convincing to me. Theoretically, if 0.999 is the \"optimal\" value, a learnable setting should be able to learn this value, instead of a way worse one. It would be better if the authors could provide an explanation for this in their revised version.\n\nOtherwise, the paper is interesting and could be a good contribution to the community. As a result, I would keep my original positive rating.", "questions": "It would be better if the authors could address the concerns raised above in the Weaknesses section. For example,\n\n* A more convincing clarification on the motivation\n\n* The necessity of the newly introduced hyper-parameter\n\n* The authors' observation or comments on the combination of FST-D and FST-W\n\n* Possibility of including more methods for the experimental comparison.\n\n* notations in the figure.", "limitations": "The authors discussed the limitations of their work and acceptable solution to address it, but did not mention the potential negative societal impact.\n\nA possible societal impact could be the bias within the learned model if there was also bias in the training data. The environmental impact could be another potential societal impact due to the large-scale and long-time training and the corresponding carbon emission.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n\\+ The idea of using a \"future student\" to update the current teacher and then using this future teacher to supervise the current student for the self-training framework is interesting and could potentially inspire following-up research.\n\n\\+ The authors perform thorough ablation studies, in which each of the proposed components was well analyzed. This could be useful for following-up research to well leverage the design of the proposed method.\n\n\\+ The proposed method performs better than existing alternative methods for the unsupervised domain adaptation task.\n\n\\+ The paper is generally well-written and easy to follow.\n\n\n**Weaknesses**\n\n\\- Although the idea of \"learning from future\" looks new, the motivation behind it is a bit unclear. Why learn from the future is beneficial, especially considering this \"future\" is also from the current state and the \"history\"? On the other hand, it also increases the ambiguity if the model (weights) is just cached without updating.\n\n\\- Please double-check if the variables were correctly labeled in Fig. 3. From the description, the teacher model  g_\\phi was trained on labled data x_l and the student model on unlabeled data x_u. But what was shown in the figure is the other way around.\nPlease also add more details to the caption to make it self-contained.\n\n\\- In the proposed future self-training (FST), a new hyper-parameter \\mu' was introduced, as in Eq. 4. But it was a bit unclear how to define this parameter and the necessity of it. The authors did an ablation study to test different \\mu', but the result did not suggest a way to set it and it is also a bit unclear the significance of adding this additional parameter.\nWhat if removing it or is that possible to change it to a learnable parameter?\n\n\\- Although the authors stated that the combination of FST-D and FST-W is more beyond the scope of their study, it would be better to have some preliminary investigation to see if these two types of learning schemes actually boost the learning.\n\n\\- It would be better to include more state-of-the-art methods for the experimental comparison of semi-supervised learning (Table 5).\n\n\\- L296, \"... of of ...\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1s3OqRWvXnm", "reviewer": "Reviewer_9B6x", "summary": "In this paper, the authors suggest a simple and interesting modification to the student-teacher self-training technique, where instead of having an exponential moving average teacher that is only reflecting the past states of the student model, they base it on a future hypothetical state of the model to prevent the confirmation bias on lower quality pseudo labels. This is achieved by one of the two variations that the authors suggest, referred to as deep future and wide future by either keeping a copy of the student model that is updated for multiple iterations over which the teacher model is updated as a moving average or by ensembling multiple updated student versions (with different data batches). The model is evaluated on unsupervised domain adaptations and semi-supervised learning and the empirical results show that the model improves as compared to the conventional teacher-student method.", "questions": "As stated above a major question is if the suggested model consistently delivers better performance when compared to baseline models with an equal total number of gradient descent updates.", "limitations": "The major limitation, i.e. higher computational costs for training is discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper aims to improve the student-teacher self-training scheme which is among the most frequently used semi-supervised techniques and therefore contributions here could be of interest to the community. \n* The empirical evaluations are fairly extensive. The method is compared against the baseline student-teacher model, state-of-the-art models in unsupervised domain adaptation, and multiple different variations in the ablation study. Besides, experimentations are also expanded to more modern baseline network architectures, making the conclusions more directly usable.\n* The paper is reasonably well-written and easy to follow.\n\nWeaknesses:\n* Some of the figures and comparisons might be somewhat misleading. More specifically I am referring to figures 2, and 4(a) where an iteration-to-iteration comparison is done among the proposed futuristic and baseline student-teacher methods. Even though the authors argue (at the end of 4.4) that the comparisons are fair as the two models are getting the same number of updates for the student model, actually each iteration of updating the student model implies multiple updates to a copy of the student model ($g_\\tilde{\\theta}$), which is e.g. K times more compute-extensive in the deep-future version. This is specifically questionable when iteration-to-iteration comparisons are made. \n* As another note, but on a relevant topic, I like that the authors bring some comparisons between the proposed multi-iteration student-teacher model and models that are trained for longer. But I find it insufficient for proving wrong a major question/hesitation if a significant part of the advantage is coming from training for more; specifically, we observe that for both those two experiments the model's advantage is less visible (less than a percent) while the two models are computationally roughly equivalent and therefore fairer to compare. I would have liked to see more evidence against this question.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept