Generalized Semi-Supervised Learning via Self-Supervised Feature Adaptation
Authors: Jiachen Liang
RuiBing Hou
Hong Chang
Bingpeng Ma
Shiguang Shan
Xilin CHEN
Venue: NeurIPS 2023 poster (2023)
Keywords: semi-supervised learning
self-supervised learning
TL;DR: A general semi-supervised learning framework based on self-supervised feature adaptation for feature distribution mismatch.

Abstract: Traditional semi-supervised learning (SSL) assumes that the feature distributions of labeled and unlabeled data are consistent which rarely holds in realistic scenarios. 
In this paper, we propose a novel SSL setting, where unlabeled samples are drawn from a mixed distribution that deviates from the feature distribution of labeled samples.
Under this setting, previous SSL methods tend to predict wrong pseudo-labels with the model fitted on labeled data, resulting in noise accumulation. To tackle this issue, we propose \emph{Self-Supervised Feature Adaptation} (SSFA), a generic framework for improving SSL performance when labeled and unlabeled data come from different distributions. 
SSFA decouples the prediction of pseudo-labels from the current model to improve the quality of pseudo-labels. Particularly, SSFA incorporates a self-supervised task into the SSL framework and uses it to adapt the feature extractor of the model to the unlabeled data. In this way, the extracted features better fit the distribution of unlabeled data, thereby generating high-quality pseudo-labels. Extensive experiments show that our proposed SSFA is applicable to various pseudo-label-based SSL learners and significantly improves performance in labeled, unlabeled, and even unseen distributions.

Generalized Semi-Supervised Learning via Self-Supervised Feature Adaptation
Jiachen Liang
Ruibing Hou', Hong Chang 1.2 Bingpeng Ma², Shiguang Shan Xilin Chen 1,2
Institute of Computing Technology, Chinese Academy of Sciences
University of Chinese Academy of Sciences jiachen. @ip..iiniiillliaacccc.mm (houruibing, changhong, sgshan, xlchen) @ @ict.ac.cn, bpma@ucas.ac.oom acas.ac.cn
Abstract
Traditional semi semi-superee supervised learning (SSL) assumes that the feature distributions of labeled and unlabeled data are consistent which rarely holds in realistic scenarios. In this paper, we propose novel SSL setting, where unlabeled samples are drawn from a mixed distribution that deviates from the feature distribution of labeled samples. Under this setting, previous SSL. methods tend to predict wrong pseudo- labels with the model fitted on labeled data, resulting in noise accumulation. To tackle this issue, we propose Self-Supervised Feature Adaptation (SSFA), a generic framework for improving SSL performance when labeled and unlabeled data come from different distributions. SSFA decouples the prediction of pseudo-labels from the current model to improve the quality of pseudo-labels. Particularly, SSFA incorporates a self-supervised task into the SSL framework and uses to adapt the feature extractor of the model to the unlabeled data. In this way, the extracted features better fit the distribution of unlabeled data, thereby generating high-quality pscudo-labels. Extensive experiments show that our proposed SSFA is applicable to various pseudo-label-ban SSL learners and significantly improves performance in labeled, unlabeled, and even unseen distributions.
1 Introduction
Semi-Supervised Learning (SSL) uses a small amount of labeled data and a large amount of unlabeled data to alleviate the pressure of data labeling and improve the generalization ability of the model. Traditional SSL methods [21, 43, 32, 28, 22, 30, 5, 6] usually assume that the feature distribution of unlabeled data consistent with that labeled one. However, many real scenarios, this assumption may not hold due to different data sources. When unlabeled data is sampled from distributions differ- ent from labeled data, traditional SSL algorithms will suffer from severe performance degradation, which greatly limits their practical application.
In real-world scenarios, it is quite common to observe feature distribution mismatch between labeled and unlabeled samples. On the one hand, unlabeled samples could contain various corruptions. For example, automatic driving, the annotated images used for training can hardly cover all driving scenes, and a plethora of images under varying weather and camera conditions are captured during real driving. Similarly, in medical diagnosis, individual differences and shooting conditions among patients could incur various disturbances in unlabeled data. On the other hand, unlabeled samples could contain unseen styles. For example, many tasks, the labeled samples are typically real- world photos, while unlabeled data collected from the Internet usually contain more styles that are not present in the labeled data, such as cartoons or sketches. Notably, although the distributions of these unlabeled data may differ from the labeled data, there is implicit common knowledge between them that can compensate for the diversity and quantity training data. Therefore, is crucial to enlarge the SSL scope to effectively utilize unlabeled data from different distributions.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this study, we focus on a more realistic scenario of Feature Distribution Mismatch SSL (FDM- SSL), i.e. the feature distributions of labeled and unlabeled data could be different and the feature distributions of test data could contain multiple distributions. It is generally observed that the performance of classical SSL. algorithss [22, 30, 6, 9] degrades substantially under feature distribution mismatch. Specifically, in the early stages of training, classical SSL algorithms typically utilize the current model fitted on labeled data to produce pseudo-labels for unlabeled data. However, when the distribution of unlabeled data deviates, the current model are not applicable to unlabeled data, resulting in massive incorrect pseudo-labels and aggravating confirmation bias [3]. Recently, some works [7. 19] attempt to address FDM-SSL. These approaches assume that the unlabeled feature distribution comes from a single source and they only focus on the labeled distribution, which do not always hold in real tasks. However, due to the unknown test scenarios, it is desirable to develop method to perform well on labeled, unlabeled and even unseen distributions simultaneously.
In this work, we propose generalized Self-Supervised Feature Adaptation (SSFA) framework for FDM-SSL which does not need to know the distribution of unlabeled data ahead of time. The core idea of SSFA is to decouple pseudo-label predictions from the current model to address distribution mismatch. SSFA consists of two modules, including the cemi-supervised learning module and the feature adaptation module. Inspired by [31] that the main classification task can be indirectly optimized through the auxiliary task, SSFA incorporates an auxiliary self-supervised task into the SSL module to train with the main task. In the feature adaptation module, given the current model primarily fitted on labeled data, SSFA utilizes the self-super task to update the feature extractor before making predictions the unlabeled data. After the feature extractor adapts to the unlabeled distribution, refined features can be used to generate more accurate pseudo-labels to assist SSL..
Furthermore, the standard evaluation protocol of SSL normally assumes that test samples follow the same feature distribution as labeled training data. However, this is too restricted to reflect the diversity of real-world applications, where different tasks may focus different test distributions. It is strongly desired that the SSL model can perform well across a wide range of test distributions. Therefore, in this work, we propose new evaluation protocols that involve test data from labeled, unlabeled and unseen distributions, allowing for a more comprehensive assessment of SSL performance. Extensive experiments are conducted on two types of feature distribution mismatch SSL tasks, i.e., corruption and style mismatch. The experimental results demonstrate that various escudo-label-bee SSL methods can be directly incorporated into SSFA, yielding consistent performance gains across wide range of test distributions.
2 Related work
Semi-Superrised Learning (SSL). Existing SSL methods [44, 7, 30, 41, 50, 9, 6] typically combine various commonly used techniques for semi-supervised tasks, such as consistency regularization(2) 43, 32,222] 28] and pseudo-label [22], to achieve state-of-the-a performance. For example, [6] expands the dataset by interpolating between labeled and unlabeled data. [5, 30] use the confident weak augmented view prediction results to generate pseudo-labels for the corresponding strong augmented view. [41, 50, 9, 44] improve the performance by applying adaptive thresholds instead of fixed thresholds. However, these works focus on traditional supervipe learning, which assumes that the labeled data and unlabeled data are sampled from the same distribution. Recently, some works [2. 53] address the issue of different feature distributions for labeled and unlabeled data under certain prerequisites. [2] assumes the test and unlabeled samples are drawn from the same single distribution. [53] studies Semi-Supervised Domain Generalization (SSDG), which requires multi-source partially-labeled training data. To further expand SSL to a more realistic scenario, we introduce a new Feature Distribution Mismatch SSL FEDM-SSL) setting, where the unlabeled data comes from multiple distributions and may differ from the labeled distribution. To solve the FDM-SSL, we propose Self-Supervised Feature Adaptation (SSFA), generic framework to improve the performance in labeled, unlabeled, and even unseen test distributions.
Unsupervised Domain Adaptation (UDA). UDA aims to transfer knowledge from source domain with sufficient labeled data to an unlabeled target domain through adaptive learning. The UDA methods can be roughly categorized into two categories, namely metric-based methods 35, 25, 34, 49] and adversarial methods [1, 16, 8, 24, 46, 29,37, 10, 42]. Metric-based methods focus on measuring domain discrepancy to align the feature distributions between source and unlabeled target domains. Adversarial methods [16, 47, 24] use domain classifier as a discriminator to enforce the

---

Table 1: Comparison between different problem settings.
Task
Labeled
Unlabeled
Train setting
Test distribution
traditional SSL
scarce
abundant abundant
pí(x) Pu(x)
p((x)
UDA
abundant
pi(x) + Pu(x)
Pu(z)
TTA
abundant
p((x)
Pv(
FDM-SSL
scarce
abundant
(I( # Pa(x)
pi(x), Pu (z), Punseen (x)
feature extractor to learn domain-invariant features through adversarial training. Another task similar to FDM-SSL Unsupervised Domain Expansion (UDE) [39, 51, 33, 45], which aims to maintain the model's performance the labeled domain after adapting to the unlabeled domain. The main differences between UDA, UDE and FDM-SSL are the number of labeled data during training and the distribution of the unlabeled data. In FDM-SSI, the labeled data is scarce, and the distribution of unlabeled data is not limited to a specific domain but rather a mixture of multiple domains. These challenges make UDA and UDE methods unable to be directly applied to FDM-SSL.
Time Adaptation (TTA). If the distribution of test data differs from that of training data, the model's 's performance may suffer performance degradation. TTA methods focus on improving the model's performance with a two-stage process: model should first adapt to the test samples and then make predictions of them. For example, [31] and [15] introduce an auxiliary self-learning task to adapt the model to test distribution. [38] modifies the scaling and bias parameters of BatchNorm layers based on entropy minimization. [52, 13, 12, 26] try to improve the robustness of test-time adaptation. Unlike most TTA methods which suppose the test samples come from single distribution, [40] assumes mixed test distribution that changes continuously and stably. However, these assumptions are still difficult to satisfy in real-life scenarios. In addition, similar to UDA, TTA methods generally require model trained well on the source domain. In contrast, FDM-SSL setting only requires scarce labeled data and has no restrictions on the distribution of unlabeled data.
Problem Setting
For Feature Distribution Mismatch SSL (FDM-SSL) problem, model observes a labeled set D: = [(x;, ) and an unlabeled set D,, = {1;)) دزار with N, >> N,, where 2; and u; are input images and y; is the label of x. The labeled image x, and unlabeled image u, are drawn from two different feature distributions pi (z)) and Pa (z), respectively. Note that different from previous works, in FDM-SSL setting the unlabeled samples may come from a mixture of multiple distributions rather than just one distribution: i.e. (x) wops (z) >k= waPā(r)) where wo and wi represent the weights of the labeled distribution (z) and the k-th unlabeled distribution pt respectively. The goal FDM-SSL to train model that generalizes well over a large range of varying test data distributions, including labeled, unlabeled and even distributions unseen during training (Punssen (T)). The differences between traditional SSL, UDA, TTA and FDM-SSL, are summarized in Table
The core of FDM-SSL is to enhance the utilization of unlabeled samples in the presence of feature distribution mismatch. On one hand, the shared information, such patterns and structures, in the unlabeled samples can provide effective cues for model adaptation. On the other hand, these mismatched unlabeled samples can facilitate the learning of more robust model by exposing it to wide range of data distributions.
Method
4.1 Overview
In the FDM-SSL setting, due to the feature distribution shift from pı(z) to pa(z), directly applying the SSL model fitted on labeled data to unlabeled data may lead massive inaccurate predictions on the unlabeled data, thereby aggravating confirmation bias and impairing SSL learning.
This motivates us to propose Self-Supervised Feature Adaptation (SSFA), a unified framework for FDM-SSI, which decouples the pseudo-label predictions from the current model to address distribution mismatch. As illustrated in Figure 1, SSFA consists of two modules: a semi-supervised
3

---

learning module and a feature adaptation module. The semi-supervised learning module incorporates a sauudo-labe-bbseee SSL learner with self-super auxiliary task that shares a portion of feature extractor parameters with the main classification task. [15] has pointed out that the main task can be indirectly optimized by optimizing the auxiliary task. To this end, the feature adaptation module designed to update the current model through self-supervised learning on unlabeled data, so to better match the unlabeled distribution and predict higher-quality pseudo-labels for semi-supervised learning. Therefore, by optimizing the auxiliary self-supervised task individually, the updated model
can make more accurate classification decisions in the main task.
Semi- Supervised Learning Module
0
1 R.)
0
0,
[
Inida Waights
gradent
0
Uge
0 /
0,
Expt
Feature Adaptation Module
Figure 1: The pipeline of SSFA. Let I, Uh and denote a batch of the labeled data, the weak augmentation and the strong augmentation of unlabeled data respectively, {-} represent the data stream.
4.2 Semi-Supervised Learning Module
In the Semi- Semi-Supervisee Learning Module, we introduce self- task auxiliary task. which is optimized together with the main task. As shown in Figure I, the network parameter comprises three parts: 0g for the shared encoder, 0₂ for the main task head, and 0g for the auxiliary task head. During training. SSL module optimizes a supervised loss Lz, an unsupervised loss Lu and self superviedd auxiliary loss Laux, simultaneously. Typically, given batch of labeled data {(rb. yo) ) 1== with size B and batch of unlabeled data {(w)) 1==1 ) with size pB, where μ is the ratio of unlabeled data to labeled data, Cg applies standard cross-entrops loss on labeled examples:
C2 = E H(To, yoi 0g. 0c),
where H (---) the cross-entropy function. Different SSL learners [30, 5] may design different Ly- One of the widely used is the pseudo-label loss.In particular, given the pseudo-label q for each unlabeled input ub, Lu in traditional SSL can be formulated as:
1 BB
C.
Muub 26; 0c),
where Q is the per-sample supervised loss, e.g., mean-square error [6] and cross-entropy loss [30].
Furthermore, introduce self-supervised learning task to the SSL module for joint training to optimize the parameter of the auxiliary task head 0.. To learn from labeled and unlabeled data distributions simultaneously, it is necessary use all samples from both distributions for training. Therefore, Lauz can be formulated as:
Lanx (2 + B)B
1
Σ E Og 0.)
Σ (u 0.) Σ (A
where l, denotes the self seff-super loss function, An and A, denote the weak and strong augmen- tation functions respectively. In order to maintain consistent optimization between the main task

---

and auxiliary task, it is necessary to optimize both tasks jointly. So we add Lave to semi-supervise learning module and the final object is:
LSSPA = Lar + Xuha + da CaCaux:
(4)
where Ay and Xa are hyper- denoting the relative weights of Lu and Lour respectively.
4.3 Feature Adaptation Module
Some classic SSI. approaches [30, 6, 5] directly use the outputs of the current classifier as pseudo- labels. In particular, [30] applies weak and strong augmentations to unlabeled samples and generates pseudo-labels using the model's predictions on weakly augmented unlabeled samples, i.e., q = Pm (yw (ux); 00;000) 0e), where Pm (y) u; 0) denotes the predicted class distribution of the model given unlabeled image u. However, as the classification model (0,, 0.) is mainly fitted to the labeled distribution, the prediction (wp) is usually inaccurate under distribution mismatch between the labeled and unlabeled samples.
To alleviate this problem, we design Feature Adaptation Module adapt the model the unlabeled data distribution before making predictions, thereby producing more reliable pseudo-labels for optimizing the unsupervised loss Lu. More specifically, before making pseudo-label predictions, firstly fine-tune the shared feature extractor 0g by minimizing the self-supervised auxiliary task loss on unlabeled samples:
1 BB Σ (um(:ruu);;
Capt
O.) 0.)-
Here 0g is updated 83 arg min Lapt- Notably, since excessive adaptation may lead deviation in the optimization direction and largely increase calculation costs, we only perform one-step optimization in the adaptation stage.
After self-supervised adaptation, we use (0'g 0o) to generate the updated prediction, which can be denoted as:
0z).
02)-
(6)
We convert %f to the hard "one-hot" label of and denote of as the pseudo-label for the corresponding
strongly augmented unlabeled sample A (us). After that, Or will be discarded without influencing
other model parts during training. In the end, Lu in the SSL module (Equation 4) can then be computed with cross-entropy loss
Lu
Σ ((ma((()) > ((.
4.4 Theoretical Insights
In our framework, we jointly train the model with a self-supervised task with loss function Ea and main task with loss function pm. Let H. be feasible hypothesis and D, = {(u,, v unlabeled dataset. Note that the ground-truth label y²" of u; is actually unavailable, but just used for analysis.
) }= be the
Lemma 1 ([31)) Assume that for all x. lmlx,, yi h) is differentiable, convex and B-smooth in h, and both V (x, y; b)]l. ([V hill S for all h E H. With a fixed learning rate r f every x, such that (Vem (z. y: h), Vl, h)) > e, we have Lm yi h²) em (x, y: h), where i is the updated hypothesis, namely h = h h V,, h)
(8)
Let Em (h, D.) EK lm (Ii vi: h) denote the empirical risk of the main task on D.. and (, Dw) ΣK (,(,;, h) denote the empirical risk of the self-supervised task on D,, Under the premise of Lemma and the following sufficient condition, with fixed learning rate n/:
(V Em (h, D.). E., D.))
(9)
5

---

we can get that:
Eh Du) Ern (hi Du);
(10)
where W is the updated hypothesis, namely // h 1 P. (h, D.)-
Therefore, in the smooth and convex case, the empir-
ical risk of the main task E,,, can theoretically tend to O by optimizing the empirical risk of self-supervine supervieed task F.,. Thus, in feature adaptation module, by optimizing the self-supervised loss for unlabeled sam- ples, we can indirectly optimize the main loss, thereby mitigating confirmation bias and making full use of the unlabeled samples. As above analysis, the gradient correlation plays a determining factor in the success of optimizing F... through E, in the smooth and convex case. For non-convex deep loss functions, we provide empirical evidence to show that our theoretical insights also hold. Figure plots the correlation between the gradient inner product (of the main and auxiliary tasks) and the performance improvement of the model on the test set, where each point in the figure represents the average result of set of test samples. In Figure 2, we observe that there a positive correlation between the gradient inner product and model performance improvement for non-convex loss functions the deep learning model. This phenomenon is consistent with the theoretical conclusion, that is, a stronger gradient correlation indicates higher performance improvement.
Figure 2: Scatter plot of the gradient inner product between the two tasks, and the im- provement from SSFA. We transform the x x-axis with log( (z) + 1 for clarity.
Experiments
In this section, evaluate our proposed SSFA framework on various datasets, where labeled and unlabeled data are sampled from different distributions. To this end, we consider two different distribution mismatch scenarios: image corruption and style change, in the following experiments. More experimental details and results are provided in the Appendix.
5.1 Experimental Setting
Datasets. For the image corruption experiments, we create the labeled domain by sampling images from CIFARIO0 [20], and the unlabeled domain by sampling images from a mixed dataset consisting of CIFAR 100 and CIFAR 100 with Corruptions (CIFAR 100-C). The CIFAR 100-C is constructed by applying various corruptions the original images in CIFAR 100, following the methodology used for ImageNet-C [18]. We use ten types of corruption as training corruptions while reserving the other five corruptions unseen corruptions for testing. The proportion of unlabeled samples from CIFAR100-C was controlled by the hyper-parameter ratio. For the style change experiments, we use OFFICE-31 [27] and OFFICE-HOME [36] benchmarks. Specifically, we designate one domain in each dataset as the labeled domain, while the unlabeled domain comprises either another domain or mixture of multiple domains.
Implementation Details. In corruption experiments, we use WRN-28-8 [48] as the backbone except for [6] where WRN-28-2 is used prevent training collapse. In style experiments, use ResNet-50 [17] pre-trained on ImageNNet [1 [14] as the backbone for OFFICE-31 and OFFICE-HOME. To ensure fairness, we use the same hyperparameters for different methods employed in our experiments.
Comparison Methods. We compare our method with three groups of methods: (1) Supervised method baseline to show the effectiveness of training with unlabeled data. Classical UDA methods including DANN [1] and CDAN [24]] and (3) Popular SSL methods including MixMatch [6]. ReMMixMatchhs], FixMatch [30], FM-Rot (joint training with rotation prediction task), FM-PPre (pre-trained by rotation prediction tasks), FreeMatch [41], SoftMatch [9], and Adamatch [7].
Evaluation Protocols. We evaluate the performance across labeled, unlabeled and unseen distribu- tions to verify the general applicability of our framework. Therefore, we consider three evaluation protocols: (1) Label-Domaie Evaluation (L): test samples drawn from labeled distribution, (2)

---

Table 2: Comparison of accuracy (%) for Feature Distribution Mismatch SSL on CIFARI00.
400 labeled
4000 laheled
10000 labeled
Method
ratio 0.5
ratio
1.D
ratio 1.0
ratio 1.0
L
UL
US
L
UL
US
L
UL
US
L
UL
US
Supervised
10.6
8.9
48.0
17.0
24.0
30.8
DANN [1]
11.3
9.3
7.0
114
9.3
7.0
46.7
30.8
27.6
614
42.7
17.8
CDAN [24]
11.6
11.6
10.0
47.1
31.3
28.8
62.6
39.1
MixMatch [6]
10.7
3.6
5.4
15.4
2.5
6.1
46.4
10.9
21.1
60.3
31.4
37.6
AdaMatch [7]
6.8
4.6
1.3
6.0
4.5
2.1
19.1
6.4
1.8
26.7
9.0
1.6
ReMixMaich [5]
39.6
20.6
52.7
39.2
19.1
3.1
68.4
15.8
58.1
75.5
40.2
63.2
RM-SSFA (ours)
43.1
35.1
43.0
23.1
35.1
71.0
37.9
60.8
63.5
FixMatch [30]
25.8
4.7
16.5
15.7
3.5
8.5
53.0
16.0
39.0
65.3
33.0 52.8
FM-SSEA (ours)
25.8
25.7
22.2
52.5
58.0
FrecMatch [41]
35.0
1.4
21.0
17.4
2.2
7.2
55.2
16.3
41.4
67.0
23.9
53.4
FrecM-SSFA (ours)
37.6
25.1
314
27.5
18.7
21.9
62.0
54.2
53.0
70.2
61.8
60.0
SoftMatch [9]
35.5
24.1
19.4
4.6
58.3
29.7
48.9
56.3
ASFS tours)
40.8
29.5
34.8
31.1
22.4
25.2
62.6
54.4
53.8
70.3
61.8
59.9
UnLabel-Domain Evaluation (UL): test samples are drawn from unlabeled distribution, and (3)
UnSeen-Domain Evaluation (US): test samples are drawn from unseen distribution.
5.2 Main Results
Image Corruption. Our SSFA framework is generic framework that can be easily integrated with existing pseudo-label-based SSL methods. this work, we combine SSFA with four SSL methods: FixMatch, ReMixMatch, FreeMatch and SoftMatch, denoted by FM-SSFA, RM-SSFA, FreeM- SSFA and SM-SSFA, respectively. Table 2 shows the compared results on the image corruption experiment with different numbers of labeled samples and ralio. We can observe that (1) the UDA methods, DANN and CDAN, exhibit very poor performance on labeled-domain evaluation (L). This is because UDA methods aim to adapt the model to the unlabeled domain, which sacrifices performance on the labeled domain. And the UDA methods are primarily designed to adapt to a single unlabeled distribution, making it unsuitable for FDM-SSL scenarios. In contrast, our methods largely outperform the two UDA methods on all evaluations. from significant performance degradation in the FDM-SSL setting, particularly in unlabeled-doman evaluation (UL). And this degradation becomes more severe when the number of labeled data is small and the proportion of unlabeled data with corruption is high, due to the increased feature mismatch degree. Our methods largely outperform these SSL methods, validating the effectiveness of our self-supervised feature adaptation strategy for FDM-SSL. (3) SSFA largely improves the performance of original SSL methods on all three evaluations. The gains on unlabeled-domain evaluation (UL) indicate that SSFA can generate more accurate pscudo-labels for unlabeled samples and reduce confirmation bias. Moreover, the gains on unseen-domain evaluation (US) validate that SSFA can enhance the model's 's robustness and generalization.
(2) The traditional SSL methods suffer
Style Change. We conduct experiments on OFFICE-31 and OFFICE- benchmarks to evaluate the impact of style change on SSL methods. Specifically, we evaluate SSL methods in two scenarios where unlabeled data is sampled from a single distribution and a mixed distribution. The results are summarized in Table 3. Compared to the supervised method, most SSL methods present significant performance degradation. One possible explanation is that style change causes an even greater feature distribution mismatch compared to image corruption, resulting more erroncous predictions for unlabeled data. As shown in Table 3(a), our method shows consistent performance improvement over existing UDA and SSL methods on OFFICE-31. In some cases, such as "W/A", the task itself may be relatively simple, so using only label data can already achieve high accuracy. The results on OFFICE-HOMEE are summarized in Table 3(b). To demonstrate that our method can indeed improve the accuracy of pssudd-labeld we add an evaluation protocol UU, which indicates the aseudo-labels accurucy of selected unlabeled samples. As shown Table 3bb,, compared to existing UDA and SSL methods, our approach achieves the best performance on all setups and significantly improves the accuracy of pseudo-labels on unlabeled data. The results provide experimental evidence that supports the theoretical analysis in Section 4.4.
7

---

Table 3: Comparison of accuracy (%) for Feature Distribution Mismatch SSL on OFFICE-31 and OFFICE-HOME
Results on OFFICE-31
Single Domain
Muliiple Domatns
Method
A/D
A/W
DA
W/A
A/DW
D/AW
L
UL
US
L
UL
US
UL
US
L
UL
US
L
UL
UL
supervised
65.9
57.1
52.0
65.9
47.8
61.0
93,0
56.8
84.9
93.6
56.2
92.4
68.0
51.9
93,3
60.7
DANN [i]
53.9
17.9
20.1
15.3
7.2
4.6
74.7
4,8
16.7
2.5
2.9
4.2
28.1
7.6
4.7
288
[24]
2.6
2,9
7.7
FixMatch [30]
58.4
46.9
53.3
56.3
9.5
19.3
82.8
4.7
50.6
78.3
9.4
62.6
55.0
48.8
80.2
40.3
FM-Rot
60.0
53.1
57.1
54.6
27.4
19.3
70.8
29.5
50.6
75.9
17.9
66.9
54.0
40.1
63.0
26.9
FM-SSFA
67.5
61.7
63.0
66.4
56.5
62.5
93.0
89.3
85.9
93.0
56.4
92.8
69.1
60.8
92.4
61.8
(b) Results on OFFICE-HOME
Method
A/CPR
C/APR
P/ACR
A/ACPR
CIACPR
R/ACPR
1.
UL
LU
L
UL
UU
L
UL
UU
L
UL
UL
1.
UL
UU
I. UL. UU
supervised
48.4
39.9
40.9
31,2
68.4
36.5
49.3
43.0
41.0
36.3
61.6
46.8
DANN [1]
45.0
37.8
31.8
7.7
53.9
25.6
49.3
42.0
30.5
20.2
44,5
32.1
CDAN
30.4
FixMatch [30]
27.2
17.7
19.3
28.1
19.6
19.4
S5.6
24.9
30.6
324
23.0
31.
36.9
30.6
36.0
42.2
31.5
35.4
37.9
24,9
FM-Rot
1.7
2.1
0.0
33.
20.1
0.7
43,4
14.6
17.2
25.6
17.7
213
37.3
31.
33.2
45.6
34.7
42.3
FM-SSEA
50.7
45.1
37.6
41.9
70.5
55.0
44.7
41.7
64.8
52.7
80.3
5.3 Feature Visualization
Figure visualizes the domain-level features generated by SSL models with/without SSFA respec- tively. In Figure 3 (a), the vanilla FixMatch model maps labeled and unlabeled samples to different clusters the feature spaces without fusing samples from different domains. Conversely, Figure 3 (b) shows that our FM-SSFA model can effectively fuse these samples. We further compute the A-distance metric [4] to measure the distributional divergence between the labeled and unlabeled distributions. The A-distance of FM-SSFA (0.91) is lower than that of FixMatch (1.10), verifying the implicit distribution alignment of SSFA.
Additionally, Figure visualizes the class-level features generated by SSL models with/without SSFA respectively. In Figure (a), FixMatch fails distinguish samples from different classes. In contrast, FM-SSFA can separate samples from different categories well, as shown in Figure (b). This result indicates that SSFA can help extract discriminative features.
(a) FixMatch
(b) FM-SSFA
labei
urlsbel0
FlaMatch
FM-SSFA
unlxbel1
unlabel
unlabel?
unlabe19
Figure 3: Visualization of domain-level features using different methods, where "label" represents the labeled data drawn from the labeled domain, and "unlabel0" to "unlabel9" represent the unlabeled data drawn from ten unlabeled domains respectively.
Figure 4: Visualization of class- level features using different meth- ods.
5.4 Ablation Study
Combined with different self-supervised tasks. Table compares different self-supervised tasks combined with our SSFA framework on the image corruption experiment. We employ different self-supervised losses corresponding to the rotation prediction task, contrastive learning task, and

---

Table Combined with different self sulervueee tasks. "Rot", "SimCIr", and "EM" represent the rotation prediction task, contrastive learning task [11] and entropy minimization task [38] respectively.
400 laheled
4000 labeled 10000 labeled
Method
ratio
ratio 1.0
ratio 1.0
UL
L
UL
US
UL US
UL
FixMatcch|301
25.8 4.7 16.5 15.7 3.5 8.5
$3.0
16.0 39.0 65.3 33.0 52.8
FM. SSFA (Rat)
23.2
25.7
FM-SSFA (SimClr)
22.8
13.7
17.4
19.2
13.6
14,1
45.3438
45.1
66.2
53.6 52.9
FM-SSFA (EM)
22.9 14.5
17.2
18.0
15.9
14.6
44.2
44.7
66.7
52.5 52.6
entropy minimization task, which are the cross entropy loss, the contrastive loss from SimCLR and the entropy loss, respectively. As shown, different self-supervised tasks can bring significant performance gains. And the rotation prediction task leads to the largest improvements compared to the other two self-supervised tasks. This can be attributed to the more optimal parameters and the direct supervision signals in rotation prediction. In the experiments, we employ the rotation prediction task as the default self-supervvsed task.
The effectiveness of Feature Adaptation Module. To evaluate the effectiveness of the Feature Adaptation Module in the SSFA framework, we compare FM-SSFA with baseline method FM-Rot, where we remove the Feature Adaptation Module and only add the self-supervieed rotation prediction task. As shown in Table 3()) and (b), FM-SSFA largely outperforms FM-Rot on OFFICE-31 I and OFFICE-HOME, especially in the UL evaluation metric. The superiority of FM-SSFA highlights that the Feature Adaptation Module helps the model to adapt to unlabeled samples from different distributions, resulting in better generalization FM-Rot only marginally better than FixMatch and, in some cases, may even perform worse. These results suggest that simply integrating the self-supervised task into SSL methods only brings limited performance gains and may even be detrimental in some scenarios.
unlabeled domain. Moreover, we can observe the
Small distribution shift between labeled and un- labeled data. To demonstrate the robustness of our proposed method, we evaluate SSFA on scenarios where there is a small distribution shift between labeled and unlabeled data. Specially, we conduct experiments on two setups: ratio=0.0 (degrade into traditional SSL) and rafio=0.1. Table shows our method can still bring significant improvements over the baseline. This indicates that our proposed SSFA framework is robust to various SSL scenarios under a low noise ratio.
Table 5: The performance of SSFA when the distribution shift between labeled and unlabeled data is small CIFAR100.
ratio
ratio 0.1
Method
L/UL US
L
UL US
FixMatch [30] 33,3 25.8 31.7 10.5 24.8
FM-SSFA
41.3 33.0 41.2 15.8 32.9
Robustness to confident threshold Figure 5 illustrates the impact of confident threshold 7 for different SSL models on OFFICE-31. As shown in Figure 5, the vanilla FixMatch model is highly sensitive to the values of 7. For example, when 7 is set to 0.95, FixMatch achieves relatively high performance with confident enough pseudo-labels. When is lowered to 0.85, the performance of FixMatch on UL and UU degrades significantly, indicating that the model has deteriorated by too many wrong pseudo-labels. In contrast, FM-SSFA is more robust to different values of
Figure 5: The impact of for different SSL models on OFFICE-31 ("A/W" task).
The number of shared layers between auxiliary and main task. Table analyzes the impact of different numbers of shared layers in the shared feature extractor of the auxiliary and main task. As shown, the performance difference negligible between sharing and 3 layers in the feature extractor, while significant performance degradation happens if we share all layers (4 layers) of the feature extractor. We argue that this is because too many shared parameters of the feature extractor may lead to aver-adaptation of the feature extractor and compromise of the main task, thus the predictions of

---

Table 6: The impact of shared layers between auxiliary and main task on OFFICE-HOME. "X layers" denotes the number of shared layers for the feature extractor.
Mathod
ACPRR
C/APR
PIACR
R/ACP
UU
UU
FM SSEA (2 layers) 48.9 39,9 70.8
46.0
39,3
69.0
70.3
42.0
67.3
61.0
40.8
67.0
FM-SSFA layers)
47.5 39.8
71.0
45.4
39.0
77.6
72.3
45.
76.8 59.5
74.4
63.4
43,9
FM-SSFA (4 layers) 44,4 39.1
57.1
1.3
244
0.0
57.9
33.4
40.3
28.7
54.3
pscudo-labels may be more erroneous. In the experiments, we set the number of shared layers to 2 by default.
6 Conclusion
In this paper, we focus on a realistic SSL setting, FDM-SSL, involving a mismatch between the labeled and unlabeled distributions, complex mixed unlabeled distributions and widely unknown test distributions. The primary challenge lies the scarcity of labeled data and the potential presence of mixed distributions within the unlabeled data. To address this challenge, framework SSFA, which introduces self-supervised task to adapt the feature extractor to the unlabeled distribution. By incorporating this self-supervised adaptation, the model can improve the accuracy of pseudo-labels to alleviate confirmation bias, thereby enhancing the generalization and robustness of the SSL model under distribution mismatch.
propose generalized
Broader Impacts and Limitations. The new problem setting takes into account the model's generalization across different distributions, which is essential for expanding the application of SSL methods in real-world scenarios. Additionally, SSFA serves as simple yet effective framework that can be seamlessly integrated with any pseudo-label-basee SSL methods, enhancing the overall performance and adaptability of these models. However, the performance of SSFA is affected by the shared parameters between the main task and the auxiliary task. We hope that SSFA can attract more future attention to explore the effectiveness of feature adaptation with the self-supervised task.
Acknowledgments
This work is partially supported by National Key R&D Program of China no. 2021ZD01 19011, National Natural Science Foundation of China (NSFC): 61976203, 62276246 and 62306301, and National Postdoctoral Program for Innovative Talents under Grant BX20220310.
References
[1] Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain- adverssraa neural networks. arXiv, 2014.
[2] Gholamali Aminian, Mahed Abroshan, Mohammad Mahdi Khalili, Laura Toni, and Miguel Rodrigues. An wnwformatioo-thhortiiie approach to semi-supervised learning under covariate- shift. In AISTATS, 2022.
[3] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias deep semi-supervised learning. In IJCNN, 2020.
[4] Shai Ben-David. John Blitzer, Koby Crammer, and Fermando Pereira. Analysis of representations for domain adaptation. In NeurIPS, 2006.
[5] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In ICIR, 2020.
[6] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019.
10

---

[7] David Berthelot. Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alexey Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. In ICLR. 2022.
[8] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In CVPR,
2017.
[9] Hao Chen, Ran Tao, Yue Fan, Yidong Wang. Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality tradeoff in semi- supervised learning. In ICLR, 2023.
[10] Minghao Chen, Shuai Zhao, Haifeng Liu, and Deng Cai. Adversarial-learned loss for domain adaptation. In AAAI, 2020.
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[12] Francesco Croce, Sven Gowal, Thomas Brunner, Evan Shelhamer, Matthias Hein, and Taylan Cemgil. Evaluating the adversarial robustness of adaptive test-time defenses. In ICML, 2022.
[13] Mohammad Zalbagi Darestani, Jiayu Liu, and Reinhard Heckel. Test-time training can close the natural distribution shift performance gap in deep learning based compressed sensing. In ICML, 2022.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[15] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A. Efros. Test-time training with masked autoencoders. In NeurlPS, 2022.
[16] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain aamaainad training of neural networks. The journal of machine learning research, 17:2096 17200962000,, 2016.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[18] Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
[19] Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In NeurlPS, 2021.
[20] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Totonto, 2009.
[21] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR. 2017.
[22] D.-H. Lee. Pseudo-label The simple and efficient semi-supervised learning method for deep neural networks. In JCML,
[23] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.
[24] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. NeurIPS, 2018
[25) Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu. Transfer feature learning with joint distribution adaptation. In ICCV, 2013.
[26] Prabhu Teja and François Fleuret. Test time adaptation through perturbation robustness. In NewrIPS, 2021.
11

---

[27] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, 2010.
[28] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic trans- formations and perturbations for deep semi-supervised learning. In NeurIPS, 2016.
[29] Swami Sankaranarayanan, Yogesh Balaji. Carlos Domingo Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In CVPR, 2018.
[30] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In NeurlPS, 2020.
[31] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
[32] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averanl consistency targets improve semi-supervised deep learning results. In ICLR (Workshop), 2017.
[33] Kaibin Tian, Qijie Wei, and Xirong Li. Co-teaching for unsupervised domain adaptation and expansion. arXiv, 2022.
[34] Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultancous deep transfer across domains and tasks. In ICCV, 2015.
[35] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv, 2014.
[36] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.
Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
[37] Riccardo Volpi, Pietro Morerio, Silvio Savarese, and Vittorio Murino. Adversarial feature augmentation for unsupervised domain adaptation. In CVPR, 2018.
[38] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
[39] Jie Wang, Kaibin Tian, Dayong Ding, Gang Yang, and Xirong Li. Unsupervised domain expan- sion for visual categorization. ACM Transactions Multimedia Computing, Communications, and Applications, 17:1-24, 2021.
[40] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In CVPR, 2022.
[41] Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj. Bernt Schiele, and Xing Xie. Freematch: Self Self-adaptive thresholding for supeevied learning. In ICLR, 2023.
[42] Guoqiang Wei, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation. In CVPR, 2021.
[43] Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In NeurIPS, 2020.
[44] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In ICML, 2021.
[45] Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In ICCV, 2021.
[46] C. Yu, J. Wang. Y. Chen, and M. Huang. Transfer leamning with dynamic adversarial adaptation network. In ICDM, 2020.
[47] Chaohui Yu, Jindong Wang, Yiqiang Chen, and Meiyu Huang. Transfer learning with dynamic adversarial adaptation network. In ICDM, 2019.
12

---

[48] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[49] Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Plate Central moment discrepancy (CMD) for domain-invariant representation learning. In ICLR, 2017.
[50] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In NeurIPS, 2021.
[51] Jing Zhang, Wanqing Li, Chang Tang. Philip Ogunbona, et al. Unsupervised domain expansion from multiple sources, arXiv, 2020.
[52] Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO: test time robustness via adaptation and augmentation. In NeurlPS, 2022.
[53] Kaiyang Zhou, Chen Change Loy, and Ziwei Liu. Semi-supervised domain generalization with stochastic stylematch. International Journal of Computer Vision, 131:2377-2387, 2023.
13


Reviews:
[{"review_id": "8wFxHv5L2x", "reviewer": "Reviewer_LyxH", "summary": "This paper focuses on Feature Distribution Mismatch SSL (FDM-SSL), which considers a mismatch between the labeled and unlabeled data. In this task setting, the challenges include the scarcity of labeled data and the mixed distributions of the unlabeled data. The paper proposes a framework named Self-Supervised Feature Adaptation (SSFA), which adapts the feature extractor to the unlabeled distribution. Experiments show that SSFA can improve SSL performance on labeled, unlabeled, and unseen data.", "questions": "See Weaknesses.", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper studies a new SSL problem named Feature Distribution Mismatch SSL. The presented SSFA method is simple and effective to address this problem.\n\n2. Extensive experiments have been conducted to evaluate the performance of SSFA.\n\n3. The paper is well written and presented. For instance, Table 2 is clear, it reveals the difference between task settings.", "weaknesses": "1. The proposed SSFA method is built upon existing SSL and unsupervised domain adaptation (UDA) methods. In SSFA, the SSL module and Feature Adaptation Module are inherited from the benchmark SSL and UDA methods, respectively. The optimization objectives $L_x$, $L_u$, and $L_{aux}$ are widely used in SSL methods. The major novelty is the Feature Adaptation Module, which uses the unlabeled data as input to generate pseudo labels $q_b'$ instead of the original pseudo labels $q_b$ generated from the labeled data. I acknowledge that this work presents some novelties in proposing and addressing the challenges in domain-shifted SSL. However, the method itself is straightforward in my opinion.\n\n2. The assumptions of Lemma 1 are somewhat strong, i.e., the loss is expected to be convex and smooth, the learning rate is fixed to $\\frac{\\epsilon}{\\beta G^2}$. In addition, the result of Lemma 1 is counter-intuitive --- 'The empirical risk of the main task can be theoretically reduced to 0 by optimizing the empirical risk of self-supervised task.' Due to these aspects, I kindly doubt that this theoretical result could work well in practice. And, the derivation only guarantees that the empirical risk is decreasing along with gradient descent, but not 'reduced to 0'. Please check this. \n\n3. I am a bit confused about the SSFA's theoretical performance on 'unseen data'. In Sections Introduction and Experiments, the authors claimed that SSFA can significantly improve SSL performance on unseen data. However, the method only adapts models to the *unlabeled* data during training. Why does SSFA improve the performance on unseen data? More theoretical evidence is encouraged to be presented to better support this claim.\n\n4. I suggest the authors show some visualizations of the gap between labeled and unlabeled data, e.g., what are the image corruptions in the experiments? What's the style change from OFFICE-31 dataset to OFFICE-HOME dataset? I find some related details in Appendix's texts. However, it would be more friendly to show some visualizations of the data gap, since domain shifting is the main challenge studied by this work.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments! We see that your main concerns are novelty, more explanations and some details. We answer your questions point by point.\n\n**W1. I acknowledge that this work presents some novelties in proposing and addressing the challenges in domain-shifted SSL. However, the method itself is straightforward in my opinion. However, the method itself is straightforward in my opinion.** \n\nThanks for acknowledging the novelty of our work. We believe that a straightforward method offers several advantages, including simplified implementations, facilitating understanding, and serving as a solid baseline for further research in Feature Distribution Mismatch SSL. \n\nPlease refer to our reply to Q1 in the global response for more details.\n\n**W2.   The result of Lemma 1 is counter-intuitive. The assumptions of Lemma 1 are somewhat strong and I kindly doubt that this theoretical result could work well in practice. Lemma 1 only guarantees that the empirical risk is decreasing along with gradient descent, but not 'reduced to 0'.**\n\nThank you for your feedback on issues related to Lemma 1. We will divide the answer to W2 into three parts.\n\n **2.1.The results of Lemma 1 is counter-intuitive --- 'The empirical risk of the main task can be theoretically reduced to 0 by optimizing the empirical risk of self-supervised task'.**\n\nPlease refer to our reply to Q2 in the general response.\n\n**2.2 The assumptions of Lemma 1 are somewhat strong, potentially impeding the effectiveness of the theoretical result in practice.**\n\nPlease refer to our reply to Q3 in the general response.\n\n**2.3 Lemma 1 only guarantees that the empirical risk is decreasing along with gradient descent, but not 'reduced to 0'.**\n\nAccording to the toy example, it is possible to indirectly reduce the main task loss to 0 by minimizing self-supervised task loss. In addition, according to Lemma 1, assuming that the loss is expected to be convex and smooth, as the gradient of self-supervised task loss decreases, the empirical risk of the main task will gradually decrease. Therefore, if we choose a proper training strategy to optimize the self-supervised task, the empirical risk of the main task can tend to 0.\n\n**W3. However, the method only adapts models to the unlabeled data during training. Why does SSFA improve the performance on unseen data?** \n\nWe explain the reason why SSFA improves the performance on unseen data from two aspects.\n\n   + **From a perspective of method.** In our SSFA, unlabeled samples are aligned with the feature distribution of labeled samples through the feature adaptation module, thereby improving the model's prediction accuracy for distribution-mismatch unlabeled samples. By making correct predictions on samples from various domains, the feature extractor becomes adept at mapping data from different domains to the same feature space, gradually eliminating the impact of domain divergence.  As a result, the model learns to capture domain-invariant features as two feature space becomes indistinguishable. \n\n   + **From a perspective of experiment.** As the domain number exposed by the model increases during training, the feature extractor tends to map more domains to the same feature space. It enhances the model's potential for generalizing to unseen domains. Therefore, the model has a higher improvement compared to the baseline on unseen data. To visually illustrate this, we visualize the domain-level features generated by SSL models with/without SSFA respectively. As Figure 1 in the PDF of the global response shows, FixMatch maps samples from different domains (1 labeled, 10  unlabeled and 5 unseen domains) to different clusters in the feature spaces. Differently, our FM-SSFA model can effectively fuse these samples. The fusion of features in the unseen domains and the seen domains indicates that our SSFA has a good generalization ability to map different domain features to the same feature space.\n\n**W4. I suggest the authors show some visualizations of the gap between labeled and unlabeled data, e.g., what are the image corruptions in the experiments?** \n\n   Thanks for your suggestions. We will add relevant visualization images in our paper to further illustrate the feature distribution gap between labeled and unlabeled data. We show the visualizations of the gap caused by image corruption and style change in Figure 2 and Figure 3 in the PDF of the global response respectively."}, {"review_id": "LZizbPrv2B", "reviewer": "Reviewer_WadC", "summary": "The authors propose an interesting and novel semi-supervised learning method with auxiliary self-supervised feature adaptation for addressing issues with heterogeneity in labelled and unlabelled data. In fact, the authors focus on heterogeneity across domains, which is not something traditionally considered in the SSL literature, but is more in the field of unsupervised domain adaptation. The authors combines ideas of feature adaptation from UDA with consistency regularization and pseudo-labelling from SSL to bridge the two fields. The results presented seem much better than the baseline methods.", "questions": "1. The Feature Adaptation Module seems like 1-directional (unlabelled to labelled) and 1step transformation, as the unlabelled are aligned to the labelled. Have the authors tried using more steps in the optimization to allow more adaptation, or does that rely on the loss and step calibration?\n2. The authors show the effect of different fixed thresholds in Eq 7 in Figure 5. Have the authors tried an adaptive threshold as in FlexMatch?\n3. While the authors have compared SSFA to more recent and state-of-the-art models in SSL such as FreeMatch and SoftMatch, the authors have not compared against the arguably the state-of-the-art in UDA such PMTrans. How does SSFA compare against PMTrans or any of the more recent leaders the Domain Adaptation leaderboard on Office-31 and Office-Home?\n4. Moreover, SSFA is especially impressive for low number of labelled data against UDA methods. How does it do in low labelled data settings against more recent UDA methods?", "limitations": "The authors have addressed some limitations of their method", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Table 1 is useful and sets the tone of the paper showing the problem settings and constraints of each type of approach.\n2. The performance metrics reported in Table 2 and 3 are very impressive compared to baselines. ", "weaknesses": "1. However, the caveat is that the baseline methods cannot do both SSL and UDA, which makes comparisons of performance metrics harder. However, SSFA does seem to be a novel method bridging the two fields. \n\nMinor points:\n1. Typo in Table 1: traditional\n2. Might be useful to point out more forcefully Eq 2 is traditional in SSL, but Eq 7 is actually used, especially as SSFA Algorithm with equation references is in Supplement. Maybe it would be useful to not call both Eq 2 and Eq 7, L_u.\n3. How was the cluster representation in Figure 4 made? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments! We address all of your concerns point by point as below.\n\n**W1. However, the caveat is that the baseline methods cannot do both SSL and UDA, which makes comparisons of performance metrics harder. However, SSFA does seem to be a novel method bridging the two fields.**\n\nCurrently, research on FDM-SSL remains limited.   Considering the similarities between the task setting of FDM-SSL and more extensively studied SSL and UDA, we also conducted experiments on SSL and UDA methods to evaluate their ability in addressing FDM-SSL in Table 2 and Table 3 of our paper. As you pointed out, our SSFA framework has the potential to serve as a universal baseline, bridging SSL and UDA in the future.\n\n**W2. Minor points:**\n\n   1. **Typo in Table 1: traditional**\n\n       Thanks for pointing out this issue.   We will correct this typo in the final version.\n\n   2. **Might be useful to point out more forcefully Eq 2 is traditional in SSL, but Eq 7 is actually used, especially as SSFA Algorithm with equation references is in Supplement. Maybe it would be useful to not call both Eq 2 and Eq 7, L_u.**\n\n      Thanks for your suggestions. We will make modifications based on the suggestions provided by the reviewer.\n\n   3. **How was the cluster representation in Figure 4 made?**\n\n      In Figure 4, we randomly select four categories and extract features from labeled and unlabeled samples using the model's feature extractor. Then, we use t-SNE to reduce the dimensionality and visualize these features. Figure 4 (a) shows that the features of different categories are mixed together, indicating that FixMatch lacks the ability to distinguish different categories under FDM-SSL settings. In Figure 4 (b), the categories are clearly distinguished without being affected by different domains, indicating that the SSFA module can greatly alleviate the classification error caused by feature distribution mismatch.\n \n\n**Q1. Have the authors tried using more steps in the optimization to allow more adaptation, or does that rely on the loss and step calibration?**\n\n   Following your suggestion, we further explored multi-step adaptation during the optimization process on CIFAR100 benchmark (with 400 labeled data and the $ratio$ is 1.0).  As shown in the table below, only one step of adaptation can bring significant improvements over the baseline. And as the number of adaptation steps increases, the performance will be further improved, but with greater computational costs. Considering the trade-off between accuracy and computational cost, we perform one-step optimization in this paper.\n\n   | Method             | L     | UL    | US    |\n   | ------------------ | ----- | ----- | ----- |\n   | FM                 | 15.7  | 3.5   | 8.5   |\n   | FM-SSFA (1 step)   | 25.7  | **22.2**  | 22.5  |\n   | FM-SSFA (5 steps)  | 26.2  | 14.9  | 16.0 |\n   | FM-SSFA (10 steps) | **41.1** | 18.7 | **33.7** |\n   \n**Q2. Have the authors tried an adaptive threshold as in FlexMatch?**\n\nIn our paper, we have compared SSFA to more recent and state-of-the-art SSL algorithms such as FreeMatch and SoftMatch. Both algorithms use self-adaptive thresholds similar to (but more complex than) FlexMatch. For further explanation,  we conduct FlexMatch in FDM-SSL and give the results on CIFAR100 benchmark (with 400 labeled data and the $ratio$ is 0.5) as follows. It can be seen that SSFA can further improve the performance of FlexMatch, especially in the distribution of unlabeled and unseen data. \n\n   | Method         | L    | UL   | US   |\n   | -------------- | ---- | ---- | ---- |\n   | FlexMatch      | 35.8 | 2.2  | 23.6 |\n   | FlexMatch-SSFA | **38.3** |**28.4** | **31.6** |\n\n**Q3. How does SSFA compare against PMTrans or any of the more recent leaders the Domain Adaptation leaderboard on Office-31 and Office-Home?**\n\nFollowing your suggestion, we evaluate PMTrans under FDM-SSL setting on OFFICE-HOME benchmark. As shown in the table below, PMTrans achieves very poor performance. In fact, we have also found that previous UDA methods (such as DANN and CDAN) may crash during training in some cases. We have mentioned the relevant reasons in the paper (lines 244 - 248). In addition, due to the fact that the PMTrans uses a more complex Swin-Transformer as the backbone, while our method uses Resnet50 as the backbone, PMTrans requires longer training time and more memory usage compared to SSFA.\n\n   | Method   | A/ACPR |      | C/ACPR |      | P/ACPR |      | R/ACPR |      |\n   | -------- | ------ | ---- | ------ | ---- | ------ | ---- | ------ | ---- |\n   |          | L      | UL   | L      | UL   | L      | UL   | L      | UL   |\n   | PMTrans  | 8.6    | 2.3  | 17.4   | 10.5 | 23.8   | 12.7 | 15.3   | 6.6  |\n   | FixMatch | 32.4   | 23.0 | 36.9   | 30.6 | 52.9   | 32.6 | 42.2   | 31.5 |\n   | FM-SSFA  | **55.0**   | **45.5** | **44.7**   | **41.7** |**71.8**   | **52.6** | **64.8**   | **52.7** |\n\n**Q4. Moreover, SSFA is especially impressive for low number of labelled data against UDA methods. How does it do in low labelled data settings against more recent UDA methods?**\n\nFollowing your suggestion, we evaluate the recent UDA method PMTrans and our SSFA in low labeled data settings (1 labeled sample per class) on OFFICE-HOME benchmark. It can be seen that in low labeled data settings, the performances of PMTrans and FixMatch are poor. However, with our SSFA, FixMatch has a significant improvement in performance.\n   \n   | Method   | A/ACPR |      | C/ACPR |      | P/ACPR |      |\n   | -------- | ------ | ---- | ------ | ---- | ------ | ---- |\n   |          | L      | UL   | L      | UL   | L      | UL   |\n   | PMTrans  | 5.8    | 1.7  | 6.6    | 3.3  | 10.4   | 6.3  |\n   | FixMatch | 4.7    | 3.5  | 10.0   | 6.6  | 6.3    | 5.7  |\n   | FM-SSFA  | **19.1**   | **20.5** | **18.4**   | **16.2** | **29.3**   | **20.0** |\n\n"}, {"review_id": "5GRNrojfnx", "reviewer": "Reviewer_wRdb", "summary": "In this work, the authors propose a generalized Self-Supervised Feature Adaptation (SSFA) framework for FDM-SSL (Feature Distribution Mismatched Semi-Supervised Learning) that does not require prior knowledge of the distribution of unlabeled data. The SSFA framework aims to address distribution mismatch by decoupling pseudo-label predictions from the current model. It consists of two modules: the semi-supervised learning module and the feature adaptation module. The authors draw inspiration from previous work on auxiliary tasks and incorporate an auxiliary self-supervised task into the SSL module to train alongside the main task. In the feature adaptation module, the current model, primarily trained on labeled data, is updated by utilizing the self-supervised task to adapt the feature extractor before making predictions on unlabeled data. This adaptation allows for the generation of more accurate pseudo-labels, which in turn assist the SSL process.", "questions": "It was not very clear from the text about how the feature adaptation model is being trained. The module is being trained for one iteration and then is being discarded. This part is confusing. Is the module trained one unlabeled sample at a time or using all unlabeled samples? If the module needs all unlabeled samples to be trained this may not reflect a real-world use case scenario and will limit the applicability of the approach. Figure 1 needs to be updated to reflect these important details. \n\nWhat kind of self-supervised loss is used in equation 5? \n\nPlease define G in Lemma 1.  What does subscript m in the loss function in Lemma 1 indicates? It was not previously used in defining loss function.\n\nPlease correct the typos in Table 1. \"Traditioal\",   Abundant/Scarce could replace Plenty/Lack. \n\n\n-------------\nYour responses have clarified some of the concerns I had with Lemma 1.  In particular, I acknowledge your explanations regarding the training process of the feature adaptation module, and the flexible nature of the self-supervised loss function. I am inclined to update my initial score to reflect these additional insights your rebuttal offered. Thanks.", "limitations": "Conclusions allude to some algorithmic limitations yet do not discuss societal impact. No negative societal impact beyond what is already present in generic machine learning algorithms has been observed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Addition of the feature adaptation module to adapt the model (trained on labeled data) to the unlabeled data distribution before generating pseudo labels seems to help with the distribution shift problem between labeled and unlabeled data. \n\n- Extensive empirical evidence provided to show that proposed SSFA method significantly improves classification accuracy on CIFAR100 and OfficeHome and Office31 datasets. Domain-level and class-level feature visualization also suggest successful feature adaptation across different domains and learning of more distinguishable representations across different classes. Ablations studies investigating the effect of different self-supervised tasks, number of shared layers between auxiliary and main task, and the sensivity of the method to different confidence thresholds are also useful.", "weaknesses": "- Lemma 1 is fine but convexity is a very strong assumption, which does not hold true in real-world deep loss functions. Some empirical evidence is provided, but it is somewhat counterintuitive to think that minimizing self-supervised empirical loss will also minimize supervised empirical loss. \n\n- Using weakly augmented samples for a supervised loss and strongly augmented samples for an unsupervised loss is not novel and first introduced in the FixMatch paper. Technical novelty is somewhat incremental though feature adaptation module can still be considered a significant improvement. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! Below are the responses to your comments.\n\n**W1. Lemma 1 is fine but convexity is a very strong assumption, which does not hold true in real-world deep loss functions. Some empirical evidence is provided, but it is somewhat counterintuitive to think that minimizing self-supervised empirical loss will also minimize supervised empirical loss.**\n\nThank you for your comments on lemma1. We will answer the questions in two parts. \n\n**1.1. It is somewhat counterintuitive to think that minimizing self-supervised empirical loss will also minimize supervised empirical loss.** \n\nPlease refer to our reply to Q2 in the general response. We provide a toy example to explain this question.\n\n**1.2. Lemma 1 is fine but convexity is a very strong assumption, which does not hold true in real-world deep loss functions.**\n\nPlease refer to our reply to Q3 in the general response. We provide a theoretical analysis and empirical evidence to prove its feasibility in the real-world.\n\n**W2. Using weakly augmented samples for a supervised loss and strongly augmented samples for an unsupervised loss is not novel and first introduced in the FixMatch paper. Technical novelty is somewhat incremental though feature adaptation module can still be considered a significant improvement.**\n\n   Thank you for your approval of method novelty. The consistency loss introduced by FixMatch is not the primary focus of our method. The key innovation is to propose a unified framework SSFA for FDM-SSL.  In our paper, FixMatch serves as an illustrative example to showcase the working pipeline of SSFA, but FixMatch can be replaced by any other pseudo-label-based semi-supervised method.\n   \n   As your mentioned, our innovation mainly lies in the proposed feature adaptation module, which introduces a self-supervised task to achieve feature alignment. This feature adaptation module offers flexibility, allowing multiple self-supervised tasks such as rotation prediction tasks and entropy minimization tasks. Overall, our work focuses on proposing a universal SSFA framework to solve the new and more realistic FDM-SSL scenario.\n\n Please refer to our reply to Q1 in the general response for more details.\n\n\n**Q1. It was not very clear from the text about how the feature adaptation model is being trained. The module is being trained for one iteration and then is being discarded. This part is confusing. Is the module trained one unlabeled sample at a time or using all unlabeled samples? If the module needs all unlabeled samples to be trained this may not reflect a real-world use case scenario and will limit the applicability of the approach. Figure 1 needs to be updated to reflect these important details.**\n\nWe are not using all unlabeled data for updating the feature adaptation module. Instead, we only utilize the unlabeled data from the current batch. Typically, the batch size is not excessively large (e.g., set to 64 in our experiment). In particular, we update the feature extractor $\\theta_ g$ to $\\theta_ g'$  using the unlabeled data from the current batch. Afterwards, the updated feature extractor $\\theta_ g'$ is used to extract new features for the unlabeled data to generate pseudo-labels. Notably,  $\\theta_ g'$ did not have any other impact on subsequent model training. We will add important details to Figure 1 in the revised version.\n   \n**Q2. What kind of self-supervised loss is used in equation 5?**\n\n   The self-supervised loss can be chosen flexibly depending on the used self-supervised task. In our paper, we employ different self-supervised losses corresponding to the rotation prediction task, contrastive learning task, and entropy minimization task, which are the cross entropy loss, the contrastive loss from SimCLR  and the entropy loss, respectively.\n\n**Q3. Please define G in Lemma 1. What does subscript m in the loss function in Lemma 1 indicates? It was not previously used in defining loss function.**\n\n   G is a constant, denoting the upper bound of $\\lVert \\nabla l_s(x,y;h)\\rVert$. $l_m$ is the loss function of the main task.\n\n**Q4. Please correct the typos in Table 1. \"Traditioal\", Abundant/Scarce could replace Plenty/Lack.**\n\n   Thanks for your suggestions. We will modify the paper based on the suggestions.\n\n**Limitations. Conclusions allude to some algorithmic limitations yet do not discuss societal impact. No negative societal impact beyond what is already present in generic machine learning algorithms has been observed.**\n\nThanks for your suggestion. We will add a section to discuss the societal impact in the paper.\n"}, {"review_id": "eqMqaGDFxj", "reviewer": "Reviewer_epDa", "summary": "The paper challenges the traditional assumption of semi-supervised learning: the feature distributions of labeled and unlabeled data are consistent. It claims that this assumption rarely holds in realistic scenarios, as: (a) unlabeled samples could contain various corruptions; (b) unlabeled samples could contain unseen styles.\n\nFor this, the authors propose Self-Supervised Feature Adaptation (SSFA), a generic framework for improving SSL performance when labeled and unlabeled data come from different distributions.\n\nSSFA consists of a semi-supervised learning module and a feature adaptation module, where the semi-supervised learning module performs semi-supervised and self-supervised learning", "questions": "(1) For Table 1, I am curious about the proposed method's performance on standard SSL benchmark, or when the ratio is low (like 0.1), this is a measure of whether the proposed method is robust to any scenarios, even if the noise is not that much.\n\n(2) The ablation study shows that the number of shared layers won't affect much as long as not sharing all, I would suggest the authors to conduct experiments on more backbones and get some empirical conclusion on how to set this hyper-parameter.\n", "limitations": "Limitation of this work is that the performance of SSFA is affected by the shared parameters between the main task and the auxiliary task.\nThe ablation study shows that the performances do not differ much as long as not all parameters are shared. \nThe number of layers to share should be a hyper-parameter to tune.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper is well written and easy to follow, presentation is good.\n\n(2) The proposed method is simple and effective. It consistently improves over its vanilla SSL algorithm counterpart.\n\n(3) The paper conducts extensive experiments on two realistic scenarios and ablation study to validate the contribution of each component and the overall method.\n\n", "weaknesses": "(1) The proposed method has limited novelty. Incorporating self-supervised learning into semi-supervised learning, or leveraging self-supervised learning for domain adaptation, both are not new conceptions. \n\n(2) Some important technical details are missing in the experimental setting section. For example, what is the number of shared layers and the self-supervised learning algorithm by default. These are important for reproducibility.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your useful comments! We see that your main concerns are novelty, experimental details and performance. We will address your concerns. \n\n**W1. The proposed method has limited novelty. Incorporating self-supervised learning into semi-supervised learning, or leveraging self-supervised learning for domain adaptation, both are not new conceptions.** \n\n   Thank you for your comments on novelty. To our knowledge, our proposed SSFA is the first to propose to correct pseudo labels for unlabeled data through feature adaptation in SSL.  Our extensive experiments in various scenarios have also verified the effectiveness of our method. Therefore, SSFA is a concise yet effective SSL framework that can be used to solve more realistic SSL problems.\n\nPlease refer to our reply to Q1 in the general response for more detailed explanations.\n\n**W2. Some important technical details are missing in the experimental setting section. For example, what is the number of shared layers and the self-supervised learning algorithm by default. These are important for reproducibility.**\n\n   Thanks for your suggestions. We will add more experimental details in the paper to enhance reproducibility. By default, the number of shared layers in our experiment is 2, the default self-supervised learning task is the rotation prediction task, and the corresponding default self-supervised loss is the cross entropy loss.\n\n\n**Q1. For Table 1, I am curious about the proposed method's performance on standard SSL benchmark, or when the ratio is low (like 0.1), this is a measure of whether the proposed method is robust to any scenarios, even if the noise is not that much.**\n\n   According to your suggestion, we conducted experiments on standard SSL setting and on the setting of low ratio(=0.1) on CIFAR100 dataset with 400 labeled data. The results are summarized in the following table. It can be seen that our method can still bring significant improvements over the baseline. This indicates that our proposed SSFA framework is robust to SSL scenarios even under a low noise ratio. The results further demonstrate that our SSFA is a generalized SSL framework.\n\n   | Method   | standard  |  SSL    | ratio  0.1 |      |      |\n   | -------- | ------------ | ---- | --------- | ---- | ---- |\n   |          | L/UL         | US   | L         | UL   | US   |\n   | FixMatch | 33.3         | 25.8 | 31.7      | 10.5 | 24.8 |\n   | FM-SSFA  | **41.3**        | **33.0** | **41.2**      | **15.8** | **32.9** |\n\n**Q2. The ablation study shows that the number of shared layers won't affect much as long as not sharing all, I would suggest the authors to conduct experiments on more backbones and get some empirical conclusion on how to set this hyper-parameter.**\n\n   Following your suggestions, we conducted relevant experiments on WiderResNet to explore the impact of the number of shared layers on CIFAR100 benchmark. In WiderResNet, \"3 layers\" represents that the main task and self-supervised task share the whole feature extractors. As shown in the following table,  combined with our SSFA, the baseline has a significant improvement. \n\n   In fact, in most cases, adjusting the number of shared layers does not result in significant performance fluctuations. In rare cases, sharing all feature extractors may lead to potential risks. Based on our findings, we provide a more suitable shared layer setting: sharing half of the parameters of the shared feature extractor which is able to ensure relatively better performance.\n   \n   | Method               | 400  |   labeled   | 4000 |   labeled    | 4000 |  labeled     |\n   | -------------------- | ----------- | ---- | ------------ | ---- | ------------ | ---- |\n   |                      | L           | UL   | L            | UL   | L            | UL   |\n   | FixMatch             | 15.7        | 3.5  | 53.0         | 16.0 | 65.3         | 33.0 |\n   | FM-shared (2 layers) | **25.7**        | **22.2** | **60.2**         | **52.5** | **69.1**         | **57.8** |\n   | FM-shared (3 layers) | 23.3        | 21.0 | 57.3         | 47.3 | 66.2         | 53.0 |\n\n\n"}, {"review_id": "LtitO9UxnD", "reviewer": "Reviewer_Pxab", "summary": "The authors claim to have addressed the new problem of semi-supervised learning with feature distribution mismatch (FDM-SSL) and to have proposed a new method called Self-Supervised Feature Adaptation (SSFA) for this problem.\n\nThe proposed SSFA consists of semi-supervised learning module and a feature adaptation module. The semi-supervised learning module optimizes the classifier based on joint minimization of supervised/unsupervised losses (to pseudo-labels) and a self-supervised auxiliary loss (for rotation prediction) with weak/strong data augmentation. The feature adaptation module aims to generate more reliable pseudo labels by updating the feature extraction backbone based on the self-supervised auxiliary loss on the unlabeled samples.\n\nComparative experiments with existing semi-supervised learning are conducted and the effectiveness of the proposed method is claimed.", "questions": "I have no specific questions for now. I would like to ask the authors to point out any misunderstandings I have about the above points I have raised as weaknesses.", "limitations": "Its limitations are discussed in the concluding section.\n\nIt would be interesting to add a discussion of the performance of the proposed method in the presence of another often discussed realistic distribution mismatch, class distribution mismatch (open set semi-supervised learning).", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper addresses the FDM-SSL problem, which has few studied examples.\n\nThe method is simple and its superiority over the major semi-supervised learning methods in the FDM-SSL problem is experimentally demonstrated.", "weaknesses": "A. Problem Novelty\n\nThe authors claim in the abstract and conclusion sections that one of the novelties of this paper is in that they tackle the novel problem of semi-supervised learning with feature distribution mismatch. However, this claim is not valid because previous work [a] has addressed a very similar problem. The differences between these two should be emphasized and clarified. \n\nFurthermore, there are also several closely related problems that have been considered in the past (e.g., semi-supervised domain generalization [b] and few-shot domain generalization [c]), but discussion of their relevance is lacking.\n\n[a] Aminian et al., An Information-theoretical Approach to Semi-supervised Learning\nunder Covariate-shift. ICML, 2022.\n\n[b] Zhou et al., Semi-Supervised Domain Generalization with Stochastic StyleMatch. NeurIPS Workshop, 2021. (Extended version is published at IJCV)\n\n[c] Yuan et al., A novel forget-update module for few-shot domain generalization. Pattern Recognition, 2022.\n\n\nB. Experiments\n\nComparisons with existing methods for the related problems I listed in Weakness A above are missing. Furthermore, since the problem of FDM-SSL can be considered a variant of domain generalization, there should be comparisons with existing domain generalization (and more recent domain adaptation) methods. Due to the lack of such an evaluation, the effectiveness of the proposed method is not adequately demonstrated.\n\n\nC. Method Novelty\n\nThe novelty of the proposed method is lacking. It is basically composed of common techniques in semi-supervised learning (such as weak/strong data augmentation or pseudo labeling with confidence thresholding). The feature adaptation module, which performs fine-tuning of the feature extraction backbone based on self-supervised auxiliary loss, has not been explored in semi-supervised learning, but the method itself is borrowed from [30].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We now answer them point by point.\n\n**W A. Problem Novelty: A discussion of several closely related problems ([a], [b] and [c]) is lacking.**\n\nOur proposed FDM-SSL problem setting is different from the previous works, we summarize the similarities and differences as follows:\n\n+ **Comparing with [a]**. Our FDM-SSL and [a] both address the issue of different feature distributions for labeled and unlabeled data. The main differences between FDM-SSL and [a] lie in: **(1) test distributions.** [a] assumes that the feature distributions of test and unlabeled data are the same, diverging solely from labeled distribution. So [a] focuses on the model's performance on unlabeled distribution. Differently, our FDM-SSL framework imposes no constraints on the test distribution. The feature distribution of test data may encompass labeled, unlabeled, or even unseen ones during the training process. This presents a bigger challenge compared to [a]. **(2) cause of feature distribution mismatch.** [a] is more concerned about the different distribution of labeled and unlabeled data caused by *selection bias*, such as the construction of labeled and unlabeled data on the MNIST dataset in [a]. In contrast, our FDM-SSL is more concerned about the mismatch in feature distribution between labeled and unlabeled data caused by *image corruption* or *style changes*. Overall, there is a significant difference between FDM-SSL and [a].\n+ **Comparing with [b]**. The main differences between our FDM-SSL and Semi-Supervised Domain Generalization (SSDG) are: **(1) domain labels.** SSDG assumes that the training data has $K$ domains, providing domain labels for each sample. However, the domain labels are not presented in our settings since we assume there is no additional information for unlabeled data. Consequently, FDM-SSL is a more realistic setting with fewer constraints. **(2) distributions of labeled and unlabeled data.** In SSDG, each domain contains a set of labeled data and a set of unlabeled data. These labeled and unlabeled data are sampled from the same distribution. In contrast, in our FDM-SSL, there are no constraints on the feature distribution of unlabeled data. In detail, unlabeled data could potentially include samples sharing the same feature distribution as the labeled data, or there may be no unlabeled data sharing the same distribution as labeled data.\n+ **Comparing with [c]**. The main differences between our FDM-SSL and Few-Shot Domain Generalization (FSDG) are as follows: **(1) number of labeled samples.** FSDG often assumes a large number of labeled samples in source domain for training with a few labeled samples in target domain for adaptation. In contrast, in FDM-SSL, the number of labeled samples is very limited but with a large number of unlabeled samples for training. **(2) distribution shift.** FSDG focuses on generalizing a model to new classes with a limited number of label examples, which implies a shift in the label space between the test set and the training set ( $p_{test} (y)  \\neq p_{train} (y)$ ). On the contrary, FDM focuses more on the robustness of feature distribution shift ( $p_ {test} (x)  \\neq p_ {train} (x)$ ).\n\n**W B. Experiments: Comparisons with existing methods for the related problems I listed in Weakness A and more recent domain adaptation methods are missing.**\n\nBased on our answer to Weakness A, our FDM-SSL is different from domain generalization. Instead, it presents a more realistic SSL problem with fewer constraints. Given the mismatch in feature distribution among samples in FDM-SSL, coupled with the similarities between domain generalization and domain adaptation,  we choose the domain generalization and domain adaptation methods for comparison in our experiments. \n\nFor domain generalization methods, considering that [a] does not have open-source code and [c] has different settings on the training set compared to SSL, we evaluate [b] in FDM-SSL setting.\nThe following table shows the results of [b] on OFFICE-HOME dataset. We can observe that our method can achieve better performance than [b].\nIn addition, [b] introduced an additional style image generation model AdaIN  to expand the training data resulting in a higher training cost (much higher memory usage and training time than our method). So our SSFA is more simple and efficient than [b].\n\n| Method| A/ACPR| |P/ACPR| |R/ACPR | |\n| ---------- | ------ | ---- | ------ | ---- | ------ | ---- |\n| |L|UL|L|UL|L|UL|\n| StyleMatch |52.0|35.3|62.9|45.9|40.3|22.3|\n| FM-SSFA|**55.0**|**45.5**|**71.8**|**52.6**|**64.8**|**52.7**|\n\nFor domain adaptation, we choose the state-of-the-art UDA methods PMTrans for comparison. Detailed experiment results and analysis can be found in the response to Q3 and Q4 for Reviewer WadC.\n\n**W C. Method Novelty**\n\nFor more explanation of this issue, please refer to our reply to Q1 in the general response.\n\n**Limitations. Add a discussion of the performance of the proposed method in open set semi-supervised learning.**\n\nAccording to the reviewer's suggestion, we conduct experiments on open set semi-supervised learning. For CIFAR10, we split the classes into known and unknown classes by defining animal classes as known (6 classes) and others as unknown (4 classes). The following table shows the accuracy.\n\n| No. of labeled samples per class |5|50|\n| :--------------------: |-----|----|\n|FixMatch|48.2|86.0|\n|FM-SSFA|**65.8**|**88.5**|\n\nIn open set semi-supervised learning, our method can still bring improvements to the baseline, and the improvement is more significant when labeled data is rarer. In addition, we also conduct experiments on standard SSL benchmark or when the ratio is low (refer to the answer to Q1 from Reviewer epDa), and the results verify that our proposed method is adaptable and effective to many scenarios. To sum up, our proposed SSFA is a more general semi-supervised framework for handling inconsistent distribution of labeled and unlabeled data."}]

Author rebuttal:
We are grateful to all reviewers and ACs for the generous effort they have invested in reviewing our work! Some important or common questions are answered.

**Q1. The novelty of the method may be limited.**

Thanks for your comments.

According to Occam's Razor, *Entities should not be multiple unnecessarily*. In addition, complex methods often incur increased training costs. Therefore, our method is intentionally designed to be simple yet highly effective. 

In our paper, we aim to alleviate the severe performance degradation caused by feature distribution mismatch between labeled and unlabeled data in SSL. We propose Self-Supervised Feature Adaptation (SSFA) from the perspective of distribution adaptation. By using a feature adaptation module to update the features of unlabeled samples, we can map input data with different feature distributions to the same feature space, thereby greatly improving the pseudo-labels accuracy for unlabeled samples. In this way, our proposed SSFA is not a specific semi-supervised method, but a universal semi-supervised framework based on feature adaptation. 
To our knowledge, our proposed SSFA is the first to propose to predict pseudo labels for unlabeled data after feature adaptation in SSL.  This method is characterized by its simplicity and remarkable effectiveness in solving more complex SSL problems.  
Furthermore, in experiments, we have demonstrated that the SSFA framework can be combined with a wide range of pseudo-label-based semi-supervised learning methods. Besides, there is a wide selection of auxiliary tasks for feature adaptation modules, which greatly improves the practicality of our proposed SSFA framework. We believe that this simple framework can serve as a solid baseline for further research in Feature Distribution Mismatch SSL.

**Q2. The results of Lemma 1 is counter-intuitive.**

We provide a toy example to illustrate this counter-intuitive conclusion pointed by the reviewer: minimizing the loss of self-supervised tasks in Lemma1 indirectly minimizes that of the main task. 

Consider a two-layer linear network parametrized by $\theta_g$ (the shared linear layer) , $\theta_c$ (the linear head for main task) and $\theta_s$ (the linear head for self-supervised task) . The predictions for the main and self-supervised task can be denoted as: $\hat y_m = \theta_c^T \theta_g x$ and $\hat y_s = \theta_s^T \theta_g x$ separately. The main task loss is $l_m(x,y; \theta_g, \theta_c)=\frac12(y_m-\hat y_m)^2$ and the self-supervised task loss is $l_s(x,y; \theta_g, \theta_s)=\frac12(y_s-\hat y_s)^2$. Since $y_s$ is known, we update the shared feature extractor $\theta_g$ by one step of gradient descent on $l_s$. The updated feature extractor $\theta_g'$ is given by:
        $$
        \theta_g'= \theta_g - \eta(y_s-\hat y_s)(-\theta_sx^T)= \theta_g - \eta(y_s-\theta_s^T \theta_g x)(-\theta_sx^T),
        $$
        where $\eta$ is the learning rate. If we set $\eta=\frac{y_m-\hat y_m}{(y_s-\hat y_s)\theta_c^T\theta_sx^Tx}$, we can find the main task loss $l_m(x,y; \theta_g', \theta_c)=0$.

This toy example demonstrates that it is theoretically possible and reasonable to reduce the loss of the main task to zero, by optimizing self-supervised tasks. 


**Q3. The assumptions of Lemma 1 are somewhat strong, which may not hold true in practice.**

Although Lemma 1 relies on somewhat strong assumptions, its conclusion can be applied to real-world scenarios where the assumptions might not hold.
We attempt to verify this point by first offering a theoretical analysis and then providing empirical evidence derived from the theory.

+ **Theoretical Analysis**: Based on lemma1, for any $\eta$,  by smoothness and convexity,
$$l_m(x,y;h-\nabla l_s(x;h))\leq l_m(x,y;h)+\eta{\langle \nabla l_m(x,y;h),\nabla l_s(x;h)\rangle}+\frac{\eta^2 \beta}2 {\lVert \nabla l_s(x;h)\rVert}^2.$$
Denote $\eta^*=\frac{\langle \nabla l_m(x,y;h),\nabla l_s(x;h)\rangle}{\beta \lVert \nabla l_s(x;h)\rVert^2}$.
The equation becomes:
$$l_m(x,y;h-\eta^*\nabla l_s(x;h))\leq l_m(x,y;h)-\frac{\langle \nabla l_m(x,y;h),\nabla l_s(x;h)\rangle^2}{2\beta \lVert \nabla l_s(x;h)\rVert},$$
 namely,
$$ l_m(x,y;h)-l_m(x,y;h-\eta^*\nabla l_s(x;h))\geq \frac{\langle \nabla l_m(x,y;h),\nabla l_s(x;h)\rangle^2}{2\beta \lVert \nabla l_s(x;h)\rVert}.$$
It can be seen that in the smooth and convex case, with the gradient inner product between the main task loss $l_m$ and the self-supervised task loss $l_s$, i.e., $\langle \nabla l_m(x,y;h),\nabla l_s(x;h)\rangle$, increasing, the updated model has a smaller loss on the main task. That is, the larger the inner product, the greater the decline of the loss function.

+ **Empirical Evidence**: For non-convex loss functions, we empirically show that our theoretical insights also hold. In Figure 2 of our paper, we plotted the correlation between the gradient inner product and the performance improvement of the model on the test set, where each point in the figure represents the average result of a set of test samples. From Figure 2, it can be seen that there is a positive correlation between the gradient inner product and model performance improvement. This observed phenomenon is consistent with the theoretical conclusion, that is, a strong gradient correlation clearly indicates higher performance improvements over the baseline.


Meta-review: The paper studies an extension of the semi-supervised learning (SSL) setting in which feature distributions of labeled and unlabeled data can be different. The challenge when applying standard SSL methods in this setting is that the pseudo-labels tend to be incorrect due to the feature distribution mismatch. The authors propose a method for solving this problem by decoupling pseudo-label predictions from the current model. The proposed method consists of two modules: SSL learning module and feature adaptation module. SSL module is based on auxiliary self-supervised task inspired by previous works. Feature adaptation module is designed to produce more reliable samples by adapting the model to the unlabeled data distribution before making predictions. The method is evaluated with image corruption and style change as distribution mismatch scenarios.

Reviewers found the proposed method to be simple and effective. The paper is well-written, well-organized and easy to follow. The experimental evaluation is comprehensive and the improvements over baselines are substantial. The ablation studies are carefully designed and validate the proposed approach.

Some reviewers expressed concerns about (1) the problem's novelty and lack of discussion and proper positioning with respect to prior works that considered related problems, (2) the usefulness of theoretical results in reality given strong assumptions, and (3) lack of comparison to more recent UDA methods. During the rebuttal, these concerns have been addressed by the authors and reviewers raised their scores.

Eventually all reviewers agree that the paper should be accepted and thus my recommendation is to accept the paper.
Decision: Accept (poster)