Title: Generalized Semi-Supervised Learning via Self-Supervised Feature Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: semi-supervised learning

self-supervised learning
Abstract: Traditional semi-supervised learning (SSL) assumes that the feature distributions of labeled and unlabeled data are consistent which rarely holds in realistic scenarios. 
In this paper, we propose a novel SSL setting, where unlabeled samples are drawn from a mixed distribution that deviates from the feature distribution of labeled samples.
Under this setting, previous SSL methods tend to predict wrong pseudo-labels with the model fitted on labeled data, resulting in noise accumulation. To tackle this issue, we propose \emph{Self-Supervised Feature Adaptation} (SSFA), a generic framework for improving SSL performance when labeled and unlabeled data come from different distributions. 
SSFA decouples the prediction of pseudo-labels from the current model to improve the quality of pseudo-labels. Particularly, SSFA incorporates a self-supervised task into the SSL framework and uses it to adapt the feature extractor of the model to the unlabeled data. In this way, the extracted features better fit the distribution of unlabeled data, thereby generating high-quality pseudo-labels. Extensive experiments show that our proposed SSFA is applicable to various pseudo-label-based SSL learners and significantly improves performance in labeled, unlabeled, and even unseen distributions.

Generalized Semi-Supervised Learning via Self-Supervised Feature Adaptation

Abstract

Traditional semi-supervised learning (SSL) assumes that the feature distributions of labeled and unlabeled data are consistent which rarely holds in realistic scenarios. In this paper, we propose a novel SSL setting, where unlabeled samples are drawn from a mixed distribution that deviates from the feature distribution of labeled samples. Under this setting, previous SSL methods tend to predict wrong pseudolabels with the model fitted on labeled data, resulting in noise accumulation. To tackle this issue, we propose Self-Supervised Feature Adaptation (SSFA), a generic framework for improving SSL performance when labeled and unlabeled data come from different distributions. SSFA decouples the prediction of pseudo-labels from the current model to improve the quality of pseudo-labels. Particularly, SSFA incorporates a self-supervised task into the SSL framework and uses it to adapt the feature extractor of the model to the unlabeled data. In this way, the extracted features better fit the distribution of unlabeled data, thereby generating high-quality pseudo-labels. Extensive experiments show that our proposed SSFA is applicable to various pseudo-label-based SSL learners and significantly improves performance in labeled, unlabeled, and even unseen distributions.

1 Introduction

Semi-Supervised Learning (SSL) uses a small amount of labeled data and a large amount of unlabeled data to alleviate the pressure of data labeling and improve the generalization ability of the model. Traditional SSL methods [21, 43, 32, 28, 22, 30, 5, 6] usually assume that the feature distribution of unlabeled data is consistent with that of labeled one. However, in many real scenarios, this assumption may not hold due to different data sources. When unlabeled data is sampled from distributions different from labeled data, traditional SSL algorithms will suffer from severe performance degradation, which greatly limits their practical application.

In real-world scenarios, it is quite common to observe feature distribution mismatch between labeled and unlabeled samples. On the one hand, unlabeled samples could contain various corruptions. For example, in automatic driving, the annotated images used for training can hardly cover all driving scenes, and a plethora of images under varying weather and camera conditions are captured during real driving. Similarly, in medical diagnosis, individual differences and shooting conditions among patients could incur various disturbances in unlabeled data. On the other hand, unlabeled samples could contain unseen styles. For example, in many tasks, the labeled samples are typically real-world photos, while unlabeled data collected from the Internet usually contain more styles that are not present in the labeled data, such as cartoons or sketches. Notably, although the distributions of these unlabeled data may differ from the labeled data, there is implicit common knowledge between them that can compensate for the diversity and quantity of training data. Therefore, it is crucial to enlarge the SSL scope to effectively utilize unlabeled data from different distributions.

2

In this study, we focus on a more realistic scenario of Feature Distribution Mismatch SSL (FDMSSL), i.e. , the feature distributions of labeled and unlabeled data could be different and the feature distributions of test data could contain multiple distributions. It is generally observed that the performance of classical SSL algorithms [22, 30, 6, 9] degrades substantially under feature distribution mismatch. Specifically, in the early stages of training, classical SSL algorithms typically utilize the current model fitted on labeled data to produce pseudo-labels for unlabeled data. However, when the distribution of unlabeled data deviates, the current model are not applicable to unlabeled data, resulting in massive incorrect pseudo-labels and aggravating confirmation bias [3]. Recently, some works [7, 19] attempt to address FDM-SSL. These approaches assume that the unlabeled feature distribution comes from a single source and they only focus on the labeled distribution, which do not always hold in real tasks. However, due to the unknown test scenarios, it is desirable to develop a method to perform well on labeled, unlabeled and even unseen distributions simultaneously.

In this work, we propose a generalized Self-Supervised Feature Adaptation (SSFA) framework for FDM-SSL which does not need to know the distribution of unlabeled data ahead of time. The core idea of SSFA is to decouple pseudo-label predictions from the current model to address distribution mismatch. SSFA consists of two modules, including the semi-supervised learning module and the feature adaptation module. Inspired by [31] that the main classification task can be indirectly optimized through the auxiliary task, SSFA incorporates an auxiliary self-supervised task into the SSL module to train with the main task. In the feature adaptation module, given the current model primarily fitted on labeled data, SSFA utilizes the self-supervised task to update the feature extractor before making predictions on the unlabeled data. After the feature extractor adapts to the unlabeled distribution, refined features can be used to generate more accurate pseudo-labels to assist SSL.

Furthermore, the standard evaluation protocol of SSL normally assumes that test samples follow the same feature distribution as labeled training data. However, this is too restricted to reflect the diversity of real-world applications, where different tasks may focus on different test distributions. It is strongly desired that the SSL model can perform well across a wide range of test distributions. Therefore, in this work, we propose new evaluation protocols that involve test data from labeled, unlabeled and unseen distributions, allowing for a more comprehensive assessment of SSL performance. Extensive experiments are conducted on two types of feature distribution mismatch SSL tasks, i.e. , corruption and style mismatch. The experimental results demonstrate that various pseudo-label-based SSL methods can be directly incorporated into SSFA, yielding consistent performance gains across a wide range of test distributions.

2 Related work

Semi-Supervised Learning (SSL). Existing SSL methods [44, 7, 30, 41, 50, 9, 6] typically combine various commonly used techniques for semi-supervised tasks, such as consistency regularization[21, 43, 32, 28] and pseudo-label [22], to achieve state-of-the-art performance. For example, [6] expands the dataset by interpolating between labeled and unlabeled data. [5, 30] use the confident weak augmented view prediction results to generate pseudo-labels for the corresponding strong augmented view. [41, 50, 9, 44] improve the performance by applying adaptive thresholds instead of fixed thresholds. However, these works focus on traditional semi-supervised learning, which assumes that the labeled data and unlabeled data are sampled from the same distribution. Recently, some works [2, 53] address the issue of different feature distributions for labeled and unlabeled data under certain prerequisites. [2] assumes the test and unlabeled samples are drawn from the same single distribution. [53] studies Semi-Supervised Domain Generalization (SSDG), which requires multi-source partially-labeled training data. To further expand SSL to a more realistic scenario, we introduce a new Feature Distribution Mismatch SSL (FDM-SSL) setting, where the unlabeled data comes from multiple distributions and may differ from the labeled distribution. To solve the FDM-SSL, we propose Self-Supervised Feature Adaptation (SSFA), a generic framework to improve the performance in labeled, unlabeled, and even unseen test distributions.

Unsupervised Domain Adaptation (UDA). UDA aims to transfer knowledge from a source domain with sufficient labeled data to an unlabeled target domain through adaptive learning. The UDA methods can be roughly categorized into two categories, namely metric-based methods [23, 35, 25, 34, 49] and adversarial methods [1, 16, 8, 24, 46, 29, 37, 10, 42]. Metric-based methods focus on measuring domain discrepancy to align the feature distributions between source and unlabeled target domains. Adversarial methods [16, 47, 24] use a domain classifier as a discriminator to enforce the

̸

̸

feature extractor to learn domain-invariant features through adversarial training. Another task similar to FDM-SSL is Unsupervised Domain Expansion (UDE) [39, 51, 33, 45], which aims to maintain the model's performance on the labeled domain after adapting to the unlabeled domain. The main differences between UDA, UDE and FDM-SSL are the number of labeled data during training and the distribution of the unlabeled data. In FDM-SSL, the labeled data is scarce, and the distribution of unlabeled data is not limited to a specific domain but rather a mixture of multiple domains. These challenges make UDA and UDE methods unable to be directly applied to FDM-SSL.

Test-Time Adaptation (TTA). If the distribution of test data differs from that of training data, the model's performance may suffer performance degradation. TTA methods focus on improving the model's performance with a two-stage process: a model should first adapt to the test samples and then make predictions of them. For example, [31] and [15] introduce an auxiliary self-learning task to adapt the model to test distribution. [38] modifies the scaling and bias parameters of BatchNorm layers based on entropy minimization. [52, 13, 12, 26] try to improve the robustness of test-time adaptation. Unlike most TTA methods which suppose the test samples come from a single distribution, [40] assumes a mixed test distribution that changes continuously and stably. However, these assumptions are still difficult to satisfy in real-life scenarios. In addition, similar to UDA, TTA methods generally require a model trained well on the source domain. In contrast, FDM-SSL setting only requires scarce labeled data and has no restrictions on the distribution of unlabeled data.

3 Problem Setting

For Feature Distribution Mismatch SSL (FDM-SSL) problem, a model observes a labeled set D l = { ( x i , y i ) } N l i =1 and an unlabeled set D u = { ( u j ) } N u j =1 with N u ≫ N l , where x i and u i are input images and y i is the label of x i . The labeled image x i and unlabeled image u j are drawn from two different feature distributions p l ( x ) and p u ( x ) , respectively. Note that different from previous works, in FDM-SSL setting the unlabeled samples may come from a mixture of multiple distributions rather than just one distribution: i.e. p u ( x ) = w 0 p l ( x ) + ∑ K k =1 w k p k u ( x ) , where w 0 and w k represent the weights of the labeled distribution p l ( x ) and the k -th unlabeled distribution p k u ( x ) respectively. The goal of FDM-SSL is to train a model that generalizes well over a large range of varying test data distributions, including labeled, unlabeled and even distributions unseen during training ( p unseen ( x ) ). The differences between traditional SSL, UDA, TTA and FDM-SSL, are summarized in Table 1.

The core of FDM-SSL is to enhance the utilization of unlabeled samples in the presence of feature distribution mismatch. On one hand, the shared information, such as patterns and structures, in the unlabeled samples can provide effective cues for model adaptation. On the other hand, these mismatched unlabeled samples can facilitate the learning of a more robust model by exposing it to a wide range of data distributions.

4 Method

4.1 Overview

In the FDM-SSL setting, due to the feature distribution shift from p l ( x ) to p u ( x ) , directly applying the SSL model fitted on labeled data to unlabeled data may lead to massive inaccurate predictions on the unlabeled data, thereby aggravating confirmation bias and impairing SSL learning.

This motivates us to propose Self-Supervised Feature Adaptation (SSFA), a unified framework for FDM-SSL, which decouples the pseudo-label predictions from the current model to address distribution mismatch. As illustrated in Figure 1, SSFA consists of two modules: a semi-supervised

learning module and a feature adaptation module. The semi-supervised learning module incorporates a pseudo-label-based SSL learner with a self-supervised auxiliary task that shares a portion of feature extractor parameters with the main classification task. [15] has pointed out that the main task can be indirectly optimized by optimizing the auxiliary task. To this end, the feature adaptation module is designed to update the current model through self-supervised learning on unlabeled data, so as to better match the unlabeled distribution and predict higher-quality pseudo-labels for semi-supervised learning. Therefore, by optimizing the auxiliary self-supervised task individually, the updated model can make more accurate classification decisions in the main task.

𝑥

𝑢

!

Initialized Weights

Semi-Supervised Learning Module

𝓛

𝒙

𝑢

!

"

𝒒′

𝓛

𝒂𝒖𝒙

Feature Adaptation Module

𝓛

𝒂𝒑𝒕

𝜽

𝒄

𝜽

𝒔

𝜽

𝒈

"

𝜽

𝒄

𝜽

𝒔

X

𝓛

𝒖

stop gradient

𝑥

,

𝑢

!

, 𝑢

"

𝑥

𝑢

"

𝜽

𝒈

𝑢

$

4.2 Semi-Supervised Learning Module

In the Semi-Supervised Learning Module, we introduce a self-supervised task as an auxiliary task, which is optimized together with the main task. As shown in Figure 1, the network parameter θ comprises three parts: θ g for the shared encoder, θ c for the main task head, and θ s for the auxiliary task head. During training, SSL module optimizes a supervised loss L x , an unsupervised loss L u and a self-supervised auxiliary loss L aux , simultaneously. Typically, given a batch of labeled data { ( x b , y b ) } B b =1 with size B and a batch of unlabeled data { ( u b ) } µB b =1 with size µB , where µ is the ratio of unlabeled data to labeled data, L x applies standard cross-entropy loss on labeled examples:

where H( · , · ) is the cross-entropy function. Different SSL learners [30, 5] may design different L u . One of the widely used is the pseudo-label loss.In particular, given the pseudo-label q b for each unlabeled input u b , L u in traditional SSL can be formulated as:

where Ω is the per-sample supervised loss, e.g. , mean-square error [6] and cross-entropy loss [30].

Furthermore, we introduce a self-supervised learning task to the SSL module for joint training to optimize the parameter of the auxiliary task head θ s . To learn from labeled and unlabeled data distributions simultaneously, it is necessary to use all samples from both distributions for training. Therefore, L aux can be formulated as:

where ℓ s denotes the self-supervised loss function, A w and A s denote the weak and strong augmentation functions respectively. In order to maintain consistent optimization between the main task

and auxiliary task, it is necessary to optimize both tasks jointly. So we add L aux to semi-supervised learning module and the final object is:

where λ u and λ a are hyper-parameters denoting the relative weights of L u and L aux respectively.

4.3 Feature Adaptation Module

Some classic SSL approaches [30, 6, 5] directly use the outputs of the current classifier as pseudolabels. In particular, [30] applies weak and strong augmentations to unlabeled samples and generates pseudo-labels using the model's predictions on weakly augmented unlabeled samples, i . e ., q b = p m ( y |A w ( u b ) ; θ g , θ c ) , where p m ( y | u ; θ ) denotes the predicted class distribution of the model θ given unlabeled image u . However, as the classification model ( θ g , θ c ) is mainly fitted to the labeled distribution, the prediction on A w ( u b ) is usually inaccurate under distribution mismatch between the labeled and unlabeled samples.

To alleviate this problem, we design a Feature Adaptation Module to adapt the model to the unlabeled data distribution before making predictions, thereby producing more reliable pseudo-labels for optimizing the unsupervised loss L u . More specifically, before making pseudo-label predictions, we firstly fine-tune the shared feature extractor θ g by minimizing the self-supervised auxiliary task loss on unlabeled samples:

Here θ g is updated to θ ′ g = arg min L apt . Notably, since excessive adaptation may lead to deviation in the optimization direction and largely increase calculation costs, we only perform one-step optimization in the adaptation stage.

After self-supervised adaptation, we use ( θ ′ g , θ c ) to generate the updated prediction, which can be denoted as:

We convert q ′ b to the hard 'one-hot' label ˆ q ′ b and denote ˆ q ′ b as the pseudo-label for the corresponding strongly augmented unlabeled sample A w ( u b ) . After that, θ ′ g will be discarded without influencing other model parts during training. In the end, L u in the SSL module (Equation 4) can then be computed with cross-entropy loss as:

4.4 Theoretical Insights

In our framework, we jointly train the model with a self-supervised task with loss function ℓ s , and a main task with loss function ℓ m . Let h ∈ H be a feasible hypothesis and D u = { ( u i , y u i ) } N i =1 be the unlabeled dataset. Note that the ground-truth label y u i of u i is actually unavailable, but just used for analysis.

Lemma 1 ([31]) Assume that for all x, y , ℓ m ( x, y ; h ) is differentiable, convex and β -smooth in h , and both ∥∇ ℓ m ( x, y ; h ) ∥ , ∥∇ ℓ s ( x ; h ) ∥ ≤ G for all h ∈ H . With a fixed learning rate η = ϵ βG 2 , for every x , y such that ⟨∇ ℓ m ( x, y ; h ) , ∇ ℓ s ( x ; h ) ⟩ > ϵ , we have

where h ′ is the updated hypothesis, namely h ′ = h -η ∇ ℓ s ( x, y ; h ) .

Let ˆ E m ( h, D u ) = 1 K ∑ K i =1 ℓ m ( x i u , y i u ; h ) denote the empirical risk of the main task on D u , and ˆ E s ( h, D u ) = 1 K ∑ K i =1 ℓ s ( x i u ; h ) denote the empirical risk of the self-supervised task on D u . Under the premise of Lemma 1 and the following sufficient condition, with a fixed learning rate η :

we can get that:

where h ′ is the updated hypothesis, namely h ′ = h -η ∇ ˆ E s ( h, D u ) .

Therefore, in the smooth and convex case, the empirical risk of the main task ˆ E m can theoretically tend to 0 by optimizing the empirical risk of self-supervised task ˆ E s . Thus, in our feature adaptation module, by optimizing the self-supervised loss for unlabeled samples, we can indirectly optimize the main loss, thereby mitigating confirmation bias and making full use of the unlabeled samples. As above analysis, the gradient correlation plays a determining factor in the success of optimizing ˆ E m through ˆ E s in the smooth and convex case. For non-convex deep loss functions, we provide empirical evidence to show that our theoretical insights also hold. Figure 2 plots the correlation between the gradient inner product (of the main and auxiliary tasks) and the performance improvement of the model on the

0

1

2

3

4

5

6

Gradient inner product

10

0

10

20

30

40

50

60

Improvement (%)

test set, where each point in the figure represents the average result of a set of test samples. In Figure 2, we observe that there is a positive correlation between the gradient inner product and model performance improvement for non-convex loss functions on the deep learning model. This phenomenon is consistent with the theoretical conclusion, that is, a stronger gradient correlation indicates a higher performance improvement.

5 Experiments

In this section, we evaluate our proposed SSFA framework on various datasets, where labeled and unlabeled data are sampled from different distributions. To this end, we consider two different distribution mismatch scenarios: image corruption and style change, in the following experiments. More experimental details and results are provided in the Appendix.

5.1 Experimental Setting

Datasets. For the image corruption experiments, we create the labeled domain by sampling images from CIFAR100 [20], and the unlabeled domain by sampling images from a mixed dataset consisting of CIFAR100 and CIFAR100 with Corruptions (CIFAR100-C). The CIFAR100-C is constructed by applying various corruptions to the original images in CIFAR100, following the methodology used for ImageNet-C [18]. We use ten types of corruption as training corruptions while reserving the other five corruptions as unseen corruptions for testing. The proportion of unlabeled samples from CIFAR100-C was controlled by the hyper-parameter ratio . For the style change experiments, we use OFFICE-31 [27] and OFFICE-HOME [36] benchmarks. Specifically, we designate one domain in each dataset as the labeled domain, while the unlabeled domain comprises either another domain or a mixture of multiple domains.

Implementation Details. In corruption experiments, we use WRN-28-8 [48] as the backbone except for [6] where WRN-28-2 is used to prevent training collapse. In style experiments, we use ResNet-50 [17] pre-trained on ImageNet [14] as the backbone for OFFICE-31 and OFFICE-HOME. To ensure fairness, we use the same hyperparameters for different methods employed in our experiments.

Comparison Methods. We compare our method with three groups of methods: (1) Supervised method as a baseline to show the effectiveness of training with unlabeled data.(2) Classical UDA methods including DANN [1] and CDAN [24]; and (3) Popular SSL methods including MixMatch [6], ReMixMatch[5], FixMatch [30], FM-Rot (joint training with rotation prediction task), FM-Pre (pre-trained by rotation prediction tasks), FreeMatch [41], SoftMatch [9], and Adamatch [7].

Evaluation Protocols. We evaluate the performance across labeled, unlabeled and unseen distributions to verify the general applicability of our framework. Therefore, we consider three evaluation protocols: (1) Label-Domain Evaluation (L): test samples are drawn from labeled distribution, (2)

UnLabel-Domain Evaluation (UL): test samples are drawn from unlabeled distribution, and (3) UnSeen-Domain Evaluation (US): test samples are drawn from unseen distribution.

5.2 Main Results

Image Corruption. Our SSFA framework is a generic framework that can be easily integrated with existing pseudo-label-based SSL methods. In this work, we combine SSFA with four SSL methods: FixMatch, ReMixMatch, FreeMatch and SoftMatch, denoted by FM-SSFA, RM-SSFA, FreeMSSFA and SM-SSFA, respectively. Table 2 shows the compared results on the image corruption experiment with different numbers of labeled samples and ratio . We can observe that (1) the UDA methods, DANN and CDAN, exhibit very poor performance on labeled-domain evaluation (L). This is because UDA methods aim to adapt the model to the unlabeled domain, which sacrifices performance on the labeled domain. And the UDA methods are primarily designed to adapt to a single unlabeled distribution, making it unsuitable for FDM-SSL scenarios. In contrast, our methods largely outperform the two UDA methods on all evaluations. (2) The traditional SSL methods suffer from significant performance degradation in the FDM-SSL setting, particularly in unlabeled-domain evaluation (UL). And this degradation becomes more severe when the number of labeled data is small and the proportion of unlabeled data with corruption is high, due to the increased feature mismatch degree. Our methods largely outperform these SSL methods, validating the effectiveness of our self-supervised feature adaptation strategy for FDM-SSL. (3) SSFA largely improves the performance of original SSL methods on all three evaluations. The gains on unlabeled-domain evaluation (UL) indicate that SSFA can generate more accurate pseudo-labels for unlabeled samples and reduce confirmation bias. Moreover, the gains on unseen-domain evaluation (US) validate that SSFA can enhance the model's robustness and generalization.

Style Change. We conduct experiments on OFFICE-31 and OFFICE-HOME benchmarks to evaluate the impact of style change on SSL methods. Specifically, we evaluate SSL methods in two scenarios where unlabeled data is sampled from a single distribution and a mixed distribution. The results are summarized in Table 3. Compared to the supervised method, most SSL methods present significant performance degradation. One possible explanation is that style change causes an even greater feature distribution mismatch compared to image corruption, resulting in more erroneous predictions for unlabeled data. As shown in Table 3(a), our method shows consistent performance improvement over existing UDA and SSL methods on OFFICE-31. In some cases, such as "W/A", the task itself may be relatively simple, so using only label data can already achieve high accuracy. The results on OFFICE-HOME are summarized in Table 3(b). To demonstrate that our method can indeed improve the accuracy of pseudo-labels, we add an evaluation protocol UU , which indicates the pseudo-labels accuracy of selected unlabeled samples. As shown in Table 3(b), compared to existing UDA and SSL methods, our approach achieves the best performance on all setups and significantly improves the accuracy of pseudo-labels on unlabeled data. The results provide experimental evidence that supports the theoretical analysis in Section 4.4.

(a) Results on OFFICE-31

(b) Results on OFFICE-HOME

5.3 Feature Visualization

Figure 3 visualizes the domain-level features generated by SSL models with/without SSFA respectively. In Figure 3 (a), the vanilla FixMatch model maps labeled and unlabeled samples to different clusters in the feature spaces without fusing samples from different domains. Conversely, Figure 3 (b) shows that our FM-SSFA model can effectively fuse these samples. We further compute the A -distance metric [4] to measure the distributional divergence between the labeled and unlabeled distributions. The A -distance of FM-SSFA (0.91) is lower than that of FixMatch (1.10), verifying the implicit distribution alignment of SSFA.

Additionally, Figure 4 visualizes the class-level features generated by SSL models with/without SSFA respectively. In Figure 4 (a), FixMatch fails to distinguish samples from different classes. In contrast, FM-SSFA can separate samples from different categories well, as shown in Figure 4 (b). This result indicates that SSFA can help to extract more discriminative features.

(a) FixMatch

(b) FM-SSFA

label

unlabel0

unlabel1

unlabel2

unlabel3

unlabel4

unlabel5

unlabel6

unlabel7

unlabel8

unlabel9

5.4 Ablation Study

Combined with different self-supervised tasks. Table 4 compares different self-supervised tasks combined with our SSFA framework on the image corruption experiment. We employ different self-supervised losses corresponding to the rotation prediction task, contrastive learning task, and

(a) FixMatch

(b) FM-SSFA

class0

class1

class2

class3

entropy minimization task, which are the cross entropy loss, the contrastive loss from SimCLR and the entropy loss, respectively. As shown, different self-supervised tasks can bring significant performance gains. And the rotation prediction task leads to the largest improvements compared to the other two self-supervised tasks. This can be attributed to the more optimal parameters and the direct supervision signals in rotation prediction. In the experiments, we employ the rotation prediction task as the default self-supervised task.

The effectiveness of Feature Adaptation Module. To evaluate the effectiveness of the Feature Adaptation Module in the SSFA framework, we compare FM-SSFA with a baseline method FM-Rot, where we remove the Feature Adaptation Module and only add the self-supervised rotation prediction task. As shown in Table 3(a) and 3(b), FM-SSFA largely outperforms FM-Rot on OFFICE-31 and OFFICE-HOME, especially in the UL evaluation metric. The superiority of FM-SSFA highlights that the Feature Adaptation Module helps the model to adapt to unlabeled samples from different distributions, resulting in better generalization on unlabeled domain. Moreover, we can observe the FM-Rot is only marginally better than FixMatch and, in some cases, may even perform worse. These results suggest that simply integrating the self-supervised task into SSL methods only brings limited performance gains and may even be detrimental in some scenarios.

Small distribution shift between labeled and un-

labeled data. To demonstrate the robustness of our proposed method, we evaluate SSFA on scenarios where there is a small distribution shift between labeled and unlabeled data. Specially, we conduct experiments on two setups: ratio =0.0 (degrade into traditional SSL) and ratio =0.1. Table 5 shows our method can still bring significant improvements over the baseline. This indicates that our proposed SSFA framework is robust to various SSL scenarios even under a low noise ratio.

Robustness to confident threshold τ . Figure 5 illustrates the impact of confident threshold τ for different SSL models on OFFICE-31. As shown in Figure 5, the vanilla FixMatch model is highly sensitive to the values of τ . For example, when τ is set to 0.95, FixMatch achieves relatively high performance with confident enough pseudo-labels. When τ is lowered to 0.85, the performance of FixMatch on UL and UU degrades significantly, indicating that the model has deteriorated by too many wrong pseudo-labels. In contrast, FM-SSFA is more robust to different values of τ .

0.5

0.6

0.7

0.8

0.9

Threshold

20

40

60

80

100

FM(L)

FM(UL)

FM(UU)

FM-SSFA(L)

FM-SSFA(UL)

FM-SSFA(UU)

The number of shared layers between auxiliary and main task. Table 6 analyzes the impact of different numbers of shared layers in the shared feature extractor of the auxiliary and main task. As shown, the performance difference is negligible between sharing 2 and 3 layers in the feature extractor, while a significant performance degradation happens if we share all layers (4 layers) of the feature extractor. We argue that this is because too many shared parameters of the feature extractor may lead to over-adaptation of the feature extractor and compromise of the main task, thus the predictions of

pseudo-labels may be more erroneous. In the experiments, we set the number of shared layers to 2 by default.

6 Conclusion

In this paper, we focus on a realistic SSL setting, FDM-SSL, involving a mismatch between the labeled and unlabeled distributions, complex mixed unlabeled distributions and widely unknown test distributions. The primary challenge lies in the scarcity of labeled data and the potential presence of mixed distributions within the unlabeled data. To address this challenge, we propose a generalized framework SSFA, which introduces a self-supervised task to adapt the feature extractor to the unlabeled distribution. By incorporating this self-supervised adaptation, the model can improve the accuracy of pseudo-labels to alleviate confirmation bias, thereby enhancing the generalization and robustness of the SSL model under distribution mismatch.

Broader Impacts and Limitations. The new problem setting takes into account the model's generalization across different distributions, which is essential for expanding the application of SSL methods in real-world scenarios. Additionally, SSFA serves as a simple yet effective framework that can be seamlessly integrated with any pseudo-label-based SSL methods, enhancing the overall performance and adaptability of these models. However, the performance of SSFA is affected by the shared parameters between the main task and the auxiliary task. We hope that SSFA can attract more future attention to explore the effectiveness of feature adaptation with the self-supervised task.

Acknowledgments

This work is partially supported by National Key R&D Program of China no. 2021ZD0111901, National Natural Science Foundation of China (NSFC): 61976203, 62276246 and 62306301, and National Postdoctoral Program for Innovative Talents under Grant BX20220310.

References

Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain-adversarial neural networks. arXiv , 2014.

Gholamali Aminian, Mahed Abroshan, Mohammad Mahdi Khalili, Laura Toni, and Miguel Rodrigues. An information-theoretical approach to semi-supervised learning under covariateshift. In AISTATS , 2022.

Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In IJCNN , 2020.

Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NeurIPS , 2006.

David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In ICLR , 2020.

David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS , 2019.

David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alexey Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. In ICLR , 2022.

Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In CVPR , 2017.

Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality tradeoff in semisupervised learning. In ICLR , 2023.

Minghao Chen, Shuai Zhao, Haifeng Liu, and Deng Cai. Adversarial-learned loss for domain adaptation. In AAAI , 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Francesco Croce, Sven Gowal, Thomas Brunner, Evan Shelhamer, Matthias Hein, and Taylan Cemgil. Evaluating the adversarial robustness of adaptive test-time defenses. In ICML , 2022.

Mohammad Zalbagi Darestani, Jiayu Liu, and Reinhard Heckel. Test-time training can close the natural distribution shift performance gap in deep learning based compressed sensing. In ICML , 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A. Efros. Test-time training with masked autoencoders. In NeurIPS , 2022.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research , 17:2096-2030, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR , 2019.

Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In NeurIPS , 2021.

A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Totonto, 2009.

Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR , 2017.

D.-H. Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. In ICML , 2013.

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In ICML , 2015.

Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. In NeurIPS , 2018.

Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu. Transfer feature learning with joint distribution adaptation. In ICCV , 2013.

Prabhu Teja S and François Fleuret. Test time adaptation through perturbation robustness. In NeurIPS , 2021.

Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV , 2010.

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In NeurIPS , 2016.

Swami Sankaranarayanan, Yogesh Balaji, Carlos Domingo Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In CVPR , 2018.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. In NeurIPS , 2020.

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML , 2020.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In ICLR (Workshop) , 2017.

Kaibin Tian, Qijie Wei, and Xirong Li. Co-teaching for unsupervised domain adaptation and expansion. arXiv , 2022.

Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In ICCV , 2015.

Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv , 2014.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR , 2017.

Riccardo Volpi, Pietro Morerio, Silvio Savarese, and Vittorio Murino. Adversarial feature augmentation for unsupervised domain adaptation. In CVPR , 2018.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR , 2021.

Jie Wang, Kaibin Tian, Dayong Ding, Gang Yang, and Xirong Li. Unsupervised domain expansion for visual categorization. ACM Transactions on Multimedia Computing, Communications, and Applications , 17:1-24, 2021.

Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In CVPR , 2022.

Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. In ICLR , 2023.

Guoqiang Wei, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation. In CVPR , 2021.

Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In NeurIPS , 2020.

Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In ICML , 2021.

Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In ICCV , 2021.

C. Yu, J. Wang, Y. Chen, and M. Huang. Transfer learning with dynamic adversarial adaptation network. In ICDM , 2020.

Chaohui Yu, Jindong Wang, Yiqiang Chen, and Meiyu Huang. Transfer learning with dynamic adversarial adaptation network. In ICDM , 2019.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC , 2016.

Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz. Central moment discrepancy (CMD) for domain-invariant representation learning. In ICLR , 2017.

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In NeurIPS , 2021.

Jing Zhang, Wanqing Li, Chang Tang, Philip Ogunbona, et al. Unsupervised domain expansion from multiple sources. arXiv , 2020.

Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO: test time robustness via adaptation and augmentation. In NeurIPS , 2022.

Kaiyang Zhou, Chen Change Loy, and Ziwei Liu. Semi-supervised domain generalization with stochastic stylematch. International Journal of Computer Vision , 131:2377-2387, 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors claim to have addressed the new problem of semi-supervised learning with feature distribution mismatch (FDM-SSL) and to have proposed a new method called Self-Supervised Feature Adaptation (SSFA) for this problem.\n\nThe proposed SSFA consists of semi-supervised learning module and a feature adaptation module. The semi-supervised learning module optimizes the classifier based on joint minimization of supervised/unsupervised losses (to pseudo-labels) and a self-supervised auxiliary loss (for rotation prediction) with weak/strong data augmentation. The feature adaptation module aims to generate more reliable pseudo labels by updating the feature extraction backbone based on the self-supervised auxiliary loss on the unlabeled samples.\n\nComparative experiments with existing semi-supervised learning are conducted and the effectiveness of the proposed method is claimed.\n\nSTRENGTHS:\nThis paper addresses the FDM-SSL problem, which has few studied examples.\n\nThe method is simple and its superiority over the major semi-supervised learning methods in the FDM-SSL problem is experimentally demonstrated.\n\nWEAKNESSES:\nA. Problem Novelty\n\nThe authors claim in the abstract and conclusion sections that one of the novelties of this paper is in that they tackle the novel problem of semi-supervised learning with feature distribution mismatch. However, this claim is not valid because previous work [a] has addressed a very similar problem. The differences between these two should be emphasized and clarified. \n\nFurthermore, there are also several closely related problems that have been considered in the past (e.g., semi-supervised domain generalization [b] and few-shot domain generalization [c]), but discussion of their relevance is lacking.\n\n[a] Aminian et al., An Information-theoretical Approach to Semi-supervised Learning\nunder Covariate-shift. ICML, 2022.\n\n[b] Zhou et al., Semi-Supervised Domain Generalization with Stochastic StyleMatch. NeurIPS Workshop, 2021. (Extended version is published at IJCV)\n\n[c] Yuan et al., A novel forget-update module for few-shot domain generalization. Pattern Recognition, 2022.\n\n\nB. Experiments\n\nComparisons with existing methods for the related problems I listed in Weakness A above are missing. Furthermore, since the problem of FDM-SSL can be considered a variant of domain generalization, there should be comparisons with existing domain generalization (and more recent domain adaptation) methods. Due to the lack of such an evaluation, the effectiveness of the proposed method is not adequately demonstrated.\n\n\nC. Method Novelty\n\nThe novelty of the proposed method is lacking. It is basically composed of common techniques in semi-supervised learning (such as weak/strong data augmentation or pseudo labeling with confidence thresholding). The feature adaptation module, which performs fine-tuning of the feature extraction backbone based on self-supervised auxiliary loss, has not been explored in semi-supervised learning, but the method itself is borrowed from [30].\n\nQUESTIONS:\nI have no specific questions for now. I would like to ask the authors to point out any misunderstandings I have about the above points I have raised as weaknesses.\n\nLIMITATIONS:\nIts limitations are discussed in the concluding section.\n\nIt would be interesting to add a discussion of the performance of the proposed method in the presence of another often discussed realistic distribution mismatch, class distribution mismatch (open set semi-supervised learning).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper challenges the traditional assumption of semi-supervised learning: the feature distributions of labeled and unlabeled data are consistent. It claims that this assumption rarely holds in realistic scenarios, as: (a) unlabeled samples could contain various corruptions; (b) unlabeled samples could contain unseen styles.\n\nFor this, the authors propose Self-Supervised Feature Adaptation (SSFA), a generic framework for improving SSL performance when labeled and unlabeled data come from different distributions.\n\nSSFA consists of a semi-supervised learning module and a feature adaptation module, where the semi-supervised learning module performs semi-supervised and self-supervised learning\n\nSTRENGTHS:\n(1) The paper is well written and easy to follow, presentation is good.\n\n(2) The proposed method is simple and effective. It consistently improves over its vanilla SSL algorithm counterpart.\n\n(3) The paper conducts extensive experiments on two realistic scenarios and ablation study to validate the contribution of each component and the overall method.\n\nWEAKNESSES:\n(1) The proposed method has limited novelty. Incorporating self-supervised learning into semi-supervised learning, or leveraging self-supervised learning for domain adaptation, both are not new conceptions. \n\n(2) Some important technical details are missing in the experimental setting section. For example, what is the number of shared layers and the self-supervised learning algorithm by default. These are important for reproducibility.\n\nQUESTIONS:\n(1) For Table 1, I am curious about the proposed method's performance on standard SSL benchmark, or when the ratio is low (like 0.1), this is a measure of whether the proposed method is robust to any scenarios, even if the noise is not that much.\n\n(2) The ablation study shows that the number of shared layers won't affect much as long as not sharing all, I would suggest the authors to conduct experiments on more backbones and get some empirical conclusion on how to set this hyper-parameter.\n\nLIMITATIONS:\nLimitation of this work is that the performance of SSFA is affected by the shared parameters between the main task and the auxiliary task.\nThe ablation study shows that the performances do not differ much as long as not all parameters are shared. \nThe number of layers to share should be a hyper-parameter to tune.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a generalized Self-Supervised Feature Adaptation (SSFA) framework for FDM-SSL (Feature Distribution Mismatched Semi-Supervised Learning) that does not require prior knowledge of the distribution of unlabeled data. The SSFA framework aims to address distribution mismatch by decoupling pseudo-label predictions from the current model. It consists of two modules: the semi-supervised learning module and the feature adaptation module. The authors draw inspiration from previous work on auxiliary tasks and incorporate an auxiliary self-supervised task into the SSL module to train alongside the main task. In the feature adaptation module, the current model, primarily trained on labeled data, is updated by utilizing the self-supervised task to adapt the feature extractor before making predictions on unlabeled data. This adaptation allows for the generation of more accurate pseudo-labels, which in turn assist the SSL process.\n\nSTRENGTHS:\n- Addition of the feature adaptation module to adapt the model (trained on labeled data) to the unlabeled data distribution before generating pseudo labels seems to help with the distribution shift problem between labeled and unlabeled data. \n\n- Extensive empirical evidence provided to show that proposed SSFA method significantly improves classification accuracy on CIFAR100 and OfficeHome and Office31 datasets. Domain-level and class-level feature visualization also suggest successful feature adaptation across different domains and learning of more distinguishable representations across different classes. Ablations studies investigating the effect of different self-supervised tasks, number of shared layers between auxiliary and main task, and the sensivity of the method to different confidence thresholds are also useful.\n\nWEAKNESSES:\n- Lemma 1 is fine but convexity is a very strong assumption, which does not hold true in real-world deep loss functions. Some empirical evidence is provided, but it is somewhat counterintuitive to think that minimizing self-supervised empirical loss will also minimize supervised empirical loss. \n\n- Using weakly augmented samples for a supervised loss and strongly augmented samples for an unsupervised loss is not novel and first introduced in the FixMatch paper. Technical novelty is somewhat incremental though feature adaptation module can still be considered a significant improvement.\n\nQUESTIONS:\nIt was not very clear from the text about how the feature adaptation model is being trained. The module is being trained for one iteration and then is being discarded. This part is confusing. Is the module trained one unlabeled sample at a time or using all unlabeled samples? If the module needs all unlabeled samples to be trained this may not reflect a real-world use case scenario and will limit the applicability of the approach. Figure 1 needs to be updated to reflect these important details. \n\nWhat kind of self-supervised loss is used in equation 5? \n\nPlease define G in Lemma 1.  What does subscript m in the loss function in Lemma 1 indicates? It was not previously used in defining loss function.\n\nPlease correct the typos in Table 1. \"Traditioal\",   Abundant/Scarce could replace Plenty/Lack. \n\n\n-------------\nYour responses have clarified some of the concerns I had with Lemma 1.  In particular, I acknowledge your explanations regarding the training process of the feature adaptation module, and the flexible nature of the self-supervised loss function. I am inclined to update my initial score to reflect these additional insights your rebuttal offered. Thanks.\n\nLIMITATIONS:\nConclusions allude to some algorithmic limitations yet do not discuss societal impact. No negative societal impact beyond what is already present in generic machine learning algorithms has been observed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an interesting and novel semi-supervised learning method with auxiliary self-supervised feature adaptation for addressing issues with heterogeneity in labelled and unlabelled data. In fact, the authors focus on heterogeneity across domains, which is not something traditionally considered in the SSL literature, but is more in the field of unsupervised domain adaptation. The authors combines ideas of feature adaptation from UDA with consistency regularization and pseudo-labelling from SSL to bridge the two fields. The results presented seem much better than the baseline methods.\n\nSTRENGTHS:\n1. Table 1 is useful and sets the tone of the paper showing the problem settings and constraints of each type of approach.\n2. The performance metrics reported in Table 2 and 3 are very impressive compared to baselines.\n\nWEAKNESSES:\n1. However, the caveat is that the baseline methods cannot do both SSL and UDA, which makes comparisons of performance metrics harder. However, SSFA does seem to be a novel method bridging the two fields. \n\nMinor points:\n1. Typo in Table 1: traditional\n2. Might be useful to point out more forcefully Eq 2 is traditional in SSL, but Eq 7 is actually used, especially as SSFA Algorithm with equation references is in Supplement. Maybe it would be useful to not call both Eq 2 and Eq 7, L_u.\n3. How was the cluster representation in Figure 4 made?\n\nQUESTIONS:\n1. The Feature Adaptation Module seems like 1-directional (unlabelled to labelled) and 1step transformation, as the unlabelled are aligned to the labelled. Have the authors tried using more steps in the optimization to allow more adaptation, or does that rely on the loss and step calibration?\n2. The authors show the effect of different fixed thresholds in Eq 7 in Figure 5. Have the authors tried an adaptive threshold as in FlexMatch?\n3. While the authors have compared SSFA to more recent and state-of-the-art models in SSL such as FreeMatch and SoftMatch, the authors have not compared against the arguably the state-of-the-art in UDA such PMTrans. How does SSFA compare against PMTrans or any of the more recent leaders the Domain Adaptation leaderboard on Office-31 and Office-Home?\n4. Moreover, SSFA is especially impressive for low number of labelled data against UDA methods. How does it do in low labelled data settings against more recent UDA methods?\n\nLIMITATIONS:\nThe authors have addressed some limitations of their method", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on Feature Distribution Mismatch SSL (FDM-SSL), which considers a mismatch between the labeled and unlabeled data. In this task setting, the challenges include the scarcity of labeled data and the mixed distributions of the unlabeled data. The paper proposes a framework named Self-Supervised Feature Adaptation (SSFA), which adapts the feature extractor to the unlabeled distribution. Experiments show that SSFA can improve SSL performance on labeled, unlabeled, and unseen data.\n\nSTRENGTHS:\n1. The paper studies a new SSL problem named Feature Distribution Mismatch SSL. The presented SSFA method is simple and effective to address this problem.\n\n2. Extensive experiments have been conducted to evaluate the performance of SSFA.\n\n3. The paper is well written and presented. For instance, Table 2 is clear, it reveals the difference between task settings.\n\nWEAKNESSES:\n1. The proposed SSFA method is built upon existing SSL and unsupervised domain adaptation (UDA) methods. In SSFA, the SSL module and Feature Adaptation Module are inherited from the benchmark SSL and UDA methods, respectively. The optimization objectives $L_x$, $L_u$, and $L_{aux}$ are widely used in SSL methods. The major novelty is the Feature Adaptation Module, which uses the unlabeled data as input to generate pseudo labels $q_b'$ instead of the original pseudo labels $q_b$ generated from the labeled data. I acknowledge that this work presents some novelties in proposing and addressing the challenges in domain-shifted SSL. However, the method itself is straightforward in my opinion.\n\n2. The assumptions of Lemma 1 are somewhat strong, i.e., the loss is expected to be convex and smooth, the learning rate is fixed to $\\frac{\\epsilon}{\\beta G^2}$. In addition, the result of Lemma 1 is counter-intuitive --- 'The empirical risk of the main task can be theoretically reduced to 0 by optimizing the empirical risk of self-supervised task.' Due to these aspects, I kindly doubt that this theoretical result could work well in practice. And, the derivation only guarantees that the empirical risk is decreasing along with gradient descent, but not 'reduced to 0'. Please check this. \n\n3. I am a bit confused about the SSFA's theoretical performance on 'unseen data'. In Sections Introduction and Experiments, the authors claimed that SSFA can significantly improve SSL performance on unseen data. However, the method only adapts models to the *unlabeled* data during training. Why does SSFA improve the performance on unseen data? More theoretical evidence is encouraged to be presented to better support this claim.\n\n4. I suggest the authors show some visualizations of the gap between labeled and unlabeled data, e.g., what are the image corruptions in the experiments? What's the style change from OFFICE-31 dataset to OFFICE-HOME dataset? I find some related details in Appendix's texts. However, it would be more friendly to show some visualizations of the data gap, since domain shifting is the main challenge studied by this work.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}