Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the universal semi-supervised learning, which combines traditional SSL and domain adaptation with the open-set problem. Under this setting, two issues are raised as the technical challenge, i.e., the class distribution shift and feature distribution shift. Accordingly, the CAFA is proposed in this paper, which identifies the class shared data among training and testing domain first, and then employs domain adaptation on these data for semi-supervised learning. Experiment results demonstrate the appilcability of the proposed method on several benchmark datasets.
main_review: Originality:
This paper tries to unify SSL and domain adaptation in a single framework. The originality of is OK. But authors seem to miss an important reference which proposes the unsupervised open domain recognition problem:

[1] Unsupervised Open Domain Recognition by Semantic Discrepancy Minimization. In CVPR, 2019.

The key difference between CAFA and [1] lies in the usage of data that does not belong to the shared class. In CAFA, these data is assumed to be discarded, while in [1] these data are encouraged to be predicted to the categories (only in target domain) with equal probability. Authors should discuss the difference between CAFA and UODTN regarding to the problem setting. 

Quality and clarity:
The quality and clarity of this paper is good.

Significance:
I think this paper has revealed a good point towards the development of SSL. However, the relation among SSL, unsupervised domain adaptation and other related problem setting (e.g., semi-supervised domain adaptation), should be discussed in an comprehensive way.

Detailed comments:

a. In CAFA, the authors choose to discard those data that does not belong to the shared class. This point still need to be justifed in a proper manner, since it has been proved that these private data can be used to enhance the prediction diversity as in [1] and [2].

[2] Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations. CVPR, 2020.

Authors should compare CAFA with these methods on these methods under similar setting.

b. How does the wrongly detected class shared data affect the accuracy?

------
I have read the author response, which addressed my concerns appropriately. I stick to my original score and vote for acceptance on this paper.



limitations_and_societal_impact: See the detailed comment.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper targets the universal semi-supervised learning problem where the class distribution and the feature distribution both mismatch between the labeled data and the unlabeled data.

The paper proposed a new framework Class-shAring data detection and Feature Adaptation (CAFA) to address the universal semi-supervised learning problem. The key insight is to design a scoring mechanism integrating domain similarity and label prediction shift to identify both the labeled and unlabeled data from the shared classes.

The paper conducts experiments on several datasets for semi-supervised learning and domain adaptation and shows competitive results on the semi-supervised learning setting with both class distribution mismatch and feature distribution mismatch.

main_review: Strength:

The problem setting tackled in this paper is quite practical and general for realistic applications. Since the unlabeled can be collected from any resource, there is potential that the unlabeled data have both different class distribution and feature distributions from the labeled data. Also, the problem setting is much more challenging than prior works with no prior knowledge of class distributions.

The paper addresses the overfitting problem of domain discriminators by training on mixup samples, which generates more discriminative domain similarities. 

Label prediction shift with the adversarial perturbation can reflect the quality of the prediction.

The experiments are conducted on both semi-supervised learning dataset and domain adaptation datasets, which can demonstrate the effectiveness of the proposed method on the setting with feature distribution shift.

Weakness:

In Eqn. (1), the semi-supervised learning loss is performed on all the unlabeled data. As demonstrated in [14,25], if performing semi-supervised learning loss on all the unlabeled data, the performance degraded severely.

No variance is reported in the experiment results.

On line 109, both samples from p^u and p^l are labeled as '0'. 
limitations_and_societal_impact: No obvious negative social impact.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper investigates a realistic situation of open-set SSL, which includes the class distribution mismatch and the feature distribution mismatch. Moreover, it assumes the class relationship between labeled data and unlabeled data is unknown. The proposed method utilizes two scores to detect the data that belong to the common classes, and conduct domain adaptation to solve the feature mismatch problem.
main_review: The main contributions are: 1) to the best of my knowledge, this paper is the first work to investigate the domain difference between labeled and unlabeled data in SSL (in both label and feature domains), and it shows that conducting domain adaptation can address such problem; 2) the proposed domain similarity training which performs Mixup is novel; 3) this work shows that the adversarial perturbation can be utilized to perform OOD data detection, which is novel in open-set SSL.

Strengths: 
- The problem setting is quite realistic, and the experimental results are also encouraging. I feel that this work will promote the application of semi-supervised learning to more real-world problems.
- The writing of this paper is good, and the details of algorithms are clearly stated.
- The proposed method is a general framework that can be deployed to many existing methods. 


Weaknesses: 
- The proposed method is a bit complex. Since it conducts both Mixup training and adversarial training in each round, the method could be computationally expensive. 
- Can the proposed method outperform other SSL methods in the closed-set SSL setting? 
- Is there any deeper analysis on why the closed-set SSL methods would fail in such a feature distribution mismatch setting? 
- Since this work is strongly related to domain adaptation (the subset mismatch is related to partial domain adaptation, the intersectional mismatch is related to open-set domain adaptation, and the proposed setting is related to universal domain adaptation), it would be nice to discuss the difference between this problem and domain adaptation. 

limitations_and_societal_impact: It seems that the proposed method is a bit complex. Besides, it would be better if the authors can discuss the relationship between the universal semi-supervised learning discussed in this paper and universal domain adaptation.
Other limitations have been indicated in the above weakness.

I think this work will attract intensive attention as this work studies a situation which is quite close to realistic applications. This work paves the way for applying semi-supervised learning to solving real-world problems.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an approach for universal semi-supervised setting that handles both class distribution and feature distribution mismatch. The key idea of the CAFA approach lies in introducing a scoring mechanism that identifies shared classes in the labeled and unlabeled data. The objective function in CAFA consists of supervised cross-entropy loss on labeled data, adversarial loss for feature adaptation between labeled and unlabeled data and class sharing data exploration term on pseudo-labels that are calibrated by weighted softmax function where weights are computed for each class. The approach is validated on three datasets CIFAR-10, Office-31 and VisDA on three tasks: (1) only class distribution mismatch,(2) only feature distribution mismatch, and (3) both class and feature distribution mismatch.
main_review: Pros:
- The proposed universal SSL setting is very realistic and there is no method proposed specifically for this setting so far.
- The way scoring mechanism for labeled and unlabeled data is computed is interesting and provides insights about relationships of weights between shared and private classes of labeled and unlabeled data.
- Proposed approach significantly outperforms SSL methods.

Cons:
- The proposed approach is very complex and consists of so many design choices that are not clearly motivated and experimentally validated. Each design decision is supported only by an intuitive explanation why it should work. 
- Why is L_ssl weighted by w_u when it does not consist of only private but also shared classes? 
- What is the effect of pseudo-target calibration by w_c^avg?  While there is an experiment showing the average weight of classes in Fig 5, there is no ablation study that shows the effect on the performance.
- In domain similarity loss authors add an extra binary cross entropy term to help prevent overfitting but mixed feature representations will be generated from both domain-private and non-private classes so it is not clear what is the effect of this besides intuitive explanations that feature representations of shared classes should be closer.
- There are many parameters that are not explained how they should be scheduled and what is their effect on the performance. For example, why are \gamma and \delta in (1) scheduled as exp(-5(1-min(iter/n,1))^2)? Why is n 8000 for \gamma and 4000 for 
\delta? How do different values affect performance? Similarly, how to schedule 
 the parameter \lambda (i.e. alpha in Beta distribution)?
- How is the separation of the private classes in the unlabeled data guaranteed, or this can not be guaranteed at all and the approach can only guarantee to separate them from shared classes?  This should be clearly explained.
- Where is domain loss in eq (5) utilized? Eq (1) consists of L_{CE}, L_{adv} and L_{ssl}, but it is not clear where domain loss is used.
- Are the accuracies in Tables 1 and 3 reported only on the shared set of classes? If that is the case, does it mean that the authors report accuracy on only 3 classes for intersectional mismatch on the CIFAR-10?
- CIFAR-10 is quite a simple dataset, and it would be desirable to report performance on a more challenging CIFAR-100 dataset.
- Why do the authors consider 9 classes for CIFAR-10 instead of 10 and 30 classes instead of 31 on the Office dataset?
- In Figure 5b is this experiment done on CIFAR-10 dataset? Why is the performance of other baseline so low in the case when there are no private classes? How are methods evaluated when all classes are private?
- Lack of baselines. For feature distribution mismatch it is desirable to compare to the state-of-the-art partial and universal domain adaptation methods such as DANCE (Saito et al. Neurips ‘20) and ETN (Cao el al. CVPR ‘19)

Other comments:
- There is a mistake in the domain similarity paragraph. Non-adversarial discriminator D’ should predict data from p^l as 1 not 0.
- There is a typo in the number of examples of the Office dataset. I believe it should be 4000.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper studies universal semi-supervised learning problem. In this setting, class distribution and feature distribution both mismatch between labeled and unlabeled data. The authors propose an algorithm CAFA and demonstrate its effectiveness on several benchmark data sets. The reviewers agree that the paper provide new insight on this problem and the setting is of practical importance. Good paper. I recommend accept.
Decision: Accept (Poster)