Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to extend semi-supervised learning techniques to address domain adaptation problems and proposes a unified method, AdaMatch, that can handle unsupervised domain adaptation (UDA), semi-supervised domain adaptation (SSDA) and semi-supervised learning (SSL). While extensively borrowing techniques from SSL, AdaMatch proposes three unique techniques, i.e., distribution alignment,  random logit interpolation, and relative confidence threshold. Extensive experiments verify the efficacy of AdaMatch for the significant improvement over existing methods for different problems.
main_review: The advantages of this paper are as follows.
1) This paper proposes a neat algorithm that extends SSL to address the DA problems. The proposed method is technically sound, simple yet highly effective. 
2) Extensive experiments are conducted to evaluate the proposed method by comparing representative existing ones. 
3) The proposed method significantly extends the state-of-the-art performance. 

I like this paper, but there are some minor issues that I hope the authors can clarify or improve in future versions.
1) In several places, the authors mentioned that early stopping is applied or the mean accuracy is calculated over many checkpoints. Does it imply that the performance of the proposed method is unstable and can vary significantly over training time?
2) Why not evaluate the method on the commonly used benchmarks? For both UDA and SSDA, there are more commonly used datasets and extensive prior work have reported performance on those datasets. How about the performance of the proposed method on those datasets? Is it still as competitive as in the datasets chosen in the paper? 
3) Some techniques have not been ablation-studied, for example, the warmup function, applying cross-entropy twice, and prevention of gradient back propagation. 

summary_of_the_review: This is a good paper overall. It proposes a simple yet highly effective method to address various DA problems. There are some issues in the experiments, which I hope the authors could address in the response or in the future version. But overall I like it and recommend for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a unified solution for unsupervised domain adaptation (UDA), semi-supervised learning (SSL), and semi-supervised domain adaptation (SSDA). The authors extend the state-of-the-art SSL method, that is FixMatch, to make it capable of handling DA setting where target data stem from a different distribution with source data. Experimental results show that the proposed method performs on par or significantly better than respective state-of-the-art methods in UDA, SSL, and SSDA.

main_review:   

Strength

- It is interesting and should be practically useful to take a unified scheme for SSL and DA.

- A thorough empirical study validates the advantage of the proposed method in the setting of UDA, SSL, and SSDA.

- This paper is well-written and easy to follow.

Weakness

- The technical novelty of the proposed method is somewhat limited. AdaMatch is based on FixMatch and contains three modifications: random logit interpolation, prediction-based distribution alignment, and relative confidence threshold.

  - The random logit interpolation is somewhat novel here, but its siginificancy seems slight (Table 4).

  - The modification on distributional alignment is incremental. It is quite natural to use model's predictions for estimating class-wise weights of loss, especially in the DA setting (e.g. [R1]).

     [R1] "Partial Adversarial Domain Adaptation," ECCV 2018.

  - The modification on confidence threshold is also incremental. In addition, the idea to dynamically change the threshold for pseudo-labeling during training can be found in recent SSL method [R2].

     [R2] "Dash: Semi-Supervised Learning with Dynamic Thresholding," ICML 2021.


- I understand the intuition on why we need relative confidence threshold, but does it work properly in the early stage of the training? When the model is not sufficiently trained, c_\tau can be relatively small, which might induce accepting many incorrect pseudo-labels. Although this issue may be alleviated by adopting \mu in Eq. (11), it would be worth discussing somewhere in the manuscript.


Minor concerns that do not affect my score

- It should be interesting to see exclude-one-out analysis in the SSL setting. As shown in Section 5.2, AdaMatch outperforms FixMatch+ in SSL with DomainNet224, but it is not clear which modification (random logit interpolation or relative confidence threshold) more contributes to this improvement. If the relative confidence threshold contributes a lot, it somewhat matches the results in [R2].

- Table 1 provides a good summarization on the relationship between SSL, UDA, and SSDA, but there should be one important aspect that is not written here: which dataset can be limited in practice. In the setting of SSL, the number of labeled (source) data should be small as shown in the experiments. In contrast, in the setting of DA, the number of target data is often small, and this is the reason why some recent studies have been focusing on zero/few-shot domain adaptation. Since AdaMatch is based on FixMatch, it seems to implicitly assume the availability of abundant target data (e.g. uratio >> 1), which might be unsuitable for the DA setting. From this perspective, it would be worth discussing when AdaMatch performs well or do not.


summary_of_the_review: Although the technical novelty is somewhat marginal, this study has a significant impact in the literature of SSL and UDA by empirically showing a unified scheme capable to solve both SSL and UDA. I vote for "weak accept."
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To boost accuracy on domain shifts, this paper proposes an AdaMatch method for unsupervised domain adaptation (UDA), semi-supervised learning (SSL) and semi-supervised domain adaptation (SSDA). The proposed AdaMatch extends the existing FixMatch in three aspects, and it achieves the state-of-the-art accuracy using the same hyper-parameters regardless of the dataset or task.
main_review: The strengths of the paper:
+ The paper is well-written.
+ The paper introduces an AdaMatch method to boost accuracy on domain shift for UDA, SSL and SSDA, which extends the existing FixMatch in three aspects.
+ The AdaMatch achieves the state-of-the-art accuracy on the SSL, UDA and SSDA tasks using the same hyper-parameters regardless of the dataset.
+ Experiments evaluation is extensive, and the ablation analysis helps understand the importance of each improvement in the AdaMatch.

The weaknesses of the paper:
- In loss function (11), how does \mu(t) boost the accuracy compared to the case that \mu(t) = 1 in the entire training process. 
- In section 5.1.1 UDA WITH PRE-TRAINING, why is MCD selected as a baseline for comparison?
- In Figure 4 (center), the accuracy increases when the unlabeled to labeled data ratio (uratio) is increased, is this Figure correctly plotted?
summary_of_the_review: The contribution is significant and somewhat new, and the experimental evaluation is extensive. However, three concerns in the weaknesses of the paper should be clarified in the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper extends FixMatch as a unified approach for semi-Supervised Learning, Unsupervised Domain Adaptation, and Semi-supervised Domain Adaptation.  The proposal learns representations with a cross-entropy loss between labels and logits for a strong and weak augmentation versions of the source data while the target data helps with regularization.  The proposal uses a random mixture the logits of the two augmentations, and normalizes the distribution of the pseudo-labels with the ratio of the expected logits of the augmentations.

The contribution is the mixture of aligning the label distributions by normalizing them through the ratio between the source and target predicted labels, using relative thresholds to compare source and target label distributions, and the random logit interpolation to regularize the representations.  This combinations let the method outperform the reported methods in the three tasks on two datasets.
main_review: Strengths:
- The description of the relation between the tasks (SSL, UDA, and SSDA) helps to understand the main problem.  For example, Table 1 was really useful.

- Strong results that improve over the compared methods on three setups.

- Ablation study that shows the contribution of each part of the model.

Weaknesses:
- The contributions are not clear.  Most of the steps used are taken from previous methods in the literature and adapted.  For instance, the distribution alignment comes from Berthelot et al. 2020, or the loss from FixMatch (Sohn et al. 2020).  Please, clearly state what are your contributions over the state-of-the-art.  If the contribution is the mixture of existing ideas, then say so, instead of misleading the reader.

- The diagram of Fig. 1 is hard to follow.  For instance, the upper stream should be named as the variable $Z''$ and the bottom one $Z'$ (if I understood it correctly).  Similarly, the relative threshold seems to depend only from the source, yet it depends on the source and target (6).  Also, what is the operation that produces the second distribution on the middle row?  Is it the alignment w.r.t. the source and target weak?  If so, shouldn't the arrow mix both instead of only coming from the source?

- It is not clear what you are trying to do with the random logit interpolation.
  
  I don't understand how the interpolation relates to rather equalizing $Z_{\text{SL}}'$ or $Z''_{\text{SL}}$.  Since it is a random variable, there is no optimization that will happen on them, but just a mixture of them.

  $Z_{\text{SL}}'$ has the information of the source and target, while $Z''_{\text{SL}}$ only the source information.  Is your interpolation mixing that information as a form of regularization?  Or is $f$ shared in a specific way that let you achieve the claim of minimizing the loss of every point between these logits? Is not only through a normalization on the batch that the logits differ?   

  Moreover, since the $\lambda$ is random, you can't guarantee that you will learn a mixture of interest (i.e., an optimal $\lambda$), but instead you are making it robust to it (somehow). 

  Could you explain what you are trying to achieve here?  Is my interpretation correct?

- It is not clear what the claim that the random logit interpolation will make the logits either equal or to belong to an infimum.  How can the interpolation control what the loss landscape looks like?

- It seems from (1) and (2) that you call the model twice for mixing the source and target data (1) to learn some representations, and then alone with only the source data (2).  However, after these equations you mention that the model is called twice for each augmentation type.  Which one is it?  Or are you doing both?

- If the model $f(\cdot)$ already outputs the logits for the classes, why do you compute the pseudo-labels through another round of softmax functions (5)?

- You should minimally describe the augmentations you use to give a reader an idea of what "weak" and "strong" means.  Leave the details in the reference, but at least make a setup for the reader.
summary_of_the_review: The proposal extends FixMatch to work on three related tasks, and outperforms existing methods on two datasets.  The proposal mixes existing approaches for normalizing label distributions to train the model with a cross-entropy loss on different versions of the data to provide regularization. Moreover, the paper also proposes a random logit interpolation as means of regularization.  However, the intent of how these pieces were selected and how they work together is not clear to me.  The paper demonstrates empirically that the parts are necessary through an ablation study, yet I don't know why they are needed.

Overall, the paper presents an engineer method that works, although why escapes me.
review_title: null

Meta-review: Thanks for your submission to ICLR!

This paper presents a novel way to combine domain adaptation with semi-supervised learning.  The reviewers were, on the whole, quite happy with the paper.  On the positive side, the results are very extensive and impressive, it's a clever way to combine domain adaptation and semi-supervised learning, and it's a fairly general approach in that it works in several settings (e.g., unsupervised vs semi-supervised domain adaptation).  On the negative side, the approach itself is somewhat limited technically.

After discussion, the one somewhat negative reviewer agreed that the paper has sufficient merit and should be accepted; thus, everyone was ultimately in agreement.  I also read this paper carefully and personally find it very interesting and promising, so I am happy to recommend acceptance.  It seems to give state of the art performance in several cases, and could possibly lead to more research down the road on methods to combine adaptation techniques with SSL.
Decision: Accept (Poster)