Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposed an NFlow-based noise-robust UDA method. Specifically, they trained an NFlow (Durkan et al. 2019)  generative model of the feature distribution for each class by using the noisy pseudo labels generated from the source. With the classwise NFlow models, they introduced a modified feature mix-up scheme (i.e. D-CFA), and a consistency regularization loss on these NFlow-based classifiers and the task classifier (i.e. GDC). The experiments show that the proposed method can improve the DA performance by 1% compared to the SOTA methods on office-home for single-source UDA, as well as MSDA and PAC for multi-source UDA.        
strength_and_weaknesses: Strength:

- The work proposed a generative model based $\\textit{noise-robust}$ UDA method. They introduced an NFlow-based feature Mix-Up scheme (D-CFA): Instead of using instance features, they use the classwise NFlow models to sample features to do feature mix-up. It can provide enough diversity while eliminating the noise from instance features. Also, they proposed a consistency constraint (GDC) on these NFlow-based classifiers and the task classifier. 
- The theoretical justification for GDC and theoretical discussions from the perspective of probability about D-CFA are given in the paper.  
- Their experiment results showed that GDC decreased the noise level by 8.01% and D-CFA further decreased the noise level by 6.70%.

Weakness:

-  I have some concerns about the theoretical discussions of  GDC in the paper. The definition of  $=>$ in EQ3 is unclear. If it is $\geq$, the deduction has to be given. 
- The experiment results are not very strong. Though the proposed method can largely decrease the noise level, the final classification performance was only marginally improved by 1% compared to SOTA.    


clarity,_quality,_novelty_and_reproducibility: The work proposed a class-wise NFlow-based UDA method. They introduced a new generative model based  "Feature Mix-Up" scheme (D-CFA), and a consistency constraint (GDC) on these NFlow-based and the task classifier. The work has enough novelty. 


In the paper, they theoretically and experimentally justified that the proposed techniques could decrease the noise level of pseudo labels ( 8.01% by D-CFA, and 6.70% by GDC)  
summary_of_the_review: In summary, the work proposed a novel generative model based noise-robust UDA method. They theoretically and experimentally justified that the proposed techniques can decrease the noise level of pseudo labels. However, there are still some concerns about the method and experiment results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a novel method to handle the noisy pseudo-label issue in UDA settings. They first train normalizing flow models to estimate the distribution of the target domain bases on generated pseudo-labels. Then they use the generative models to construct the D-CFA on features in the source domain. Besides, they use the GDC loss to require that predictions of the discriminative model and generative model (i.e., the normalizing flows) are consistent. Finally, they conduct experiments on several datasets to prove the effectiveness of the methods.
strength_and_weaknesses: The authors study an important problem and propose novel techniques to deal with the noisy pseudo-label issue in UDA settings. The experiments are comprehensive and ablation studies are provided. The paper is generally well-written. However, I still have several concerns.

For the method part, I have several questions about the theoretical analysis. As a result, I doubt the effectiveness of the methods from a theoretical perspective.
1. For the probability view of D-CFA against label noise, the approximation steps are unclear and need further theoretical analysis. Firstly, the authors suppose that $y_i$ and $\hat{y}_i$ are independent, which may not be correct. The reason is that the pseudo-labels are generated to be close to the true label, making the two labels dependent. Secondly, even if the labels are independent, it is still unclear how to get Equation (4) from Equation (3). For example, why $p(f_i|\hat{y}_i,y_i)$ can be approximated as $p(f_i|\hat{y}_i)$?
2. For the theoretical insights in Section 3.1.1. I do not agree with the authors' claim on the $H\Delta H$ divergence. Since $h$ and $h'$ are optimized according to the sup function in the divergence, they are not trained on the same labeled source function. As a result, it may not be correct to claim that the distance can be approximated by the supreme in the target distribution.

For the experiment part.
1. I do not understand the results in the Office-Home dataset. The proposed method achieves superior results when the target domain is Ar or Cl while they are not the highest results when the target domain is Pr or Rw. Could the authors explain this phenomenon?

Minor issues:
1. The third line in Section 3.1.1: $k1$ $\rightarrow$ $K > 1$
clarity,_quality,_novelty_and_reproducibility: Details are mentioned in the Strength And Weaknesses section.
summary_of_the_review: As mentioned in the Strength and Weakness section, I think the experiments are comprehensive while the theoretical analysis needs further explanation. As a result, I am negative about the paper in this round.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel UDA method that utilizes generative models trained in the feature space to alleviate a domain gap. The generative model is trained in a feature space for each subset of the target data categorized into a common pseudo-class, and generated samples from this model are used for mixup-like augmentation of extracted features. Learning with this augmentation effectively reduces a domain gap, which makes a classifier trained on top of the feature perform well on the target data. Experimental results with several popular benchmark datasets show that the proposed method performs on par or better than state-of-the-art UDA methods.

strength_and_weaknesses: <Strength>

- The proposed method seems to be applicable to a wide range of model architectures.
- This manuscript is well-organized and easy to follow.


<Weakness>

- The proposed method requires training of several NFlow models, which may lead to prohibitive computation cost especially when the number of classes is large. In spite of this, the performance gain seems not so significant in the experiments.
- To train the NFlow models, we also need a sufficiently large number of target samples for every class, which may limit the applicability of the proposed method in practice. 

clarity,_quality,_novelty_and_reproducibility: <Clarity>

- This manuscript is well-organized and easy to follow.


<Quality>

- It is not clear why utilizing NFlow models can reduce label noise in pseudo-labeled data. Since NFlow models are trained based on pseudo labels, the model may learn inaccurate class-wise data distribution. It should lead to degrade the performance of the discriminative classifier, which will make label noise in pseudo-labeled data much worse.
- In Section 3.2, the authors state "We further assume that y_i and \hat{y}_i are independent," but is it reasonable? In many cases, x_i stems from p(x|y_i), and \hat{y}_i is determined based on x_i. Consequently, there should be some depedency between these two.
- Class prior probabilities are not considered in the computation of p^G. Why is it?
- Just after Eq. (9), the author state "we assume that their discrepancy should be intrinsically large to fit the maxpD,pG part," but this is not reasonable, because neither classifier is not trained to maximize the discrepancy between pG and pD. On the contrary, the discriminative classifier is trained to make pD consistent to pG, which is apparently doing the opposite thing.
- From Table 5, it is not clear to see if the proposed algorithm really improves the accuracy of pseudo labels, because the threshold \tau seems to be fixed. In addition, recent methods such as [R1] and [R2] adopt some sophisticated strategies to determine the threshold for more accurate pseudo labels. It would be better to compare the proposed method with such methods to show the effectiveness of D-CFA+GDC.

[R1] "AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation," ICLR 2022.

[R2] "Dash: Semi-Supervised Learning with Dynamic Thresholding," ICML 2021.


<Novelty>

Somewhat limited. The proposed method is based on FixMatch (as well as ICT[R3]), and one of the sample pair to mixup is replaced from a real sample to a generated one from NFlow. Using NFlow models might provide some novelty, but there seems to be no ingenuity in how to effectively utilize NFlow models specifically for domain adaptation.

[R3] "Interpolation Consistency Training for Semi-Supervised Learning," IJCAI 2019.


<Reproducibility>

Good. Several important implementation details are provided in the appendix, and the authors state "Our code will be released."

summary_of_the_review: The design of the proposed algorithm is interesting, but I have several concerns on its validity. I vote for "weak reject."
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a new domain adaptation method for both single-source and multi-source conditions. The main idea is to alleviate the error accumulation of target pseudo labels and reduce the domain gap at the prediction level. Technically, they use a generative model (normalizing flows) to model the class-wise distribution of the target data and solve a probability maximization problem. Experiments on three benchmarks demonstrate the effectiveness.
strength_and_weaknesses: Strength:
1. The paper is nicely written and the proposed method is well described.
2. Useful illustrations are provided that illustrate essential parts of the problem or method.
3. The empirical evaluation is thorough and conducted in diverse settings, including single- and multi-source settings.

Weaknesses:
1. The main concern for me is the difference with TSA (Li et al. 2021b). First, the idea of using the class-wise target distribution to perform augment is similar to TSA. There should include more comparisons and discussions. Second, this work exploits NFlow instead of Gaussian distribution to model the class-wise distribution of target data but the motivation is not clear and is not well-proven. For example, in Relate Work section, the authors claim that NFlow models can be performed to bridge the domain gap and alleviate noise accumulation. How about Gaussian-based models?  Third, it would be nice to revisit TSA before introducing the proposed D-CFA.

2. Concern about noise correction. From Fig. 1(a), we can see that target samples around the decision boundary of the discriminative classifier (red star) are prone to be mixed. My concern is that would these samples be corrected over and over again.

3. More explorations are needed. It is a bit unclear that two assumptions "the pseudo-labels are intact but the features are corrupted" and "the features are intact but the labels are corrupted" are in the Introduction.

4. A doubt about the feasibility of class-wise distribution. D-CFA samples the features from the class distribution of the same class (depending on pseudo labels predicted by the generative models), however, due to domain shift, untrustworthy pseudo-labels may affect the construction of the distribution. Thus, the sampled features might be incorrect.

5. About experimental results. First, the improvement on Office-Home is limited (72.1 vs TSA: 71.2 & SRDC: 71.3) and more recent results should be included. Second, it might be better to provide the results of TSA on MSDA in Table 3 and Table4. 
clarity,_quality,_novelty_and_reproducibility: As discussed above, insufficient novelty is contained in the proposed method. Clarity and reproducibility lack sometimes, among others for the above weaknesses.
summary_of_the_review: In summary, the paper proposes an interesting approach to a relevant problem. Clarity and reproducibility could be improved with a minor revision. At this point, my main concern is the difference with TSA. This point requires clarification and some solid empirical support before warranting acceptance of this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Strength aspects of the proposed approach relate to a certain novelty (not unanimously) and as containing some theoretical contributions, also corroborated by a complete experimental stage. The paper was also considered in general well presented, but still subject to improvement.

Conversely, weak points regard some parts to be clarified concerning the technical and theoretical contributions, the latter containing hypotheses or approximations not better motivated or explained, justifications of the approach not so well highlighted, and just slightly increased performance wrt the state of the art, especially when compared to the heavier computational cost. Indeed, novelty was argued by a couple of reviewers.

Rebuttal was provided, but reviewers were not fully convinced of some of the replies, and none changed his/her ratings, which resulted all below threshold. 

For these reasons, the paper is deemed not acceptable to ICLR 23 in this form.
Decision: Reject