Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper provides two upper bounds for two notions of generalization errors; this first is the gap between the population risk in the target domain and the population risk in the source domain; the second is the gap between the population risk in the target domain and the empirical risk in the source domain. In addition, the paper presents two techniques for improving generalization in UDA and validate them experimentally.
strengths_and_weaknesses: ### Originality

The paper's idea is not extremely novel, since there have been a lot of upper bounds for the first gap, such as [2].

### Quality

The technical part of the paper seems correct as far as I can see.

### Clarity 

The paper is well-written. 

### Significance

The question studied in the paper is significant. The generalization gap bounds inspire insights into algorithm designs.
questions: - The upper bounds seem very loose. What is the first upper bound's superiority to the previous bounds mentioned in [2]?
limitations: The limitations of the presented analysis could be discussed in more detail.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents an information-theoretic analysis for unsupervised domain adaptation, where the authors considered two notions of the generalization errors in this context and present novel learning bounds. Some of these bounds recover the previous KL-based bounds under different conditions, and other bounds are algorithm-dependent that exploits the unlabelled target data, which inspired the designing of learning algorithms. The authors also demonstrate the effectiveness of these schemes on standard benchmark datasets.
strengths_and_weaknesses: Some concerns:

1. To some extend, the results in section 4 were not surprised. The upper bounds for the population risks expressed by KL divergence were well established in the transfer learning literature, say the works by Ben-David. The issue is that such bounds are usually hard to compute and applied to algorithm designs. In addition, it is also very difficult to verify high tight such bounds are. Since this paper neither presented new proving techniques nor showed any tightness results on their presented upper bounds, I may question the usefulness of such bounds for addressing the core difficulty of transfer learning problems.

2. It seems that the number of samples in both source and target training data may play important roles in domain adaptation, which is referred to as the sample complexity issue in transfer learning. However, I do not see such characterizations in the bound presented in this paper. How does more target domain samples can effect the learning algorithms as well as the performances? This may need more detailed studies and discussions.

3. The authors mentioned "This paper presents the first information-theoretic analysis for unsupervised domain adaptation". This is a bit over strong statement, and I would not say this is the first paper to present such analyses, since there are many other theoretical papers in transfer learning area that had done such kind of analyses. The paper may present some new bounds, but shall not be the first in this area.

4. The most valuable part of this paper to me is the algorithm designs in section 5. However, it seems to me that the core idea of the proposed algorithm from the theoretical analyses in to add a L2 regularization term in the gradient decent updating, which has also been applied in many previous works. I wonder if the contribution of this paper is to provide another theoretical justification for the regularization term, or indeed there are novel steps in the algorithm that were not been considered before?

5. The equation between line 105 and 106, the second term on the right hand side should be R_{\mu} instead of R_{\mu'}. 
questions: No.
limitations: No.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper provides two types of information-theoretic bounds for unsupervised domain adaptation. The first is the PP generalization error, which bounds the gap between the source population risk and the target population risk. The second is the expected EP generalization error that bounds the expected gap between the target population risk and the source empirical risk. The PP generalization error is connected to the KL divergence-based marginal distribution matching method. And the EP generalization error bound is algorithm-dependent and intrigues two simple approaches that improve the KL method on rotated MNIST and digits dataset. 
strengths_and_weaknesses: ### Strengths
The whole paper is well-written and easy to follow. And the proposed algorithms seem to boost the performance of KL-divergence-based marginal distribution matching methods on the two datasets. 

### Weakness

1. The originality and significance of the work are not obvious

The PP generalization bound related to KL divergence is actually a similar version to the JS divergence bound of [1]. 
Both the PP and EP bound are related to the KL divergence of the two domain distributions $\mu, \mu^\prime$. However, this term is fixed and irrelevant to the algorithm given the domain distributions. Even though the paper mentioned some insights for learning the representation via minimizing the KL divergence, no rigorous theoretical analysis is provided.
The fundamental difficulty of UDA is not studied in this paper, e.g., the theoretical validity of the pseudo label. 

2. Not enough related work discussion

The paper is highly related to [2] and [3], which also studied UDA with information-theoretic tools, but no discussion is provided. 

3. Experimental evaluation is not adequate

The proposed algorithms (gradient penalty/controlling label information) in fact add a regularization for learning the source domain to avoid overfitting and improve cross-domain generalization.  Similar approaches exist in previous works like [1], but no comparison is conducted. 

4. Technique Concerns

See the questions in the following section.

[1] Shui, C., Chen, Q., Wen, J., Zhou, F., Gagné, C. and Wang, B., 2020. Beyond H-divergence: Domain adaptation theory with Jensen-Shannon divergence.

[2] Wu, Xuetong, et al. "Information-theoretic analysis for transfer learning." 2020 IEEE International Symposium on Information Theory (ISIT). IEEE, 2020.

[3] Jose, Sharu Theresa, and Osvaldo Simeone. "Information-theoretic bounds on transfer generalization gap based on Jensen-Shannon divergence." 2021 29th European Signal Processing Conference (EUSIPCO). IEEE, 2021.
questions: ### Major Concerns

1. Equation (1) in the main paper seems problematic. $E_{W,S}[R_{\mu^\prime}(W) - R_S(W)]$ equals to $E_{W,S,S^\prime_{X^\prime}}[R_{\mu^\prime}(W) - R_S(W)]$ when $W,S$ and $S^\prime_{X^\prime}$ are independent. However, $W=\mathcal{A}(S, S^\prime_{X^\prime})$ for UDA. 

2. The first equation in Appendix Line 678 is unclear to me. 


### Minor concerns
1. The subguassian assumption of loss function should be related to the specific distribution, for $(X,Y)\sim\mu$ and $(X^\prime, Y^\prime)\sim\mu^\prime$. Assumption 2 is w.r.t. $\mu$, while Theorem 5.1 is w.r.t $\mu^\prime$.
2. Line 112, typo PP -> EP
limitations: Yes, the authors have claimed that this work does not touch upon the fundamental difficulty in UDA or the lower bounds of generalization errors.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the generalization bound of Unsupervised Domain Adaptation (UDA). The authors discuss two kinds of generalization errors and present the upper bounds of them respectively. The theoretical analysis are based on information theory and give inspiring insights into algorithm designs. The experiments validate the effectiveness of the algorithm on benchmark datasets.
strengths_and_weaknesses: ## Pros:
1. Quality: This paper has adequate contribution to the generalization bound of UDA problem. They focuse on two kinds of generalization bounds. For the first error bound, their works are similar to the traditional analysis, but they generalize them that they do not require specific loss function and specific classification settings. For the second error bound, their bounds involve $W$, which motivates the algorithm designs. They derive the bounds under different assumptions and divergence measures.
2. Originality: This is the first work to analyse the generalization bounds of UDA from an information-theoretic perspective, as the authors claim. In fact, I am not familiar with the information theoretical framwork (Donsker-Varadhan representation of KL divergence) used in this paper. **So I am not very sure that the derivations in this paper are non-trival and novel.**
3. Clarity: The writing is clear and easy to follow. The paper is well-organized.
4. Significance: The generalization bounds of UDA are worth exploring. This paper is of great significance to this research field.

## Cons:
1. Some extra explanations and remarks should be added. Some contexts and symbols about information theory are somewhat hard to read.  For example, above Line 241, $I^{X_j'}(W;Z_i)$ is the disintegrated mutual information of $W$ and $Z_i$. According to the Definition 1, it involves the terms $P_{W,Z_i|X=X_j'}$ and $P_{W|X=X_j'}$. What does it mean for the probability of $W$ conditioned on an unlabeled sample $X$? It benifits for the readability of the paper if the authors add some explainations or remarks here.
2. Above Line 204, what is the distribution of $W,W'$ in $\mathbb{E}_{W,W',X}$ ?
3. Although I understand the meaning of $\ell(w,z)$, it seems has a conflic with the previous notion $\ell(f_w(X),Y)$.
4. In Theorem 5.1, 5.2, can we remove the term $\frac{1}{mn}\sum^{m}_{j=1}\sum^{n}_{i=1}$, since the samples $ \{X_j'\},\{Z_i\} $ are i.i.d. for all i,j?
questions: Q1. Compared Theorem 5.2 with Theorem 4.4, the difference between them is the first term of the bound in Theorem 5.2. And we know that $Err$ can be bounded by $\widetilde{Err}$ plus a standard generalization error term on source domain. So, what is the difference between the first term of the bound in Theorem 5.2 and the standard generalization error term on source domain? Which one is tighter?

Other questions can be found in Cons.
limitations: Yes.
review_title: null

Meta-review: metareview: The reviewers agreed that the paper's novelty is limited and lacks proper justification. The extensive authors' discussion shows that they could improve the manuscript, but the number of modifications in the revised version would require another round of reviews.

Therefore, I encourage the authors to submit an improved version of their work to an upcoming venue. 
confidence: Certain
Decision: Reject