Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on making use of latent invariant representations for unsupervised domain adaptation. The goal is to address two State-Of-The-Art existing methods issues. The first is there is no principled way to ensure that marginal invariance (P^S(Z) = P^T (Z)) preserves conditional invariance (P^S (Z|Y) = P^T (Z|Y)). The second is there are cases (especially in lower-dimensional datasets) where the encoding function Phi needs domain-specific information, in addition to the features X. 
This paper introduces an invariant representation learning method that uses the data-generating process to justify and efficiently use domain-specific information in the encoder Phi and constraint it to ensure that Z contains relevant label-specific information.
Results on three real datasets and one synthetic dataset show the proposed Domain-Specific Adversarial Network (DSAN) model’s performances (accuracy and Wilcoxon p-value) compared with existing methods and an unregularized DSAN (DSAN-U) version.

main_review: #Originality:

Using the data-generating process under conditional shift based on the causal mechanism of generating X from Z with Theta_X represents the changing parameters of P(X|Y) across domains is a brilliant idea. I appreciated how the authors motivated the use of two separate encoders to infer Z in chapter 2. Indeed, model Theta_X as a latent variable and provide it as an input to Phi in addition to X and, then, mimic the causal mechanism generating X via a decoder constraining the influence of Theta_X to be minimal (acting as a regularizer on the first encoder and forcing it to preserve semantic information when inferring Z) is very interesting and well intuitively explained.

#Quality:

The submission technically sounds well, but there are mathematical notation problems: in line 21, k and m_S, m_T are not defined. Line 152, in proposition 1, the output space of Phi should be corrected. The notation of A elements must be homogenized (we meet a x in proposition 1 and an a in proposition 2). In proposition 2 it could be more appropriated to put the indices risks such as \epsilon_S^A(g) instead of \epsilon ^A(g) _S. Line 217 the second “reconstruct” loss has to be modified by “classif” loss. In Theorem 1 some comas are missed in distribution formulations. After line 256, there are some mistakes in the equations: the index j seems to be corrected by c. Line 259 denoted \lambda_T could be confusing notation because T is yet used for the target domain. After line 260, in the equations, \rho and h_a are not defined. 
It could be appreciated that authors give more intuitions of some concepts such as “Markov Blanket” (line 141). Also, the authors have to improve the end of chapter 3 because it is not intuitively and difficult to follow. 
Then, in line 210, authors must cancel one “label” word. Line 253, “RHS” is not defined. 
To conclude, this work seems to be in progress and should be improved because of all these points.

#Clarity: 

Please, read the previous remarks about “Quality” to understand why the submission is not clearly written.  

#Significance:

The experimental results sound well bust MSTN approach is announced line 312 but missing in tables. Also, in line 326, the authors have to rectify the Figure 4(b) reference instead of “Table 4(b) because there is no table 4 in the paper. 


limitations_and_societal_impact: The authors write no elements about limitations and societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: **Summary and contributions:**
The paper considers representation learning for unsupervised domain adaptation (UDA). They show that, in some cases, it is necessary to learn encoders that vary across domains, something which is rarely done in practice. They thus propose a new method for incorporating domain-dependent features in the encoding process.

The end model/objective contains many different components: it is essentially a DANN with a decoder that aims to reconstruct the inputs by using domain-dependent and independent features. Furthermore, the decoder is forced to rely as little as possible on the domain-dependent features. 

main_review: **Overall thoughts**: I think that the realization (and reasons) for using domain-dependent encoders are interesting and valuable. But I find the proposed objective too complicated (with 5 losses/components !)  to really be used by the community.

**Strengths**:
- **Valuable realization: domain-dependent encoders can be necessary**: The authors provide simple theoretical results that characterize some settings where domain-dependent encoders are necessary (e.g. conditional shift with shared support). Although very intuitive, I do not know of any previous work that pointed this out and I think that it is a valuable contribution to the UDA + representation learning subfield.
- **Useful/thorough appendices**: the appendices are useful and of good quality: extended related work, ablation, experimental settings… I especially appreciate the ablation study that investigates whether all the components of the final objective are necessary.
- **Decent performance**: On the presented datasets, the proposed method significantly outperforms the baselines. Similar results on standard UDA benchmarks would nevertheless be much stronger (see weaknesses below).

**Weaknesses**:
- **Very complicated objective**: I find the final objective to be a combination of too many terms (5 different components with their hyperparameters) to be practical. Ablation study shows that each component help, but there must be a simpler way of reaching the desired result.
- **Missing important benchmark**: the standard UDA benchmark for image classification is OfficeHome/Office31. These results are crucially missing. Even more so given that (1) the results on ImageCLEF seem strong; (2) you seem to provide code for OfficeHome (in `mdd.py`). 


**Questions:**
- Important: how many seeds were used to generate the results from Fig. 4.b? Were the results averaged over all seeds or was the best selected? These are important experimental details that are missing. I found that information for other datasets but not ImageCLEF. Given that in the original table standard deviations were provided, please add standard deviations.
- Why do you bold the results from DSAN in table 4.b. when they are not the best (P->C or I->C)? I initially thought that the statistical difference might not have been significant but that wouldn’t explain why CDAN I->C is not bolded.
- I’m having difficulties understanding why the results are much lower for every model in the ablation study and somehow make a huge jump when everything is together. Did you use the same hyperparameter strategy for the ablation study? If so do you have an idea as to why that happens? Providing standard deviations would really help to make sense of those results.

**Suggestions**:
- explain somewhere that for ImageCLEF you are working directly from features from a Resnet50 pre-trained on ImageNet (if that is the case).
- use “section” instead of “chapter”
- define what you mean by “minimality”, currently, it’s simply thrown to you in line 277 without saying why this is what you need/meant.

**Minor suggestions**:
- line 21: $y^{\mathcal{S}}$ -> $y_k^{\mathcal{S}}$
- line 21: $x^{\mathcal{S}}$ -> $x_k^{\mathcal{S}}$
- line 49: domain -> domain
- line 50: .However -> . However
- line 83: $Z$ to contain -> $Z$ contains
- line 104: makes, -> makes <???>,
- line 108: of which -> of, which
- line 132: i.e. -> i.e., 
- line 132: low-dimensionality -> low information (? dimensionality has nothing fo with minimality)
- Proposition 1: I think you are pissing an assumption in the proposition that states that in the support the density is different (you say it in the text but it should appear in the proposition).
- Theorem 1: define $c_1,c_2$ (constants that upper bound [...])
- Figure 3: it’s currently hard to parse. Although it might have more to do with the complexity of the objective than the illustration. - I would also use a darker green to improve readability. 
- Figure 4.b.: remove newline before “)”.
- Table 1: DANN t1-t2 ((1.3) -> (1.3)
- Table 1: what does parenthesis mean? if standard deviation then please use $\pm$
- Figure 5: withour -> without
- References: many duplicated entries e.g. 21-22 or 23-24 or 33-34. Please clean that up.
- Checklist: remove instructions (see line 486-487)


limitations_and_societal_impact: Some limitations are briefly mentioned, e.g., that the method might not be as useful in high dimensions (line 177-181) "high dimension [...] no overlap [...] $\phi$only a function of X is sufficient". The discussion of limitations could be extended, e.g., the fact that the objective is very complicated and what are the downsides (e.g. hyperparameter tuning for balancing each component).
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper analyzes the mechanism of domain adaptation and proposes a new strategy to reduce the gap between the two domains. The valuable information to achieve accurate classification can be well preserved. Extensive experiments are conducted on multiple dataset, such as Wi-Fi localization and Amazon Review.
main_review: Advantages:

- It is interesting to understand the mechanism behind domain adaptation and this paper analyzes it from both empirical and theoretical perspectives. These discussions may bring some insights to the community.

- The experimental results show that the proposed method achieve higher performance compared with the existing methods, which verifies the effectiveness of the proposed method.

- This paper really make a contribution to the community.  Beyond the invariant features, the authors discuss that the domain-specifc information is also valuable. It has been empirically demonstrated by extensive experiments.

Concerns: 
- The paper proposes a question about domain adaptation, i.e., 'What Transformations to Learn'. However, this question seems to not be answered explicitly. Could the authors conclude it with two or three sentences?
- The presentation of the paper can be refined.  For example, the fonts of captions in Figure 3 is very different from that in the main paper. The titles of Figure 4 can be improved.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a unsupervised domain adaptation method through the lens of learning invariant representations. The paper validates theoretically shows some limitations of current methods when learning from distinct source and target distributions. The paper validates experimentally on a large scale image dataset and 2 other datasets.
main_review: Although the paper achieves promising results and the paper provides some theoretical justifications for their method, overall the paper is not motivated well and can be challenging to understand. For example, figure 2 is not explained very well in the text or caption, and figure 3 does not detail their proposed architecture. The architecture bears resemblance to Cycle-GAN and other related work. The paper cites CYCADA as related work but does not compare against it. The paper also does not cite very relevant work such as [1,2,3] and many others. The notation used can also be confusing in the text. The main novelty over CYCADA seems to be the L_{inv}^{trans}  which is used to minimized the KL divergence between marginals of the source and target distributions, however such ideas have existed in the community for some time. 


[1] https://arxiv.org/abs/2004.07780
[2] https://openreview.net/forum?id=lQdXeXDoWtI
[3] https://arxiv.org/abs/2003.00688
limitations_and_societal_impact: The paper does not discuss any societal impacts.
review_title: null

Meta-review: The authors study unsupervised domain adaptation. They point out that if the data supports overlap, the encoder might need domain-specific information to recover an invariant representation which elicits a high-accuracy classifier. They propose a method which models this domain-specific information as a latent variable while encouraging it to have minimal influence on a decoder.

Reviewers agreed that the motivating analysis (necessity of domain-dependent encoders) was informative. Most reviewers (DkiW, 8KhW, 219a) thought the experiments were promising (reviewer d96Z remarked that OfficeHome results were missing, but the authors address this in the rebuttal). Reviewers also tended to agree that the method itself was overcomplicated and this might hurt its usefulness in practice (319a, d96Z), but I feel this shouldn't by itself prevent acceptance.
Decision: Accept (Poster)