Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers a domain adaptation setting where the features X and label Y are both influenced by latent features Z, the distribution of which may shift across domains. The idea of the paper is to fit a latent variable model to learn Z, and then use the learned variables as the basis for an invariant predictor. The bulk of the paper is concerned with showing this is indeed possible, which they aim to do by adapting results from non-linear independent component analysis.
strength_and_weaknesses: Strengths

This work fits in well with a recent line of attacks on domain adaptation that assume an underlying (causal) structure involving latent variables, then define the allowed class of shifts in terms of shifts on these latent variables. Such approaches seem promising for articulating more refined (and possibly even useful) domain shift assumptions. The idea of actually trying to learn the latent variables also seems sound, and relatively underexplored.

Weaknesses

The main problem with this paper is that the main contribution is to give conditions for when the latent-variable modelling approach will work, but then these conditions get omitted!  Proposition 4.1 just references "certain assumptions"! The issue is that all of the promise of this approach comes down to whether or not these certain conditions are actually plausible in real problems. As such, one would expect the bulk of the paper to be dedicated to this question. Instead, the conditions aren't even given. 

A secondary---related---weakness is that the treatment of related work fails to make it clear how other approaches to domain adaption relate to the currently proposed one. This is most obvious with respect to latent variable based approaches---e.g., 1,2,3,4 at end of this review. But it's also important with respect to other baselines. For example, if Z_c can be perfectly reconstructed from X, then because Z_c is sufficient for Y under the assumed causal model, we would expect vanilla ERM to work for this problem! It's important to discuss how this, and other distributional approaches, might fail. 

Finally, there are some serious problems with clarity. As particular examples,
1. it's unclear what the data setup here is. What qualifies as out of domain? What do we assume access to at training time? The setup seems to be non-standard, but it's not explicitly spelled out 
2. How exactly are n and z meant to differ? As latent variables, they are literally indistinguishable in the model in figure 2 (the nodes could be collapsed together without any observable implication)


Some related work on domain adaptation with latent variables following a causal structure: 
1. https://proceedings.neurips.cc/paper/2021/hash/a8f12d9486cbcc2fe0cfc5352011ad35-Abstract.html
2. https://arxiv.org/abs/2006.07500
3. https://arxiv.org/abs/2106.00545
4. https://arxiv.org/abs/2208.06987
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: Although the core idea of applying non-linear identification to domain adaptation is interesting, this paper is not ready for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies multi-source domain adaptation. Instead of learning domain-invariant features to address the conditional shift, this paper assmes latent covariate shift and proposes latent causal model to formulate the data and label generating process. The identifiability of the proposed model is analyzed. By integrating the identifiability with the dependence between nc and y, this paper proposes a method to learn the invariant conditional distribution pD(y|zc). Experiments on both synthetic and real datasets are conducted.
strength_and_weaknesses: Strengths:
1. The idea of considering latent covariate shift is interesting and the motivation is clearly explained with examples.
2. By combinining difference components, the proposed method achieves superior performance as compared to the compared baselines.


Weaknesses:
1. The latent casual model is naive simply by combining existing disentanglement and casual methods. The insights behind such combination are insufficient.
2. The technical contribution is limited. The different components of the method, such as Gaussian prior, the mutual information and its approximation, entropy regularization, independence enhancement, are all existing techniques.
3. The commonly used datasets for multi-source domain adaptation, such as Digits-five, Office-Home, and DomainNet, are not considered. Due to page limit, I expected to see the results in supplementary material.
4. It is strange that when KL increases from 0.3 to 0.7, the performance of the baselines increase while the proposed method decreases. There is no analysis on the reasons.
Commonly used visualizations for domain adaptation, such as GradCam, are missing. t-SNE is not convincing.
5. The contributions of this paper are not well summarized.
6. Some closely related recent methods on multi-source domain adaptaion are not introduced and compred, such as "Self-paced Supervision for Multi-Source Domain Adaptation", "MADAN: multi-source adversarial domain aggregation network for domain adaptation".
7. The presentation needs to be improved. There are some simple grammar errors, such as "a image"->"an image", "nc play a role"->"nc plays a role". The full names of some abbreviations should be given for the first time use, such as ICA. The format of references, especially the names of conferences and journals, is inconsistent.

clarity,_quality,_novelty_and_reproducibility: The assumption on latent covariate shift for domain adaptation is interesting and makes sense. However, the technical contribution is limited and the experiments are insufficient, which restrict the originality and qulity. The main idea and method are well organized. Important implementation details are missing and the code is not provided, based on which the method and results are not easy to be reproduced.
summary_of_the_review: Interesting idea, limited technical contribution, insufficient experiments and analysis and just so-so presentation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors argue that existing works on domain adaptation are restricted to assuming covariate shift or conditional shift to interpret the change of the joint distribution and thus have their limitations in real applications. They propose a new latent casual model to formulate the generative process of input features and labels, they evaluated their proposed approach on synthetic and two real datasets. 
strength_and_weaknesses: This paper proposes to solve an important question in multi-source domain adaptation. Even though the latent content variable can only be identified together with the nonlinear ICA, the proposed framework shows the potential to solve this challenge. However, I think the current results section is weak, and the interpretation is not enough to demonstrate the superiority of this method.
clarity,_quality,_novelty_and_reproducibility: The idea to solve the multi-source domain adaptation through non-linear identification is interesting, however, I think the current paper demonstration is not enough to support its claim. 
summary_of_the_review: My overall impression is that the current experiments and results section is somewhat weak in supporting the paper's claim. For example:

1. The authors probably should consider adding evaluation metrics in the experiment section and why it is valid.

2. The introduction for the two real datasets is not clear to me, e.g. the motivation to choose these datasets; what does L7, L28, L43, and L46 mean in Terra Incognita?; because the authors modified the original PACS dataset to perform the experiments will that introduce inductive bias for the evaluation? 

3. The Interpretation of the results is not convincing to me.  In Table 1, the authors argued that with the increase of KL divergence of label distribution, the performance of MCDA, M3DA(typo here, it should be M3SDA), LtC-MSDA and T-SVDNet gradually degenerates. However, in Table 1 we can see that MCDA and LtC-MSDA performed better when DKL=0.5 compared to DL=0.3 and then decreased under DKL=0.7 which is contradictory with the conclusions in the results. Secondly, there is only one sentence: "due to our theoretical supports", to support why iLCC-MSDA performs the best compared with IRM, IWCDAN and LaCIM.

4. L7 appeared in the results section very suddenly, it was not mentioned earlier and I feel it is very difficult to follow the logic there.

Besides, I also want to ask the authors how do you distinguish nc and ns, zc and zs in your model framwork?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Although this paper received good appreciations about the idea of latent covariate shift, reviewers' ratings are all below threshold, and the situation is not improved after authors' rebuttal.
Main raised issues concern the insufficient novelty and technical contributions, unclear assumptions under which this approach should work and the positioning wrt the current domain adaptation methods (while also missing to quote some related paper in the state of the art), weak experimental analysis, insufficient to justify the paper claims and, especially, the clarity of the presentation. 
Authors provide replies for such remarks, but they did not result all so convincing, and the reviewers maintained their original evaluations.
For these reasons, this paper cannot be accepted to ICLR 2023.
Decision: Reject