Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of reweighting training samples to improve model performance on out-of-distribution test samples. The approach formulates real distribution shifts (covariate and concept related) using the exponential tilt assumption. With this assumption, the problem of improving performance on OOD samples simplifies to learning data importance weights. The paper has some theoretical analysis on the properties of exponential tilting. The paper also applies this method to improve performance on Waterbirds and BREEDS-Entity30. 
strength_and_weaknesses: Strengths 

- The paper is well-written. The exponential tilt model is clearly explained. 
- Analyzing learned importance weights. 


Weaknesses
- Exponential tilt assumption unjustified. It is not clear why complex distribution shits in practice should be parameterized with the exponential tilt model. The paper states that this problem can be applied to concept drift settings but has no experiments / analysis on concept drift.

- When does this re-weighting approach fail? A more quantitative approach to this question would be insightful. The paper loosely talks about this  (i.e. distribution needs some overlap etc).

- Limited empirical evaluation. Waterbirds and Entity30 are datasets where you know what the "ground-truth" importance weights should look like (https://proceedings.mlr.press/v177/idrissi22a.html). Showing that the proposed methods works well in these settings is a good first paper, i think the paper will be significantly stronger if the paper improves performance on more general / realistic distribution shifts (e.g. CIFAR 10.2, ImagenetV2) and then analyzes the learned importance weights. The empirical section should have additional baselines (e.g. https://proceedings.mlr.press/v162/zhou22d/zhou22d.pdf, https://proceedings.mlr.press/v177/idrissi22a.html) to clearly contrast this approach from previous methods.

- Theoretical analysis somewhat tangential and not insightful vis-a-vis the paper's main focus. I would rather first read whether this method works well on realistic distribution shifts and then discuss properties like consistency and identifiability of the parameters. 

Overall the paper is well-written and focuses on a principled approach (reweighting) to improve OOD performance. However, the paper has two major issues: (a) exponential tilt assumption is not clearly justified, (b) empirical evaluation is quite limited.

clarity,_quality,_novelty_and_reproducibility: See strengths and weaknesses above
summary_of_the_review: See strengths and weaknesses above
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a method for domain adaptation when you have labelled data from source P and unlabelled data from target Q. Their method learns a reweighting of the source data sample through a learned exponential tilting, where a neural network learns the sufficient statistics of the tilting. They show that this tilting is identifiable when "anchor sets" exist (a subset of points which only have probability mass for their class label) in the source distribution, and show that when identifiable, the parameter estimate converges. Experimentally, they show that this method is competitive on domain generalization benchmarks.
strength_and_weaknesses: This seems like technically strong work, and the proofs look sound and interesting. The experimental work supports the theoretical components and shows that this method is an interesting possibility for domain adaptation/generalization without group labels.

Some feedback:
- the authors argue that their method moves beyond covariate shift to concept drift - however, the experimental datasets (Waterbirds + Breeds), do not provide any instances of concept drift, only testing covariate shift. Experiments that test the method in a concept drift setting would make the paper's argument stronger
- Prop 4.2: I don't understand what the "T(S_k) is p-dimensional" portion of this assumption means - I think if I understood this piece, then the identifiability argument + concept drift application might be clearer to me. Also, in the appendix, it seems to say d-dimensional instead
-Assumption 4.5 - I don't totally understand this - I've seen constructions like it before but I don't recall how the sequence r_np is used in this statement and it's not clear from the body of the text: would be good to have this be explicit somewhere.
- Experiments: I don't understand why the method "\pi T -> T" doesn't perform close to perfectly on some of these target domain groups. In Waterbirds, some pairs of domains have all the same label (I think for instance environment 0 and 1 both are all Y=0). Then, shouldn't finetuning on data from those domains yield a really easy fine-tuning problem - learning a constant output?

- page 3, choosing T: the intuition provided her around how T should identify "subpopulations" isn't quite clear to me, would appreciate a bit more of a walkthrough of this
- Thm 4.6 - where is M used here? I don't see it but might be missing it
- Lemma 4.8: remind us what G is here, I don't remember

-middle of p7: guaranty -> guarantee


clarity,_quality,_novelty_and_reproducibility: Clarity: mostly good, a couple of small confusions
Quality: I think the technical work here is pretty good (although not an expert)
novelty: seems like a novel approach to a domain generalization-type problem without group information
Reproducibility: details provided are pretty good, seems reproducible
summary_of_the_review: I think the technical work here provides a clean, effective direction to building approaches for domain adaptation/generalization without group labels. My main question/confusion with the work is re: its connection to concept drift, and I'm not convinced by the paper that this approach is relevant. But even without that, I think the work is good, so I'm recommending acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a framework for computing importance weights for labeled source data with respect to a target domain where only unlabeled target data is available. Unlike previous methods, these weights allow for concept drift in addition to covariate shift. The paper proves the statistical consistency of the estimators and generalization bounds and demonstrates the efficacy of the methods over two data sets.
strength_and_weaknesses: Strengths
- The exponential tilt model is well described and the procedure for obtaining the importance weights is clear
- The weights demonstrate value in both domain adaptation and evaluation
- I did not thoroughly check the theoretical proofs but the general steps seem correct

Weaknesses
- Since the method is supposed to work also on domain adaptation, it would be nice to have presented results on some more standard domain adaptation tasks and baselines (e.g., Office).
clarity,_quality,_novelty_and_reproducibility: The work is clearly described. Although importance weights are common in domain shift problems, the paper presents an interesting way to use them that permits novel use cases.
summary_of_the_review: Overall, the paper seems strong albeit with limited experiments vs baselines. The theoretical results are nice and the insights are well-supported.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates the distribution shift problem in which train and test distributions can differ and one has access to the unlabeled data from the test domain. The paper adopts a reweighting approach, which is quite common in the literature. The goal is to learn a weight function w(X,Y) such that the moment conditions

E[w(X_P,Y_P)g(X_P,Y_P)] = E[g(X_Q,Y_Q)]

are fulfilled for all real-valued function g(X,Y). Motivated by the rich theory of exponential families in statistics, the authors propose a parametric model of the weight function, called the exponential tilt model

w(x,y) = exp(a_y.T(x) + b_y)

where T(x) is a vector of sufficient statistics and a_y,b_y are the parameters. The paper proposes ways to construct T(X), to learn the parameters a_y,b_y as well as the corresponding weights. The paper also provides theoretical analyses of the proposed method. The empirical results on two standard benchmark datasets are promising.

Overall, there is a general consensus among the expert reviewers that this is a strong paper despite some reservations. The aforementioned moment conditions are well-known and the reweighting approach is standard in the distribution shift literature. What makes this paper interesting however is the introduction of a "parametric exponential tilt model" on the weight function, allowing the in-depth theoretical analysis and capability of the model to deal with the "concept drift" scenario. Note that this concept drift scenario would have been much more challenging in a nonparametric setting. I find this aspect of the paper quite interesting and would be beneficial for other researchers working in this area. 

Some caveats of the paper include weak justification of the exponential tilt model and limited experiments/baselines. I would strongly encourage the authors to improve these two limitations in the camera-ready version.

As a result, I recommend this paper with expected minor improvements in the camera-ready for publication at ICLR 2023.
justification_for_why_not_higher_score: There are some reservations from the reviewers. 
justification_for_why_not_lower_score: Despite some reservations, the paper proposes a well-supported method for dealing with the distribution shift problem.
Decision: Accept: poster