Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This paper derives estimators for regression under additive Gaussian noise and distributional shift (mostly covariate and also some model shift) that enjoy minimax properties.

### Strengths
+ Sample complexity bounds, let alone matching lower bounds, are only recently being proposed for various realistic frameworks with distributional shift. This paper is thus timely and well-motivated.
+ The presentation is clear, and most analysis steps are well-delineated.
+ The insight that linear sufficient statistics could achieve minimax rates is valuable and the comparison to ridge regression, especially when accounting for the target, is enlightening.

### Weaknesses
- Some of the assumptions are rather strong. Assuming that the target covariance $\Sigma_T$ is known is somewhat acceptable. But when dealing with model shift, the paper assumes knowing the entire marginal distributions in the source $p_S$ and target $p_T$ (to get the importance weights $\mathbf{w}$), which effectively sidesteps the covariate shift entirely.
- ~~*[Major]* In the proof of Claim 3.1 (page 12), a key step appears wrong to me: namely the step that says “Cross term vanishes”. While it’s true that  $W^\intercal X_S = 0$, the outer product $X_S W^\intercal$ is not, and neither is $X_S A_1^\intercal A_2 W^\intercal$ that appears in the cross product and which could contribute negatively with varying $A_2$. Now it could be that this claim can be patched, which is why I judge the rest of the paper by assuming it’s true. I do the same for Claims 3.6 and 3.8, which suffer from the same issue. (Claim 3.8 just relies on the previous two, and Claim 3.6 omits the same  cross term on the third line of page 16, which invalidates the argument on the first two lines of the paragraph that follows “it only appears in the third and last non-negative terms”.)~~

### Comments
* Throughout $X$ is assumed to be centered, I believe. Best to state this explicitly.
* The definition of a sufficient statistic in footnote 2 is not the right one to use, since we cannot talk of independence when $\beta^\star$ is an unknown parameter. (A corresponding justification of (1) as sufficient statistic should then be given, using sufficiency of optimal estimation.)
* Typos:
   - the dimension of $X_S$ is $n_S \times d$ (subscript $S$ missing)
   - in the last equation of page 4 I believe the reference should be to Claim 3.1
   - in section 4 $\bar\beta_T$ needs a division by $n_T$
   - in the proof of Claim 3.1, “This is to show […] minimax *linear* estimator”
   - in the proof of Theorem 3.2, second line, divide $\mathbf{y}_T$ by $n_S$ and insert a missing $=$ sign before $\theta^\star_T$; in the third line change the subscript of $\Sigma$ to $T$ in the definition of $\Theta$; generally make sure the bolding convention is kept in this (otherwise rather nice) proof

### Overall

Minimax estimators for distributional shift can be very useful. But the paper has some lack of clarity and potential flaws, which is the only reason behind my current rating.

_[Edit: The authors clarified that the flaw I was talking about isn't there. I updated my review accordingly. The score is based on the fact that the paper can use some polish to improve clarity and the fact that some of the assumptions, even if traditional, are too strong (in essence, sample complexity bound will become meaningless if what is assumed known itself is too sample-intensive to approximate).]_

review_title: Good paper on minimax linear estimators under distributional shift, with some lack clarity.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary
The authors propose a near minimax optimal linear estimator under distribution shift. They have estimators for when there is a covariate shift (i.e. the underlying data distribution) or model shift (i.e. the distribution of the label given the features of the data).

## Strengths
* The authors provide bounds for both data coming from linear and non-linear generative models
* The authors analyze model shift in addition to covariate shift
* Experiments showing the results on simulated data

## Weaknesses
* No experiments for the non-linear data generation model

Update: after reading the arguments and comments from my fellow reviewers, I was convinced by their reasoning that the paper did not merit my original high score. I would like to therefore change my score to a 6.



review_title: Paper that has good contributions for analyzing covariate shift. Algorithms and theory is supported by experiments.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the problem of domain adaptation for linear regression problem.

1. First the authors consider a well-specified model where only the distribution of the covariates changes. For this case the authors propose a linear estimator, and show that under assumptions which guarantees the reduction of the problem to Gaussian sequence, this estimator is within a constant factor from the minimax risk of non-linear estimators. 
2. Then, the authors turn their attention to the miss-specified case of the above model. They again develop an estimator whose maximal risk is shown to match the minimax one (for linear estimators).
3. The authors consider a well-specified model allowing the vector $\beta^*$ to be changes from source to target. Similar results as for the previous case are derived.
4. The authors conduct simulation study comparing their estimators with the ridge regression.


pros: the developed estimators are theoretically grounded

cons: after several readings I still cannot understand why this work is on domain adaptation. Since the authors consider the problem of linear regression with fixed design, the domain adaptation part simply reduces to the re-weighting of the norm.

I am leaning towards rejection, but I am willing to raise the score if the authors are able to address my questions below.

Questions:
1. The authors work with the fixed design setup, that is, the expectation in the risk is taken only w.r.t. the signal $y_S$. What is the meaning of source and target data when the designs are fixed? For instance, in section 3.3. can we simply pull together $y_S, y_T$ and $X_S, X_T$ and apply the result of previous section? 
2. The authors should clarify how their first contribution is different from that of **Blaker, Minimax estimation in linear regression under restrictions, 2000** (Theorem 1 and the discussion after it). 
3. In Theorem 3.4 there is a condition that $\Sigma_S, \Sigma_T$ should commute. Isn't it $\hat{\Sigma}_S$ instead? If not, what is the reason to introduce this assumption if the design is fixed?
4. If the covariance matrix on the target is unknown, we would have to estimate it using unlabelled sample. Did you investigate a theoretical or empirical impact on the problem?

Suggestion: It might be useful to write few lines explaining why the excess risk on the target reduces to the 2-norm re-weighted by the target covariance.


After rebuttal: They authors have addressed my questions. I hope that the authors will carefully review classical literature on weighted minimax linear estimator in their final version. I raise my score to marginally above acceptance threshold, since I am still not convinced in the importance of domain adaptation for fixed design regression. 
review_title: Review

Meta-review:  
This paper derives estimators and minimax guarantees for regression under additive Gaussian noise and distributional shift 
Despite, some merits raised, limitations on too strong assumptions (knowing the entire marginal distributions, sample complexity bound that could become meaningless if what is assumed known itself is too sample-intensive to approximate) were considered important drawbacks by the reviewers.
Last but not least, the importance of domain adaptation for fixed design regression was questioned and the answer to that point was not convincing enough. 
Decision: Reject