Maximum Likelihood Estimation is All You Need for Well-Specified Covariate Shift
Authors: Jiawei Ge
Shange Tang
Jianqing Fan
Cong Ma
Chi Jin
Venue: ICLR 2024 poster (2024)
Keywords: Covariate shift; Maximum Likelihood Estimation; Out-of-Distribution generalization;
Primary area: learning theory

Abstract: A key challenge of modern machine learning systems is to achieve Out-of-Distribution (OOD) generalization---generalizing to target data whose distribution differs from that of source data. Despite its significant importance, the fundamental question of ``what are the most effective algorithms for OOD generalization'' remains open even under the standard setting of covariate shift.
This paper addresses this fundamental question by proving that, surprisingly, classical Maximum Likelihood Estimation (MLE) purely using source data (without any modification) achieves the *minimax* optimality for covariate shift under the *well-specified* setting. That is, *no* algorithm performs better than MLE in this setting (up to a constant factor), justifying MLE is all you need.
Our result holds for a very rich class of parametric models, and does not require any boundedness condition on the density ratio. We illustrate the wide applicability of our framework by instantiating it to three concrete examples---linear regression, logistic regression, and phase retrieval. This paper further complement the study by proving that, under the *misspecified setting*, MLE is no longer the optimal choice, whereas Maximum Weighted Likelihood Estimator (MWLE) emerges as minimax optimal in certain scenarios.

Published as a conference paper at ICLR 2024
MAXIMUM LIKELIHOOD ESTIMATION IS ALL You NEED FOR WELL-SPECIHEDD COVARIATE SHIFT
Jiawei Ge
Shange Tang
Jianqing Fan
Cong Ma
Chi Jin
ABSTRACT
A key challenge modern machine learning systems is to achieve Out-of- Distribution (OOD) generalization generalizing tion differs from that of source data. Despite its significant importance, the fun- damental question of "what the most effective algorithms for OOD general- ization" remains open even under the standard setting of covariate shift. This paper addresses this fundamental question by proving that, surprisingly, classical Maximum Likelihood Estimation (MLE) purely using source data (without any modification) achieves the minimax optimality for covariate shift under the well- specified setting. That is, no algorithm performs better than MLE in this setting (up to constant factor), justifying MLE very rich class of parametric models, and does not require any boundedness con- dition on the density ratio. We illustrate the wide applicability by instantiating it to three concrete examples- linear regression, logistic regres- sion, and phase retrieval. This paper further complement the study by proving that, under the misspecified setting, MLE is no longer the optimal choice, whereas Maximum Weighted Likelihood Estimator (MWLE) emerges as minimax optimal in certain scenarios.
g to target data whose distribu-
you need. Our result holds for a
our framework
INTRODUCTION
Distribution shift, where the distribution of test data (target data) significantly differs from the dis- tribution of training data (source data), is commonly encountered in practical machine learning scenarios (Zou et al., 2018; Ramponi & Plank, 2020; Guan & Liu, 2021). A central challenge of Dii-r---iitiiittonn (OOD) generalization, where learned
modern machine learning is to achieve models maintain good performance in the target domain despite the presence of distribution shifts. To address this challenge, a variety of algorithms and techniques have been proposed, including vanilla empirical risk minimization (ERM) (Vapnik, 1999; Guirajani & Lopez-Paz, 2020), impor- tance weighting (Shimodaira, 2000; Huang et al., 2006; Cortes et al., 2010b; Cortes & Mohri, 2014), learning invariant representations (Ganin et al., 2016; Arjovsky al., 2019; Wu al., 2019; Rosen- feld et al., 2020), distributionally robust optimization (DRO) (Sagawa et al., 2019), etc. See the recent survey (Shen et al., 2021) for more details. These results claim the effectiveness of the corre- sponding proposed algorithms in different regimes. This leads to natural fundamental question:
What are the most effective algorithms for OOD generalization?
This paper consider a widely-studied formulation of 200D-generllizaioonnccom covariate shift, the marginal distributions of the input covariates X vary between the source and target domains, while the conditional distribution of output given covariates Y X remains the same across domains. We consider learning a model from a known parametric model class under well-specified setting, where well-specification refers to the problems where the true conditional distribution Y lies the given parametric model class. We argue that well-specified setting
shift. Under
"equal contribution 'Department of Operations Research and Financial Engineering, (ig5300, shangetang, (arffan)prrineeeee edu Department of Statistics, University of Chicago; congmluchts 6 Department of Electrical and Computer Engineering, Princeton University; chi jēpri incetoo. edu
Princeton
University;

---

Published as a conference paper at ICLR 2024
becomes increasingly more relevant in modern learning applications, because these applications typ- ically use large-scale models with an enormous number parameters, which are highly expressive and thus make the settings "approximatel"" well-specified.
Unfortunately, even under the basic setup of well-specified covariate shift, the aforementioned high- lighted problem remains elusive while the seminar work Shimodaira (2000) provides the first asymptotic guarantees for classical Maximum Likelihood Estimation (MLE) algorithm under this setup, and proves its optimality among a specific class of weighted likelihood estimators, his results leave two critical questions open: (1) Does MLE remain effective in the practical non-asymptotic scenario when the number of data is limited? (2) Do there exist smart algorithms beyond the class of weighted likelihood estimators that outperform MLE? This paper precisely addresses both critical questions and thus resolving the highlighted problem under well-specified covariate shift.
Our contributions. Concretely, this paper makes following contributions:
1. We prove that, for a large set of well-specified covariate shift problems, the classical Maximum Likelihood Estimation (MLE) which is computed purely based source data without using any target data- - finds the optimal predictor on the target domain with prediction loss decreases as O(Tr(ZzIz')/n) I/n). Here Tr(-) standards for trace, Ig, Ir are the fisher information under source and target data distribution respectively, and is the number of source data. Our result does not require any boundedness condition on the density ratio, and is, to our best knowledge, the first general, non-asymptotic, sharp result for MLE on rich class of covariate shift problems.
2. We provide the first minimax lower bound under well-specified covariate shift for any algorithm, matching the error rate of MLE. This implies that MLE is minimax optimal, and no algorithm is better than MLE in this setting (up to constant factor), justifying "MLE is all you need".
3. We instantiate our generic results by considering three representative examples with distinct prob- lem structures: linear regression, logistic regression and phase retrieval. We verify preconditions, compute key quantities, and directly give covariate shift guarantees for these applications.
4. We further complement the study of this paper by considering the mis-specfied setting where MLE ceases work. We establish the first general, con-asymptotic upper bound for the Maxi- mum Weighted Likelihood Estimator (MWLE) provided bounded likelihood ratio. We prove that MWLE is minimax optimal under certain worst-case mis-specification.
MLE versus MWLE. This paper shows that importance weighting should not always be the go-to algorithm for covariate shift problems. Despite MWLE works under more general mis-specified setting given bounded density ratio, in the well-specified regime, MLE does not require bounded density ratio, and is provably more efficient than MWLE in terms of sample complexity. MLE is all you need for well-specified covariate shift problem.
1.1 RELATED WORK
Parametric covariate shift. The statistical study of covariate shift under parametric models can be dated back to Shimodaira (2000), which established the asymptotic normality of MWLE and pointed out that vanilla MLE is asymptotically optimal among all the weighted likelihood estimators when the model is well-specified. However, no finite sample guarantees were provided, and the optimality of MLE is only proved within the restricted class of weighted likelihood estimators. In contrast, this paper establishes non-asymptotic results and proves the optimality of MLE among all possible estimators under well-specified models. Cortes et al. (2010a) studied the importance weighting under the statistical learning framework and gave a non-asymptotic upper bound for the generalization error of the weighted estimator. However, their rate scales as O(1/ Vn) compared to our rate (1/nn). where n is the sample size. A recent line of work also provide non- asymmpttiiicc analyses for covariate shift under well-specified setting, however they focus on linear regression or a few specific models which are more restrictive than our setting: Mousavi Kalan et al. (2020) introduces a statistical minimax framework and provides lower bounds for OOD generalization in the context of linear and one-hidden layer neural network regression models. When applied to covariate shift, their lower bounds are loose and no longer minimax optimal. Lei et al. (2021)
considers the minimax optimal estimator for linear regression under fixed design, the estimator they proposed is not MLE and is much more complicated in certain regimes. Finally, Zhang et al. (2022) considers covariate shift in linear regression where the learner can have access to small number of
2

---

Published as a conference paper at ICLR 2024
target labels, this is beyond the scope of this paper, where we focus on the classical covariate shift setup in which target labels are not known.
Nonparametric covariate shift. Another line of work focuses well-specified nonparametric models under covariate shift. Kpotufe& & Martinet (2018) presented minimax results for nonpara- metric classification problem, which was controlled by transfer-exponent that measures the dis- crepancy between source and target. Hanneke & Kpotufe (2019) delves deeper into the advantages of unlabeled target data and demonstrates that typical importance sampling methods do not offer any improvements over the minimax rate already achieved by ERM. Inspired by the aforementioned work, Pathak et al. (2022) studied nonparametric regression problem over the class of Hölder contin- uous functions with a more fine- the-grained similarity measure. When considering reproducing kernel Hilbert space (RKHS), Ma et al. (2023) showed kernel ridge regression (KRR) estimator with a properly chosen penalty is minimax optimal for large family of RKHS when the likelihood ratio uniformly bounded, and reweighted KRR using truncated likelihood ratios minimax opti- mal when the likelihood ratio has finite second moment. Later, Wang (2023) proposed learning strategy based on pseudo-labels. When the likelihood ratio is bounded, their estimator enjoyed the optimality guarantees without prior knowledge about the amount of covariate shift. Although these works focused on covariate shift problems, they considered nonparametric setting, and hence are not directly comparable our work. As an example, Ma et al. (2023) showed that MLE (empiri- cal risk minimization in their language) is provably suboptimal for addressing covariate shift under nonparametric RKHS assumptions. In contrast, we show that MLE optimal for covariate shift for a well-specified parametric model. We also highlight that our lower bound is instance dependent in the sense that depends on the source and target distributions. This is in contrast prior work (e.g. Ma et al. (2023); Kpotufe & Martinet (2018); Hanneke & Kpotufe (2019)) that consider the worst-case scenario over certain classes of source-targer target pairs (e.g., bounded density ratios).
Maximum likelihood estimation. A crucial part of this work is analyzing MLE, which is a domi- nant approach in statistical inference. There exists a variety of work studying the behavior of MIL under the standard no-distribution-she
setting. It is well known that MLE is asymptotically normal (Casella & Berger, 2021) with the inverse of Fisher information as the asymptotic variance. Cramér (1946); Rao (1992) established the famous Cramer-Rai Rao bound for unbiased estimators, which also showed that no consistent estimator has lower asymptotic mean squared error than the MLE. White (1982) gave the asymptotic distribution of MLE under the mis- mis-specified setting. More recently, non-asymptotic behaviours of MLE are studied under certain models. Bach (2010); Ostrovskii & Bach (2021) established the non-asymptotic error bound for MLE in logistic regression using self- concordance. This line of work does not consider covariate shift, which is an indispensable part of this paper.
Importance reweighting algorithms. Lastly, importance reweighting (or importance sampling) classical method to use independent samples from a proposal distribution to approximate ex- pectations w.r.t. target measure (Agapiou et al., 2017). Chatterjee & Diaconis (2018) studied the sample size (depending on the KL divergence between two distributions) required for importance sampling to approximate single function. Sanz-Alonso (2018) extended analysis to the case with general /ddivergences. Zhai et al. (2022) studied overparametrized linear models and showed that Generalized Reweighting (GRW) algorithms, upon achieving zero training error, yield the same results as ERM, thus offering no advantage over it. In addition to correcting covariate shift, impor- tance reweighting has been central in offline reinforcement learning. For instance, Ma et al. (2022) showed a truncated version of importance reweighting is minimax optimal for estimation the value of target policy using data from behavior policy. For learning the optimal policy from the be- havior data, Swaminathan & Joachims (2015) presented upper bounds of an importance-reweighe estimator. This spurs long line of work of using importance weighting in offline RL. See the recent work Gabbianelli et al. (2023) and the references therein.
2 BACKGROUND AND PROBLEM FORMULATION
In this section, we provide background on the problem of learning under covariate shift. We also review two widely adopted estimators: maximum likelihood estimator and maximum weighted like- lihood estimator.
3

---

Published as a conference paper at ICLR 2024
Notations. Throughout the paper, to denote universal constants, which may vary from line to line.
2.1 COVARIATE SHIFT AND EXCESS RISK
Let X X be the covariates and Y y be the response variable that aim to predict. In general out-of-distribution (OOD) generalization problem, we have two domains of interest, namely source domain S and a target domain T. Each domain associated with data generating distribution over (X,Y): Y): (X, Y) for the source domain and (r Y) for the target domain. Given n i.i.d. labeled samples {(r;, }=
Ps(X, Y) from the source domain, the goal of OOD generalization is to learn prediction rule X + Y that performs well in the target domain. In this paper, we focus on the covariate shift version of the OOD generalization problem, in which the marginal distributions (X( and (X)) of the covariates could differ between the source and target domains, while the conditional distribution Y X is assumed to be the same on both domains.
More precisely, adopt the notion of excess risk to measure of the performance of an estimator under covariate shift. Let FF := {f(u] x; B) 33 E Rd) be a parameterized function class to model the conditional density function ply x)) Y X. A typical loss function is defined using the negative log-likeldoood function E(r,y. = log fly x;; B). The excess risk at / is then defined
(1) where the expectation Er taken over Pr(X,Y). Y). When the model well-specified, i.e. when the true density p(y æ) = f(y æ; 3 for some 8*, we have inf Er[((x, y, B)] = Er [f(z, (8)]] As a result, we evaluate the loss at B against the loss at the true parameter (3*. contrast, in the case of mis-specification, i.e., when p(y x) & F, the loss at 33 is compared against the loss of the best fit in the model class.
R(3) =- Er [6( y, 3)] - inf Er [((x, y, B]],
2.2
MAXIMUM LIKELIHOOD ESTIMATION AND ITS WEIGHTED VERSION
In the no-covariate-shift case, maximum likelihood estimation (MLE) is arguably the most popular approach. Let
(n( ΣΣ Mxi 3).
be the empirical negative log-likelihood using the samples ((x, 1)) from the source domain. The vanilla MLE defined as SMLE == arg min Cn
One potential "criticism" against MLE in the covariate shift setting is that the empirical nega- tive log-likelihood not faithful estimate of the out-of-distribution generalization performance, ie., Er [f(x, ()]. In light of this, weighted version of MLE is proposed. Let w(x) == dPr (z) s (x) be the density ratio function and
(3) = | Σm w(x;))(x; B).
be the weighed loss. Then the maximum weighted likelihood estimator is defined as 3MWLE == arg ming Cy (B). It easy to see that the weighted loss is unbiased estimate of EE [f(z, v, 3)].
To ease presentations later, we would also recall the classical notion of Fisher information-an im- portant quantity to measure the difficulty of parameter estimation. The Fisher information evaluated at on source and target is defined as
Is (3) (x)Ps(X).v a~ mud)
V(( B) (r) E Ex-Pr(X),) 2~f(u x1))
V V(((x,,,8)
Here, the gradient and Hessian are taken with respect to the parameter B.
3 WELLI-SPECIFIED PARAMETRIC MODEL UNDER COVARIATE SHIFT
In this section, we focus on covariate shift with a well-specified model, that is, the true conditional distribution falls in our parametric function class. This setting aligns with the practice since in modern machine learning we often deploy large models whose representation ability are so strong that every possible true data distribution almost falls in the function class. We assume there exists
4

---

Published as a conference paper at ICLR 2024
some 3* such that p z) = fly 2; 8*), and denote the excess risk evaluated at 3 under true model
parameter 1 as (gg.(8),cc ie..
Rg. (B) == E y fy)l:())))
P~Pr(X)
[e(x), y, B)]
E
a~Pr(X) v|a~f(g(e;3")
[e(x, y, 8,)]_
(4)
While the objective of MLE not an unbiased estimate of the risk under the target domain, we will show in this section that MLE is in fact optimal for addressing covariate shift under well-specified models.
More specifically, in Section 3.1, we provide the performance upper bound for MLE under generic assumptions on the parametric model. Then in Section 3.2, we characterize the performance limit of any estimator in the presence of covariate shift. As we will see, MLE is minimax optimal as it matches the performance limit.
3.1 UPPER BOUND FOR MLE
In this subsection, establish non-asymptotic upper bound for MLE under generic assumptions on the model class.
Assumption A. We make the following assumptions on the model class F:
A.I There exist B1, B2, (8)) and absolute constants c, such that for any fixed matrix A Raxd any E (0, 1). and any n > N (6), with probability at least 1 6:
V log
B ||||A|12
log
- -IV S <oV
+ Bil|AA12 log"
(5)
VV
n
log Th
[|v2)n(8)) - E| ( - B21
(6)
n
where == n. -
111 is the variance.
A.2 There exists some constant B3 such that [V y, 3)112 S for all x E X's Ur E
y, B E Rd, where Xs (resp. XT) is the support If s(X) (resp. Pr(X)).
A.3 The empirical loss (.(:) defined in (2) has a unique local minimum in Rd, which is also the global mininum..
Several remarks on Assumption A are order. Assumption A.1 is a general version of Bernstein inequality (when
0 it reduces to classical Bernstein inequality). which gives concentration on gradient and Hessian. This assumption naturally satisfied when the gradient and Hessian are bounded (see Proposition D.2 for details). Assumption A.2 requires the third order derivative of log- likelihood to be bounded, which is easy to satisfy (e.g., linear regression satisfies this assumption with B3 0). Assumption A.3 ensures the MLE is unique, which standard in the study of the behaviour of MLE. We can see that it naturally applies to traditional convex losses. It is worth noting that our general theorem can also be applied under a relaxed version of Assumption A.3, which will be shown in Theorem 4.5. In Section 4, we will see that Assumption A is mild and easily satisfied for a wide range of models.
Now we are ready to present the performance upper bound for MLE under covariate shift.
Theorem 3.1. Suppose that the model class F satisfies Assumption A. Let IT := IT(3*) and Is == Is(3t) For any E 1), if n 2 emax(N log(d/8), (6)), then with probability at least 28, we have R33- (BMLE) < Tr(IrTs") +) log for absolute constant Here N
Poly(d, B1, B2, B3. IZ 11,, 1 'I3 11 ))
For an exact characterization of the threshold N one can refer to Theorem A.1 in the appendix.
Theorem 3.1 gives non-asymptotic upper bound for the excess risk of MLE: when the sample size exceeds a certain threshold of max ( N log(d/8), (()).. MLE achieves an instance dependent risk bound Tr //n. is worth noting that our analysis does not require boundedness on the den- sity ratios between the target and source distributions (as have been assumed in prior art (Ma al., 2023)), which yields broader applicability. In Section 4, we will instantiate our generic analysis on three different examples: linear regression, logistric regression and phase retrieval.
5

---

Published as a conference paper at ICLR 2024
3.2 MINIMAX LOWER BOUND
In the previous section, we have established the upper bound for the vanilla MLE. Now we turn to the complementary question regarding the fundamental limit of covariate shift under well-specified models. To establish the lower bound, we will need the following Assumption B that a slight vari- of Assumption A. Different from the upper bound, the lower bound is algorithm independent and involve model class rather than a fixed ground truth. Hence, Assumption B focuses on population properties of our model as opposed to Assumption A, which is on the sample level.
Assumption B. Let 90 E TRd and B > O. We make the following assumplions on the model class F:
I Assumption A.2 holds.
B.2 There exist some constants Ls, LT 2 such that for any B1, .2 E Ba, (B):
(31) - Is((3)ll2 S Ls 82ll2, [[r((3) Ir (322 S Ir 82l12.
B.3 For any 8* (B),(B), the excess risk R3- (3) defined (4) is conver in 3 E Rd.
B.4 We assume [3) and (B) are positive definite for all Bs (B).
Assumption B.2 essentially requires the Fisher information will not vary drastically in a small neigh- bourhood of 30- This assumption is easy hold when the fisher information has certain smoothness (e.g., in linear regression, the fisher information does not change when 8 varies). Since Assumption is a slight variant of Assumption A, both assumptions are often satisfied simultancously for wide range of models, as we will show in Section 4.
Theorem 3.2. Suppose the model class F satisfies Assumption B. As long as n 2 No, we have
1
inf sup 8 BB
Tr (Z
E xxFFs(X)
Rg. (B) z 50n'
36 w|
where No = Poly(d, B-1 B3, Ls, LT, (b0)l ([t) 1,, (Bo)) | (5o) ll2).
For an exact characterization of the threshold No. one can refer to Theorem A.4 in the appendix.
Comparing Theorem 3.1 and 3.2, we can see that, under' Assumptions A and B, then for large enough sample size n, Tr (I, (5* 5 (5*)) /n exactly characterizes the fundamental hardness of covariate shift under well-specified parametric models. It also reveals that vanilla MLE is minimax optimal under this scenario. To gain some intuitions, I5 captures the variance of the parameter estimation, and Ir measures how the excess risk on the target depends on the estimation accuracy of the parameter. Therefore what really affects the excess risk (on target) the accuracy of estimating the parameter, and vanilla MLE is naturally the most efficient choice.
4 APPLICATIONS
In this section, we illustrate the broad applicability of our framework by delving into three distinct statistical models, namely linear regression, logistic regression and phase retrieval. For each model, we will demonstrate the validity of the assumptions, and give the explicit asymppotii upper bound on the vanilla MLE obtained by our framework as well as the threshold of sample size needed to obtain the upper bound.
4.1 LINEAR REGRESSION
In linear regression, we have Y = 33 + negative log-likelihood function (i.e. the loss function) is given by E(x, v, = 1 x" 3)2. We assume X ~ (0, Ia)
+ where s [O0 1) and E X. The corresponding
on the source domain and X ~ NN o2 on the target domain.
Proposition 4.1. The aforementioned linear regression model satisfies Assumption A and B with y= 1, N, = d logo( (1 /5), B = =Vd, B2 = evd. B3 and Lg = Lor = O. Moreover, we have Tr(I = || + odd
It is worthy to point out that, it is not hard for Assumptions A and B to be satisfied simultaneously, These assumptions will hold naturally when the domain is bounded and the likelihdddodd is of certain convexity and smoothness, will show in the next section by several concrete examples.

---

Published as a conference paper at ICLR 2024
By Theorem 3.1 and Theorem 3.2, since Assumption A and B are satisfied, we immediately demon- strate the optimality of MLE under linear regression. The following theorem gives the explicit form of excess risk bound by applying Theorem 3.1:
Theorem 4.2. For any å E (0, in 2 ONN log $), then with probability at least 1 - 26, we have Rg- (ŚMLE) S [Ilc (3) log where N == d (1+ IaalBa+oad Ialltooa
Remark (Excess risk). Regarding the upper bound of the excess risk, we categorize it into two scenarios: large shift and small shift. In the small shift scenarios (i.e.. ||a|13 S a²d), the result is the same as that in scenarios without any mean shift, with a rate of o²d/n. n. On the other hand, in the large shift scenarios (i.e.,
shift at a rate of ||a=|//m1 [| 2 dd), the upper bound of the excess risk increases with the mean
Remark (Threshold N). For a minor mean shift, specifically when ||a||2 = co for a given constant the threshold is N d. This aligns with the results from linear regression without any covariate shift. On the other hand, as the mean shift increases (i.e., |a/2 = ad for some 0 < k < 1/2), the threshold becomes N d'k+1, increasing with the growth of k. In scenarios where the mean shift
significantly surpasses the scaling shift, denoted 2 a dd, the threshold reaches N = d3.
4.2 LOGISTIC REGRESSION
In the logistic regression, the response variable Y E {0 obeys F(Y = 1| = x) = 1/(1 en "a), IY 0| = x) 1/(1 + x (*). The corresponding negative log-likelihood
function (i.e. the loss function) is given by {(z, y, B) == log(1 + === yxx² B). We assume X ~ Uniform(s(-1(Vd)) on the source domain and X ~ Uniform(s*- + on the target domain, where (v := {i E Rd ||]|| = vd). In the following, we will give the upper bound of the excess risk for MLE when v = r where 31 represents a vector perpendicular to 8 (i.e., 31 =0). Without loss of generality, we assume || = || ll2 1.
Proposition 4.3. The aforementioned logistic regression model satisfies Assumption A and B with 0. (6) 0, B₁ cvd, B2 = cd, B3 = (vd + r);, Ls d=. and LT = (Vd + r) Moreover, we have Tr(IrIg') x r2.
By Theorem 3.1 and Theorem 3.2, since Assumption A and are satisfied, we immediately demon- strate the optimality of MLE under logistic regression. The following theorem gives the explicit form of excess risk bound by applying Theorem 3.1:
Theorem 4.4. For any 8 E (0, 1), in O(N log 4 then with probability at least 1 28, we have R3. (SMLE) S (d+t") log where := d1(( + +
n
Remark (Excess risk). The bound on the excess risk incorporates a 7 term, which is a measurement of the mean shift. This is due to the fact that the MLE does not utilize the information that u 3 0. Therefore, v 3MLE is not necessarily zero, which will lead to an additional bias. Similar to linear regression, can categorize the upper bound of the excess risk into two scenarios: large shift (r 2 vd) and small shift (r S Va).
Remark (Threshold N). We admit that the N here may not be tight, as we lean on a general frame- work designed for variety of models rather than specific one.
4.3 PHASE RETRIEVAL
As we have mentioned, our generie framework can also be applied to the scenarios where some of the assumptions are relaxed. In this subsection, we will further illustrate this point by delving into the phase retrieval model. In the phase retrieval, the response variable = 3+)2 + e, where (0, 1) and IL X. We assume (s(X) and (X) follow the same distribution that in the logistic regression model (i.e., Section 4.2). Note that both the phase retricval model and the logistic regression model belong generalized linear model (GLM), thus they are expected have
similar properties. However, given the loss function ((3. == (y -xT 2))) it is obvious that Assumption A.3 is not satisfied, since if B is global minimum of bn. -B is also global minimum. The following theorem shows that we can still obtain results similar to logistic regression though Assumption A.3 fails to hold.
7

---

Published as a conference paper at ICLR 2024
Theorem 4.5. For any ò E (0, if ON log 2). then with probability least 1 28, we have Rg. (BMLE) ((4+)) )lobrrr)ooo log where N == (1 + r²)
MIS-SPECIFIED PARAMETRIC MODEL UNDER COVARIATE SHIFT
In the case of model mis-specification, we still employ parameterized function class F {f(y) a;; (8) E [R to model the conditional density function ply I) might not be in F. As previously showed, under well-specified parametric model, the vanilla MLE is minimax optimal up to constants. However, when the model is mis-specified, the classical MLE may not necessarily provide good estimator.
X. However, the true density
Proposition 5.1. There exist certain mii--peecffeed scenarios such that classical MLE is not consis- tent, whereas MWLE is.
Proposition 5.1 illustrates the necessity adaptation under model mis-specification since the clas- sical MLE asymptotically gives the wrong estimator. In this section, study the con-asymplotic property of MWLE. Let M be the model class the ground truth Y X, and M M the ground truth model for Y |XX
We denote the optimal fit on target as 3 (M) = arg min , B U. 3)]. The excess risk y/2~
evaluated at 3 is then given by RM (X)
[L(z, v. B)] (X) 1(
5.1 UPPER BOUND FOR MWLE
In this subsection, we establish the non- con-syymtoiiic upper bound for MWLE, as an analog Theo- rem 3.1. We make the following assumption which is modification of Assumption A.
Assumption C. We assume the function class F satisfies the follows:
C.1 There exists some constant W > 1 such that the density ratio w(x) S. W for all æ E X's U Nr. C.2 There exist B1, B2 and N (6)), and absolute constants c, such that for any fixed matrix A E Rdxd any 8 E (0, 1), and any (()), with probability at least 1 8:
Vlog
WB || || 12 VV
log
[ pM (3* (B* V
S SoV
+ WB ||/2////114 log"
n
n
log
BV
W BBV
n
where V n. EI (M)) B (M))) is the variance.
C.3 Assumption A.2 holds.
C.4 There exists (8) such that for any 8 E (0, 1) and any n 2 (8). with probability at least
1 8, the empirical loss f (-) defined in (3) has unique local minimum in RRd. which is also the global minimum.
Assumption C.1 is density ratio upper bound (not required for analyzing MLE), which is essential for the analysis of MWLE. Assumption C.2 is an analog of Assumption A.1, the sense that the empirical loss &, is replaced by its weighted version f Assumption C.4 is weaker version of Assumption A.3 in the sense that it only requires E"r has unique local minimum with high probability. This is due the nature of reweighting: when applying MWLE, w(x;) can sometimes be zero, which lead to the degeneration of 6 (with a small probability). Therefore we only require the uniqueness of local minimum holds with high probability.
To state our non-as asymtttiiiiicc upper bound for MWLE, we define the following "weighted version" of Fisher information:
(M) := E9 ExxPs(X) y/ M/w M
( (M)))
(M) := Ex-Ps(X) M|8~M
[2<x) 8 (
= Ex-Pr(X) 2 M/
8

---

Published as a conference paper at ICLR 2024
Theorem 5.2. Suppose the function class F satisfies Assumption C. Let G == (M) and H then with probability at least 1 36, we have RM (BAWLE) cc- [r.G.. (_ log for absolute constant c. Here N Poly(W, B1, B2, B3, || Tr( Tr H=
(M). For any E (0,1), if n 2 cmax{ N [(),
For an exact characterization of the threshold N one can refer to Theorem C.I in the appendix.
Compared with Theorem 3.1, Theorem 5.2 does not require well-specification of the model, demon- strating the wide applicability of MWLE. The excess risk upper bound can be explained as fol- lows: note that Tr can be expanded as Tr H. As shown by Shimodaira (2000), the term (1((/WLLE - 3*) converges asymptotically to a normal distribution, denoted as NO, af G o. Thus, the component H G.. characterizes the variance of the estimator, corresponding to the Is term in Theorem 3.1. Additionally, the excess risk's dependence on the parameter estimation is captured by H. as a counterpart of Tr in Theorem 3.1.
However, establish Theorem 5.2, does not appear in Theorem 3.1. Moreover, when the model is well-specified, by Cauchy-Schwarz ineqaulity, we have .r 2 Tr(IrIg which implies the upper bound for MWLE is larger than the vanilla MLE. This observation aligns with the results presented in Shimodaira (2000), which point out that when the model is well specified, MLE more efficient than MWLE in terms of the asymptotic variance.
necessary to assume the bounded density ratio, which
5.2 OPTIMALITY OF MWLE
To understand the optimality of MWLE, it is necessary to establish a matching lower bound. How- ever, deriving lower bound similar Theorem 3.2, which holds for any model classes that satisfies certain mild conditions, is challenging due to hardness of capturing the difference between M and F. As a solution, we present a lower bound tailored for certain model classes and data distributions in the following.
Theorem S.3. There exist (x(X) + PT((), a model class M and prediction class F satisfying Assumption such that when n is sufficiently large, we have
-1 & SUPMEM Tr (M)((M)
E (x(() M.lla
[Ra((3) 2 4.
By Theorem 5.2, the excess risk of MWLE is upper bounded by Tr 1/n. Therefore, Theo- rem 5.3 shows that there exists a non-triveen scenario where MWLE is minimax optimal.
Notice that Theorem 5.3 presents a weaker lower bound compared to Theorem 3.2. The lower bound presented in Theorem 5.3 holds only for certain meticulously chosen (X) (P model class M and prediction class F. In contrast, the lower bound in Theorem 3.2 applies to any (X()), (x), and class FF that meet the required assumptions.
6 CONCLUSION AND DISCUSSION
To conclude, we prove that MLE achieves the minimax optimality for covariate shift under a well- specified parametric model. Along the way, we demonstrate that the term Tr(InIz characterizes the foundamental hardness of covariate shift, where Is and Ir are the Fisher information on the source domain and the target domain, respectively. To complement the study, we also consider the misspecified setting and show that Maximum Weighted Likelihood Estimator (MWLE) emerges as minimax optimal specific scenarios, outperforming MILE.
Our work opens up several interesting avenues for future study. First, it is of great interest to extend our analysis to other types of OOD generalization problems, e-g., imbalanced data, posterior shift, etc. Second, our analyses relies on standard regularity assumptions, such the positive definiteness of the Fisher information (which implies certain identifiability the parameter) and the uniqueness of the minimum of the loss function. Addressing covariate shift without these assumptions is also important future directions.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
Cong Ma is partially supported by the National Science Foundation via grant
REFERENCES
S. Agapiou, O. Papaspiliopoulos, D. Sanz-Alonso, and A. M. Stuart. Importance sampling: intrinsic dimension and computational cost. Statist. Sci., 32(3):405-431, 2017. ISSN 0883-4237. doi: 10.1214/17-STS111 URL htttps://oo/ org/ 10. 121111111111111111
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization.
arXiv preprint www.iv:197.0028000 2019.
Francis Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics, (none):384+ 414, 2010. doi: 141211409-EJ8521.. URL https:// http://a.......1..11121111 09 EJS521.
Liyuan Cao. Some useful expected values with multivariate normal distribution and uniform dis- tribution on sphere. https: / /corallll ise. lesse.lehige MVNuseful pdf, February 2020.
George Casella and Roger I. Berger. Statistical inference. Cengage Learning. 2021.
Sourav Chatterjee and Persi Diaconis. The sample size required in importance sampling. The Annals of Applied Probability, 28(2):1099-119) 2018.
Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, et al. Spectral methods for data science: A statistical perspective. Foundations and Trends@ Machine Learning. 14(5):566-806, 2021.
Corinna Cortes and Mehryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103-126, 2014.
Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010a. URL https: http:/://ppccceeeeeeeeeee inw.reeeeddnngs.nniiiisss.s neurips /eeee//333311884222222 wbe (55 saaperppcf.
Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting.
Advances in neural information processing systems, 23, 2010b.
Harald Cramér.
Mathematical methods of statisties.
1946.
URL https:: //api:
semant cachhischolaccc wwwg//oorguoor..e 1e:::22200222
Genmano Gabbianelli, Gergely Neu, and Matteo Papini. importance-weighee weighted offline learning done right. arXiv preprint arXiv:2309.15777 15771, 2023.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Domain-adversarial training of neural net-
Laviolette, Mario Marchand, and Victor Lempitsky. Domain adversarial works. The journal of machine learning research, 17(1):206-2208 2016.
Rong Ge, Chi Jin, and Yi Zheng. No spurious local minima in nonconvex low rank problems: A unified geometric analysis. In International Conference on Machine Learning, pp. 1233-1242. PMLR, 2017.
Richard D Gill and Boris Y Levit. Applications of the van trees inequality: bayesian cramér-rao bound. Bernoulli, pp. 59-79, 1995.
Hao Guan and Mingxia Liu. Domain adaptation for medical image analysis: survey. IEEE Trans- actions on Biomedical Engineering, 6773)1113311185, 2021.
Ishaan Gulrajani and David Lopez-Paz. In search lost domain generalization. arXiv preprint arXiv:2007.01433, 2020.
10

---

Published as a conference paper at ICLR 2024
Steve Hanneke and Samory Kpotufe. On the value of target data in transfer learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bor Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL. https: /ptp:////////////oeeooooooc ngs neur cttpsc/cpppppa..ae prpprsreeprpppppppppppr________ File/b9r w4f4444446660 (at.ea4fffaaassacaaaa85555---000
Jayyaa Huang, Arthur Gretton, Karsten Borgwardt, Bemhard Schölkopf. and Alex Smola. Correct- ing sample selection bias by unlabeled data. Advances in neural information processing systems, 19, 2006.
Vladimir Koltchinskii, Karim Lounici, and Alexandre B Tsybakov. Nuclear-nome penalization and optimal rates for noisy low-rank matrix completion. 2011.
Samory Kpotufe and Guillaume Martinet. Marginal singularity, and the benefits of labels in covariate-shift. In Conference On Learning Theory, pp. 1882-1886. PMLR, 2018.
Qi Lei, Wei Hu, and Jason D Lee. Near-optimal linear regression under distribution shift. arXiv preprint arXiv: 106. 12108, 2021.
Cong Ma, Banghua Zhu, Jiantao Jiao, and Martin Wainwright. Minimax off-po evaluation for multi-armed bandits. IEEE Transactions on Information Theory, 68(8):5114-533) 2022.
Cong Ma, Reese Pathak, and Martin Wainwright. Optimally tackling covariate shift in rkhs-based nonparametric regression. The Annals of Statistics, 51(2):738-761 2023.
Mohammadreza Mousavi Kalan, Zalan Fabian, Salman Avestimehr, and Mahdi Soltanolkotabi. Min- imax lower bounds for transfer learning with linear and one-hidden layer neural networks. Ad- vances in Neural Information Processing Systems, 33:1959-1966 2020.
Dmitrii M Ostrovskii and Francis Bach. concordance. 2021.
samite-sample analysis of m-estimators using self-
Reese Pathak, Cong Ma, and Martin Wainwright. A new similarity measure for covariate shift with applications to nonparametric regression. In International Conference on Machine Learning, pp- 17517-17530. PMLR, 2022.
Alan Ramponi and Barbara Plank. Neural unsupervised domain adaptation in nlp- survey. arXiv preprint 2rXiv:2006.00632 2020.
C. Radhakrishna Rao. Information and the accuracy attainable in the estimation of statistical param-
eters. 1992. URL https: : //ppipppp../// sema nt cacholas aocoolrr.oo wwwooo/oooopeeeee ID: :0:11 7034671.
Alessandro Rinaldo. Advanced statistical theory lecture 13: February 26, February 2018.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. arXiv preprint arXiv:2010.05761, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- 2019.
tion. arXiv preprint 1uXiv:1911.088
Daniel Sanz-Alonso. Importance sampling and necessary sample size: an information theory ap- proach. SIAM/ASA Journal on Uncertainty Quantification, 6(2):867-879, 2018.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: survey, arXiv preprint arXiv:2108. 1322008..3644,. 2021.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227-22 2000. ISSN 0378-3758. doi: http////oorr////.0oo//////////// sci rect.. comm/cciencee wwt.arrtiiice http://pi11130870
URL https: /wwww
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research, 16(1):1731-1755, 2015.
11

---

Published as a conference paper at ICLR 2024
Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions neural net- works, 10(5):988-99) -998 1999.
Roman Vershynin. /ligh-dimensional probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018.
Kaizheng Wang. Pseudo-labeling for kernel ridge regression under covariate shift. arXiv preprint 1rXiv22302.10160, 2023.
Halbert White. Maximum likelihood estimation of misspecified models. Econometrica: Journal of the econometric society, pp. 1-25, 1982.
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmetrically-rrlaxdd distribution alignment. In International conference machine learn- ing. pp. 6872-6881, PMLR, 2019.
Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Understanding why generalized reweighting does not improve over erm. arXiv preprint arXiv:2201. 12293, 2022.
Xuhui Zhang, Jose Blanchet, Soumyadip Ghosh, and Mark S Squillante. A class of geometric
structures in transfer learning: Minimax bounds and optimality. In International Conference Artificial Intelligence and Statistics, pp. 3794-38222 3820. PMLR, 2022.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for se- mantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pp. 289-305, 2018.
12


Reviews:
[{"review_id": "z2yoDUectM", "reviewer": "Reviewer_qbMA", "summary": "This study investigates the effectiveness of maximum likelihood estimation (MLE) in the out-of-distribution optimization problem. The authors prove the effectiveness by showing the minimax optimality.", "questions": "1. Why do authors consider the minimax optimality of parametric models under a covariate shift? Is it difficult to derive corresponding results for nonparametric models? Or can the authors show uniform optimality?\n2. To the best of my knowledge, it has been known that correctly specified parametric models can adapt the OOD, at least in statistics. Therefore, the contributions of this study mainly lie in the finite-sample minimax optimality. Is my understanding correct?\n\n\n**Title**\nBy the way, I think that the title may not be appropriate. As I discussed and the authors mention in the draft, \"MLE is all we need\" has been known. The true contribution of this study is the finite-sample minimax optimality of MLE for correctly specified parametric models. If so, the title should express the true contributions. For example, if I were the authors, I name the study \"Minimax Optimality of Maximum Likelihood Estimation for Covariate Shifted Norm.\" Anyway, the authors should focus on their contributions more to clarify the claims of this study.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This study enlightens the use of the MLE under correctly specified models. The reported results are practically important because we can avoid using advanced methods for addressing the covariate shift problem. The main strength of this study lies in the finite-sample minimax optimality of MLE under a covariate shift.", "weaknesses": "Although the points raised by the authors are practically important, I cannot find novel findings. \n\n**OOD under the correctly specified models**\nFirstly, as the authors mention, it is known that the covariate shift problem is not serious when models are correctly specified (Shimodaira, 2000). Although it seems that recent studies tend to omit discussing model misspecification when discussing the covariate shift problem, classical studies motivate the use of covariate shift adaptation by considering the model misspecification setting. Those studies and classical statistics agree that correctly specified models can address the OOD generation. Therefore, when models are correctly specified, \"MLE is all we need\" has been known to researchers, though recent studies often omit this point.\n\n**Finte sample optimality**\nThen, my question for this study is its contribution. The authors insist that the contributions lie in the finite-sample minimax optimality of MLE under correctly specified parametric models. However, if we consider parametric models and minimax optimality, I think that such a result has already been shown by existing studies with more general forms or is trivial, though I do not raise some specific related work. This is because when considering parametric models $f_\\theta(x)$ parametrized by $\\theta$, it is enough to consider the minimax optimality of estimation of $\\theta$. Once we establish the optimality, we can extend the result to show the minimax optimality of estimation of $f_\\theta(x)$ using the Taylor expansion for each $x$ or uniformly over $x$. There are various results to discuss the optimality of $\\theta$ estimation, and we can just employ them.\n\n**Uniform convergence**\nSeveral existing studies discuss the uniform or point-wise minimax optimality of estimation of $f_\\theta(x)$, which directly implies the minimax optimality under a covariate shift. I believe that we can easily obtain such a result if we only consider parametric models, and the results shown by the authors can be more strengthened.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rqcROELEPD", "reviewer": "Reviewer_xCis", "summary": "The authors consider the problem of generalization for parametric models in the presence of covariate shift. Specificallly, we are given $(X,Y)$ pairs where the distribution of $X$ might be different during training and testing, but the conditional $Y|X$ is the same. They first consider the well specified regime, where the conditional distribution $Y|X$ lies in the parametric family considered. In that regime, they prove that for general parametric families under minimal assumptions (identifiability of the model, convexity of the likelihood), the Maximum Likelihood Estimator (MLE) is consistent and achieves the minimax-optimal sample complexity, without any assumptions on the density ratio between source and target distribution. They also instantiate their bounds for some concrete settings: linear regression, logistic regression and phase retrieval. They then focus on the misspecified setting, where the conditional $Y|X$ doesn't necessarily lie in the parametric class. They prove that MLE is no longer consistent and that a modification called Weighted Maximum Likelihood Estimation (WMLE) is indeed consistent. Finally, they provide a sample complexity lower bound for a specific parametric class which matches the upper bound for WMLE.", "questions": "-The finite sample guarantees that are presented hold when the number of samples $n$ is greater than some threshold, which depends on many parameters of the problem, including for example $\\|\\mathcal{I}_T^{1/2}\\mathcal{I}_S^{-1}\\mathcal{I}_T^{1/2}\\|_2$, which also measures how close the source and target distributions are in some sense. Since the sample complexity bounds only hold above that threshold, it seems that an important question is what is the dependence of the optimal sample complexity on quantities like $\\|\\mathcal{I}_T^{1/2}\\mathcal{I}_S^{-1}\\mathcal{I}_T^{1/2}\\|_2$. Have the authors considered this question?\n\n-The authors instantiate their general bound for the well-specified setting in the case of linear regression. It would be interesting to compare the results that they get with the guarantees of prior work on this problem, such as Lei et al., if such a comparison can be made.\n\n-Since there are two different distributions of source and target, I think it would help with exposition if the authors had some notation about taking expectation wrt each of the two distributions instead of hiding this dependence. For example, in the proof of Lemma A.2 the expectation is wrt the source distribution.\n\n-In page 14, in the derivation of inequality (13) in the second line, shouldn't there be an expectation above the equality sign? ($\\mathbb{E}[\\nabla l(\\beta^*)] = 0$)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper tackles a well-motivated problem which has not been studied in prior work. I believe it is a nice step towards understanding the sample complexity of learning parametric models under covariate shift. It offers a comprehensive set of results, first tackling the well-specified case and then the mis-specified one, thus providing a clear picture of the state of the art in that area. A nice feature of the result in the well-specified case is that it gives a tight answer about the sample complexity, identifying exactly how the mismatch in training and test distribution will impact the performance. The paper is also written clearly, with claims that are explained adequately and there is a nice discussion about prior work.", "weaknesses": "Perhaps this is not much of a weakness, but some of the main results do not involve significant novelty on a technical level. For example, the proof of the upper bound follows using standard arguments in parametric estimation. First they prove that the parameter $\\beta$ can be learned with accuracy $\\sqrt{\\frac{Tr(\\mathcal{I}_S^{-1})}{n}}$(Lemma A.2), which is expected, as $\\mathcal{I}_S$ is the Fischer information evaluated on the source distribution. This is shown using the standard Taylor expansion up to third order and utilizing the bounds on the first three derivatives. Then they compute how this guarantee on $\\beta^*$ translates to a guarantee about the target distribution, which is how the final factor of $Tr(\\mathcal{I}_T \\mathcal{I}_S^{-1})$ is obtained. Overall, this is similar to the standard consistency arguments in parametric statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0zyoRq8hwH", "reviewer": "Reviewer_Vzp3", "summary": "This paper working under the classic covariate shift seating where the marginal distribution of X is different between source and target data but the conditional distributions of Y|X are the same, and proved the surprising yet elegant result that classical Maximum Likelihood Estimation (MLE) purely using source data achieves the minimax optimality  under the well-specified setting. This result holds for a large family of parametric models and the authors illustrated this in linear regression, logistic regression, and phase retrieval, where no boundedness condition on the density ratio is required. They further proved that for the misspecified setting, MLE can perform poorly, and the Maximum Weighted Likelihood Estimator (MWLE) emerges as minimax optimal in specific scenarios, outperforming MLE.", "questions": "In the well-specified case where the MLE estimator is minimax optimal, is there anything to gain if there is also access to X and Y in the target population?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The presentation of the covariate shift setting and the author's result is very clean and well-written, which is also reflected in the authors' choice of using linear, logistic models to illustrate their main result. The upper and lower matching bounds on the MLE estimator is nicely presented with necessary conditions along with the MWLE estimator under the misspecified model, which imposes stronger conditions.", "weaknesses": "It would be great it the authors could present more intuition on why simple MLE works so well in well-specified model, where the estimator uses purely source data. In addition, perhaps the authors could cite and compare with some growingly popular and relevant literature in nonparametric settings of covariate shift that leverages conformal prediction, such as https://arxiv.org/abs/1904.06019 and https://arxiv.org/abs/2203.01761;\nSome comparisons between prediction and estimation problem under covariate shift could be very beneficial to the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "adcOOtnfIq", "reviewer": "Reviewer_jKTp", "summary": "This paper studies the problem of covariate shift, where a model is trained on a dataset sampled from a source distribution and aims to achieve minimal generalization error on a target distribution. It is presumed that both the source and target distributions share the same conditional distribution of label $Y$ given covariates $X$ (i.e., the so-called well-specified case) but differ in their distribution over the covariates. Under specific assumptions about the model class and the source and target distributions, the authors demonstrate that the minimax generalization error on the target distribution is achieved by the MLE (Maximum Likelihood Estimation) using only data sampled from the source. The authors further apply their findings to several (generalized) linear models. In conclusion, the authors show that there are instances in the miss-specified case where the MLE is not consistent, whereas a weighted version of MLE achieves tight sample complexity.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This is a technically solid paper, which demonstrates, for certain natural models, that the MLE achieves the minimax optimal sample complexity for covariate shift in the well-specified case. The authors also apply their findings to several (generalized) linear models, which may have potential real-world applications. The examples distinguishing between the well-specified and misspecified cases are also intriguing.", "weaknesses": "My primary reservation about this paper is that it might overly exaggerate its contributions. While the title suggests that MLE is \"All You Need\" for well-specified covariate shift, the authors introduce a series of quite non-trivial assumptions in Assumption A. It seems to me that Theorem 3.1 is essentially a direct corollary of these assumptions. Technically, these assumptions appear to essentially state that the MLE converges to the optimal parameter — the very result that Theorem 3.1 claims to prove. Additionally, regarding applications, the paper limits its focus to only (generalized) linear models. This narrow scope hardly convinces that the MLE is indeed \"All You Need\".  \n\nI would urge the authors to represent their results more accurately. Specifically, I recommend changing the title to \"Minimax Optimal Sample Complexity for Well-Specified Covariate Shift via MLE\" or something along those lines.\n\nI would like to outline the following further comments:\n\n- Assumption A and C apply to both the model class and the source and target distributions, not only to the model class as claimed by the authors.\n- In Section 4, regarding the applications, it would be beneficial to also mention the lower bounds for each example.\n- The current length of the paper, spanning 50 pages, is excessive. It would be beneficial to provide an outline of the primary proof techniques. This would help readers in distinguishing between essential steps and routine procedures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)