For high-dimensional hierarchical models, consider exchangeability of effects across covariates instead of across datasets
Authors: Brian L. Trippe
Hilary Finucane
Tamara Broderick
Venue: NeurIPS 2021 Poster (2021)
Keywords: Hierarchical Bayesian modeling
linear models
exchangeability
frequentist properties

Abstract: Hierarchical Bayesian methods enable information sharing across regression problems on multiple groups of data. While standard practice is to model regression parameters (effects) as (1) exchangeable across the groups and (2) correlated to differing degrees across covariates, we show that this approach exhibits poor statistical performance when the number of covariates exceeds the number of groups. For instance, in statistical genetics, we might regress dozens of traits (defining groups) for thousands of individuals (responses) on up to millions of genetic variants (covariates). When an analyst has more covariates than groups, we argue that it is often preferable to instead model effects as (1) exchangeable across covariates and (2) correlated to differing degrees across groups. To this end, we propose a hierarchical model expressing our alternative perspective. We devise an empirical Bayes estimator for learning the degree of correlation between groups. We develop theory that demonstrates that our method outperforms the classic approach when the number of covariates dominates the number of groups, and corroborate this result empirically on several high-dimensional multiple regression and classification problems.

For high-dimensional hierarchical models, consider exchangeability of effects across covariates instead of across datasets
Brian L. Trippe
Hilary K. Finucane
MIT CSAIL
Broad Institute
btrippe@mit.edu
Tamara Broderick
MIT CSAIL
cbroderick@csall. wwtp:/ccsal..i......m miit.ed
Abstract
Hierarchical Bayesian methods enable information sharing across regression prob- lems on multiple groups of data. While standard practice is to model regression parameters (effects) as (1) exchangeable across the groups and (2) correlated to differing degrees across covariates, we show that this approach exhibits poor statis- tical performance when the number of covariates exceeds the number of groups. For instance, in statistical genetics, we might regress dozens of traits (defining groups) for thousands of individuals (responses) ants (covariates). When an analyst has more covariates than groups, we argue that it is often preferable instead model effects (I) exchangeable across covariates and (2) correlated to differing degrees across groups. To this end, we propose a hierarchical model expressing Bayes estimator for learning the degree of correlation between groups. We develop theory that demonstrates that method outperforms the classic approach when the number of covariates dominates the number of groups, and corroborate this re- sult empirically on several high-ddmensional multiple regression and classification problems.
up millions of genetic vari-
alternative perspective. We devise empirical
Introduction
Hierarchical modeling is a mainstay of Bayesian inference. For instance, in (generalized) linear models, the unknown parameters are effects, each of which deseribes the association of particular covariate with a response of interest. Often covariates are shared across multiple related groups, but the effects are typically allowed to vary both by group and by covariate. A classic methodology, dating back to Lindley and Smith (1972) [44], models the effects as conditionally independent across groups, with latent (and learnable) degree of relatedness across covariates. From practical standpoint, the model is motivated by the understanding that it "borrows strength" across different groups [24, Chapter 5.6]. Mathematically, the model motivated by assuming effects are exchangeable across groups and applying a de Finetti theorem [44, 35]. The methodology of Lindley and Smith ubiquitous when the number of groups is larger than the number of covariates. It is standard component of Bayesian pedagogy [[23, Chapter 13.3]; [24, Chapter 15.4]] and software; e.g. it is used in the mixed modeling package 1me4 [5], which has over 16 million downloads the time of writing.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Despite its resounding success when there are more groups than covariates, we show in the present work that this standard methodology performs poorly when there are more covariates than groups. To address the many-covariater case, we turn for inspiration to statistical genetics, where scientists commonly learn linear models relating genetic variants (covariates) to traits (corresponding to different groups) across individuals (which each exhibit a response). These applications may exhibit millions of covariates, thousands of responses, and just a handful of groups. In these cases, [39, 12, 56, 69, 46, 53] use multivariate Gaussian prior akin to that of Lindley and Smith, but assume conditional independence across covariates and prior parameters that encode correlations across groups, rather than than the other way around.
As we will see, this alternative modeling approach may be motivated from a Bayesian perspective when one begins from an assumption of a priori exchangeability of the effects across covariates (rather than across groups). This exchangeability assumption is reasonable in statistical geneties, where we have little knowledge to distinguish our expectations about the effects of different genetic variants; we argue this modeling approach can be effective other modern high-dimensional analyses of multiple groups of data (beyond statistical genetics) in which large collections of covariates are frequently treated monolithically, e.g. by applying ridge regression. Namely, when there are more covariates than groups, we propose to model the effects as exchangeable across covariates (rather than groups) and learn the degree of relatedness of effects across groups (rather than covariates). In what follows, we refer to this framework as ECov. for exchangeable effects across covariates, and distinguish it from exchangeable effects across groups or EGroup.
While the existing methods in statistical genetics for modeling multiple traits obtain as a special case of ECov, to the best of our knowledge this approach absent from existing literature on hierarchical Bayesian regression. Brown and Zidek (1980) [10] and Haitovsky (1987) [28] form two exceptions, but these two papers (I) consider only the situation in which a single covariate matrix is shared across all groups (or equivalently, for each data point all responses are observed) and (2) include only theory and no empiries. While Lindley and Smith (and others) discuss priori exchangeability across covariates in the context of analysis of a single group, to our knowledge no other work has pushed this idea forward to share strength across multiple groups
We suspect that the historical origins of the methodology in statistical genetics may have hindered earlier expansion of this class of models to a wider audience. particular, this literature traces back to mixed effects modeling for cattle breeding [57]; here, an even-earlier notion of the genetic contribution of trait correlation (i.e. "genetic correlation;" see Hazel (1943) [29]) informs the covariance structure of random effects. Although genetic correlation is now commonly understood to describe the correlation of effects of DNA sequence changes on different traits [12]. its provenance predates even the first identificatinn of DNA the genetic material in 1944 [3]. As such, this older motivation obviated the need for a more general justification grounded in exchangeability. See Appendix A for further discussion of related work, including more recent works from within the machine learning community on sharing strength across multiple groups of data.
In the present work, we propose ECov as a general framework for hierarchical regression when the number of covariates exceeds the number of groups. We show that the classic model structure from statistical genetics can be seen as an instance of this framework, much as Lindley and Smith give (complementary) instance of an EGroup framework. To make the ECov approach generally practical, we devise an accurate and efficient algorithm for learning the matrix of correlations between groups. We demonstrate with theory and empirics that ECov is preferred when the number of covariates exceeds the number of groups, while EGroup is preferred when the number of groups exceeds the number of covariates. Our experiments analyze three real, non-genetic groups in regression and classification, including application to transfer learning with pre-trained neural network embeddings. We provide proofs of thcoretical results in the appendix.
2 Exchangeability and its applications to hierarchical linear modeling
We start by establishing the data and model, motivating exchangeability among covariate effects (ECov), and motivating our Bayesian generative model.
Setup and notation. Consider Q groups with D covariates. Let No be the number of data points in group q. For the gth group, the No x D real design matrix X collects the covariates, and Y4 is the veecor of responses. The nth datapoint in group g consists of covariate D-vector X and
2

---

scalar response Yr We let D == (Xx,,))) denote the collection of data from all Q groups.
We consider the generalized linear model Y X2, 39 indep p(+ p(- (39 with unknown D-vector of real effects ,34. We collect all effects in a D x Q matrix / with (d. entry -9. The linear form of the likelihood allows interpretation of 39 as the association between the dth covariate and the response in group 9. In linear regression, the responses are real-valued and the conditional distribution is Gaussian. In logistic regression, the responses are binary, and we use the logit link. The independence assumption conflicts with some models that one might use, for example in some cases when the different groups partially overlap.
Example. As a motivating non-g -ennetenett example, consider a study of the efficacy of microcredit. There are seven famous randomized controlled trials of microcredit, each in a different country [48]. We might be interested in the association between various aspects of small businesses (covariates), including whether or not they received microcredit, and their business profit (response). In this case, the dth element of X would be the dth characteristic of the nth small business in the qth country, and y is the profit of this business. See the experiments for additional examples in rates of policing. web analytics, and transfer learning.
Exchangeable effects across groups (EGroup). To fully specify Bayesian model, we need to choose a prior over the parameters B. Lindley and Smith assume the effects are exchangeable across groups. Namely, for every Q-permutation a, p(3) (3,...,,.9) Assuming exchangeability holds for imagined growing Q and applying de Finetti's theorem motivates a conditionally independent prior. Concretely, Lindley and Smith take 39 i.i.d. N(,, r), for D-vector & and D covariance matrix Т. The (d, d') entry of I captures the degree of relatedness between the effects for covariates d and d'. Both and I may be learned in an empirical Bayes procedure. However, when is large relative to Q, learning these parameters can present both computational and inferential challenges, as the O(D)) degrees of freedom in I outnumber the O(D)) effects.
p(,30(1), (0(()) (²))))..
Exchangeable effects across covariates (ECov). We here argue for a complementary approach in settings where D Q. In the microcredit example, notice that D > Q will arise whenever the experimenter records more characteristics of small business than there are locations with microcredit experiments; that is, > 7 in this particular case. Concretely, let 3a be the Q-vector of effects for covariate d across groups. Then, in the ECov approach, we will assume that effects are exchangeable across covariates instead of across groups. Namely, for every D-permutation a, p(31,33. , 3p)) = p( . B.(D)). We will see theoretical and empirical benefits to ECov in later sections, but note that the ECov assumption often consistent with prior beliefs in high dimensional settings. For instance, regarding microcredit, we may have no prior knowledge about how effects differ for distinct small-business characteristics. And we may a priori believe that different countries could exhibit more similar effects and wish to learn the degree of relatedness across those countries.
We may apply similar rationale as Lindley and Smith to motivate a conditionally independent i.i.d. model. Analogous to Lindley and Smith, we propose a Gaussian prior: 33 ~ (0, E). Σ is now Q x Q covariance matrix whose (q. entry captures the similarity between the effects in the g and o' groups. For simplicity, we restrict to BiBa]] = 0; see Appendix E.3 for discussion. Another potential benefit to ECov relative to EGroup is that we might expect statistically easier problem, with ( rather than (D2) values to learn in the relatedness matrix. We provide a rigorous theoretical analysis in Sections 4 and 5.
3
Our method
We next describe our inference method for specific instances of the exchangeable covariate effects model of Section 2. We compute the posterior and take an empirical Bayes approach to estimate Σ. We find that an expectation maximization (EM) algorithm estimates Σ effectively; Appendix A.2 compares our approach to existing methods for the related problem of estimating I' for EGroup.
Notation. We identify estimates of B and E with hats. For instance, BLS is the least squares estimate, with SL == (X4T - Ty9., y9, We will sometimes find it useful to stack the columns of B or its estimates into a length DQ vector; we denote such vectors with arrow; for example,
3

---

Algorithm 2 E--tep: Linear Regression
Algorithm Expectation Maximization for Ex-
1: 1., - E|3 E Var 3 2
changeabilityy Among Covariate Effects
2: for d 1...., do
1: // Initialize covariance
3:
Pd f- (ed @ IQ) 1
2: Σ(0) < lQ
4: Va f (ea 2 Io) V(ea @ 12)
3: // Run EM algorithm
5: Return M1,,,, pn, VI...., Vn
4: for = 1...... do
5: // Expectatioon step
Algorithm E-Stpp: Logistic Regression
6: M1...., MD, Vi,..., VD + E_Step(2(())
7:
Jí + arg max; log 2)
8:
// Maximization step
2: V + - -V3 [V p D 2:) B=p=]
[(+1) <- D-1 Ee=l + Va)
9:
3: for = 1,..., do
10:
4: Id + (ea 10) 1t'
11:
Return ((++))
5: Va + (ea IQ) V(ea & Iq)
6: Return Pi.... PD. VI..... VD
3:= [3LT , ,...,,897 , For a natural number N, we use IN, Iv, and ex to denote the N N identity matrix, N-vector of ones, and Nth basis vector, respectively. We use ) to denote the Kronecker product.
3.1 Posterior inference with a Gaussian likelihood
n. ,99 indep N(x; (8, o where og group-specific variance. When the relatedness matrix known, natural estimate of B is its posterior mean. We obtain the full posterior, including its mean, via standard conjugacy argument; see Appendix B.l:
We first consider a Gaussian likelihood: for each group & and observation n, we take Y Xx,,
Proposition 3.1. For each covariate d, let Sa i.d. N) a priori. For each group indep Nxxx 3a, og) Then EDD ~ N V) for ı ~
and data point n, let X2 n: 99 3 2T X1 ,...,, Q QT X and - y diaagia T ..... oo XQT XQ),
...., QQ x denotes DQ DQ block dickonaa matrix. where diag(o, 2 2x
At first glance, the posterior mean jł for this model might seem to introduce a computational challenge because exact computation of V involves an D(D matrix inversion. Our experiments (Section 6). however, involve on the order of DQ R 1,000 parameters, so direct inversion V demands less than single second. Moreover, in much larger problems zž may still be computed very efficiently using the conjugate gradient algorithm [49, Chapter 5], with convergence in a small number of O(D²Q) time iterations; Appendix B.2.
3.2 Empirical Bayes estimation of Σ by expectation maximization
The posterior mean of in Proposition 3.1 requires Σ. which is typically unknown, Accordingly, we propose an empirical Bayes approach of estimating Σ by maximum marginal likelihood:
Becor
E D where E arg max (D E).
Ero
Equation (1) defines two step procedure. In the first step, we learn the similarity between groups via estimation of Σ. In the second step, we use this similarity to compute an estimate, $pcov that correspondingly shares strength. Though we have been unable to identify general analytic form for E. we can compute it with an expectation maximization (EM) algorithm [47, Chapter 1.5].
Algorithm 1 summarizes this procedure; see Appendix B.3 for details.

---

3.3 Classification with logistic regression
We can extend the approach above to inference for multiple related classification problems. We assume logistic likelihood; for each y and n., Y IX ,39 indep Bern[(1 + exp(-X3 (*)))* In the classification case, we cannot use Gaussian conjugacy directly, so we apply approximation. Specifically, we adapt the original E-step Algorithm by using a Laplace approximation to the posterior [7. Chapter 4.4]. We approximate the posterior mean of 8 by the maximum posterion value. We leave extensions to other generalized linear models to future work.
4 Theoretical comparison of frequentist risk
In this section, prove theory that suggests ECov has better frequentist risk than EGroup when D is large relative to Q. Analyzing θксоv directly is challenging due its non-differentiabiiity as a function of the data, so we take a multipart approach. First, in Theorem 4.3, we show that an AMM
ECov estimate based on moment- matching (MM), PBCov dominates least squares, ÔLS, when D large relative Q; fLS in turn dominates 3MM EGroup (a similar estimator for EGroup). Second, in 2MM
Theorem 4.5, we show that B uniformly improves on PECov
Setup. Take fixed value of 3 and an estimator B. We use squared error risk, R(3,
E [| - 31 $ as our measure of performance. || ||e is the Frobenius norm of matrix, and the expectation is over all observations Y ..... y jointly. We require the following orthogonal design condition.
Condition 4.1. For each group q. 0 2x X 5 II for some shared variance o².
Though restrictive, this condition useful for theory, as other authors have found; see Appendix C.I. We empirically demonstrate that our theoretical conclusions apply more broadly in Section 6.
ECov EGroup when using moment matching in high dimensions. For ECov, the fol- lowing estimate for E is unbiased under correct prior specification: SMM D-
x (1:)), where i denotes the Moore-Penrose pseudoinverse of matrix and f1. is the least squares estimate. We define ECov B[ ΣMM] to be the resulting analogously for EGroup; see Appendix C.2 for details. While PECov and PEGroup are naturally defined only when D 2 Q and D s Q, respectively, we find it informative to compare how their performances depend on D) and Q nonetheless.
d-Adiag || - .g x
GMM
parameter estimate, and define BMM 3MM
2MM
Before our theorem, a lemma provides concise expressions for the risks of BBov and 3MM
EGroup"
Lemma 4.2. Under Condition 4.1 and when D 2 Q. R(3, ÊMM = = dDD(D 2
at 11
B8. Additionally, when D S Q, R(3, 9MM
EGroup) = o²DQ o'Q(Q 2
2D E
BGroup that are surprisingly simple. The symmetry between the forms and risks of these estimators, however, is intuitive; under Condition 4.1, 3MM PBCov and GMM PEGroup can be seen respectively arising from the same procedure applied to Bts and its transpose.
Lemma 4.2 reveals forms for the risks of BCov and 3MM
3MM
aMM With Lemma 4.2 in hand, we can now compare the risk of gEoo fr.s, and EGrrup
Theorem 4.3. Let Condition 4.1 hold. Then (1) if D > 2Q + 2, 9MM EECvv
dominates IT.s with respect PECov R(B, BtS). Additionally, (2) if
squared error risk. In particular, for any B, R(3, 2MM
2MM D Q/2 1, ECroup is dominated by Bis-
Since 3LS is minimax [41, Chapter 5], Theorem 4.3 implies that S has minimax risk in the high-dimensional setting. It follows that, regardless of how well the ECov prior assumptions hold, 3MM PBCov will not perform very poorly.
5

---

for some Further improvement with maximum marginal likelihood. The moment based approach analyzed above has limitation: with positive probability, EMM is not positive semi-definite (PSD). Though our expression for 9MM
PEC remains well-defined in this case, this non-positive definiteness obscures the interpretation of 3MM as a Bayes estimate. We next show that performance further improves if E is instead estimated by maximum marginal likelihood (Equation (1)) and is thereby constrained to be PSD.
Our next lemma characterizes the form of the resulting estimator, BROwE and establishes a connection the positive part James-Stein estimator [4].
Lemma 4.4. Assume D > Q and consider the singular value decomposition Bis = Vdiag(Aà YUT where V and U satisfy V = U = lq: and X is a Q-vector of non-negative reals. Un- (D-'X-o'1q)) -0=1o)+] UT and BEC = Vdliag | @ (Iq - ooo-oDDX->)+ UT, where (-)+ is shorthand for max(--0) element-wise, ④ is the Hadamard (i.e. element-wise) product, and the powers in X and X are applied element-wise.
der Condition 4.1, Equation (1) reduces to Σ = ===0 Udiag [(D-'A -
Lemma 4.4 allows us to see $gCov as shrinking åLS toward 0 in the direction of each singular vector to an extent proportional to the inverse of the associated singular value. The transition from 4MM PECov to 8BCov is then analogous to the taking the "positive part" of the James-Stein estimator in vector estimation, which provides uniform improvement in risk [4]. Though R(3, $gcov) ) is not easily available analytically, we nevertheless find that it dominates its moment- based counterpart.
Theorem 4.5. Assume D > Q + 1. Under Condition 4.1 θ dominates 2MM
PBCov with respect to
squared error loss, achieving strictly lower risk for every value of 3.
We establish Theorem 4.5 using a proof technique adapted from Baranchik [4]; see also Lehmann The standard approach we build upon is complicated by the fact that the directions in which we apply shrinkage themselves random.
and Casella [4m][Thm. 5.5.4].
GMM Theorem 4.5 provides strong line of support for using 8 over PECov that does not rely any assumption of "correct" prior specification; in particular the risk improves without any subjective assumptions on B. We discuss related earlier work in Appendix A.4.
5 Gains from ECov in the high-dimensional limit
The results of Section 4 give promising endorsement of ECov but face two important limitations. First, the domination results relative to least squares do not directly demonstrate that fr.c attains improvements by leveraging similarities across groups in meaningful way; indeed for a single group (i.e. Q = 1) 8ECov can be understood as a ridge regression estimate [31], and Theorems 4.3 and 4.5 provide that 3RcON dominates 31. for D >>> 3. Second, domination results reveal nothing about the size of the improvement how it depends on any structure of [3; intuitively, we should expect better performance when 3 is in some way representative of the assumed prior. To address these limitations, we analyze the size of the gap between the risk of (1) ågcov and (2) our method applied to each group independently (ID), which we denote by ÂID. We will characterize the dependence of this gap on 3.
Reasoning quantitatively about the dependence of the risk on the unknown parameter poses significant analytical challenges. In particular, Lemma 4.2 shows that R(B, 8MM depends on 3 through E l [:3:: however, || is the sum of the eigenvalues of a non-central inverse Wishart matrix, notoriously challenging quantity work with; see e.g. [42, 30). To regain tractability, we (1) develop an analysis asymptotic in the number of covariates 1 and (2) shift to Bayesian analysis in order to sensibly consider a growing collection of covariate effects. In particular, we consider
sequence of regression problems, with parameters { distributed 3a i.i.d
distribution म. Accordingly, instead of using the frequentist risk as in Section 4, we now use the Bayes risk to measure performance. Specifically, for a group with covariates and an estimator 3,
Our approach BRew is well defined in the Q 1 single group case; for each group q, we obtain 3% by computing 8ECov on the group D ((X9YY9)

---

the Bayes risk is (B)) E, where R(B, (3) is the usual frequentist risk. In the following, we describe the results of this analysis with proofs and additional details left to Appendix D.
For a single metric characterizing the benefits joint modeling, we will define the asymptotic gain as the relative performance between our two estimators of interest here, BBC and ÅID.
Definition 5.1. Consider a sequence of datasets of Q regression problems with an increasing number of covariates D, (DD D=1- Assume that for each group Condition 4.1 is satisfied with variance o² and that each Ba t.1.d. т. The asymptotic gain of joint modeling Gain o²) lim (Q)DDQ)= (9ID) ) - RRp (BECov)].
The factor of o² DQ in Definition 5.1 puts Gain( 0²) on scale that is roughly invariant to the size and noise level of the problem; for example, (GRDQ)=TA (3Ls) for any æ, D, and Q. In Appendix D.5 discuss how this asymptotic formulation may allow relaxation of Condition 4.1 one considers certain random design matrices; for simplicity, the present analysis considers only fixed designs.
Our next lemma gives an analytic expression for Gain(a, od) that provides a starting point for understanding its problem dependence.
Lemma 5.2. Assume Σ := Var, [Bi] finite and has eigenvalues A1:..., dq. If Condition 4.1 satisfied asymptotically, Gain(T,, o²) = 0²Q E (Ag + - (Eg + +0²)-1]
Lemma 5.2 reveals that the diagonals and eigenvalues and Σ are key determinants of Gain o²), but does not directly provide an interpretation of when 8Ecov offers benefits over ÂID. Our next theorem demonstrates when an improvement can be achieved from joint modeling.
Theorem 5.3. Gain(T, od) O, with equality only when Σ Var, [31] is diagonal.
Proof. From Lemma 5.2 we see Gain( o²) is the difference between strictly Schur-convex function applied to the eigenvalues of E and its diagonals (since (xr + -1 is convex on RR,). By the Schur-Horn theorem, the eigenvalues of Σ majorize its diagonals, providing the result. O
Theorem 5.3 tells that śgCOv succeeds at adaptively learning and leveraging similarities among groups in the high-dimensional limit. In particular, Gain(T, reduces to zero only when the eigenvalues of Σ are arbitrarily close to the entries of its diagonal, which occurs only when the covariate effects are uncorrelated across groups. However, when covariate effects correlated, obtain an improvement.
Our next theorem quantifies this relationship through upper and lower bounds.
Theorem 5.4. Let X and &t denote the eigenvalues and diagonals of E. respectively, sorted in descending order. Then Gain(T, o²) S - ||||2||ett [+112/(Amii + 02)s and Gain( o²) Z Q'Q--1let x-11/ [/m/ + 02)3. where Amax and Amiin are the largest and smallest, respectively, eigenvalues E. Σ.
Theorem 5.4 allows us to see several aspects of when our method will and will not perform well. First, the presence of llet 1+112 in both the upper and lower bounds demonstrates that Gain(T, o²) will be small when the eigenvalues are close to the diagonal entries, with Euclidean distance as an informative metric.
As we find in our next corollary, Theorem 5.4 additionally allows us to see that nontrivial gains may be obtained only in intermediate signal-to-noise regime, where signal is given by the size of the covariate effects and noise is the variance level a². Notably, under Condition 4.1, o² relates directly to the variance of B1.s, and is influenced by both the residual variances and the group sizes; see Appendix C. particular we interpret Amin as a proxy for signal strength since captures the magnitude of typical Ba's 's along their direction of least variation.
Corollary 5.5. Gain(T, o²) < 4k² Xwin/o² and Gain( d²) S 4r (Nminnin /02) where :=
Amax/ / Ami is the condition number of
7

---

Correlated Effects
Independent Effects
Mean Est. Err. I1l A13//D)
Q=D
Least Squares [hts]
10
Exch. Group GMM
Exch. Group [RaGsmp]
Bayes. Indep. [9m]]
Exch. Cov. [
Exch. Cov. [
10
10
10
10a
Covariates [D]
Covariates
[D]
Figure I: Dimension dependence of parameter estimation error in simulation. Covariate effects are either [Left] correlated or [Right] independent across the 10 groups. Each point the mean
11EE across 20 replicates.
Corollary 5.5 formalizes the intuitive result that with enough noise, the little recoverable signal is insufficient to effectively share strength. And furthermore, in the low-noise and high-signal regime âID is very accurate its own and there is little need for joint modeling. However, when there is large gap between the largest and smallest eigenvalues of Ê, leading to be large, the gain could be larger. will large, for example, when the covariate effects are very correlated across groups.
Experiments
6.1 Simulated data
We first conduct simulations, where directly control the relatedness among groups and where we know the ground truth values of the parameters, We show that ECov is more accurate than EGroup when covariates outnumber groups, whether effects are correlated across groups not.
In particular, we simulated covariates, parameters, and responses for Q = 10 groups across a range of covariate dimensions. We generated covariate effects 3g i.i.d. NO E). We chose so that effects were either correlated (Figure Left) or independent (Figure 1 Right) across groups; see Appendix for details. We compare performance of six estimates on these groups. These are estimates assuming EGroup/ECov using moment matching and maximum marginal likelihood to choose Σ/T (&MM (Crooup /BMM and / respectively), as well as least squares (BLs), and ECov applied to each group independently (31D).
Figure reinforces our theoretical conclusions that (1) PECav is more accurate when covariates outnumber groups and (2) 8 is more accurate when groups outnumber covariates. Our simulated X matrices somewhat relaxed from strict orthogonal design (Appendix E), these experiments suggest that our conclusions hold beyond Condition 4.1. Additionally, BECor and PEGroupp
Brcroup both outperform their moment based counterparts, PEoo and 2MM
Even for the simulations with independent effects, Theorem 4.3 suggests 8ECOV should still outper- form ÔLS and 8gGroup in the higher dimensional regime, and of Figure Additionally, agreement with Theorem 5.3, BECOv does not improve over Åip in the presence of independent effects, and the performances of these two estimators converge as D grows.
this behavior in the right panel
6.2
Real data
We find that ECov beats FGroup, as well as least squares and independent estimation, across three real groups. We describe the datasets (with additional details in Appendix E.4) and then results.

---

Community level law enforcement in the United States. Policing rates vary dramatically across different communities, mediating disparate impacts of criminal law enforcement across racial and socioeconomic groups [64, 54]. Understanding how demographic and socioeconomic attributes of communities relate to variation in rates of law enforcement crucial to understanding these impacts. Linear models provide the desired interpretability. We use a dataset [51] consisting of D = 117 community characteristics and their rates of law enforcement (per capita) for different crimes. We consider Q = 4 group subsets corresponding to distinct (region, crime) pairs: (Midwest, Robbery), (South, Assault), (Northeast, Larceny), and (West, Auto-theft). This data setup illustrates small Q and accords with the independent residuals assumption in the likelihood shared by ECov and EGroup (Section 2). Across 9. No represents between 400 and 600 communities.
Blog post popularity. We regress reader engagement (responses) on D = 279 characteristics of blog posts (covariates) [13]. We divided the corpus based on an included length attribute into Q = groups, corresponding to (1) long posts, (2) short posts, and (3) posts from an earlier corpus with missing length attribute. We hypothesized that the relationships between the characteristies of posts and engagement would differ across these three groups. We randomly downsampled to N 500 posts in each group to mimic a low sample-size regime, in which sharing strength is crucial.
Blog Popularity
Law Enforcement
CIFAR10
Prediction Error [MSE]
Prediction Error [MSE]
0.6
Classification Error [%]
24
1.0
Least Sqrs./MLE
Least Sqrs. / Pool
0.5
22
Indep. Bayes.
0.8
Exch. Group
0.4
+
Exch. Cov.
20
0.6
0.3
Figure 2: Prediction performance on held out data in three applications (mean +1SEM across 5-fold
cross-validation splits).
Multiple binary classifications using pre-trained neural network embeddings on CIFARIO. Modern machine learning methods have proved very successful large datasets. Translating this success to smaller datasets is one of the most actively pursued algorithmic challenges machine learning. It has spurred the development of frameworks from transfer learning [65] to one-shot learn- ing [62] to meta-learning [21]. One common and simple strategy starts with a learned representation (or "embedding") from expressive neural network fit to a large group. Then embedding as a covariate vector for classification tasks with few labeled data points.
can this
We take a D 128 dimensional embedding of the CIFAR10 image group [37, 60). We create Q 8 different binary classification tasks using the classes in CIFARIO (Appendix E.4). We downsampled to No varying from 100 to 1000 to mimic setting in which we hope to share strength from large groups to improve performance on smaller datasets.
Discussion of evaluation and results. In previous sections we have focused on parameter estimation. Here we instead evaluate with prediction error on held-out data since the true parameters are not observed. Specifically we perform 5-fold cross-validation and report the mean squared errors and classification errors on test splits. To reduce variance of out-of-sample error estimates on the applications in which we downsampled, we also evaluate on the additional held-out data. Because the residual variances were unknown, we estimated these for each application and group as o :=
[| 1'122/N,, D), where Px == IN, X* X9)-1x" (see e.g. [23, Chapter 18.1]). All methods ran quickly on 36 CPU machine: computation of BEcIv including the EM algorithm, required 2.04 + 0.64, 6.89 + 3.19 and 37.14 + 3.39 seconds (mean st-dev across splits) on the law enforcement, blog, and CIFARIO tasks, respectively.
Our results further reinforce the main aspects of our theory. BeCor outperformed BeCroup indepen- dent Bayes estimates (3ID), and least squares (Bts) in all applications (at > 95% nominal confidence

---

with a paired t-test). 2 Additionally, BEcov outperformed the baseline of igmoring heterogeneity, pooling groups together. and using the same effect estimates for every group ("Least Sqrs./MLE Pool").
Appendix E. includes additional results and comparisons. In particular, we provide the performance of the estimators on each component group for each application. Additionally, we report the performances of (I) stable and computationally efficient moment based alternatives to $goov and BEGroup and (2) variants of /BC.r and BEGroup that include a learned (rather than zero) prior mean. Appendix E.5 reports the licenses of software we used.
Discussion
The Bayesian community has long used hierarchical modeling with priors encoding exchangeability of effects across groups of data (EGroup). In the present work, we have made case for instead using priors that encode exchangeability across covariates (ECov) - in particular, when the number of covariates exceeds the number of groups. We have presented corresponding concrete model and inference method. We have shown that ECov outperforms EGroup in theory and practice when the number of covariates exceeds the number of groups.
Our approach is, of course, not a panacea. In some settings, priori exchangeability among covariate effects will be inconsistent with prior belief's. For example, imagine in the CIFARIO IO application if meta-data covariates (such as geo-location and date) were available, in addition to embeddings. Then we might achieve better performance by treating meta-data covariates as distinct from embedding covariates. Additionally, we focused on Gaussian prior for convenience. In cases where practitioners have more specific prior beliefs about effects, alternative priors and likelihoods may be warranted, though they may be more computationally challenging. Moreover, while relatively interpretable, linear models have their downsides. The linear assumption be overly simplistic in many applications. It common to misinterpret effects as causal rather than associative. Both the linear model and squared error loss lend themselves naturally reporting means, but in many applications median other summary is more appropriate; so using a mean for convenience can be misleading.
Many exciting directions for further investigation remain. For example, the covariance Σ may provide an informative measure of task similarity; this similarity measure can be useful in, e.g., meta learning [34] and statistical genetics [12]. Additionally, we here explored two approaches to choosing the covariance matrices in the empirical Bayes step: more sophisticated approaches to covariance estimation may provided improved performance. It also remains extend our methodology to other generalized linear models.
Acknowledgments and Disclosure of Funding
The authors thank Sameer K. Deshpande, Ryan Giordano, Alex Bloemendal, Lorenzo Masoero, and Diana Cai for insightful discussions on the manuscript, and the anonymous reviewers for their constructive suggestions. This work was supported in part by ONR Award N00014-18-S-F006 and an NSF CAREER Award. BLT is supported by NSF GRFP.
References
[1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Matthieu Dean, Jeffrey an Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga. Sherry Moore, Derek Murray, Benoit Steiner, Paul Tucker. Vijay Vasudevan. Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In 12th USENIX symposiun on operating systems design and implementation (OSDI 16), pages 265-283, 2016.
[2] Tomohiro Ando and Arnold Zellner. Hierarchical Bayesian analysis of the scemingly unrelated regression and simultaneous equations models using combination of direct Monte Carlo and importance sampling techniques. Bayesian Analysis, 5 5(1):65-95, 2010.
We did develop extension akin Algorithm 3 for EGroup, and so do not report 8E for CIFARI0. Additionally, report maximum likelihood estimate (MLE) instead of ÔLS for CIFARI0.
10

---

[3] Oswald T Avery, Colin M MacLeod, and Maclyn McCarty. Studies on the chemical nature of the substance inducing transformation of pneumococcal types: induction of transformation desoxyribonucleic acid fraction isolated from pneumococcus type III. The Journal of Experimental Medicine, 79(2):3771158 1944.
[4] Alvin J Baranchik. Multiple regression and estimation of the mean of multivariate normal distribution. Technical report, Stanford University, 1964.
[5] Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting linear mixed-effects models using 1me4, Journal of Statistical Software, 67(1):1-48, 2015.
[6] Anindya Bhadra and Bani K Mallick. Joint high nigh-dimensional Bayesian variable and covariance selection with an application to cQTL analysis. Biometrics, 69(2):44-4457, 2013.
[7] Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer. 2006.
[8] Robert C Blattberg and Edward George. Shrinkage estimation of price and promotional elasticities: Seemingly unrelated equations. Journal of the American Statistical Association, 86(414):304- 315, 1991.
19] Leo Breiman and Jerome H Friedman. Predicting multivariate responses in multiple linear regression. Journal of the Royal Statistical Society: Series B, 59(1):3-54, 1997.
[10] Philip J Brown and James Zidek. Adaptive multivariate ridge regression. The Annals of Statistics, 8(1): 64-74, 1980.
[11] Philip J Brown, Marina Vannucei, and Tom Feam. Multivariate Bayesian variable selection and prediction. Journal of the Royal Statistical Society: Series B, 60(3):627-641, 1998.
[12] Brendan Bulik-Sulliva,, Hilary K Finucane, Verneri Antila,, Alexander Gusev, Felix R Day, Po-Ru Loh, Laramie Duncan, John RB Perry, Nick Patterson, Elise Robinson, Mark Daly, Alkes Price, and Benjamin M Neal. An atlas of genetic correlations across human diseases and traits. Nature Genetics, 47 (11):1236, 2015.
[13] Krisztian Buza, Feedback prediction for blogs. In Data Analysis, Machine Learning and Knowledge Discovery, pages 145-152. Springer, 2014.
[14] Diana Cai, Rishit Sheth, Lester Mackey, and Nicolo Fusi. Weighted meta-learning. arXiv preprint arXiv:2003.09663 2020.
[15] Siddhartha Chib and Edward Greenberg. Hicrarchical analysis of SUR models with extensions correlated scrial errors and time-varying parameter models. Journal of Econometrics, 68(2):339 360, 1995.
[16] A Philip Dawid. Some matrix-variate distribution theory: notational considerations and Bayesian application. Biometrika, 68(1):265-274, 1981.
[17] Sameer K Deshpande, Veronika Ročková, and Edward George. Simultaneous variable and covariance selection with the multivariate spike-and-slab lasso, Journal of Computational and Graphical Statistics. 2019.
[18] Bradley Efron and Carl Morris. Empirical Bayes vector observations: An extension of Stein's method. Biometrika, 59(2)3335-347, 1972.
[19] Bradley Efron and Carl Morris. Limiting the risk of Bayes and empirical Bayes estimators Part II: The empirical Bayes case. Journal of American Statistical Association, 67(337):130-133 1972.
[20] Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 134445)) 2001.
[21] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pages 1126 1135, PMLR. 2017.
[22] Alan E Gelfand, Susan E Hills, Amy Racine-Poon Poon, and Adrian FM Smith. Illustration of Bayesian inference in normal data models using Gibbs sampling, Journal of the American Statistical Association, 85 (412):972-4 985, 1990.
[23] Andrew Gelman and Jennifer Hill. Data Analysis using Regression and Multilevel/ftierarchhcc Models. Cambridge University Press, 2006.
[24] Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian Data Analysis. Chapman and Hal//CRC, 2013.

---

[25] Amos Golan and Jeffrey M Perloff. Comparison of maximum entropy and higher-order entropy estimators. Journal of Econometricx, 107(1-2):15-211, 2002.
[26] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting maddient-basedd meta-learning as hierarchical Bayes. In International Conference Learning Representations, 2018.
[27] William Griffiths. Bayesian inference in the seemingly unrelated regressions model. Computer-Aided Econometrics, pages 287-314 CRC Press, 2003.
[28] Yoel Haitovsky. On multivariate ridge regression. Biometrika, 74(3):663-570, 1987.
[29] Lanoy Nelson Hazel. The genetic basis for constructing selection indexes. Genetics, 28(6):476-490, 1943.
[30] Grant Hillier and Raymond Kan. Properties of the inverse of noncentral Wishart matrix. Available SSRN 3370864, 2019.
[31] Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67, 1970.
[32] Alfred Horn. Doubly stochastic matrices and the diagonal of a rotation matrix. American Journal of Mathematics, 76(3):620-630, 1954.
[33] James and Charles Stein. Estimation with quadratic loss. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probubility, 1:361-379, 1961.
[34] Ghassen Jerfel, Erin Grant, Thomas Griffiths, and Katherine Heller. Reconciling meta-learning and continual learning with online mixtures of tasks, Advances Neural Information Processing Systems, 32, 2019.
[35] Michael Jordan. Bayesian nonparametric learning: Expressive priors for intelligent systems. Heuristics. Probability and Causality: Tribute to Judea Pearl, 11:167-185, 2010.
[36] Diederik P Kingma and Max Welling, Auto-encoding variational Bayes. arXiv preprint ar1iv:1312.6114, 2013.
[37] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Puper. University of Toronto, 2009.
[38] Nan Laird and James H Ware. Random- -efflotseeffets models for longitudinal data. Biometrics, 38(4);963-4) 31ometricx,38(449963-999 1982.
[39] Sang Hong Lee, Jian Yang, Michael E Goddard, Peter M Visscher, and Naomi R Wray. Estimation of pleiotropy between complex diseases using single-mucleotide polymorphism-derivee genomic relationships and restricted maximum likelihood. Bioinformatics, 28(19)2550-25 25442, 2012.
[40] Seunghak Lee, Jun Zhu, and Eric P Xing. Adaptive multi-task lasso: with application to eqil detection. 2010.
[41] Erich L Lehmann and George Casella. Theory of Point Estimation. Springer Science & Business Media, 2006.
[42] Guy Letae and Hélene Massam. A tutorial on non central Wishart distributions. Technical Paper, Toulouse University, 2004.
[43] Alex Lewin, Habib Saadi, James E Peters, Aida Moreno-Moral, James C Lee, Kenneth GC Smith, Enrico Petretto, Leonardo Bottolo, and Sylvia Richardson. MT-HESS: an efficient Bayesian approach for simultancous association detection in OMICS datasets, with application to eQTL mapping in multiple tissues. Bioinformatics, 32(4):523-532, 2015.
[44] Dennis Lindley and Adrian FM Smith. Bayes estimates for the linear model. Journal of the Royal Statistical Society: Series B, 34(1):1-18, 1972.
[45] David Luenberger. Introduction to Linear and Nonlinear Programming. Addison- Wesley Reading, MA, 1973.
[46] Robert Maier, Gerhard Moser, Guo-Bo Chen, Stephan Ripke, Cross-Disorder Working Group of the Psychiatric Genomics Consortium, William Coryell, James B Potash, William Scheftner, Jianxin Shi, Myrna M Weissman, Christina M Hultman, Mikael Landén, Douglas F Levinson, Kenneth S Kendler. Jordan W Smoller, Naomi R Wray, and S Hong Lee. Joint analysis of psychiatric disorders increases accuracy of risk prediction for schizophrenia. bipolar disorder, and major depressive disorder. The American Journal of Human Genetics, 96(2)2283-29 2015.
12

---

[47] Geoffrey J McLachlan and Thriyambakar Krishnan. The EM Algorithm and Extensions. volume 382. John Wiley & Sons, 2007.
[48] Rachael Meager. Understanding the average impact of microcredit expansions: A Bayesian hierarchical analysis of seven randomized experiments. American Economic Journal: Applied Economics, 11(1):57-91, 2019.
[49] Jorge Nocedal and Stephen Wright. Numerical Optimization. Springer Science & Business Media, 2006.
[50] Guillaume Obozinski, Ben Taskar, and Michael Jordan. Multi-task feature selection. Statistics Depurtment. UC Berkeley, Tech. Rep, 2(2.2).2, 2006.
[51] Michael Redmond and Alok Baveja. data-driven software tool for enabling cooperative information sharing among police departments. Eumpean Journal of Operational Research, 141 141(3):660-678, 2002.
[52] Gregory C Reinsel. Mean squared enror properties of empirical Bayes estimators a multivariate random effects general linear model. Journal of the American Statistical Association, 80(391):642-666 650, 1985.
[53] Daniel E Runcie, Jiayi Qu, Hao Cheng, and Lorin Crawford. McgaLMM: Mega-scale lincar mixed models for genomic predictions with thousands of traits. BioRxiv, 2020.
[54] Lee A Slocum, Beth M Huchner, Claire Greene, and Richard Rosenfeld. Enforcement trends in the city of St. Louis from 2007 to 2017: Exploring variability in arrests and criminal summonses over time and across communities. Journal of Community Psychology, 48(1):36-67, 2020.
[55] Michael Smith and Robert Kohn. Nonparametric seemingly unrelated regression. Journal of Econometrics. 98(2):257-281, 2000.
[56] Matthew Stephens. A unified framework for association analysis with multiple related phenotypes. PloS One, 8(7)se65255, 2013.
[57] Robin Thompson. The estimation of variance and covariance components with an application when records are subject to culling. Biometrics, pages 527-550, 1973.
[58] Hisayuki Tsukuma. Admissibility and minimaxity of Bayes estimators for normal mean matrix. Journal of Multivariate Analysis, 99(10):251-2264 2008.
[59] A Van Der Merwe and James V Zidek. Multivariate regression analysis and canonical variates. Canadian Journal of Statistics, 8(1):27-33) 1980.
[60] Arnaud Van Looveren, Giovanni Vacanti, Janis Klaise, and Alexandru Coca. Alibi-Detect: Algorithms for outlier and adversarial instance detection, concept drift and metrics. 2019. https:tttttttieeee teaitttub.co com/ SeldonIO/el Saaoonnn/lllbiiii alibi-detect..
[61] Wessel N van Wieringen. Lecture notes on ridge regression. arXiv preprint
2015.
[62] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 3637-3645, 2016.
[63] Martin J Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University
Press, 2019.
[64] David Weisburd, Malay K Majmundar, Hassan Adea, Anthony Braga, Jim Bueermann, Philip J Cook, Phillip Atiba Goff, Rachel A Harmon, Amelia Haviland, Cynthia Lum, Charles Manski, Stephen Mastrofski, Tracey Mcares, Daniel Nagin, Emily Owens, Steven Raphacl, Jerry Rateliffe, and Tom Tyler. Proactive policing: A summary of the report of the national academies of sciences, engineering, and medicine. Asian Journal of Criminology, 1472):145-177, 2019.
[65] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal Big Data, 3(1):-40,, 2016.
[66] Xiaolin Yang. Seyoung Kim, and Eric Xing. Heterogeneous multitask learning with joint sparsity con- straints. Advances in neural information processing systems, 22:2151-2155 2009.
[67] Arnold Zellner. An efficient method of estimating seemingly unrelated regressions and tests for aggregation bias. Journal of the American Statistical Association, 57(298):348-366 338, 1962.
[68] Arnold Zellner and David S Huang. Further properties of efficient estimators for seemingly unrelated regression equations. International Economic Review, 3(3):300-33 313, 1962.
13

---

[69] Xiang Zhou and Matthew Stephens. Efficient multivariate linear mixed model algorithms for genome-wide association studies. Nature Methods, 11(4):407, 2014.
[70] Jim Zidek. Deriving unbiased risk estimators of multinormal mean and regression coefficient estimators using zonal polynomials. The Annals of Statistics, pages 769-782, 1978.
14


Reviews:
[{"review_id": "pGIt9K0utRu", "reviewer": "Reviewer_g5kM", "summary": "The authors of this paper study exchangeable prior distribution for regression coefficients in high dimensional regression problems, where the number of covariates is typically much larger than the number of groups.  They argue that when the number of covariates are large, it is preferable to model the covariates as exchangeable rather than the datasets. The authors provide theoretical results by comparing frequentist risk of different empirical Bayes estimators and demonstrate the value of their approach empirically.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this is an interesting, well written paper.  That said, I’m not really familiar enough with this literature to know for sure how impactful and insightful their results really are.  My gut instinct is that, while interesting, the results are not particular profound, if only because they are quite intuitive: in high dimension, low sample size settings covariance estimation is known to be quite hard, so using an empirical Bayes procedure to infer the covariance of Beta (across covariates) might be quite bad.  \n\nSetting that aside, I have another main “complaint” which doesn’t really affect my overall assessment of the quality of the paper: I find it strange that this paper was submitted to neurips.  In my opinion, this paper is much more appropriate for a statistics journal.  I say this for two reasons: 1) As somebody who rarely cares about this distinction, I still found myself thinking that this paper is very clearly a statistics paper dealing with canonical statistical issues around normal means estimation, James Stein, regression etc.  2) this is a 9 page paper with a 40 page supplement— much of the supplement is explanatory text (not just proofs and additional figures).  It seems like this might be better in a longer journal paper.  I would encourage the authors to think about whether they should consider a more appropriate venue.\n\nRegarding the comment about statistical power at the top of page 3, is this the main driver for improvements in the ECov over EData? Since Sigma is shared across all D covariates, as D->infinity you can consistently estimate Sigma, whereas with EData D > Q implies a very poor empirical Bayes estimator of Gamma.  \n\nRelatedly, I’m curious about an estimator in which you learn an approximately low rank covariance matrix under EData, e.g. with Gamma = AA^T + sigma^2I which be a better competitor? For example, perhaps A is an D x k matrix k <= Q, so that the number of parameters in the covariance matrices to learn in EData and much smaller than the general exchangneable model. I think there is something to be learned here from the high dimensional covariance estimation literature.  In short, I think to make this really useful I’d like to know if I wouldn’t be better off just modeling exchangeability among datasets under a structured prior for the covariance (e.g low rank or sparse).  That may solve some of the problems encountered in the large D, small Q case.\n\nSection 6.1.  It seems like a truly fair comparison would include data generated under the EData assumption, e.g. correlation between covariates within datasets, Beta_q ~ N(xi, Gamma). I’d be curious how ECov performs when D < Q and when Q > D for this assumption.  ECov may still be better when D >> Q even if the “right” prior is exchangeability across covariates just do to statistical power issues.\n\nMinor comments:\n- Notation: it would be much more intuitive if there were D datasets and Q covariates (D for dataset), rather than the reverse.  This confused me many times.\n- Relatedly, I think of hierarchical modeling as models for data which can be naturally categorized into groups rather than separate datasets.  In my opinion, it would probably be much more natural to refer to Q groups rather than Q datasets.\n- Is Beta_ID just iid priors on all coefficients for all datasets across all covariates?\n- I’m curious about doubly exchangeable prior distributions? E.g. exchangeable across datasets and covariates.  Have you considered that at all?\n- Have you looked at full Bayes instead of empirical Bayes? May not be viable in very high dimensions but I’d be curious about the performance here.  The algorithms should be much simpler and likely could be implemented entirely in something like Stan.\n- The lemma about shrinkage along singular vector directions is interesting.  One comment here is that the eigenvalues of BetaBeta’ and Beta’Beta are connected via the singular vectors of Beta which relates the two approaches.  This might provide another interesting direction for thinking about this problem.  There’s also question about whether both data and covariates are correlated.  Efron comments about this in a paper about detecting row and column correlations in microarray data.  See also: “Inference with transposable data: modelling the effects of row and column correlations” (Allen and Tibshirani, 2012).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors largely address limitations, though I'd like to see more discussion about possible extensnions as mentioned in my review:\n1) structured (low rank) covariance matrices\n2) full Bayesian inference (e.g. a hyperprior on Sigma)\n3) more transparent simulations ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "o9ys1d2ql5b", "reviewer": "Reviewer_RrAL", "summary": "Paper contains a simple and effective result on analyzing datasets shared across a domain or in other words multiple sources with promising theoretical results supported by some experiments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: + The main contribution of the paper is an Alternating minimization procedure. To describe it formally, in each step a covariance type matrix is estimated for each dimension of the covariate based on regressors for that dimension -- importantly, these are estimated with data from all the sources. The paper proposes to use Conjugate Gradient (CG) algorithm algorithm to solve the inner least squares (or logistic) regression problems. \n+ Explains how block diagonal structure can be exploited when implementing CG algorithm. Experimental results with three different datasets (in addition to simulations) compared with show that the proposed method \n\nWeakness: - The theoretical justification presented in Sections 4 and 5 for the proposed algorithm seems correct, although it is a bit out of place. At a high level, Section 4 in the paper argues by associativity that the proposed estimator using ECov dominates the naive estimator EData by exploiting the closed form solution available for Least Squares problem. However, it is not clear whether the results presented can be adapted for classification setups. In Section 5, near Line 226 the paper mentions that risk depending on non-central inverse Wishart matrix is challenging, and that is the main motivation to use average case risk as done in Bayesian analysis. However, the results presented in Lemma and Theorem both use eigenvalues in the Gain. It would have been nice to read why we do not face the challenge anymore. The paper presents some simulations to support the theory presented but it falls on the short side -- for example, it is not clear whether any of the benefits of Theorem 5.3 can be realized in practice since only simulation results are shown. \n- Somewhat related to the previous point, the paper has no discussion on PAC-Bayes methods which are also a standard way to perform Bayesian generalization analysis.\n- Only preliminary experiments are presented paper which is surprising given that CG algorithm is indeed scalable, especially in the sparse settings considered here.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Addressed appropriately.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "VBBqU3AOpaA", "reviewer": "Reviewer_rxbW", "summary": "This paper studies a setting in which we analyze $Q$ datasets with $D$ covariates. The design matrix for the $q$-th dataset is denoted by $X^q$, a $N_q \\times D$-dimensional matrix and the response is a $N_q$ dimensional vector denoted by $Y^q$. The question this paper addresses is the following. How can one share information across datasets and covariates to improve the predictive performance for each individual regression? The paper focuses mostly on the case of linear as well as logistic regression.\n\nThe motivation for the paper is that most existing approaches to the above problem posit exchangeability across datasets, for example, that the coefficient vector $\\\\beta^q \\\\in \\\\mathbb R^D$ in the $q$-th dataset, is drawn from the same prior as $\\\\beta^{q'}$ for another dataset $q'$. In contrast, the main suggestion in this paper is to assume exchangeability across covariates and that $(\\\\beta^{1}_j, ..., \\\\beta^{Q}_j) \\\\in \\\\mathbb R^Q$ (the vector that collects the coefficient of covariate $j$ across all datasets) is drawn from the same prior distribution for all covariates $j=1,...,D$. It is claimed, that the second approach may be preferable when the number of covariates is larger than the number of datasets.\n\nTo implement the abstract proposal above, a multivariate normal prior is assumed for  $(\\\\beta^{1}_j, ..., \\\\beta^{Q}_j)$ and an empirical Bayes approach is developed and the prior covariance matrix is estimated by marginal maximum likelihood through the EM algorithm.\n\nThere are real data analyses that demonstrate that the approach performs well. There is also theoretical justification. In one part of the theory, James-Stein-type dominance results are established (in a frequentist setting where the true coefficients are deterministic), and in the second part, some results are established in a Bayesian setting (where a prior exists and is well-specified) with asymptotics as $D \\to \\infty$.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper is well-written. I liked the James-Stein type results and the applications of the new method. As the authors argue, their approach is particularly well-suited to situations where it is important to develop interpretable models. Furthermore, the new approach is effective at sharing information. Below are my main comments on the manuscript.\n\n\n1) I do not mind the assumption of orthonormal sample covariance in Section 4, because the results derived are strong (James-Stein-type results). However, for Section 5, where an asymptotic approach is pursued, I feel the assumption is too strong. Instead, a Random Matrix theoretic approach (taking the covariates as being random, with suitable rescaling, and perhaps also letting $N_q$ grow as $D \\to \\infty$) it should be possible to provide sharp results without the very stringent Condition 4.2.\n\n\n2) The main motivation for the new approach does not seem very to be sufficiently justified. In particular, I am not convinced by the introductory message that everyone in machine learning (in contrast to genetics) is following the Lindley-Smith approach of exchangeability across datasets. For example, I think it would be important that the authors provide references to previous NeurIPS papers (or machine learning papers more generally) that pursue the approach against which they argue. In fact, the basic idea of exchangeability across covariates, appears to have been embraced by the machine learning community previously. For example, a common approach for multitask learning is to seek the same sparsity pattern across datasets (for example, to use a group-lasso penalty on the coefficients of each covariate across all datasets, enforcing that some covariates will be zeroed out across all datasets). Such approaches also have a formal Bayes justification (as MAP estimators with priors that posit exchangeability across covariates). Some example references are the following:\n\nObozinski, Guillaume, Ben Taskar, and Michael Jordan. \"Multi-task feature selection.\" Statistics Department, UC Berkeley, Tech. Rep 2.2.2 (2006): 2.\n\nYang, X., Kim, S. and Xing, E., 2009. Heterogeneous multitask learning with joint sparsity constraints. Advances in neural information processing systems, 22, pp.2151-2159.\n\nLee, S., Zhu, J. and Xing, E., 2010. Adaptive Multi-Task Lasso: with Application to eQTL Detection. Advances in Neural Information Processing Systems, 23, pp.1306-1314.'\n\n\n\n3) The claim \"it is used by default in the mixed modeling package lme4 [5], which has over 13 million downloads at the time of writing\" needs some clarification. What is meant by \"default\" here? \n\n4) I would be interested in seeing the covariance matrices estimated in the real data examples.\n\n5)   Supplement D4, line 1113, there seems to be a typo in the proof (first equality).  \n\n6) This is very minor, but I found the notation $Q$ for number of datasets and $D$ for number of covariates a bit confusing and I kept needing to remind myself that $D$ is not the number of datasets.\n\n*Update*: I would like to thank the authors for a very well-thought and detailed response. Correspondingly I have raised my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "TgvhTV2Qof", "reviewer": "Reviewer_ZNBH", "summary": "This paper argues that when the number of covariates exceeds the number of datasets, it is more natural to model effects as exchangeable across covariates than across datasets. The paper illustrates the setting using a microcredit example.   It presents a hierarchical model to formalize the perspective and derive an efficient algorithm. The paper evaluates the model using three real-world datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros\n- the idea is interesting: why shouldn't we treat the effect as exchangeable among covariates? \n- The paper establishes that theoretically, Ecov may have better frequentist risk than Edata. Further, it uses Bayes risk to show that Ecov may improve the estimation quality in the high dimensional settings.\n- The proposed method, Ecov, is evaluated on a range of different real-world applications from image classification to social science study.\n\nMy main concerns are about (1) the motivation of the method and (2) the writing. \n\n(1) The core of the paper is arguing that one should treat the effect to be exchangeable among covariates.\nIn various parts of the paper, e.g., 7, 42, 100, the authors claim that Ecov is more “natural”. However, it’s not clear what is meant here. In line 78, the authors explain the intuition using a micro-credit dataset. However, in the experiment, there was no intuition or justification for the method. A priori, shouldn’t it be the case that what is exchangeable is dependent on the question at hand? This issue was only raised briefly in the discussion.  It should be discussed in detail in the introduction and throughout the paper.\n\n(2) The overall writing is not clear. Notably, section 5 is supposed to justify how Ecov leverages similarities across datasets in a meaningful way. However, the writing is not precise and is difficult to follow. For example, in line 222, it is unclear what “our method applied to each dataset independently” means. \n\nOther comments\n- Please use a reference package. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were initially concerned that this paper was not well-suited for NeurIPS, but better suited to a statistics journal. However, during the discussion phase the authors were able to convince the reviewers that they were able to make the necessary changes to make this right for NeurIPS. Therefore I move to accept. The authors should make sure to make the serious changes suggested by the authors and then this paper will be a nice contribution to the conference.
Decision: Accept (Poster)