For high-dimensional hierarchical models, consider exchangeability of effects across covariates instead of across datasets

Hierarchical Bayesian methods enable information sharing across regression problems on multiple groups of data . While standard practice is to model regression parameters ( effects ) as ( 1 ) exchangeable across the groups and ( 2 ) correlated to differing degrees across covariates , we show that this approach exhibits poor statistical performance when the number of covariates exceeds the number of groups . For instance , in statistical genetics , we might regress dozens of traits ( defining groups ) for thousands of individuals ( responses ) on up to millions of genetic variants ( covariates ) . When an analyst has more covariates than groups , we argue that it is often preferable to instead model effects as ( 1 ) exchangeable across covariates and ( 2 ) correlated to differing degrees across groups . To this end , we propose a hierarchical model expressing our alternative perspective . We devise an empirical Bayes estimator for learning the degree of correlation between groups . We develop theory that demonstrates that our method outperforms the classic approach when the number of covariates dominates the number of groups , and corroborate this result empirically on several high-dimensional multiple regression and classification problems . 1 Introduction . Hierarchical modeling is a mainstay of Bayesian inference . For instance , in ( generalized ) linear models , the unknown parameters are effects , each of which describes the association of a particular covariate with a response of interest . Often covariates are shared across multiple related groups , but the effects are typically allowed to vary both by group and by covariate . A classic methodology , dating back to Lindley and Smith ( 1972 ) [ 44 ] , models the effects as conditionally independent across groups , with a latent ( and learnable ) degree of relatedness across covariates . From a practical standpoint , the model is motivated by the understanding that it “ borrows strength ” across different groups [ 24 , Chapter 5.6 ] . Mathematically , the model is motivated by assuming effects are exchangeable across groups and applying a de Finetti theorem [ 44 , 35 ] . The methodology of Lindley and Smith is ubiquitous when the number of groups is larger than the number of covariates . It is a standard component of Bayesian pedagogy [ [ 23 , Chapter 13.3 ] ; [ 24 , Chapter 15.4 ] ] and software ; e.g . it is used in the mixed modeling package lme4 [ 5 ] , which has over 16 million downloads at the time of writing . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Despite its resounding success when there are more groups than covariates , we show in the present work that this standard methodology performs poorly when there are more covariates than groups . To address the many-covariates case , we turn for inspiration to statistical genetics , where scientists commonly learn linear models relating genetic variants ( covariates ) to traits ( corresponding to different groups ) across individuals ( which each exhibit a response ) . These applications may exhibit millions of covariates , thousands of responses , and just a handful of groups . In these cases , [ 39 , 12 , 56 , 69 , 46 , 53 ] use a multivariate Gaussian prior akin to that of Lindley and Smith , but assume conditional independence across covariates and prior parameters that encode correlations across groups , rather than than the other way around . As we will see , this alternative modeling approach may be motivated from a Bayesian perspective when one begins from an assumption of a priori exchangeability of the effects across covariates ( rather than across groups ) . This exchangeability assumption is reasonable in statistical genetics , where we have little knowledge to distinguish our expectations about the effects of different genetic variants ; we argue this modeling approach can be effective other modern high-dimensional analyses of multiple groups of data ( beyond statistical genetics ) in which large collections of covariates are frequently treated monolithically , e.g . by applying ridge regression . Namely , when there are more covariates than groups , we propose to model the effects as exchangeable across covariates ( rather than groups ) and learn the degree of relatedness of effects across groups ( rather than covariates ) . In what follows , we refer to this framework as ECov , for exchangeable effects across covariates , and distinguish it from exchangeable effects across groups or EGroup . While the existing methods in statistical genetics for modeling multiple traits obtain as a special case of ECov , to the best of our knowledge this approach is absent from existing literature on hierarchical Bayesian regression . Brown and Zidek ( 1980 ) [ 10 ] and Haitovsky ( 1987 ) [ 28 ] form two exceptions , but these two papers ( 1 ) consider only the situation in which a single covariate matrix is shared across all groups ( or equivalently , for each data point all responses are observed ) and ( 2 ) include only theory and no empirics . While Lindley and Smith ( and others ) discuss a priori exchangeability across covariates in the context of analysis of a single group , to our knowledge no other work has pushed this idea forward to share strength across multiple groups . We suspect that the historical origins of the methodology in statistical genetics may have hindered earlier expansion of this class of models to a wider audience . In particular , this literature traces back to mixed effects modeling for cattle breeding [ 57 ] ; here , an even-earlier notion of the genetic contribution of trait correlation ( i.e . “ genetic correlation ; ” see Hazel ( 1943 ) [ 29 ] ) informs the covariance structure of random effects . Although genetic correlation is now commonly understood to describe the correlation of effects of DNA sequence changes on different traits [ 12 ] , its provenance predates even the first identification of DNA as the genetic material in 1944 [ 3 ] . As such , this older motivation obviated the need for a more general justification grounded in exchangeability . See Appendix A for further discussion of related work , including more recent works from within the machine learning community on sharing strength across multiple groups of data . In the present work , we propose ECov as a general framework for hierarchical regression when the number of covariates exceeds the number of groups . We show that the classic model structure from statistical genetics can be seen as an instance of this framework , much as Lindley and Smith give a ( complementary ) instance of an EGroup framework . To make the ECov approach generally practical , we devise an accurate and efficient algorithm for learning the matrix of correlations between groups . We demonstrate with theory and empirics that ECov is preferred when the number of covariates exceeds the number of groups , while EGroup is preferred when the number of groups exceeds the number of covariates . Our experiments analyze three real , non-genetic groups in regression and classification , including an application to transfer learning with pre-trained neural network embeddings . We provide proofs of theoretical results in the appendix . 2 Exchangeability and its applications to hierarchical linear modeling . We start by establishing the data and model , motivating exchangeability among covariate effects ( ECov ) , and motivating our Bayesian generative model . Setup and notation . Consider Q groups with D covariates . Let Nq be the number of data points in group q . For the qth group , the Nq ⇥D real design matrix Xq collects the covariates , and Y q is the Nq-vector of responses . The nth datapoint in group q consists of covariate D-vector Xqn and scalar response Y qn . We let D : = { ( Xq , Y q ) } Q q=1 denote the collection of data from all Q groups . We consider the generalized linear model Y qn |Xqn , q indep⇠ p ( ·|Xq > n q ) with unknown D-vector of real effects q . We collect all effects in a D⇥Q matrix with ( d , q ) entry qd . The linear form of the likelihood allows interpretation of qd as the association between the dth covariate and the response in group q . In linear regression , the responses are real-valued and the conditional distribution is Gaussian . In logistic regression , the responses are binary , and we use the logit link . The independence assumption conflicts with some models that one might use , for example in some cases when the different groups partially overlap . Example . As a motivating non-genetics example , consider a study of the efficacy of microcredit . There are seven famous randomized controlled trials of microcredit , each in a different country [ 48 ] . We might be interested in the association between various aspects of small businesses ( covariates ) , including whether or not they received microcredit , and their business profit ( response ) . In this case , the dth element of Xqn would be the dth characteristic of the nth small business in the qth country , and Y qn is the profit of this business . See the experiments for additional examples in rates of policing , web analytics , and transfer learning . Exchangeable effects across groups ( EGroup ) . To fully specify a Bayesian model , we need to choose a prior over the parameters . Lindley and Smith assume the effects are exchangeable across groups . Namely , for every Q-permutation , p ( 1 , 2 , . . . , Q ) = p ( ( 1 ) , ( 2 ) , . . . , ( Q ) ) . Assuming exchangeability holds for an imagined growing Q and applying de Finetti ’ s theorem motivates a conditionally independent prior . Concretely , Lindley and Smith take q i.i.d.⇠ N ( ⇠ , ) , for D-vector ⇠ and D⇥D covariance matrix . The ( d , d0 ) entry of captures the degree of relatedness between the effects for covariates d and d0 . Both ⇠ and may be learned in an empirical Bayes procedure . However , when D is large relative to Q , learning these parameters can present both computational and inferential challenges , as the O ( D2 ) degrees of freedom in outnumber the O ( DQ ) effects . Exchangeable effects across covariates ( ECov ) . We here argue for a complementary approach in settings where D > Q . In the microcredit example , notice that D > Q will arise whenever the experimenter records more characteristics of a small business than there are locations with microcredit experiments ; that is , D > 7 in this particular case . Concretely , let d be the Q-vector of effects for covariate d across groups . Then , in the ECov approach , we will assume that effects are exchangeable across covariates instead of across groups . Namely , for every D-permutation , p ( 1 , 2 , . . . , D ) = p ( ( 1 ) , ( 2 ) , . . . , ( D ) ) . We will see theoretical and empirical benefits to ECov in later sections , but note that the ECov assumption is often consistent with prior beliefs in high dimensional settings . For instance , regarding microcredit , we may have no prior knowledge about how effects differ for distinct small-business characteristics . And we may a priori believe that different countries could exhibit more similar effects – and wish to learn the degree of relatedness across those countries . We may apply a similar rationale as Lindley and Smith to motivate a conditionally independent model . Analogous to Lindley and Smith , we propose a Gaussian prior : d i.i.d.⇠ N ( 0 , ⌃ ) . ⌃ is now a Q⇥Q covariance matrix whose ( q , q0 ) entry captures the similarity between the effects in the q and q0 groups . For simplicity , we restrict to E [ d ] = 0 ; see Appendix E.3 for discussion . Another potential benefit to ECov relative to EGroup is that we might expect a statistically easier problem , with O ( Q2 ) rather than O ( D2 ) values to learn in the relatedness matrix . We provide a rigorous theoretical analysis in Sections 4 and 5 .

Summary: This paper studies a setting in which we analyze $Q$ datasets with $D$ covariates. The design matrix for the $q$-th dataset is denoted by $X^q$, a $N_q \times D$-dimensional matrix and the response is a $N_q$ dimensional vector denoted by $Y^q$. The question this paper addresses is the following. How can one share information across datasets and covariates to improve the predictive performance for each individual regression? The paper focuses mostly on the case of linear as well as logistic regression.  The motivation for the paper is that most existing approaches to the above problem posit exchangeability across datasets, for example, that the coefficient vector $\\beta^q \\in \\mathbb R^D$ in the $q$-th dataset, is drawn from the same prior as $\\beta^{q'}$ for another dataset $q'$. In contrast, the main suggestion in this paper is to assume exchangeability across covariates and that $(\\beta^{1}_j, ..., \\beta^{Q}_j) \\in \\mathbb R^Q$ (the vector that collects the coefficient of covariate $j$ across all datasets) is drawn from the same prior distribution for all covariates $j=1,...,D$. It is claimed, that the second approach may be preferable when the number of covariates is larger than the number of datasets.  To implement the abstract proposal above, a multivariate normal prior is assumed for  $(\\beta^{1}_j, ..., \\beta^{Q}_j)$ and an empirical Bayes approach is developed and the prior covariance matrix is estimated by marginal maximum likelihood through the EM algorithm.  There are real data analyses that demonstrate that the approach performs well. There is also theoretical justification. In one part of the theory, James-Stein-type dominance results are established (in a frequentist setting where the true coefficients are deterministic), and in the second part, some results are established in a Bayesian setting (where a prior exists and is well-specified) with asymptotics as $D \to \infty$.