Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Paper contains a simple and effective result on analyzing datasets shared across a domain or in other words multiple sources with promising theoretical results supported by some experiments.
main_review: Strength: + The main contribution of the paper is an Alternating minimization procedure. To describe it formally, in each step a covariance type matrix is estimated for each dimension of the covariate based on regressors for that dimension -- importantly, these are estimated with data from all the sources. The paper proposes to use Conjugate Gradient (CG) algorithm algorithm to solve the inner least squares (or logistic) regression problems. 
+ Explains how block diagonal structure can be exploited when implementing CG algorithm. Experimental results with three different datasets (in addition to simulations) compared with show that the proposed method 

Weakness: - The theoretical justification presented in Sections 4 and 5 for the proposed algorithm seems correct, although it is a bit out of place. At a high level, Section 4 in the paper argues by associativity that the proposed estimator using ECov dominates the naive estimator EData by exploiting the closed form solution available for Least Squares problem. However, it is not clear whether the results presented can be adapted for classification setups. In Section 5, near Line 226 the paper mentions that risk depending on non-central inverse Wishart matrix is challenging, and that is the main motivation to use average case risk as done in Bayesian analysis. However, the results presented in Lemma and Theorem both use eigenvalues in the Gain. It would have been nice to read why we do not face the challenge anymore. The paper presents some simulations to support the theory presented but it falls on the short side -- for example, it is not clear whether any of the benefits of Theorem 5.3 can be realized in practice since only simulation results are shown. 
- Somewhat related to the previous point, the paper has no discussion on PAC-Bayes methods which are also a standard way to perform Bayesian generalization analysis.
- Only preliminary experiments are presented paper which is surprising given that CG algorithm is indeed scalable, especially in the sparse settings considered here.
limitations_and_societal_impact: Addressed appropriately.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper argues that when the number of covariates exceeds the number of datasets, it is more natural to model effects as exchangeable across covariates than across datasets. The paper illustrates the setting using a microcredit example.   It presents a hierarchical model to formalize the perspective and derive an efficient algorithm. The paper evaluates the model using three real-world datasets.
main_review: pros
- the idea is interesting: why shouldn't we treat the effect as exchangeable among covariates? 
- The paper establishes that theoretically, Ecov may have better frequentist risk than Edata. Further, it uses Bayes risk to show that Ecov may improve the estimation quality in the high dimensional settings.
- The proposed method, Ecov, is evaluated on a range of different real-world applications from image classification to social science study.

My main concerns are about (1) the motivation of the method and (2) the writing. 

(1) The core of the paper is arguing that one should treat the effect to be exchangeable among covariates.
In various parts of the paper, e.g., 7, 42, 100, the authors claim that Ecov is more “natural”. However, it’s not clear what is meant here. In line 78, the authors explain the intuition using a micro-credit dataset. However, in the experiment, there was no intuition or justification for the method. A priori, shouldn’t it be the case that what is exchangeable is dependent on the question at hand? This issue was only raised briefly in the discussion.  It should be discussed in detail in the introduction and throughout the paper.

(2) The overall writing is not clear. Notably, section 5 is supposed to justify how Ecov leverages similarities across datasets in a meaningful way. However, the writing is not precise and is difficult to follow. For example, in line 222, it is unclear what “our method applied to each dataset independently” means. 

Other comments
- Please use a reference package. 
limitations_and_societal_impact: see above
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies a setting in which we analyze $Q$ datasets with $D$ covariates. The design matrix for the $q$-th dataset is denoted by $X^q$, a $N_q \times D$-dimensional matrix and the response is a $N_q$ dimensional vector denoted by $Y^q$. The question this paper addresses is the following. How can one share information across datasets and covariates to improve the predictive performance for each individual regression? The paper focuses mostly on the case of linear as well as logistic regression.

The motivation for the paper is that most existing approaches to the above problem posit exchangeability across datasets, for example, that the coefficient vector $\\beta^q \\in \\mathbb R^D$ in the $q$-th dataset, is drawn from the same prior as $\\beta^{q'}$ for another dataset $q'$. In contrast, the main suggestion in this paper is to assume exchangeability across covariates and that $(\\beta^{1}_j, ..., \\beta^{Q}_j) \\in \\mathbb R^Q$ (the vector that collects the coefficient of covariate $j$ across all datasets) is drawn from the same prior distribution for all covariates $j=1,...,D$. It is claimed, that the second approach may be preferable when the number of covariates is larger than the number of datasets.

To implement the abstract proposal above, a multivariate normal prior is assumed for  $(\\beta^{1}_j, ..., \\beta^{Q}_j)$ and an empirical Bayes approach is developed and the prior covariance matrix is estimated by marginal maximum likelihood through the EM algorithm.

There are real data analyses that demonstrate that the approach performs well. There is also theoretical justification. In one part of the theory, James-Stein-type dominance results are established (in a frequentist setting where the true coefficients are deterministic), and in the second part, some results are established in a Bayesian setting (where a prior exists and is well-specified) with asymptotics as $D \to \infty$.
main_review: 
The paper is well-written. I liked the James-Stein type results and the applications of the new method. As the authors argue, their approach is particularly well-suited to situations where it is important to develop interpretable models. Furthermore, the new approach is effective at sharing information. Below are my main comments on the manuscript.


1) I do not mind the assumption of orthonormal sample covariance in Section 4, because the results derived are strong (James-Stein-type results). However, for Section 5, where an asymptotic approach is pursued, I feel the assumption is too strong. Instead, a Random Matrix theoretic approach (taking the covariates as being random, with suitable rescaling, and perhaps also letting $N_q$ grow as $D \to \infty$) it should be possible to provide sharp results without the very stringent Condition 4.2.


2) The main motivation for the new approach does not seem very to be sufficiently justified. In particular, I am not convinced by the introductory message that everyone in machine learning (in contrast to genetics) is following the Lindley-Smith approach of exchangeability across datasets. For example, I think it would be important that the authors provide references to previous NeurIPS papers (or machine learning papers more generally) that pursue the approach against which they argue. In fact, the basic idea of exchangeability across covariates, appears to have been embraced by the machine learning community previously. For example, a common approach for multitask learning is to seek the same sparsity pattern across datasets (for example, to use a group-lasso penalty on the coefficients of each covariate across all datasets, enforcing that some covariates will be zeroed out across all datasets). Such approaches also have a formal Bayes justification (as MAP estimators with priors that posit exchangeability across covariates). Some example references are the following:

Obozinski, Guillaume, Ben Taskar, and Michael Jordan. "Multi-task feature selection." Statistics Department, UC Berkeley, Tech. Rep 2.2.2 (2006): 2.

Yang, X., Kim, S. and Xing, E., 2009. Heterogeneous multitask learning with joint sparsity constraints. Advances in neural information processing systems, 22, pp.2151-2159.

Lee, S., Zhu, J. and Xing, E., 2010. Adaptive Multi-Task Lasso: with Application to eQTL Detection. Advances in Neural Information Processing Systems, 23, pp.1306-1314.'



3) The claim "it is used by default in the mixed modeling package lme4 [5], which has over 13 million downloads at the time of writing" needs some clarification. What is meant by "default" here? 

4) I would be interested in seeing the covariance matrices estimated in the real data examples.

5)   Supplement D4, line 1113, there seems to be a typo in the proof (first equality).  

6) This is very minor, but I found the notation $Q$ for number of datasets and $D$ for number of covariates a bit confusing and I kept needing to remind myself that $D$ is not the number of datasets.

*Update*: I would like to thank the authors for a very well-thought and detailed response. Correspondingly I have raised my score.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors of this paper study exchangeable prior distribution for regression coefficients in high dimensional regression problems, where the number of covariates is typically much larger than the number of groups.  They argue that when the number of covariates are large, it is preferable to model the covariates as exchangeable rather than the datasets. The authors provide theoretical results by comparing frequentist risk of different empirical Bayes estimators and demonstrate the value of their approach empirically.
main_review: Overall, I think this is an interesting, well written paper.  That said, I’m not really familiar enough with this literature to know for sure how impactful and insightful their results really are.  My gut instinct is that, while interesting, the results are not particular profound, if only because they are quite intuitive: in high dimension, low sample size settings covariance estimation is known to be quite hard, so using an empirical Bayes procedure to infer the covariance of Beta (across covariates) might be quite bad.  

Setting that aside, I have another main “complaint” which doesn’t really affect my overall assessment of the quality of the paper: I find it strange that this paper was submitted to neurips.  In my opinion, this paper is much more appropriate for a statistics journal.  I say this for two reasons: 1) As somebody who rarely cares about this distinction, I still found myself thinking that this paper is very clearly a statistics paper dealing with canonical statistical issues around normal means estimation, James Stein, regression etc.  2) this is a 9 page paper with a 40 page supplement— much of the supplement is explanatory text (not just proofs and additional figures).  It seems like this might be better in a longer journal paper.  I would encourage the authors to think about whether they should consider a more appropriate venue.

Regarding the comment about statistical power at the top of page 3, is this the main driver for improvements in the ECov over EData? Since Sigma is shared across all D covariates, as D->infinity you can consistently estimate Sigma, whereas with EData D > Q implies a very poor empirical Bayes estimator of Gamma.  

Relatedly, I’m curious about an estimator in which you learn an approximately low rank covariance matrix under EData, e.g. with Gamma = AA^T + sigma^2I which be a better competitor? For example, perhaps A is an D x k matrix k <= Q, so that the number of parameters in the covariance matrices to learn in EData and much smaller than the general exchangneable model. I think there is something to be learned here from the high dimensional covariance estimation literature.  In short, I think to make this really useful I’d like to know if I wouldn’t be better off just modeling exchangeability among datasets under a structured prior for the covariance (e.g low rank or sparse).  That may solve some of the problems encountered in the large D, small Q case.

Section 6.1.  It seems like a truly fair comparison would include data generated under the EData assumption, e.g. correlation between covariates within datasets, Beta_q ~ N(xi, Gamma). I’d be curious how ECov performs when D < Q and when Q > D for this assumption.  ECov may still be better when D >> Q even if the “right” prior is exchangeability across covariates just do to statistical power issues.

Minor comments:
- Notation: it would be much more intuitive if there were D datasets and Q covariates (D for dataset), rather than the reverse.  This confused me many times.
- Relatedly, I think of hierarchical modeling as models for data which can be naturally categorized into groups rather than separate datasets.  In my opinion, it would probably be much more natural to refer to Q groups rather than Q datasets.
- Is Beta_ID just iid priors on all coefficients for all datasets across all covariates?
- I’m curious about doubly exchangeable prior distributions? E.g. exchangeable across datasets and covariates.  Have you considered that at all?
- Have you looked at full Bayes instead of empirical Bayes? May not be viable in very high dimensions but I’d be curious about the performance here.  The algorithms should be much simpler and likely could be implemented entirely in something like Stan.
- The lemma about shrinkage along singular vector directions is interesting.  One comment here is that the eigenvalues of BetaBeta’ and Beta’Beta are connected via the singular vectors of Beta which relates the two approaches.  This might provide another interesting direction for thinking about this problem.  There’s also question about whether both data and covariates are correlated.  Efron comments about this in a paper about detecting row and column correlations in microarray data.  See also: “Inference with transposable data: modelling the effects of row and column correlations” (Allen and Tibshirani, 2012).

limitations_and_societal_impact: The authors largely address limitations, though I'd like to see more discussion about possible extensnions as mentioned in my review:
1) structured (low rank) covariance matrices
2) full Bayesian inference (e.g. a hyperprior on Sigma)
3) more transparent simulations
review_title: null

Meta-review: The reviewers were initially concerned that this paper was not well-suited for NeurIPS, but better suited to a statistics journal. However, during the discussion phase the authors were able to convince the reviewers that they were able to make the necessary changes to make this right for NeurIPS. Therefore I move to accept. The authors should make sure to make the serious changes suggested by the authors and then this paper will be a nice contribution to the conference.
Decision: Accept (Poster)