Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes an efficient optimization method for estimating the full noise covariance in a hierarchical Bayesian framework. It's shown in the experiment that the optimization method could recover the true noise covariance in a simulated example and estimating the full covariance has better performance than homo- and heteroscedastic covariance.

I think the proposed method is an effective tool to estimate the full noise covariance especially for the problem setting in this paper. But the overall novelty and contribution are not strong enough for the ICLR community.

Papers in fMRI literature [Michael Shvartsman et al 2017, Anqi Wu et al 2019] have proposed to work with full noise covariance in more complicated models such as factor analysis, Gaussian process regression. The basic model in this paper is a bit too simple compared with other models preventing from making significant methodological contributions. It might fit a signal processing or brain source imaging specialized publication better.

Also in many applications (especially with brain data), it's shown that a full rank noise covariance is not always preferable given that there are usually some correlations among measurements that lead to lower dimensional subspace at the noise level. So I'm not quite sure whether a full covariance without any structural or subspace assumption would really outperform low-rank full covariance when applying to the real data.

Another issue in this paper is there is no real data application. I'm not very convinced that simulated data generated from a realistic lead field matrix is considered as the real-world data. 

review_title: an effective optimization method for full noise covariance estimation but the novelty is not strong enough

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a methodology for type-II maximum likelihood on a hierarchical Bayesian model for EEG signals. The particular feature of the model, which separates it from other EEG models, is the consideration of a full covariance matrix which makes the noise correlated and heteroskedastic. 

The model, as claimed by the authors, is fully Gaussian and therefore tractable. As a consequence, the inference poses no challenges other than the computational complexity. To address this, the authors propose a mechanism for, what they claim is, efficient optimisation. This contribution alone is not sufficient (over the standard literature) for publication as a theoretical improvement. 

Given the lack of a theoretical advancement, I was hoping that the contribution of the article came in the experimental treatment, however, it was not the case.  A single set of experiments using synthetic data was considered, where the proposed method was compared against other benchmark. It is far form surprising when the authors deal with exact inference on a model where the observations where produced under the same statistical assumptions.

I also would like to emphasise that the discussion of the paper states that  "This paper proposes an efficient optimization algorithm for jointly estimating...." and "The benefits of our proposed framework were evaluated within an extensive set of experiments ". None of these claims are true or at least they not validated by any supporting evidence in the paper. 

Perhaps with the stated future work and stronger experimental results (real data), this paper can be improved. 


review_title: Well known setting in the literature, limited experimental validation. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
The paper generalised Type-II ML regression models for scenarios where different noise dimensions cannot be assumed independent, but instead one needs to model the full covariance structure. This is clearly an important problem and it is well motivated in the work.

Reasons for score:
I recommend rejecting the paper even though it represents high-quality work in statistics, because I think it is somewhat tangentially related to ICLR and the contribution would be better appreciated in a different venue.

Strong points:  (1) Addresses an important problem. (2) Seems to work well in practice

Weaknesses: (1) Limited conceptual novelty. (2)Technical contribution hidden in Appendix

Detailed review:
The work addresses a relevant statistical question of accounting for correlated noise in hierarchical linear regression, but feels somewhat of a poor fit for ICLR. It formally fits within the scope, but still feels out of place in the sense that neither readers interested in the theoretical contributions nor people looking to apply these methods would consider ICLR as a natural venue to look for the information. The development is restricted to a specific, relatively simple, model family that is frequently used in several fields but that is not at the core of the ICLR community. This is highlighted also by the fact that the technical contribution is largely in statistical properties of the covariance estimator, and for this audience gets hidden in the Appendix. Consequently, I believe that paper would much more naturally fit into a publication forum in statistics.

The proposed approach itself is sound and well developed. Accounting for correlated noise is a very obvious thing to do, but the technical details are non-trivial. The authors rely on Riemannian optimisation for covariance matrices and are able to use the recent Champagne algorithm for SBL. The detailed derivation of Theorem 2 shows non-trivial technical contribution, but remains somewhat isolated as it is hidden in the Appendix. For example, there is no discussion on whether the result derived here would have uses also in other model families. I can see several potential uses for better tools for learning full covariance noise e.g. in matrix factorisation models (e.g. probabilistic CCA relies on covariance estimates) or non-linear regression models, but the authors do not discuss this at all. A proper discussion on this would be important to link the work more closely to the broader activities in the field, to extend the contribution beyond the current viewpoint of a very specific model.

The empirical experiments are well carried out and demonstrate the value of learning the full covariance matrix compared to methods that only operate with diagonal noise. This is sufficient, since no clear comparison methods accounting for full covariance are available.
review_title: Improved treatment of correlated noise for linear regression, but somewhat odd choice of publication forum

final_score: rating: 5: Marginally below acceptance threshold
review_content: Joint Learning of Full-structure Noise in Hierarchical Bayesian Regression Models

Summary:

The paper argues that modeling the full covariance structure in a sparse bayessian learning setting leads to significantly better results in eeg inverse problems. The paper details a majorization-minimization type algorithm leading to a set of fairly simple update rules. The proposed method is evaluated on simulated data.

Positive:

1. The proposed method is well motivated and the problem is highly relevant.
2. The mathematical details regarding the algorithm are presented in sufficient detail.
3. The paper is well written and easy to follow for the most part.
4. Experiments are reasonable and presented clearly.

Negative:

1. The abstract could be improved to more clearly describe the problem and contributions of the paper in a self-contained manner.
2. Has this particular problem (sparse bayesian regression with full covariance noise) not been considered by others? The main contribution, in my view, is algorithmic; which other algorithms have been used previously for this type of problem? (I think both ML-II and MCMC and possibly other methods have previously been used.) I would have liked a review and experimental comparison.
3. While the experimental evaluation is reasonable, I think the paper would benefit from a demonstration and benchmarking with competing approaches on a real data task.
4. Experiments on simulated data highlighting more clearly the *algorithmic* advantages of the proposed method would be appreciated.
5. I did not notice a link to software implementing the proposed method? Sharing software implementations will significantly strengthen the contribution and allow the community to reproduce the results.

Recommendation:

Weak reject.



review_title: Well written and interesting. Experimental evaluation could be improved. Novelty limited.

Meta-review: The paper presents hierarchical Bayesian methods for modelling the
full covariance structure in cases where noise dimensions cannot be
assumed independent.

This is an important problem with potential practical importance. The
work is solid.

Conceptual novelty in the work is somewhat limited.

The method is applied in the paper on hierarchical linear
regression. It is claimed to be applicable to other methods as well,
and the claim is plausible, but to be fully convincing, results and
comparisons would need to be shown. The new extended discussion does
help somewhat.

There was also discussion about whether ICLR is the best match for
this work. This is not a strereotypical ICLR paper though is relevant.

Authors are encouraged to continue this line of work.

Decision: Reject