Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper explores the problem of deep heteroskedastic multivariate regression where the goal is to regress over symmetric positive definite matrices; that is, the deep learning model should take as input data points, and produce a conditional covariance matrix as the output. The key challenge in this setting is how to ensure the predicted matrix is positive definite (and thus follows the non-linear geometry of these matrices), how the neural network can be trained for this task, and what loss function can be used for effective training. The paper proposes a neural network with bilinear layers in this regard, and uses the von Neumann divergence as the loss function to regress the predicted covariance against a ground truth SPD matrix. The gradients of the von Neumann divergence are provided for learning via backpropagation. Experiments on several synthetic datasets and small scale datasets are provided, showcasing some benefits. 

Pros: 
1. The use of von Neumann divergence as a loss for this task is perhaps novel.
2. The use of \alpha-derivatives, while computationally demanding, is perhaps novel in this context as well. 

Cons:
1. I do not think the problem setting or the proposed framework is entirely new or is the best choice of its ingredients. Specifically, the idea of using second-order neural networks have been attempted in several prior papers, including the ones the paper cite (such as Ionescu et al. ,2015). Several other papers in this regard are listed below. 
[a] Second-order Temporal pooling for action recognition, Cherian and Gould, IJCV, 2018
[b] Deep manifold-to-manifold transforming network, Zhang et al, ICIP, 2018
[c] Statistically motivated second-order pooling, Yu and Salzmann, ECCV, 2018

In comparison to these methods, it is not clear how the proposed setup is novel, or in what way is method better. There are no comparisons to these methods, and thus it is difficult to judge the benefits even empirically. 

2. There are also models that predict the mean and covariance matrices directly from the model, such as the works below. The paper should also include and perhaps compare to their datasets.
[d] Deep Inference for Covariance Estimation: Learning Gaussian Noise Models for State Estimation, Liu et al, ICRA, 2018

3. I do not think the use of von Neumann divergence as a loss is the best choice one could have, esp. for a deep neural network learning setting. This divergence includes the matrix logarithm, which is perhaps computationally expensive. This is a problem when using other popular loss/similarity functions on SPD matrices (such as the log-Euclidean metric). Perhaps a better option is to use the Jensen-Bregman log-det divergence, as suggested in [a] above; this divergence is symmetric and also has computationally efficient gradients. It is unclear why the paper decided to use von Neumann. 

4. The experiments are not compelling, there are no comparisons to alternative models and the datasets used are small scale. Thus, it is unclear if the design choices in the paper have any strong bearing in the empirical performances.

Overall, the paper makes an attempt at designing neural networks for learning SPD matrices. While, there are some components in the model that are perhaps new, the paper lacks any justifications for their choices, and as such these choices seem inferior to alternatives that have been proposed earlier. Also, the experimental results are not convincing against prior works. 


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This manuscript proposes a novel formulation of the MLP to address predicting a symmetric positive definite (SPD) matrix from an input vector or matrix.  While the field has had methods for years to estimate SPD matrices (such as the covariance matrix estimate in the reparameterization trick), this manuscript proposes a markedly different approach based on a different layer structure and repeated normalization steps based on Mercer Kernels.  Additionally, the loss used can be modified to use more PSD-specific losses, such as the symmetrized von Neumann divergence, rather than the traditional quadratic loss.  This loss appears to give significantly better solutions on synthetic data.

While there is some interesting and potentially useful novelty in the approach, I have some concerns about the empirical evidence and modeling to truly determine the mMLP's utility.

First, in the synthetic data the mMLP with the l_QRE loss outperforms the l_quad loss with regards to both E_QRE and E_quad.  Why does mMLP/l_QRE outperform on E_quad? As l_quad is actually designed to minimize this error, this is surprising and needs additional explanation.  Additionally, there are two major changes between mMLP and the MLP.  One is the network structure, and the second is the trace being normalized; which change really induces the improvements in the performance? If the l_QRE loss was used in the MLP, would you get similar improvements?

The network structure as a whole needs greater validation.  A major difference in (3) is that the weight matrices W_l are applied as a both a left and right multiplication.  Given that the \mathcal{H} operation symmetrizes and normalizes the matrix, a symmetric operation isn't strictly necessary here.  Using both multiplications leads to quadratic properties, which in my experience are less stable in the optimization.  Can the authors validate this structure versus the simpler structure of simply using left multiplications?  Or, in other words, is this weight multiplication structure helpful or do the benefits really come from the \mathcal{H} operation.

I think that the heteroscedastic regression experiments don't evaluate on one of the key issues, which is uncertainty estimation.  The trace-1 normalization is highly restrictive, so I imagine that that this method is getting the uncertainty incorrect.  Also, heteroscedastic regression has a long history in neural networks, dating back to at least Nix and Wiegand in 1994.  The manuscript needs to be updated to reflect the historical work and current literature on the topic.

Please check Table 5(a), which states that you are only using a small number of training samples.  Also, given the relatively small sample size of these datasets, please comment on the uncertainty of the results.  How confident are you that the methods actually improve the prediction?  How were the competing models tuned and optimized?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper generalized neural networks into case where a semidefinite positive matrix is learned at the output. The paper presents theoretical derivations that look sound, and validating experiments on synthetic and real data. I must say my expertise does not really correspond to what is done in this paper, but I do not see any obvious flaws and the results look solid. I appreciated the discussion of limitations in section 6. I vote for acceptance with the weakest possible confidence level since it is likely I missed many important points. 
review_title: Official Blind Review #2

Meta-review: This paper introduces a novel architecture and loss for estimating PSD matrices using neural networks.  There is some theoretical justification for the architecture, and a small-scale but encouraging experiment.

Overall, I think there is a sensible contribution here, but there are so many architectural and computational choices presented together at once that it's hard to tell what the important parts are.

The main problems with this paper are:
1) The scalability of the approach O(N^3)
2) The derivation of the architecture and gradient computations wasn't clear about what choices were available and why.  Several alternative choices were mentioned but not evaluated.  I think the authors also need to improve their understanding of automatic differentiation.  Backprop through eigendecomposition is already available in most autodiff packages.  It was claimed that a certain kind of matrix derivative provided better generalization, which seems like a strong claim to make in general.
3) The experimental setup seemed contrived, except for the heteroskedastic regression experiments, which lacked competitive baselines.  Why were the GP and MLPs homoskedastic?

As a matter of personal preference, I found that having 4 different "H"s differing only in font and capitalization for the network architecture was hard to keep track of.

I agree that R1 had some unjustified comments and R2's review was contentless.  I apologize for these inadequate reviews. 
Decision: Reject