Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper uses an information geometric view on hierarchical models to discuss a bias - variance decomposition in Boltzmann machines, presenting interesting conclusions, whereby some more care appears to be needed for making these claims. 

The paper arrives at the main conclusion that it is possible to reduce both the bias and the variance in a hierarchical model. The discussion is not specific to deep learning nor to Boltzmann machines, but actually addresses hierarchical exponential family models. The methods pertaining hierarchical models are interesting and presented in a clear way. My concern are the following points: 

The main theorem presents only a lower bound, meaning that it provides no guarantee that the variance can indeed be reduced. 

The paper seems to ignore that a model with hidden variables may be singular, in which case the Fisher metric is not positive definite and the Cramer Rao bound has no meaning. This interferes with the claims and derivations made in the paper in the case of models with hidden variables. The problem seems to lie in the fact that the presented derivations assume that an optimal distribution in the data manifold is given (see Theorem 1 and proof), effectively making this a discussion about a fully observed hierarchical model. In particular, it is not further specified how to obtain θˆB(s) in page 6 before (13). 

Also, in page 5 the paper states that ``it is known that the EM-algorithm can obtain the global optimum of Equation (12) (Amari, 2016, Section 8.1.3)''. However, what is shown in that reference is only that:  (Theorem 8.2., Amari, 2016) ``The KL-divergence decreases monotonically by repeating the E-step and the M-step. Hence, the algorithm converges to an equilibrium.'' A model with hidden variables can have several global and local optimisers (see, e.g. https://arxiv.org/abs/1709.05276). The critical points of the EM algorithm can have a non trivial structure, as has been observed in the case of non negative rank matrix varieties (see, e.g., https://arxiv.org/pdf/1312.5634.pdf). 

OTHER

In page 3, ``S_\beta is e-flat and S_\alpha ... '', should this not be the other way around? (See also page 5 last paragraph of Section 2.) Please also indicate the precise location in the provided reference.  

All pages up to page 5 are introduction. Section 2.3. as presented is very vague and does not add much to the discussion. 

In page 7, please explain E ψ(θˆ )^2 −ψ(θ∗ )^2=0 

review_title: This paper uses an information geometric view on hierarchical models to discuss bias a variance in Boltzmann machines, presenting interesting conclusions, whereby some care seems to be needed in the derivations and discussion. 

final_score: rating: 7: Good paper, accept
review_content: Summary: The goal of this paper is to analyze the effectiveness and generalizability of deep learning. This authors present a theoretical analysis of bias-variance decomposition for hierarchical graphical models, specifically Boltzmann Machines (BM).  The analysis follows a geometric formulation of hierarchical probability distributions. The authors describe a general log-linear model and other variations of it such as the standard BM, arbitrary-order BM and Restricted BM to motivate their approach. 

The authors first define the bias-variance decomposition of KL divergence using Pythagorean theorem followed by applying Cramer-Rao bound and show that the variance decreases when adding more parameters in the model. 

Positives:
-The paper is clearly written and the analysis is helpful to show the effect of adding more parameters on the variance and bias in a general architecture (the Boltzmann Machines)
-The authors did a good job covering general probabilistic models and progression of models starting with the log-linear model.
-The authors provided an example to illustrate the theory, by showing that the variance decreases with the increase of model parameters.

Questions:
-How does this analysis apply to other deep learning architectures such as Convolutional Neural Networks?
-How does this analysis apply to other frameworks such as variational auto-encoders and generative adversarial networks?
review_title: Bias-Variance Decomposition for Boltzmann Machines Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary of the paper:
The paper derives a lower bound on the expected  squared KL-divergence between a true distribution and the sample based maximum likelihood estimate (MLE) of that distribution modelled by an Boltzmann machine (BM) based on methods from information geometry. This  KL-divergence is first split into the squared KL-divergence between the true distribution and MLE of that distribution,  and the expected squared KL-divergence between the MLE of the true distribution and the sample based MLE (in a similar spirit to splitting the excess error into approximation and estimation error in statistical learning theory). The letter is than lower bounded (leading to a lower bound on the overall KL-divergence) by a term  which does not necessarily increase if the number of model parameters is increased. 


Pros:
- Using insights from information geometry  opens up a very interesting and (to my knowledge) new approach for analysing the generalisation ability of ML models.
- I am not an expert on information geometry and I did not find the time to follow all the steps of the proof in detail, but the analysis seems to be correct.

Cons:
- The fact that the lower bound does not necessary increase with a growing number of parameters does not guarantee that the same holds true for the KL-divergence (in this sense an upper bound would be more informative). Therefore, it is not clear how much of insights the theoretical analysis gives for practitioners (it could be nice to analyse the tightness of the bound for toy models).
- Another drawback reading the practical impact is, that the theorem bounds the expected  squared KL-divergence between a true distribution and the sample based MLE, while training minimises the divergence between the empirical distribution and the model distribution ( i.e. the sample based MLE in the optimal case),  and the theorem does not show the dependency on the letter. 

I found some parts difficulty to understand and clarity could be improved  e.g. by
- explaining why minimising KL(\hat P, P_B) is equivalent to minimising the KL-divergence between the empirical distribution and the Gibbs distribution \Phi.
- explaining in which sense the formula on page 4 is equivalent to “the learning equation of Boltzmann machines”.
- explaining what is the MLE of the true distribution (I assume the closest distribution in the set of distributions that can be modelled by the BM).

Minor comments:
- page 1: and DBMs….(Hinton et al., 2006) : The paper describes deep belief networks (DBNs) not DBMs 
- \theta is used to describe the function in eq. (2) as well as the BM parameters in Section 2.2 
- page 5: “nodes H is” -> “nodes H are” 



REVISION:
Thanks to the reviewers for replying to my comments and making the changes. I think they improved the paper. On the other hand the other reviewers raised valid questions, that led to my decision to not change the overall rating of the paper.
review_title: The paper presents a interesting analysis revealing the usefulness of analysing the generation ability of ML models based on insights from information geometry. However,  a lower bound on the KL-divergence is less informative in practice. 

Meta-review: This paper presents a bias/variance decomposition for Boltzmann machines using the generalized Pythagorean Theorem from information geometry. The main conclusion is that counterintuitively, the variance may decrease as the model is made larger. There are probably some interesting ideas here, but there isn't a clear take-away message, and it's not clear how far this goes beyond previous work on estimation of exponential families (which is a well-studied topic).

Some of the reviewers caught mathematical errors in the original draft; the revised version fixed these, but did so partly by removing a substantial part of the paper about hidden variables. The analysis, then, is limited to fully observed Boltzmann machines, which have less practical interest to the field of deep learning.

Decision: Reject