Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper analyzes the approximation of the log-ratio (called normalized evidence in the paper) (the log-probability under the model minus the log-probability under the true model). Since the model is singular, the Laplacian approximation of the normalized evidence needs a new method. The mean-field distribution and a resolution map allow us to estimate two important numbers (lambda and m) in the approximation. 

The paper has made several contributions to the study of this problem in several aspects. First, the paper improves previous results by relaxing the assumption. Second, the paper actually uses a normalizing flow to actually learn the transformation. 



main_review: This is the first time I hear about the theory. I think the topic is very interesting. 

However, the paper has several issues. First, I feel that the theoretical improvements over previous methods are not significant. Most theoretical results are just minor improvements such as making assumptions less restricted. The main result in 5.1 is still similar to [Bhattacharya et al. 2020], which already has the conclusion that a mean-field distribution allows the estimation of lambda and m. 

Section 3 seems to be purely from previous work. 

Second, the experiment is done on only two toy problems. The dataset has 5000 instances while the network has two hidden layers. Do you actually check whether the model is singular or not? I hope to see experiments based on realistic datasets. Can we also include other variational inference methods as the baseline when estimating the normalized evidence? 

Third, I don't know how significant it is to use normalizing flow to learn the resolution map? Is the analysis still valid for the complex flow model? In practice, the method has competitors from other variational inference methods. How practice is this method in estimating the normalized evidence. 

Fourth, the writing of the paper can be greatly improved. A lot of my understanding depends on previous work by [Bhattacharya et al. 2020]. Here is a list of detailed issues that block my understanding. 
1. I think the paper should make clear the Laplacian approximation from the beginning and also the significance of estimating lambda and m. 
2. Equation 6, "in the asymptotic expansion of ..." what expansion? The paper may need to explain the expansion a little. 
3. Should section 3 go to the introduction since it is not the contribution of this paper? 

summary_of_the_review: The paper studies an interesting topic. However, there are several issues such as minor improvements over previous methods, practical value, and also writing problems. Overall, this is an interesting paper, but the quality is not very high.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Inspired by asymptotic results from singular learning theory, the authors of the paper propose using a generalised gamma mean-field distribution with a normalising flow (that targets the "desingularization map") to perform variational inference. The authors additional build on prior work to derive a tighter bound on the log normalised evidence for variational inference using this approximate posterior combined with the "correct" desingularization map. 

Note: while I am familiar with the Bayesian Neural Network literature, I am not familiar with singular learning theory. This is reflected in my confidence score. As such, I was unable to verify the theory in the paper, and assess the significance of the theoretic results included in the paper.
main_review: Points marked with (*) are especially important, and should be addressed if the authors would like me to reconsider my score. 

Strengths:
1) I enjoyed the discussing relating the ELBO score with the quality of the variational posterior predictive, and how good ELBO scores do not imply good posterior predictive distributions. Thank you for including this.

Weaknesses:
1) *As I understand, Eq. (1) shows for non-singular models, the posterior distribution can be expressed in a specific form i.e., a mixture of standard form distributions. However, this result only applies "when n is large". The discussion of what it means for "n to be large" is extremely limited. I expect this to depend on the network in question. Further, it is entirely unclear to me whether practical Bayesian Neural Networks are in a "large n" regime. This issue, at the very least, should be discussed within the paper. 

2) *The technical contribution of the paper, at least in terms of a practical inference method, seems highly limited in terms of novelty. The paper proposes using a normalising flow approximate posterior with a Gamma source. However, for expressive flows, the choice of source distribution can always be absorbed into the flow. And flows have regularly been used for posterior inference. The contribution here is thus highly limited.  
3) The readability and presentation of the paper could be improved. For example, equations could be named when referred to within the text. Intuition could be added on why neural networks are singular models. The figure captions are hard to understand without referring back to the text extensively. Experiments could be named, much more of the results could be in the main paper, intuition behind theorem conditions could be provided. I think this is especially valuable as singular learning theory is not yet well known within the machine learning community. 
4) *The experimental evaluation of the paper is highly limited. I understand that simple networks are required to apply singular learning theory,  but the proposed method could be applied to much larger networks. The experiments within the related work, which is almost exclusively work within the Bayesian Deep Learning community, typically uses experiments with a much larger scale. Further, not all runs reached convergence, which undermines the claims made in the paper. 
5) *The technical results of the paper assume that b \propto 1 (page 6), but the authors write that this condition is not expected to hold in reality, even for very simple experiments. This seems to be a major concern. This should, at the least, be mentioned when the theoretical results is discussed within the paper. The authors go on to write "we do not believe this assumption is critical", which as far as I can tell, is conjecture, and not convincingly discussed within the paper. 
6) Although the authors use a Gamma source distribution, in practice, they approximate this with a Gaussian distribution. This seems to further undermine the technical novelty of the paper. 
7) *The flow that is used within the paper is very shallow—it only has two coupling layers. It is well known that many layers of affine couplings to have expressive distributions. The authors claim "though learning the variational parameters, k, in the source distribution goes against conventional wisdom in normalizing flows, our experiments suggest some performance may be gained by learning the optimal untruncated generalized gamma source distribution at the same time as learning the resolution map". This claim is not well supported; the experimental results use particularly non-expressive flows, and as such, this claim only stands in this context. And within this context, I expect that learning parameters of the source distribution is increasing the expressivity of the flow. The conventional wisdom holds primarily for expressive flow distributions.  
8) The authors write "for large n, the posterior is not Gaussian, but a mixture of standard forms". However, note that this is true only *post flow transformation. i.e., the distribution of the weights when transformed into a new coordinate system follows this form. However, in general, flows can express arbitrary distributions. This result seems to be somewhat oversold. 

Clarifications:
1) As far as I can tell, the theory assumes that the space of parameters is a compact set. However, in BNNs, we typically place prior support over the entire reals. This seems to be a theory-practice mismatch. Is this problematic?
2) The asymptotic expansion of the normalised model evidence holds only for large values of n, right? I'm concerned about how this affects the validity of the theory. 
3) The authors write "we are aided by the fact that a resolution map can attain the optimal value of Psi_K(q, g)". How/where is this demonstrated? Please clarify.  
4) Singular learning theory provides this resolution map, which in practice is learnt using a normalising flow. However, the flow is actually learning by maximising the ELBO objective, which is a lower bound on the likelihood. I don't understand why **in the context of singular learning theory** which maximising the likelihood (approximately) is a sensible target for the flow. I think it is a sensible target, when considered in the context of variational inference.
summary_of_the_review: I have a number of concerns about the paper. In short, (i) limited technical novelty (ii) the theory does not seem to apply in practice (iii) the experimental evaluation is limited, and uses only small scale experiments (iv) room for improvement in terms of presentation e.g., many of the results are included only in the appendix. 

As a result, I do not believe that this submission, in its current form, is of interest to the ICLR community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper builds on the recent theoretic results of [Bhattacharya et al. (2020)],
which shows that a mean-field variational approximation with carefully chosen approximation family
leads to an ELBO $\Psi$ which is sharp up to a constant C(d) which only depends on the dimensionality d of the parameter space.
For the proof, [Bhattacharya et al. (2020)] assumes that the approximation family is a Gamma distribution truncated to [0,1]. 
Therefore the authors of this paper conjecture that the Gamma distribution truncated to [0,1]
is a good choice for the source distribution of a normalizing flow.
Their experimental results suggest that this choice leads a higher ELBO than using as source distribution a normal distribution.
main_review: Strength:
- The authors make an effort to exploit recent progress in singular model theory
for developing practical variational methods
- The summary of the related work is well presented, and appears to be comprehensive.
- The authors are clear about the limitations of their approach.

Weakness:
- Theorem 5.1. does not seem to tell more than Theorem 3.1 in [Bhattacharya et al. (2020)].
Am I missing here something?
- The main contribution seems to be the effort to exploit the theoretic idea of a mean-field variational approximation with truncated Gamma distributions. However, the relationship between the theoretic idea and their experiments appears ad-hoc since in their experiments they do not use a mean-field approximation, but rather use a normalizing flow with a Gaussian source distribution that 
tries to approximate an untruncated Gamma distribution. Why is it an untruncated Gamma and not a truncated Gamma distribution?
What is the relationship between the mean-field approximation and the normalizing flow?
- Since the authors use a Gaussian approximation for the Gamma distribution, it would be interesting if the authors could show whether and when the approximation is justified in their setting. (It appears to be correct, since n is rather large)
- The presentation of the experimental results needs improvement.
The experimental results in Table 2ff are essential, but can only be found in the appendix.
They should be summarized in the main paper.
- Actually it is not clear to me why the Theorem 3.1 in [Bhattacharya et al. (2020)] suggests that 
the ELBO found with a gamma mean-field approximation is better than a normal mean-field approximation.
summary_of_the_review: The idea of trying to use recent singular model theory to improve variational inference is interesting.
However, this work, in its current form, leaves too many open questions about the relation between theory and their experiments.
Furthermore, it is unclear what the take-home message is.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Working with Bayesian neural networks, this paper proposed a variational algorithm to approximate posterior distribution of the network weights. To overcome model singularities, the authors used the idea of normalizing flow by transforming the weights through an affine coupling network, and subsequently worked on the desingularized parameter space. In addition, they derived an asymptotic expression for the ELBO, and compared the Gaussian and generalized gamma approximating families in the experiments.
main_review: The writing is generally clear. Although there are innovations, such as using affine coupling network to learn the desingularization map, but I am not quite sure about the contribution of this paper. Equation (8) is misleading because it is not exact equality but should be $\asymp$, i.e., $\sup_{q\in\mathcal{Q}_{(0,1]}}\Psi_K(q,g)$ is bounded above and below by the right hand side up to constant multiples (see (15)). Therefore the lower bound of (8) is similar to the lower bound obtained by Bhattacharya et al. (2020) in (7). To improve upon their result, you need to show that $-\log{C(d)}$ is smaller than their constant.

I don't think recovering the leading order term of $\log{\bar{Z}_K(n)}$ guarantees that the mean field generalized gamma family is the optimal approximating family (as suggested in Pages 20-21 in Bhattacharya et al., 2020 and implicitly in this paper). There are many examples in the literature where dependent approximating families give clear improvement with respect to mean field (see "Variational Inference with Normalizing Flows" for example).
 
Here are some other questions:
1. Why do you work with $\Psi_K(q,g)$ and $\bar{Z}_K(n)$ in (4) and (5)? The results would be more significant if their stochastic counterparts $\Psi(q,g)$ and $\bar{Z}(n)$, which are the quantities of main interest, are considered instead.

2. Why did you choose the generalized gamma in constructing the mean field approximation? It seems to me that other distributions with tails heavier than Gaussian, such as the $t$-or Cauchy distributions, would also work. 

#########################################################################################################
TYPOS:
1. In Section 2 the first display, there should be a $dw$ in $\bar{Z}(n)$
2. In Page 8 the last two paragraphs, write that Tables 1-4 are in Appendices D and E

summary_of_the_review: I have doubts about the contribution of this paper and its significance to the literature.
review_title: null

Meta-review: The paper proposes a variational inference based on singular learning theory (SLT), where the resolution of singularity is learned by normalizing flow so that the latent distribution is factorized.

Pros:
- A unique idea to use SLT for variational inference.

Cons (only serious concerns):
- Goal is unclear.  The authors say that they propose variational inference based on SLT.  But apparently, they propose it not as an alternative to the state-of-the-art variational inference for neural networks (if so the experiments shown are far from the acceptable level).  The authors must clearly say for what purpose they propose a new method.  I would guess the proposed method is for analyzing singular models to compute their RLCT.  In that case, the authors should compare with existing methods for evaluating RLCT, e.g., MCMC based methods:

K. Nagata and S. Watanabe, "Exchange Monte Carlo Sampling From Bayesian Posterior for Singular Learning Machines," in IEEE Transactions on Neural Networks, vol. 19, no. 7, pp. 1253-1266, July 2008, doi: 10.1109/TNN.2008.2000202.

and discuss pros and cons of the proposed method.  For DNN, you should use the state-of-the-art MCMC sampling methods like

Wenzel, F., Roth, K., Veeling, B., Swiatkowski, J., Tran, L., Mandt, S., Snoek, J., Salimans, T., Jenatton, R. &amp; Nowozin, S.. (2020). How Good is the Bayes Posterior in Deep Neural Networks Really?. <i>Proceedings of the 37th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 119:10248-10259 Available from https://proceedings.mlr.press/v119/wenzel20a.html.

as a baseline.  Approximating the posterior with normalizing flow can be another baseline.

- Large n issue.  SLT can be seen as a generalization of the asymptotic learning theory for the regular model, where the model complexity is represented by the parameter dimension d, and "asymptotic" means n >> d.  Watanabe revealed that the model complexity cannot be represented by d in singular models, and therefore the definition of "asymptotic" is not as clear as the regular case.  But it is known that typical neural networks are overparameterized and can achieve zero training error.  I have seen no work arguing that SLT holds in this regime.  If the authors insist that their method is applicable to deep neural networks, they should cite references where it would be proved that SLT holds in the overparameterized regime or prove it by themselves.

There are many more concerns including those pointed out by reviewers, and the paper is not ready for publication.
Decision: Reject