Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a multi-scale method to design invertible network and apply it to bayesian inverse problems.

The method itself is quite concise and beautiful in its simplicity (a nice case of “obvious once you see it”), and looks rather simple to implement. The results are also rather promising.

With that said, there are some steps that are lacking. Notably the results leave some things open (e.g. the SotA-ness is highly debatable given more recent methods such as routing transformers) and for the Bayesian inverse problems it’s hard to judge given that the only baselines are “in house” rather than externally defined. I also find the theoretical exposition to be a bit hard to follow, especially given the generous use of appendices.

I have some further comments:

* The existence of a prior conditioning operator in the general case seems somewhat debatable (consider odd cases like singular distributions, distributions with weird topologies and discrete distributions), I’m quite sure you need at least some further conditions. I’d also be fine with a “if it exists” style statement.
* The optimisation described in appendix D is quite critical to the method as is (especially since using Jeffreys divergence is even mentioned in the abstract as a main point) and should be in the body of the article.
* The statement “the subsurface flow simulation at the scale of meters approximates well the simulation at the scale of centimeters” is more or less repeated twice verbatim. It’s also not the most accessible example.
* Spelling errors like dimebsion, groundtruth. Overall the language could use some polishing throughout the article.
* All figures should be in vector format
* The statement “ q(x(s1, s2)) = q(x(s1, 1 − s2))” does not afaik imply more than one mode, consider e.g. a mode centered at s2=0.5.
* Showing the “worst of 3” seems rather dishonest. Why was this done?
* Table 2 is quite lacking, several SotA methods are not in there (which makes this method not SotA).

review_title: Multiscale Invertible Generative Networks for High-Dimensional Bayesian Inference

final_score: rating: 6: Marginally above acceptance threshold
review_content: # Paper Summary
This paper presents a model and a corresponding training approach for multi-scale invertible models. The presented model is defined on multiple scales with information on finer scales being conditioned on coarser scales. Data generation is hence done sequentially from a coarser to finer scale. The authors argue that this multi-scale sampling helps in addressing the curse of dimensionality problem by allowing to sample from high density regions more efficiently.

As a core part of the model, the authors present a linear layer that combines information from a coarse scale and a noise vector to predict the data on a finer scale. This prior conditioning layer, as the authors call it, exploits the fact that the downsampling operation from a finer scale to a coarser scale is a linear operation. Under the assumption that the fine scale data follows a multivariate normal distribution, the downsampling operation can be analytically inverted.

Training is done by minimizing Jeffreys divergence, i.e. the loss is calculated bi-directinally in the forward and the backward direction of the model, akin to Ardizzone et al. (2019). The authors argue that Jeffreys divergence is less prone to the mode dropping that can happen when training with KL divergence. The authors justify this argument by citing Nielsen & Nock (2009), by doing ablation studies on toy data, and by giving a toy example in Appendix C.

Due to the multi-scale structure and the fact that the coarse scale data can be generated from a fixed corruption process (the downsampling model), it is possible to train this model by decoupling the different scales. The authors suggest to train the model from a coarser to finer scale in a multi-stage process. The authors claim the superiority of this approach to regular end-to-end training.

In an extensive experimental section, the authors evaluate their approach on two 'Bayesian Inverse Problems', as well as on generative modeling of the Celeb A 64 data set. In further experiments, the authors perform ablation studies on a synthetic data set in order to justify their presented approach. In all experiments, the presented approach yields the highest performance.

# Assessment Summary

The presented model and training approach exploit the hierarchical nature of data in a smart way. This in turn results in good performance on a number of data sets. The experimental section is rigorous and convincing. These aspects of the work should put me in favor of acceptance if it weren't for the lack of clarity in places (see below). In its current state the paper does not explain the method well enough in order to get a full understanding of the approach. I therefore tend to reject the paper at this stage. With appropriate improvements in the writing I would be happy to improve my score.

# Positives
- A multi-scale approach is a reasonable choice in many real world applications. The presented model exploits our world knowledge by utilizing a fixed downsampling operator to generate views of the data on different scales. This kind of inductive bias is simple to interpret and it appears to be a promising aid in model training.
- The experimental section is extensive. The authors evaulate their approach on multiple problems, synthetic and real world. Especially the ablation study should be pointed out.
- The ablation study aims to confirm the authors claims that the presented approach results in better mode capture. To do that the authors compare different model architectures, different training losses, and the multi-stage training to end-to-end training. Their results suggest that the multi-scale model trained in multiple stages using Jeffreys divergence indeed performs best.

# Negatives
- One of the biggest issues in this paper is the lack of clarity in places, which in turn makes the work hard to digest. This is often owed either to a poor choice of notation or to abbreviated explanations. Especially the methods section should be pointed out in this regard. I will give a few examples below.
	- In equation (1) the authors surrogate $q(x|y)$ with $q(x)$. This choice makes it difficult to follow equations (5) and (6) as it is not clear anymore where we are dealing with marginal or conditional probabilities.
	- The use of $\rho(x)$ and $L(y|x)$ does not follow the ICLR style guide for probabilities.
	- Section 2.1 is not clear. Here, the choice of abbreviations in (5) and (6) hinders an understanding. It is not made clear which role variable y plays in the network. It should be made more clear by the authors that the forward process of the multiscale model is sampling from the posterior distribution.
	- Figure 1 only shows the sampling process, but does not help to understand the training. $\tilde{x}_l$ is represented with smaller images than $x_l$ although both have the same dimensionality.
	- Section 2.3 discusses biefly the use of Jeffreys divergence with importance sampling for training the model. The authors do not make clear what $p(x)$ and $q(x)$ stand for in this context. As such it is not possible to make out how to train the model with the suggested loss.
- The prior conditioning layer (upsampling layer) exploits the fact that the downsampling operation is linear and the prior over fine-scale data is assumed to be a multivariate normal distribution. As such, the upsampling can be implemented with a linear function of the coarse data and a noise vector. It is well known that this kind of inversion, although analytically correct, is computationally infeasible in high dimensional data. The authors do not address this issue in their text.
- With respect to the multi-scale structure of the model, the paper does not position itself well in the literature. Invertible models such as in Dinh et al. (2016) already use a multi-scale structure. The authors should point out the difference of their approach more explicitly. Using explicit downsampling as an inductive bias was used successfully before for training GANs in Denton et al. (2015). This work might be a good starting point to discover other related multi-scale approaches.


# Recommendations
- Figure 1 could be improved to visualise the overall approach better. It would be worth visualising the sampling process for estimating Jeffreys divergence. Further, I would suggest to use an example from the Celeb A data set in order to get a better understanding of the intermediate scales. In addition, the downsampling operation $\mathcal{A}$ could be emphasided in this figure.
- Notation should be improved and made more consistent. Is there a way to reduce the amount of variables used? Is it possible to use consistent notation to denote probabilities?
- Section 2.1 should be clarified. Is there a simpler way to explain this? For example by exploiting the Markov property of the model.

# Questions
- In Lemma 2.1 it is assumed that the fine-scale signal follows a multivariate normal distribution. Is this a reasonable assumption? Does this not contradict the idea of multiple modes in the data?
- Figure 6: Why do the interemediate coarser scale samples not look like downsampled versions of the high resolution image?
- What happens if the models on each scale are trained completely decoupled? Would it be fair to call the model a stacked super-resolution model?

# References

Ardizzone, L., Kruse, J., Wirkert, S., Rahner, D., Pellegrini, E. W., Klessen, R. S., & Köthe, U. (2018). Analyzing inverse problems with invertible neural networks.
 
Denton, E. L., Chintala, S., & Fergus, R. (2015). Deep generative image models using a￼ laplacian pyramid of adversarial networks. 

Dinh, L., Sohl-Dickstein, J., & Bengio, S. (2016). Density estimation using real nvp.

Nielsen, F., & Nock, R. (2009). Sided and symmetrized Bregman centroids.

---------------------------------------------------------

# Review Update
Based on the response and update from the authors I will improve my score from '5: Marginally below acceptance threshold' to '6: Marginally above acceptance threshold'. I generally believe that the presented approach merits publication. The methodology is sound and the experiments are convincing. I appreciate the fact that the authors made an effort to improve the writing.



review_title: The lack of clarity is in the way of making this a good paper that I am willing to accept. (Update: Marginal accept)

final_score: rating: 5: Marginally below acceptance threshold
review_content: While the use of an hierarchical flow-based model to obtain samples of finer scales seems interesting, I have to admit that I did not understand much of the setup the authors propose. The methodology section is too dense and many of their claims are not really motivated at all. 

I do not see this paper accepted the way it is, but it might be true that a person more expert in flow-based models and inference than me thinks completely the opposite.
review_title: I believe the paper has its merits, but frankly I could not understand much

Meta-review: The paper considers sample generation in high-dimensional bayesian inference and proposes a multi scale procedure that performs coarse-to-fine multi-stage training and enables interpretability of intermediate activations at coarse scales.  The method is simple, elegant and addresses a very important bottleneck of high-dimensional bayesian inference. The clarity of the paper has been greatly improved based on the reviewers suggestions. 

However some concerns remain regarding the evaluation that are needed to clearly demonstrate the value of the approach. In particular, it would be important to assess the impact of the number of levels, and how quickly the dimensions grow from one level to the next. The number of forward simulations does not provide a sufficient picture of the computational cost of the approaches. It would be also important to provide wall-clock time.  Figure 2a should also provide the best of 3  independent experiments,  or better, more experiments should be run, and curves with shaded areas should be provided so one could visualize variability w.r.t runs. 
Decision: Reject