Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a new multi-scale architecture for flow-based generative models. Unlike prior work on multi-scale flow architectures which use fixed dimension-splitting heuristics, the proposed approach learns which dimensions to process further. The features are chosen for further processing based on a heuristic motivated by each feature's contribution to the total likelihood. This contribution is given by the each features' contribution to the log-determinant term in the change of variables formula. The model is trained in a two-step process. First a flow model with no multi-scale architecture is trained. Then each feature's  importance is calculated based on its contribution to the log-determinant. Then these scores are used to rank the features for the second, multi-scale model which is retrained from scratch. The authors demonstrate the performance of their approach on density modeling on standard image datasets. They demonstrate an improvement over the standard real-nvp architecture.

Overall, I would recommend to reject this paper. While the proposed method is interesting, I do not feel that the authors  experiments demonstrated its performance relative to other additions on top of coupling-layer based flow models. For example, one could view the invertible 1x1 convolutions from GLOW as a similar way to "``learn" how the features should be ordered. The authors do not provide a comparison with more modern flows such as Glow, Residual Flows -- both of which perform considerably better than Real-NVP with the proposed architecture. 

While I think it is possible that Glow, Residual Flows and other recent flows could benefit from the proposed architecture, the results presented here do not convince me of this fact. If the authors presented a more comprehensive ablation study comparing their architecture to the various additions of other recent work on large-scale flow-based generative models, and their model performed favorably, then I could be convinced to raise my score. 

__________________
POST REBUTTAL
__________________

I appreciate the great efforts the authors have gone through to address my initial concerns. Most of these concerns had to do with the limited scope of the paper's initial experiments. The authors have expanded their experiments to incorporate a few more flow-based generative models and have demonstrated their proposed approach can improve their likelihoods across the board. I am somewhat disappointed that they did not attempt to improve upon the current SOTA on CIFAR10 (residual flow) despite the fact that code is available for this model. If the authors had used their method to improve upon the SOTA then their experiments would have been considerably more convincing. 

Despite that, I find these new results much more compelling and I would like to make that clear to the area chairs. Given the harsh discretization of the scoring system (1, 3, 6, 8) I cannot in good faith increase my score from a 3 to a 6, but I would like to make it very clear that these changes to the experiments do improve my view of the paper -- moving from a weak reject to neutral, though that is a not an official option. 

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper propose a heuristic algorithm for deciding which random variables to be Gaussianized early in flow-based generative models. The proposed algorithm involves first training a flow without multi-scale training, for example, 32*32*c  - 32*32*c - 32*32*c. Then, it computes the logdet term for each variable at each layer. It then spatially partition the first flow block by two halves of shape 16*16*2c based on max-pooling the logdet term. Then it recursively Gaussianize one half, and partition the other half as 8*8*4c, still using the pre-computed logdet tensors (Ld in the paper). After partitioning, they train a multi-scale model with the learned partition.

While I agree adaptive multi-scale architecture is a topic worth researching, and the paper does have some positive experimental results. I think the writing of the paper is very vague and the techniques are not sensible. 

Writing: the main algorithm is just depicted in the last paragraph of Page 4. There are not any equations or pseudocode on what exactly does the proposed algorithm do. Figure 1 and 2 are not detailed enough. For example, Figure 2 doesn't explain how to "Perform splitting based on log-det heuristic and spatial constraints". I can only guess what the algorithm is. I suggest the authors make the algorithm more clear, and avoid using large paragraphs of natural language to depict the algorithm. 

Technique: 
1. While I agree partitioning based on logdet term makes some sense, I think *recursively* partitioning without updating the logdet terms is problematic. If the flow only have one layer, the proposed algorithm makes sense. However, for a multi-layer flow model. After the first partitioning, the network changes. For example, for a two layer flow 32*32*c - 32*32*c - 32*32*c, the two layers both have 3*3*c*c filters. However, after partitioning the first output layer as two 16*16*2c, the filter of the second layer should have shape 3*3*2c*2c now. It is not clear how to translate the original, single-scale model into a multi-scale one. And the logdet tensor for the second flow layer doesn't mean anything now. 

2. For affine coupling layer, we can indeed compute the logdet term dimension wise. However, it is not clear how to do the computation for other types of flows. For example, invertible ResNets, which estimates the log-det term for each ResBlock with an unbiased estimator.

3. I am also not sure with the training algorithm after pretraining. Do we need to remember what pixel to pickup for each max-pooling operation? That has O(s*s*c*L) space complexity. Does the "gather" operation baesd on the memorized locations time consuming?

4. Training a single-scale model has higher time complexity than training a multi-scale model. Is this time complexity too high?

=================

Update: thank for the authors for their significant effort on revising this paper.

Writing is indeed much better. However there are still many typos (e.g. jabocian, algorithm). Algorithm 1 is better than the original plain-text version, it still doesn't look like even a pseudocode though. I suggest converting the bullets into actual code, e.g., (try to minimize the amount of natural language since it is vague)

For each layer l 
Ld1, Ld2 <- Maxpooling(sth), Minpooling(sth)
...
EndFor

I still don't think my concern 1 is addressed. Imagine a single-scale flow

y = AffineCoupling(x)
z = AffineCoupling(y)
h = AffineCoupling(z)

vs a multi-scale flow

y = AffineCoupling(x)
y1, y2 = LCMA-Split(y)
z = AffineCoupling(y1)
z1, z2 = LCMA-Split(z)
h = AffineCoupling(z1)

I don't think |dh/dz1| of model 2 is a submatrix of |dh/dz| of model 1. Because z1 of model 2 is not a part of z of model 1 in the first place. In model 2, z1 is computed with only y1, while in model 1, z1 is computed with the full y.

Concern 3:
The authors partially addressed my question. However, the proposed algorithm is still more expensive than a fixed multi-scale architecture, right? A fixed multi-scale architecture such as RealNVP (next scale block is s/2 * s/2 * 2c) is cheaper than a single-scale architecture (next scale block is s/2 * s/2 * 4c). I guess the time complexity of the proposed approach is the letter one instead of the first one. So the improved likelihood still comes with time complexity cost (comparing with a fixed multi-scale architecture).

To summarize, I can increase my score to 3 as a positive feedback for the author's effort. But I really think this paper still has a long way to go to be complete.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary: 
The paper investigates a strategy for factoring out dimensions in multi-scale generative flow models. The strategy relies on a heuristic that employs the availability of per-dimension log-determinant terms (e.g. as affine couplings) to decide which half of the dimensions to factor out. Quantitative experimental results support the proposed strategy’s efficacy.

Recommendation: Weak Reject
The paper misses several important comparisons and baselines - both in terms of placing the proposed method in a broader context (e.g. how could the method be used in multiscale i-ResNet), as well as providing experimental comparisons that would definitively demonstrate the method’s contribution (e.g. is it the heuristic dimension splitting or the unfactored pre-training that provides the benefit.

The applicability of the proposed method appears to be overstated. The paper suggests that the method is applicable to any multi-scale flow architecture. However, it depends on the availability of per-dimension log-determinants. As the paper states, these values are readily available for methods that rely on affine couplings (e.g. RealNVP, Glow), but not for methods such as i-ResNets or FFJORD, that also employ multi-scale architectures to factor out variables.


Major comments:
1. Section 2.1 introduces flows as a mapping from data x to the latent code z, and gives the log-likelihood formula (2) that can be optimized during model training. This presentation is at odds with Section 3, where the authors talk about sampling a latent variable that fixes the “log-latent density” and only leaves the log-determinant term to be optimized (e.g. “[...] the log-latent-density term depends on the choice of the latent distribution and if fixed given the sampled latent variable. [...] once the variable is sampled, maximizing the log-det term results in maximized likelihood.” The two presentations are at odds with each other and should be reconciled, ideally by altering Section 3 since it does not reflect the way flow models are trained.
2. The authors made an implicit assumption that the log-det term is decomposable into per-dimension component. As mentioned earlier in this review, generally speaking this is not true and methods such as i-ResNets or FFJORD do not (directly?) permit such decomposition, while they do employ multi-scale architectures akin to that of RealNVP. This assumption manifests itself in several places in the paper, which are listed below. Either the authors could make this assumption and the limitations imposed by it very clear, or they could provide additional information on how their method extends to methods that do not provide per-dimension log-det contributions.
a) In Section 3 “The total log-det term is nothing by the sum of log-det terms for contributed by each dimension”. This is obvious (and possibly holds) only for affine couplings.
b) In Section 3 “Nevertheless, such an analogy can be extended for other flow models which involve a multiscale architecture [...]”. As mentioned, I am not convinced this is true.
c) In Section 4 “Since our method focuses individually on the dimensions using a heuristic which is always available in flow model training, it can prove to be have more versatility in being compatible with generic multi-scale architecture”. Same as above - in fact I believe that the opposite may be true.
d) Section 4 does not mention methods that employ multi-scale architectures whil not allowing for discerning per-dimension contributions of the log-det. Mentioning these methods is important for accurately positioning the authors contribution relative to the existing literature.
e) The method is only applied to the RealNVP architecture. The claim of it being generally applicable to multi-scale architectures would be much stronger if the comparison would include other multi-scale architectures. Here, I strongly recommend i-ResNets or a similar architecture based on log-det approximation to the entire coupling layer.
3. Unless I missed it, the authors do not mention how they aggregate the log-det values. They are available per datapoint. Are they aggregated over the entire dataset?
4. In Section 4 the authors imply that the contribution of a dimension towards the total likelihood is dominated by the log determinant (“[...] as the contribution by the variable (dimension) towards the total log-det (~ total log-likelihood)”). This is not obvious, and I am not sure it is always true - it probably depends on the choice of the base distribution p(z).
5. The authors show experimentally that the use of their variable factorisation heuristic improves log-likelihoods over the baseline RealNVP model; they further perform ablation studies to confirm that preferential factorisation of low log-det dimensions provides best log-likelihoods and qualitative results on the CelebA dataset. However, several points remain unclear.
a) What are the relative contributions of pre-training without dimension factorisation and the proposed factorisation heuristic? Do the experiments shown in Table 2 all employ the pre-training scheme? If so, this should be clearly stated. If not, this set of experiments should be performed.
b) The networks used in the experiments use between m=1 and 4 factorisation layers. So in total, there are 2^m possible decisions of which half of the variables at those layers to factorise. These numbers are not so large as to prevent us from training all possible combinations of the networks. How does the proposed heuristic compare to just training all (or a few sampled) factorisations decided a priori? This comparison should also investigate the effect of using pre-training.
6. In the qualitative comparison (Section 5.2) the authors argue two points (i) RealNVP with the proposed method generates finer details on CelebA; and (ii) provides a reasonable latent space that can be interpolated in.
a) With regards to (i), I personally find it difficult to argue about fine details in what are effectively low-resolution images. The differences (which are already hard to see and easy to imagine) could be present due to minute changes in the training code/procedure/random seed/etc. I would be more confident about these differences if the authors could confirm that the methods were trained using the same code base, random seeds, dataset shuffle and framework versions.
b) How do the qualitative interpolation results compare to RealNVP without the proposed factorisation heuristic?

Minor comments:
- In Section 2.1 “Let x be a high-dimensional random vector with unknown true distribution x ~ p(x) [...]”. Here x appears to refer to the data or a sample from the data distribution, and not just a random vector. This could be made clearer.
- In Section 2.1 the “jacobian” should be written with a capital “J”
- In Section 2.1 and elsewhere, when denoting outputs of intermediate flows f_i as y_i, it is mentioned that y_0 = x, but not that y_k = z. If the latter is true, mentioning this would improve clarity.
- Section 3 “In a multi-scale architecture, it is apparent that the variables getting exposed to more layers of flow will be more expressive in nature [...]”. I do not find this to be an obvious fact. Perhaps the authors could provide an intuition for why this is true or demonstrate this on a toy example? My confusion may stem from the fact that I am unsure what it means for a variable (i.e. a dimension we factor out) to be more expressive.
- Some phrasing is a little unusual (e.g. “less (more) log-det tems” instead of “smaller (larger) log-det term”; “to be have”).
- In Section 5.1 “code length” is introduced without context. The connection between code length, log-likelihood and compression can be made more clear.

------------------------------------------------
Post rebuttal update
------------------------------------------------

I’d like to thank the authors for additional elaboration on their baseline experiments that consider their heuristic-based scheme for factoring out dimensions (the low-low-low scheme) and the “anti-heuristic” (which considers the high-high-high scheme). I agree with the authors that both schemes would have been within the 2^m possibilities I encourage them to consider, however I do not fully agree with some of the conclusions the authors draw for these experiments.

First, given that the method proposed by the authors is a heuristic, it is not obvious that the high-high-high and the low-low-low schemes indeed correspond the worst and best possible of the 2^m schemes. Running the 2^m experiments could demonstrate this empirically and strengthen the manuscript.

Second, if we speak about the high-high-high and low-low-low factorizations, then we’ve already gone through the computationally expensive effort of pre-training a model that does not factor out any variables. It is precisely because of this that I’d like to see a baseline that doesn’t do any pre-training, but simply enumerates the 2^m schemes and selects the one with the best validation score. It is not obvious that this approach would computationally more expensive than the pre-training step -- another point I believe is important to cover.

To summarize, I do not believe that the baselines currently present in the paper are a substitute for the ones requested.
Including new results requires a review of those new results, so I would not be in favour of adding new results at the camera ready stage. I therefore retain my score.

review_title: Official Blind Review #2

Meta-review: The authors propose a multi-scale architecture for generative flows that can learn which dimensions to pass through more flow layers based on a heuristic that judges the contribution to the likelihood. The authors compare the technique to some other flow based approaches. The reviewers asked for more experiments, which the authors delivered. However, the reviewers noted that a comparison to the SOTA for CIFAR in this setting was missing. Several reviewers raised their scores, but none were willing to argue for acceptance. 
Decision: Reject