Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work provides the novel approach for searching flow architectures. Compared to the standard approaches used to find the best deep architecture in standard network networks this problem is more challenging due to the need of invertible transform and requirement for determinant of the Jacobian to be easy to calculate. To solve the problem the authors propose to apply the weighting for candidate transformations. In order to enforce invertible properties of the model the authors suggest to use the mixing distribution approach instead of mixing the base transformations. They formulate the problem of learning best weights and show that optimal solution for soft weights is not optimal for binarised versions. Therefore they propose to optimize the upper bound of the proposed loss function instead. Further they show how to deal with the problem with larger number of layers. Some experiments are also performed to show the quality of the approach with respect to the baseline that is expert-based selection. 
main_review: Strengths

The problem considered in this work is novel and important for the community that works with flows. The problem is significantly more challenging than standard architecture search. The proposed solution is well-motivated. The theoretical claims seems to be correct. The flow of the paper is easy to follow and each step of the proposed approach is justified. The novelty and contribution of the paper is high in my opinion. The idea of using mixing the probabilities and the application of upper bound instead of direct optimisation is non-trivial.
 

Weaknesses 

The empirical evaluation of the proposed method should be more extensive. The selection manual flow baseline is a bit tricky to be. How far is the manual approach from the optimal combination of transformations on validation set? It would be also beneficial to create the baseline where the transformation are selected randomly. Such a baseline would deliver the information what is the gain on NLL with respect to the random approach. I would suggest also to take under consideration evolution approach that optimize the binary vectors that stays behind the selection process as reference method. 

My second concern is about the generalisation of the proposed approach to other types of flows. The approach seems to be generic and scale to any possible transformations, but the experimental evaluation is mainly focused on autoregressive flows. Is it possible to adopt that approach to various types of layers that represent the dynamics in CNF? It would be also beneficial to see what is the quality of that approach for various CNF layers in experimental part. 

The third concern is about the limitation of the approach due the fact that complexity grows exponentially and some decomposition methods are essential to apply architecture search effectively. For this part it would be interesting how much we loose during the decomposition process. 
summary_of_the_review: Concluding, the problem consider by the authors is novel and worth investigating, the approach seems to be adequate, the evaluation is limited but I think the paper is worth accepting.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed to adapt a differentiable architecture search formulation (Liu et al., 2019) based on learned weighting of an ensemble of modules to automated search for Normalizing Flow architectures. The authors made several adaptations to the original approach for the normalizing flow problem, due to the invertibility constraints that prevent direct linear summation of different transform operations. Furthermore, the authors proposed to optimize the full network using an approximated upper bound of the KL divergence, instead of directly optimization. The authors proposed two methods to decompose the optimization problem: grow method, which is more straightforward and greedy, and block method, that alternatively adjusts each block. The authors experimentally compared their proposed method with manually specified architectures across various datasets, including POWER, GAS, HERMASS, MINIBOONE and BSDS300. The results seem mixed, as the searched model outperforms the manual model in some contexts but not others.
main_review: Pros:

Overall the paper is well written and easy to follow. The main idea is to adapt the differentiable formulation for the NAS towards normalizing flow architecture optimization, and the authors made interesting theoretical contributions for reformulating the mixture-of-experts setup towards normalizing flow models, as well as for proposing novel optimization strategies. Overall the proposed method is sound and coherent.

Cons:

- The experimental evaluation seems relatively weak. Though the optimized architectures seem to consistently have lower train, forward and inverse costs, the test performance is mixed, and in many scenarios worse-performing than the manually design architecture. 
- The search space that the authors experimented with is extremely limited, limited to only planar flows and radial flow, with the weights between the two transformations being the only architectural hyperparameter learned, not accounting for other hyperparameters such as number of stacked flows, the network complexity (number of feature layers, etc) for each network, which has little generalizability to more modern and useful architectures (e.g., RealNVP, GLOW, FFJORD).
- On the novelty side, though the authors made problem-specific adaptations for the differentiable architecture search algorithm for normalizing flows, the main idea is very much adapted from Liu et al. 2019 and somewhat marginally novel.

summary_of_the_review: As a summary, the overall novelty of the proposed approach is rather incremental, the search space is too restrictive and limited to be practically useful for designing practical NF architectures (only searching for a binary choice between two architectures, not accounting for other more complex bijective transformations), and the experimental results are insufficient to illustrate the usefulness of the proposed approach for improving NF architectures.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a DARTS-like method for searching automated normalization flow models. Instead of directly using the output ensembles, which leads to infeasible flow models, this work proposed distribution mixture to guarantee that the supernet is always a valid flow model. The upper bound of the loss function is optimized jointly with resource constraints. Experiments on small-to-medium scale datasets valid the effectiveness of the proposed method.
main_review: [Strength]

1. This is the first work using NAS to optimize flow models

2. Although the proposed method is based on DARTs, it requires some efforts to make it work on flow models, such as distribution mixture.

[Weakness]

1. The paper is a bit difficult to follow if the reader is not very familiar with flow model. Especially, the following things need to be further clarified in great detail:

  a) How did you get Eq. (7) from Eq. (6)? In Eq. (7), $W_k=\prod_{i}w_i$, so the right hand side does not contain $k$ at all?

  b) The proof of upper bound optimization should be further clarified. It is hard to follow (at least to me).

2. The upper bound argument seems questionable. While I could understand that we need to binarize the weight $\alpha$ in order to get a simple and valid flow model, I still cannot understand why optimizing the Jensen's upper bound is a good (or better) idea. At least from the experiments, it seems that binarization  is still a necessary step.

3. The proof of Proposition 1 seems questionable. The second $\geq$ is not obvious. 

4. The experiments focus on density estimation problems. It is somewhat insufficient. I would expect more real-world applications and comparing to strong baseline methods.

5. Table 1 need improvements. For example, it is better to explicitly align with one cost and then compare the test score. For now it is difficult to compare results since they have different costs.

6. The proposed method almost directly follows DARTS. Although the distribution mixture is novel, it is still more or less an incremental improvement.

summary_of_the_review: To my best knowledge, this work is the first work aiming to design more efficient flow models via a DARTs-like NAS method. However, there are still many issues to be addressed. This work can be made significantly stronger if the aforementioned issues are well addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors present an automated normalizing flow(NF) architecture search method. The method employs a mixture distribution formulation that can construct an optimal flow model with n layers of transformations from the transformation set. Besides, the authors introduce a block-wise optimization method to deal with exponentially growing optimization complexity. In the experiment, the authors proved the effectiveness of the optimization method which via approximate upper bound. And AutoNF has a better performance-cost trade-off than hand-tuned SOTA flow models. 
main_review: Positive points:
1. In this paper, the authors propose an automated normalizing flow architecture search method which can find the best distribution for each layer from a set of given distribution sequences. 
2. When constructing each layer of the model, the authors used a weighted summation of probability density instead of each distribution to ensure the reversibility of the model and the simple calculation.
3. The authors optimized the model via approximate upper bound instead of using KL divergence between the target distribution and the mixed flow, so that the model can go out of the local minimum.
sssssss
Negative points:

1. The novelty of this paper seems limited. The authors directly apply NAS techniques to the field of Normalizing Flow (NF). It would be better to clarify whether there are some technical contributions regarding to the search algorithm.

2. There seem some errors in Eqn. (4). The mixed transformation should be the weighted sum of T_i^j rather than T_i.

3. Eqn. (8) should use the symbol of expectation instead of directly using capital E.

4. As mentioned in Section 3.2, the global minimum may not be the desired architecture. Why can optimizing the upper bound find the desired architecture? It seems that both cases suffer the same issue.

5. Since this paper is essentially a NAS paper, it is necessary to compare the proposed method with existing NAS methods, e.g., DARTS [a], ENAS [b], MNasNet [c].

6. The performance-cost trade-off seems to depend on a parameter lambda that needs to be manually adjusted. Thus, the impact of lambda should be investigated in the experiment section.


Reference:

[a] Darts: Differentiable architecture search. ICLR 2019.

[b] Efficient neural architecture search via parameters sharing. ICML 2018.

[c] Mnasnet: Platform-aware neural architecture search for mobile. CVPR 2019.



summary_of_the_review: It is unclear the essential technical contributions compared to existing NAS methods.
review_title: null

Meta-review: This paper proposes two methods to learn the architecture of normalizing flows models; Their framework is inspired by (Liu et al., 2019) which uses ensembles/mixtures with learnable weights for architecture search. The application of these ideas to NFs requires a trivial modification to respect the invertibility constraint; which consists in building a mixture model over all possible sequences of compositions of transformations from a fixed set.

The paper proposes to use an upper-bound to the forward KL instead of the fKL directly. The reasoning is that this will lead to a "pure" model after optimization, that is, the mixture weights will be in {0, 1}. Mathematically, this simply corresponds to treating the mixture as a latent-variable model and performing MAP-inference over discrete latent variables, assuming that all mixture components have the same prior weights in the mixture.

The experimental results across various datasets are very mixed, and the family of transformations considered in the experiments is quite restricted.
Decision: Reject