AutoNF: Automated Architecture Optimization of Normalizing Flows Using a Mixture Distribution Formulation
Authors: Yu Wang
Jan Drgona
Jiaxin Zhang
Karthik Somayaji NS
Frank Y Liu
Malachi Schram
Peng Li
Venue: ICLR 2022 Submitted (2022)
Keywords: normalizing flow
architecture optimization

Abstract: Although various flow models based on different transformations have been proposed, there still lacks a quantitative analysis of performance-cost trade-offs between different flows as well as a systematic way of constructing the best flow architecture. To tackle this challenge, we present an automated normalizing flow (NF) architecture search method. Our method aims to find the optimal sequence of transformation layers from a given set of unique transformations with three folds. First, a mixed distribution is formulated to enable efficient architecture optimization originally on the discrete space without violating the invertibility of the resulting NF architecture. Second, the mixture NF is optimized with an approximate upper bound which has a more preferable global minimum. Third, a block-wise alternating optimization algorithm is proposed to ensure efficient architecture optimization of deep flow models. 

Under review as a conference paper at ICLR 2022
AUTONF: AUTOMATED ARCHITECTURE OPTIMIZA- TION OF NORMALIZING FLOWS USING A MIXTURE DISTRIBUTION FORMULATION
Anonymous authors Paper under double-bling review
ABSTRACT
Although various flow models based on different transformations have been pro- posed, there still lacks a quantitative analysis of performance-cost trade-off's be- tween different flows as well as a systematic way of constructing the best flow architecture. To tackle this challenge, we present an automated normalizing flow (NF) architecture search method. Our method aims to find the optimal sequence of transformation layers from given set of unique transformations with three folds. First, a mixed distribution is formulated to enable efficient architecture op- timization originally the discrete space without violating the invertibility of the resulting NF architecture. Second, the mixture NF is optimized with an ap- proximate upper bound which has more preferable global minimum. Third, a block-wise alternating optimization algorithm is proposed to ensure efficient ar- chitecture optimization of deep flow models.
INTRODUCTION
Normalizing flow (NF) is probabilistic modeling tool that has been widely used in density estima- tion. generative models, and random sampling. Various flow models have been proposed in recent years improve their expressive power. Discrete flow models are either built based on elemental- wise monotonical functions, named autoregressive flow or coupling layers (Papamakarios et al.) 2017 or built with transformations where the determinant of the flow can be easily calculated with matrix determinant lemma (Rezende & Mohamed, 2015). In the continuous flow family, the models are constructed by neural ODE (Grathwohl et al. 2019).
Despite the variety of flow models, there's yet perfect flow concerning the expressive power and the computation cost. The flow models with higher expressive power usually have higher computa- tional costs in either forward and inverse pass. In contrast, flows which are fast to compute not able to model rich distributions and are limited to simple applications. For instance, autoregressive flows (Papamakarios al., (017) universal probability approximators but are D times slower to invert than forward calcu lation, where D is the dimension of the modeled random variable æ (Papamakarios al.) 2021). Flows based on coupling layers (Dinh al., 2015 2017: Kingma & Dhariwal 2018) have an analytic one-pass inverse but are less expressive than their autoregressive counterparts. Other highly expressive NF models (Rezende & Mohamed 2013 Behrmann et al. 2019 cannot provide an analytic inverses and relies on numerical optimizations.
For different applications, the optimal flow model can be drastically different, especially the com- putation cost is taken into consideration. For generative models (Dinh et al.] 2015 Kingma & Dhariwal 2018), flows with the fast forward pass are preferable since the forward transformations need to be applied to every sample from the base distribution. For density estimation (Papamakarios ctt al. 2017 Rippel & Adams, 2013), flows with cheap inverse will prevail. For applications where flow is utilized as co-trained kernel (Mazoure et al.) 2020), the computation cost and performance trade-off are more important, i.e., having a fast model with relatively good performance. However, in the current body of work, the architecture designs of the flow models are all based manual configuration and tuning. To this date, there is lack of a systematic way that could automatically construct optimal flow architecture with preferred cost.

---

Under review as a conference paper at ICLR 2022
In this paper, we propose AutoNF, automated method for normalizing flow architecture optimiza- tion. AutoNF has a better performance- cost trade-off than hand-tuned SOTA flow models based on given set of transformations. Our approach employs mixture distribution formulation that can search a large design space of different transformations while still satisfying the invertibility require- ment of normalizing flow. The proposed mixture NF is optimized via approximate upper bound which provides better optimization landscape for finding the desired flow architecture. Besides, to deal with exponentially growing optimization complexity, we introduce block-wise optimization method to enable efficient optimization of deep flow models.
2 RELATED WORK
Various normalizing flow models have been proposed since the first con- cept in (Tabak & Turnr 2013). Current flow models can be classified into two categories: finite flows based on layer structure, and continuous flow based on neural ODE (Grathwohl et al.] (2019). al., The finite flow family includes flows based elemental-wise transformation (Papamakarios
Normalizing Flows:
2017| Kingma & Dhariwal, 2018) and flows whose transformations are restricted to be contractive (Behrmann et al.] 2019). In elemental-wise transformation flows, autoregressive flow and coupling layers are two major flavors and extensive work has been proposed to improve the expressive power of both flow models. In Huang et al.) (2018), the dimension-wise scalar transformation is imple- mented by sigmoid neural network, which increases the expressive power at the cost of being not analytically invertible. Durkan et al.] al.] (2019). piecewise splines are used as drop-in replace- ment of affine or additive transformations (Dinh et al. 2015 2017) and is the current SOTA flow model. Consequently many recent research efforts have been devoted to closing the gap of expres- sive power, albeit at the cost of more complex and expensive transformations. Moreover, there has been no quantitative trade-off analysis between the performance and cost among different flows.
Neural Architecture Search: Many algorithms have been proposed or applied for neural architec- ture search. For instance, reinforcement learning (Zoph & Le 2017), 1. genetic algorithm (Real et al. 2017 Suganuma et al. 2018, Liu et al., 2018). Monte Carlo tree search (Negrinho & Gordon 2017) or Bayesian optimization (Kandasamy et al.) 2018). However, these methods all face the challenge of optimizing on large discrete space and can take thousand of GPU days find good archi- tecture. To address this issue, DARTS (Liu et al. 2019) proposes to relax the search space from discrete to continuous and allows efficient differentiable architecture search with gradient method which could reduce the search time to a single GPU day while still producing the SOTA architecture. However, all current NAS methods focus on optimizing traditional neural network structures (CNN, RNN) and there has yet been any implementation on normalizing flow.
Despite various transformations proposed in the literature, there is no perfect transformation with strong expressive power and low computational cost. Autoregressive flows have better expressive power, but the inverse computa- tion cost grows linearly with data dimension. Coupling layers' inverse calculation is as fast as the forward pass, but their expressive power is generally worse than autoregressive flow with the same element wise transformation. Even in the same autoregressive flow or coupling layer family, flows with different element wise transformations have different performance and computation costs. For instance, additive or affine coupling layers (Dinh et al.) 2017 2015) have very fast forward and in- verse calculation with limited expressive power while the flow in (Durkan et al., 2019) are highly expressive but are more demanding on computation. In most applications, is necessary find the best performance while minimizing at least one specific component of the cost. Unfortunaely,, the current design of flow models is empirical and therefore cannot ensure the optimal trade-offs.
Necessity for the Trade-off Between Performance and Cost:
METHOD
In this work, aim to tackle the challenge of finding an optimal flow model for given task via an automated architecture search algorithm.
Assumptions: In the remaining part of this paper, without losing generality, assume that the transformation is properly modeled such that during the training process, only forward computation is needed. Under this assumption, when the flow model is used for density modeling (Durkan)
2

---

Under review as a conference paper at ICLR 2022
et al. 2019), the forward calculation is the dominant computation. When the flow model is used for random sampling (Kingma & Dhariwal 2018). the inverse calculation computationally intensive. When the flow model is utilized as module and trained together with other components, e.g.. policy network in maximum entropy learning (Mazoure et al. 2020), the training cost of the flow model is important consideration.
Problem Definition: Given transformation set with m options "T..T the goal is to construct an optimal flow model with layers of transformations from the set. The flow model PNr(2; 0) = PT.Ts...T, (a:: 0) should minimize the KI. divergence between the target distribution (x) and itself while minimizing transformation in the flow model. In this paper, function (Papamakarios et al. 2021):
computational cost CNF- Here, 0 are the parameters of the
the forward KL divergence as target loss
mig 0 arg min {DKL (a) PT (x; (c:0)] X CNF}
s.t. T₁ E (T*,
While À is tuning factor capturing the relative importance the performance-cost trade-off. Find- ing this optimal flow model is a discrete optimization problem with exponential complexity, To enable efficient architecture optimization, we use proposed method of relaxing the discrete search space to continuous space as suggested in Liu et al. (2019).
3.1 MIXED FLOW ENSEMBLE
For the in transformation layer with m options, we introduce a corresponding weight w/ for each option T which reflects how likely the transformation will be selected. The weight parameterized by vector and made continuous via softmax:
exp(a.)
ン==1 exp(a?)
By applying this parameterization for each transformation layer, we can construct a mixed flow ensemble Parer 0, a),, where each layer in this mixed model reflects weighted combination of the effect of all possible transformations. is reduced to learning the weight vector for each layer and, at the end of the optimization process, weights will be binarized and the transformation with the highest weight in one layer will be selected as the final transformation. The mixed flow ensemble thus degrades to normal flow model. The whole procedure is illustrated in Fig. (let).
this case, the architecture optimization problem
As adopted in (Liu al., 2019), training of the flow ensemble becomes joint optimization of the architecture parameter a and the model parameter over the training and validation datasets, which could be written the following bi-level optimization problem:
a arg min Deal [p"
(æ) PMix (w; 0", a)) + X CMiz(a)
0" arg min DRLi" (p" (x) PMii (x; a)),
TT PMis TE (T (T,,T2,..T
While the optimization problem well defined, the key challenge is construct the flow ensemble within the normalizing flow framework. This is different from traditional neural architecture search, which can mix various operations with no additional issue. Normalizing flow has its unique re- quirement for the invertibility of transformations and preferred simple Jacobian calculation, which requires careful handling.
The mixed flow ensemble PMrxe ; 0" αc) must satisfy two requirements. First, it must be legal density function such that it can be optimized by the KI. divergence formulation. Second, each transformation layer in PM (æ: 0* a.) should represent weighted combination of all possible transformations. Consider the ith layer in the mixed flow ensemble with input random variable ain and output random variable Couts and Pa.. (I:.) and Prost (Wout) are their corresponding density functions. This layer has transformation options in (T] T?? T.? and w is the corresponding
3

---

Under review as a conference paper at ICLR 2022
Flow
Mixed transformation
14
W
2
T
Nont
Prout (out)
oout
22
T
Xin
((a) 1f (Ktm)
(Xm)
71
W
2
Ti
Binerization:
Pxue
T1
T2
Mixed distribution
oout
W1
7
22
T
Xin
Xaut
Prout (oore)
7
wh
Tn
Popt(tin)
Pep Pri(
Normal flow model:
Xoue
T1
X1
12
X2
Xin
Pxi (x
Figure I: Left-top: the relaxation of search space and the flow ensemble is shown in Fig. T Left- middle: binarization of weights. Left-bottom: degradation to normal flow architecture. Right-top: construction flow ensemble by mixed transformations. Right-bottom: construction of flow ensemble by mixing distributions. The blue line in right indicates transformation on random variables and the orange line reflects change in distributions.
weight for each transformation. As discussed in Assumption, we assume all transformations directly model the inverse transformation, i.e. Fin TV (Fout). Two approaches can be used to construct the mixed flow ensemble.
Construction by Mixed Transformations: The straight forward way of building the ish mix flow ensemble layer is to mix all transformations by weighted summation, as shown in Fig, [ (right-top) The final weighted transformation for this layer can be thus represented as:
(1 Σ mi .Ti (Xout) 1=1
(4)
There are drawbacks of this formulation despite its simplicity. First, definition normaliz- ing flow requires the mixed transformation T; be invertible and differentiable in order to ensure Proci (Fout) legal density function. However, this invertibility is not guaranteed even if all candi- date transformations are invertible. Second, even if the mixed transformation is invertible, there is no easy way to calculate the Jacobian determinant of this weighted summation of transformations. Meeting the requirement of invertibility and case of calculating Jacobian determinant brings strict restrictions on the candidate transformations and prevents the optimization of flow architectures on a wider search space. As result, the construction of the mixed flow ensemble by weighted summation of transformations is not adopted in this paper.
Construction by Mixed Distributions: An alternating way is to build the mixed flow ensemble by mixing distributions. For a given transformation T/' in this is layer, applying the transformation to the input random variable will result in new distribution:
Pry (Fout) = Pa... (T] (Iout)) dot Jyp (Xout)l
(S)
By applying this to every transformation option in [T, butions, and it is possible mix all the density functions together by their weighted summation, to get a mixture model as shown in (6.
we can obtain different distri-
PT.(
Σ
wi Pry (Xout)
1=1
4

---

Under review as a conference paper at ICLR 2022
An illustration of this process is shown in Fig. I (right-bottom). Different from the previous ap- proach, the mixture model has legal density function as: pr, (Zout). By the definition of normal- izing flow,
assume that there exists invertible and differentiable transformation T, which transforms Din to Pout, although the transformation itself can not be explicitly written out.
For the next (i 1) layer, the density of the mixture model will be used the input density function Pæ... (mm)) as the previous layer. By applying this formulation for n layers, the final mixed flow ensemble can be written
m
PMix (2;0 0.
Wk TT. (x,
Σ
We pe(a;; 0.)
where each
We
II wi and Σ Wx =1
Each wy is defined in eq. and (= 0x) to represent "normal flow architecture" with transformation layers. Clearly, the final mixed flow ensemble is a legal density function which is in fact, weighted summation of all possible flow models built with layers of transformations.
3.2 OPTIMIZATION WITH APPROXIMATED UPPER BOUND
Optimizing the forward KL divergence between the target distribution and the mixed flow ensemble can be written as:
cp DEL (x) || PMix (z; 0, ax]
m
= -p
(log(
W (x: Σ
We will demonstrate that direct optimization of this original loss can lead to underside mixture models. In the whole search space of the flow ensemble, architectures" points, i.e. the points where the weight of one architecture is and others are all O. However,
interested only in "normal flow
be easily proven that the global optimum of CO may be the desired normal flow architecture (the red points in Fig. 2 Instead, optimization is very likely to end up in a mixture model that globally optimal with similar weight for each possible flow architecture (the green point in Fig. ( In this case, we will encounter difficulty when extracting normal flow architecture with the search result. common practice in differentiable architecture search (Liu et al.) 2019) is to binarize the weights and select corresponding transformations. However, there is no guarantee that the binarized architecture will have a lower loss, and finding this nearest binarization may lead to performance drop. As a result, optimization with the original loss function is not suitable, and could be risky.
Upper bound: Pari
Original loss: C
Normal flow architecture (Search target)
(1.00;)
Mixture model (Not desired)
(0,1.8))
(0.45,0.55.0,.0)
Figure 2: An illustrative example of the original loss and upper bound for flow ensemble with possible architectures. The red points indicate desired normal flow architectures and the green point indicates the global minimum of LO L which is a mixture model. The parameters (a, b, 0 f2) refer to the weight of architecture 1, architecture and their corresponding parameters.

---

Under review as a conference paper at ICLR 2022
In this paper, propose to optimize upper bound of the original loss function provide better global optimum for the search of best normal flow architectures. Our method utilizes Jensen's inequality log(Z W x) 2 Σ W. log(z) follows, since we have Σ W 1 and the log function concave, we can obtain an upper bound of the KI. divergence given as:
m
m
C PMax = - Epp log
Σ
Wi (x;2;;
S Sp = =- Ep (an) 1 Wi log(px (9)
The benefit of optimizing the upper bound can be summarized as follows:
Proposition! 1: The global minimum point of C is defined by normal flow architecture.
Suppose each flow model (x: 0x) has an optimal parameter 0% that minimizes the KL divergence between p (z) and it:
Proof:
-Ep (log(p 0,]] - Ep (m)]lo((ps 0x)]
(10)
There also exists flow architecture (p= (2; 02)) that has the minimal KL divergence:
Byp (w) [og(p= (2;; 03) S Ep (x) (og(pp 03), Wk m
(11)
We can then prove the proposition by showing that:
m
Cyarc = - (x)
W log(pe 0x)il 2 - -Ep >
We log(px(z;
(12)
2 -pp W& (o; = - (a) (log(p
Proposition 2: At normal architecture points (W₁ 1, W- 0), C C
The proof proposition 2 is apparent.
With the above propositions and under the assumption that the global optimum can be reached the end of the optimization. we can show that the solution set, ie. all possible normal flow architectures the same in both C.y... and C. C and can do optimization with proposed upper bound without violating the original definition. Furthermore, since the global optimum the upper bound will always lead to normal flow architecture, will not end up in finding mixture model with the need to do heuristic and risky binarization of weights W.
3.3 EFFICIENT ARCHITECTURE OPTIMIZATION FOR DEEP FLOW MODELS
While the flow ensemble by mixed density formulation could reflect the weighted effect of all pos- sible transformation combinations, the architecture optimization complexity grows exponentially with respect to the number of considered transformation types and the number of transformation layers. In this scenario, efficient optimization of the whole flow architecture will not be possible. It natural to decompose the original problem into sequential optimization of few different blocks, where each block could be optimized in one time with a limited number of layers. We propose two methods to decompose the problem.
The first approach is a straightforward greedy method which we call "Grow". Each time, a block is optimized until convergence, and the weights of the transformation layer are binarized. The searched transformations in this block will be directly added to the searched layer the previous block. The architecture optimization of later blocks will be based on the existing layers and, the growth of layers stops when reaching the total number of layers constraint. Despite its simplicity, the downside of the "Grow" method is that the optimization short-sighted. The block being optimized has information about the architectures which could be added later, and the whole architecture is more likely to be trapped in local minimum.
Grow Method:
To avoid the issue of getting stuck in local minimum, we propose another method named "Block" optimization. Blocks B in this approach are optimized alternatively to allow each block adjust their architectures with respect to other blocks. In fact, the first "Grow" approach is a specific case of the "Block" method, where all the blocks are initialized as identity transformations and optimized only once.
Block Method:

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Algorithm flow for AutoNF
Require: Transformations: {T), ,.m, Blocks: B = (B1, Bz, B.....Br ..). Cost: Cıi
Ensure: n-layer flow model:
1: while not converged do
2: for each B; E do
3: while not convergence do
4:
a B = arg min minan;
KL [p (a)) PMax (z; o3 RBB.] + X Caria(Oe,,)
5:
OB = arg milleg D (*))) PMix a 0B 0.)]]
6:
end while
7: Fix architecture for B,
8: end for
9:
end while
3.4 COST MODEL AND ALGORITHM FLOW
As discussed in section II, we are interested in modeling the training cost (forward calculation cost) and the inverse calculation cost, since each of them plays different role based on desired ap- plications. We use an independent experiment to model the cost of different types of flows and summarized in a table which are included in Appendix B. With the cost model, the total cost of the mixed flow ensemble could be extracted based on emphasize on different costs, e.g. training cost is the major concern, only training cost of different flows will be calculated. This total cost CMiz is then added as an regularization term into the training loss function.
In paper, gradient based method is used for optimization which is efficient in this very high dimensional search space. The architecture parameter a and the flow model parameter are opti- mized alternatively with first order approximation in (Liu et al. 2019). The final algorithm flow of our proposed AutoNF method can be summarized in Algorithm
4
EXPERIMENTS
4.1 EVALUATION OF PROPOSED UPPER BOUND
We use simple example to demonstrate the necessity of doing optimization with our proposed upper bound. We use AutoNF to build a 4 layer flow model with transformation options including planar flow and radial flow from (Rezende & Mohamed 2015). We use the POWER dataset as the target and optimize with original loss (name MI) and our proposed upper bound (named M2). We use Adam optimizer for both architecture parameter and model parameter with learning rate of 0.002. The batch size 512 and the training iteration 10000.
Setup:
The results shown in Fig For both MI and M2, we present the weight for planar and radial flow for each layer as well as the training and validation loss during the search process. The final weight for each layer, searched architectures after binarization and the test score shown in the right-bottom table.
Analysis: Optimization with our proposed upper bound (M2) shows a concrete convergence of weight 0 or 1 for each layer, which leads desired normal flow architecture, while the opti- mization with the original loss function (MI) ends up in a mixture model instead of a normal flow architecture, shown in Fig Fleft.. This is within in our expectation as shown in Fig Moreover, although the mixture model is mostly likely to be the optimal in the original loss, the normal flow architecture after binarization however, not an optimal model. As shown in the right- right-bottom table, the architecture found by M2 has a significantly better test score than MI, and this clearly supports our statement of doing optimization with our proposed upper bound.
4.2 SEARCH FOR FLOW MODELS WITH BEST PERFORMANCE COST TRADE-OFF
Transformation Options and Reference Designs: To evaluate our AutoNF framework, we setup experiments with four types of non-linear flows and one linear flow. In autoregressive family,

---

Under review as a conference paper at ICLR 2022
wegght of laye
weght of loger
Training loss
Yep sdadion
weight
planar
radial
planar
radial
Dager
0.968
0.032
0.988
Claner Flnur
0,723
0.277
0.001
Final W (Layer 31
0,014
0.96
0.998
0,002
W Bayer
0.889
0.061
Searched Arcliteturee
[planar,
Salener, planer,
Text Score
4/470.00
5.7090.077
The result of optimization of a 4-layer flow ensemble with transformation options between planar flow and radial flow with original loss and proposed upper bound. The left four figures are the weight for each layer during the search process. The right-top figures are the training and vali- dation loss during training. The right-bottom table collects final weight for each layer, the searched architecture, and their test score (lower the better).
Figure 3:
choose affine autoregressive flow (Papamakarios et al. 2017) and rational quadratic autoregressive flow (Durkan et al. 2019). Affine autoregressive flow has limited expressive power but the com- putation cost is lower, while the later has the state of art performance in autoregressive family with higher cost. Affine coupling layer (Dinh et al. 2015) and rational quadratic coupling layer (Durkan) 2019) are selected from coupling layer family. For linear transformation, we combine a re- verse permutation and an LU linear layer together as a single layer. Random permutation (Durkan
et al.
2019 Oliva et al. 201 8) is not used since it is difficult to reproduce in architecture optimiza- tion. Every non- linear transformation layer paired with a linear transformation layer suggested by Durkan al. (2019) final transformation option, i.e., layer in experiment contains reverse permutation, an LU-linear layer and one of the non-lineat transformation layer listed above.
al.
We use the rational quadratic flows family, including rational quadratic autoregressive flow (RQ-AF) and Rational quadratic coupling layer (RQ-C) in (Durkan et al. (209)) which have top 2 performance use RQ-AF as the baseline when emphasizing forward cost
the baseline. For fair comparison, since it has better performance and use RQ-C as the baseline when emphasizing inverse cost since RQ-C has significantly lower inverse cost.
Evaluation Metric and Datasets: Evaluating the performance-cost trade-off an open question in NF, we propose to use a new metric to address the difficulty of negative log-likelyhood (NLL). NLL is a common measurement for density estimation (lower, the better), however, the order of magnitude of NLL is different across different datasets and it is not suitable to use percentage dif- ference to measure how model is exactly better than another.
In this paper, We proposed to utilize density and coverage (Nacem et al. 2020) to evaluate the performance of NF models. Density and coverage are recently proposed method to evaluate the sample quality of generative models. The density metric reflects the fidelity of the model and is consistent with NLL metric. Across different datasets, density and coverage are at the same order of magnitude and allows evaluation of architecture across datasets. In our experiments, 10000 samples are drawn from the trained flow models and compare with 10000 samples from the test data. The results of three independent runs averaged the final reported results.
To evaluate the performance-cost trade-off, we define a figure of merit (FOM) as FOM = cost re- duction% density drop% compared to reference SOTA designs. In principle, the weight of the two terms can be manually adjusted to reflect the importance. For demonstration purpose, we use the equally weighted summation to report the results.
The performance of the flow models are evaluated with density estimation for UCI (Dua & Graff 2017) and BSDS330 (Martin al. 2001) datasets.
8

---

Under review as a conference paper at ICLR 2022
The architecture search results are reported in Table which includes the test NLL, density, coverage, cost and corresponding FOM. T shows that our AutoNF clearly helps to find architectures that have better performance-com trade-off. out AutoNF reach up to 3.66X cost reduction and up to 75.2% improvement in FOM compared with SOTA literature results. Across all five different datasets, AutoNF demonstrates average improvement of 58.67% on FOM with emphasis on forward cost and an average improvement of 52.57% on FOM with emphasis on inverse cost.
Analysis:
Table 1: Performance and cost trade-off between searched architectures and human designed archi- tectures UCI density estimation datasets. When the emphasizing forward cost, the cost column reports forward cost, when emphasizing on inverse cost, the cost column reports inverse cost. The test NLL (lower the better), density, coverage (higher the better) cost and FOM are reported.
Datasets
Cost Hmphasize
Architectures
Test NLL
Density
Coverage
Cost
FOM
RQ-AF
+0.66
0.9909
0.9637
16.64
0
Forward
AutoNF
-0.2410.01
0.9677
0,9558
4.41
+74,4%
POWER
RO-C
-0.64 0.01
0,9715
0.96
13.65
0
Inverse
AutoNF
-0.210.01
0.957
0.9547
3.73
+75.2%
RQ-AF
-13009 = 0.02
0.7939
0,8982
16.64
0
Forward
AutoNF
-9.1240.02
0.6233
0.769
5.41
+46.00%
GAS
ROC
-13.09
0.02
0.7874
0,8965
13.65
0
Inverse
AutoNF
1..5500.33
0.5186
0.687
4.73
+31.21
RQ-AF
14.01 +: 0.03
0.9428
0,9588
16.64
0
Forward
AutoNF
17.3610.02
0.9074
0.9549
6.49
+57.23%
HEPMASS
RQCC
14.7510.03
0.7822
0.9556
13.65
0
Inverse
AutoNF
19.32.10.02
0.8841
0,9471
5.73
+54,84%
RQ-AF
9.22 1: 0.48
0.48
0.9034
0.922
16.64
0
Forward
AutoNE
10.91+0.44
0.7957
0.9274
3.94
+64,40%
MINIBOONE
RQ-C
9.67 + 0.47
0.8257
0.8982
13.65
0
Inverse
AutoNE
13.56+0..57
0.6816
0.8768
3.73
+55 +55.22%
RQ-AF
-157.31+0.28
0.8696
0.896
16.64
0
Forward
AutON
131.73=00.28
0.7817
0.8758
6.41
+31.36%
BSDS300
RQ-C
-157.5420.28
0.8624
0,9069
13.65
0
Inverse
AutoNE
-149.79=0.28
0.7848
0.8332
6.1
+46.40%
DISCUSSION
Normalizing flow is highly parameterized module and designing a flow model and use for appli- cation requires a lot of hands-on experience and domain knowledge. In this paper, we show that the AutoNF framework is very effective in balancing performance- plex flow models. Moreover, although not demonstrated in this paper, the framework could also be used to help decide hyper parameters in complex flow model, e.g. the hidden features and number of bins in the SOTA coupling layer (Durkan et al. 2019). In additional, the proposed optimization method with upper bound can be easily extended to other suitable probabilistic kernels. one ex- ample is identify the best parameterized distribution(s) within a mixture model. We believe our framework will be very useful in many machine learning applications where normalizing flows are needed.
trade-offs when building com-

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Joern-Henrik Jacobsen. Invertible residual networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma- chine Learning Research, pp. 573-582, Long Beach, California, USA, 09-15 Jun 2019. PMLR. m
URL http::/proceediinn
9a html
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. CoRR, abs/ 1410.5516 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URLhttps://opperriiee: net / forumm 2o-
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http::////ac..
1cs. www.m.ee.mmmmm.m...
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Asso- ciates, Inc., 2019, URL 7ac71d433f 221.2222222223442444442222222222
pdf
Will Grathwohl, Ricky Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: free-form continuous dynamics for scalable reversible generative models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpp::/ ew wwtpiiew...eeeeeee..eeeee..eeeee
EUxgknCcK7
Chin-Was Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Con- ference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp- 2078 2087- PMLR, 10-15 Jul 2018. URL huangl8di html
ngs. ml mir.press//8C.
Kirthevasan Kandasamy, Willie Neiswanger. Jeff Schneider, Barnabas Poczos, and Eric Xing. Neural architecture search with bayesian optimisation and optimal transport. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL httt::///ppoeeeer....ere.../ss/p
neurips wtt.eeepp....peeeeereeeeeeeeeeee cctppapppr//11///1eeeee
Paper pdf
Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolu- tions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N.
and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https:
neurips pcc//papee/201//lleee pdf
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hier- archical representations for efficient architecture search. International Conference on Learning Representations, 2018. URL https: http:///eeepaaeeee.e.eeee//eeeas
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations 2019. URL Utt/ittts://opeereee net (eterrummddsSSeeeeee
D. Martin, C. Fowlkes, D. Tal, and Malik. A database human segmented natural images and its application evaluating segmentation algorithms and measuring ecological statisties. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pp. 416-423 vol.2, 2001. doi: 10.110//CC2.2011.99939
10

---

Under review as a conference paper at ICLR 2022
Bogdan Mazoure, Thang Doan, Audrey Durand, Joelle Pineau, and R Devon Hjelm. Leveraging exploration in off- algorithms via normalizing flows. Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 430-444. PMIR, 30 Oct-01 Nov 2020. URL
ml prw.sressss.. v1 wtt.11000maaaaeeee00 .ttml
Muhammad Ferjad Nacem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. 2020.
Renato Negrinho and Geoff Gordon. Deeparchitect: Automatically designing and training deep architectures, 2017.
Junier Oliva, Avinava Dubey, Manzil Zaheer, Barnabas Poczos, Ruslan Salakhutdinov, Eric Xing, and Jeff Schneider. Transformation autoregressive networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3898-3907. PMLR, 10-15 Jul 2018. URL httpss:/pxxceeddngg press wtt.ppeeee////olalaeese...
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL httppsss/pppooccedass....om wttp://www...aaee.eo.......o...e
pdf
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference, 2021.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alex Kurakin. Large-scale evolution of image classifiers, 2017.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Francis the 32nd International Conference Machine Learn- ing, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR. html
Bach and David Blei (eds.), Proceedings the
ml
Oren Rippel and Ryan Prescott Adams. High-dimensional probability estimation with deep density models, 2013.
Masanori Suganuma, Mete Ozay, and Takayuki Okatani. Exploiting the potential of standard con- volutional autoencoders for image restoration by evolutionary search, 2018.
E. G. Tabak and Cristina V. Turner. A family of nonparametric density estimation algorithms. Com- munications on Pure and Applied Mathematics, 66(2); 145-164, February 2013. ISSN 0010-3640. doi: 10. 10.1002/epa.21423
Barret Zoph and Quoc Le. Neural architecture search with reinforeement learning. In 5th In- ternational Conference on Learning Representations, ICLR 2017, Toulon, France. April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:://////..... /tss:///ppeeriee. net. net/forumm? pemamiidee r
l He&g


Reviews:
[{"review_id": "v2AIBV2tNX6", "reviewer": "Reviewer_ZCmD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\n\n1. This is the first work using NAS to optimize flow models\n\n2. Although the proposed method is based on DARTs, it requires some efforts to make it work on flow models, such as distribution mixture.\n\n[Weakness]\n\n1. The paper is a bit difficult to follow if the reader is not very familiar with flow model. Especially, the following things need to be further clarified in great detail:\n\n  a) How did you get Eq. (7) from Eq. (6)? In Eq. (7), $W_k=\\prod_{i}w_i$, so the right hand side does not contain $k$ at all?\n\n  b) The proof of upper bound optimization should be further clarified. It is hard to follow (at least to me).\n\n2. The upper bound argument seems questionable. While I could understand that we need to binarize the weight $\\alpha$ in order to get a simple and valid flow model, I still cannot understand why optimizing the Jensen's upper bound is a good (or better) idea. At least from the experiments, it seems that binarization  is still a necessary step.\n\n3. The proof of Proposition 1 seems questionable. The second $\\geq$ is not obvious. \n\n4. The experiments focus on density estimation problems. It is somewhat insufficient. I would expect more real-world applications and comparing to strong baseline methods.\n\n5. Table 1 need improvements. For example, it is better to explicitly align with one cost and then compare the test score. For now it is difficult to compare results since they have different costs.\n\n6. The proposed method almost directly follows DARTS. Although the distribution mixture is novel, it is still more or less an incremental improvement.\n", "summary_of_the_paper": "This paper proposes a DARTS-like method for searching automated normalization flow models. Instead of directly using the output ensembles, which leads to infeasible flow models, this work proposed distribution mixture to guarantee that the supernet is always a valid flow model. The upper bound of the loss function is optimized jointly with resource constraints. Experiments on small-to-medium scale datasets valid the effectiveness of the proposed method.", "summary_of_the_review": "To my best knowledge, this work is the first work aiming to design more efficient flow models via a DARTs-like NAS method. However, there are still many issues to be addressed. This work can be made significantly stronger if the aforementioned issues are well addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "GOUVo8DqYsS", "reviewer": "Reviewer_bHa6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points:\n1. In this paper, the authors propose an automated normalizing flow architecture search method which can find the best distribution for each layer from a set of given distribution sequences. \n2. When constructing each layer of the model, the authors used a weighted summation of probability density instead of each distribution to ensure the reversibility of the model and the simple calculation.\n3. The authors optimized the model via approximate upper bound instead of using KL divergence between the target distribution and the mixed flow, so that the model can go out of the local minimum.\nsssssss\nNegative points:\n\n1. The novelty of this paper seems limited. The authors directly apply NAS techniques to the field of Normalizing Flow (NF). It would be better to clarify whether there are some technical contributions regarding to the search algorithm.\n\n2. There seem some errors in Eqn. (4). The mixed transformation should be the weighted sum of T_i^j rather than T_i.\n\n3. Eqn. (8) should use the symbol of expectation instead of directly using capital E.\n\n4. As mentioned in Section 3.2, the global minimum may not be the desired architecture. Why can optimizing the upper bound find the desired architecture? It seems that both cases suffer the same issue.\n\n5. Since this paper is essentially a NAS paper, it is necessary to compare the proposed method with existing NAS methods, e.g., DARTS [a], ENAS [b], MNasNet [c].\n\n6. The performance-cost trade-off seems to depend on a parameter lambda that needs to be manually adjusted. Thus, the impact of lambda should be investigated in the experiment section.\n\n\nReference:\n\n[a] Darts: Differentiable architecture search. ICLR 2019.\n\n[b] Efficient neural architecture search via parameters sharing. ICML 2018.\n\n[c] Mnasnet: Platform-aware neural architecture search for mobile. CVPR 2019.\n\n\n", "summary_of_the_paper": "In this paper, the authors present an automated normalizing flow(NF) architecture search method. The method employs a mixture distribution formulation that can construct an optimal flow model with n layers of transformations from the transformation set. Besides, the authors introduce a block-wise optimization method to deal with exponentially growing optimization complexity. In the experiment, the authors proved the effectiveness of the optimization method which via approximate upper bound. And AutoNF has a better performance-cost trade-off than hand-tuned SOTA flow models. ", "summary_of_the_review": "It is unclear the essential technical contributions compared to existing NAS methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GG558wScuFJ", "reviewer": "Reviewer_gQWy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe problem considered in this work is novel and important for the community that works with flows. The problem is significantly more challenging than standard architecture search. The proposed solution is well-motivated. The theoretical claims seems to be correct. The flow of the paper is easy to follow and each step of the proposed approach is justified. The novelty and contribution of the paper is high in my opinion. The idea of using mixing the probabilities and the application of upper bound instead of direct optimisation is non-trivial.\n \n\nWeaknesses \n\nThe empirical evaluation of the proposed method should be more extensive. The selection manual flow baseline is a bit tricky to be. How far is the manual approach from the optimal combination of transformations on validation set? It would be also beneficial to create the baseline where the transformation are selected randomly. Such a baseline would deliver the information what is the gain on NLL with respect to the random approach. I would suggest also to take under consideration evolution approach that optimize the binary vectors that stays behind the selection process as reference method. \n\nMy second concern is about the generalisation of the proposed approach to other types of flows. The approach seems to be generic and scale to any possible transformations, but the experimental evaluation is mainly focused on autoregressive flows. Is it possible to adopt that approach to various types of layers that represent the dynamics in CNF? It would be also beneficial to see what is the quality of that approach for various CNF layers in experimental part. \n\nThe third concern is about the limitation of the approach due the fact that complexity grows exponentially and some decomposition methods are essential to apply architecture search effectively. For this part it would be interesting how much we loose during the decomposition process. ", "summary_of_the_paper": "This work provides the novel approach for searching flow architectures. Compared to the standard approaches used to find the best deep architecture in standard network networks this problem is more challenging due to the need of invertible transform and requirement for determinant of the Jacobian to be easy to calculate. To solve the problem the authors propose to apply the weighting for candidate transformations. In order to enforce invertible properties of the model the authors suggest to use the mixing distribution approach instead of mixing the base transformations. They formulate the problem of learning best weights and show that optimal solution for soft weights is not optimal for binarised versions. Therefore they propose to optimize the upper bound of the proposed loss function instead. Further they show how to deal with the problem with larger number of layers. Some experiments are also performed to show the quality of the approach with respect to the baseline that is expert-based selection. ", "summary_of_the_review": "Concluding, the problem consider by the authors is novel and worth investigating, the approach seems to be adequate, the evaluation is limited but I think the paper is worth accepting.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7IUEQWB8Vqt", "reviewer": "Reviewer_ucQj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nOverall the paper is well written and easy to follow. The main idea is to adapt the differentiable formulation for the NAS towards normalizing flow architecture optimization, and the authors made interesting theoretical contributions for reformulating the mixture-of-experts setup towards normalizing flow models, as well as for proposing novel optimization strategies. Overall the proposed method is sound and coherent.\n\nCons:\n\n- The experimental evaluation seems relatively weak. Though the optimized architectures seem to consistently have lower train, forward and inverse costs, the test performance is mixed, and in many scenarios worse-performing than the manually design architecture. \n- The search space that the authors experimented with is extremely limited, limited to only planar flows and radial flow, with the weights between the two transformations being the only architectural hyperparameter learned, not accounting for other hyperparameters such as number of stacked flows, the network complexity (number of feature layers, etc) for each network, which has little generalizability to more modern and useful architectures (e.g., RealNVP, GLOW, FFJORD).\n- On the novelty side, though the authors made problem-specific adaptations for the differentiable architecture search algorithm for normalizing flows, the main idea is very much adapted from Liu et al. 2019 and somewhat marginally novel.\n", "summary_of_the_paper": "In this paper, the authors proposed to adapt a differentiable architecture search formulation (Liu et al., 2019) based on learned weighting of an ensemble of modules to automated search for Normalizing Flow architectures. The authors made several adaptations to the original approach for the normalizing flow problem, due to the invertibility constraints that prevent direct linear summation of different transform operations. Furthermore, the authors proposed to optimize the full network using an approximated upper bound of the KL divergence, instead of directly optimization. The authors proposed two methods to decompose the optimization problem: grow method, which is more straightforward and greedy, and block method, that alternatively adjusts each block. The authors experimentally compared their proposed method with manually specified architectures across various datasets, including POWER, GAS, HERMASS, MINIBOONE and BSDS300. The results seem mixed, as the searched model outperforms the manual model in some contexts but not others.", "summary_of_the_review": "As a summary, the overall novelty of the proposed approach is rather incremental, the search space is too restrictive and limited to be practically useful for designing practical NF architectures (only searching for a binary choice between two architectures, not accounting for other more complex bijective transformations), and the experimental results are insufficient to illustrate the usefulness of the proposed approach for improving NF architectures.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes two methods to learn the architecture of normalizing flows models; Their framework is inspired by (Liu et al., 2019) which uses ensembles/mixtures with learnable weights for architecture search. The application of these ideas to NFs requires a trivial modification to respect the invertibility constraint; which consists in building a mixture model over all possible sequences of compositions of transformations from a fixed set.

The paper proposes to use an upper-bound to the forward KL instead of the fKL directly. The reasoning is that this will lead to a "pure" model after optimization, that is, the mixture weights will be in {0, 1}. Mathematically, this simply corresponds to treating the mixture as a latent-variable model and performing MAP-inference over discrete latent variables, assuming that all mixture components have the same prior weights in the mixture.

The experimental results across various datasets are very mixed, and the family of transformations considered in the experiments is quite restricted.
Decision: Reject