InfoCNF: Efficient Conditional Continuous Normalizing Flow Using Adaptive Solvers
Authors: Tan M. Nguyen
Animesh Garg
Richard G. Baraniuk
Anima Anandkumar
Venue:  (2020)
Keywords: continuous normalizing flows
conditioning
adaptive solvers
gating networks

Abstract: Continuous Normalizing Flows (CNFs) have emerged as promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation. However, conditioning CNFs on signals of interest for conditional image generation and downstream predictive tasks is inefficient due to the high-dimensional latent code generated by the model, which needs to be of the same size as the input data. In this paper, we propose InfoCNF, an efficient conditional CNF that partitions the latent space into a class-specific supervised code and an unsupervised code that shared among all classes for efficient use of labeled information. Since the partitioning strategy (slightly) increases the number of function evaluations (NFEs),  InfoCNF also employs gating networks to learn the error tolerances of its ordinary differential equation (ODE) solvers for better speed and performance. We show empirically that InfoCNF improves the test accuracy over the baseline  while yielding comparable likelihood scores and reducing the NFEs on CIFAR10. Furthermore, applying the same partitioning strategy in InfoCNF on time-series data helps improve extrapolation performance. 

Under review as a conference paper at ICLR 2020
INFOCNF: EFFICIENT CONDITIONAL CONTINUOUS NORMALIZING FLOW USING ADAPTIVE SOLVERS
Anonymous authors Paper under double-blind review
ABSTRACT
Continuous Normalizing Flows (CNFs) have emerged promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation. However, conditioning CNFs on signals of interest for conditional image generation and downstream predictive tasks is inefficient due to the high- dimensional latent code generated by the model, which needs be of the same size as the input data. In this paper, we propose InfoCNF, an efficient conditional CNF that partitions the latent space into a class-specific supervised code and an unsupervised code that shared among all classes for efficient use of labeled information. Since the partitioning strategy (slightly) increases the number of function evaluations (NFEs), InfoCNF also employs gating networks to learn the error tolerances of its ordinary differential equation (ODE) solvers for better speed and performance. We show empirically that InfoCNF improves the test accuracy over the baseline while yielding comparable likelihood scores and reducing the NFEs CIFAR10. Furthermore, applying the same partitioning strategy in InfoCNF on time-series data helps improve extrapolation performance. Additional details (tttts: google www.////ww..////////////o..ttee. http:///sss.....ooo.o..o//c///ea
INTRODUCTION
Invertible models are attractive modelling choice in a range of downstream tasks that require accurate
densities including anomaly detection (Bishop) 1994.
Chandola et al.] 2009) and model-based
reinforcement learning (Polydoros & Nalpantidis 2017). These models enable exact latent-variable
inference and likelihood estimation. A popular class of invertible models is the flow-based generative
models (Dinh et al. 2017] |Rezende & Mohamed 2015 Kingma & Dhariwal 2018 88 Grathwohl et al. 2018) that employ a change of variables to transform a simple distribution into more complicated ones while preserving the invertibility and exact likelihood estimation. However, computing the likelihood in flow-based models is expensive and usually requires restrictive constraints on the architecture in order to reduce the cost of computation. Recently, (Chen et al. (2018) introduced a new type of invertible model, named the Continuous Normalizing Flow (CNF), which employs ordinary differential equations (ODEs) to transform between the latent variables and the data. The use of continuous-timg transformations in CNF, instead of the discrete ones, together with efficient numerical methods such as the Hutchinson's 's trace estimator (Hutchinson, 1989). helps reduce the cost of determinant computation from (dd²) to (d), where d is the latent dimension. This improvement opens up opportunities scale up invertible models to complex tasks on larger datasets where invertibility and exact inference have ad advantages.
Until recently. CNF has mostly been trained using unlabeled data. In order take full advantage of the available labeled data, a conditioning method for CNF which models the conditional likelihood, as well the posterior, of the data and the labels is needed. Existing approaches for conditioning flow-based models can be utilized, but we find that these methods often do not work well on CNF. This drawhack is because popular conditioning methods for flow-based models, such as in (Kingma Dhariwal 2018). make use of the latent code for conditioning and introduce independent parameters for different class categories. However, in CNF, for invertibility, the dimension of the latent code needs to be the same as the dimension of the input data and therefore is substantial, which results in many unnecessary parameters. These additional but redundant parameters increase the complexity of the model and hinder learning efficiency. Such overparametrizatioo also has negative impact other flow- generative models, as was pointed out by (Liu et al. 2019), but is especially bad in the case of CNF. This is because the ODE solvers in CNF are sensitive the complexity of the

---

Under review as a conference paper at ICLR 2020
model, and the number of function evaluations that the ODE solvers request in a single forward pass (NFEs) increases significantly as the complexity of the model increases, thereby slowing down the training. This growing NFEs issue has been observed in unconditional CNF but to a much lesser extent (Grathwohl et al. 2018). It poses a unique challenge to scale up CNF and its conditioned variants for real-world tasks and data.
Our contributions in this paper are as follows:
Contribution 1:
We propose simple and efficient conditioning approach for CNF, namely InfoCNF. Our method shares the high-level intuition with the InfoGAN (Chen et al. 2016), thus the eponym. In InfoCNF, we partition the latent code into two separate parts: class-specific supervised code and unsupervised code which is shared between different classes (see Figure We use the supervised code condition the model on the given supervised signal while the unsupervised code captures other latent variations in the data since it is trained using all label categories. The supervised code is also used for classification, thereby reducing the size of the classifier and facilitating the learning. Splitting the latent code into unsupervised and supervised parts allows the model to separate the learning of the task-relevant features and the learning of other features that help fit the data. We later show that the cross-entropy loss used to train InfoCNF corresponds to the mutual information between the generated image and codes in InfoGAN, which encourages the model to learn disentangled representations.
Contribution 2: We explore the speed-up achievable in InfoCNF by tuning the error tolerances of the ODE solvers in the model. ODE solvers can guarantee that the estimated solution is within given error tolerance of the true solution. Increasing this tolerance enhances the precision of the solution but results in more iterations by the solver, which leads to higher NFEs and longer training time. However, when training neural network, it might not be necessary to achieve high-precision activation, i.e. the solution of the corresponding ODE, each layer. Some noise in the activations can help improve the generalization and robustness of the network (Gulcehre et al. 2016 Bengio et al. 2013 Nair& & Hinton| 2010; Wang et al., 2018a). With carefully selected error tolerances, InfoCNF can gain higher speed and better performance. However, the process of manually tuning the tolerances con-ccosuming and requires a large amount of computational budget. To overcome this limitation, propose new method to learn the error tolerances of the ODE solvers in InfoCNF from batches of input data. This approach employs learnable gating networks such as the convolutional neural networks to compute good error tolerances for the ODE solvers.
Contribution 3:
We study methods to improve the large-batch training of InfoCNF including tuning and learning the eиror tolerances of the ODE solvers, well increasing the learning rates.
We conduct experiments on CIFARIO and show that InfoCNF equipped with gating networks outperforms the baseline Conditional Continuous Normalizing Flow (CCNF) in test error and NFEs in both small- batch and large-batch training. In small-batch training, InfoCNF improves the test error over the baseline by 12%, and reduces the NFEs by 16%. When trained with large batches, InfoCNF attains a reduction of 10% in test error while decreasing the NFEs by 1 11% compared to CCNF. InfoCNF also achieves slightly better negative logliikll (NLL) score than the baseline in batch training, but attains slightly worse NLL score in small-bateh training.
In order better understand the impact of the gating approach to learn the error tolerances, we compare InfoCNF with and without the gating networks. In small-batch training, InfoCNF with gating networks achieves similar classification and density estimation performance as the same model without the gating networks, but reduces the NFEs by more than 21%. When trained with large batches, gating networks help attain a reduction of 5% in test error and small improvement in NLLs. We also confirm the benefits of our gating approach on unconditional CNF and observe that on CIFAR 10 learning the error tolerances helps reduce the NFEs by 15% while preserving the NLL.. Furthermore, we explore the potential benefit of the partitioning strategy for time-series data. In our experiments, when the latent code is partitioned in the baseline LatentODE (Rubanova et al. 2019). the model achieves better performance in curve extrapolation.
EFFICIENT CONDITIONING VIA PARTITIONED LATENT CODE
We will begin by establishing our notation and provide brief review of the flow-based generative models. Throughout this paper, we denote vectors with lower-case bold letters (e.g.. x) and scalars with lower-case and no bolding (e.g.. x). We use x, y, and 0 to denote the input data/input features, the
2

---

Under review as a conference paper at ICLR 2020
Stage
Stage
CNN Gate
CNN
ODESoIv
supervised code
ODESo
unsupervised
Input
Figure I: InfoCNF with CNN gates that learn the tolerances of the ODE solvers using reinforcement learning. The latent code InfoCNF split into the supervised unsupervised codes. supervised code used for conditioning and classification. The unsupervised code captures other latent variations in the data.
supervised signals (e.g., the class labels), and the model's parameters respectively. The superscript i used to indicate the layer i. For example, x; is the set of input features into layer i.
2.1
BACKGROUND ON FLOW-BBSED GENER MODELS
Invertible flow-based generative models such as RealNVP (Dinh et al. 2017 and Glow (Kingma & Dhariwal 2018) have drawn considerable interest recently. These models composed of bijective
transforms whose Jacobian matrices are invertible with tractable determinants. Let (zz; 0) denote
such transform applied on the latent variable generate the data Because of its bijective structure, the transform ff not only allows exact inference of z given x but also enables exact density evaluation via the change of variable formula:
log (x)
pz() - log det (3f(z:
The exact inference and exact density evaluation are preserved when stacking the bijective transforms into deep generative model. The chain formed by the successive probability distributions generated by these transforms is called a normalizing flow, and the resulting generative models are called flow-based generative models. The distribution of the latent code at the top of the model is usually chosen to be a factorized standard Gaussian to simplify the computation, and the parameters are learned by maximizing the exact log-likelihood log p(gp(X 0) where X is the training set containing all training data x. While flow-based generative models enjoy nice properties, the requirements to ensure the invertibility and tractable computation restrict the expressive power of the model.
Recently, the Continuous Normalizing Flows (CNFs) have been explored in (Chen et al. 2018 to bypass the restrictive requirements in the flow-based generative models and allow the models be expressive for more complicated tasks. CNF defines the invertible transforms via continuous-time dynamics. models the latent variable z and the data x as values of continuous- time to and (1, respectively. Given z, CNF solves the initial value problem to find x
variable z(t)
(to) z. (z(())/(rrrrrr rx;
The change in deg-denstty under this model follows the instantancous change of variables formula
l log f" Tr(@f(u()::)))d(()))
log log((z(n)) =
Thus, CNF reduces the (dd²) cost of computing the determinant to the (dd)) cost of computing the trace. Taking advantage of the Hutchinson's 's trace estimator (Hutchinson, 1989). this computation cost can be reduced to (dl) (Grathwohl et al. 2018) where d' is the dimension of latent code z.
2.2 CONDITIONING CNF VIA PARTITIONED LATENT CODE
Conditional CNF: To the best of our knowledge, there has not been a conditioning method partic- ularly designed for CNF, However, since CNF belongs to the flow-based generative model family, conditioning methods for flow-based models can also be applied on CNF In particular, the condition- ing approach via a Gaussian Mixture Model (GMM) and an auxiliary classifier proposed in (Kingmal & Dhariwal 2018) has been widely used. We refer CNF equipped with this type of conditioning as the Conditional Continuous Normalizing Flow (CCNF). We use CCNF as the baseline for comparison in our experiments. In CCNF, the distribution of the latent code follows GMM whose means
3

---

Under review as a conference paper at ICLR 2020
and scales are functions of the conditional signal and parameterized by simple neural networks. Furthermore, an additional predictive model is applied on z to model the distribution p(ylz) via an auxiliary predictive task
z~ (Z))
-0))))0))-0)0 p(y|z)= qo(z).
(3)
Here, Qo(y) and (z(z) are usually chosen to be neural networks whose parameters are learned during training. While this conditioning method has been shown to enable flow-based models to do conditional image generation and perform predictive tasks such as object classification in (Kingma & Dhariwal, 2018), it is in fact rather inefficient. This is because the size of the latent code z in flow-based generative models is the same as the size of the input image and therefore often large. As a result, the conditioning network used to synthesize z and the predictive network qe applied on z introduce a significant amount of additional parameters for the model to learn.
InfoCNF only uses portion of the latent code for conditioning. In particular, In- foCNF splits z into two non-overlapping parts supervised latent code z, and unsupervised latent code such that z Z Zu] The supervised latent code zy captures the salient structured semantic features of the data distribution. In particular, denote the set of structured latent variables which account for those semantic features by y1, : Vr.. For simplicity and efficient computation, we as- sume that these latent variables follow a factored distribution such that P(y1.y2, '.. -Yz) I P(y)) The supervised code Zy is the concatenation of . Zyz .... Zyz where Zy; is the code that captures the latent variable y;. We use Z, for conditioning the model. Similar to the conditional Glow, the distribution p(z,) is modeled by GMM (Z)
InfoCNF:
whose centers and scales are functions of the conditional signal y. As in Eq. (3) these functions are parameterized by a neural network (r)... The posterior p(y z.) is then approximated by another neural network (Z,) applied on z, to solve the corresponding predictive task. The unsupervised code zu ~ (z 00 /) can be considered as source of incompressible noise which accounts for other latent variations in the data.
We learn InfoCNF by optimizing the supervised loss from qu (Z(Z)) and the conditional log-likelihood
log p(xly) of the model. The learning objective of InfoCNF is given by
J = CNL (xly) - Cxent (f,y),
where Lxem (f.y) is the cross-entropy loss between the estimated label y and the ground truth label y. is the weighting factor between the cross-entropy loss Exem (f,y) and the conditional -og-likelihood loss (x))) LNLL (xl) is given by
K Σ/ l.' (1(0
(xly = pxly( = poz)()))y) + (ogp(z.)
(5)
1=1
where are indices for layers in the network and (2)
are calculated from the formula for log-likelihood of a Gaussian distribution. In our notation, we set Zς and zo = x. For each integral of the trace of the Jacobian in Eqn. 2 without generality, we choose to 0 and f
The mutual information between the gener- ated images and the codes in InfoGAN approximated by a variational lower bound via an "auxiliary" distribution, which is chosen to be neural network. Since InfoCNF is an invertible model, the generated images from the model given the codes matches the input images. Thus, maximizing the mutual information between the generated images and the codes is equivalent maximizing the cross-entropy loss between the estimated label and the ground truth label, which is the loss (f.y) in Eqn. 4 Thanks to the invertibility of InfoCNF, "auxiliary" network.
Connection to the mutual information in InfoGAN:
eliminate the need of using an additional
Compared CCNF, InfoCNF needs slightly fewer parameters since the size of the supervised code zy is smaller than the size of z. For example, in our experiments, z, is only half the size of and InfoCNF requires 4% less parameters than CCNF, This removal of unnecessary parameters helps facilitate the learning. As discussed in Section 4.3) our experiments on CIFARIO suggest that InfoCNF requires significantly less NFEs from the ODE solvers than CCNF. This evidence indicates that the partition strategy in InfoCNF indeed helps alleviate the difficulty during the training and improves the learning of the model.
LEARNING ERROR TOLERANCES OF THE ODE SOLVERS
Tuning the error tolerances: We explore the possibility of improving InfoCNF by tuning the error tolerances of the ODE solvers in the model. The advantage of this approach is two-fold. First, it
4

---

Under review as a conference paper at ICLR 2020
reduces the number of function evaluations (NFEs) by the ODE solvers and, therefore, speeds up the training. Second, we hypothesize that the numerical errors from the solvers perturb the features and gradients, which provides additional regularization that helps improve the training of the model.
Learning the error tolerances: We extend our approach of tuning the error tolerances of the ODE solvers by allowing the model to learn those tolerances from the data. We propose InfoCNF with learned tolerances, which associates each ODE solver in InfoCNF with gating network that computes the error tolerance of the solver such that the model achieves the best accuracy and negative log-likelihood with the minimal NFEs. These gates are learnable functions that map input data or features into the tolerance values. In our experiments, we use CNNs for the gates (see Figure !I.
The eπror tolerance decides how many iterations the solvers need to find the solution. This process discrete and non-differentiable, which creates a unique challenge for training InfoCNF with learned tolerances. We exploit the reinforcement learning approach to solve this non-differentiable optimization and learn the parameters of the gating networks. In particular, each gate, we formulate the task of learning the gating network as a policy optimization problem through reinforcement learning (Sutton & Barto 1998) to find the optimal error tolerances.
In InfoCNF, we assume that the error tolerances can be modeled by Gaussian distribution. The sample sequence of the error tolerances drawn from our policy starting with input x is defined as g [g). .gv] ~ Aa.. where Fg Fie.... _Fve] is the sequence of network layers with parameters and &i ~ (g.l of) where [u, o = (x)) are the outputs of the gating network layer i. Here, the policy is to decide which error tolerance to use and is defined as a function from input x; to the probability distribution over the values of the error tolerances, π (x;; = P = g:))- The CNN in the gating network is used to estimate the parameters of the probability distribution over the values of the error tolerances. We choose the rewards function R; -NFE, the negative of the number of function evaluations by the solver at layer i, so that our policy tries to fit the model well and do good classification while requiring less computation. The overall objective is given as:
min ((0) min Ex, Be Lxem (xxm((l))) (f,)) + (xIL(x,))
a N
(g(g)
(6)
where the a balance between minimizing the prediction/NLL loss and maximizing the rewards.
Employing REINFORCE (Williams 1992). we can derive the gradients VeJ. Defining TF, (x) Po L = Lxem (F,y) + (x,) and r; -If Ey Rj]. the gradients VoJ is given by
VoJ(O)= = BL Exy g
e log (g)()))
The first part of Eq. iis gradients of the cross-entropy and NLL loss while the second part corresponds to the REINFORCE, gradients in which r; is the cumulative future rewards given the error tolerance estimated by the gating networks. This combination of supervised, unsupervised, and reinforcement learning encourages the model to achieve good performance on classification and density estimation while demanding less number of function evaluations by the ODE solvers.
4 EXPERIMENTAL RESULTS
In this section, we empirically demonstrate the advantage of InfoCNF over the baseline CCNF when trained on CIFARIO 10. Throughout the experiments, we equip InfoCNF with the gating networks to learn the emror tolerances unless otherwise stated. Compared to CCNF, InfoCNF achieves significantly better test errors, smaller NFEs, and better (in large- batch training) or only slightly worse (in small- batch training) NLL. Furthermore, improve InfoCNF in all criteria except for a slightly worse NLL in small-batcl training and a similar NFEs in large-batch training. We also describe how evaluate our model to make sure that reported results are not biased by the numerical error from the ODE solvers in section 4.2
observe that learning the error tolerances of the solvers helps
4.1 IMPLEMENTATION DETAILS
Dataset: We validate the advantages of our models on the CIFARIO dataset. Uniform noise is added to the images, and during training, the data randomly flipped horizontally.
5

---

Under review as a conference paper at ICLR 2020
Networks: We use the FFJORD multiscale architecture composed of multiple flows in (Grathwohl al. 2018) for our experiments. The network details can be found in Appendix [ When conditioning separate 1-layer linear networks for the classifier qe and the condition encoder do- The parameters of the networks are initialized to zeros. In InfoCNF, we use half of the latent code for conditioning. We apply dropout (Srivastava et al., (014) of rate 0.5 on the linear classifier.
the model,
Training: We train both CCNF and InfoCNF with the Adam optimizer (Kingma & Baj (013). When using the batch size of 900, we train for 400 epochs with learning rate of 0.001 which was decayed to 0.0001 after 250 epochs. When using the batch size of 8,000, we train for 400 epochs with learning rate of 0.01 which was decayed to 0.001 at epoch 120.
4.2 EVALUATION PROCEDURE
The adaptive ODE solvers perform numerical integration and therefore have errors inherent in their outputs. When evaluating the models, we need to take these numerical errors into account. Since our method of learning the error tolerances is purely for training, a reasonable evaluation tolerances to a small value at test time and report the results on the test set. In order to find which small value of the error tolerance to use for evaluation, we train InfoCNF on 1-D synthetic data. Since a valid probability density function needs to integrate to one, we take InfoCNF trained on the synthetic 1-D data sampled from mixture of three Gaussians and compute the area under the curve using Riemann sum at different error tolerance values starting from the machine precision of 10- (see Appendix forr more details). Figure 2 shows that the numerical errors from InfoCNF is negligible when the tolerance is less than equal to 10 Thus, in our experiments, we tolerances to 10 at test time. In other to validate that a tolerance of 10 10-5 still yields negligible numerical errors on complex datasets like CIFARIO 10, we also evaluate our trained models using tolerances of 10 10-7, and 10-8, 8 We observe that when using those values for tolerances, our trained models yield the same test errors and NLLs as when using tolerance of 10 5, The NFEs of the ODE solvers reported in our experiments are computed by averaging the NFEs over training epochs. Figure 2p shows the Iearned error tolerances from InfoCNF trained on CIFARIO using small batches of size 900. We validate that the learned tolerances from the train and test sets are similar and thus the learned tolerances from our model do not only overfit the training data. Evaluation using the learned error tolerances with different batch sizes is discussed in Appendix D
set the error
2.50
10-1
(a)
Solver Tolerance
2.25
(b)
tain
test
2.00
10a
3.30
10
3.25
1.09
10-1
30-R
30.7
10
10 10ta
10-2
10
10
Solver Tolerance
Layer
Figure 2: (a) Log plot that shows that the density under the model does integrate to one given sufficiently low tolerance. (b) Distribution of error tolerances learned by InfoCNF at each layer over CIFARI0 train and test sets.
4.3 INFOCNF VS. CCNF
CNF
CNF Ieamedd tolcrances
fixed lekeanoss
Figure 33 shows that compared to CCNF, InfoCNF improves the test classification error CIFARIO by 12% when training with small batches. Interestingly, the advantage of InfoCNF over CCNF still holds when we train the models with large batch size of 8k (10% improvement in test classification error, see Fig- ure (t). While InfoCNF facilitates classification. also attains similar NLLs compared to CCNF in small- -bach training and better NLLS in large-batch training (Figure 3p. e). Figure 33 and f show the evolution of NFEs during training of InfoCNF and
NLL (bitsddmm)
3.7
3.2
50
100
150
211 Epach
250
300
350
21.0
(b)
FFnccin Evaluations
CCNF, In small-batch experiments, InfoCNF is much more ef- ficient than CCNF in the first 240 epochs but then the NFEs of InfoCNF increases and exceeds the NFEs of CCNF. Overall, the NFEs from InfoCNF during training is 16% less than the NFEs from CCNE. When training with large batches, InfoCNF requires 11% less NFEs than CCNE.
300
790
600
20
100
150
200
353
300
250
Epoch
Figure 4: (a) Test NLLs, (b) NFEs of CNF with/without learned tolerances in small- batch training CIFARIO.

---

Under review as a conference paper at ICLR 2020
CCNF (baseline)
InfoCNF fixed tolerances
InfoCNF learned tolerances
0.5
(b)
Functonn Evalustoons
2120
(c)
0.3
NILL (sloadim))
2230
339
139
0.25
-0
230
0.3
0.20
Epech
Epoch
Epoch
Small-Batee Training (batch size 900)
(d)
(e)
Function Bvaluations
123
0.53
MLLL (btts)8m)))
0.45
0.25
033
0.5
123
352
23.9 2.00
352
182
250
Epoch
Large-Batch Training (batch size 8,000)
Figure 3: Evolution of test errors, test NLLs, and NFEs from the ODE solvers during the training of InfoCNF with learned tolerances InfoCNF with fixed tolerances vs. CCNF (baseline) on CIFARIO 10 using small batch size (top) and large batch size (bottom). Each experiment averaged over 3 runs.
4.4 IMPACTS OF GATING NETWORKS
We study the benefits of using gating networks learn the error tolerances by comparing In- foCNF with learned error tolerances and with error tolerances set to 10-5. When training both models with large batches. InfoCNF with learned tolerances achieves the test error of 4% less than the test сгror attained by the same model with fixed error tolerances. InfoCNF with learned tolerances also yields slightly better NLLs (3.64 vs. 3.67). small-batch training, both models achieve similar test errors and NLLs. Unlike with the test error and likelihood, the InfoCNF with learned tolerances sig- nificantly reduces the NFEs compared InfoCNF with fixed error tolerances when both are trained with small batches-a reduction of 21%. In large-batch training, the NFEs from both models are similar. summary, when InfoCNF with learned tolerances has the advantage over InfoCNF with tolerances set to 50-5, it is notable improvement. Otherwise, InfoCNF with learned tolerances is as good as or only slightly worse than the its counterpart. We summarize the comparison between InfoCNF and CCNF in Table 1
Table Test errors, test NLLs, and NFEs of InfoCNF and CCNE
Test error
NLLs
NFEs
Small-batch
CCNF
33.09
3.511
22..12
InfoCNF with fixed tolerances InfoCNF with learned tolerances
21.50 1:000-10... 0.29
3.533 + 0.005
984.92 1 10.34
0.67
3.568 0.003
7
Large-batch
CCNF
38.14-1 1.14
3.710
660.71 4.70
InfoCNF with fixed tolerances
3.663
3.674 0.002
582.22 10.16
InfoCNF with learned tolerances
27.88
3.638 0.006
589.69 1 11.20
Automatic Tuning vs. Manual Tuning: Figure3d, e, and f, InfoCNF is trained with the manually- tuned ODE solvers' tolerances since otherwise the models are hard to train. Thus, for large-batch training, we are comparing the learned tolerances in InfoCNF with the manually tuned tolerances. Furthermore, Figure 10 in Appendix shows similar comparison for small-batch training. In both experiments, we observe that our automatic approach learns the tolerances which outperform the manually- tuned ones in both classification and density estimation while being only slightly worse in term of NFEs. Also, our automatic approach via reinforcement learning requires much less time and computational budget to find the right values for the tolerances compared to the manual tuning.
7

---

Under review as a conference paper at ICLR 2020
CCNF without large learning rate and tuned tolerance
InfoCNF without large learning rate and tuned tolerance
CCNF
InfoCNF
60
(a)
(b)
904
(c)
004
NILL (altsdin))
5
754
Text
50.
14.
100
250 Epoch
250
350
200
300
351
100
253
250
300
Epoch
Eboch
Figure 5: (a) Test crrors, (b) test NLLs, and (c) NFEs of InfoCNF and CCNF trained on CIFARIO using large batch size with and without large learning rate and manually tuned error tolerance.
Unconditional CNF: Inspired by the success of InfoCNF, we explore whether the same reinforcement learning approach still work for unconditional models. We compare the baseline CNF in (Grathoohl ct al. 2018) with CNF which uses the CNN gates to learn the error tolerance of the ODE solvers in small-bated training setting. While both models yield similar NLLS (3.37 bits/dim), CNF with learned tolerances significantly reduces the NFEs by 15% compared to the baseline CNF (see Figure (
4.5
IMPROVING LARGE- BATCH TRAINING OF INFOCNF AND CCNF
Training neural networks using large batches is much faster than small batches, but suffers from poor generalization (Keskar et al. 2017). In order to conduct meaningful experiments with large-batch training, we study methods to improve the performance of CCNF and InfoCNF. Our experiments confirm that tuning the error tolerance of the ODE solvers, together with using larger learning rate as suggested in (Keskar et al.) 2017] Goyal et al. 2017). helps enhance the performance of both InfoCNF and CCNF, resulting better test error, lower NLLs, and smaller NFEs (see Figure
4.6 CONDITIONING VIA PARTITIONING ON TIME-SERIES DATA
We study the potential benefit of applying the conditioning method via partitioning the latent code in InfoCNF on time-series data. In this ex- periment, we choose the LatentODE (Chen et al.) 2018) as the baseline model and conduct the experiment on the synthetic bi-directional spi- ral dataset based on the one proposed in (Chen et al.) 2018). In particular, a fixed set of 5,000 2-dimensional spirals of length 1,000 ground truth data: 2,500 curves are clockwise and the other 2,500 curves are counter-clockwise. These spirals have different parameters. We then randomly sample one spiral of 200 equally-spaced time steps from each of these ground truth spirals. We add Gaussian noise of mean and standard deviation 0.3 these small spirals to form the training set for our experiments. The test set is generated in the similar way. We train the LatentODE with and without our conditioning strategy for trajectory fitting and test the trained models for trajectory fitting and extrapolation on data in test set. More details on the dataset, network architecture, and training details are provided in Appendix [E] We observe that the LatentODF equipped with our conditioning strategy outperforms the baseline Latent ODE on trajectory fitting and extrapolation tasks, especially in unseen domains.
Tres injectery
Leamned trajectory
Sampled data
LatenOnF (o)
LateHOnE Parittioiin
(b)
first generate
Figure 6: Trajectories learned by the LatentODE (a) without and (b) with the conditioning via partitioning the latent code.
5 RELATED WORK
Conditional generative models: NNalinnick et al., 2019) applies a generalized linear model on the latent code of a flow-based generative model to compute both p(x) and p(ylx) exactly. Their method does not consider splitting the latent code and is complimentary to our partitioning approach. The conditioning approach proposed in (Ardizzone et al. 2019) is close to InfoCNF. However, in contrast to this method, InfoCNF does not penalize the mismatch between the joint distribution of the supervised and the unsupervised code p(zy. zn) and the product of their marginal distributions p(z)) InfoCNF also does not minimize the MMD distance between the distribution of the backward prediction against the prior data distribution. Instead, we maximize the likelihood of the encoder to generate the
input image given the label pxxiy) p(z,))))(())) (Liu et al. 2019)
8

---

Under review as a conference paper at ICLR 2020
partitioned latent code for Glow. This encoder is trained by optimizing adversarial loss as in the generative adversarial networks (GANs) [Goodfellow et al. 2014) so that its generated latent codes indistinguishable from the latent codes computed by Glow for real data. Our InfoCNF directly splits the latent code without using an extra complicated architecture like GANs, which might introduce more instability into the ODE solvers and slow down the training.
Adaptive computation: The use of gating networks and reinforcement learning to learn policy for adaptive computation has been studied enhance the efficiency of the neural networks. (Wang et al. 20186 employ gating networks to decide which blocks to skip in residual networks. (Liu et al. 2018) develops reinforcement learning framework to automatically select compression techniques for a given DNN based on the usage demand. Other works including (Graves) 2016 Jernite et al. 2016 Figurnov et al. 2017| Chang et al. 2017) study methods to select the number of evaluations in recurrent and residual networks. To the best of our knowledge, our InfoCNF with learned tolerances is the first that learns the error tolerances of the ODE solvers CNF.
Large-batch training: Various methods have been proposed to improve the generalization of neural networks trained with large batches by scaling the learning rate and scheduling the batch size. Among them are (Keskar et al. 2017; Goyal et al. 2017, Smith et al., 20188 You et al. 2017). These methods are complimentary to our approach of tuning the error tolerances.
CONCLUSIONS
We have developed an efficient framework, namely InfoCNF, for conditioning CNF via partitioning the latent code into the supervised and unsupervised part. We investigated the possibility of tuning the error tolerances of the ODE solvers to speed up and improve the performance of InfoCNF. We invented InfoCNF with gating networks that learns the error tolerances from the data. We empirically show the advantages of InfoCNF and InfoCNF with learned tolerances over the baseline CCNF. Finally, we study possibility of improving large-batch training of our models using large learning rates and learned error tolerances of the ODE solvers.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with
invertible neural networks. In International Conference Learning Reprexentations, 2019. URL
https://cpenrevieew
[DeX
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 340.3332...43 2013.
Christopher M Bishop. Novelty detection and neural network validation. IEE Proceedings-Vise Vision, Image and Signal processing, 141(4))217-22,, 1994.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41()):5,, 2009.
Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert, Multi-level residual networks from dynamical systems view. In International Conference on Learning Representations, 2017. URL https
OgR-
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary diffcrential equations. In Advances in Neural Information Processing Systems, pp. 6571-6583, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances neural information processing systems, pp. 2172-2180, 2016.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In Interna- tional Conference on Learning Representations, 2017. URL RRR/tttps///opeeevllww wwtp:/ewwwwwwwweeeeeeeeeeeeeeeee net net/forum? 1d=
HkponH91x
Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1039-1048, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozais, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680, 2014.
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: /ttl::/700.00000 2017.
Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In International Conference on Learning Representations, 2018.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603. 08983, 2016.
Caglar Gulcehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. Noisy activation functions. In Interna- tional conference on machine learning, pp. 5059-3068, 2016.
M. F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines (corr: V19 p433-450). Communications in Statistics: Simulation and Computation, 18:1059-100 1076, 1989.
Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in recurrent neural networks. arXiv preprint arXiv: 1.06188, 2016.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Rui Liu, Yu Liu, Xinyu Gong, Xiaogang Wang, and Hongsheng Li. Conditional adversarial generative flow for controllable image synthesis. arXiv preprint arXiv: 190400178, 2019.
10

---

Under review as a conference paper at ICLR 2020
Sicong Liu, Yingyan Lin, Zimu Zhou, Kaiming Nan, Hui Liu, and Junzhao Du On-demand deep model compression for mobile devices: usage-driven model selection framework. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys 18, pp. 389 400, New York, NY, USA, 2018. ACM. ISBN 978- 1 1-083----0----00 5720--3.. doi: 10.1 10.114553210244021103377 URL
http:/ / acm. org/10. 11 114511133333 30155/22024000 52337
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807-814, 2010.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Hybrid models with deep and invertible features. In International Conference Machine Learning, 2019.
Athanasios S Polydoros and Lazaros Nalpantidis. Survey of model-based reinforcement leamiing Applications on roboties. Journal of Intelligent & Robotic Systems, 86(2):153 173, 2017.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pp. 1530-1538, 2015.
Yulia Rubanova, Ricky TQ Chen, and David Duvenaud. Latent odes for prrgularly-samplle time series. arXiv preprint arXiv: 039077, 2019.
Samuel L. Smith, Pieter-Jan Kindermans. and Quoe V. Le. Don't decay the learning rate, increase the batch size. In International Conference Learning Representations, 2018. URL httpps://oppereeeeeww net/
Eorum?Id=BLyy
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15: 1929-1958, 2014. URL http: httpppp///mm.rroroopppeerr/////s
4a. htmm
Richard Sutton and Andrew G. Barto. Reinforcement Learnigg: An Introduction. MIT Press, Cambridge. MA. USA, 1998, ISBN 0- 19398-.---. URL/bttp:/wwww wwwberta. wtw.aaaaaaaaaaaaaaaaaaaaattttt.. ebook / the book.
Bao Wang, Binjie Yuan, Zuoqiang Shi, and Stanley J Osher. Enresnct: Resnet ensemble via the feynman-kuc formalism. arXiv preprint arXiv: 1.8881111000 10745,, 2018a.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. Skipnet: Learming dynamic routing in convolutional networks. In Proceddigss of the European Conference on Computer Vision (ECCV), pp- 409-424, 2018b.
Yue Wang, Tan Nguyen, Yang Zhao, Zhangyang Wang, Yingyan Lin, and Richard Baraniuk. Energynet: Energy-efficient dynamic inference. 2018c.
Ronald J Williams. Simple statistical gradicnt-followingg algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-22) 256, 1992.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv: / 708. 03888, 6, 2017.

---

Under review as a conference paper at ICLR 2020
APPENDIX
A GENERATED MNIST IMAGES FROM INFOCNF
We validate InfoCNF on the MNIST dataset. On MNIST, InfoCNF with learned error tolerances, InfoCNF with fixed error tolerances, and the baseline CCNF achieve similar NLLs and test errors. However, the InfoCNF with learned and fixed error tolerances are 1.5x and 1.04x faster than the baseline CCNF, respectively (416 NFEs/epoch 589 NFEs/epoch 611 NFEs/epochs). We include the detailed results Table 2 All experiments are conducted with batch size 900, and the results averaged 3
Table Test errors, test NLLs, NFEs of InfoCNF and CCNF MNIST.
Test error
NLLs
NFEs
Small-batch
CCNF
0.02
1.016
611 7
InfoCNF with fixed tolerances
0.60
1.030
589
InfoCNF with Iearned tolerances
0.61 0.02
1.018
416
Below is the MNIST images generated by InfoCNFwith learned error tolerances.
0
2
3
4
5
6
7
8
9
0
/
2
3
4
5
6
7
g
9
0
7
3
y
5
4 6 6
7
0
9
0 0(13
1
3
4
5
7
8
9
0 1
0123
3
4
5
7
8
9
0
2
3
4
5
6
7
8
9 9
0 1
3
4
5
6
1
8
D
1
1
3
4
5
6
7
8
9 9
0
0121
1
2
3
4
5
6
7
9
0
0
2
3
3
6
7
8
9
Figure
MNIST images generated by InfoCNF.
B NETWORK ARCHITECTURE FOR EXPERIMENTS ON CIFAR10
For experiments on 10, we use the FFJORD multiscale architecture composed of multiple flows as (Grathwohl et al. 2018) for our experiments. The network uses scale blocks, each of which contain 2 flows, a "squezze" operator, and then other 2 flows. Each flow is made of convolutional layers with 64 filters whose kernel size is 3. The squeeze operators are applied between flows to down-sample the spatial resolution of the images while increasing the number of channels. We apply the softplus nonlinearity at each layer. This architecture tries to reduce the dimensionality of the latent representation each level while preserving invertibility. It based the multiscale architecture (Dinh et al., (017)), which has been widely used as the base architecture for invertible models. Also, parameterize de and qo by separate linear networks.
C SYNTHETIC 1-D USED TO ESTIMATE THE VALUE OF ERROR TOLERANCES OF THE ODE SOLVERS AT TEST TIME
order to estimate the value of error tolerances the ODE solvers which yields negligible numerical errors during test time, train InfoCNF 1-D synthetic dataset and calculate the area under the curve at different error tolerance values starting from the machine precision of 10 Examples in this dataset are sampled from the mixture of Gaussians shown by
As mentioned in Section 4.2
12

---

Under review as a conference paper at ICLR 2020
the blue curve in Figure The orange curve in Figure represents the distribution learned by
InfoCNF when the error tolerances are set to 10-3
Groundtruth distribution
Learned distribution
M
M
0.15
0.10
0.05
0.00
Figure 8: Distribution from which we sample the 1-D synthetic data for computing the value of error tolerances used to evaluate the trained InfoCNF (blue curve) and the distribution learned by InfoCNF (orange curve)
D EVALUATING THE TRAINED INFOCNF USING THE LEARNED ERROR TOLERANCES
We explore if the error tolerances computed from batches of input data can be used to evaluate the trained InfoCNF. First, repeat the experiment 1-D synthetic data described in Section and and Appendix C above using the learned error tolerances instead the fixed error tolerances. We observe that the numerical error in this case is 0.00014, which is small enough. We further use the learned eтror tolerances to evaluate our trained InfoCNF on CIFAR 10 with small-batches. Distribution of these error tolerances different layers be found in Figure Zp. The test and NLL we obtain are 20.85 1.48 and 3.566 +: 0.003, which are close enough to the results obtained when setting the error tolerances 10 (test eπтоr and NLL in this case are 20.99 0.67 and 3.568 + 0. 0.003, respectively, as shown in Table Furthermore, when using the learned eтror tolerances to evaluate the trained model, we observe that the trained InfoCNF achieves similar classification errors, negative log-likeli lielliielihoods (NLLs), and number of function evaluations (NFEs) with various small values for the batch size (e.g. 1, 5000, 900, 1000, 2000). However, when we use large batch sizes for evaluation (e.g. 4000, 6000, 8000), those metrics get worse. This sensitivity to test batch size is because the error tolerances InfoCNF are computed for each batch of input data.
E DATASET, NETWORK ARCHITECTURES, AND TRAINING DETAILS FOR EXPERIMENTS ON SYNTHETIC TIME-SERIES DATA
E.1 DATASET
Bi-directional Spiral Dataset of Varying Parameters: In the experiments time-series data in Section (46) we use the synthetic bi-directional spiral dataset based on the one proposed in (Chen first generate fixed set of 5,000 2-dimensional spirals of length 1,000
et al. 2018). In particular, as ground truth data: 2,500 curves are clockwise and the other 2,500 curves are counter-clockwise These spirals have different parameters. The equations of the ground truth spirals are given below:
Clockwise: R= R-atbx 50
R cos(r) 55; y- sin(r)
(8)
Counter-Clockwise: - x R cos(r) +5; y= sin(r).
Here a and serve as the system parameters and are sampled from the Gaussian distributions (1110, (1.0..00 0.08) and (0.25, 0.00). respectively.
13

---

Under review as a conference paper at ICLR 2020
12.5
clock
10.0
5.0
2.5
0.0
-2.5
-5.0
-10
-5
10
Figure 9: Ground truth spirals: One spiral is clockwise and another is counter-clockwise.
We then randomly sample one spiral of 200 equally-spaced time steps from each of these ground truth spirals. We add Gaussian noise of mean 0 and standard deviation 0.3 to these small spirals to form the training set for our experiments. The test set is generated in the similar way.
E.2 NETWORK ARCHITECTURES
In our experiments, the baseline is the LatentODE model (Chen et al., 2018). The RNN encoder that estimates the posterior 4o (Z) > " Xxx) from the observations is fully connected and has 25 hidden units. The latent state are 5-dimensional vectors. Different from the LatentODE, in our model, the first three dimensions of the initial latent state 2, are for the supervised code Z, and the other two dimensions are for the unsupervised code zu. Like the LatentODE, given the initial latent state Zio our model computes the future states Zi -...... .Zny by solving the equation az(1)/Or- = f (z((1, 0f). The dynamic function f is parameterized with a one-hidden-laye network of 20 hidden units. Also, the decoder that reconstructs & from Z, is another one-hidden-laye network of 20 hidden units. Additionally. in model, the conditioning function Oe and the supervised function ga are linear networks. our experiments, the conditioning/superiisce signal y are the values of the parameters b, and the direction of the spiral (i.e. clockwise counter-clockwee
E.3 TRAINING DETAILS
We train the model with the Adam optimizer (Kingma & Ba 2015). We use batch training with the learning rate of 0.001. The training is run for 20,000 epochs.
F AUTOMATIC TUNING VS. MANUAL TUNING IN SMALL-BATCH TRAINING SETUP
We show the test error, NLLs, and the NFEs of InfoCNF with manually-tuned error tolerances for small-batch training on CIFAR10 10 (the yellow curves) the Figure 10 in comparison with the results from CCNF, InfoCNF, and InfoCNF with learned tolerances in the main text. As can be seen, InfoCNF with learned tolerances, which learns the error tolerances, still as good as InfoCNF with manually-tuned error tolerances (except for the slightly worse NFEs) when trained with small batches.
G MARGINAL LOG-LIKELIHHOO RESULTS ON CIFARIO
The NLLs discussed in the main text are the conditional negative log wiw-liikklhosss--o log p(xly) We would also like to compare the marginal negative log-like log-likelsoooss - logg p(x) of CCNF, InfoCNF, and InfoCNF with learned tolerances. Figure TT shows that like in the case of the conditional NLLs, InfoCNF with learned tolerances results in better marginal NLLs in large-batch training but slightly worse NLLs in small-batch training compared to InfoCNF and CCNF.
Improving Large-Batel Training of InfoCNF and CCNF: We would like to understand if using large learning rate and tuned error tolerances of the ODE solvers help improve the marginal NLLs in large-batch training. Figure confirms that both InfoCNF and CCNF trained with large learning
14

---

Under review as a conference paper at ICLR 2020
InfoCNF
CCNF (baseline)
InfoCNF fixed tolerances
manual-tuned tolerances
InfoCNF
learned tolcrances
A
(b)
(c)
6.5
NLL (8ttsomm
1000
640
0.35
030
630
Epach
Epoch
Epoch
Figure 10: Evolution of test classification errors, test negative oog-likelhhood, and the number function evaluations in the ODE solvers during the training of InfoCNF with Iearned tolcrances (blue) vs. InfoCNF with manually-tuner tundd error tolerance (yellow) vs. InfoCNF with tolcrances=100 (green) vs. CCNF (red, bascline) on CIFAR1O 10 using small batch size. Each experiment is averaged over 3 runs.
CCNF (baseline)
InfoCNF
InfoCNF learned tolerances
fixed tolerances
(b)
-logp(x) (bits:dim)
4.2
(bitsydim)
6.2
4.0
6.1
0.0
3.8
3.
a.
1.00
1.5
200
250
50 1.50
200
200
350
Epoch
Epach
Figure 11: Evolution test marginal negative log-likelihoods during the training of InfoCNF with learned
tolerances (blue) vs. InfoCNF with fixed tolerances (green) vs. CCNF (red, bascline) on CIFAR 10 using small batches of size 900 (left) and large batches of size 8,000 (right). Each experiment is averaged over 3 runs.
rate and tuned error tolerances yield much better marginal NLLs in large-batch training compared to the baseline training method which uses small learning rate and constant error tolerances.
CCNF without large learning rate and tuned tolerance
CCNF
InfoCNF without large learning rate and tuned tolerance
InfoCNF
5.0
x) (bits/dim)
4.6
4.4
4.2
3.6
100
150
200
350
Epoch
Figure 12: Test marginal NI.Ls InfoCNF and CCNF trained on CIFARIO using large hatch size with and without large learning rate and manually-tuned error tolerance.
H LARGE-BATCH TRAINING vs. SMALL-BATCH TRAINING
Given promising results from training our models with large batches, would like to compare large-batch with small-batch training in term of speed. Currently, the CNF-based models studied in paper yield better test classification errors and negative log-likelihoods CIFAR1O 10 when trained
15

---

Under review as a conference paper at ICLR 2020
with small batches than with large batches. However, since large-bated batch training attains smaller NFEs than small-batch training, we would like to explore the model trained with large batches can reach certain test classification errors and NLLs faster than the same model trained with small batches. In our experiments, we choose to study InfoCNF with learned tolerances since it yields the best test error and NLLS in large-batch training while requiring small NFEs. We compare InfoCNF with learned tolerances trained with large batches and with small batches. We also compare with the baseline CCNF trained with small batches. Figure 13 shows that InfoCNF with learned tolerances trained with large batches achieves better test error than CCNF trained with small batches while converging faster. However, still lags behind CCNF trained with small batches term of NLLs and InfoCNF with learned tolerances trained with small batches in both test error and NLLs. This suggests future work for improving large-batch training with the CNe-based models.
InfoCNF learned tolerances (small batch)
InfoCNF
CCNF (baseline)
learned tolerances (large batch)
0.8
a
0.7
0.5
Test
0.3
0.2
50000
100000
150000
200000
250000 300000
# Function Evaluations
5.0
(b)
4.8
NLL (bits/dim)
4.6
4.0
3.8
50000
100000 150000 200000 250000 300000 # Function Evaluations
Figure 13: (a) Test error and (b) NLLs vs. NFEs during training of CNF and InfoCNF (with learned tolerances) on CIFARIO.
I INFOCNF WITH LEARNED TOLERANCES FACILITATES TRAINING OF LARGER MODELS
Since InfoCNF with learned tolerances reduces the NFEs while improving the test error and NLL, we would like explore the possibility of using InfoCNF with learned tolerances for training larger models to gain better performance in classification and density estimation. We train InfoCNF with learned tolerances with flows per scale block (compared model) CIFARIO using large batches and compare the results with the original InfoCNF with learned tolerances and the baseline CCNF, Here we follow the same notation as in (Grathwohl et al.) 2018) to describe the models. We call the large InfoCNF with learned tolerances with 4 flows per scale block the 2x- Ix-InfoNNF with learned tolerances. Figure [4 shows that the 2x-InfoCNF with learned tolerances yields better test errors and NLLS while increasing the NFEs compared both InfoCNF and CCNE.
flows per scale block in the original
16

---

Under review as a conference paper at ICLR 2020
InfoCNF leamed tolerunces
InfoCNF with 2x flows per scale block
CCNF (baseline)
learned tolerances
(b)
(c)
Furciion Leekoo
120e
(btt/dim)
845
025
700
630
625
Epoch
Epoch
Epoch
Figure 14: Evolution of test classification errors, test negative oogllikelhhodds, and the number function evaluations in the ODE solvers during the training of the large size InfoCNF with 2x more flows per scale block and learned tolerances (brown) vs. InfoCNF with learned tolerances (blue) vs. CCNF (red, baseline) on CIFAR1O 10 using large batchs of size 8,000.
17


Reviews:
[{"review_id": "r1eoUaJ0KB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines the problem of extending continuous normalizing flows (CNFs) to conditional modeling. The authors propose a model, InfoCNF, which models a latent code split into two partitions: one that is unique to a class, and another that is more general. InfoCNF relies on the accuracy of ODE solvers, so the paper also proposes a method that learns optimal error tolerances of these solvers. They perform experiments on CIFAR10, showing that InfoCNF outperforms a baseline in accuracy and negative log-likelihood. They also ablate through experiments that demonstrate the utility of learning the ODE error tolerances. \n\nCNFs are an exciting tool, and it's an important problem to devise methodology that applies CNFs to conditional modeling tasks. The idea of splitting latent codes into two separate components -- one supervised, the other more general -- is interesting. And the approach to learning error tolerances is a good idea.\n\nThe main drawback of this paper is the lack of clarity. It is poorly written and the presented model is not clearly motivated. Even after reading through the paper multiple times, I find it difficult to understand various aspects of InfoCNF. Below are some examples of this lack of clarity:\n\n- When motivating Conditional CNF (CCNF), the details for training the model are unclear. What is the loss function, and how does it balance between modeling x and learning the auxiliary distribution? Although these may seem like small details, since InfoCNF builds off on CCNF, it is crucial to solidify an understanding of the training procedure and how the auxiliary task relates to modeling x. Moreover, the cited reference (Kingma and Dhariwal, 2018) does not contain these details (and there is no mention of object classification in the paper, contrary to the claim on page 3). It would be helpful to cite other references when mentioning that this approach is widely used. \n\n- The definition of InfoCNF is unclear. The variable y has been used to denote the image label, so why are there now L latent variables y_1, ..., y_L? The following terms of equation 4 are undefined: L_{NLL}, L_{Xent}, and y_hat. Although some readers would be able to understand these definitions from context (flow-based NLL, cross entropy loss, and the logits provided by the auxiliary distribution q), they are never explicitly defined and result in confusion and ambiguity. Most importantly, p(x|z,y) is never made explicit; although one can infer from context that  is transformed to x using CNFs, it is never made explicit in the definition (and that these flows only appear in L_{NLL}). Overall, the motivation for splitting the latent code into two pieces is not clearly explained, and the paper should spend more time arguing for this.\n\nThe paper compares InfoCNF to a single baseline (CCNF). I understand that the paper is proposing the first method that combines CNFs with conditional modeling, but there are plenty of non-CNF flow-based conditional approaches that could've been compared. The paper goes over these approaches in Section 5 and discusses their differences with InfoCNF, but these are never experimented with. It seems possible that these models could be extended in a straightforward manner to use CNFs instead of other normalizing flows. Even comparing with these models without CNFs would have been interesting. I think that using a single baseline, instead of applying a more complete set of comparisons, hurts the persuasiveness of their method.\n\nIn addition to comparing to a single baseline, the paper only compares for a single dataset (CIFAR10). A more convincing experiments section would compare against more models on more than a single dataset.\n\nThe paper proposes a promising idea to an important problem. Due to the lack of clarity throughout the paper and incomplete comparisons, I would argue to (weakly) reject.  ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJgxY5YRFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a conditional CNF based on a similar intuition of the InfoGAN that partitions the latent space into a class-specific supervised code and an unsupervised code shared among all classes. To improve speed, the paper further proposed to employ gating networks to learn the error tolerance of its ODE solver. The experiments are performed on the CIFAR-10 dataset and synthetic time-series data.\n\nThe paper has addressed an important issue of investigating efficient conditional CNF. The general idea of the paper is clear, but I found certain parts can be improved, such as the formulation of InfoCNF. It seems the authors assume readers know InfoGAN well enough, which might not be the case. \n\nMy main concern is the limited evaluation as all the experiments are performed on the CIFAR-10 and synthetic data. Since the paper address efficient conditional CNF, it would make the claim much stronger if more experiments could be performed on larger images: if not the original imagenet, maybe imagenet-64 or imagenet-128 or image benchmarks with higher resolutions.\n\nWhy does InfoCNF achieve slightly worse NLL in small batch training, while it outperforms CCNF in all the other metrics? Do you have any explanations?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxVhmb6cH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper suggests a methodology of partitioning latent code to a set of class-specific codes, to solve the inefficiency from the large size of the latent code in conditional normalizing flow. This inefficiency problem has been a weak point in the existing conditional normalizing flow models. Also, this work addresses the increase of the number of function evaluations which is caused by code partition, by optimizing the error tolerance with a reinforcement learning approach.\n\nThis paper has a novel contribution, outperforms the baseline (CCNF: CNF (Chen et al.) + GMM / auxiliary classifier (Kingma & Dhariwal, 2018)) on several evaluation metrics. I agree to accept this paper, but I vote for ‘weak accept’ because of the following weaknesses:\n\n1. The (three) contributions described by the authors seem to be somewhat exaggerated.\n- For the second contribution, the authors’ way is quite similar to Wang et al. (SkipNet). For both studies, the purpose is the efficiency of certain neural architectures and training scheme is hybrid reinforcement learning with REINFORCE (but with different reward design).\n- For the last contribution, I think it is a minor contribution comparing to two first bullets and overlapping with the second one.\n\n2. There is a lack of explanation to support the efficiency of the proposed model.\n- The authors claim that InfoCNF needs fewer parameters than the baseline. Then, why didn't you show the actual number of parameters? The only line that I found was “... InfoCNF requires 4% less parameters than CCNF. (Section 2)”.\n- Also, it would be better if there were a direct computation and comparison between the size of InfoCNF and CCNF.\n- Finally, is there any constraint on the length of merged latent code z? Since InfoCNF is also an invertible model, it should have the same size as the input, but I cannot find descriptions about it.\n\n3. It is skeptical that automatic tuning of error tolerance has achieved its original purpose.\n- For me, it is questionable whether automatic tuning has achieved its original purpose: reducing NFEs for better speed (and performance).\n- In figure 3c and 3f, we can find NFE of InfoCNF learned tolerances eventually exceeds the NFE of InfoCNF fixed tolerance. It looks like learning tolerance increases NFE in large epochs, and the timing seems to depend on the batch size. If so, the batch size is a really important hyper-parameter in this framework, how can we determine the appropriate size?\n- In section 4.4 (Automatic Tuning vs. Manual Tuning), the authors state that “our automatic approach learns the tolerances which outperform the manually-tuned ones in both classification and density estimation while being only slightly worse in term of NFEs.”. But as abstract says, is reducing NFEs the original goal of automatic tuning?\n- Lastly, how can we know/confirm “our automatic approach via reinforcement learning requires much less time and computational budget”. I cannot see any explanation about this claim.\n\n4. Minor things.\n- What is the stage stated in Figure 1?\n- The abbreviation of the CCNF first appears in section 1 but its full name first appears in section 2.2.\n- In figure 3a and 3b (and table 1), why is test NLLs of CCNF lower than InfoCNF’s where test error of CCNF is larger than InfoCNF’s with high margin? Is there any possible explanation?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxHzbIT5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a conditioning approach for CNF and explore speed-up by tuning error tolerance of the ODE solvers.\n \nOverall, this is a mediocre paper which directly use a similar structure introduced in InfoGAN but absolutely lose beautiful insights in the construction of the loss function in InfoGAN, i.e., the mutual information between the generated image and codes. In other way, this paper is just an incremental paper with even less insights than the seminal paper. \nAt least in the loss function Eq. (4), I didn't see any mutual information regularization used here. Instead, the authors use a GMM but I am totally not sure why a GMM is better than the mutual information regularization. At the same time, in equation (4), I didn't see the specific definition of L_xent and L_NLL and thus I am not even able to verify that the use of the loss function is correct. For the current version, I am not be able to gain any insight from the loss function. It seems to be an ensemble of several existing works and definitely not innovative. \n\nFor the tuning error of the ODE solvers, I didn't even see the problem statement.  Is it possible to make the problem more clear? It seems that the first time when the authors mentioned error tolerance is in contribution 2, but I didn't see the definition of the error tolerance. I am not sure whether it is a good idea to introduce two problems in one paper. At the same time, I do not know why the problem should be formulated in the form of the reinforcement learning problem. I didn't see any advantage.  Intuitively, learning a generative model can be time consuming  and solving a reinforcement learning problem is also hard. I do not understand why combining them together would be beneficial and even time efficient? For me, it seems that authors are just trying to make the problem unnecessarily more complicated and thus they can use fancy tools to solve it.  \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a conditional CNF based on the InfoGAN structure to improve ODE solvers. Reviewers appreciate that the approach shows improved performances over the baseline models. 

Reviewers all note, however, that this paper is weak in clearly defining the problem and explaining the approach and the results. While the authors have addressed some of the reviewers concerns through their rebuttal, reviewers still remain concerned about the clarity of the paper.

I thank the authors for submitting to ICLR and hope to see a revised paper at a future venue.
Decision: Reject