Sparse Flows: Pruning Continuous-depth Models
Authors: Lucas Liebenwein
Ramin Hasani
Alexander Amini
Daniela Rus
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative modeling
neural ODEs
continuous normalizing flows
pruning
sparsity
compression
generalization

Abstract: Continuous deep learning architectures enable learning of flexible probabilistic models for predictive modeling as neural ordinary differential equations (ODEs), and for generative modeling as continuous normalizing flows. In this work, we design a framework to decipher the internal dynamics of these continuous depth models by pruning their network architectures. Our empirical results suggest that pruning improves generalization for neural ODEs in generative modeling. We empirically show that the improvement is because pruning helps avoid mode-collapse and flatten the loss surface. Moreover, pruning finds efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy. We hope our results will invigorate further research into the performance-size trade-offs of modern continuous-depth models.

Sparse Flows: Pruning Continuous-depth Models
Lucas Liebenwein'
Ramin Hasani' MIT CSAIL rhasani &mit.edu
Alexander Amini MIT CSAIL amini tmimi edu
Daniela Rus MIT CSAIL rus@csail mit.iil.i.eeei...
MIT CSAIL Jucas@csail.mtn mit.edu edu
Abstract
Continuous deep learning architectures enable learning of flexible probabilistic models for predictive modeling as neural ordinary differential equations (ODEs), and for generative modeling as continuous normalizing flows. this work, we design a framework to decipher the internal dynamics of these continuous depth models by pruning their network architectures. Our empirical results suggest that pruning improves generalization for neural ODEs in generative modeling. We empirically show that the improvement is because pruning helps avoid mode- collapse and flatten the loss surface. Moreover, pruning finds efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy. We hope our results will invigorate further research into the performance-size trade-off's of modern continuous-deptt models.
1 Introduction
The continuous analog of normalizing flows (CNFs) (Chen et al., 2018) efficiently (Grath- wohl et al., 2019) maps a latent space to data by ordinary differential equations (ODEs), re- laxing the strong constraints over discrete nor- malizing flows (Dinh et al., 2016, Durkan et al.,
Segative Log Likellhood
Pruring Neural ODEs improvess generolization
MAF
SOS
3.7
Pruning
FFOORD
RO-NSF
Glow
CP
2019, Huang et al., 2020, Kingma and Dhari- wal, 2018, Papamakarios et al., 2017, Rezende and Mohamed, 2015). CNFs enable learning flexible models by unconstrained neural net- works. While recent works investigated ways to improve CNFs' efficiency (Finlay et al., 2020, Grathwohl et al., 2019, Li et al., 2020), regu- larize the flows (Onken al., 2020, Yang and Kariiadakis, 2020), or solving their shortcomings such as crossing trajectories (Dupont et al., 2019, Massaroli et al., 2020), less understood about their inner dynamics during and post training.
Res Flow
50k
100k
300k
1.2M
3M
7M
20M
40M
100M
Number of Parameters
Figure 1: Pruning Neural ODEs improves their generalization with at least order of magnitude less parameters. CIFAR-10 density estimation. Values and methods are described in Table 3.
this paper, we set out to use standard pruning algorithms to investigate generalization properties of sparse neural ODEs and continuous normalizing flows. In particular, we investigate how the inner dynamics and the modeling performance of a continuous flow varies we methodologically prune its neural network architecture. Reducing unnecessary weights of neural network (pruning) (Han et al., 2015b, Hassibi and Stork, 1993, LeCun et al., 1990, Li et al., 2016) without loss of accuracy results in smaller network size (Hinton et al., 2015, Liebenwein al., 2021a,b), computational efficiency (Luo et al., 2017, Molchanov et al., 2019, Yang et al., 2017), faster inference (Frankle and Carbin, 2019), and enhanced interpretabilityy (Baykal al., 2019a.b. Lechner et al., 2020a, Liebenwein et al., 2020). Here, our main objective to better understand CNFs' dynamics density estimation tasks as we increase network sparsity and show that pruning can improve generalization in neural ODEs.
denotes authors with equal contributions. Code: https://githuu: http:////ttttcccmmaaaaaa///rraaa
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Pruning improves generalization in neural ODEs.
Our results consistently suggest that certain ratio of pruning of fully connected neural ODEs leads to lower empirical risk density estimation tasks, thus obtaining better generalization. We validate this observation on large series of experiments with increasing dimensionality. See example here in Figure 2.
0.80
Loss (NLL)
0.75
0.70
0.65
0.60
Pruning flattens the loss surface of neural ODEs. Ad- ditionally, we conduct Hessian based empirical investi- gation on the objective function of the wnws-under-test in density estimation tasks to better understand why pruning results in better generalization. We find that for Neural ODEs, pruning decreases the value of the Hessian's 's eigen- values, and as a result, flattens the loss which leads better generalization, c.f., Keskar et al. (2017) (Figure 3).
20
40
60
8D
Prune Ratio
Figure 2: Pruning enhances generaliza- tion of continuous-depth models. Struc- tured pruning (green), unstructured prun- ing (blue). More details in Section 4.
Pruning helps avoiding mode-collapse in generative modeling. In series of multi-modal density estimation tasks, we observe that densely connected CNFs often get stuck in sharp local minimum (See Figure 3) and as result, cannot properly distinguish different modes data. This phenomena is known as mode-collapse sparsify the flows, the quality of the density estimation increases significantly and consequently mode-collapse does not occur.
Test loss
Train
Once we
Flat minimum
Sharp
minimum
Figure 3: Flat minima result better generalization compared to sharp min- ima. Pruning neural ODEs flattens the loss around local minima. Figure is re- produced from Keskar et al. (2017).
Pruning finds minimal and efficient neural ODE rep- resentations. Our framework finds highly optimized and efficient neural ODE architectures via pruming. In many instances, we can reduce the parameter count by 70-98% (6x-50x compression rate). Notably, one cannot directly train such sparse and efficient continuous-deper models from scratch.
Background
In this section, we describe the necessary background construct our framework. We show how to perform generative modeling by continuous depth models using the change variables formula.
Generative modeling via change of variables. The change of variables formula uses an invertible mapping f IR." + IR," to wrap normalized base distribution p= (z). to specify a more complex distribution. In particular, given ~ pz (z), a random variable, the log density for function f(z) = can be computed by Dinh al. (2015):
log Pa (x) = log (z) - log det 0f(z) Oz
(1)
where af(x)
-3 is the Jacobian of f. While theoretically Eq. demonstrates simple way to finding the log density, from a practical standpoint computation of the Jacobian determinant has a time complexity of O(D²). Restricting network architectures can make its computation more tractable. Examples include designing normalizing flows (Berg et al., 2018, Papamakarios et al., 2017, 1,1,,0 Rezende and Mohamed, 2015), autoregressive transformations (Durkan et al., 2019, Jaini et al., 2019, Kingma et al., 2016, Müller et al., 2019, Oliva et al., 2018a, 8a, Wehenkel and Louppe, 2019), partitioned trans- formations (Dinh et al., 2016, Kingma and Dhariwal, 2018), universal flows (Kong and Chaudhuri, 2020, Teshima et al., 2020), and the use of optimal transport theorem (Huang et al., 2020).
Alternative to these discrete transformation algorithms,, one can construct generative model similar to (1), and declare f by a continuous-time dynamics (Chen et al., 2018, Grathwohl et al., 2019, Lechner et al., 2020b). Given sample from the base distribution, one can parametrize an ordinary differential equations (ODEs) by function fz((1), I, 1,0), and solve the ODE to obtain the observable data. When f is neural network, the system called neural ODE (Chen et al., 2018).

---

Neural ODEs. More formally, neural ODE is defined by finding the solution to the initial value problem (IVP): @x(() (z((t,,, z(fo) Zo. with Zu Pzo (Zo). get z(t,.) the desired output observations terminal integration step 7 (Chen al., 2018).
Continuous Normalizing Flows. If z(t,,. ) is set to our observable data, given samples from the base distribution 2, Psp (Zo), the neural ODE described above forms continuous normalizing flow (CNF). CNFs modify the change in log density by the left hand-side differential equation and as a result the total change in log-density by the right hand-side equation (Chen al., 2018, Grathwohl et al., 2019):
log p(z(t)) -Tr 0l
0f ) logp(z((t.) = log p(z(()) Oz((1)
ts Tr S( to
0f Oz(()
dt.
(2)
The system of two differential equations (the neural ODE (z(t) = (()((), 0)) and (2) can then be solved by automatic differentiation algorithms such as backpropagation through time (Hasani et al., 2021, Rumelhart et al., 1986) the adjoint sensitivity method (Chen et al., 2018, Pontryagin, 2018). Computation of Tr Zx(t)
costs O(D2). A method called the Free-form Jacobian of Reversible Dynamics (FFJORD) (Grathwohl et al., 2019) improved the cost to O(D) by using the Hutchinson's trace estimator (Adams al., 2018, Hutchinson, 1989). Thus, the trace of the Jacobian can be estimated by: Tr (z(1) = Epte) e" a _f_e] क where p(e) is typically set to a Gaussian or Rademacher distribution (Grathwohl al., 2019). Throughout the paper, investigate the properties of FFJORD CNFs by pruning their neural network architectures.
os
3 Pruning Neural ODEs
We enable sparsity in neural ODEs and CNFs by removing, i.e., pruning, redundant weights from the underlying neural network architecture during training. Pruning can tremendously improve the parameter efficiency of neural networks across numerous tasks, such as computer vision (Liebenwein al., 2020) and natural language processing (Maalouf et al., 2021).
3.1 A General Framework for Training Sparse Flows
Our approach to training Sparse Flows is inspired by iterative learning rate rewinding, recently proposed and broadly adopted pruning framework as used by Liebenwein et al. (2020), Renda et al. (2020) among others.
In short, our pruning framework proceeds by first training an unpruned, i.e., dense network to obtain warm initialization for pruning. Subsequently, we proceed by iteratively pruning and retraining the network until we either obtain the desired level of sparsity, i.e., prune ratio or when the loss for a pre-specified hold-out dataset (validation loss) starts to deteriorate (early stopping). We note that our framework readily applicable to any continuous-deple model and not restricted FFJORD-like models. Moreover, we can account for various types of pruning, i.e unstructured pruning of weights and structured pruning of neurons or filters. An overview of the framework provided in Algorithm and we provide more details below.
3.2 From Dense to Sparse Flows
TRAIN a dense flow for a warm initialization. To initiate the training process, we first train a densely-connected network obtain warm initialization (Line 2 of Algorithm 1). We Adam with a fixed step learning decay schedule and weight decay in some instances. Based on the warm initialization, start pruning the network.
PRUNE for Sparse Flow. For the prune step (Line 5 of Algorithm 1) we either consider unstructured or structured pruning, i.c., weight neuron/filter pruning, respectively. At fundamental level,
"One can design more expressive representation (Hasani et al., 2020, Vorbach et al., 2021) of continuous- depth models by using the second-orde approximation of the neural ODE formulation (Hasani et al., 2021). This representation might give rise better neural flows which will be the focus of our continued effort.
3

---

Algorithm SPARSEFLOW(S 4 Purin PR. e)
Input: f: neural ODE model with parameter set 0: Prain hyper- parameterss for training; PR: relative prune ratio; e: number of training epochs per prune-cycle.
Output: [(5 0): Sparse Flow; mn sparse connection pattern.
I: Oo RANDOMINIT(
Initial training stage with dense neural ODE ("warm start"). 3: <- 1 > Initialize binary mask indicating neural connection pattern.
2: 8 <- TRAIN(GU, Puain: e)
4: while validation loss of Sparse Flow decreases do
5: m + PRUNE(M @ 0, PR)
> Prune PR% of the remaining parameters and update mask.
6: 0 < TRAIN(m @ 0, & train, e)
Restart training with updated connection pattern.
7: end while
8: 0 < m @ and return f(:0), m
unstructured pruning aims at inducing sparsity into the parameters of the flow while structured pruning enables reducing the dimensionality of each flow layer.
For unstructured pruning, we use magnitude prun- ing (Han et al., 2015a)), where prune weights across all layers (global) with magnitudes below a pre-defined threshold. For structured pruning, we the fi-norm of the weights associated with the neu- ron/filter and prune the structures with lowest norm for constant per-layer prune ratio (local) as proposed by Li et al. (2016). See Table for an overview.
Table I: Pruning Methods.
Unstructured (Han et al, 201
Structured (1a al. 2016)
Target Score
Weights Wayl
Neurons WWlll Local
Scope
Global
TRAIN the Sparse Flow. Following the pruning step, we re-initiate the training with the new sparsity pattern and the unpruned weights (Line 6 of Algorithm 1). Note that we do not change the training hyperparameters between the different stages training.
ITERATE for increased sparsity and performance. Naturally, we can iteratively repeat the PRUNE and TRAIN step to further sparsify the flow (Lines 4-7 of Algorithm 1). Moreover, the resulting perrstt-performmcce trade-off is affected by the total number of iterations, the relative prune ratio PR per iteration, and the amount training between PRUNE steps. We find that good trade-off to keep the amount of training constant across all iterations and tune it such that the initial, dense flow is essentially trained to (or close to) convergence. Depending on the difficulty of the task and the available compute resources we can then adapt the per-iteration prune ratio PR. Note that the overall relative sparsity after n iterations is given by (1 - PR)". Detailed hyperparameters and more explanations for each experiment are provided in the supplementary material.
4 Experiments
We perform diverse set of experiments demonstrating the effect of pruning the generalization capability of continuous-depth models. Our experiments include pruning ODE- flows in density estimation tasks with increasing complexity, well pruning neural-ODEs in supervised inference tasks. The density estimation tasks were conducted on flows equipped with Free-form Jacobian of Reversible Dynamics (FFJORD) (Grathwohl al., 2019), using adaptive ODE solvers (Dormand and Prince, 1980). We used two code bases (FFJORD from Grathwohl et al. (2019) and TorchDyn pruning framework."
(Poli al., 2020a)) over which we implemented
Baselines. In complex density estimation tasks we compare the performance of Sparse Flows to a variety of baseline methods including: FFJORD (Grathwohl et al., 2019), masked autoencoder density estimation (MADE) (Germain et al., 2015), Real NVP (Dinh et al., 2016), masked autoregressive flow (MAF) (Papamakarios et al., 2017), Glow (Kingma and Dhariwal, 2018), convex potential flows (CP-Flow) (Huang et al., 2020), transformation autoregressive networks (TAN) (Oliva et al., 2018b), neural autoregressive flows (NAF) (Huang al., 2018), 8), and sum-of- -uuuor-squares polynomial flow (SOS) (Jaini et al., 2019).
All code and data are available ouline at: https://githut: httpss//gttaabb.om///sssseeeeeee http://ghhab.cococcccaaaaee/eere

---

Unstructured Pruning
Structured Pruning
1.8
1.35
0.75
Loss (NLL)
1.7
0.70
1.25
Loss
1.20
0.65
0.60
40
60
80
20
40
80
20
40
60
80
Prune Ratio (%))
Prune Ratio (%)
Prune Ratio (%)
(a) Gaussians
(b) Gaussian Spiral
(c) Spiral
Figure 4: Negative log likelihood of Sparse Flow as function of prune ratio. In general, structured pruning a more constrained problem as we constrain the type of sparsity. Hence, for almost any pruning experiment we can expect that structured pruning performs worse than unstructured pruning starting at a certain prune ratio.
4.1 Density Estimation on 2D Data
PR=0%
In the first set of experiments, FFJORD on a multi-modal Gaussian distri- bution, multi-model set of Gaussian distri- butions placed orderly on a spiral as well as spiral distribution with sparse regions. Fig-
train
PR-30%
ure (first row) illustrates that densely con-
nected flows (prune ratio 0%) might get stuck in sharp local minima and as result in- duce mode collapse (Srivastava al., 2017a).
Once we perform unstructured pruming, we ob- serve that the quality of the density estimation in all tasks considerably improves, c.f. Fig- 5 (second and third rows). If we continue sparsifying the flows, depending on the task hand, flows get disrupted again.
PR=50%
PR=70%
Therefore, there is a certain threshold for prun- ing flows required to avoid generative model- ing issues such as modc-collapse in contin- uous flows. We validate this observation by plotting the negative log-likelihood loss as a function of the prune ratio in all three tasks with both unstructured and structured prun- ing. As shown in Figure we confirm that sparsity in flows improves the performance of continuous normalizing flows.
PR=90%
Figure 5: Pruning FFJORD (PR= Prune ratio).
We further explore the inner dynamics of the flows between unpruned and pruned networks on the multi-modal case, with the aim of understanding how pruning enhances density estimation performance. Figure represents the vector-field constructed by each flow to model 6-Gaussians independently. We observe that sparse flows with PR of 70% attract the vector-field directions uniformly towards the mean of each Gaussian. In contrast, unpruned flows do not exploit this feature and contain converging vectors between distributions. This how the mode-collapse occurs.
4.2 Density Estimation Real Data Tabular
We scale our experiments to set five real-world tabular datasets (prepared based on the in- structions given by Papamakarios et al. (2017) and Grathwohl et al. (2019)) to verify our empirical observations about the effect of pruning on the generalizability of continuous normalizing flows. Table 2 summarizes the results. We observe that sparsifying FFJORD flows substantially improves

---

PR = 20%
PR = 70%
PR 90%
Unstructured
Pruming
Unpruned
Structured
Pruming
Figure 6: Multi-moda Gaussian flow and pruning. We observe that Sparse Flows attract the vector- field directions uniformly towards the mean of each Gaussian distribution, while an unpruned flow does not exploit this feature and contains converging vectors in between Gaussians. See Supplements Section S2, for a detailed explanation of these observations.
Table 2: Negative test log- likelihood (NLL) in nats of tabular datasets from (Papamakarios et al.,
2017) and corresponding architecture size in number of parameters (#params). Sparse Flow (FFJORD with unstructured pruning) with lowest NLL and competing baseline with lowest NLL are bolded.
POWER
GAS
HEPMASS
MINIBOONE
BSDS300
Model
nats #params
nats #params
nats #params
nats
#params
nats #params
MADE (Genauia et 2015)
3.08
-3.56
20.98
147K
15.59
164K
148.85
621K
Real NVP (Dinh
-0.17
212K
-8.33
216K
18.71
5.46M
13.84
5.68M
113.228
22.3M
MAF (ppumaarii 2017)
-0.24
59.0K
-10.08
62.0K
17.70
1.47M
11.75
1.64M
-155.69
6.21M
Glow (Kirgms and Diarival, 2016
-0.17
-8.15
N/A
18.92
N/A
11.35
-155.07
N/A
CP-Flow dlaarg
-0.52
5.46M
-10.36
2.76M
16.93
2.92M
10.58
379K
-154999
2.15M
TAN (Oiia oti.
-0.60
N/A
-1.06
N/A
13.78
N/A
11.01
N/A
-159.80
N/A
NAF (Huagg 2018)
-0.62
451K
-11.96
443K
15.09
10.7M
8.86
8.03M
-17.733
42.3M
SOS Janii 2019)
-0.60
212K
-11.99
256K
15.15
4.43M
8.90
6.87M
157.48
9.09M
FFJORD (Gmihwahi 2009)
-0.35
43.3K
-8.58
279K
17.53
547K
10.50
821K
128.33
6.70M
-0.45
30K
-10.79
194K
16.53
340K
10.84
397K
145.62
4.69M
-0.50
23K
-11.19
147K
15.82
160K
10.81
186K
-148.72
3.55M
Sparse Flow
-0.53
13K
-11.59
85K
15.60
75K
9.95
32K
-150.45
2.03M
-0.52
10K
-11.77
64K
15.99
46K
10.54
18K
-151.34
1.16M
their performance in all tasks. In particular, we gain up to 42% performance gain in the POWER, 35% in GAS, 12% HEPMASS, 5% in MINIBOONE and 19% in BSDS300.
More importantly. this is achieved with flows with 1 to orders of magnitude less parameters compared other advanced flows. On MINIBOONE dataset for instance, found sparse flow with only 4% of its original network that outperforms its densely-connectee FFJORD flow. On MINIBOONE, Autoregressive flows (NAF) and -qquoresquares models (SOS) which outperform all other models possess 8.03 and 6.87 million parameters. In contrast, we obtain a Sparse Flow with only 32K parameters that outperform all models except NAF and SOS.
Let us now look at the loss versus prune-ratio trends in all experiments to conclude our empirical ob- servations real-worlh tabular datasets. As shown in Figure we observe that pruning considerably improves the performance of flows at larger scale as well.
4.3 Density Estimation Real Data Vision
Next, we extend our experiments to density estimation for image datasets, MNIST and CIFAR10. We observe similar case on generative modeling with both datasets, where pruned flows outperform densely-connected FFJORD flows. On MNIST, a sparse FFJORD flow with 63% of its weights pruned outperforms all other benchmarks. Compared the second best flow (Residual flow), our

---

-0.35
-0.5
17.5
-0.5
17.0
Loss (NL
16.5
Loss
-1.3
-0.5
15.5
20
50
80
20
40
50
30
60 Prune Ratio (%)
50
Prune Ratio (16)
Prune Ratio (96)
(a) Power
(b) Gas
(c) Hepmass
12.0
-150
1005 (NLL)
13.5
-135
Unstructured Pruning
11.0
Structured Pruning
LOSS
10.5
20
40
80
20
40
62
60
Prune Ratio (%)
Prune Ratio (%)
(d) Miniboonc
(c) Bsds300
Figure 7: Negative log- likeliidl versus prune ratio tabular datasets.
Table 3: Negative test log-likelihood (NLL) in bits/dim for image datasets and corresponding architecture size number parameters (#params). Sparse Flow (FFJORD with unstructured pruning) with lowest NLL and competing baseline with lowest NLL are bolded.
MNIST
CIFAR-10
Model
bits/dim
#params
bits/dim
#params
MADE (Germain et al., 2015)
1.41
1.20M
5.80
11.5M
Real NVP (Dinh et al., 2016)
1.05
N/A
3.49
N/A
MAF (Papamakarios et al., 2017)
1.91
12.0M
4.31
115M
Glow (Kingma and Dhariwal, 2018)
1.06
N/A
3.35
44.0M
CP-Flow (Huang al., 2020)
1.02
2.90M
3.40
1.90M
TAN (Oliva et al., 2018b)
1.19
N/A
3.98
N/A
SOS (Jaini et al., 2019)
1.81
17.2M
4.18
67.1M
RQ-NSF (Durkan al., 2019)
3.38
11.8M
Residual Flow (Chen al., 2019)
0.97
16.6M
3.28
25.2M
FFJORD (Grathwohl et al., 2019))
1.01
801K
3.44
1.36M
Sparse Flows (PR=20%)
0.97
641K
3.38
1.09M
Sparse Flows (PR=38%)
0.96
499K
3.37
845K
Sparse Flows (PR=52%)
0.95
387K
3.36
657K
Sparse Flows (PR=63%)
0.95
302K
3.37
510K
Sparse Flows (PR=71%)
0.96
234K
3.38
395K
Sparse Flows (PR=77%)
0.97
182K
3.39
308K
Sparse Flows (PR=82%)
0.98
141K
3.40
239K
Sparse Flows (PR=86%)
0.97
109K
3.42
186K
sparse flow contains 70x less parameters (234K vs 16.6M). On CIFAR1O, we achieve the second best performance with over 38x less parameters compared to Residual flows which performs best.
Furthermore, on CIFARIO, Glow with 44 million parameters performs on par with our sparse FFJORD network with 657k parame-
Loss (NLL)
9:42
ters. It is worth mentioning that FFJORD ob- tains this results by using simple Gaussian prior, while Glow takes advantage of learned base distribution (Grathwohl al., 2019). Fig- ure 8 illustrates the improvement of the loss (negative likelihoohh)) density estimation on image datasets result of pruning neural
3
Prune Rabio (%)
Prune
(a) MNIST
(b) CIFARIO
Figure Loss vs. prune ratio for MNIST and CI- FAR 10 with unstructured and structured pruning.

---

Table 4: Percentage of good quality samples as measure of mode-collapse. structured pruning.
Pruning method:
GAUSSIAN
GAUSSIAN-SPIRAL
MNIST
Percentage of good quality samples as measure of mode-collapse
Prune ratio (%)
std =
std =
std =
std =
Prune ratio (%)
std 3
std = 5
0.0
72.89%
89.34%
66.65%
84.01%
0.0
6.94%
58.24%
20.0
79.46%
93,45%
74.29%
90.64%
20.0
8.02%
65.98%
25.6
81.40%
94.90%
78.22%
93.02%
37.8
9.35%
68.11%
52.0
84.30%
96.85%
80.40%
94.29%
51.7
8.70%
67.24%
71.2
81.83%
94.86%
78.06%
93.41%
77.3
8.59%
66.24%
75.1
68.20%
86. 80%10%
78.99%
93.67%
89.3
6.07%
57.28%
98.0
36.01%
76.54%
9.27%
20.32%
91.7
0.02%
32.75%
ODEs. Around 60% sparsity of a continuous normalizing flow leads better generative modeling compared to densely structured flows.
4.4 Pruning Helps Avoid Mode-collapse
To quantitatively explore the effectiveness of pruning avoiding mode-collapse (as shown qualita- tively Figure 5, we developed measure adopted from (Srivastava et al., 2017b): The percentage of Good Quality samples- We draw samples from trained normalizing flow and consider a sample of "good quality" if is within n (n typically chosen be 3, or 5) standard deviations from its nearest mode. We then can report the percentage of the good quality samples measure of how well the generative model captures modes.
Table summarizes the results on some synthetic datasets and the large-scale MNIST dataset. We can see that the number of "good quality samples", which stand for quantitative metric for mode-collapse improves consistently with certain percentage of pruning in all experiments. This observation is very much aligned with our results on the generalization loss, which we will discuss in the following.
4.5 Pruning Flattens the Loss Surface for Neural ODEs
What could be the potential reason for the enhanced generalization of pruned CNFs besides their ability to resolve mode-collapse? investigate this further, we conducted Hessian-based analysis on the flows-under-test. Dissecting the properties of the Hessian by Figenanalysis allows us to gain useful insights about the behavior of neural networks (Erichson et al., 2021, Ghorbani al., 2019, Hochreiter and Schmidhuber, 1997, Lechner and Hasani, 2020, Sagun et al., 2017). We PyHessian (Yao al., 2020) tool-set to analyze the Hessian w.r.t. the parameters the CNF This enables to study the curvature of the loss function the eigenvalues of the Hessian determines (locally) the loss gradients' changes.
Table 5: Eigenanalysis of the Hessian in terms of the largest eigenvalue (Amaz), trace (tr), and condition number (k) of pruned and unpruned continuous normalizing flows the mixture of Gaussian task. Numbers are normalized with respect the unpruned flow. See more results on other datasets in the supplements Section S3.
Model
NLL
Amaa (1)
tr(H)
k(H)
Unpruned FFJORD
1.309
1.000
1.000
1.000
Sparse Flows (PR=20%)
1.163
0.976
0.858
0.825
Sparse Flows (PR=60%)
1.125
0.356
0.583
0.717
Sparse Flows (PR=70%)
1.118
0.295
0.340
0.709
Sparse Flows(PR=90%)
1.148
0.416
0.366
0.547

---

PR=0%
DinL
Depihh
PR=84%
10
08
0.
Dm
PR=96%
DnL
1.0
Figure 9: Robustness of decision boundaries for pruned networks. Column 1 is the decision boundary. Column 2 = state-space, and column the flow of data points: it shows the evolution of single samples (each narrow line represents an individual sample's dimension I path) from input their output through the depth of the network. How does the attenuation of the vector field intensity indicate that the classification boundary is column and row entry: We see that the classification boundary between the blue half-moon and the orange half-moon very close to one tail of the blue half-moon. This means that if perturb the blue half-moon data, some of them might get classified incorrectly based on the extremely pruned network's decision boundary. As the intensity of the vector field shrinks the center of the vector field (it is attenuated in the areas distant from the center), the decision boundary is set poorly as opposed networks with more parameters.
sensitive perturbations? Let's look at the first
Larger Hessian eigenvalues therefore, stand for sharper curvatures and their sign identifies upward downward curvatures. In Table 5, we report the maximum eigenvalue of the Hessian Amaz. Hessian's 's Trace, and the condition number к = Amaz flow found a flatter minimum. As shown in Figure 3, a flat minimum leads to a better generalization error as opposed to sharp minimum. We find that up certain prune ratio, the maximum of the Hessian decreases and increases thereafter. Therefore, we claim that pruned continuous flows finds flatter local minimum, therefore, it generalize better than their unpruned version.
Smaller Amaz and tr H) indicates that our normalizing
Xmizy
Moreover, the Hessian condition number x could be an indicator of the robustness and efficiency of a deep model (Bottou and Bousquet, 2008). Smaller corresponds to obtaining more robust learned agent. We observe that x also follows the same trend and up to a certain prune ratio shrinks and then increases again confirming our hypothesis.
4.6
On the Robustness of Decision Boundaries of Pruned ODE-based Flows
While pruning improves performance, it is imperative to investigate their robustness properties in constructing decision boundaries (Lechner et al., 2021). For feedforward deep models it was recently shown that although pruned networks perform as well as their unpruned version, their robustness significantly decreases due to smaller network size (Liebenwein et al., 2021a). this also the case for neural ODEs?
We design experiment investigate this. We take simple 2-Dimensional moon dataset and perform classification using unpruned and pruned neural ODEs. This experiment (shown in Figure
"Thi experiment was inspired by the Hessian-hased rohustness analysis performed Erichson et al. (2021). "This experiment is performed using the TorchDyn library (Poli et al., 2020a).

---

9) demonstrates another valuable property of neural ODEs: We observe that neural ODE instances pruned up 84%, manage to establish safe decision boundary for the two half moon classes. This insight about neural ODEs which was obtained by our pruning framework is another testimony of the merits of using neural ODE in decision-critical applications. Additionally, we observe that the -ouuddary gets very close to one of the classes for a network pruned up to 96% which reduces robustness. Though even this network instance provide the same classification accuracy compared to the densely connected version.
The state-space plot illustrates that a neural ODE's learned vector-field becomes edgier as we keep pruning the network. Nonetheless, we observe that the distribution of the vector-field's intensity (the color map in the second column) get attenuated as we go further away from its center, in highly pruned networks. This indicates that classification near the decision boundary is more sensitive to perturbations extremely pruned networks.
Moreover, Column 3 of Figure shows that when the networks are not pruned (column 3, row 1) or are pruned up to certain threshold (column 3, row 2), the separation between the flow of samples of each class (orange or blue) is more consistent compared to the very pruned network (column 3, row 3). In Column 3, row 3 we see that the flow of individual samples from each class is more dispersed. As a result, the model is more sensitive to perturbations on the input space. This is very much aligned with our observation from column 1 and column 2 of Figure 9.
5 Discussions, Scope and Conclusions
We showed the effectiveness of pruning for continuous neural networks. Pruning improved generaliza- tion performance of continuous normalizing flows in density estimation tasks at scale. Additionally, pruning allows to obtain performant minimal network instances with at least one order of mag- nitude less parameter count. By providing key insights about how pruning improves generative modeling and inference, we enabled the design of better neural ODE instances.
Ensuring sparse Hessian computation for sparse continuous-depth models. As we prune neural ODE instances, their weight matrices will contain zero entries. However the Hessian with respect to those zero entries is not necessarily zero. Therefore, when we compute the eigenvalues of the Hessian, we must ensure to make the decompositon vector sets the Hessian of the pruned weights to zero before performing our eigenanalysis.
Why experimenting with FFJORD and not CNFs? FFJORD is an elegant trick to efficiently compute the log determinant of the Jacobians the change of variables formula for CNFs. Thus, FFJORDs are CNFs but faster. To confirm the effectiveness and scalability of our approach, we pruned some CNFs on toy datasets and concluded very similar outcomes (see these results in Appendix).
What are the limitations of Sparse Flows? Similar to any based learning system, the compu- tational efficiency of sparse flows is highly determined by the choice of their ODE solvers, data and model parameters. As the complexity of any of these fronts increases, the number of function evalua- tions for a given task increases. Thus we might have slow training process. This computational overhead can be relaxed in principle by the use of efficient ODE solvers (Poli et al., 2020b) and flow regularization schemes (Massaroli al., 2020).
What design notes did we learn from applying pruning to neural ODEs? We performed an ablation study over different types of features of neural ODE models. These experiments can be found in the supplements Section S4. Our framework suggested that to obtain a generalizable sparse neural ODE representation, the choice of activation function important. In particular activations that are Lipschitz continuous, monotonous, and bounded are better design choices for density estimation tasks.
Moreover, we found that the generalizability of sparse neural ODEs is more influenced by their neural network's width than their depth (number of layers), c.f., Appendix Section S4. Furthermore, pruning neural ODEs allows us to obtain better hyperparameters for the optimization problem by setting a trade-off between the value of the learning rate and weight decay.
In summary, we hope to have shown compelling evidence for the effectiveness of having sparsity in ODE-based flows.
10

---

Acknowledgments
This research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-194 1-1300-9-2--0000 1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Govermment. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This work was further supported by The Boeing Company and the Office of Naval Research (ONR) Grant N00014-18-1-2883
Funding Transparency Statement
Authors declare no competing interests. Funding in direct support of this work: The Bocing Company through the Explainable Control Project, National Science Foundation (NSF) Graduate Research Fellowship Program, the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000 and The Boeing Company and the Office of Naval Research (ONR) Grant N00014-18-1-2330.
References
Ryan P Adams, Jeffrey Pennington, Matthew J Johnson, Jamie Smith, Yaniv Ovadia, Brian Patton, and James Saunderson. Estimating the spectral density of large implicit matrices. arXiv preprint arXiv:1802.034511 2018.
Cenk Baykal, Lucas Liebenwein, Igor Gilittchennki, Dan Feldman, and Daniela Rus. Data-dependent coresets for compressing neural networks with applications to generalization bounds. In Inter- national Conference Learning Representations, 2019a. URL htttps://ppenteeee //oppr/ee/e//eeeee///// net/
forum?id hiimmidd=JJfw2A55
Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Sipping neural networks: Sensitvitt-infooreed provable pruning of neural networks. arXiv preprint arXiv:1910.0542. 2019b.
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomczak, and Max Welling. Sylvester normalizing flows variational inference. arXiv preprint IrXiv:180:3 090.005649 2018.
Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. In Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2008. URL https: http:://://prrceee http:////rcceennss///ee///ss www.inuuu..... wttp:ipp..cppppp/p/0///////
Paper pdf.
Ricky T. Q. Chen, Jens Behrmann, David K Duvenaud, and foem--Henri Jacobsen. Residual flows for invertible generative modeling. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buc, E. Fox, and R. Gameet,, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: 5a0d5594d24f09955548888
www.ffdddd-pappp pdf.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In Advances in neural information processing systems, pages 6571-6583, 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. In International Conference on Learning Representations, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605 100.08800000.033, 2016.
John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.

---

Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In Advances in Neural Information Processing Systems, pages 3134-3144, 2019.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows, arXiv preprint arXiv:1906. (.Xi::1906.00032,, 2019.
N. Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W. Mahoney. Lipschitz recurrent neural networks. In International Conference on Learning Represen- tations, 2021. URL hwwpt::ttttttteeee httppss///ppeerreeeee wtt.://oooooomiooc-c/-P........
Chris Finlay, Jörn- Jacobsen, Levon Nurbekyan, and Adam M Oberman. How train your neural ode. arXiv preprint arXiv:2002.02799 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable
neural networks. In International Conference Learning Representations, 2019. URL https:
//openreview www./neee/////eeee
///peerereer..ntt//rrr////// fi b3RcF7.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 881-889, Lille, France, 07-09 Jul 2015. PMLR. URL http: /tttt:///prcceeings.. mlr. wwt.press/vve v37/ /ttr///rmainn11 html.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pages 2232- 2241. PMLR, 2019.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. International Conference on Learning Representations, 2019.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.001499, 2015a. URL
http:// /ttp:///aaxv...eo www/abs/////oeeoooooo..ooe.eeere awtaaabs/1100 00149.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume Volume pages 1135-1143, 2015b.
Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. natural lottery ticket winner: Reinforcement learning with ordinary neural circuits. In International Conference on Machine Learning, pages 4082-4093 PMLR, 2020.
Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu, Liquid time- constant networks. Proceedings of the AAAI Conference on Artificial Intelligence, 359977577766, May 2021.
Babak Hassibi and David G Stork. Second order derivatives for network pruming: Optimal brain surgeon. Morgan Kaufmann, 1993.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 103iv:1503.02531, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Confer- ence on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2078- 2087. PMLR, 10-15 Jul 2018. URL httphttp://prrceedang mlr. press/ w80//vaaaaaaaa 18d. html.
Chin-Was Wei Huang. Ricky TQ Chen, Christos Tsirigotis, and Aaron Courville. Convex potential flows: Universal probability distributions with optimal transport and convex optimization. arXiv preprint arXiv:2012.05942, 2020.
12

---

Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications 1989.
and Computation, 183)
Priyank Jaini, Kira Selby, and Yaoliang Yu. Sum-of-squares polynomial flow. In International Conference on Machine Learning, pages 3009-3018. PMLR, 2019.
Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-hatch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Diederik P Kingma and Prafulla Dhariwal. Glow: generative flow with invertible 1 1 convolutions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 10236-1011 10245, 2018.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow. arXiv preprint arXiv:1606. 04934, 2016.
Zhifeng Kong and Kamalika Chaudhuri. The expressive power of class of normalizing flow models. In International Conference on Artificial Intelligence and Statistics, pages 3599-3609, PMLR, 2020.
Mathias Lechner and Ramin Hasani. Learning long-term dependencies in iregularly-sampled time series. arXiv preprint arXiv:2006.04411 04 2020.
Mathias Lechner, Ramin Hasani, Alexander Amini, Thomas A Henzinger, Daniela Rus, and Radu Grosu. Neural circuit policies enabling auditable autonomy. Nature Machine Intelligence, 2(10): 642-652, 2020a.
Mathias Lechner, Ramin Hasani, Daniela Rus, and Radu Grosu. Gershgorin loss stabilizes the recurrent neural network compartment of an end-to-end robot learning scheme. 2020 IEEE International Conference Robotics and Automation (ICRA), pages 5446-5452. IEEE, 2020b.
Mathias Lechner, Ramin Hasani, Radu Grosu, Daniela Rus, and Thomas A Henzinger. Adversarial training not ready for robot learning. arXiv preprint arXiv:2103.08187 2021.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 16Xxv:66080087000 2016.
Xuechen Li, Timg-Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pages 3870-3883 PMLR, 2020.
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In International Conference on Learning Representations, 2020.
URL https: / /ttp:/:://///eeeee/eeee wttp:/neeee/ooomidadmedeeeoee
Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. Lost in pruning: The effects of pruning neural networks beyond test accuracy. Proceedings of Machine Learning and Systems, 3, 2021a.
Lucas Liebenwein, Alaa Maalouf, Oren Gal, Dan Feldman, and Daniela Rus. Compressing neural networks: Towards determining the optimal layer-wise decomposition. arXiv preprint arXiv:2107.1 1442, 2021b.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEFE international conference on computer vision, pages 5058-5066, 2017.
13

---

Alaa Maalouf, Harry Lang, Daniela Rus, and Dan Feldman. Deep learning meets projective clustering. In International Conference on Learning Representations, 2021. URL https://openreviii net/forume htt::/fouuridd-Efffee 1d=EffpuFF3bbb
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asma. Dissecting neural odes. In 34th Conference on Neural Information Processing Systems, NeurlPS 2020. The Neural Information Processing Systems, 2020.
Molchanov, Tyree, Karras, Aila, and J Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017-Conference Track Proceedings, 2019.
Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. Neural importance sampling. ACM Transactions on Graphics (TOG), 38(5):1-19, 2019.
Junier Oliva, Avinava Dubey, Manzil Zaheer, Barnabas Poczos, Ruslan Salakhutdinov, Eric Xing, and Jeff Schneider. Transformation autoregressive networks. In International Conference Machine Learning, pages 3898-3907. PMLR, 2018a.
Junier Oliva, Avinava Dubey, Manzil Zaheer, Barnabas Poczos, Ruslan Salakhutdinov, Eric Xing, and Jeff Schneider. Transformation autoregressive networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1898-3907. PMLR, 10-15 Jul 2018b. URL http:/ httpp://proceedings mlr, wwwppppsssss.eee /v00////////////// http///000//ll11/11 olival8a..a...... html....................
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. Ot-flow: Fast and accurate continuous normalizing flows via optimal transport. arXiv preprint arXiv:2006.00 00104, 2020.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. arXiv preprint MwXivv:70S.07000 2017.
Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Torchdyn: A neural differential equations library. arXiv preprint arXiv:2009.09346. 2020a.
Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, Jinkyoo Park, et al. Hy- persolvers: Toward fast continuous-depth models. Advances in Neural Information Processing Systems, 33, 2020b.
Lev Semenovich Pontryagin. Mathematical theory of optimal processes. Routledge, 2018.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing fine-tuning and rewinding in neural network pruming. In International Conference on Learning Representations, 2020. URL https://openreviet: wtt.:eeeoorrmmddd1111100000000
Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows. In International Conference Machine Learning, pages 1530-1538. PMLR, 2015.
David Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back-propagating emror. nature, 32366088)5533-5366 1986.
Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the
hessian of over-parametrized neural networks. arXiv preprint 170.00400004454 2017.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles Sut- ton. Veegan: Reducing mode collapse in gans using implicit variational learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017a. URL Paaldddappp...ee pdf. (4220044955aaaa88044eeeeeea
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. Proceedings of the 31 International Conference Neural Information Processing Systems, pages 3310-3320, 2017b.
14

---

Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling- based invertible neural networks are universal diffeomorphism approximators. arXiv preprint arXiv:2006. / /469, 2020.
Charles Vorbach, Ramin Hasani, Alexander Amini, Mathias Lechner, and Daniela Rus. Causal navigation by continuous-time neural networks. arXiv preprint arXiv:2106.0833 2021.
Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks. arXiv preprint 19X8::9908.05164 2019.
Liu Yang and George Em Karniadakis. Potential flow generator with 12 optimal transport regularity for generative models. IEEE Transactions Neural Networks and Learning Systems, 2020.
Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural networks using merrg--awaee pruning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 5687-5695, 2017.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE International Conference on Big Data (Big Data), pages 581-590. IEEE, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "KuVcPUkrVED", "reviewer": "Reviewer_LH4D", "summary": "This paper shows that it is possible to drastically prune the network parameterising the neural ODE in a CNF and that this can result in improved generalisation as measured by the LL on a held-out test dataset. The authors hypothesise that this is due to the flattening of the loss surface around the local minima.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the idea of this paper is not particularly original – the authors apply existing pruning techniques to an existing CNF method – the simplicity is a plus in this case. A simple paper is in a prime position to have a big practical impact, and smaller neural networks, which are *also more performant*, are of course going to be of great interest to ML practitioners. However, the paper's potentially large practical significance is, unfortunately, let down by a few half-baked ideas, some overly strong claims, and lack of clarity in a few places. \n\n## Half-baked ideas & overly strong claims\n\n1. *Structured vs unstructured pruning.* The authors mention that their method can be applied to both kinds of pruning, and shows that this is true for one of their toy experiments. However, they don't demonstrate that this holds for larger-scale experiments. Nor do they provide any analysis comparing/contrasting these methods in the context of CNF pruning which is a shame. These kinds of insights would be very interesting and useful! \n\n2. *Key insights enabling the design of better neural ODE instances.* It wasn't clear to me what the insights were that would allow for the design of better neural ODEs. Furthermore, I would want to see some proof-of-concept experiments that show that this is true. The authors might be referring to the paragraph \"What design notes did we learn from applying pruning to neural ODEs?\" but most of these insights don't seem to be related to pruning. Furthermore, none of the claims here were demonstrated in the paper. Experiments should be provided to back these claims up, at the moment it isn't clear where they come from.\n\n3. *Pruning helps to avoid mode-collapse in generative modelling.* This is a strong claim to make given that this is only demonstrated on a few toy datasets.\n\n4. *Pruning Flattens the Loss Surface for Neural ODEs.* This is also a strong claim to make, given evaluation on a single dataset – results for more datasets (including tabular and image) would be helpful.  Relatedly, \"We find that up to a certain prune ratio, the maximum of the Hessian decreases and increases thereafter\" is also not shown in the results. Specifically, no increase is shown, and only a few prune ratios are shown – a plot with a range of ratios would be much more illustrative. \n\n5. *One cannot directly train sparse and efficient CNFs from scratch.* While this sounds like a reasonable statement, it is not necessarily true, a small experiment to demonstrate this would help! For example, in tables 2 & 3, add results for some FFJORD models with the same number of parameters as the pruned versions. \n\n6. It would be great to have a few experiments showing whether or not the results from this paper do indeed also apply to standard CNFs (without FFJORD) or if there is something special about FFJORD which makes pruning work well.\n\n## Clarity issues/questions\n\n1. It isn't clear where the 98% pruning mentioned in the abstract comes from. \n\n2. It isn't clear what the term \"decipher the internal dynamics\" in the abstract means.\n\n3. Please be more specific about the pruning procedure. From reading the text I am not sure I'd be able to replicate the results. Saying that the procedure is \"inspired\" by iterative learning rate rewinding is too vague! Similarly, the term \"pre-defined threshold\" isn't specific enough, it isn't clear exactly what is meant by \"prune the structures\" (what kinds of structures are being considered?), and it isn't clear what \"we generally follow the same hyperparameters\" means?\n\n4. I did not understand figure 6. A more detailed description, some annotations in the figure, and perhaps adding another dataset would help to know what I should be seeing. \n\n5. What are the pruning ratios in table 2, and how were they chosen? Why do some datasets have more pruning variations than others?\n\n6. Why do figures 8 a), b), and e) not have the full range of pruning ratios? \n\n7. What does the phrase \"flows-under-test\" mean?\n\n8. Regarding figure 9:\n  * What does it mean for a vector field to be \"edgier\"? Annotating the figure might help here.\n  * How does the attenuation of the vector field intensity indicate that the classification boundary is more sensitive to perturbations? A little intuition might be helpful here. \n  * What should I be looking at in column 3 of figure 9? Some analysis would be helpful.\n\n## Other comments & suggestions\n\n1. I think the NICE paper would be a better citation for eqn 1.\n\n2. Add NSF as a benchmark for the tabular experiments.\n\n3. Split eqn 2 into two equations, and add a numbered equation for the neural ODE formula on line 100. Then it would be much easier/clearer to reference these equations in the paragraph \"Continuous Normalizing Flows\". \n\n4. Add a citation for the use of back-prop through time to train CNFs?\n\n5. In figure 5, it would be very helpful to show the ground truth densities. Otherwise, it is hard to know how the method is performing.\n\n## Summary  \n\nOverall, I think this paper has a lot of potential to be both interesting and useful. However, I think the current iteration falls short of this promise. As a result, I will be giving a score of 4. I would be very happy to increase my score if the authors can expand on what they have already and provide a more comprehensive and clear story of pruning in CNFs.\n\n## Update during the discussion phase\n\nThe authors have addressed a number of my concerns and clarified many sources of confusion. I do still have a number of questions that I've discussed in responses to the authors. However, I am happy to increase my score to a 5 given the response so far and would be willing to increase it further depending on the response to my remaining concerns. \n\n## Final update during the discussion phase\n\nThe authors have addressed my remaining concerns and have added substantial additional experimental results. As a result, I have decided to increase my score to a 7. I have, however, decreased my confidence as it is hard to know exactly what the camera-ready version of the paper will now look like. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations of their work but did not touch on any societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "IZadfd5TUmK", "reviewer": "Reviewer_AFJg", "summary": "The authors introduce sparse-flows, which is trained by iteratitively pruning a continous normalizing flow network (in this case FFJORD). \n\nThrough their experiments the authors show that the sparsified FFJORD model often preforms better than FFJORD and other flow-based models like GLOW, real-NVP etc. They show (on a variety of datasets) that upto a limit, sparsifiying the model can reduce the NLL of the data. The authors also show that pruning flattens the loss surface for Neural-ODEs and show that classifiers trained through sparse-ODEs are more robust than ODEs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and motivated and the experiments provide good insights on the effects of sparsifying continuous normalizaing flows.\n\nWhile the results are good, my concern is with the overall novelty of the paper. The authors tested out the results of pruning in a new field, namely continuous flow, however their findings although intriguing are not so suprising given the recent results papers that show that neural networks architectures are sparse and thus can be pruned to reduce their size while keeping the performace intact.\n\nSparse Flow, the algorithm introduced by the authors, is based on FFJORD (which I presume is more efficient to train than other models). However, given that the main contribution of the paper is how sparsity improves the performance of flow-based models, the methodology should be applied to other models as well, especially to verify the performance gain is not only limited to sparsifying FFJORD.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately discussed the limitations of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "D89plo_Um_F", "reviewer": "Reviewer_uTzg", "summary": "The paper presents a finding that pruning improves generalisation for neural ODEs in generative modelling alongside with reducing the number of parameters to up to 98%. \n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Pros:*\n\nThe proposed method is an attractive way of improving the number of parameters and at the same time regularising continuous generative models. The method has a potential for good impact for the neural ODE community and practitioners using the methods. \n\n*Cons:* \n\nSome of the description could be clarified (see below)\n\n*Clarity:*\n\nLine 132- 143: It would be good to tell  here how exactly this structured pruning method, local and global, is implemented for the continuous models, as the original methods are designed for (discrete) deep neural networks. Also, it would be good to define $W_ij$ in Table 1\n\n*Ablation studies*\n\nIt looks convincing from the experiment that pruning has a good effect on performance, however, some additional ablation studies could help understand it further why it happens (decorrelation of parameters similar to the dropout, smaller number of parameters decreasing the chance of falling into undesirable local minima):\n- how would the results compare with reducing the width of the dense network? If that helps too, it would suggest that just smaller number of parameters could improve learning.  I appreciate that it may or may not be feasible to disentangle it in the experimental setting from Neural ODE dimensionality as every continuous layer of neural ODE has the same dimension, however it would be good to hear the author's opinion on it in the response and in the paper. \n- I wonder if keeping the number of parameters the same but introducing the analogue of dropout could help understand whether it gives the regularisation result similar to the dropout (and whether in that case the same number of parameters as in the original Neural ODE architecture would be beneficial).\n\n*Other comments:* \n\n- Table 2: Would it be possible to output confidence intervals as well?\n\n- lines 299-306: the authors discuss the activation function choice, it would be good if the authors could clarify whether they have more evidence on its impact (and maybe put that evidence in the appendix). \n\n== After rebuttal ==\n\nAs the authors sufficiently addressed the concerns in the rebuttal and carried out new ablation studies which could significantly improve the content of the paper, the score is increased from 6 to 7. However, as the reviewer could not open the main file with figures of the ablation study due to technical error and only could see the contents and conclusion of the ablation study, the confidence is decreased from 4 to 3.\n\n**Update:** the confidence is increased back to 4 as the authors provided necessary support for ablation studies. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of computational complexity and accuracy are thoroughly discussed. The societal impact relates to the general concerns about inappropriate use of machine learning methods and neural ODE based density estimation in particular.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "5QDvSz5NIE", "reviewer": "Reviewer_nNvs", "summary": "**Summary**\n\nThe paper describes the effect of using pruning during neural ODEs training. Namely, the authors propose to train neural ODEs iteratively by alternating training (fine-tuning) and pruning steps. They demonstrate the benefits of applying the proposed training procedure to the density estimation tasks. Also, the paper considers a simple classification task to compare the robustness of decision boundaries of pruned ODE-based flows.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nTraining procedure that alternates pruning and fine-tuning steps is not new. The novelty of the proposed paper is application of this idea to neural ODEs.  The paper investigates the properties of pruned (sparse) neural ODEs. They provide examples to show how the usage of pruning (structural/unstructural) improves generalization, flattens the loss surface, and helps to avoid mode-collapse. They make an overview of important papers related to generative modelling and pruning.\n\n**Quality**\n\nI'd consider this paper as a complete piece of work that requires some clarifications. There are several questions concerning empirical experiments that should be specified (please, see the Questions section below). Also, I'd recommend writing a passage about time computational costs (both wall clock and number of function evaluations) during training/inference required for sparse neural ODEs comparing to dense neural ODEs.\n\n**Clarity**\n\nBoth background and method description are clearly written. I'd suggest adding the following paper when overviewing efficient neural ODEs training: (Daulbaev, T. et al. Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs. 2020)\n\n**Significance**\n\nThough the training with alternating pruning/fine-tuning steps is not new, its application to neural ODEs leads to several benefits, particularly significant improvement in parameters/loss trade-off for density estimation tasks.\n\n**Questions**\n- What are time costs (wall clock and number of function evaluations) during training/inference?\n- Suppose we start with two neural ODEs of different layer widths and end up with two models of the same number of parameters. What difference we'll see in the properties of the resulting models? (If there is any)\n- What is a motivation to use different optimizers in experiments (sometimes Adam, sometimes AdamW)?\n- What induces the bigger difference for structural and unstructural pruning in the case of Gaussian distribution? (Figure 4)\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Concerning limitations, there would be nice to have more information about time complexity (see Questions).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The following is a summary of the pros and cons raised by the reviewers:

pros:
* complete work (nNvs)
* Method and background sections are clearly written. (nNvs, AFJg)
* application of pruning to neural ODEs leads to several practical benefits such as generalization and reduced parameter count and its simplicity is a plus (nNvs, uTzg, LH4D)

cons:
* Requires additional experimental results:
    - wall-clock time and number of function evalutions during training and inference should be reported to see how pruning influences 
this. --> preliminary results described in rebuttal, no reply from reviewer. (nNvs)
    - More evidence is needed to support claims such as flattening the loss surface. --> addressed during discussion period. (LH4D)
    - missing ablation studies that could help improve understanding of why pruning has benefits --> addressed in rebuttal.(uTzg)
    - structured vs unstructured pruning should be investigated on larger scale experiments, not only toy datasets --> addressed during discussion period. (LH4D)
    - pruning should be applied to other continuous normalizing flows, not just ffjord. --> authors have provided results on other cnf's on a toy dataset. (AFJg, LH4D) 
* clarifications are required with respect to the pruning strategy and optimization choices. --> addressed in rebuttal (nNvs, uTzg)
* findings are unsurprising (AFJg).

The authors and 3 out of 4 reviewers engaged in active and fruitful discussions during the discussion period. The authors included many additional results that were requested by the reviewers, which led to the main concerns being addressed and 3 out of 4 scores being raised. The recommended decision for this submission is accept.
Decision: Accept (Poster)