Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: **Summary**

The paper describes the effect of using pruning during neural ODEs training. Namely, the authors propose to train neural ODEs iteratively by alternating training (fine-tuning) and pruning steps. They demonstrate the benefits of applying the proposed training procedure to the density estimation tasks. Also, the paper considers a simple classification task to compare the robustness of decision boundaries of pruned ODE-based flows.
main_review: **Originality**

Training procedure that alternates pruning and fine-tuning steps is not new. The novelty of the proposed paper is application of this idea to neural ODEs.  The paper investigates the properties of pruned (sparse) neural ODEs. They provide examples to show how the usage of pruning (structural/unstructural) improves generalization, flattens the loss surface, and helps to avoid mode-collapse. They make an overview of important papers related to generative modelling and pruning.

**Quality**

I'd consider this paper as a complete piece of work that requires some clarifications. There are several questions concerning empirical experiments that should be specified (please, see the Questions section below). Also, I'd recommend writing a passage about time computational costs (both wall clock and number of function evaluations) during training/inference required for sparse neural ODEs comparing to dense neural ODEs.

**Clarity**

Both background and method description are clearly written. I'd suggest adding the following paper when overviewing efficient neural ODEs training: (Daulbaev, T. et al. Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs. 2020)

**Significance**

Though the training with alternating pruning/fine-tuning steps is not new, its application to neural ODEs leads to several benefits, particularly significant improvement in parameters/loss trade-off for density estimation tasks.

**Questions**
- What are time costs (wall clock and number of function evaluations) during training/inference?
- Suppose we start with two neural ODEs of different layer widths and end up with two models of the same number of parameters. What difference we'll see in the properties of the resulting models? (If there is any)
- What is a motivation to use different optimizers in experiments (sometimes Adam, sometimes AdamW)?
- What induces the bigger difference for structural and unstructural pruning in the case of Gaussian distribution? (Figure 4)


limitations_and_societal_impact: Concerning limitations, there would be nice to have more information about time complexity (see Questions).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a finding that pruning improves generalisation for neural ODEs in generative modelling alongside with reducing the number of parameters to up to 98%. 


main_review: *Pros:*

The proposed method is an attractive way of improving the number of parameters and at the same time regularising continuous generative models. The method has a potential for good impact for the neural ODE community and practitioners using the methods. 

*Cons:* 

Some of the description could be clarified (see below)

*Clarity:*

Line 132- 143: It would be good to tell  here how exactly this structured pruning method, local and global, is implemented for the continuous models, as the original methods are designed for (discrete) deep neural networks. Also, it would be good to define $W_ij$ in Table 1

*Ablation studies*

It looks convincing from the experiment that pruning has a good effect on performance, however, some additional ablation studies could help understand it further why it happens (decorrelation of parameters similar to the dropout, smaller number of parameters decreasing the chance of falling into undesirable local minima):
- how would the results compare with reducing the width of the dense network? If that helps too, it would suggest that just smaller number of parameters could improve learning.  I appreciate that it may or may not be feasible to disentangle it in the experimental setting from Neural ODE dimensionality as every continuous layer of neural ODE has the same dimension, however it would be good to hear the author's opinion on it in the response and in the paper. 
- I wonder if keeping the number of parameters the same but introducing the analogue of dropout could help understand whether it gives the regularisation result similar to the dropout (and whether in that case the same number of parameters as in the original Neural ODE architecture would be beneficial).

*Other comments:* 

- Table 2: Would it be possible to output confidence intervals as well?

- lines 299-306: the authors discuss the activation function choice, it would be good if the authors could clarify whether they have more evidence on its impact (and maybe put that evidence in the appendix). 

== After rebuttal ==

As the authors sufficiently addressed the concerns in the rebuttal and carried out new ablation studies which could significantly improve the content of the paper, the score is increased from 6 to 7. However, as the reviewer could not open the main file with figures of the ablation study due to technical error and only could see the contents and conclusion of the ablation study, the confidence is decreased from 4 to 3.

**Update:** the confidence is increased back to 4 as the authors provided necessary support for ablation studies. 

limitations_and_societal_impact: The limitations of computational complexity and accuracy are thoroughly discussed. The societal impact relates to the general concerns about inappropriate use of machine learning methods and neural ODE based density estimation in particular.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper shows that it is possible to drastically prune the network parameterising the neural ODE in a CNF and that this can result in improved generalisation as measured by the LL on a held-out test dataset. The authors hypothesise that this is due to the flattening of the loss surface around the local minima.
main_review: While the idea of this paper is not particularly original – the authors apply existing pruning techniques to an existing CNF method – the simplicity is a plus in this case. A simple paper is in a prime position to have a big practical impact, and smaller neural networks, which are *also more performant*, are of course going to be of great interest to ML practitioners. However, the paper's potentially large practical significance is, unfortunately, let down by a few half-baked ideas, some overly strong claims, and lack of clarity in a few places. 

## Half-baked ideas & overly strong claims

1. *Structured vs unstructured pruning.* The authors mention that their method can be applied to both kinds of pruning, and shows that this is true for one of their toy experiments. However, they don't demonstrate that this holds for larger-scale experiments. Nor do they provide any analysis comparing/contrasting these methods in the context of CNF pruning which is a shame. These kinds of insights would be very interesting and useful! 

2. *Key insights enabling the design of better neural ODE instances.* It wasn't clear to me what the insights were that would allow for the design of better neural ODEs. Furthermore, I would want to see some proof-of-concept experiments that show that this is true. The authors might be referring to the paragraph "What design notes did we learn from applying pruning to neural ODEs?" but most of these insights don't seem to be related to pruning. Furthermore, none of the claims here were demonstrated in the paper. Experiments should be provided to back these claims up, at the moment it isn't clear where they come from.

3. *Pruning helps to avoid mode-collapse in generative modelling.* This is a strong claim to make given that this is only demonstrated on a few toy datasets.

4. *Pruning Flattens the Loss Surface for Neural ODEs.* This is also a strong claim to make, given evaluation on a single dataset – results for more datasets (including tabular and image) would be helpful.  Relatedly, "We find that up to a certain prune ratio, the maximum of the Hessian decreases and increases thereafter" is also not shown in the results. Specifically, no increase is shown, and only a few prune ratios are shown – a plot with a range of ratios would be much more illustrative. 

5. *One cannot directly train sparse and efficient CNFs from scratch.* While this sounds like a reasonable statement, it is not necessarily true, a small experiment to demonstrate this would help! For example, in tables 2 & 3, add results for some FFJORD models with the same number of parameters as the pruned versions. 

6. It would be great to have a few experiments showing whether or not the results from this paper do indeed also apply to standard CNFs (without FFJORD) or if there is something special about FFJORD which makes pruning work well.

## Clarity issues/questions

1. It isn't clear where the 98% pruning mentioned in the abstract comes from. 

2. It isn't clear what the term "decipher the internal dynamics" in the abstract means.

3. Please be more specific about the pruning procedure. From reading the text I am not sure I'd be able to replicate the results. Saying that the procedure is "inspired" by iterative learning rate rewinding is too vague! Similarly, the term "pre-defined threshold" isn't specific enough, it isn't clear exactly what is meant by "prune the structures" (what kinds of structures are being considered?), and it isn't clear what "we generally follow the same hyperparameters" means?

4. I did not understand figure 6. A more detailed description, some annotations in the figure, and perhaps adding another dataset would help to know what I should be seeing. 

5. What are the pruning ratios in table 2, and how were they chosen? Why do some datasets have more pruning variations than others?

6. Why do figures 8 a), b), and e) not have the full range of pruning ratios? 

7. What does the phrase "flows-under-test" mean?

8. Regarding figure 9:
  * What does it mean for a vector field to be "edgier"? Annotating the figure might help here.
  * How does the attenuation of the vector field intensity indicate that the classification boundary is more sensitive to perturbations? A little intuition might be helpful here. 
  * What should I be looking at in column 3 of figure 9? Some analysis would be helpful.

## Other comments & suggestions

1. I think the NICE paper would be a better citation for eqn 1.

2. Add NSF as a benchmark for the tabular experiments.

3. Split eqn 2 into two equations, and add a numbered equation for the neural ODE formula on line 100. Then it would be much easier/clearer to reference these equations in the paragraph "Continuous Normalizing Flows". 

4. Add a citation for the use of back-prop through time to train CNFs?

5. In figure 5, it would be very helpful to show the ground truth densities. Otherwise, it is hard to know how the method is performing.

## Summary  

Overall, I think this paper has a lot of potential to be both interesting and useful. However, I think the current iteration falls short of this promise. As a result, I will be giving a score of 4. I would be very happy to increase my score if the authors can expand on what they have already and provide a more comprehensive and clear story of pruning in CNFs.

## Update during the discussion phase

The authors have addressed a number of my concerns and clarified many sources of confusion. I do still have a number of questions that I've discussed in responses to the authors. However, I am happy to increase my score to a 5 given the response so far and would be willing to increase it further depending on the response to my remaining concerns. 

## Final update during the discussion phase

The authors have addressed my remaining concerns and have added substantial additional experimental results. As a result, I have decided to increase my score to a 7. I have, however, decreased my confidence as it is hard to know exactly what the camera-ready version of the paper will now look like. 
limitations_and_societal_impact: The authors discussed the limitations of their work but did not touch on any societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce sparse-flows, which is trained by iteratitively pruning a continous normalizing flow network (in this case FFJORD). 

Through their experiments the authors show that the sparsified FFJORD model often preforms better than FFJORD and other flow-based models like GLOW, real-NVP etc. They show (on a variety of datasets) that upto a limit, sparsifiying the model can reduce the NLL of the data. The authors also show that pruning flattens the loss surface for Neural-ODEs and show that classifiers trained through sparse-ODEs are more robust than ODEs.
main_review: The paper is well written and motivated and the experiments provide good insights on the effects of sparsifying continuous normalizaing flows.

While the results are good, my concern is with the overall novelty of the paper. The authors tested out the results of pruning in a new field, namely continuous flow, however their findings although intriguing are not so suprising given the recent results papers that show that neural networks architectures are sparse and thus can be pruned to reduce their size while keeping the performace intact.

Sparse Flow, the algorithm introduced by the authors, is based on FFJORD (which I presume is more efficient to train than other models). However, given that the main contribution of the paper is how sparsity improves the performance of flow-based models, the methodology should be applied to other models as well, especially to verify the performance gain is not only limited to sparsifying FFJORD.
limitations_and_societal_impact: The authors have adequately discussed the limitations of their work.
review_title: null

Meta-review: The following is a summary of the pros and cons raised by the reviewers:

pros:
* complete work (nNvs)
* Method and background sections are clearly written. (nNvs, AFJg)
* application of pruning to neural ODEs leads to several practical benefits such as generalization and reduced parameter count and its simplicity is a plus (nNvs, uTzg, LH4D)

cons:
* Requires additional experimental results:
    - wall-clock time and number of function evalutions during training and inference should be reported to see how pruning influences 
this. --> preliminary results described in rebuttal, no reply from reviewer. (nNvs)
    - More evidence is needed to support claims such as flattening the loss surface. --> addressed during discussion period. (LH4D)
    - missing ablation studies that could help improve understanding of why pruning has benefits --> addressed in rebuttal.(uTzg)
    - structured vs unstructured pruning should be investigated on larger scale experiments, not only toy datasets --> addressed during discussion period. (LH4D)
    - pruning should be applied to other continuous normalizing flows, not just ffjord. --> authors have provided results on other cnf's on a toy dataset. (AFJg, LH4D) 
* clarifications are required with respect to the pruning strategy and optimization choices. --> addressed in rebuttal (nNvs, uTzg)
* findings are unsurprising (AFJg).

The authors and 3 out of 4 reviewers engaged in active and fruitful discussions during the discussion period. The authors included many additional results that were requested by the reviewers, which led to the main concerns being addressed and 3 out of 4 scores being raised. The recommended decision for this submission is accept.
Decision: Accept (Poster)