Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper discusses using piecewise constant or piecewise linear basis functions to represent time-dependent parameters in an ODE-net. This can also be used to represent parameters in a time-dependent BatchNorm layer. A change of basis can be performed to reduce the number of parameters or increase the discretization step sizes.
main_review: Overall, this paper uses concepts from continuous-time/depth neural networks to improve their discretized variants, e.g. a ResNet. I like that the paper shows models with higher-order discretizations and continuous basis functions behave more similarly to continuous-depth neural networks, whereas the piecewise-constant (ResNet) counterpart behaves worse, especially during basis function compression.

However, the scope of the paper seems to be limited to learning simple mappings from input to output, and does not discuss applications that ODEs are more useful at, e.g. irregularly-sampled time series prediction and continuous normalizing flows. These applications either require more complex trajectories or directly make use of the right-hand-side of ODEs, and both benefit significantly from adaptive step size solvers in practice.

With the use of fixed step size solvers, it’s not clear whether the discretized model is still an approximation to the true ODE solution. The observation that piecewise-constant basis functions do not compress well may simply be due to them not being a close approximation of an ODE. It’d be more convincing to see experiments on the aforementioned applications.

Another concern with the use of piecewise-constant/linear basis functions is the lack of smoothness for higher-order solvers. When using a method such as RK4, the step sizes must correspond to the basis function’s Delta t. This again goes against the use of adaptive step sizes. This property makes it difficult to perform long-range predictions, e.g. for irregularly-sampled time series predictions. Why not explore more complex or smoother basis functions?

Regarding continuous-time batch normalization, it doesn’t seem clear to me that using this is beneficial in practice. While the paper discusses efficiently storing a time-dependent mean/var/bias/scale, it’s not clear if this actually improves convergence during training. Normalization seems like a weird operation to have in an ODE since it forces the model to predict outputs based on relative (and stochastic due to mini-batching) differences instead of absolute position. This dependence makes its interpretation as a dynamical system following a vector field less intuitive. In any case, it would be more convincing if the authors can run ablation experiments to discuss the merits of using continuous-time batch normalization.

Furthermore, the lack of a Lipschitz bound for normalization is concerning. If a hidden unit is the same value (e.g. zero) for all samples in a mini-batch, then there is a division by zero. Likewise, it was shown that multihead attention layers are not Lipschitz [1]. Though I suspect these aren’t major concerns unless one tries to solve the ODEs extremely accurately.

[1] “The Lipschitz Constant of Self-Attention” Kim et al.

----

Update after author response: Thanks for the clarifications. I generally agree with the response; however, the paper likely should be rewritten to make it clear that this is regarding discrete-time networks and to re-iterate on the scope and expectations of this work. The naming convention for continuous-time BN can also be misleading, since this is not applied in a continuous-time setting. Just my two cents, but answering these questions may be helpful: why is compressing an ODE-net an important task? What if you simply said you're compressing ResNets (and mention extending to networks that are parameterized by more sophisticated ODE solver schemes)? This may help increase the number of readers while also clarifying scope.
limitations_and_societal_impact: The scope is limited to discretized networks that are simply expected to behave like ODE solutions. This expectation is a small leap of faith and should be discussed more in the paper. It’d be more convincing to see experiments making use of adaptive step size solvers and/or applications involving strong reliance on accurate ODE solutions. The batch normalization method should be ablated against as it's not clear if this leads to faster training or performance improvements. Furthermore, the numerics involved with solving an ODE with potentially very large or unbounded Lipschitz is not discussed sufficiently.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes weight compression technique for learnt ODE-Nets without retraining the model while maintaining near state-of-the-art performance. Their proposal includes treating the weight matrices as the linear combination of continuous-in-time basis functions, where the linear combination coefficients are learnt by optimizing the loss function between the learnt weights and the approximate weights (computed by combining basis functions). Empirical evidence suggests that such a scheme compresses ODE-Nets on image classification and sentence-tagging tasks.
main_review: 
Strengths:
---------------

- Compression of the weight matrices is done post training and does not require re-training or any feed-forward passes through training data. 


Weakness:
---------------

- Novelty of the compression scheme seems incremental.

- Model compression using the proposed scheme only guarantees drop in model size. Paper lacks computational cost of performing inference ( Algorithm~1 suggests that the inference cost will be higher than forward pass of the uncompressed model ). 

Questions for Authors:
---------------

- It would have been better to showcase the inference time comparison between the compressed and uncompressed scheme. If for each input, the weights are constructed on the fly with the basis function, this additional cost is non-trivial and needs to be accounted for. Instead if the weights are simply constructed from the compressed model, then the inference proceeds as in the uncompressed scheme, then we did not gain anything from the computational standpoint. 


- Could authors elaborate a bit as to why is it hard to incorporate BatchNorm as one of the layers in the general function $f$ for the ODE $\frac{dx}{dt} = f(x)$?

- Is there any ablative study as to what benefits do you get by incorporating CT-Batch Norm in your proposed scheme?

- Do you count the basis function parameters as well as linear coefficients in the number of parameters shown in the tables?

- Did you run any of the baselines learning schemes with the architectures used in the experiments section? Its not clear from the paper if the baselines used the same architecture design as ContinuousNetV2 (barring the continuous time batch-norm).

Writing Clarity:
---------------

- Line 33: Clearly state what are gradient parameters and how they differ from state parameters.

- Missing related works 

	ODE/PDEs:
	Continuous Time RNN (by Rosenblatt),
	Neural Controlled Differential Equations for Irregular Time Series (https://proceedings.neurips.cc//paper/2020/file/4a5876b450b45371f6cfe5047ac8cd45-Paper.pdf), 
	Incremental RNNs (https://openreview.net/forum?id=HylpqA4FwS), 
	NeuPDE ( http://proceedings.mlr.press/v107/sun20a.html),
	Time Adaptive RNNs (https://openaccess.thecvf.com/content/CVPR2021/papers/Kag_Time_Adaptive_Recurrent_Neural_Network_CVPR_2021_paper.pdf)
	
	Related works on the deep network compression is missing.
	
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a new neural ODE with time-continuos depth and time-dependent parameterization of the network. The time-dependent parameterization is composed of a linear combination of time-dependent basis function and learnable “base parameters”. 

By relying on this continuous reparameterization using basis function the authors highlight the close connection with the compressibility of the network by doing a basis change during inference time to a lower-dimensional space of basis function, where the “base parameters” in the lower dimensional space can be computed either via projection or linear interpolation from the higher-dimensional basis space. 

Additionally, the authors introduces a novel notion of time-continuous batch normalization and they present theoretical results pertaining to the implicit regularization potential of their model when solved via RK-methods and empirical results for image classification and time-continuous analogues of transformers. 
main_review: ## Score

Overall, I definitely appreciated the novelty and the rigor of the proposed time-continuous model and I believe there is scientific merit and the potential to influence the broader community. However, I am little concerned about the current presentation of the results. Specifically, they are covering lots of grounds from introducing a new type of model to theoretical results to compression to empirical results. Consequently, some of the presentation is very superficial and it’s hard to understand the important details that might help other researcher pick up on this work and build upon it. 

So in short, I imagine the core contribution to be quite interesting but this has to be backed by the corresponding depth and detailed discussions of the method in the paper. Right now, there is lots of breadth and little depth across the sections.


## Ways to Improve My Score

Please address the weaknesses I am outlining below. I would specifically focus on trying to clean up some of the writing and re-structuring it to be able to go into depth about individual aspects of the work while potentially removing other aspects.

## Strengths

* Intuitive and powerful model that promises a lot of flexibility down the road. 

* Interesting experimental results pertaining both to image classification and NLP/transformers. 

* Good introduction and explanation about the concept of basis functions. 

* I enjoyed learning about the combination of using a NN parameterization based on basis functions  + time-continuous models. 

## Weaknesses

* By far the biggest weakness to me is that there is so much going on that it is very hard to formulate a take-away message at the end of the paper. Sure, I might be able to download the code, re-run the experiments, and modify the models a little bit to adapt them to my task. But I don’t think the papers contributes a deep understanding of the model that someone could leverage in their own research in order to come up with new and innovative solutions.

* Introduction: I don’t think it’s really useful to touch upon so many concepts in the introduction. Try re-structuring the core concepts, simplifying them, shortening the intro, and then go into more details about each aspect later on in the paper. 

* The time-continuous batch normalization: This concept has been mentioned many times throughout the sections and I suppose I can somewhat imagine how it works but overall the language of the paper never goes beyond the conceptual idea of what the authors want to achieve. For example, looking at Algorithm 1 I cannot figure out where $\hat\theta^{s*}$ is being used? It seems to be an output of the algorithm but what are we doing with it? Is it part of the model as a batch norm layer? Where would this layer be? It’s confusing but still seems to be very important so it would be crucial to understand how this fits into the model. 

* I cannot really find a take-away message from Section 5. It seems to be a somewhat interesting result from the space of dynamical systems and uses a lot of very atypical concepts that I cannot imagine are common knowledge among the broader NeurIPS community (or even the more narrow neural ODE community). If you really want to include these theoretical results, please simplify the section in the main paper and put a much more in-depth discussion into the appendix. In the current shape, I am left to simply take the authors’ word for granted about the connection between their theorem and the implicit regularization potential without really being able to back it up myself. 

* The experiments are good but I would ask the authors to significantly expand upon the concept of (multi-level) refinement. This seems to be a central empirical “trick” in order to make their model work in practice. What exactly is the concept behind it? How to tune the different levels? What is the crucial insight there that makes the difference in the empirical performance? 

* Choice of basis functions: The basis functions seem overly simplistic (either convex combination or piecewise). What about other basis functions and what constitutes the motivation for these basis functions? The concept and the notation are very heavy in order to introduce these basis functions. So is there any benefit beyond these simple basis functions? 

* Related to my question above: It appears to me that the authors rely on a fixed-step solver like RK4. In addition, they often use piecewise-constant basis functions. My question is how this model even differs from a regular NN that explicitly models multiple layers? Specifically, if for each time step in RK4 there is a different “piecewise-constant basis function” that is activated this implies that at any given depth you call the _same layer architecture_ with a _distinct set of parameters_ much like you would simply repeat the same layer architecture with separately trained parameters in a regular NN with residual connections? And even if there is more time steps in RK4 than there are basis functions it very much still seems to boil down to a regular NN with a little bit of parameter sharing, right? And same goes for the piecewise linear basis functions, right? I am not saying that the proposed formulation isn’t more expressive in theory but it seems that all instantiations of the model framework are essentially very close to  a regular network with multiple layers of the same layer architecture stacked on top of each other. Could the authors comment on this observation? 

* I would also appreciate if the authors could spend more time in the introduction and the related work on how their work differs from prior work. It seems basis functions have been explored before from what I understand from their paper and also of course neural ODEs are not a new concept. So it would be very helpful to contextualize their work within existing work more clearly and delineate their contributions. 

--- 

## Update after rebuttal 

I am raising my score to 6 notwithstanding some of the open issues that I feel need to be addressed, whether during a potential camera-ready version of the paper or a potential re-submission depending on the outcome of the review process. More details can be found below in my discussion with the authors. 
limitations_and_societal_impact: See above weaknesses especially with regards to the refinement in the training process. 

Additionally, I glanced over Section 4.1 in the appendix regarding the implementational details of the projection step. It seems to require various derivatives including a Hessian computation. How does the projection step scale with the model? Is that something that’s feasible for larger models and datasets? What is the practical runtime for this step? It would be good to understand that.
review_title: null

Meta-review: This paper presents a compressed representation of ODE-based neural networks, and analyzes its computational and implicit regularization properties. Experiments cover a good range of architectures, including both convolutional and transformer-based.

The reviewers believe there are worthwhile ideas in the paper and haven't identified anything that looks like a critical flaw. They raised some concerns about clarity and identified numerous ways in which the work could be extended. Reviewers also expressed skepticism of the usefulness, either because compression might not translate into practical gains, or because this architecture might lose various benefits associated with neural ODEs. 

My own impression is that the submission is pretty well written by the standards of a NeurIPS paper, and the methods are described clearly even if the motivation could sometimes be made more explicit. The reviewers' suggestions seem to me (as a non-expert on this topic) to be sensible ones which would indeed improve the paper, but any paper that tried to do all of them would be unreasonably expansive. Regarding significance, the proposed architecture seems like a natural one to explore, and a paper which does it carefully (as this one does) is making a worthwhile contribution. Therefore, I recommend acceptance despite the slightly-low scores.

Decision: Accept (Poster)