Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on identifying and empirically verifying the ways that Neural ODEs can be regularized towards improved generalization. Sparsity in weights and features are explored in particular, where a new regularization technique called PathReg is developed as an analogue to other common regularization techniques in deep neural network training. There are experiments performed on mocap and RNA-sequencing datasets to assess the generalization capabilities of continuous-depth neural networks.

strengths_and_weaknesses: ## Novelty, relevance, significance
The work is a thorough empirical survey of the proposed PathReg method which is a unification of L0 regularization (which specifying the weights in a neural net) and LassoNet (which selects features based on how much an input affects the output) with residual weight regularization. The loss is modified in the usual way to include these extra regularization terms and generally the results point at the fact that combining feature and model selection, building upon improvements shown by either methods already achieved independently. 

I generally find data augmentation types of papers difficult to rate in terms of high novelty. It also makes sense that better out-of-distribution generalization is achieved when the model is not trained to over-fit to in distribution and hence can better ignore spurious features.

## Soundness / Correctness & Clarity
The derivations for the new loss and adaptation of existing two methods for regularization / sparsification are generalized correctly to the NODEs setting. L0 regularization of terms is also standard and hence sound.

## Quality of writing/presentation
Writing and presentation are well-done. Figures are clear and labeled.

questions: Do these changes lead to more favorable ODE solvers over others? What are the speedups in general?

Would you be able to compare again other neural ODE techniques and summarize their differences.

limitations: Not discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors propose a novel feature regularization method called PathReg for NODEs, a neural net architecture that can reconstruct and extrapolate time series data. This method allows the NODE to learn parameters with 2 types of sparsity: 1) weight sparsity and 2) feature sparsity. While methods exist to regularize NODEs with feature sparsity (C-NODE and Group Lasso), there is currently no approach that also integrates weight sparsity. The primary contribution of this work is to first adapt two methods for weight sparsity in regular neural networks to NODEs (L0 and LassoNet) and then combine both types of sparsity in PathReg. The authors evaluate their method on a set of convincing simulated and real datasets against several baselines using reasonable metrics. Furthermore, they provide some interesting insight on the data by interpreting the feature correlations derived from their model. The results and experiments described in the manuscript are quite promising. However, there are several claims that need to be clarified and additional experiments that would be needed to support claims or "nice to have" to improve the paper. Details are given below.
strengths_and_weaknesses: (1) 
First, we very much appreciate the author’s thorough evaluation of the existing literature in this space and the clear statement of their contribution. It was easy to recognize exactly how the author’s are uniquely innovating in the context of other approaches in the field. 
(2)
The author’s describe several advantages of sparsity. These include: 1) improved speed and efficiency, 2) better generalization, and 3) improved interpretability. 
(a)
Would it be possible to show runtime benchmarks for each method? The reason for requesting this benchmark is that C-Node performs comparably (and sometimes better on certain datasets) on the reconstruction metrics. One reason to justify the use of PathReg would be the runtime. Is it the case that learning fewer weight parameters by combining both feature and weight regularization makes the approach faster? 
(b)
Is it possible to perform a similar interpretation on the A matrix in C-NODE as was done on the A matrix in PathReg? If not, can the interpretability of C-NODE and PathReg results be compared in some other way?
(3)
It is not completely clear what the overarching premise of combining both feature and weight sparsity actually achieves in practice. In other words, why should one care that the weight metric is higher for PathReg in cases when the generalization error is comparable. 
(4)
It is not clear what Method Ranking and Overall Score mean in Figure 3 and 6. Regardless, these are not useful metrics and could be discarded to make the figures more clear. 
(5)
It is not clear what path weight means (x-axis in right most plot in Figure 2).
(6)
The interpretation of the correlation matrix A for both the motion capture data and single cell data are quite exciting. We would encourage the authors to consider adding some examples to the main text and to elaborate on subjective claims such as “learn sensible and arguably correct adjacency matrices (derived from joint connectedness)”. For example, were leg and arm joints more connected in the walking data (where both appendages move) compared to the golf data (where legs do not move). In general, we did not find Figure 4 or Figure 5 (top) to be particularly useful for illustrating the main claims in the paper and could be replaced.
(7)
The C. elegan embryo development dataset in Packer et al. 2019 has single cell time series data with experimentally derived time. This could be used instead of the inferred pseudotime. The approach used to infer pseudotime is also not currently state of the art. Please refer to Saelen’s et al. 2019 for thorough benchmarks of over 70 methods. 
(8)
RNASeq batches should be corrected for batch effect. Consider methods like COMBAT or RUVSeq. 
(9)
It is not clear how spurious features are defined in Figure 6. How did the authors determine that a pair of genes did not directly affect each other’s expression? What does it mean for a transcription factor to be inferred by PathReg as “important”? One approach to determine whether certain TFs affect other genes is to take knockout data (such as from ENCODE) to perform differential expression between control and TF KO data.
(10)
Please elaborate on the contribution for releasing data. What aspect of the data was modified to warrant re-release under this publication? 
(11)
The Base model has significantly worse generalization error compared to other methods for gene expression. This was not so much the case with the motion data where the generalization error was comparable to the other methods for some datasets. This is quite an interesting result. Why do the authors think this is the case and can their model’s improved interpretability aid in figuring out the answer?  

Minor errors and comments:
-----------------------------------------
In the Figure 1 feature sparsity figure, shouldn’t x0 in the input be connected to x0 in the output? How can it be possible for a feature to have no effect on reconstructing itself? Even though this is a cartoon, this is a bit misleading.

Typo line 241: PsthReg -> PathReg

Perhaps highlight that S1D2 was the training data in Figure 6

Is the feature sparsity % of exactly 75.52 duplicated multiple times in Table 1 correct? 

questions: See above.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes modifications (suggestions) to improve the performance of the Neural Ordinary Differential Equations towards better generalization. Specifically, for the neural network based function in X’=f(X, t), they propose a regularization technique for the weights of `f’ based on a combination of path-norm and L0 norm. The proposed regularization targets inducing feature sparsity, which ensures that the output variable depends on a few number of parents X_i’ = f(pa(X_i, t) and also the weight sparsity, which is traditionally shown to help with the generalization capability of the neural networks. This work primarily introduces a differentiable L0-based regularizer (PathReg) and extends it to NODEs. The authors argue empirically that the technique proposed by them improves over the state-of-the-art methods over a range of time-series tasks. 
strengths_and_weaknesses: (previous work summary) C-NODE uses the upper bound of L1(path-norm) and GroupLasso uses the group lasso norm as regularization. LassoNet introduces a skip connection and then enforces sparsity by putting constraints on the skip connection weights and the first layer of the feedforward NN. 

Before I can assess the quality, novelty and impact of the work, I request the authors to address the points raised in the Questions & suggestions section. I am not able to follow certain aspects of the work. 

Update: After the clarifications provided by the authors, I have updated my ratings.
questions: Questions:
1. (Line 175) $W_j^(1)$ represents the first layer weights of $h_\theta$ ?
2. What are the choices of the distribution `q’ used for the L0 reg? 
3. I understand that the equation 10 & 11 enforce the same constraint. In eq. (11), can you please expand on the form of the distribution $q^l( z | \phi)$ ? I believe that the $\phi$ params will be learned during the training process. 
4. In equation (12), the expectation term samples `s’, as per my understanding, gives a set of G’s. Is the ||A_z|| term changing with every sample? How does the $\phi$ term come into picture here? 

I request the authors to make these above points clearer in the paper as well. I went through the supplementary material too but I was not able to find clear descriptions of the procedure. 

Experiment suggestions:
5. The \lambda_1 ||A||_1 term is introducing weight sparsity. How does the performance change if this term is dropped and only \lambda_0 ||A_z||_1 is used? 

limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a method PathReg to improve sparsity in NODE models. The authors backup their claim that PathReg improve sparsity and performance by experiments in 3 different datasets.
strengths_and_weaknesses: - The figures are nice and well-represent the experiments
- The effect of sparsification of PathReg is significant
- The performance of PathReg is descent. 

questions: - The design of PathReg seems to be fairly straight forward with feed-forward networks. What is the reason you choose NODE-based models over general feed-forward networks?
- Gating are dependent on initialization so probably multiple experimental runs are needed for an accurate evaluation. How many times have you run each of your experiments?
- Gating could also potentially slow down the training process. What is the time consumption compared to other models?
limitations: As far as I know, there will hardly be any negative societal impact of the work.
review_title: null

Meta-review: metareview: The paper proposes a new sparsity inducing regularization scheme for continuous-depth neural networks. The reviewers acknowledged the relevance of the proposed method and generally appreciated the results. The paper is nicely written and provides a range of interesting experiments demonstrating the effectiveness of the proposed method. I want to thank the authors for their detailed responses that helped in answering some of the reviewers' questions. (The reviewers have provided detailed feedback in their reviews, and we strongly encourage the authors to incorporate this feedback when preparing a revised version of the paper. I also would like to encourage the authors to carefully revise the related work section on  continuous-depth neural nets to better acknowledge work that has appeared in the last 2 years.) In summary, the feedback of the reviewers is positive and thus I recommend accepting this paper.
confidence: Certain
Decision: Accept