Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a continuous time version for Fast Weight Programmers family of sequence classification models. It presents continuous time analogues to Hebbian learning rules. It introduces continuous time versions of multiple FWPs variants. The continuous time versions are empirically tested three standard sequence classification tasks, and compared to other sequence classifiers. In the appendix, the method is also applied to two basic RL problems.

strengths_and_weaknesses: The paper is overall well presented with a thorough explanation. The continuous analogous of FWPs is an original contribution to the literature of neural ODEs. A particular novelty is the consideration of Hebbian learning methods.

A wide variety of experiments are performed, which is a strength of the paper. However, the main weakness of this paper is that each experiment only has one baseline method as comparison, and does not have enough details, which makes it unclear if the proposed method is significantly better than existing methods. 

This is a minor aspect of the work: the “memory saving” adjoint method is mentioned as a “pro” of their method. A questionable claim is “Memory Efficient Backpropagation for FWPs”. The “memory-saving” adjoint method only works if the equation is reversible (not all ODEs can be integrated in both directions c.f. heat equation). Otherwise, checkpointing of the forward pass of the ODE is required.   (See, e.g. Zhuang et al. “Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE'' 2020). Additionally, the baseline methods could also use it (with the same error) so it is not a pro unique to their method. 

questions: ### Experiments:

In the experiments, why were these particular baselines chosen? Why were only ODE based methods chosen as comparison? Could a non-ode method be included in these problems?

Tables 1 & 2:  Are the models comparable? What are the parameter counts of each row? What are the runtimes of training and inference? What are the architecture hyperparameters? 

Table 2: Are the results for “Direct NODE” or “NCDE”?

Line 264: Did the authors perform the experiment comparing NODE and NCDE equations for this problem? It is not clear from the text or results.

### Usage of “memory efficient” adjoint 

As mentioned, it is not clear that the checkpoint-free adjoint method can be used to these equations. Are the equations of the continuous FWP models reversible, and guaranteed to remain so?

Is the adjoint method only applied on the fast weight learning rule, or the entire network?

Did you compare using the checkpoint-free adjoint method vs. regular backpropagation in your examples? 

Were the baseline methods also able to use the “memory saving” adjoint method for training?

### Clarifications

Line 211: Why is it not practical? Is it because of the number of parameters, or computational cost, something else?

Line 217: Is the Latent ODE RFWP used anywhere in this work?
Line 218: This last sentence is confusing. “While this case” Which is “this case”? Does this paragraph mean to say that all of section 3.3 is not of central interest? 

Line 263: What is an RDE?

### Minor

This is a very superficial comment, but the paper title makes it difficult to understand what the paper is about. The phrase “Learning to Program” suggests a different field of program synthesis or general optimization algorithms, which does not fit. The title could be simplified.

limitations: The authors focus on how their method mitigates limitations of the original family of methods. The authors have some discussions about the cost limitations of the family methods, and hypotheses on the mechanism by which their method performs well with low-rank methods when previous variants do not.

There are no potential negative societal impacts that are particular to their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose several parameterisation of the vector field in neural ODEs/CDEs/ODE-RNNs based on FWP and learning rules.
strengths_and_weaknesses: **Strength: topic**

There has been relatively little convincing work on good architectures for RNNs, NCDEs, etc. This work addresses a meaningful problem.

**Strength: originality**

To the best of my knowledge the ideas presented in this paper (combining NCDEs with FWP, etc.) are new.

The scalability, as compared to the original formulation of neural CDEs, is an important contribution.

**Strength: literature review**

This work ties together many individual lines of work -- neural CDEs, neural RDEs, long sequence modelling, learning rules, FWP, etc. -- and so far as I can determine does an excellent job providing complete references for all of them.

**Strength: experiments**

The experiments look to have been carefully conducted. They go into a great deal of detail. Moreover the results obtained demonstrate meaningful improvements on existing benchmarks.

**Strength: presentation**

The prose is carefully worded and precise. I enjoyed reading this paper.

**Weakness: experiments**

I believe the experiments are overly-focused on neural CDE and neural RDE comparisons. Whilst these are the natural choice given the framing of the paper, I think it would improve the paper to include more quantitative comparisons against other works, e.g. [0], [Ref. 34, 46, 47 from the paper]. Indeed there have been quite a few papers proposing novel architectures for RNNs/etc.

**Weakness: presentation**

At least for me -- I am largely unfamiliarS with the work on learning rules, FWP etc. -- I had to wait until the experimental section to determine that what is being proposed is essentially a new architecture, for the purposes of supervised learning etc. I would suggest explicitly stating this kind of thing up-front.

**Weakness: "the adjoint sensitivity method" (line 66)**

The term "adjoint" has become heavily overloaded. In the current literature is is often used to mean specifically optimise-then-discretise, but in older literature it was often used to mean specifically discretise-then-optimise. Following [1, Remark 5.5] I recommend avoiding it altogether.

In passing, since backpropagation is performed via OtD, then the authors may find speedups by utilising the technique of [2].

**Weakness: numerical/software**

The choice of computational framework, and choice of numerical solvers, seems to go entirely undiscussed. (I did not see it mentioned in the appendix either.)

This is a meaningful limitation: the choice of numerical solver, its choice of tolerance, etc., can greatly affect results.

**Minor issues**

Line 33: I wouldn't use the word "ancient". Some of us were around back then!

Line 78: I would argue that natural cubic splines should not be mentioned here. As [Ref. 31 of this paper] shows, there are now much better alternatives, so I think it would be better to avoid discussing natural cubic splines altogether. (e.g. they are not even implemented in [3])

**Commentary**

Line 85: I'm not sure I completely agree with this analysis. As per the next heading ("Second") then the absolute value of $x$ may be treated as an input as well, by recording and replaying it. In practice it's not been demonstrated that this actually occurs in practice, so I think it possible that this paper's side-note may be true in practice, if not in theory.

I quite like the idea of directly providing $x$ or $x'$ to the vector field, in addition to leaving $x'$ outside, as in equation (23). Most works have usually framed the inside/outside placement of $x$ or $x'$ as a dichotomy.

**References**

[0] Coupled Oscillatory Recurrent Neural  Network (coRNN): An accurate and (gradient) stable architecture for  learning long time dependencies, Rusch and Mishra, ICLR 2021

[1] On Neural Differential Equations, Kidger, Doctoral Thesis, University of Oxford 2021

[2] "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms, Kidger et al, ICML 2021

[3] Diffrax, https://github.com/patrick-kidger/diffrax
questions: I have direct questions for the authors. I have made some suggestions above; to highlight them:

1. I think the experiments could be improved by moving beyond just the CDE/RDE case.
2. The choice of numerical method and software should definitely be discussed.
limitations: There are no ethical concerns with this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work introduces variants of Fast Weight Programmers, where the Fast Weight matrix W evolves via neural DEs, with vector field structures inspired by learning rules (namely Hebb, Oja, and Delta). These can be applied to continuous learning tasks. 

The structure of the neural DEs here fall under a subclass considered by Kidger et al in the paper introducing neural CDEs (that is, the neural CDEs subsume these methods). However, this paper demonstrates empirically that the specific restriction to these models seem to perform well compared with optimising over the broader class of neural CDEs, as well as providing scalability benefits.
strengths_and_weaknesses: This is a very nicely written paper, which presents the work done in a clear and concise manner. The background and literature is explained well. There is full transparency, the code is available for validation and reproduction of the results.

An observation is that this is pooling together a few ideas from different areas, so the improvement is more incremental, however this is not necessarily a weakness, and the empirical results and benefits seem to be good.

In terms of weaknesses, although it is not realistic to be comparing with all models out there, this paper does not discuss that in Kidger et al, the model that performed best for sepsis no-OI is in fact GRU-D, with a performance of 85% plus/minus 1.3, which is superior to the best performance of the three models introduced in this paper.

Furthermore, most of the comparison here is with Kidger et al. (2020), which has been superseded by Morrill et al. (2021) in Neural CDEs for online prediction tasks. It would be good for come comments and comparison with the results in the latter.

Some minor issues:
"ancient" for the 80s/90s can be slightly offensive!
Eqn (27) should be RFWP?
Table 1 should be AUC rather than AUR.
questions: In Section 3.3, it is assumed that the continuous control is not available and only discrete observations are accessible. In practice, this is almost always the case, in the paper by Morrill et al. [31], there is an extensive discussion on the construction of a continuous control path from the discrete observations. Perhaps the authors can highlight and clarify, when might one choose to take the discrete observations and apply methods in Section 3.3 rather than apply interpolation and use Neural CDEs?
limitations: The authors have discussed that some of their results have rather large variances, in particular Delta achieved the highest single seed performance for sepsis no-OI, but on average Hebb performed best in this case.

As discussed in weaknesses, the authors do not discuss comparisons with other baselines from non-ODE based neural networks. A discussion on where this work sits in the broader literature and any limitation of application of these new networks in broader problems would be beneficial.

Negative societal impact not applicable for this paper.
review_title: null

Meta-review: metareview: A novel blend of linear transformers and "Fast Weight Programmers" is proposed where slow weights generate an embedding and learning rule parameterizations of a Neural ODE based evolution of "fast weights" of another network. The new architectures appear to provide superior performance on several benchmarks in comparison to NCDE/NRDE benchmarks. The reviewers agree on the core contributions and recommend strengthening some presentation aspects of the paper, possibly reorganizing the main and supplementary materials so the paper is more self-contained.
confidence: Certain
Decision: Accept