Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: 

This paper aims at defining a new architecture, Dynamic Recurrent Neural Network, that would be based on discrete differential equations of basic linear system transfer functions known from dynamic system identifiation. They show also an application example.

The paper is correctly written, and the subject is of interest. But I don't really see the point as it looks like the method removes all of what is 'recurrent', as no weights from the new unit are learned during the training. The learning is only on some fully connected layers afterwards: I feel that the authors just made an (interesting) 'feature extraction' instead of a recurrent unit, by using standard transfer functions without any weights to be learned (or haven't I understood correctly?).

The claim that the paper is the first to present a method where the sequence sampling rate, or 'delta t', is a parameter that can be modified without needed to re-train the network seems odd. In fact, I am not very familiar with this field but it looks to me that a lot of works that are now combining neural nets and dynamical systems are by definition able to do that. For example, I know this work 'Learning Dynamical Systems from Partial Observations', Ayed et al. 2019, which has a section called 'Benefits of Continuous-Time.', where I can read 'this allows us to accommodate irregularly acquired observations, and as demonstrated by the experiments, allows interpolation between observations.' So it is not something 'new' to the community and I would guess that papers closer to yours would also have the same feature?

Questions/ remarks:
- please explain better how your network is still recurrent : where is the 'hidden memory' that is passed? And where are the weights inside the DYRNN that are learned during the training? For me it looks like only the FC model placed after, that is moreover without any non-linearity (ReLU, etc), contains weights that are updated during training.
- 'and are state of the art layer types for text based sequence-to-sequence problems like machine translation or text processing --> no, the recurrent NN are not state of the art in translation anymore... transformers are. But RNN could be state of the art in other problems.
- 'The number of output channels per layer n layer amounts to (base component count * n oc = 5 * n oc ), as shown in Figure 3b.' --> not clear: there is 5 output channels, so noc = 5, but what is this 'base component count'? Why do we have to multiply it, and what is a layer? Or do you mean that there are 5 DYRNN units? or that every 'component' (P, I, ..) outputs the 5 outputs? This is not what is represented here.
- Fig. 6 might not be necessary, we can understand the concept without it.
- Modell --> Model

review_title: Interesting topic but the method seems to just be a feature extraction + fully connected layers.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The theory of dynamical systems has such a rich and extensive history dating to more than a century ago, that Poincaré himself would marvel to see what we can do now with modern technology.  Perhaps he’d not be pleased to see that extensive research in dynamical systems from the previous century is ignored in this paper, however the authors are aiming at reaching more interpretability on NN models as stated in the abstract.

I think the paper has definitely technical merits, but I remain puzzled with the lack interpretability section (3.4). I do not see how I can use the methodology to reconstruct the underlying physical processes that drive the physical phenomena. Since the authors are claiming in the abstract that interpretability is a distinctive feature of the methodology, I was expecting to see a full fleshed content, but it’s essentially just a couple of paragraphs.

These are the questions that I’d love the authors to address:
1/ Since the authors are inserting an explicit integration method by virtue of the Euler step through Eqs 3 to 8, the method inherently carries A-unstable. Can you estimate the conditions for any system that would prevent the A-stability risk?
2/ What are the differences of the methodology for Hamiltonian and dissipative dynamical systems? 
3/ If the methodology that the authors are proposing works, it should be the case that any transient trajectory can be explained by DYRNN. It means that a trained DYRNN on a small set of trajectories (or one sufficiently large one if we have an ergodic system), can fully reproduce any other transient trajectory of the dynamical system. In other words, produce a train and test trajectories and prove that the error on the test set is small enough.
4/ At the core of classical dynamical system theory is understanding the set of local and global bifurcations that explain the physical/chemical/biological phenomena. If DYRNN can predict the bifurcation points respect to a control parameter, then the method is rock&roll, otherwise it’s just another method.
5/ Interpretability: If the authors can map back the DYRNN to a set of ODEs/PDEs then I’d be fully convinced on the value of the method. Otherwise, I remain unconvinced on the value of the interpretability.

I am not ready to accept this paper in its current form, but if the authors could prove (1,2) or 3 or 4 or 5 then I’ll happily change my rating.

review_title: Where’s the interpretability exactly?

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper aims at proposing Dynamic Recurrent Network to understand the underlying system properties of RNNs. By first showing five basic linear transfer functions in dynamic systems theory, the paper formulates DYRNN units. To solve the increasing number of layers issue, they concatenate inputs and intermediate results before passing into an FC layer. It is interesting to see how adjusting
\deta t is related to the model’s robustness. Though not fully explained, this paper provides a method to partially explain the RNN insights through FC layers learnt weights.  

The paper is well written to convey the central ideas. The overall idea is interesting, and experiments are clear to demonstrate the proposed method. It will be better to test the method on some benchmark datasets so that it will be easy to compare with the state-of-the-art.

A small advice: do you mean varying instead of variing in section 6?
review_title: overall this is a good submission

final_score: rating: 3: Clear rejection
review_content: The authors present a method for incorporating basic concepts from linear systems theory into the standard structure for training artificial neural networks. They compare results of their approach against standard approaches for 3 simple datasets.

Broadly speaking the work is quite unclear, and takes several passes over to have a basic sense of the approach. There are too many shortcomings to enumerate them all, so I will just present one. Figure 5 is presented in the section "example architecture" which might lead one to believe the authors implement this network (which it appears they do not). I believe this is included only to indicate a hypothetical architecture, but the presentation is too poor to glean this with any quickness. This is of course, in itself, not sufficient grounds for rejection, but speaks broadly to the poor presentation of the work. It does not seem ready for publication.

As for the significance, the work clearly falls short. Although the motivation of constructing a "more explainable model" is a good one, this should not come at an extreme cost of model expressivity. It seems obvious that richer models, such as LSTMs etc., correctly trained, should be able to account for the linear transformations the authors include in their "novel layer." That their work is competitive with these richer models is simply an indication of the simplicity of the tasks they chose, which (as far as I can tell) can all be accounted for using linear systems analysis (although it's hard to say, since they work so poorly explains the second two tasks). It's completely unclear how effective the authors' approach would be over standard, richer models, on tasks that cannot be accounted for by linear systems analysis, and I am doubtful that the suggested approach could offer much over these richer models.

Likewise, an alternative view of the authors' work is as a learnable filter bank applied to data to create a representation of the data better suited for post-hoc learning with a richer model, which is certainly an useful idea, but it is not clear to me (and the authors haven't shown) that their choice for this filter-bank is superior to many other choices (e.g. convolutional layers applied prior to FC layers, which is standard for deep networks).
review_title: Clear reject. Lacks clarity. Findings not likely to be very general (weak significance).

Meta-review: This paper proposes a new RNN architecture called Dynamic RNN which is based on dynamic system identification.

Reviewers questioned the expressivity of the proposed model, practical application/impact of the proposed model, and interpretability of the proposed model. Even though the authors attempted to convince the reviewers, 3 out of 4 reviewers think that this work is not ready for publication. 

Specifically, R4 recommends 5 ways to strengthen the paper. I recommend the authors to incorporate this feedback and make a stronger resubmission in the future.
Decision: Reject