Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the nonlinear dynamical system on time series data. It proposed a piecewise-linear (PL) recurrent neural network (RNN) and attempted to couple it with dendritic computation, a computing paradigm from the neural computation discipline. The training method of the proposed model is through variational inference, and BPTT with teacher forcing.

main_review: The motivation of the model design in this paper is ambiguous. The proposed model is a simplified hybrid linear and nonlinear dynamic system. The reasoning of coupling this model with dendritic computation is not clear. The claimed interpretability of the proposed model lacks both theoretical and experimental justification. The tractability of the training and inference method seems to stem from the VI and BPTT, thereby making the paper lack technical contribution.

Eq. (2) and (4) refer to the same function denoted by \phi, which incurs confusing definitions. Which function is exactly used in the model? 

The proposed dynamical system is termed with piecewise-linear RNN. In time series analysis, "piecewise" often refers to the method that separately models the segments of time series [1]. In this paper, the implication of "piecewise" is not well explained.

One of the training methods applied in this paper is based on VI. Since VI on time series/sequence is heavily studied in recent research works, e.g. [2-5], the authors are expected to detail the design of their VI based training and the comparison to existing methods. 

In the experiment section, the result is unconvincing in that the proposed method failed to exhibit consistent out-performance on the metrics/datasets, i.e. mostly only on half of the datasets, the proposed methods can outperform.  

[1] 2007, SDM. A better alternative to piecewise linear time series segmentation

[2] 2015, NeurIPS, A Recurrent Latent Variable Model for Sequential Data

[3] 2016, NIPS, Sequential Neural Models with Stochastic Layers

[4] 2017, AAAI, Structured Inference Networks for Nonlinear State Space Models

[5] 2021, ICLR, Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models

summary_of_the_review: The motivation of this paper needs more clarification. The proposed model is a simple hybrid dynamic system and the training method is based on existing VI and BPTT and thus the paper lacks novelty and technical contribution. The experiment is not solid enough to support the claim advantages of the model, i.e. interpretability and approximating ability.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to modify the nonlinearity of neuron couplings in PLRNN with a linear spline basis expansion. This modification is claimed to boost PLRNN's capacity of capturing arbitrary nonlinear dynamics in low-dimensions. The paper also introduces two training frameworks, variational inference and BPTT. The proposed method is evaluated on various nonlinear dynamical systems.
main_review: Post-rebuttal: Thanks for addressing my concerns. The authors have updated the manuscript and added experiments. I will not change my recommendation.

#####
Pros: 

+ Capacity to capture nonlinear dynamics

+ Competitive performance
 
+ Simple form and better interpretability 

Concerns: 

* As a modification to PLRNN, PLRNN could be a baseline and comparing to it would be confirmative.

* SINDy and LSTM require fully observed data. Do the authors consider to compare with other SSM methods like LFADS?

* To demonstrate the expressive power, apart from chaotic attractors, there are many dynamical systems such as fixed attractors, continuous attractors and etc that are commonly used to describe e.g. neural computational (Wang 2006, Mante 2013). Such experiments could be useful for qualitative demonstration. 

* Lack for real-world data example.
summary_of_the_review: Overall, I vote for accepting. The paper is well written. The method is evaluated on various data and compared to other methods. Hopefully the authors can address my concern in the rebuttal period.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed an extension of PLRNN with a dendrite-like formulation of the nonlinear activation function. They showed that the proposed dendPLRNN maintains the mathematical tractability and dynamical systems interpretation as PLRNN. They compared with alternatives for DS reconstruction with an overall better performance.

main_review: I think the paper is well-written and easy to follow. The spline basis extension is straightforward and explained well by comparing it with PLRNN. However, I think the overall novelty is not that strong enough. The spline basis expansion is quite common in many problems, like in mixture models, instead of using one model, people extend to mixture models with a linear combination; in kernel study, instead of using a single kernel, people use a mixture of kernels. In early papers such as [Chan et al ACC 1998], people have studied spline bases to model the nonlinear relationship between input and outputs in RNN. Splines are a common choice of basis functions to capture nonlinear functional relationships in a variety of neural systems [Qian et al Neural Computation 2018, Huang et al Neural Computation 2009, Frank et al Journal of Neuro 2002]. Therefore, although spline basis performs well in the context of PLRNN, I don't think the technical contribution is much significant to the community of ICLR.

Some detailed comments:

1. It's claimed that "we can essentially reformulate a high-dimensional PLRNN in terms of an equivalent lower-dimensional dendPLRNN". I think the equivalence is only true when the high-dimensional PLRNN has a unique structure of the connection weight W as shown in eq. 40. Therefore, although a dendPLRNN can be formulated as a high-dimensional PLRNN, the opposite direction is not always true. 

2. Following point 1, at the end of page 6, "it becomes clear that the basis expansion enables to reduce the model’s overall dimensionality without compromising performance." I think this is only true if the structured assumption of W is valid. If the authors want to compare PLRNN and dendPLRNN in the same space, e.g., M*B, it's needed to show that how much approximation error there will be. 














summary_of_the_review: The spline basis expansion works empirically but the technical novelty is not strong enough.
review_title: null

Meta-review: Inspired by dendritic nonlinearity, this paper extends previous work on PLRNN/PWL dynamical system modeling by Durstewitz's group. The extension replaces the ReLU nonlinearity with a linear combination of ReLUs. This preserves the theoretical properties of PLRNN, however, the dimensionality of the latent dynamics remains the same, increasing the expressive power of prior PLRNNs. I (area chair) actually read this paper since not all reviewers provided high-quality reviews and one key reviewer is having a personal emergency. Though I appreciate the premise, detailed numerical evaluations, and the inference approach, the novelty is marginal and I do not buy the theoretical advantage of this class of models as presented (see below). Therefore I cannot recommend this paper to appear at ICLR at this time.

Some additional weaknesses that reviewers did not point out:
1. Dendritic nonlinearity is summarized as a point nonlinearity; It lacks the interesting phenomena of dendrites such as nonlinear summation and calcium spikes with its own internal dynamics.
2. The many analytical properties of PLRNN may sound nice on paper, but very impractical. To search for the fixed points and cycles, the amount of required computation exponentially increases as the number of neurons and cycle length increases. In addition the boundary effects cannot always be ignored. In general detailed analysis can become quite non-trivial quickly, e.g., https://arxiv.org/abs/2109.03198
3. High-dimensional PLRNN that approximates a low-dimensional dynamical system due to model mismatch won't have the same topological stability structures. Theoretical analysis of higher-dimensional DS may be very misleading.
Decision: Reject