Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces an approach for capturing long-term dependencies of complex time series. The paper incorporates some ideas from the classical dynamical system modeling literature into machine learning.
strength_and_weaknesses: Strengths:
i) The proposed method is very comprehensive. It is tailored in fine details including computational feasibility.
ii) The paper reports a large list of experiments, compares against competitive modern machine learning approaches such as transformers, and appears to outperform all of them.

Weaknesses:
i) The method involves calculation of iFFT somewhere in the pipeline. I wonder how much it could preserve end-to-end differentiability of the eventual loss function if it is used together with encoders, for instance in high-dimensional sequences such as videos. 
clarity,_quality,_novelty_and_reproducibility: The paper is written in a clear and precise scientific language. The proposed method is novel in its own context, even though probably not as novel for the signal processing domain. The paper gives sufficient amount of details to reproduce the experiments.
summary_of_the_review: Solid piece of work. The approach is a bit different from the commonplace approaches of the core machine learning community, which could make its accessibility a bit questionable. On the other hand, this could also be an opportunity for the community to get exposed to ideas from different domains.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper generalizes the recently proposed S4 model with an input dependent time constant used in state transition matrix and conduct thorough experiments to illustrate its benefits.
strength_and_weaknesses: Strengths:
- Input dependent transition mechanism in state space models is a novel, important and natural extention of the original S4 model. I really like that this paper thoroughly explores it.
- The paper conducts thorough experiments on multiple datasets and get impressive improvements over the original S4 model.

Weaknesses:
The main set of weaknesses I see in this paper is related to computational complexity and cost.

- This paper can be much stronger with a simpler diagonal version of Liquid-S4. Some of the accelerators don't support (or are super expensive) the full set of operations required for implementing S4. 
- The paper also doesn't report relative increase in computational cost or wall clock time compared to S4 and other alternatives. I would be interested in knowing that.
- [Optional] Some of the the baselines reported on this paper on LRA doesn't match what's reported in the original papers e.g. S5 paper. Although, its possible that they were updated recently. It's likely that some of the improvements are orthogonal can improve results of this paper and make it even stronger.
clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and presents novel ideas.
summary_of_the_review: While the paper introduces very interesting ideas and impressive results, it can be made much stronger by also looking at computational costs attached to introducing these ideas.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an extension on the popular S4 model by allowing the state transition matrix to be input dependent. They show significant improvements on the state of the art on many long range sequence prediction problems. 
strength_and_weaknesses: Significant advance on the state of the art. I only have minor comments. 

1. Why is the same B used for modifying the state transition matrix and additive input in Equation 8 ? 
2. It is not clear why the best linearization of $A + Bu(t)$ equal to $\bar{A} + \bar{B}u(t)$. Ideally, the linearization must be done on $A + Bu(t)$ directly. 
3. It seems that the empirical analysis only focuses on PB mode. Why? Since the paper presents two modes, it is best to present results with both modes in all tables (Liquid-S4-KB and Liquid-S4-PB). 
4. It will be useful to get insights on the relationship between optimal $p$ and the "complexity" of $u$. Why does ListOps and IMDB require higher $p$ compared to Pathfinder and path-X datasets? Add Pathfinder and path-X to Figure 1 for completeness.
clarity,_quality,_novelty_and_reproducibility: Well written.

There seems to be typo on page 6 .. .. "to use parametrization of $\Lambda - PP^*$ instead of $\Lambda - PP^*$"
summary_of_the_review: The paper presents a signficant advance. Minor changes can make the paper clearer.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper combines liquid time constant models with the recently published S4 model.  This extends S4 to have input-dependent dynamics.  This can be cast back into the original convolutional framework, and the efficient Cauchy kernel can be re-used for computing the additional terms.  The method achieves good results on the standard benchmarks.  

strength_and_weaknesses: 
## Strengths
The method appears to perform very strongly on a good range of standard benchmarks.  The method also combines two existing methods nicely.  The (short) derivation of how the truncated liquid kernel can be computed, and its parallels to the S4 kernel are also nice.  Highlighting how (effectively) time-varying kernels can be used in S4 is nice.

## Weaknesses
My main criticisms with the paper are that it is well below the threshold in terms of presentation, some questions on the core of the model, and that the contribution feels a little incremental.

(a) There are numerous typos, typographical errors, inconsistencies etc that make the paper feel incredibly rushed.  I have tried to outline some of these below.  

The paper is also very wordy for what is quite a simple idea.  It feels like the authors have engineered paragraphs to cram dozens of citations in to make the work feel more thoroughly referenced; when really the work is just LTC models + S4.  For instance, listing the baselines and their citations in the paragraph “Baselines” is totally unnecessary (since they are also in Table 1).  The same criticism goes for much of Section 2.  

I think this paper could very easily be reduced to say six pages, and that a shorter manuscript would dramatically improve its impact.  

I also wished the authors separated S4 into its own background section.  Although the core concepts predate S4, the math in eg. (4) clearly is drawn from S4.  In that sense, I think “Background” should actually be called “S4”.  Similarly, the bullet points in “How to compute LS4 Kernel Efficiently” should also be in this background section.  

Really, I feel like your sections should be:  “S4”, “LTCMs”, and then the methods section is just “S4+LTCMs”.  This will dramatically simplify the exposition for the reader, and make it clear what is a novel contribution in **this** paper.  

(b)  Can the authors clarify some things in the relation to LCTMs for me:  You actually use the "PB" model variant in all of your experiments, which means that you aren't actually using the discrete-time LTC model in (9).  This should be highlighted in the table, where your method is labelled as "Liquid-S4", as opposed to "Liquid-S4-PB".  Is there a reason why you don't evaluate "KB"?  Secondly, I am unconvinced that the discretisation from (8) to (9) is correct.  The bilinear (or ZOH) transform is non-linear, and so discretising (8) is not as simple as discretising $A$ and $B$ separately.  I think the correct discretisation is $\overline{A} \cdot \overline{(Bu_k)}$.  This scheme would make computing the recurrence in (11) more difficult.  I am willing to entertain that I have mis-evaluated this, but this means that the model for which results are presented is actually quite a drastic deviation from the initially motivated LS4 model.  This is okay, but does mean that much of the prose up until (15) is a touch misleading.  I invite the authors to comment on this.  

I also query if the learned PB model can be iterated in a recurrent mode for autoregressive generation?  If this is not the case, then a major plus-point of the original S4 work has been lost.  

(c)  My final major criticism is that the work feels a little incremental.  If we believe the extra cross-terms in (linear) LTC models are useful, as prior work has demonstrated, then there is no reason to think they won’t improve S4 models.  Adding more flexibility into a model should improve performance.  The only “novel” work therefore is showing that these cross-terms can be cast in the same terms as the S4 update, which is nice, but is also not totally surprising given the nature of the cross-terms.  

The ablation presented in Figure 1 is useful.  I would like to see the ablation for all the datasets:  did increasing p degrade performance?  Or did performance plateau?  What happens if we consider p=100?  

Do any of the theoretical results from S4/DSS/S4D carry across to LS4?  If not, why not?  Is tying the $B$ matrices in the transition and input important?  

Side note on Figure 1:  why was p=4 not evaluated? 

Are you also able to clarify whether the figures in the table are the validation or test scores? 



## Minor Weaknesses / Typographical Comments.
(i)  Throughout:  Inconsistent use of \eqref{}, \ref{}, Equ. \ref{} etc. 

(ii)  Throughout:  The math is weirdly spaced, cf. (1) or the second line on page two.

(iii)  Throughout:  Only proper nouns should be capitalized.  (i.e. “State” -> “state”)

(iv)  Throughout:  $x$ (and other terms in select places) should probably be boldfaced as they are vectors.

(v)  Reference:  “KALMAN” -> “Kalman”

(vi)  Pg 2:  “parameter” -> “parameters”.

(vii)  Pg 3:  The logic in attributing the performance of S4 isn’t quite correct.  Numerical performance is derived from HiPPO.  Memory and runtime performance is derived from the NPLR representation.  

(viii)  Eqn. (8):  The element-wise addition is very non-standard.  I would include multiplication with a row vector of ones to signify this in math, as opposed to in the text.

(ix)  Alg 1:  The subscript “liquid” was in \mathrm elsewhere.

(x)  Eqn. (11):  The statement of this is far too informal.  It obfuscates the message.  Please write out the term fully in math. 

(xi)  Throughout:  “Structural” -> “structured”  (in the context of S4).

(xii)  Throughout:  Footnotes should be avoided at all costs.

(xiii)  Throughout (cf. footnote 2):  \citep and \citet are used incorrectly in places.

(xiv)  RAW and FULL do not need to be capitalised. 

(xv)  Citations are inconsistent in their use of initial, full first name, middle initial etc.  URLs should also be removed.  

(xvi)  Appendix B:  it is nice to re-state the proposition prior to proving it.  

(xvii)  Figure 1:  text-wrapped figures should be avoided. 

(xviii):  You used an extra layer (9), compared to S4d's eight, in the ListOps experiment.  Why?  Is it possible to re-run LS4 with eight layers? 

clarity,_quality,_novelty_and_reproducibility: The work is not especially well presented.  The prose is littered with typographical mistakes and sentences that don’t read especially well.  This makes the paper feel very rushed.  The prose is also very long in places.  The paper would benefit from cutting a lot of the “fat” from the main text (see (a) above).

The work is novel in the sense that no one has combined LTC models with S4.  Deriving the efficient form for the liquid kernel is nice.  However, the paper does have somewhat of an incremental feel to it.  

I believe that the work would be fully reproducible. 

summary_of_the_review: The results in this paper speak for themselves.  However, the paper offers limited additional intuition or insight beyond “existing LTC ideas can improve S4”.  I also have reservations that the LTC ideas are carried across correctly as is currently presented.  This, coupled with the below-par presentation mean I am scoring this paper as a reject.  I am willing to upgrade my review if that authors can allay my concerns.  

I also note that this is one of as many as six S4 extensions submitted to ICLR.  I would expect to see these at least commented on (although certainly not directly compared to) in a revised draft.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper extends the S4 model by allowing the state transition to be input-dependent, thus improving results on long sequences. 

On the positive side, the paper is supported by a large number of experimental results.

On the downside, one can turn that around and say the paper is incremental and heavily experimental.  It does not add to science in a big way.  


justification_for_why_not_higher_score: see above
justification_for_why_not_lower_score: Indeed, I was inclined to go for a reject at first.  Rebuttal of the authors was convincing though, and the final paper will get enough attention to merit publication.
Decision: Accept: poster