Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: The S4 model is a potentially new up and coming work that improves existing efficient transformers by leaps and bounds on the LRA benchmark. When the S4 model came out, it was a daunting read and not very intuitive since it is quite mathematically dense. Many folks that I have talked to about the S4 paper admitted confusion about the intuition behind the S4 paper. 

Here comes this timely blog post to the rescue! Not only does this blog post is of extremely high quality and value in writing, it actually comes with full code that is so clearly annotated that you wonder why not more people are doing this! 

Notably, this submission is not exactly a commentary but more of a tutorial-style blogpost but is very very very helpful to the community. I also like the fact that the authors use JAX and justified their choice of JAX. 

I think this manuscript totally embodies the spirit of the ICLR blog track. When I think of the ICLR blog post track, this type of work immediately comes to mind. 

I strongly vote that this paper to be accepted and if there is a best paper award of any sort for the blog post track - that this paper to be considered for it. 

Admittedly, it was hard to not chance upon this blog post on social media and as a result, I already know the identities of the authors as a result of this. However, I declare I have no professional/personal relationship to the authors of the original S4 paper and also the authors of this blogpost. 

This review comes from a satisfied customer, as this blog post has already benefited me in the understanding of the S4 work.

Some minor comments:
1. Is the code and functions in the paper or code adhering to any Python style guide. Trying out some of the code sends our linter into frenzy. Just a nit here and also perhaps different people use different style guides but the casing on some of the functions like make_NPLR_HiPPO is kind of weird that it has staggered alternating lower and uppercases.
2. Might be more value to have a simple LM example in the main blogpost over MNIST? since generative modeling of images is already used as an example...might as well just do a LM task?

 









review_title: The gold standard for a submission in the blog post track

final_score: rating: 6: Marginally above acceptance threshold
review_content: # Overall

**Is there a significant added value in comparison to the cited papers?**

Yes, and the added value is not significant enough. This blog is the first step towards this goal of gaining intuition, linking concrete code implementations with literate explanations from the cited ICLR paper. However, the code link is already provided in the ICLR paper and the structure of this blog is just a reconstruction of the cited ICLR paper. 

**Is this added value supported by accurate, convincing, and clear arguments?**

Yes, and this blog also implements S4 and makes it in practice, e.g., S4 can operate as a CNN for training, but then convert to an efficient RNN at test time, and can also generate images from pixels and sounds directly to preview the results. 


# Details

**Contribution**:

This blog is well-written and easy to understand. 
S4 paper has released its codes at https://github.com/HazyResearch/state-spaces. This blog has implemented the code and provided explanations. 
In fact, I think the blog authors have revisited the ICLR paper by mapping

* "BACKGROUND: STATE SPACES" into "Part 1: State Space Models (Modeling)"
* "METHOD: STRUCTURED STATE SPACES (S4)" into "Part 2: Implementing S4 (Advanced)"
* "EXPERIMENTS" into "Part 3: S4 in Practice (NN Implementation)"


**Limitations**: 

I think the first shortage lies in **Clarity**. 
The equations are all in latex grammar while not being parsed, and they lack readability. 
Besides, there are also some garbled words in this blog. 

The second shortage lies in **Originality**.
As the code is provided by ICLR paper authors, the blog authors just implement and explain it. The blog is also just a reconstruction of the cited ICLR paper. 


**Doubly Anonymous Reviewing**: 
I have already seen a [blog](https://srush.github.io/annotated-s4/) post with author names online which is almost the same as this one. 
I doubt if it is allowed for anonymous reviewing. 

review_title: A blog on annotating the model and code for the cited ICLR paper (S4)

Decision: Accept