Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an offline reinforcement learning approach which is able to capture longer range dependencies than a traditional sequence modeling approaches such as a Decision Transformer (DT). Similar to DT, Decision S4 views RL as a sequence modeling problem, but using the implicit S4 model instead of attention blocks. The offline approach is trained by taking individual transitions and rewards to go, and predicting the actions. The paper also introduces an offline-to-online training approach by first freezing the actor and then the critic, as well as freezing the S4 kernel. Results show that a much smaller model can achieve a similar performance to DT on offline RL tasks. 
strength_and_weaknesses: Strengths: 

- To my knowledge this paper presents a novel approach to tackle offline RL
- Applications of the S4/LSSM architectures to control and RL are under-explored and could be beneficial for the community 
- S4 does provide almost similar performance to DT with fewer parameters 
- Offline-to-online finetuning approach works quite well 
- paper is well written 

Weaknesses: 

- I don't particularly understand why the offline training is capturing long range dependencies, if it is predicting a single action at a time conditioned on the current state and reward to go? Where is this captured? It's important to explain this well. 
- How long does it take to train the model vs DT and what is the inference time vs DT? 

I would be willing to increase my rating if the questions above are answered. 


I would be willing to incre
clarity,_quality,_novelty_and_reproducibility: Paper is clear, presents a novel approach and is well written. 
summary_of_the_review: This paper provides an interesting use of the S4 parameterization and shows that the method works well compared to alternatives (such as Decision Transformer). Details and explanations about what the S4 blocks actually capture in the context of RL are missing which would make the claims in the paper stronger.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes replacing Learning (RL) methods based on Transformers with the S4 family of models. Two algorithms are proposed for both the on-policy and off-policy setting, showing strong empirical results.

strength_and_weaknesses: **Strengths:**
- The method is well-motivated. RL can require both efficient batch training as well as efficient step-by-step unrolling, and makes sense as an application of state-space models (SSMs) which can leverage both an efficient parallel training (convolutional mode) and recurrent inference (recurrent mode) of SSMs.
- The on-policy algorithm involves a methodological innovation that involves training with the recurrent view of S4, which was previously only used during inference time.
- The empirical evaluation is comprehensive and the model performs well on standard benchmarks with many established baselines.
- There are extensive empirical ablations, such as on the importance of long-term dependencies, the comparison against more basic RNNs, and on the model size.

**Weaknesses:**
- The methodological innovation is limited. However, the paper is clearly positioned as an empirical work that applies a recent model to a new domain, so this is not a significant weakness.
- It isn't clear how much tuning is required for the proposed method over DT baselines. Also, it isn't clear how much of the proposed algorithms is tailored to DS4; it could be even more compelling if the same algorithms worked for DT but improved with DS4 as a drop-in replacement.


clarity,_quality,_novelty_and_reproducibility: This paper focuses on empirical rather than technical innovation, and from this standpoint is clearly written and positioned. The experiments are extensive and thorough, including many useful ablations analyzing the appropriateness of the proposed method to the problem, as well as the importance of various components of the method.

summary_of_the_review: This work applies a recent method (S4) to a new domain (RL), showing promising results in performance and efficiency.

--------
Post-rebuttal: The authors added substantial new content in Appendices explaining several design choices and contrasted the tunability of the model compared to baselines. More improvements have been applied, increasing the performance of the method. All my concerns are addressed and I have raised my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for the recent decision-making-as-sequence-modeling paradigm. It follows the recent trend of treating RL as a sequence modeling problem: given the state-action-reward history, the current state, and the desired reward-to-go, predict the next action. The proposed method is to replace the commonly used transformer model backbone with S4 (Gu et al., 2021) for better long-range sequence modeling capability and reduced model size. The paper also proposes a offline-online learning scheme where the actor is trained with offline data using the decision-transformer-like objective and finetuned online using an actor-critic algorithm.
strength_and_weaknesses: Strengths: Overall I like the motivation of the paper: a lean model that can capture long-range dependency is desirable for any RL agent, and S4 seems to be the perfect backbone for the job. The method proposes a relatively straightforward drop-in replacement of recurrent S4 architecture for the more heavy-duty transformer backbone and shows that the new architecture performs favorably both in task and computational performance. 

Weaknesses: 
- To me, the offline-online learning scheme is an awkward orthogonal component to the work that’s neither novel nor tightly connected with the core DS4 method. It appears that S4 architecture is not uniquely attuned to online finetuning (compared to transformer / regular RL agents)? And the proposed online finetuning scheme is not tailored to the DS4 architecture either. And the online finetuning is not mentioned in other aspects of evaluation, e.g., long-range dependency and parameter reduction. In the future iteration of this paper, I’d suggest the author remove this component and focus on testing S4 against more challenging domains such as domains that require strong long-term memory.
- The paper claims runtime latency to be a major advantage of S4-based architecture. I'd like to see some empirical results that substantiate this point.
- Minor: the writing needs improvement. Most sentences are way too long for the message they carry. 
clarity,_quality,_novelty_and_reproducibility: Quality: solid execution of an conceptually simple idea
Clarity: Structure is good. Writing is ok. Can be improved
Originality: to my knowledge applying S4 to DT-like model is novel.
summary_of_the_review: Overall I think the paper is a solid execution of a conceptually simple idea, although some part of the method is not strictly necessary and may muddle the key message of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes Decision S4 (DS4) by replacing the Transformer backbone of Decision Transformer by S4. In addition, it also introduces an offline-to-online training method. The experiments are performed on AntMaze-v2 and some MuJoCo tasks (Hopper, HalfCheetah, and Walker2D) and showed that DS4 outperforms DT.

Strength. S4 is a new sequence modeling approach which does not currently have many application success stories yet. So this is good to know that S4 works for offline RL tasks. It is an important knowledge to the community both S4 and RL. Thanks to the efficiency of S4. The proposed method is 5x faster than DT and also requires smaller memory footprint. The paper is clearly well written. Also contains extensive ablations studies. Also in the rebuttal the authors said that it does not require much hyperparameter tuning.

Weakness. The contribution might still seem incremental if we put this work into the framework of simply replacing Transformer in DT by S4. Some reviewers see that the offline-online learning scheme part is an awkward orthogonal component, but I think the author's response makes sense. It's good to see its working in the online setting as well.


justification_for_why_not_higher_score: The novelty can be seen incremental.
justification_for_why_not_lower_score: The result is worth sharing with both the S4 and RL communities.
Decision: Accept: poster