Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: This work proposes Adaptive-Saturated RNNs (asRNN), which combines the capacity of vanilla RNNs with the training stability of orthogonal RNNs. The asRNN dynamically adjusts the saturation level between these two types of RNNs, allowing it to avoid the vanishing gradient problem and perform well on challenging sequence learning benchmarks.
strengths_and_weaknesses: ### Strengths:
1. The paper is well-motivated, clear and the proposed method is supported by theoretical proofs. 

### Weaknesses:
1. My only concern is about the applicability of this work in applied settings. Can the authors explain more on areas where asRNN might offer an advantage over LSTMs specifically?
suggested_changes: 1. How many experimental trials were used for reporting the mean and standard error in Tables 1? 
2. "Unless specified otherwise, asRNN is optimizes using RMSprop with α = 0.99" - Does this imply that the other methods were optimized using some other optimizer setting?
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: This paper proposes the Adaptive-Saturated RNN (asRNN) model, which can dynamically adjust the activation function saturation, so as to achieve high memory capacity and not be affected by the VGP.
strengths_and_weaknesses: # Strengths
1. This paper proposes a new RNN variant, asRNN, which does not suffer from the vanishing gradient problem while having high memory capacity and expressive power.
2. The method in this paper is simple and elegant, and the method is proved in principle.
3. Extensive experiments demonstrate that asRNN is as effective and reliable as baseline methods in terms of long-term memory.

# Weaknesses
1. First of all, I believe that this article has not been compared with RNN variants that are relatively recent in time, such as UniCORNN. The baseline methods are relatively outdated and cannot fully explain the advantages of asRNN.
2. The paper does not have a comparative experiment on the time cost of asRNN and baseline methods in training and reasoning, which is a pity.
3. It is well known that the long-term memory of RNN is limited by its hidden size. Can the authors consider adding parameter sensitivity experiments on the hidden size?
4. The structure of the paper is incomplete, and the main body even lacks the Section on *Conclusion*.
suggested_changes: In general, I think this paper is a solid work, but it needs to consider the four points mentioned in the *Weakness* to further optimize the paper. Simultaneously, in view of  [the submission requirements of Tiny Paper](https://iclr.cc/Conferences/2023/CallForTinyPapers): the paper should not exceed two pages and the practice of moving the majority of the content into appendices is discouraged, I prefer that this paper can be optimized and published as a long paper instead of a Tiny paper .
review_title: null

Meta-review: Slightly over two pages. Please correct in camera-ready.
Decision: Invite to present