Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: In this submission, the authors run experiments on an alternative training loss (Salle & Prates (2019)) when running a Recurrent Neural Network (RNN) over the same input multiple times (Bansal et al., 2022). They show that the Delta loss enables the same “extreme extrapolation” (scaling to larger instances than seen during training), while being more efficient.
strengths_and_weaknesses: Strength:
- Interesting experiments, a good indication that the usage of the Delta loss can save many resources
- The delta loss performs as well as the proposed loss of Bansal et al.
- The delta loss is hyper-parameter free
- Implementation available

Weaknesses:
- The “robustness” claim is not well-backed
- Limited experimental results only indicate that the Delta loss will outperform
suggested_changes: I have no particular suggested changes for this submission in this track. I would love to see a deeper analysis on the robustness claim in the future.
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: This paper proposes to use the Delta Loss to train RNN to solve iterative reasoning problems. The results show encouraging results of Delta Loss compared to the state-of-the-art Progressive Loss.
strengths_and_weaknesses: Strengths
- Empowering (deep) neural networks with the iterative reasoning capability is an important research topic.
- The empirical results are encouraging, showing that Delta Loss can perform comparably with the state-of-the-art progressive loss, while being simpler and more computationally efficient.

Weaknesses
- In Think Again Networks, the authors proposed an abstraction framework with the Delta Loss, and also mentioned RNN as one possible implementation (in the abstract). Given this, the technical contribution and extendibility of this work is quite limited. 
- The empirical results are not really strong: on all three tasks, both Delta and Progressive losses perform mostly the same, albeit Delta Loss even took more reasoning steps to converge.
- The authors might consider plotting the accuracy against the training time to highlight the training efficiency of Delta Loss.
- A more detailed discussion on the future work is preferred.
suggested_changes: Please see Weaknesses.
review_title: null

Decision: Invite to present