Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The authors propose a novel approach to recursive reasoning with Neural Networks, wherein they augment a DNN with a stack. They demonstrate the performance of their approach on the CLRS algorithmic reasoning benchmark.
strengths_and_weaknesses: Strengths: 
- The paper proposes an interesting idea to tackle the problem of algorithmic reasoning
- The method has been described intuitively and clearly
- Ablation has been done to explicate the impact of adding the stack to the training

Weaknesses: 
- The writing can be made more self-contained. Currently, there are a lot of references to prior work which makes it a bit harder to understand the impact of the obtained results. For example,
- HPs have not been explicitly reported. I am not sure how reproducible the results are.
- I am missing baselines in this experiment. While it is clear that the performance is being evaluated on a benchmark for recursive reasoning, what is not evident to me is the notion of a good performance for this approach. Further details need to be provided explaining what the results mean quantitatively i.e. what is an expected test accuracy, what has been obtained and how do we explain the difference etc.
suggested_changes: - I would suggest explaining the connection between expected performances in the CLRS benchmark and how indicative is that performance to the original problem of recursive reasoning. While this has been discussed in the cited paper, I think it makes sense to add a small explanation to make the writing more self-contained
- It has been mentioned at multiple points that the method differs from  RNNs. Can RNN baselines be formulated for this task? If so, I would recommend adding that to the plot. If not, I would recommend indicating the performance of previous approaches on the benchmark to give the reader a notion of what the obtained accuracy means.
review_title: null

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: The paper proposes a new neural network architecture which is augmented with a stack to save and restore states across time. This work makes reasoning recursively with neural networks possible and leads to better constructing generalist neural algorithmic learners. 
strengths_and_weaknesses: Thank you for submitting this work to Tiny Papers @ ICLR 2023. The paper targets an important aspect of neural networks and presents an interesting solution to incorporate stacks with the architecture.

Strength
- The clarity is very-well satisfied. The problem that neural networks cannot reason recursively is presented clearly in the paper. The proposed architecture, training and testing pipeline, and the results are described explicitly as well. The idea of incorporating stacks with neural networks is very interesting and has certain degree of novelty.
- The results (i.e., nearly 30% increase in accuracy) justifies their hypothesis.
- The authors are aware of the importance of the reproducibility of the experiments and mention that their implementation will be provided in the camera-ready version.
- This paper successfully follows the basic formatting requirements, page limit, and the ICLR code of conduct.

Weakness
- There are several relevant literature and some components of their experiments (e.g., hints and  the CLRS algorithmic reasoning benchmark) mentioned in the paper. However, it would be better to describe the experiment setting (i.e., the hyperparameters, datasets, neural network backbone, etc.) in more detail. 
- For the accuracy reported in the result section, I would suggest to state more about what the accuracy means and whether it is possible to compare the experimental results with some published baseline results to make the stated improvement more convincing.
- The paper does claim the goal and importance of this new architecture, but it would be better to describe some concrete applications of it. For example, some specific tasks that current architectures could not/struggle to solve but can be easily addressed by this proposed architecture.
suggested_changes: The proposed idea on augmenting neural network with stack is innovative and promising to make some impact in the field. Here are some suggestions to further improve the work. 
- Explicitly describe the experimental setting (i.e., hyperparameters, neural network backbone, and datasets) and the result accuracy. Adding some reports on the performance (i.e., time and space complexity) would be beneficial.
- Compare the results of the proposed architecture with SOTA architectures.
- For future directions, apply this proposed architecture to solve some tasks that are otherwise difficult for other architectures.
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: This work introduces a simple strategy to algorithmic reasoning by incorporating a stack to exactly save and recall the network's states. 
strengths_and_weaknesses: Strengths
- This paper studies an important research question of algorithmic reasoning
- The proposed method is simple and yet effective

Weaknesses
- The writing could be improved, while the high level idea of the proposed method is clear, many of its implementation details are missing. Particularly, how did the authors modify the network to interact with the stack, i.e. as the stack size is always changing, what is the last layer of the network? 
- Is it fair to compared to the non-stack variant as this method requires call stack hints? What is the complexity overhead when introducing the stack?
- In Appendix A, "replacing the predicted computation state of the last time step with the ground truth with 50% probability during training" is commonly referred to as "scheduled sampling" [A] rather than teacher forcing.

[A] Bengio, Samy, et al. "Scheduled sampling for sequence prediction with recurrent neural networks." Advances in neural information processing systems 28 (2015).
suggested_changes: See Weaknesses
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: In this submission, the authors consider the problem of applying neural networks (a GNN in this case) to recursive problems. They introduce a neural architecture that incorporates a stack. They provide a prototypical experiment on learning to depth-first-search.
strengths_and_weaknesses: Strengths:
- Considering the training of neural networks on recursive or inductive problems is an important topic 
- In their prototype exerpiment, a neural network with a stack improves the mean accuracy from 60.7% to 89.1%

Weaknesses:
- The authors consider one architecture (GNNs) and one recursive algorithm (DFS), but the title and introduction are way to broad.
- The stack operation selection is supervised
suggested_changes: The title and introduction are way too broad. I would recommend the authors specify this.
review_title: null

Meta-review: metareview: The paper proposes a novel approach to recursive reasoning with Neural Networks, which is an important aspect of neural networks. The method is clearly presented and the results demonstrate its effectiveness. The submission can be improved by providing more details about the experimental setting, hyperparameters, and comparing the results with other state-of-the-art architectures.
confidence: 3: The area chair is somewhat confident
Decision: Invite to present