Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
The authors study the mechanisms that a GRU network uses to store and retrieve memory in a variable delay copy task. the authors show that the GRU uses slow manifolds to do this, and construct a network for a smaller variant of the task by hand.
main_review: Studying the mechanism by which an RNN model stores and retrieves memory is an important question that can give insights into how these networks work.

This paper's approach of trying to reverse engineer the mechanism of a trained GRU to understand this is interesting. Moreover the finding that slow manifolds are used for the purpose of storing memory of a sequence is interesting.

But the paper does not consider or test alternate hypotheses other than slow manifolds. For example, no attempt is made to empirically show that the GRU is not using pseudo-line attractors. This is a major flaw of the paper.

It is also not clear why the authors specifically consider the GRU architecture, especially since it seems like using an LSTM would make it easier to reason about when information enters and exits the cell.

The writing of the paper is also very confused and hard to parse in many places. The sequence of arguments made for slow-manifolds is sometimes hard to follow.

For example:
- Only 3 units are shown from the trained GRU, with text indicating that this is representative of all other neurons. But later the authors say only 73 neurons were analysed. Might be a good idea to show the plots for all neurons, possibly in the supplement.
- It might help to explain formally what a slow manifold is.
- The perturbation based experiment is hard to understand, as is the PCA analysis.
- Minor: Fig. 3: what are the colors?
summary_of_the_review: Overall, while the goals of the paper are well-founded, and their analysis seems potentially interesting, a combination of lack of clarity and some gaps in their overall argument weakens the paper significantly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper attempts to understand how a RNN goes about solving the Variable Copy Delay Memory task in fine detail. Their model has been trained perfectly on this task and so they are able to focus on how it goes about changing its cell values in the case. The authors present different metrics to track resetting and memorization behavior by the GRU's neurons.
main_review: In terms of strengths, I think the authors definitely show that for this specific task and architecture, the GRU is learning to use working memory in a more or less explainable way.

In terms of weaknesses, I feel that this paper lacks novelty. Their work is not technically novel because they are using a GRU and using a super well studied task. I don't think I took away any sort of new information from their analysis either. Hence I am not sure what the contribution of this work is to the community. I would also like to comment on how busy the author' figures and experiment descriptions are and how hard they are to follow. 

If the authors wanted to improve the novelty of their work, they can maybe focus on state of the art architectures and more complex tasks.
summary_of_the_review: This work lacks novelty, is hard to follow, and shouldn't be included in ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper analyzes GRU's underlying mechanisms that store and retrieve information in the delay copy task of a sequence of K symbols. It proposes a perturbation-based method to determine which neurons are responsible for encoding a step-element pair. The paper then shows that at each step of the decoding phase, certain neurons are reset by GRU, generally mapping to those tuned to hold information of step-element pair.  Finally, the paper provides a synthetic solution to the delay copy task for the case K=2.
main_review: The paper shed light on the encoding and decoding process of GRU, explaining its capability and behaviours. Overall, the authors' hypotheses are well-validated through empirical evidence and informative visualization of the synthetic task. Among the findings,  the synthetic solution is the most interesting and gives novel insights into GRU's operation. On the other hand, the writing is hard to follow. Here, the paper limits to demonstrating the phenomenon and leave the action for future work, which is somewhat incomplete. 

Questions and comments:
- The term "slow manifold" is mentioned frequently without any explanation. From the writing, it seems to simply refer to the fact that during the delay phase, some neurons change slowly. To make the paper self-contained, a brief definition would be appreciated. 
- Algorithm 1 is critical for readers to understand Section 3. Please describe it in more detail in the main manuscript. Or maybe add a simplified version of Algorithm 1. 
- Section 3 seems to suggest that GRU "memorizes" all pairs of step-element during the encoding phase. If so, it cannot generalize to test data with a slightly different S. Did the authors test this case?
- Fig. 3 is hard to read. Top's y-axis: what does "accuracy" mean? Is there any meaning in the point's colour?
- Fig. 4-left, what is the meaning of the big horizontal line at neuron 60?
- Fig. 5-left, the label K is confused with the number of symbols
- Fig.5-right, any explanation for the exception at t=3? How does the GRU perform at this step? If it still decodes correctly at t=3, what makes it possible?
- Sec. 5, did you verify your synthetic solution by comparing it with the parameters of the trained GRU. Is this possible that, after training, GRU can converge to your synthetic solution?
summary_of_the_review: An interesting paper, yet the writing needs improvement. It is important to understand the underlying mechanism of models like GRU. However, I am unsure if the presented result is useful enough for designing a better mechanism for RNN/GRUs.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies GRU cell dynamics on the variable-length memory problem in depth.
Building on top of the uncovered mechanisms a synthetic solution is proposed.
main_review: #### Strengths:
- Thorough experimentation and exciting insights.
    - Through perturbation analysis, the paper finds that only 73 of 250 state elements encode memory.
        - PCA analysis solidifies this finding in Figure 4.
    - A reset measure $\kappa$ is defined in section 4, the new measure allows visualization of decoding resets in Figure 5.
- Using the discovered mechanisms the paper proposes a synthetic solution to the variable length memory problem.
- Source code with a few docstrings is available.
- The paper is a good fit for ICLR.


#### Weaknesses:
- Related Work: Previous work by Karpathy et al. studies the behaviour of various recurrent cell gates across recurrent architectures for natural language processing tasks. The study appears to be relevant for this article. Authors may want to be fair and consider a citation?
    - Karpathy, Andrej et al. "Workshop Track - ICLR 2016 Visualizing and Understanding Recurrent Networks." (2016).

- The description of the experiments was often hard to read. Speaking of 'neurons' instead of reset-gate-neurons,
  update-gate-neurons or hidden-state-neurons, for example, made it harder at first.
  Readers can infer the exact setting from context but should not have to do that.

#### Questions:
- Figure 1: Hidden state neurons mean we are looking at the output from the tanh in equation (3)?
- Figure 2: I personally found it tricky to understand what is going on here.
    I could follow the description of the argmax,
    but what is the difference between the "Projection to hyperplanes from hidden state" and the white points?
    If the white points are the entries of $\mathbf{y}_0, \dots , \mathbf{y}_n $$ aren't these points the projection?
    Why is the projection along the value axis?
- Figure 3, 4: We are looking at hidden neurons, aren't we?
- I think neuron is always shorthand for hidden-neuron?
  If that is true, the authors may want to consider explicitly stating this somewhere?

#### Minor remarks:
- Figure 4, caption: ...every eight column corresponds... ( switch plural columns to singular )
summary_of_the_review: To the best of my knowledge, the paper appears experimentally sound and different from previous work.
I firmly believe that we need more work trying to uncover the inner workings of neural networks.
I am therefore recommending acceptance in the proceedings, assuming that the differences to the analysis from Karpathy et al. (citation in main review) will be discussed briefly.
review_title: null

Meta-review: This paper studies how recurrent neural networks, and more specifically GRUs, store and access information. The authors analyze the solution obtained by gradient descent to the variable delay copy memory task for discrete sequences. They use concepts from dynamical systems, such as slow-manifold, to understand the behavior of the learned model. Finally, based on this analysis, the authors propose a synthetic solution to a simplified version of the delay copy memory task.

Overall, while the scores for the paper are rather positive, I still have concerns about the paper, based on the reviews and discussion. I do not believe that these concerned were well addressed by the authors in their rebuttal. First, I tend to agree that the paper is somewhat lacking novelty and insightful findings (reviewers TN5R, afLb, MToe). For example, I think that tools from dynamical systems are mostly useful to analyze RNNs when the input is constant (Jordan et al., 2019). In the case of the copy task, this corresponds to the "delay" period, where in practice the hidden state is almost constant. This behavior is easily explained by the value of the update gate, close to 1. I thus agree that other hypotheses than slow manifold should be discussed to explain how GRUs store and access information, and that the benefits of using dynamical systems is not obvious. Moreover, I believe that previous solutions to the copy task (eg, from Henaff et al.) could be extended to the variable setting by adding a gating mechanism to these solutions. In particular, Henaff et al. claimed that LSTM could solve this task empirically, while the authors claim otherwise.

Second, after reading the revised version a couple of times, I still find the paper hard to follow (MToe, afLb, TN5R). For example, I think that the concept of slow manifold is not introduced properly, and in particular, how it applies to the learned solution is not clear. More generally, I found the sections regarding how information is stored and accessed a bit confusing. Finally, I think that the studied task is simple, and probably does not provide strong insight about the working of recurrent networks. Specifically, LSTMs tend to perform similarly or slightly better than GRUs on many tasks, while the authors claim that this architecture cannot solve the studied task.
Decision: Reject