Gating Mechanisms Underlying Sequence-to-Sequence Working Memory
Authors: Ian D Jordan
Piotr A Sokol
Il Memming Park
Venue: ICLR 2022 Submitted (2022)
Keywords: Working Memory
RNN
Dynamical Systems
Slow Manifold
Gating

Abstract: Working memory is the process by which a system temporarily stores information across a necessary duration. Memory retention and manipulation of discrete sequences are fundamental building blocks for the underlying computation required to perform working memory tasks. Recurrent neural networks (RNNs) have proven themselves to be powerful tools for such problems, as they, through training, bring rise to the dynamical behavior necessary to enact these computations over many time-steps. As of yet, the means by which these learned internal structures of the network result in a desired set of outputs remains broadly elusive. Furthermore, what is known is often difficult to extrapolate from due to a task specific formalism. In this work, we analyze an RNN, trained perfectly on a discrete sequence working memory task, in fine detail. We explain the learned mechanisms by which this network holds memory and extracts information from memory, and how gating is a natural architectural component to achieve these structures. A synthetic solution to a simplified variant of the working memory task is realized. We then explore how these results can be extrapolated to alternative tasks. 

Under review as a conference paper at ICLR 2022
GATING MECHANISMS UNDERLYING SEQUENCE-TO- SEQUENCE WORKING MEMORY
Anonymous authors Paper under double-blind review
ABSTRACT
Working memory is the process by which a system temporarily stores informa- tion across necessary duration. Memory retention and manipulation of discrete sequences are fundamental building blocks for the underlying computation re- quired to perform working memory tasks. Recurrent neural networks (RNNs) have proven themselves to be powerful tools for such problems, as they, through training, bring rise the dynamical behavior necessary to enact these computa- tions over many time-steps; a feat not obtainable using a finite memory system, such as transformer networks and reservoir computing. As of yet, the means by which these learned internal structures of the RNN result in desired set of out- puts remains broadly elusive. Furthermore, what is known is often difficult extrapolate from due to task specific formalism. In this work, we analyze RNN, trained perfectly on a discrete sequence working memory task, in fine de- tail. We explain the learned mechanisms by which this network holds memory and extracts information from memory, and how gating is a natural architectural component achieve these structures. synthetic solution to simplified variant of the working memory task is realized. We then explore how these results can be extrapolated to alternative tasks.
INTRODUCTION
Recurrent neural networks (RNNs) transform stimuli across multiple time-points produce linear working memory representations that can be used to solve complex tasks (Elman) 1990, Hochreiter & Schmidhuber 1997; Mante et al. 2013). Memorization and manipulation of dis- crete sequences of elements are a common low level requirement to many broad families of such problems (Hochreiter & Schmidhuber 1997; Jordan al.) 2021 Yang et al., 2019). However, full understanding of how the underlying dynamics learned by these networks accurately bring rise to the necessary computations remains an open area of research (Sussillo Barak 2013). That is, from a dynamical system's 's point of view (Guckenheimer & Holmes] 1983 Jordan et al. 2021). how does the network's internal phase-flow and attractor structures, brought forth by training, play a part in the found solution to the desired task? Furthermore, many of the known properties of trained RNNs' learned dynamical mechanisms are specific to individual problems (Henaff et al.] 2016 Jarne 2020 Ichikawa & Kaneko 2021). Therefore, due to their narrow formalism, such attributes are often difficult to extrapolate to alternative tasks of the same family. It has been demonstrated
Encoding
Variable Length Delay
Decoding
Neuron Index
#84
Ww
#82
Time
Figure 1: Superimposed trajectories for neurons 82 84 across one thousand trials of VDCM, from perfectly trained GRU. Neurons 82 and 83 demonstrate slow manifold dynamics during the delay period of each trial. These three neurons represent the qualitative behavior across all 250 neurons.

---

Under review as a conference paper at ICLR 2022
that the underlying attractor structures of RNNs successfully trained on an individual task are often similar, if not topologically equivalent, across networks, regardless of architecture, initialization, and syper-parameters (Maheswaranathan al. 2019). However, one would also expect commonal- ities between the underlying structures of RNNs trained on similar, but none the less, different tasks (Flesch et al. 2021 Yang al. 2019) research feat not yet well explored.
To better train and interpret the solutions found by RNN models we require a finer grain analy- sis, where results can be broadened to very general classes of problems. (Karpathy al.) 2016) demonstrated a more in depth empirical exploration of GRU and LSTM architectures computing with sequential data, and identified different failure cases that can arise when training these models. However, this work studied the existence of underlying dynamical mechanisms indirectly from their effects on network output and single-neuron behavior, leaving out details on the functionality of each mechanism at the population level. If done, found mechanisms can be further studied and synthet- ically recreated to be made more understandable. Moreover, the synthetic realization of dynamical mechanisms, inspired by those obtained through gradient based optimization, can be combined and extrapolated from to form synthetic RNN solutions to related tasks.
We surgically analyze single RNN trained on discrete sequence working memory task. The network performs with no mistakes on a sufficiently sized validation set of trials. Inspired by the behavior discovered in the network, we design and experimentally validate synthetic solution to a simplified version of the same task, realizable in relatively low dimensions. We then discuss how such findings apply to networks trained on different but mechanistically similar tasks, including sequence-to-sequenc translation task.
2
VARIABLE DELAY COPY MEMORY TASK AND RNN MODEL
In choosing appropriate task, we look at copy memory; standard benchmark to evaluate a neural network's ability to accurately recall information seen many time-steps in the past (Hochreiter & Schmidhuber 1997 Henaff et al., 2016 Arjovsky et al., 2016). Let A {as) be set of K symbols. We then pick S, E N. The input is a vector of categories. length T 22S,, where each entry one-hot encoded. Each trial of the task consists of three phases, of length S, T', and S respectively. During the first phase (encoding), the network is presented with with S entries uniformly sampled from {a. ) to be remembered sequentially. During the second phase (delay period), the network is fed T - inputs of ag+1. a blank category, indicating no important information is entering the network. At the final time-step of the second phase S S T, a delimiter dx+2 is input to the network, indicating that the RNN should output the original S entries of input in the same order by which they appeared in the first phase of the trial beginning at the next time-step (third phase - decoding). During these last S time-steps, the inputs are all set to the blank category ak+1- During the first T + S time-steps (encoding and delay period), the network should output aK+1- The task is to minimize the average cross-entropy of the outputs at every time-step. As such, the networks should remember a sequence of S elements for T time-steps.
Henaff, Szlam, and LeCun developed and experimentally verified synthetic solution to this task (Henaff et al. 2016). However, if we allow T to vary from trial to trial, the underlying dynamical mechanisms allowing the RNN to properly enact the computation remains elusive. We will refer to this task as variable delay copy memory (VDCM), as coined by (Henaff et al.) 2016). We suc- cessfully trained a GRU network (Cho et al. 2014), by novel means (Anonymous). with a linear readout on VDCM, such that it performs perfectly on all the test trials. We were unable to train other network architectures on VDCM, including LSTM. For each trial we set K = 8, 10, and T ~ (100,, 101, ..., 120). The model used is represented as follows:
2: o [W T. biz U + bhe)
(1)
F; 0 [W, ,,, + bir + U,h,11 + byr))
(2)
h; (1 2,) O tanh(W,ar WW,, + bin + -r (URRi + ban)) + 2t h--1
(3)
y: = Vouth; + bout
(4)
ychoice arg max (axx((y1lm)) [31]m), E [0, 1, 1,.,,... [..,K] K] C (
(5)
where h2 E IRd, d = 250, H. E R K22 W., W,, W, E Raxs and U2, U, E Rxd are the parameter matrices, biz, ber. bih, bbz, bar, bab IRdd are bias vectors, represents element-wise
2

---

Under review as a conference paper at ICLR 2022
multiplication, and (22) 1/(1 + + is the element-wise logistic sigmoid function. For the linear readout Yt E RK+1, bout E RS-1 and Vout E [[ Ychoice is taken the largest element of ye at each time-step, and represents the chosen class readout.
3 ENCODING ON SLOW MANIFOLDS
The computations required for VDCM fall into two main parts. The first is the memory structure. How does the GRU retain information about the elements presented to the network at each of the ten encoding time-steps? The second is memory recall. How does the GRU pull stored information from memory in the correct order? In this section we will focus on the former, and explain the structure by which our trained RNN uses to properly encode inputted information.
Fig. I demonstrates the behavior of hidden-state neurons 82 - 84 (i.e. [he]] for index j E (82, 83, 84)) of the trained network, while performing VDCM. Let newron refer to hidden-state neu- ron unless otherwise specified. The trajectory of these selected neurons across one thousand trials superimposed, clearly indicating the three segments of the task. If we look the delay period, beginning at / = 11, we notice that the neural activity appears to be near constant across each trial. The selection of neurons shown in Fig. I are representative of the behavior of across most neurons in the network, with the exception of several oscillatory modes that are rare and inconsistent across trials. A complete collection of neurons across trials can be found the appendix E What of primary interest are the neurons that significantly vary trial to trail during the delay period, such as neurons 82 and 83. These varying neurons indicate the existence of slow manifold, an observation in line with recent research (Ghazizadeh & Ching 2021). A slow manifold is similar to attractor (i.e. a fixed point, an attracting line, an attracting ring, etc.), where the state of the system h, lies on the attractor it will not change unless perturbed. However, the manifold is not entirely made up of fixed points, rather the speed of the phase flow in these regions is arbitrarily slow in a subset of directions. In the GRU architecture, such behavior results from either a pseudo-line attractor (Jor- dan et al.] (021). or from the influence of the update-gate 2₁ (Cho et al., (014). If we look at the update-gate for dimensions with analogous behavior to 82 and 83, what we find that, during the delay period, most demonstrate a value close to 1 for z₁. Such activity simplifies equation 3 to the following approximate form:
[he]: [he-1];
(6)
where i indexes the neurons with a high update-gate gate during the delay period. As such, the network retains near perfect memory of the past in these directions. We assume that the neurons' update- gates are the primary mechanism to enact mow-manifolds in our trained network. the case of pseudo-line attractors, such slow flow is the result of the nullclines of the underlying continuous- time dynamical system, by which the network can be interpreted as a numerical approximation of, existing sufficiently close together in the hidden state-space (Jordan et al. (02T). However, the null- clines of this system cannot be oriented such that they form pseudo-line line attractor in any canonical direction (in the direction of single neuron). We will see in the next section that the means by which our network is decoding are canonical in nature, and so we disregard this mechanism.
Given that this is the most likely mechanism used to encode information, how exactly this computa- tion enacted? We trained this network to be able to encode 810 possible sequences, K 8 elements to choose from at each time-step, across S 10 time-steps. If such a computation is implemented by carefully placing each trajectory on a slow manifold, the manifold can be segmented into regions where each individual readout element is outputted. Due to the implementation of the argmax func- tion for VCDM, the largest element in the readout vector will be chosen as the class. A cartoon representation of such regime depicted in Fig. 2
To empirically show that a slow manifold is the dynamical feature used and to determine how it organized, we implement perturbation based experiment. The low level details can be found in appendix A We can assume an encoding time-step q [1,..., 10), and an element p E [O,...7] If across two trials, we input a sequence where at time-step g the element to be encoded is p. and an identical sequence where at time-step q the element is not p. we can determine how the neuron representation differs after the encoding phase of each trial (t = 11). By comparing this difference over many sessions, often each neuron is used, and approximate the expected value each neuron takes. We can then test the accuracy of our q - p representation by inputting many sequences through the network, where
determine which neurons are used to encode for this q - p pair, how
3

---

Under review as a conference paper at ICLR 2022
Projectos
Largest E lement Chosen
Radoout
"y2"
Slow Manifold
Value
System State
Superimposed Readouts
Figure 2: Left: the system state is on slow manifold (blue). The output projection y, (h), can ensure that the a specified dimension in the output space has the largest value (green circle), Due to the argmax function, this dimension will be chosen as the class. Right: we express each element of y/ as a linear combination of the values of each neuron (defined by the learned output weights), which be interpreted as hyperplane, where our position on this hyperplane is determined by he The hyperplanes can be arranged such that regions in the hidden-state space exists, where each possible readout takes on a higher value than the others. The white points indicate the class decision boundaries, and the blue points indicate each element of y, from an example ht:.
the elements at = q are not p, but set, at 11, the most frequently used neurons in the representation to their approximate expected values for the representation. If the network output remains unaltered, but at decoding time-step q the readout is incorrectly changed to p, we consider the trial success. The results of this method are displayed in Fig. 3
Figure 3: Results of our method to map out the structure of the slow-manifold in our trained RNN. Top: probability of a successful per- turbation at the beginning the delay period (1 - 11) for each possible element, indicated by color, across each encoding time-step. Suc- indicates that the neurons perturbed cessfully altered the appropriate decoding time- step readout to the desired element without al- tering the readouts at any other time-step. Bot- tom: number of neurons used for each per- turbation. All pairs of time-steps and elements required least 5 neurons.
Accuracy
0.6
0.4
Neurons Used
1.6
14-
12.
10-
Encoding Time-Step
Only 73 of the 250 neurons in the network may be used to encode memory. We reorder these 73 neurons by their center of mass with respect to the 80 possible combinations of g and p. where sweep through all values p can take for q = 1, then q 2, etc. Fig. 4 (Left) depicts such a reordering, revealing a block-like structure. We see that nonempty set of neurons account for every element at each specific time-step q. The plot is colored by the expected value that neuron (row) takes when presented with a specific element at a given time-step (column).
To analyze the finer details the manifold's 's structure, let's consider the neurons primarily tuned to single time-step; those tuned to encode information at time-step 9, for example. We can use principal component analysis (PCA) to visualize the activity of these selected neurons in low dimensions (Bishop) 2006). We project down the state of the set of neurons tuned to time-step 9 at 11, across the set of test trials as demonstrated in Fig. 4 (Right). Data points are colored by which element K was presented to the network at 9, and form eight separable clusters, one for each element. The points in each cluster vary in a single direction, indicated by the blue arrow. Since VDCM is deterministic task, the only source for this variability is the influence of the input the network at all time-steps preceding t = 9. This suggests that the neurons primarily tuned to encode information / = 9 are not fully decoupled from the neurons tuned to other time-steps.
4

---

Under review as a conference paper at ICLR 2022
10
2.2
Time-Step: 9
1.5
20
0.5
Neuron
10
0.00
-0.25
-1.0-
-1.
-0.75
70
-20
-2
Time-Step/ Element Pair
PC1
Figure 4: Left: the 73 neurons exhibiting slow manifold dynamics during the delay period. Every cight columns corresponds to the different elements that can be given at a specific encoding time-step. Matrix elements displayed in white indicate that the corresponding neuron (row) do not respond to changes in these element time-step pairs. Further color indicates the expected value that neuron takes when a given element input at the corresponding time-step. Right: PCA projection of the activity of the neurons tuned to encode information from / = 9 (shaded region on the left 11 for each trial plotted
plot) at the beginning of the delay period = 11). The activity as data point colored by which element was presented to the network at t = 9. The background coloring indicates what class most likely to be readout at the ninth step of decoding, if that point in PC space was projected back up to the hidden subspace of neurons tuned to time-step 9 at / 11, replacing them.
However, this observation brings about important point regarding the neurons in our trained network that are primarily tuned to input presented at / 9. While PCA brings about the dimensions with the highest variability across trials, it does not indicate which dimensions are most crucial for enacting the computation. While insightful in understanding how training brought forth various sub-mechanisms that make up the finer memory structures in the network, this direction allocated to previous inputs may not be used to indicate class during the ninth step of decoding. In the following section, we will dive into the second major computation required for VDCM, decoding. It will be shown that analysis of decoding will enable us to better understand the mechanism used for encoding. For all other encoding time-steps, plots analogous to Fig. (right) can be found in appendix B
4 DECODING AND THE ROLE OF GATING
How does information from memory get readout in the desired order? The readout at each time-step determined by equation 4 and equation 5 The linear readout matrix V ous and readout bias vector b bout separate the hidden-state space h, into classification regions with linear decision boundaries. Prior receiving the readout signal, the network remains regime where only the delimiter symbol "8" is outputted. The readout signal perturbs the hidden-state of the network into a decoding subspace, where some mechanism for determining which decoding time-step the network on must activate (i.e clock). Decoding the first element of the sequence is easy enough, as the network needs only to perturb the hidden-state to a region where the neurons tuned to encoding the first time-step of each trial are dominant in readout (Fig. 2). But then how does the network properly output the remaining elements of the input sequence? The most natural means to enact this behavior built into the GRU architecture, the added gates, 2; and Tı, act forget reset information stored in the hidden-state. If some neuron, of index p. in the network has corresponding update-gat sadate-gate and reset-gate values close zero, the evolution of that neuron can be approximated as follows:
[hipp tanh([W,], wlpp x, [bb))p)
where [Wlp indicates row of W.. Therefore, the network loses all information previously stored in this neuron of the system (direction of the hidden state-space). Moreover, the neuron is overwritten by the input x,-.
We want to analyze our trained GRU network to see if the decoding regime exploits this behavior. Prior to doing this, it is important define what exactly is meant for neuron to be "reset." We
5

---

Under review as a conference paper at ICLR 2022
[0, 1), where the overhead bar represents sample mean. At a given time-step. if neuron p is to forget all previous information held within (1
will define the metric
Imelp) -1-
(relp) = 1. The closer to zero this term is, the more prior information is retained. We run 1000 randomly generated input sequences through the network and determine A. for every neuron each decoding time-step. We then determine a threshold Ç, where neurons with x >> Çat at a given time-step are "reset." We chose Ç 0.255, and detail our reasoning in appendix
[=t]p][ (1-[==lp)1
Using this threshold, we reorder the neurons of the network by which decoding time-step each is reset after for the first time (i.e. the first time that the expression > is true after the readout signal is presented). Fig. 5 (Left) depicts this reordering, allowing us to visualize for each neuron using our choice of Ç. We notice clear staircase pattern, indicating the existence of a non-empty set of neurons that reset for the first time after every time-step of decoding. Furthermore, due the methods used to obtain our value of Ç. it is assured that at least one neuron in each of these sets is tuned the equivalent time-step during the encoding phase of each trial.
28
26
40
TO
Neuron
Neuron
Decoding Time-Step
Time-Step /leement Pair
Figure 5: Left: mean activity of our defined metric k =1 [2ep) [Te]p) for each neuron when we line up the decoding phases across trials. We reordered the neurons such that those which "reset" time-step of decoding are on top, followed by those which "reset" for the first time at time-step 2, etc. Doing so reveals the demonstrated staircase pattern, indicating that there exists a distinct set of neurons which are reset at each step the decoding phase. Right: a recoloring of Fig. 4 indicating at what time-step each neuron active in memory retention during the delay period is reset in the decoding phase. We note that every time-step of encoding has at least one corresponding neuron that is appropriately reset at the equivalent step of decoding, with the exception of those tuned to 3.
We recolor Fig. (left) by which time-step of decoding each of the 73 neurons involved encoding are reset after. This is demonstrated in Fig. 5 (Right). For the set of neurons tuned to each time-step of the encoding phase, at least one neuron is reset after its corresponding decoding time-step. This isn't surprise, as we ensured this was the case when choosing Ç. However, what is surprising is that for each set of neurons, with the exception of = 3 at least one neuron exists that tuned to every element of its corresponding time-step. Therefore. this regime demonstrates of the most important neurons responsible for performing VDCM. We say important to indicate that these neurons are the primary carriers of information (i.e. hold memory) throughout each trial of the task, while the others play a more supporting role; either managing internal mechanisms for telling time during the encoding/decoding phases of the task or fine tuning the discussed dynamical mechanisms, such as the direction allocated for noise demonstrated in Fig. 4 (right). Furthermore, our analysis suggests that the system is primarily canonical in its encoding and decoding mecha- nisms. This is a result of the GRU architecture, as both gates are canonical in their functionality, meaning information is maintained or reset the directions of individual neurons (Cho al., a 22014). Furthermore, the internal clock is finely tuned to ten time-steps. We have tested if the network can generalize to different values of If we vary S, not a single trial's readout is entirely correct. S < 10, the first S elements are properly encoded/decoded. If S > 10, the first 10 elements properly encoded/decoded.
complete picture
'Information from 1 = 3 is reset after the fourth time-step of decoding, for some values of K. suggesting possible intersection between the regions of the slow manifold, and or that the geometry of theses regions are complex, requiring the neurons tuned additional time-steps aid enacting the computation.

---

Under review as a conference paper at ICLR 2022
5 SOLUTION TO THE K = CASE OF VDCM
Given what we know about the inner functionality of our trained network, we'd like to develop a complete solution for performing VDCM based off of found behavior. The solution should layout a specific set of parameters for the GRU network. No training will be required and the internal dynamical behavior must be interpretable. We will incorporate a slow manifold to encode the input sequence. Furthermore, we will design an appropriate readout projection and sequentially reset the neurons allocated memory as to output each element of the sequence in the desired order. This will be done with a separately designed clock mechanism, which will be used for determining the number of time-steps which have passed during both the encoding and decoding phases of each trial.
We restrict our solution to the K = 2 case of VDCM, such that each sequence to be encoded is binary string of length S. Any information expressible as a binary string elements can be encoded and later decoded in this context, with 2 different states achievable with single network of 2 (8' +'++1 neurons, as we shall demonstrate below. Furthermore, this allows for a more pedagogical approach in our presentation, our goal is to build deep intuition for these mechanisms and demonstrate proof of concept in extrapolation. As we'll Ill see, only one neuron is needed to hold memory from a given time-step in the K = 2 case of this problem. Fig. (left) depicts schematic representation of our solution. For easy visualization, Fig. 6 (right) illustrates the decoding mechanism for the S K 2 case.
hz
START
1
....
S-1
STOP
44, A)
Reset In
SYSTEM MODE
Reset
Encodal
(B, A}
h
h
fa, B)
ENCODE
hi
DECODE
SLOW MANIFOLD
8 B}
Schematic representation of the synthetic solution to VDCM for arbitrary S and K = where elements are notated as either A or B. The system acts in one of two modes, either encoding or decoding. During the encoding phase of each trial, elements to be remembered are presented to the network, setting the values of the first S neurons (a slow manifold) sequentially, as dictated by an internal clock. After the first S time-steps, the system switches out of the encoding mode into decoding. The clock is not reactivated until the readout signal is given, which will begin reseting the values of the S neurons allocated to memory to zero, again sequentially. Done properly, the network will readout the encoded elements in their original order. Right: An example of the memory structure for the S = K = 2 case of our solution. The decision boundary separates the hi - ha plane into regions associated with each element. Four subregions exist, where information can be encoded such that each of the four possible sequences can be readout.
Figure 6: Left:
We begin by reiterating what each dimension of our input x; E 4 describes. and 22 are the two possible sequence elements, A and B respectively, 43 is the blank symbol and x4 is the readout signal. All unspecified parameter matrices and bias vectors only contain zeros in their entries. Fig. deppcct complete picture of all nonzero parameter matrices and bias vectors used in our synthetic solution. From this, we can further define each subcomponent. We allocate S neurons initialized at zero to memory, and S +1 neurons, all initialized at except for the second to last, which is initialized at 1, to the clock. Furthermore switch will determine if the system is finished encoding or not, by saturating near -1 (low) or (high). We also choose two parameters, and 3, such that worked well for us. All unspecified subcomponent entries are considered to be zero.
allocate neuron initialized at -1 1 to switch. The
>>>>>>>>>>> 0. a 10 and = 3 has
Starting from the upper left, dn encodes information directly into memory. Each row of An is (r-) -(r -.r- We chose r = The larger r is, the greater in value S can be while maintaining perfect accuracy the task. A21 ensures the clock doesn't indicate two simultaneous time-steps
7

---

Under review as a conference paper at ICLR 2022
Input
Readout Projection
Figure 7: Nonzero parameter matrices and bias vectors used in our synthetic solution VDCM. Each is broken up into subcomponents, where nonzero subcomponents are denoted in gray. The dimensions of each subcomponent are depicted in blue. The functional connectivity of each block matrix can be obtained by the purple and red la- beling. For example, consider V Vout, which takes part in the readout projec- tion from the hidden-state. Whether an or a B is outputted is dependent only on the hidden-state dimensions al- located to memory.
Memory Clock
AlL sui
A12
La
J3
C:
Vout
bost
Am Seniin
Azz Seliia
J --h0e
J Lnd
C²
Aa
A32
Dutour from memory when loutts" and taas arevcc3dd
The
Internal Dynamics
Hemory
Clock
Switch
Mamory
Swittch
Hi
H12
H13
G11 5-d
G12 Sixdiiti
G13
Un
Mzz
G21
G23
Hat
M1z Lxp-0)
H33
G
G Sid--a
G3: 1+1
1+5
2x5
during the encoding phase. Every element in the last two rows of X21 are aa. The submatrix Aạι = i-a -a] holds the switch low during the encoding phase. The last row of A22 is [-a a]. ensuring the clock remains deactivated during the delay period and reactivates upon receiving the readout signal. A32 = [aa a] keeps the switch high after the encoding phase.
In regards to the hidden-state dynamies, H22 has a for every entry along the super- uppe-diagonal. This the clock mechanism, which works in isolation, causing the clock neurons to sequentially rise from a low to a high value. Every entry of the bias sub-blocks F2 and F33 are set to -B. F22 ensures that after neuron allocated to the clock takes at the next time-step. We chose 3 < a to ensure that neurons multiplied by a can overcome the functional effects of those multiplied by B. Similarly, F3 keeps the neuron allocated to the switch low during the encoding phase. The entries Σ5= - -) for each j column of G12 are all a, and sequentially reset each of the S neurons allocated memory after each time-step dictated by the clock during the decoding phase. Every entry of F1 is B. as to ensure the reset-gate for each neuron in memory is kept high prior to decoding, thereby retaining memory.
high value, it will be brought back to a low value
The primary component to readout from memory is J11. The first row of J11 is always alternating. The second row takes similar structure and is [0 -1 0 alternating. In conjunction with the placement of the encoded information on the slow manifold, as defined by the two geometric series which make up the columns of A11. this ensures that information encoded at earlier time-steps takes precedence until reset. Fig. (right) depicts the resultant decision boundary when 5 2. All but the far right entry of J22 ensure that, unless both the switch is high and the clock is activated, only the blank symbol will be outputted. Thus completes our synthetic solution to the K = 2 case for VDCM. We validated this solution the same test of trials used for traditionally trained network, and it performs with perfect accuracy. Examples of the hidden-state dynamics for the S = 2 and S = 10 cases are depicted in Fig. g and the code to generate these plots are in the supplementary material.
0
0...J
-1...]
again
a. J23 aa, and C2- = 2 a. These
3-28
DISCUSSION
We've identified and analyzed set of underlying dynamical mechanisms which enact the compu- tations necessary to perfectly perform VDCM, a sequence-to-sequeeee working memory task, with GRU network architecture. The internal dynamical behavior of the RNN constitutes the execution of both an encoding mechanism. where information can then be held indefinitely in memory, and a decoding mechanism, which makes use of and deletes information held in memory for accurate readout. The encoding regime stores information the hidden-state space of the system. Regions of this manifold are tuned to specific time-steps the encoding period, where subregions within each region are structured to differentiate between what input was presented to the network at the time-step the region was tuned to. The decoding mecha- nism deletes information sequentially from memory, allowing data stored at later time-steps to take precedence in the readout. Both of these mechanisms take advantage of gating to behave properly. We then detailed out a synthetic solution to the K 2 case of VDCM and demonstrated its func-
slow manifold, which makes up a subspace of
8

---

Under review as a conference paper at ICLR 2022
Hidden-Stafe Dimension Index
Switch 1000X
Clock
XXXXXX
Memory
60
20
40
183
200 140
Time-Steps
Time-Steps
Figure 8: Hidden-state trajectories for 1000 superimposed trials of VDCM using our synthetic solution, for both the S = 2 case (left) and the S = 10 case (right). Neurons fall into one of three functional categories. S neurons are allocated to holding memory. S + neurons create the clock, which indicates what time-step of encoding or decoding present, and one neuron is allocated to switch. When low, the switch indicates the trial is in its encoding phase. When high, the trial has finished encoding. A high switch in conjunction with the readout signal begins the decoding phase.
tionality. In this section, we want convince the reader that this analysis does not so narrowly apply only to RNNs trained on VDCM, but on alternative tasks as well.
The mechanisms realized in synthetic solution can be applied in variety of different contexts, so long as the mechanistic function required of each is retained across tasks. Consider the parity bit problem (Wegener & Pruim 2005). The RNN given binary string of length S and has to simultancously output a binary string, indicating if the number of I's presented to the network is even or odd. Unaltered, this task only requires an XOR operation between the input and output at the previous time-step. This suggests the need of a memory structure (or buffer) that holds one piece information the last element outputted. However, if we change the task so that the RNN needs to readout the desired binary sequence arbitrarily long after then entire input sequence has been presented, then the task is nearly identical to VDCM (Fig. 9 (middle)). The one caveat that the information encoded in memory is not the input itself, but the output from the XOR operation of the input and information about the last element stored in memory. Let's construct another task; a
VDCM
Parity Bit
Translation
Figure 9: Schematic representation of the mechanistic connectivity of the processes and structures used accurately perform VDCM, modified parity bit, and a binary character translation task. Left: input is encoded in memory, which is self sustaining, and is later decoded and outputted. Middle: nearly iden- tical to VDCM, but what is encoded is the out- put of XOR operation between the input and the previous encoded value in memory. Right: same as VDCM, but memory must undergo nonlinear computation prior to decoding.
Input
Input
Input
Memory
Memory
Memory
Output
Output
Output
Decada
Computation
sequence-to-sequenee translation task with binary character language and variable delay between receiving the input binary sequence and needing to output a desired binary sequence. Suppose that the input sequence ends in the desired output sequence is the input sequence in reverse order, beginning with the last element seen. Furthermore, if the sequence ends in a O, the desired output and vice versa. Such
sequence is the input sequence with all the bits flipped every 0 becomes a task is also nearly identical to VDCM. The memory structure can be made the same, but must undergo some nonlinear computation prior or during each decoding time-step (Fig. 9 (right)). We were unable to train an RNN this latter task by any conventional means (appendix DD. suggesting a need for more sophisticated training methods which leverage the dynamical behavior required of important
the RNN complete the task. We leave this to future work, and note this result example of why the work done in this manuscript is important for RNNs when performing more sophisticated tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrett neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, pp. 1120-1128, New York, NY, USA, June 2016. JMLR.org.
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statis- fics). Springer Verlag, Berlin, Heidelberg, 2006. ISBN 0387310738.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcchre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN En- coder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar, 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1119. URL http: //aclweb. hww.//e...rr//attoooooooo/------
Jeffrey Elman. Finding structure in time. Cognitive Science, 14(2):199-211, April 1990. ISSN
0364-0213. doi: 10. 0221000660364--21111990000222220 URL https:/ httt:::://wwwsttt www.:ssceeeeeeeeeeeoo
Timo Flesch, Keno Juechems, Tsvetomira Dumbalska, Andrew Saxe, and Christopher Summerfield. Rich and lazy learning task representations brains and neural networks. bioRxiv, 2021.
doi: 10.1101/20211111111111111111100 04.20...33... 441 24112244220.42224441122 URL httpp:://wwtt http:o/www.crr/ooeeett/aaaaaaaa
2021/04/23/20222 04. 04. 441128
Elham Ghazizadeh and ShiNung Ching. Slow manifolds within network dynamics encode
working memory efficiently and robustly. PLOS Computational Biology, 17(9):1-0,, 09 2021. doi: f0.137//ourraal..cb..00000
httpst::: BtLtttts////otorrre ottp
pcbi 1obi.100936.
John Guckenheimer and P. J. Holmes. Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields. Applied Mathematical Sciences. Springer- Verlag, New York, 1983. ISBN 978- doi:
0-387-90819-9.
1-6177-----6121-140000 1140-2. URL https: spr wpgengerrirrrrreeeee.rrerrreeeee com/
gp/book//7803879000
Mikael Henaff, Arthur Szlam, and Yann LeCun. Recurrent Orthogonal Networks and Long- Memory Tasks. In International Conference on Machine Learning, pp. 2034-2042. PMLR, June 2016. ISSN: 1938-7228.
URL httph:://ppcoeeeeeeeeeeeeee
html
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, doi: (00116//neco.1199.988.1 URL
18:11735-1880, November 1997. ISSN 0899-7667.
wet../.......................... 1997. 9...99...9...3...
Kohei Ichikawa and Kunihiko Kaneko. Short term memory by transient oscillatory dynamics in recurrent neural networks. arXiv:2010.15300 August 2021. URLattp://aaxxv..rrgg//////
201 0. 15308
arXiv: 2010.15308.
Cecilia Jarne. The dynamics Recurrent Neural Networks trained for temporal tasks and the eigenvalue spectrum. arXiv:2005. /3074, June 2020. URLhttpp//arxii wwtp://ggg/bbb//bb 13074 arXiv: 2007.13074.
Ian D. Jordan, Piotr Aleksander Sokót, and II Memming Park. Gated Recurrent Units Viewed Through the Lens of Continuous Time Dynamical Systems. Frontiers in Computational Neu- roscience, 15:67, 2021. ISSN 1662-5188. doi: 10.3389/fncom.2022 20211228111..... URL https:
// //www. f front e wtw.:cniiersiiioorrrrri.iii. 3389/ fncom. 2021. 678158
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and Understanding Recurrent Net- works. pp. 11, 2016.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Sto- ica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv: 18070500 2018.
10

---

Under review as a conference paper at ICLR 2022
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations recurrent net- works. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL http:: ptatpp::/prroceed:: ngs neurips cctppapprr/211///11ee BE5G1726667//BBBEBFF1CCCC22322  Papero pdf
Valerio Mante, David Sussillo, Krishna V. Shenoy, and William T. Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 503(7474):78 84, November 2013. ISSN 1476-4687. doi: 00.10388nature12742 URL https: / www. nature. com/
atticles/naturel27.22
David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural Computation, 25(3):626-644 649, March 2013. ISSN 1530-888X. doi: 10. 10.116//ECO.aa0000
Anonymous. Quasi-periodic attractors carry stable gradients.
Ingo Wegener and R. Pruim. Complexity Theory: Exploring the Limits of Efficient Algorithms. Springer-Veriiee Verlag, Berlin, Heidelberg, 2005. ISBN 3540210458.
Guangyu Robert Yang. Madhura R. Joglekar, H. Francis Song, William T. Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cogni- tive tasks. Nature Neuroscience, 22(2):297-300 February 2019. ISSN 1097-6256, 1546-
1726. doi: 10. 10.1008//41999--08-03002222 URL http: /tt::///www... http://wwwwatureeeeeeeee nawure.ataa..... com/ articles/ 641593-018-0310-2

---

Under review as a conference paper at ICLR 2022
A
SLOW MANIFOLD MAPPING
Algorithm 1: Slow Manifold Map
1:
procedure MAP(S, K)
2:
Q
P- (0 1, 1,...
1}
time-steps available symbols
3:
1
4:
for each element i @
5:
for each element j of P do
6:
q - Qi p P.
7:
8:
for iterations do Generate
> we chose < 1000 random input sequence the RNN where 7/ 7
9:
10:
V/ 7
Uq --p p
12:
Run both fy and v through the RNN and save the hidden-states of each at t = S Record which & neurons are sufficiently different threshold of 10-3 was used Approximate E hyg] when v is inputted to the RNN Vk for M iteration do
14:
15:
we chose M <- 1000 and n 4 D random input sequence to the RNN where %g * Run 7 through the RNN, but alter h4 E[(hs] for the most frequently recorded neurons at / = S during the forward pass
16:
Generate 7
if Ychoice = p at step q decoding and all other outputs are correct then record that this iteration was a success
19:
number of successes less than M and n S Temax then we chose mmaz 15
21:
72 f- 12 +
22:
go 15 else if n >> Umaz then
23:
24:
Disregard all but the n recorded neurons for n which gave the most successes
25:
else
26:
Disregard all but the n recorded neurons return The remaining values for each set of recorded neurons for all qp-pairs
12

---

Under review as a conference paper at ICLR 2022
B ALL PCA PLOTS
Time-Step:
Step:
Time-Step:
15
10
55
50
-15
-1.
Time-Step:
Time-Step:
Time- Step:
PC2
20
1.2
03-
0.-
455
-15
-2.0
-28
-2.6
-15
-16
-65
60
25
1.
35
24
-2
-1.5 -3.0 -55
Eo
05
10
15 25
Time Step: 7
Time-Step:
2.-
Time-Step: 10
15-
1.0
45-
65
02
-15-
-12
16-
-33
25
15 20
1.0
PC1
Figure 10: Equivalent plots Fig. (Right), for each of neurons used to encode each additional time-step during the encoding phase of VDCM.
C RESET THRESHOLD
We'd like our threshold be relatively low (i.e close zero), where exhibits little variance about Ç. Furthermore, we need to ensure that nonempty set of neurons exists for each set of neurons tuned each encoding time-step. such that each reset for the first time during the correct decoding time-step. For example, if neuron is tuned to encode information presented at t 5, then when the decoding period of the trial begins, the neuron should remain "not-reset" up until the fifth time-step of readout, where it will then properly reset. This will also indicate which neurons described in section 3 are most important for holding onto memory at each encoding time-step. Fig. further details reasoning for choice of Ç, which we to 0.255.
13

---

Under review as a conference paper at ICLR 2022
0.20
Density
0.15
16
0.10
2
0.05
Time- Steps
0.0
0.2
0.4
0.6
0.8
1.0
(1 z1) -t)
Figure 11: Plot indicating how we chose threshold Ç on K, where values under this threshold are considered "not-reset", and those above are considered "reset." Top: histogram showing the sample mean values each neuron takes across steps the decoding phase, over one thousand trials. Notice the bimodal distribution, The green curve represents the standard deviation of the average values of for each bin of the histogram. We want pick with relatively low variance ensure that individual neurons do not cross back and forth over the threshold trial to trial. Bottom: For varying k, white indicates values of ( where neurons tuned analysis. We want our threshold to not only be small, but to contain at least one neuron for each encoding sime to be reset appropriately.
this metric for most neurons is usually close to zero
given time-step in the encoding phase shown reset by
D TRAINING RESULTS
were chosen using Bayesian optimization (Ax https://ax.dev/ from the Ray Tune library) (Liaw et al. 2018) with validation likelihood used as the decision criterion. Specific values for the learning rate, learning rate scheduler, learning rate decay factor and gradient clipping were chosen over 100 iterations of the tuning algorithm All models were trained using Adam, with a batch size of 200; models used for hyper parameter optimization were trained for at most 100 epochs, while the models that were ultimately chosen were trained for fixed 200 epochs. Standard cross- entropy loss was used as the loss functional. Unless implied otherwise by the name, models were initialized using the default method in PoTorch, which internally samples from uniform distribution u(- VN VN with being the number of hidden units.
14

---

Under review as a conference paper at ICLR 2022
Test Performance on VDCM Task
Loss
Acc. per timestep
Acc. per sequence
HNN onth mii.
GRU
KNN
Orthagonal RNN
0.6
0.6
LSTM
0.75
0.4
0.4
0.50
0.25
0.2
0.2-
0.0
0.0
Training Epochs
Figure 12: Test performance for trained RNNs on VDCM. Colors indicate mean activity of each architecture and initialization strategy used (10 networks each). The shaded region about each curve depicts the variance across networks. Left: cross-entropy loss across epochs. Middle: traditional network accuracy. Right: depicts what fraction of test trials the network outputted the sequence with mistakes. Notice that network other than GRU got close to learning VDCM.
Test Performance Parity Task
Loss
Acc. per timestep
per sequence
1.50
10-
10-
RNN
1.25
08-
0.-
GRU
1.00
Orthogonal KNN
0.6
0.6
6.00
0.0
0.0
50
100
150
200
50
100
150
200
50
100
150
200
Training Epochs
Test performance for trained RNNs on modified parity bit task. Colors indicate mean activity of each architecture and initialization strategy used (10 networks each). The shaded region about each curve depicts the variance across networks. Left: cross-entropy loss across epochs. Middle:
Figure 13:
traditional network accuracy. Right: depicts what fraction of test trials the network outputted the sequence with no mistakes. LSTM does manage to learn this task.
Test Performance on Translation Task
Loss
Acc. per timestep
Acc. per sequence
RNN ortho init.
0.8
GRU
RNN
0.6
0.6
LS TM
0.75
0.4
100
0.4
0.90
0.25
0.2-
0.2-
0.0
Training Epachs
Figure 14: Test performance for trained RNNs a binary character translation task. Colors indicate mean activity of each architecture and initialization strategy used (10 networks each). The shaded region about each curve depicts the variance across networks. Left: cross-entropy loss across epochs. Middle: traditional network accuracy. Right: depicts what fraction of test trials the network outputted the sequence with no mistakes. LSTM leams this task the best, but is still unable to consistently output entire the desired sequences with no mistakes.
15

---

Under review as a conference paper at ICLR 2022
E
TRAINED NETWORK HIDDEN-STATES ACROSS TRIALS OF THE TEST SET
10
Hidden-State State Dimension Index
8
7-
5
2
20
40
60
80
100
120
140
time-steps
Figure 15:
Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
20
Hidden-Staie State Dimension Index
19
17
16
14
13
12
11
20
40
60
80
100
120
140
time-steps
Figure 16: Superimposed trajectories thousand trials of VDCM, from perfectly trained GRU.
16

---

Under review as a conference paper at ICLR 2022
30
Hidden-State Dimension Index
29
28
27
26
25
24
23
22
21
0
20
40
60
80
100
120
140
time-steps
Figure 17:
Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
30
Hidden-State Dimension Index
29
28
27
26
25
24
23
22
21
0
20
40
60
80
100
120
140
time-steps
Figure 18: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
17

---

Under review as a conference paper at ICLR 2022
50
Hidden-State Dimension Index
49
48
47
46
45
44
43
42
41
0
20
40
60
80
100
120
140
time-steps
Figure 19:
Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
60
Hidden-State Dimension Index
59
58
57
56
55
54
53
52
51
0
20
40
60
80
100
120
140
time-steps
Figure 20: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
18

---

Under review as a conference paper at ICLR 2022
70
Hidden-State Dimension Index
69
68
67
66
65
64
63
62
61
0
20
40
60
80
100
120
140
time-steps
Figure 21:
Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
80
Hidden-State Dimension Index
79
78
77
76
75
74
73
72
71
0
20
40
60
80
100
120
140
time-steps
Figure 22: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
19

---

Under review as a conference paper at ICLR 2022
90
Hidden-State Dimension Index
89
88
87
86
85
84
83
82
81
0
20
40
60
80
100
120
140
time-steps
Figure 23: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
100
Hidden-State Dimension Index
99
98
97
96
95
94
93
92
91
0
20
40
60
80
100
120
140
time-steps
Figure 24: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
20

---

Under review as a conference paper at ICLR 2022
110
Hidden-State Dimension Index
109
108
107
106
105
104
103
102
101
0
20
40
60
80
100
120
140
time-steps
Figure 25: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
120
Hidden-State Dimension Index
119
118
117
116
115
114
113
112
111
20
40
60
80
100
120
140
time-steps
Figure 26: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
21

---

Under review as a conference paper at ICLR 2022
130
Hidden-State Dimension Index
129
128
127
126
125
124
123
122
121
0
20
40
60
80
100
120
140
time-steps
Figure 27: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
140
Hidden-State Dimension Index
139
138
137
136
135
134
133
132
131
0
20
40
60
80
100
120
140
time-steps
Figure 28: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
22

---

Under review as a conference paper at ICLR 2022
150
Hidden-State Dimension Index
149
148
147
146
145
144
143
142
141
0
20
40
60
80
100
120
140
time-steps
Figure 29: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
160
Hidden-State Dimension Index
159
158
157
156
155
154
153
152
151
0
20
40
60
80
100
120
140
time-steps
Figure 30: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
23

---

Under review as a conference paper at ICLR 2022
170
Hidden-State Dimension Index
169
168
167
166
165
164
163
162
161
0
20
40
60
80
100
120
140
time-steps
Figure 31: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
180
Hidden-State Dimension Index
179
178
177
176
175
174
173
172
Mc MCXMAWWOO
171
0
20
40
60
80
100
120
140
time-steps
Figure 32: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
24

---

Under review as a conference paper at ICLR 2022
190
Staden-SSate Dimension Index
189
188
187
186
185
184
183
182
181
0
20
40
60
80
100
120
140
time-steps
Figure 33: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
200
Hidden-State Dimension Index
199
198
197
196
M
195
194
193
192
191
0
20
40
60
80
100
120
140
time-steps
Figure 34: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
25

---

Under review as a conference paper at ICLR 2022
210
Hidden-State Dimension Index
209
208
207
206
205
204
203
202
201
0
20
40
60
80
100
120
140
time-steps
Figure 35:
Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
220
Hidden-State Dimension Index
219
218
217
216
215
214
213
212
211
20
40
60
80
100
120
140
time-steps
Figure 36: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
26

---

Under review as a conference paper at ICLR 2022
230
Hidden-State Dimension Index
229
228
227
226
225
224
223
222
221
20
40
60
80
100
120
140
time-steps
Figure 37: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
240
Hidden-State Dimension Index
239
238
237
236
235
234
233
232
231
0
20
40
60
80
100
120
140
time-steps
Figure 38: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
27

---

Under review as a conference paper at ICLR 2022
250
Hidden-State Dimension Index
249
248
247
246
245
244
243
242
241
20
40
60
80
100
120
140
time-steps
Figure 39: Superimposed trajectories one thousand trials of VDCM, from perfectly trained GRU.
28


Reviews:
[{"review_id": "wL0O4MAYiQX", "reviewer": "Reviewer_MToe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shed light on the encoding and decoding process of GRU, explaining its capability and behaviours. Overall, the authors' hypotheses are well-validated through empirical evidence and informative visualization of the synthetic task. Among the findings,  the synthetic solution is the most interesting and gives novel insights into GRU's operation. On the other hand, the writing is hard to follow. Here, the paper limits to demonstrating the phenomenon and leave the action for future work, which is somewhat incomplete. \n\nQuestions and comments:\n- The term \"slow manifold\" is mentioned frequently without any explanation. From the writing, it seems to simply refer to the fact that during the delay phase, some neurons change slowly. To make the paper self-contained, a brief definition would be appreciated. \n- Algorithm 1 is critical for readers to understand Section 3. Please describe it in more detail in the main manuscript. Or maybe add a simplified version of Algorithm 1. \n- Section 3 seems to suggest that GRU \"memorizes\" all pairs of step-element during the encoding phase. If so, it cannot generalize to test data with a slightly different S. Did the authors test this case?\n- Fig. 3 is hard to read. Top's y-axis: what does \"accuracy\" mean? Is there any meaning in the point's colour?\n- Fig. 4-left, what is the meaning of the big horizontal line at neuron 60?\n- Fig. 5-left, the label K is confused with the number of symbols\n- Fig.5-right, any explanation for the exception at t=3? How does the GRU perform at this step? If it still decodes correctly at t=3, what makes it possible?\n- Sec. 5, did you verify your synthetic solution by comparing it with the parameters of the trained GRU. Is this possible that, after training, GRU can converge to your synthetic solution?", "summary_of_the_paper": "The paper analyzes GRU's underlying mechanisms that store and retrieve information in the delay copy task of a sequence of K symbols. It proposes a perturbation-based method to determine which neurons are responsible for encoding a step-element pair. The paper then shows that at each step of the decoding phase, certain neurons are reset by GRU, generally mapping to those tuned to hold information of step-element pair.  Finally, the paper provides a synthetic solution to the delay copy task for the case K=2.", "summary_of_the_review": "An interesting paper, yet the writing needs improvement. It is important to understand the underlying mechanism of models like GRU. However, I am unsure if the presented result is useful enough for designing a better mechanism for RNN/GRUs.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3IjamT7um3_", "reviewer": "Reviewer_AM8u", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths:\n- Thorough experimentation and exciting insights.\n    - Through perturbation analysis, the paper finds that only 73 of 250 state elements encode memory.\n        - PCA analysis solidifies this finding in Figure 4.\n    - A reset measure $\\kappa$ is defined in section 4, the new measure allows visualization of decoding resets in Figure 5.\n- Using the discovered mechanisms the paper proposes a synthetic solution to the variable length memory problem.\n- Source code with a few docstrings is available.\n- The paper is a good fit for ICLR.\n\n\n#### Weaknesses:\n- Related Work: Previous work by Karpathy et al. studies the behaviour of various recurrent cell gates across recurrent architectures for natural language processing tasks. The study appears to be relevant for this article. Authors may want to be fair and consider a citation?\n    - Karpathy, Andrej et al. \"Workshop Track - ICLR 2016 Visualizing and Understanding Recurrent Networks.\" (2016).\n\n- The description of the experiments was often hard to read. Speaking of 'neurons' instead of reset-gate-neurons,\n  update-gate-neurons or hidden-state-neurons, for example, made it harder at first.\n  Readers can infer the exact setting from context but should not have to do that.\n\n#### Questions:\n- Figure 1: Hidden state neurons mean we are looking at the output from the tanh in equation (3)?\n- Figure 2: I personally found it tricky to understand what is going on here.\n    I could follow the description of the argmax,\n    but what is the difference between the \"Projection to hyperplanes from hidden state\" and the white points?\n    If the white points are the entries of $\\mathbf{y}_0, \\dots , \\mathbf{y}_n $$ aren't these points the projection?\n    Why is the projection along the value axis?\n- Figure 3, 4: We are looking at hidden neurons, aren't we?\n- I think neuron is always shorthand for hidden-neuron?\n  If that is true, the authors may want to consider explicitly stating this somewhere?\n\n#### Minor remarks:\n- Figure 4, caption: ...every eight column corresponds... ( switch plural columns to singular )", "summary_of_the_paper": "The paper studies GRU cell dynamics on the variable-length memory problem in depth.\nBuilding on top of the uncovered mechanisms a synthetic solution is proposed.", "summary_of_the_review": "To the best of my knowledge, the paper appears experimentally sound and different from previous work.\nI firmly believe that we need more work trying to uncover the inner workings of neural networks.\nI am therefore recommending acceptance in the proceedings, assuming that the differences to the analysis from Karpathy et al. (citation in main review) will be discussed briefly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2Rg8dhI7LAl", "reviewer": "Reviewer_afLb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In terms of strengths, I think the authors definitely show that for this specific task and architecture, the GRU is learning to use working memory in a more or less explainable way.\n\nIn terms of weaknesses, I feel that this paper lacks novelty. Their work is not technically novel because they are using a GRU and using a super well studied task. I don't think I took away any sort of new information from their analysis either. Hence I am not sure what the contribution of this work is to the community. I would also like to comment on how busy the author' figures and experiment descriptions are and how hard they are to follow. \n\nIf the authors wanted to improve the novelty of their work, they can maybe focus on state of the art architectures and more complex tasks.", "summary_of_the_paper": "This paper attempts to understand how a RNN goes about solving the Variable Copy Delay Memory task in fine detail. Their model has been trained perfectly on this task and so they are able to focus on how it goes about changing its cell values in the case. The authors present different metrics to track resetting and memorization behavior by the GRU's neurons.", "summary_of_the_review": "This work lacks novelty, is hard to follow, and shouldn't be included in ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "2KJo2uP8WTt", "reviewer": "Reviewer_TN5R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Studying the mechanism by which an RNN model stores and retrieves memory is an important question that can give insights into how these networks work.\n\nThis paper's approach of trying to reverse engineer the mechanism of a trained GRU to understand this is interesting. Moreover the finding that slow manifolds are used for the purpose of storing memory of a sequence is interesting.\n\nBut the paper does not consider or test alternate hypotheses other than slow manifolds. For example, no attempt is made to empirically show that the GRU is not using pseudo-line attractors. This is a major flaw of the paper.\n\nIt is also not clear why the authors specifically consider the GRU architecture, especially since it seems like using an LSTM would make it easier to reason about when information enters and exits the cell.\n\nThe writing of the paper is also very confused and hard to parse in many places. The sequence of arguments made for slow-manifolds is sometimes hard to follow.\n\nFor example:\n- Only 3 units are shown from the trained GRU, with text indicating that this is representative of all other neurons. But later the authors say only 73 neurons were analysed. Might be a good idea to show the plots for all neurons, possibly in the supplement.\n- It might help to explain formally what a slow manifold is.\n- The perturbation based experiment is hard to understand, as is the PCA analysis.\n- Minor: Fig. 3: what are the colors?", "summary_of_the_paper": "\nThe authors study the mechanisms that a GRU network uses to store and retrieve memory in a variable delay copy task. the authors show that the GRU uses slow manifolds to do this, and construct a network for a smaller variant of the task by hand.", "summary_of_the_review": "Overall, while the goals of the paper are well-founded, and their analysis seems potentially interesting, a combination of lack of clarity and some gaps in their overall argument weakens the paper significantly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies how recurrent neural networks, and more specifically GRUs, store and access information. The authors analyze the solution obtained by gradient descent to the variable delay copy memory task for discrete sequences. They use concepts from dynamical systems, such as slow-manifold, to understand the behavior of the learned model. Finally, based on this analysis, the authors propose a synthetic solution to a simplified version of the delay copy memory task.

Overall, while the scores for the paper are rather positive, I still have concerns about the paper, based on the reviews and discussion. I do not believe that these concerned were well addressed by the authors in their rebuttal. First, I tend to agree that the paper is somewhat lacking novelty and insightful findings (reviewers TN5R, afLb, MToe). For example, I think that tools from dynamical systems are mostly useful to analyze RNNs when the input is constant (Jordan et al., 2019). In the case of the copy task, this corresponds to the "delay" period, where in practice the hidden state is almost constant. This behavior is easily explained by the value of the update gate, close to 1. I thus agree that other hypotheses than slow manifold should be discussed to explain how GRUs store and access information, and that the benefits of using dynamical systems is not obvious. Moreover, I believe that previous solutions to the copy task (eg, from Henaff et al.) could be extended to the variable setting by adding a gating mechanism to these solutions. In particular, Henaff et al. claimed that LSTM could solve this task empirically, while the authors claim otherwise.

Second, after reading the revised version a couple of times, I still find the paper hard to follow (MToe, afLb, TN5R). For example, I think that the concept of slow manifold is not introduced properly, and in particular, how it applies to the learned solution is not clear. More generally, I found the sections regarding how information is stored and accessed a bit confusing. Finally, I think that the studied task is simple, and probably does not provide strong insight about the working of recurrent networks. Specifically, LSTMs tend to perform similarly or slightly better than GRUs on many tasks, while the authors claim that this architecture cannot solve the studied task.
Decision: Reject