Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary
Authors: Takashi Morita
Venue: Accepted by TMLR ()

Abstract: This study reports an unintuitive finding that positional encoding enhances learning of recurrent neural networks (RNNs). Positional encoding is a high-dimensional representation of time indices on input data. Most famously, positional encoding complements the capabilities of Transformer neural networks, which lack an inherent mechanism for representing the data order. By contrast, RNNs can encode the temporal information of data points on their own, rendering their use of positional encoding seemingly redundant/unnecessary. Nonetheless, investigations through synthetic benchmarks reveal an advantage of coupling positional encoding and RNNs, especially for handling a large vocabulary that yields low-frequency tokens. Further scrutinization unveils that these low-frequency tokens destabilizes the gradients of vanilla RNNs, and the positional encoding resolves this instability. These results shed a new light on the utility of positional encoding beyond its canonical role as a timekeeper for Transformers.

Published in Transactions on Machine Learning Research (11/2024)
Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary
Takashi Morita
tmorita@alumm mit. com
Academy of Emerging Sciences Center for Mathematical Science and AT
Chubu Universily
Reviewed on OpenReview: https: openreview. net/ forun? id= PtnwId13SF
Abstract
This study reports an unintuitive finding that positional encoding enhances learning of re- current neural networks (RNNs). Positional encoding a high-dimensional representation of time indices on input data. Most famously, positional encoding complements the capa- bilities of Transformer neural networks, which lack an inherent mechanism for representing the data order. By contrast, RNNs can encode the temporal information of data points on their own, rendering their use of positional encoding seemingly redundant/ /unnecessary. Nonetheless, investigations through synthetic benchmarks reveal an advantage of coupling positional encoding and RNNs, especially for handling a large vocabulary that yields low- frequency tokens. Further scrutinization unveils that these low-frequency tokens destabilizes the gradients of vanilla RNNs, and the positional encoding resolves this instability. These results shed a new light on the utility of positional encoding beyond its canonical role as a timekeeper for Transformers.
1 Introduction
Since thcir invention in 2017, Transformer neural networks (Vaswani et al., 2017) have taken over the gold standard processor/eee generator of time series data, from the more traditional models in the form of recurrent neural networks (RNNs; Elman, 1990). One of the most striking differences between the two models is the way they represent the temporal information of the data (i.c., the order of individual data points, or tokens, in the time series). On the one hand, RNNs encode temporal information by updating their internal state based on the input observations as well as the previous state. On the other hand, Transformers per se do not have mechanism represent the temporal order of data points; consequently, they rely on an external "clock" called positional encoding.
processor/generator
Positional encoding is a high-dimensional representation of time indices on input data (Gehring et al., 2017). For instance, its most basic and popular implementation utilizes sinusoidal waves of various predefined frequencies (Vaswani et al., 2017). Positional encoding "timestamps" input tokens by addition concatenannnnnnn of these vectors the corresponding input embeddings. Unlike RNNs, the time representation via iosational encoding invariant input values (i.c., autonomous) until they are processed together by network.
alternative form of time representation that replaces RNNs (in combination with Transformers), positional encoding and RNNs not fundamentally incom- patible; that is, inputs to RNNs can be "rodundantly" augmented by position-cncoding vectors. Indeed, autonomous activities of biological neurons such as neural oscillations are thought to play an important role time perception (Matell & Meck, 2004; Buhusi & Meck, 2005), as well as other perceptual processes (including vision, Milner, 1974; Eckhorn et al., 1988; Gray et al., 1980; and odors, Milner, 1974; Eckhorn et al., 1988; Gray et al., 1989; Wehr & Laurent, 1996; Perez-Orive et al., 2002) and motor control (Marder & Bucher, 2001; Gross et al., 2002; Proctor et al., 2010).
Although positional encoding has been considered

---

Published in Transactions on Machine Learning Research (11/2024)
Accordingly, the present study explores "redundant" positional encoding imputs to RNNs, utilizing syn- thetie benchmarks. The experiments will unveil that positional encoding aids RNNs in handling a wider variety of discrete inputs (i.e., a larger vocabulary) than those without positional encoding.
The contributions of this study are summarized as follows:
Difficulties training RNNs on a large vocabulary are demonstrated through systematically designed benchmark tasks. This problem has not been identified in the previous studics- or at most has received little attention from the research community despite its potential relevance to empirical applications.
This identified problem with training RNNs on a large vocabulary is clucidated by the gradient insta- bility induced by fow-freqenccy tokens, which necessarily arise from the expansion of the vocabulary size.
A novel efficacy of positional booddnggbesides its timestamping function for Transformers- -ss shown by coupling it with RNNs. Specifically, positional encoding will be shown to mitigate the large-vocabulary problem by stabilizing the gradients of RNNs against disruptions caused by low- frequency tokens.
2 Related Studies
2.1 Theoretical and Empirical Computational Power of (Vanilla) RNNs
Mathematically, RNNs are known to be Turing-complete; they can simulate Turing machines their weights have infinite precision and are ideally tuned (Siegelmann & Sontag, 1992; 1994; 1995; Siegelmann, 1996; 1999; Chen et al., 2018).1 Indeed, even RNNs with random recurrent and input-to-hidden weights (called reservoir computers; Maass et al., 2002; Jaeger & Haas, 2004) can achieve the universal approximation property if their hidden-to-outpput weights are idealized (Grigoryeva & Ortega, 2018; Gonon & Orlega, 2020). These theoretical results have motivated the use of RNNs for processing complex time series such as human languages (Sundermeyer et al., 2012; Graves, 2013) and weather (Shi et al., 2015).
In practice, however, RNN weights are bounded by finite precision and must be optimized based on finite obsorvations of data. These settings impose limitations on the empirical capabilities of RNNs (Chen et al., 2018; Weiss et al., 2018). For example, empirical RNNs cannot store infinitely many observations in their memory, state vector(s), and the memorized information decays over time. This latter problem with the memory duration has attracted the interest of rescarchers, leading to extensive exploration of RNN architectures for longer-lasting memory (Hochreiter & Schmidluber, 1997; Arjovsky et al., 2016; Neil et al., 2016; Chang el al., 2017; Jing et al., 2017; 2019).
More recently, research into prolonged memory retention has shifted towards continuous-time models (Voelker et al., 2019; Gu al., 2020). Instead of representing the memory of an input sequence through discrete-time modifications of latent state, these models approximate the input history by linear combination of or- thogonal polynomials in continuous-tina space. Consequently, the coefficients of these polynomials yield fintee-diennsonal representation of the input sequence (termed the High-Order Polynomial Projection Op- erator, or HiPPO) and the dynamics of these coefficients can be articulated through an ordinary differential equation (ODE; Gu et al., 2020). This concept of contimuous-time memory representation has subsequently been extended to neural state-space models by replacing the fixed state matrix in the ODE corresponding manually selected basis of polynomials with a learnable one, while constraining its structure to a diag- onal (plus row-rank) matrix (Gu et al., 2021; 2022a;b; Gu & Dao, 2024). Most strikingly, with additional refinements, the latest state-space model has achieved superior language modcling performance, rivaling that of Transformer-basee models.
II order be Turing-complete, RNNs must also be allowed to read entire input prior their output emission; they at most context-sensitive they have return an output at each time step upon the receival of input token (Chen al., 2018; Weiss al., 2018).
2

---

Published in Transactions on Machine Learning Research (11/2024)
2.2 Positional Encoding
Positional encoding is a ligh-dinennsooal representation of temporal structures in input data (Gehring et al., 2017). The primary demand for this representation scheme comes from Transformers. Unlike RNNs, Transformers lack an inherent mechanism for representing input arrangements. Accordingly, input tokens to Transformer are "time-stamped" via
of a position-encoding vector.
In the original implementation of Transformer, token positions were encoded by sinusoidal waves of various predefined frequencies (Vaswani et al., 2017). While this original encoding scheme is powerful enough for a wide range of tasks, researchers have explored for other possibilities as well. For example, the well-known BERT pretraining for natural language processing employed learnable embeddings to encode token positions (Devlin et al., 2019). Some studies also suggested that the combination of the sinusoidal and learnable encoding improves model performance (Dai et al., 2019). There also an option to encode the distance between tokens rather than the elapsed time from the sequence onset (Shaw et al., 2018; Dai et al., 2019).
Besides Transformers, positional encoding is also used to represent the elapsed time in diffusson processes (Ho et al., 2020; Song et. al., 2021). Moreover, the effectiveness of positional encoding is not limited to lem- poral information; previous studies in three-dimensional mesh/ mesh/point-clous /point-cloud modeling report that sinusoidal transformation of spatial data improves model performance compared to the raw coordinate represcntation (Mildenhall et al., 2020; Jun& & Nichol, 2023).
Despite the widespread use of positional encoding in various areas of machine learning today, its applications to pure RNNs remain largely unexplored. To the best of the author's knowledge, only two studics have previously explored position-encoded RNNs.2 Karaníkolos & Refanidis (2019) reported that position- encoded LSTM outperformed a vanilla LSTM well a shallow Transformer (with four layers) in text summarization. In another study predating the proposal of sinusoidal positional encoding in the deep learning community- Limcere-Lamarre et al. (2016) showed that oscillatory signals at random frequencies improved the performance of random RNN (i.e., reservoir computer) in a timing task, assessing the model's 's memory duration by its ability to produce a (smoothod) output pulse after a specified time interval from an onset signal.
Similarly, the time index in time series data has rarely been utilized directly by RNNs, presumably because of its "redundancy" along the functionality of RNNs. As exception, Neil et al. (2016) proposed a periodic gating mechanism for updating the state and memory cell of LSTM. This periodic gating was scheduled according to a triangular wave interspersed with a plateau at the floor value (= 0.0; the frequency, phase, and the duration of the wave phase were learnable paramoters).
3 Methods
3.1 Task
Effects of positional encoding on RNNs were investigated through the reverso-ordering task;² given a sequence of random integers, RNNs were trained to reconstruct them in reverse order (e.g. 8,29, 2, 11 11, 29, 8; Fig. 1).
3.2 Model Architecture
The investigations in this study were based single-layer gated recurrent unit (GRU; Cho et al., 2014) and long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) combined with input-embedding and cntput-projectto layers (Fig. 1). Besides these RNNs, a neural state-space model, S4D (i.e., S4 with a diagonal state matrix; Gu et al., 2022a), was also investigated. Fach the integers in the input sequences was first embedded and concatenated with the positional encoding, and then fed to the RNN/SAD. After
PIn other studies, positional encoding and RNNs have served submodules within complex models. typically in conjunction with attention mechanism (Kim et al., 2018; Song al., 2020).
s Investtigations of additional tasks are reported in Appendix A.
4The concatenation the positional encoding and input embeddings inflates the number of learnable parameters in the input-to-hidde hidden weights of RNNs in comparison to the vanilla models. However, this additional parameterization is designed
3

---

Published in Transactions on Machine Learning Research (11/2024)
Reversed 11
2
29
8
1
1
Linear + argmax
RNN
RNN
RNN
RNN
RNN
RNN
RNN
RNN
Positional Encoding
L+1
27.
21.
2L
B
8
H
Output Query
Embedding
(Leanable Vector)
Input
29
2
11
(Random Integers)
Figure Illustration of the model structure and the reversc-ordering task.
reading the entire input sequence, the network received a command to return the output. This command represented in the form of time-invariant learnable vector, and fed to the RNN in place of the input embedding (ef. Arjovsky et al., 2016). The outputs from the RNN/S4D module were linearly projected into the classification logits, whose cross-entropy loss against the target sequence was used to optimize the entire network. Model predictions in the testing phase were defined by the argmax these logits for each time step.
This study adopted the canonical sinusoidal positional encoding designed for Transformers (Vaswani et al., 2017, see Appendix for discussions on alternative implementatiinns); specifically, each time step I encoded by the Dpos- dimenninni vector, (PE,,..... ..e.....,,PP.,......... )T, defined as follows:
PF122 == sin
(1)
2(1-1)
110000
Poor
PE,2141 = cos
pos
(2)
21--11
10000
For the sake of learning stability, the positional encoding was divided by Dposs2 so that the encoding vectors had the unit L2-norm. Note that the time step increased throughout both the input and output phases (i.c., t 1, 1,... L, L 1,...,24 2L where L represents the input length), without any hard-coded association between the input and output positions (i.e., no shared tinestamps).
3.3 Implementation Details
Across the experiments, the dimensionality of the hidden layer of the RNNs was set to 512. The embedding of the input integers and the memory cell of the LSTM also had the same dimensionality of 512. Similarly, the
not to impact the learning of input embeddings. Appeudix experimentally demonstrates that merely increusing the model size docs not culance the performance of RNNs.
The adjustment of the time and frequency indices ( and "i 51- 17) in Eqs. and aligns the 1-based indexing this paper (adopted for better readability) with the O-based indexing in Python.
4

---

Published in Transactions on Machine Learning Research (11/2024)
GRU LSTM
S4D
Vanilla
Position Deittin-Enncoee
32
256
256
64
Vocabulary Size
1024
1024
128
4096
4096
256
16384
16384
0.00
0.25
0.50
0.75
1.00
40 Daameau-Levenshter Distance
20
Accuracy
Figure 2: Token-wise accuracy (left) and sequence-wise reconstruction errors (right) of the reverse-ordering task performed by GRU/LSTMM/AD with and without positional encoding (labeled as "Position-Fanooedd" and "Vanilla" respectively). The input length fixed 64. The error bars represent the 95% confidence interval estimated from 10,000 bootstrapped samples of five training-test test trials with different random seeds. Each of the five trials held out 1024 random sequences (= 65,536 tokens) for computing the test accuracy.
hidden dimensionality of S4D was set to 512, while its state size (or the order of the Legendre polynomials) was maintained at the default value of 64.
The models were trained 300,000 iterations using the Adam optimizer (Kingma Ba, 2015) with the parameters (31, 92) = (0.9, 0.999) and no weight decay. The learning rale was linearly warmed up from 0.0 0.001 for the first 1,000 iterations, and then annealed according to the cosine schedule (Loshchilov & Hutter, 2017). The batch size was 512.
All the experiments were implemented in PyTorch (ver. 2.1.1; Paszke et al., 2017; 2019) and each training- test trial was executed on a single NVIDIA A100 GPU (with 80GB VRAM) hosted by the Academic Center for Computing and Media Studies, Kyoto University. The source code is available https: //gi thut.hhub.oo...... com/ skc-morita/positte tinitiiion-oiee encoded_rrme rnn.
4 Results
4.1
Key Findings
As a result, positional encoding improved the ability RNNs to handle larger vocabulary in the reverse- ordering task (Fig. 2). The position-encoded GRU and LSTM successfully reversed the input sequences of
TTe S4D implemented based the official code provided https://github. htwp://bb.ccoeetttteeeepccceee http:///tttttepppcc///////////aa models/s4ssdd.

---

Published in Transactions on Machine Learning Research (11/2024)
64 integers' drawn uniformly at random from the vocabularies of size 32-256 and 256-16,384, respectively, achieving the token-wise accuracy above 95%. By contrast, the performance of the vanilla models without positional encoding degraded as the vocabulary size increased. Similarly, positional encoding enhanced the capacity of S4D to handle large vocabularies. These improvements are also evident in the reduced sequence- wise reconstruction errors, measured by the Dameraue-Levenshtinn distance. It is of note that neither extra training iterations nor greater batch sizes improved the performance of the vanilla models.
4.2
Frequency Matters
The most apparent consequence of the increased vocabulary size was the reduced chance of observing indi- vidual vocabulary items (e.g., 1/256 + 1/16,384). Accordingly, additional experiments were conducted with non-uniformly distributed tokens to investigate the relation between their frequency vs. RNN performance. Specifically, the input vocabulary was evenly divided into FREQUENT and RARE groups, and the FREQUENT tokens had three times the probability of the RARE tokens; that is, the probability of each FREQUENT token was 7/8 x 2/K (where K denotes the total vocabulary size and was set to 64, 1024, 2048 for GRU, LSTM, and S4D respectively) whilist the probability of each RARE token was 1/8 2/K.
The training data consisted of 64 independent samples from this dual dual-frequency vocabulary. By con- trast, the Lest data were systematically constructed that each sequence included single "target" token (FREQUENT/ RARE) whose retrieval was evaluated for accuracy assessment, along with 63 "disturbants" that were either all FREQUENT or all RARE.
The experiment revealed that it was the disturbant lokens whose Trequency siguificantly impacted the per- formance of the vanilla RNNs and S4D (Fig. 3). On the one hand, the RARE targets were successfully retrieved as long as they were surrounded by the FREQUENT disturbants. On the other hand, the vanilla GRU struggled to recover the FREQUENT targets when the other imput tokens were filled with the RARE disturbants. The LSTM performance was also degraded, especially when the targets were positioned in the first quarter of the imput sequence (1 S t S 16). Similarly, the RARE disturbants were detrimental to the S4D; unlike the RNNs, however, the accuracy was worst when the targets were located in the middle of the input sequences (17 $2)..
By contrast, the position-encoded RNNs exhibited robustness to the frequency of the target and disturbant tokens. They achieved nearly perfect accuracies in most cases, except when the GRU processed the fully RARE data whose target was located in the first half of the sequence (1 I t S 32). Likewise, positional encoding enhanced the resilience of the S4D against the influence of RARE disturbants.
4.3 Analysis of Gradient Stability
To delve deeper into the influence of token frequency on the RNN performance, the gradients of the RNN latent states were scrutinized. the analysis, pairs of input sequences were processed by the RNNs trained on the dual-frequency vocabulary (comprising FREQUENT and RARE items; Fig. 4). Each pair of sequences shared the same initial token (t = 1; "target") but varied in the subsequent tokens (2 L L. "disturbants"). Then, gradients were computed for the distant mapping between the first and last updated states (i.e., time ( = 1 and 2L) of the RNNs using backpropagation through lime. The slabillly of RNN learning assessed by measuring the dot-product similarity of the gradients between the paired input sequences (after normalization over output dimensions).
Formally, the paired input sequences, denoted as A and B, established two distinct, but ideally similar map- pings, f(4) and f(B), from the first to the last latent state of the RNNs (Eig) (Z1), where E (A) The gradient stability of the RNNs was defined by the dot-product similarities between the normalized gradients of these paired mappings:
See Appendix B. for discussion robustness to variable input. length.
6

---

Published in Transactions on Machine Learning Research (11/2024)
Disturbant FREQUENT
RARE
Target
FREQUENT
RARE
Vanilla
Position-Eneoded
1-16
17-32
GRU
33-48
49-64
1-16
17-32
USTS
33-48
49-64
Target Position in Input
1-16
17-32
S4D
33-48
49-64
Accuracy
10
Figure 3: Token-wise wise accuracy of the roverso-ordering task performed by GRU/LSTM/S4D with and without positional encoding (labeled "Position-Encodees and "Vanilla" respectively). The vocabulary evenly split into FREQUENT and RARE groups (32+-32 for GRU, 512+512 for LSTM, and 1024+1024 for S4D), and the former was sampled three times more frequently than the latter. The input length was fixed at 64. The error bars represent the 95% confidence interval estimated from 10,000 bootstrapped samples of five training-test trials with different random seeds. Fach of the five trials held out 4096 test sequences
(= 262,144 tokens), consisting of a single "target" token (frequent or rare) surrounded by 63 "disturbants" (all frequent or rare). That is, sixteen test sequences were held out for each condition (Irequent/rare
target frequent/ prauuennt/rrreeeeee /rare disturbants target positions).

---

Published in Transactions on Machine Learning Research (11/2024)
Evaluate
Normalized Gradients:
(B)
0h
24
(B)
LB)
Dot-Product Similarity
021,1
(2)
IS
(4) aN
Normalized Gradients:
22,
(4)
20,4
821.1
021,
ISSDD
RNNN
RNN
RNN
RNN
RNN
RNN
RNN
RNN
Embedding (++ Positional Encoding)
Output Query (-> Positional Encoding)
1
1
29
11
Disturbants A Assessing Robustness
Target
8
18
31 Disturbants B
Figure Schematic illustration of the analysis of gradient stability. The RNN trained on the reverse- ordering task processed a pair of input sequences that shared the initial token (t 1; blue) but differed the rest (2 < L; referred to "Disturbant A/B", and colored in orange/purple) For each dimension
of the final RNN output h2L at time 2L, the distant gradient
[8haL.:
0hoz.s at the first updated
0=s,1 ..... 021.c2
latent state ž (= h in GRU; concatenation of the hidden and cell states LSTM, doubling the total dimensionality to 2D) was computed per input sequence via backpropagation through time (dashed lines). The gradient stability was defined by the dot-product similarity of the paired gradients normalized over the output dimensions by the coefficients a; ( {A, B}), whose definition is provided in Eq.
(TA Σ
D
(A) (a / 1!
(z),
(Z)),
(zi)
Σ 1=1
a
(A)
(B)
(2)
2
0h
(A)
ah
(3)
@zi.j
where the coefficients aj ( normalized the raw gradients (i.e., over the row vectors of the Jacobian matrix):
V (Z1) over the output dimensions i:= 1....;
O = V (1)1l ΣΦ / (Z)I)
(1
Σ
22)
2
(a)
/
Σ
D
Σ
D
=
d2i.5
221.5
(4)
Consequently, the stability metric emphasizes the consistency the paired gradients that both have greater L2-norm across the output dimensions.
It deserves special note that the mapping from the first. to the last RNN state was conditioned on the disturbant tokens occurring at 2 < t < L. Nevertheless, the oeverse-ordering task trained the networks to retrieve the initial token as their final output whatever tokens intervened in the middle. Thus, a well-trained RNN would maintain invariance its final state over the disturbants. Conversely, consistent gradient directions across varied disturbants would lead to successful learning, which premises the proposed analysis.
Unlike the RNN models, both the vanilla and position-encoded S4Ds achieved high accuracy over 96% for the initial target token (t = 1), regardless of the frequency of the target and disturbants. Accordingly, for the analysis of S4D, the target token was positioned in the middle at t = 23, where the vanilla model exhibited its poorest accuracy with the RARE disturbants. The disturbants were prefixed and suffixed to this target to construct input sequences. The prefix disturbants were shared between the paired sequences, thereby ensuring that the latent dynamics of the model remained identical up to the target token.
It should also be noted that the latent states of S4D are complex-valle valued (while its outputs are real-valued), and consequently, the gradients and their dot-product similarities are also complex-valued. For the sake of
8

---

Published in Transactions on Machine Learning Research (11/2024)
Disturbant FREQUENT
RARE
Target
FREQUENT RARE
Vanilla
Position- Encoded
0.8
0.6
CRU
0.4
0.2
0.8
0.6
0.4
Gradient Stability
0.98
DUS
0.96
0.94
2
3
2
Training Iteration
x10
Figure Gradient stability of GRU/LSTM/S4D trained on the reverse-ordering task with and without positional encoding (labeled as "Position-Eucoded and "Vanilla" respectively). For the GRU and LSTM, the stability was defined by the dot-product similarity of latent-to-latenl gradients after normalization over the output dimensions, conditioned on two input sequences sharing the initial "target" token (whose FRE- QUENT RARE distinction is represented by the line color), followed by FREQUENT RARE disturbants (represented by the solid vs. dashed lines). For the S4D, the targel token was positioned at (= 23, where the vanilla model scored the worst accuracy with the RARE disturbants. The disturbants were prefixed and suffixed to the target to construct input sequences. The prefix disturbants were shared between the paired sequences, ensuring that the latent dynamics of the model was guaranteed to remain identical up to the target token. The total input length was 63 = 22 + 1 41 = 64. The average similarity over 1024 input pairs times five trials plotted for every 5000 training iterations.
the present analysis, the complex-valued gradicnts were treated as the double-sized real arrays, and real- valued similarity was defined by Eq. 3. This equivalent to taking the real component of the complex-valued similarity, and intuitively natural given that a perfect alignment between complex gradient directions yields real-valued score of 1.0 (= the norm of a normalized complex vector). Additionally, the extra dimension the latent states representing the order of the Legendre polynomials was merged with the channel dimension, and the entire state treated flattened vector.

---

Published in Transactions on Machine Learning Research (11/2024)
Monitoring the gradients at training checkpoints revealed that RARE disturbants destabilize the learning of vanilla RNNs (Fig. 5). The similarity of the paired gradients decreased gradually (GRU) or rapidly (LSTM) when the networks were exposed to the RARE disturbants.
And most remarkably, positional encoding endowed the RNNs with robustness to these RARE disturbants. Both the GRU and LSTM maintained the high similarity of the paired gradients across the different tar- get/disturbant conditions.
By contrast, the impact of positional encoding on the gradient stability of the S4D was marginal; unlike the RNNs, the vanilla S4D was highly stable by itself against RARE disturbants throughout the training, even though there was a visible relative destabilization due to RARE disturbants compared to FREQUENT disturbants in the early stages of training, as well as an observable improvement by positional encoding. It is also noteworthy that the difference between the FREQUENT vs. RARE disturbants diminished after 10,000 training itcrations. Consoquently, gradient stability does not fully account for the decline in the accuracy of S4D in the presence of RARE disturbants, nor does it explain the enhancement brought about by positional encoding.
5 Discussion
5.1
Difficulties in Handling a Large Vocabulary
The present study introduced a novel challenge in training (vanilla) RNNs: large vocabularies. While inves- tigations into the manageable vocabulary size of RNNs appear to be a pertinent research arca-being crucial for empirical applications such as natural language processing- prres-uus studies were primarily dedicated to evaluating and improving the memory duration of RNNs, and the vocabulary size in these studics was typically set small (= eight; Arjovsky et al., 2016; Neil al., 2016; Chang et al., 2017; Jing et al., 2017; 2019; Voelker et al., 2019; Gu et al., 2020).
This study examined the RNN gradients and identified their destabilization when processing low-frequency tokens, which are necessarily included in large vocabulary. Specifically, inputs that do not contribute to baddient-based optimization at a target time step (e.g., tokens at 2 S t S L upon the retrieval of the initial token at t 2L in the reverso-ordering task) were found to be detrimental.
In general cases of time series processing, data points carrying crucial information for specific time steps become irrelevant otherwise. Consequently, each token exhibits dual nature both crucial and noisy- throughout the task, and processing rare tokens is particularly challenging presunably because they are irrelevant at most of the time while making a large impact on the learning through the greater loss to compensate for their fewer learning opportunities. Dealing with such "unignorable noise" presents a pervasive challenge for RNNs.
5.2 Functionality of Positional Encoding beyond the Timekeeper for Transformers
Although low-frequency tokens destabilize the gradient-based learning of RNNs, the present study also dis- covered that this issue can be alleviated by positional encoding. This enhancement of RNNNs via positional encoding is noteworthy because RNNs were specifically designed to process time series data on their own; hence, unlike Transformers, they are presumed to function without relying on an "external clock" (Siegelmann & Sontag. 1992; 1994; 1995; Siegelmann, 1996; 1999). Consoquently, position-cncoded RNNs have remained largely unexplored, with only two exceptions to the best of the author's knowledge (Vincent-Lamarre et al., 2016; Karanikolos & Refanidis, 2019). The findings of the present manageable vocabulary size due to the enhanced gradient stabiliy-broaddn the currently limited under- standing of the impact of positional encoding on RNNs.
the improvement in the
Additionally, the results of this study shed a new light on the utility of positional encoding. While positional encoding has been viewed as nothing more than input timestamps for Transformers, the present study demonstrated its efficacy in stabilizing the gradients of RNNs against disruption by low-frequency tokens. This novel functionality of positional encoding would not have been visible in Transformer studies, as the
10

---

Published in Transactions on Machine Learning Research (11/2024)
model can dynamically adjust the relevance of imput tokens through their attention mechanism and thus inherently mitigate the impact of disturbant tokens.
5.3 Limitations and Future Directions
A primary unresolved question in this study pertains to the mechanism behind the gradient stabilization by positional encoding. All the findings here are based on experimental investigations, lacking rigorous mathematical explanations for how and why the gradients of RNNs are destabilized by infrequent tokens and stabilized by positional encoding. Moreover, the present study primarily focused on the canonical implementation of sinusoidal positional encoding designed for Transformers (Eqs. 1,2), leaving it open which parameters of the sinusoidal waves (i.c., frequencies and phascs) are critical for gradient stabilization. Future research may broaden its scope to encompass more general forms of positional encoding, such wavelets and non-periodic signals (see Appendix D for preliminary comparison among sinusoidal, learnable, and randomly-fixed encodings).
Moreover, the analysis of gradient stability did not fully address the enhanced performance of the position- encoded state-space model (S4D). In terms of accuracy, the positioned-encode -ncodedd S4D) exhibited greater ro- bustness to infrequent tokens compared to the vanilla model, resembling the behavior observed in RNNs. However, the gradients of the vanilla S4D were too stable account for this descline in performance. This leaves the question open how positional encoding influences gradient-based learning of state-space models. Additionally, future studies may investigate broader range of state-space models including the state-of- the-art archilecture of Mamba (Gu & Dao, 2024)- to achieve comprehensive understanding of the interplay between positional encoding and these models.
In addition to these scientifically oriented questions, future studies could also address practical applications of position-encoded RNNs and neural state-space models. Although positional encoding enhanced model performance across different synthetic tasks (see Appendix A), the extent of this enhancement is task- dependent. Indeed, while Karanikolos & Refanidis (2019) reported the effectiveness of positional encoding for LSTM lext summarizer, the present study found no empirical advanlage for the language modeling task, aside from a slightly more rapid decline in training loss (sce Appendix E). Thus, positional encoding is not a panacea for arbitrary tasks, and further investigations are necessarily to determine when it is effective.
Acknowledgments
This study supported by JST ACT-X (JPMJAX21AN) and Core Research for Evolutional Science and Technology (JPMJCR17A4, JPMJCR22P6); JSPS Grant-in-Aid for Early-Career Scientists (JP21K17805) and for Scientific Research A (JP24H00774), B (JP22H03914), and C (JP2AK15087); and Kayamori Foun- dation Informational Science Advancement (K35XXVI16200) The author also gratefully acknowledges the support of the ACCMS, Kyoto University, regarding the use of their supercomputer system.
References
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Maria Flo- rina Balcan and Kilian Q. Weinberger (ods.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1128, New York, New York, USA, 20 22 Jun 2016. PMLR.
Catalin V. Buhusi and Warren H. Meck. What makes us tick? functional and neural mechanisms of interval timing. Nature Reviews Nearoscience, 6(10):755-776) 2005. doi: 10.1038/nmm1764.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. In 1. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. Recurrent neural networks weighted language recognizers. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of
11

---

Published in Transactions on Machine Learning Research (11/2024)
the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp- 2261-2271, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 118653//11/N881120
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN mcoder-decoder for statis- tical machine translation. In Proceedings of the 2014 Conference on Empirioal Methods in Natural Tan- guage Processing (EMNLP), pp. 1724 1734, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-111
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer- XL. Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988, Florence, Italy, July 2019. Association for Compulational Linguistios. doi:
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- Minneapolis, Minnesola, June 2019. Association for
ume (Long and Short Papers), pp. Computational Linguistics. doi: 10. 10.18653/ 1111///1111111
Reinhard Eckhorn, Roman Bauer, Wolfgang Jordan, Michael Brosch, Wolfgang Kruse, Matthias Munk, and Herbert J. Reitboeck. Coherent oscillations: A mechanism of feature linking in the visual cortex? Biological Cybernelies, 60(2):121- 1988. doi: 10.1007/BF002028991
Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179-211 211, 1990. doi: 10.1207/ s15516709cog1400 1.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learming. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Reseorch, pp. 1243 1252. PMLR, 06-11 Aug 2017.
Lukas Gonon and Juan-Pablo Ortega. Reservoir computing universality with stochastic inputs. IEEE Transactions on Newral Networks and Learning Systems, 31 (1):1000-112, 2020. doi: 10.1 109/TNNT.S.2019 2899649.
Alex Graves. Generating sequences with recurrent neural networks. arXiv:130S.085: 2013.
Charles M. Gray, Peter König, Andreas K. Engel, and Wolf Singer. Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature, 338(3333):331 337, 1989. doi: 10.1038/336334al
Lyudmila Grigoryeva and Juan-Pablo Ortega. Echo state networks are universal. Neural Networks, 108: 495 508, 2018. ISSN 0893-6080. doi: 10.1016//..eunne.2018... 20...00.0..
Joachim Gross, Lars Timmermann, Jan Kujala, Martin Dirks, Frank Schmitz, R. Salmelin, and Alfons Schnitzler. The neural basis of intermittent motor control in humans. Proceedings of the National Academy of Sciences, (4):22229 2302, 2002. doi:
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In Proceedings of the First Conference on Language Modeling, 2024.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. HiPPO: Recurrent memory with optimal polynomial projections. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (ods.), Advances in Neural Information Processing Systems, volume 33, pp. 4474-1487 Curran Associates, Tnc., 2020.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and contimuous-time models with linear state space layers. Tn M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 572-583. Curran Associates, Inc., 2021.
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.); Advances in Neural Information Processing Systems, volume 35, pp. 1999833..... Curran Associates, Inc., 2022a.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modcling long sequences with structured state spaces. In Proceedings of the Tenth International Conference on Learning Representations (ICLR). Open- neti 2022b.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Tin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840 6851. Curran Associates, Inc., 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 988):1735-1780, 1997. ISSN 0899-7667. doi: 10.1162/noco.1997. 199...................
Herbert Jaeger and Harald Haas. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless commmication. Science, 304(5667):77-- 80, 2004. ISSN 0036-8075. doi: 0122///cienc.11990
Li Jing, Yichen Shen, Tena Dubcek, John Peurifoy, Scott Skirlo, Yann LeCun, Max Tegmark, and Marin Soljačió. Tunable efficient unitary neural nelworks (EUNN) and their application to RNNs. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1741. PMLR, 06-11 Aug 2017.
Li Jing, Caglar Gulcchre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacie, and Yoshua Bengio. Gated orthogonal recurrent units: On learning to forget. Neural Compulation, 31(4):765 783, 2019. doi: 10.1162/neet 01174.
Heewoo Jun and Alex Nichol. Shap-E: Generating conditional 3D implicit functions, 2023.
Apostolos Karanikolos and Ioamnis Refanidis. Encoding position improves recurrent neural text summarizers. In Mourad Abbas and Abed Alhakim Freihat (eds.), Proceedings of the 3rd International Conference on Natural Language and Speech Processing, pp. 142-150, Trento, Italy, September 2019. Association for Computational Linguistics.
Jun-Seong Kim, Junghoe Kim, SeungUn Park, Kwangyong Lee, and Yoonju Lee. Modeling with recurrent neural networks for open vocabulary slots. In Emily M. Bender, Leon Derczynski, and Pierre Isabelle (eds.), Proceedings of the 27th International Conference on Computational Långuisties, pp- 1778-2790 Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of 3rd International Conference on Learning Representations (ICLR), San Diego, California, 2015.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.18853/v1/ 101113//1//DD8-0122
Tya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In Proceedings of the 5th International Conference on Learning Representations (ICLR). OpenReview.nee .net, 2017.
Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, 14(11):2531-22) -5660 2002. doi: 00.1122/080070002704079755.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Eve Marder and Dirk Bucher. Central pattern generators and the control of rhythmic movements. Current Biology, 11 (23):R986- 2001. doi:
y00581-4.
Matthew S. Matell and Warren H. Meek. Cortico-striatal circuits and interval timing: coincidence detection of oscillatory processes. Cognitive Brain Research, (2)):1339117) 2004. ISSN 0926-6410. doi: 10.1016/j- Rgebrainres.2004.06.00 Neuroimaging of Interval Timing.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In Proceedings of the 5th International Conference on Learning Representations (ICLR). OpenReview.nee! 2017.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Proceedings of the European Conference Com- puler Vision (ECCV), pp. 405 421, Cham, 2020. Springer International Publishing, ISBN 978-3-030- 58452-8. doi: 10.1007/978-.0 0300000--0000 58452-8 __25282-2..22 24.
Peter M. Milner. A model for visual shape recognition. Psychological Review, 81(6):521 535, 1974. ISSN 0033-295X. doi: 10.1037/h0037143
Daniel Neil, Michael Pfeifler, and Shih-Chii Liu. Phased Ista:: Accelerating recurrent network training for long or event-based sequences. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Aduances in Newral Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep lcarning library. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d' d'Ach Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Javier Perez-Orive, Ofer Mazor, Glenn C. Turner, Stijn Cassenaer, Rachel I. Wilson, and Gilles Laurent. Oscillations and sparsening of odor representations in the mushroom body. Science, 297(058)):359) 365, 2002. doi: 10.1126 /eti2//cceence.0005000
Joshua L. Proctor, Raghavendra P. Kukillaya, and Philip J. Holmes. A phase-reduced meuro-mechanical model for insect locomotion: weed-forward stability and proprioceptive feedback. Philosophical Trans- actions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 368(1930):3087-5100 2010. doi: 10.1008 2000//rtta..00.0334.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- lional Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464 468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-110
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun WOO. Con- volutional LSTM network: A machine learning approach for precipitation nowcasting. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 28. Curran Associales, Inc., 2015.
Hava T. Sicgelmann. Recurrent neural networks and finite automata. Computational Intelligence, 12(4): 567-50 574, 1996. doi: 00.111//.446--6611199....022222
Hava T. Siegelmann. Neural Networks and Analog Computation: Beyond the Turing Limit, Birkhauser Boston Inc., Cambridge, MA, USA, 1999. ISBN 0-8176-3949-7.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Hava T. Siegelmann and Eduardo D. Sontag. On the computational power of neural nets. In Proceedings of the Fifth Annual Workshop Compulational Learning Theory, COLT '92, pp. 440-449, New York, NY, USA, 1992. Association for Computing Machinery. ISBN 089791497X. doi: 10.1145/ 10.1145/130365.10044 1300885.333.33...
Hava T Siegelmann and Eduardo D Sontag. Analog computation via neural networks. Theoretical Computer Science, (31(2):331-30) 1994. ISSN doi:
Hava T. Siegelmann and Eduardo D. Sontag. On the computational power of neural nets. Journal of Computer and System Sciences, 50(1):132 150, 1995. ISSN 0022-0000. doi: (0.1006/jcss.195.. 1995.1013.
Tianbao Song. Jingbo Sun, Yinbing Zhang, and Weiming Peng. An RNN model for gencrating sentences with a desired word at a desired position. Technical Gazelle, 27(1):81 88, February 2020. ISSN 1848-6339.
doi: 10. 10.17559/tv- 20-599/tv22111100 222-200999995322000
Yang Song, Jascha Docl-DDckstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. LSTM neural networks for language modeling. In Proceedings of INTERSPEECH, pp. 194- 2012. doi: 10.21437 417/Interspeecch220 2012-66..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. Tn I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998 6008 Curran Associates, Inc., 2017.
Philippe Vincent-Lamarre, Guillaume Lajoic, and Jean-Philippe Thiviorge. Driving reservoir models with oscillations: a solution to the extreme structural sensitivity of chaotic networks. Journal of Computational Neuroscience, (133:300--322 322, 2016. doi: 10.1007/110227-016-06111
Aaron Voelker, Ivana Kajié, and Chris Eliasmith. Legendre memory units: Contimoous-time representation in recurrent neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Tnc., 2019.
Michael Wehr and Gilles Laurent. Odour encoding by temporal sequences of firing in oscillating neural assemblies. Nature, 384(6605):162-1666 166, 1996. doi: 10. 00.0083344162a0.
Gail Weiss, Yoav Goldberg, and Eran Yahav. On the practical computational power of finite precision rmns for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Popers), pp. 740 745. Association for Computational Linguistics, 2018.
A
Other Tasks
This section demonstrates the effectiveness of positional encoding RNNs across different tasks, besides the reverse ordering task discussed in the main text.
A.1 Reverse-Ordering Delayed-Addition
This section reports the performance of position-encoded RNNs than the reverse ordering of input sequences. Extending the reverso-ordering task, the models received additional random input integers during the output phase, and added each of them to the corresponding token in the reverso-ordered input sequence (modulo the vocabulary size, so that the output range was bounded; Fig. 6).
complicated, combinatorial task
This task was too challenging GRUs- even after reducing the input length L 16-so so only the results from LSTMs reported below. Also, the network was trained for 600,000 iterations (i.e., twice longer
15

---

Published in Transactions on Machine Learning Research (11/2024)
11 + 5 (mod K)
2 18 (mod K)
29 +
8+13 (mod K)
Reversed
(mod K)
Linear _ argmax
RNN
RNN
RNN
RNN
RNN
RNN
RNN
RNN
Positional
L+1
2L
2
2L
27
Encoding
H
B
2
H
H
Embedding
1
Input
29
2
18
13
(Random Integers)
Figure 6: Illustration of the reverse-ordering + delayed-additiin task. The modulus K of the addition is equal to the vocabulary size.
Vanilla LSTM Encooie LSTM
Vocabulary Size
896
960
1024
1088
0.0
0.2
0.4
0.6
0.8
1.0
Accuracy
Figure Token-wise accuracy of the reverse-ordering + delayed-addition task performed by the LSTM with and without positional encoding (labeled as "Position- "Posidion-Encodeed" and "Vanilla" respectively). The input length was fixed bootstrapped samples of five training-test trials with different random seeds. Each of the five trials held out 1024 random sequences (= 16,384 tokens) for computing the test accuracy.
16. The error bars represent the 95% confidence interval estimated from 10,000
than the other tasks) for ensuring the convergence. The other as reported in the main text.
were the same
Consequently, positional encoding improved the model performance as the vocabulary size grew from 896 to 1088 (Fig. 7).
A.2 Sorting
In the reverse ordering task, the order of input integers was important information for accomplishing the task. Thus, positional encoding may play its originally intended role in encoding the temporal information.
16

---

Published in Transactions on Machine Learning Research (11/2024)
GRU LSTM
Vanilla
Position- Dosition-Encoded
256
512
1024
Vocabulary Size
2048
4096
8192
16384
0.00
0.25
0.50
0.75
1.00
10
20
30
40
Accuracy
Damerau- LeveesshheLeennsheiiinn Distance
Figure 8: Token-wise accuracy (left) and sequenco-wisc reconstruction crrors (right) of the sorting task performed by RNNs with and without positional encoding (labeled as "Position-Encodee" and "Vanilla" respectively). The input length was fixed at. = 64. The error bars represent. the 95% confidence interval estimated from 10,000 bootstrapped samples of five training test trials with different random seeds. Each of the five trials held out 1024 random sequences (= 65,536 tokens) for computing the test accuracy.
This section reports the effectiveness of positional encoding for a task in which the order of input observations was completely irrelevant; the learning objective was to simply sort the input integers in their inherent ascending order (e.g. 8, 29, 2, 11 + 2, 8, 11, 29). The input integers were uniformly randomly sampled with replacement, allowing for ties the sorting process.
As result, positional encoding also proved effective for RNNs to handle larger vocabulary in the sorting task (Fig. 8), though the improvement. remained marginal compared to the reverse-ordering task.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Predecessor 29
(1 -= 2)
Linear argmax
RNN
RNN
RNN
RNN
RNN
Positional
1
1+1
Encoding
1
H
B
Embedding
Input
29
11
Query
Mion-Repe Random Integers)
3)
Figure 9: Illustration of the precedent-query task.
A.3 Predecessor Query
Finally, this section presents benchmark results for the predecessor-query task, illustrated in Fig. 9. The network first received a sequence of non-repeating random integers, ,1,..., Subsequently, one of the non-initial input integers, Ztquery (2 < Equery I L), was randomly selected and reintroduced to the network at time 1. The learning objective to return the predecessor of the reviewed integer (= pit The predecessor-query task evaluates the capacity of RNNs to integrate information regarding both the order and content of input sequences.
As in the reversc-ordering + delayed-addition task, the input sequence was reduced to L = 16 due to the complexity of the task, and the experiment focused on the LSTM. The number of training iterations was maintained at 300,000.
Similar to the other benchmarks, positional encoding improved the LSTM's capacity to manage the larger vocabularies.
B Robustness to Variations in Input Length
So far, all the tasks were experimented using fixed-length inputs (T. = 64). One might wonder if positional encoding is ezceptionally effective under this setting, informing RNNs with the exact timing when each input token should be returned as the output. Thus, it remains unclear whether or not position-encoded RNNs can also handle a larger vocabulary even when the input length variable and, thus, the exact timing of the output emission is not identifiable from the positional encoding attached the inputs.
To assess the robustness to variations in the input length, an additional experiment was conducted the LSTM, with the input length varied between 32 and 64. In this setup, the maximum imput length (= 64) covers the entirety of the shortest input sequence plus its reversed reconstruction (= 32 + 32). Consoquently, the positional encoding per se cannot even distinguish the input vs. output phases at f = 33, 64. The vocabulary size was set to 16,384.
As a result, the positional encoding still improved the LSTM's performance on the reverse-ordering task against the perturbations in the input length (Fig. 11). This result suggests that the effectiveness of the positional encoding for RNNs not limited to strictly scheduled tasks.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Vanilla LSTM
Position-Encoded LSTM
512
Vocabulary Size
640
896
1021
0.0
0.2
0.4
0.6
0.8
Accuracy
Figure 10: Accuracy of the precedent-query task performed by the LSTM with and without positional encoding (labeled as "Position-Facodee/ and "Vanilla" respectively). The input length was fixed at T, := 16. The error bars represent the 95% confidence interval estimated from 10,000 bootstrapped samples of five training- test trials with different random seeds. Each of the five trials held out 1024 random sequences (= 65,536 tokens) for computing the test accuracy.
C Effects of Additional Parameters in Position-Encoder RNNs
The concatenation of positional encoding with input embeddings inflates the number of learnable parameters in the input-to-hidden projection weights. This additional parameterization per se does not influence the learning of the input embeddings, and therefore does not elucidate the enhanced performance of position- encoded RNNs. This section substantiates this argument. by equalizing the number of learnable parameters between the vanilla and position- residion-encoded models.
Specifically, the equalization was achieved by concatenating two identical copies of the input embeddings and feeding them to the LSTM. This size of the input-to-hidden weight for each gate in the LSTM, aligning it with that of the position-encoded LSTM, while maintaining all other parameters, including the dinensionality of the (non-repeated) input embeddings.
wwwtrrattionnenncooore termed "double vanilla"- effeectivee doubled the
As illustrated Fig. 12, the double vanilla LSTM did not yield any improvements in the reverso-ordering sorting tasks. These results affirm that the reported enhancement of RNNs is not merely attributable to the additional parameterization associated with the positional encoding.
Alternative Implementations of Positional Encoding
While this study implemented positional encoding by sinusoidal waves (Vaswani et al., 2017), there are alternative implementations proposed in the previous studies. For instance, the BERT-based models typically encode cach token position by a learnable embedding (Devlin et al., 2019). Morcover, the original study of Transformer pointed out that even random vectors can function positional encoding (Vaswani et al., 2017).
Accordingly, these two alternative forms of positional encoding were tested on the LSTM performing the reverse-ordering task. The random position-encoding vectors were uniformly and independently sampled from the (512-1)-dimed) dil-eniadimensionall hypersphere. The learnable embeddings were implemented using the canonical embedding module of PyTorch (torch. nn. Embedding). The input length and vocabulary size were set and 16,384 respectively. As shown Fig. 13, both the random vectors and learnable embeddings improved the performance of L.ST.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Vanilla LSTM
Desition-Encoder LSTM
32
33
35
36
37
38
40
42
43
44
45
Test Input Length
46
47
49
50
51
52
54
56
57
58
59
61
62
63
64
0.2
0.6
1.0
Accuracy
Figure 11: Token-wise accuracy of the reverse-ordering task performed by the LSTM with and without positional encoding (labeled as Prosition-Encodeen and "Vanilla" respectively). During training, the input length was randomly selected from range of 32 to 64. The vocabulary size was to 16,384. The error bars represent the 95% confidence interval estimated from 10,000 bootstrapped samples of five training-test trials with different random seeds. Each of the five trials held out 1024 random sequences per length for computing the test accuracy.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Positional Encoding
Reverse- Reverse-Ordering
Sorting
Vanilla
Double Vanilla
Sinusoidal
0.0
0.5
1.0 0.0
0.2
0.4
0.6
Accuracy
Figure 12: Token-wise accuracy of the reverse-ordering and sorting tasks performed by the LSTM with and without positional encoding. The "double vanilla" model concatenated two copies of the identical imput embeddings to match the number of parameters with the position-encodel model. The input length was fixed at L: = 64. The vocabulary size was set to 16,384. The error bars represent the 95% confidence interval estimated from 10,000 bootstrapped samples of five training-test trials with different random seeds. Each of the five trials held out 1024 random sequences (= 65,536 tokens) for computing the test accuracy.
Positional Encoding
Vanilla
Sinusoidal
Random
Learned
0.4
0.6
0.8
1.0
Accuracy
Figure 13: Token-wise accuracy of the reverse-ordering task performed by the LSTM with and without positional encoding. Three variants of positional encoding (sinusoidal, randomly fixed vectors, and learnable embeddings) were tested. The imput length was fixed at L : 64. The vocabulary size was set to 16,384. The error bars represent the 95% confidence interval estimated from 10,000 bootstrapped samples of five training- test trials with different random seeds. Each of the five trials held out 1024 random sequences (= 65,536 tokens) for computing the test accuracy.
Among the different implementations of positional encoding, the sinusoidal encoding outperformed the two alternatives. The advantage of the sinusoidal encoding became more apparent when the input length was variable between 32 and 64 (Fig. 14); the sinusoidal encoding was more robust to the variations in the input length than the others.
E Language Modeling
This section reports benchmark results for the language modeling task. Single-layer LSTMs with and with- out sinusoidal positional encoding were trained and tested on the WikiText-103 dataset (Merity et al., 2017). Due to constraints in computational resources, the vocabulary was reduced from the original size of 267,735 768 by retokenizing the raw data using SentencePiece (Kudo & Richardson, 2018). The headings were removed, and the main text segmented by paragraphs (separated by the line break). Additionally, only the first 1024 tokens of each paragraph were utilized for training and testing, ensuring that the absolute posi- tional encoding always aligned with the beginning of each paragraph. The hyperparameters were configured as specified in $3.3.
As illustrated Fig. 15, positional encoding proved effective only for marginally faster learning during the initial phase of training. The difference diminished around 10,000/30.000 iterations, and the test perplexities of the position-encoded model inferior those of the vanilla model.
21

---

Published in Transactions on Machine Learning Research (11/2024)
Vanilla
Sinusoidal
Random
Learned
32
49
33
50
34
51
35
52
36
53
37
54
38
55
Test Input Length
39
56
40
57
41
58
42
59
43
60
44
61
45
62
46
47
63
48
64
0.0
0.2
0.4 Accuracy
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 14: Token-wise accuracy of the reverse-ordering task performed by the LSTM with and without positional encoding. Three variants of positional encoding (sinusoidal, randomly fixed vectors, and learnable embeddings) were tested. During training, the input length was randomly selected from range of 64. The vocabulary size set 16,384. The error bars represent the 95% confidence interval estimated from 10,000 bootstrapped samples of five training-tost trials with different random seeds. Each of the five trials held out 1024 random sequences per length for computing the test accuracy.
22

---

Published in Transactions on Machine Learning Research (11/2024)
Vanilla LSTM
10a
LSTM
Perplexity
6 x 10a
x 10"
10a
10
Training Iteration
Figure 15: Training perplexities on the WikiText-103 dataset. Each of the five trials with a different random seed is represented by single line.
Table 1: Test perplexities on the WikiText-103 datasel. The minimum, mean, and maximum are obtained from five trials with different random sceds.
Model
Min
Mean
Max
Vanilla LSTM
36.8257
37.7731
38.916589
Position- Encooed LSTM
38.0685
38.5384
38.893656
23


Reviews:
[{"review_id": "DHnCxb1RI9", "reviewer": "Reviewer_wCde", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The authors present an unexpected and significant discovery about the benefits of positional encoding (PE) for RNNs, which challenges the existing assumptions about RNNs' inherent ability to handle temporal information without external mechanisms like PE. \n2. The design of task and model architecture is simple and clear, which helps to understand the underlying mechanisms in depth. \n\nWeaknesses: \n1. Although the paper mentions other tasks in the appendix, all experiments are conducted on synthetic data. While these toy examples are useful for exploring potential mechanisms, it raises the question of whether the findings can generalize to larger-scale, real-world data.\n2. The related works section could be expanded to include a discussion on state-space models and models like Mamba. It would be valuable to explore the differences and connections between the role of PE in these models versus RNNs.\n3. In the appendix, the authors present results from multiple implementations of positional embeddings, showing that random embeddings outperform learned ones. This aspect needs further discussion. Additionally, it would be beneficial to include a comparison of absolute, relative, and other forms of positional embeddings to enrich the analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the impact of postional embedding in RNN models and points out that positional embedding can stabilize the graident of RNNs to handle larger vocabularies. The authors design the reverse-ordering task and test GRU and LSTM to analyze the gradient stability to understand the mechanism of improvement. In the experiments, positional encoding was found to improve RNN performance when handling rare tokens surrounded by more frequent tokens, especially in long sequences.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As noted in the weaknesses, the authors are encouraged to expand the experiments and provide more in-depth discussions. Specifically, it would be beneficial to explore how the findings generalize to larger-scale, real-world datasets and to compare the role of positional encoding in RNNs with its use in state-of-the-art models like Mamba [1]. A deeper examination of the relationship between positional encoding and the mechanisms in models like Mamba would further strengthen the paper.\n\nOverall, the paper presents a novel and valuable contribution. However, more comprehensive experiments and a more thorough discussion would enhance its impact.\n\n[1] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RKFAfUSsKe", "reviewer": "Reviewer_93n9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This work proposes the idea of using positional encodings in RNNs to improve their capabilities in settings of large vocabulary and low-frequency alphabets. While similar ideas have been studied before, this specific motivation seems new.\n\nWeaknesses:\n\n- The main glaring weakness is the lack of either strong theoretical insights or convincing real-life experimental evidence for the proposed work. The work only studies simple synthetic benchmark tasks such as reversing and sorting and it's not clear how this proposed technique will fare in more real-life applicatios of RNNs such as in NLP.\n\n- The synthetic tasks also essentially involve positionality concepts heavily, such as reversing an array. Therefore, it may possibly be the case that positional embeddings specifically help such narrow tasks whereas be not important in potentially other tasks such as summarization, semantics, classification, etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes improving the performance of RNNs to handle sequential/temporal datasets better by adding positional embedding modules to the inputs embeddings. While RNNs inherently have a temporal structure due to their design, the authors use a simple synthetic benchmark to argue that in the cases of large vocabulary and low frequency, gradient instability is an issue. To mitigate this, they propose to use sinusoidal embeddings that are common in transformer. The improvements in accuracy are shown for some synthetic benchmarks such as reversing an array and also gradient stability is shown to have improvement. Some small-scale modifications are also performed, such as modifying the task or changing the sequence length. The target audience are people interested in improving RNNs or using them for task that rely heavily on temporal structure.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical for recommending acceptance: While the work is potentially interesting to some, I would like to see a more convincing reason (either using theoretical ideas or extensive experiments) that using positional embeddings will largely benefit RNNs across a wide variety of tasks.\n\nNot critical: Typo in section 5: carring -> carrying", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2wfvdbHmXU", "reviewer": "Reviewer_7TzK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I genuinely like the setup of the paper. Please read the strengths in the bullet points below.\n\n1. The overall \"topic\" of the paper is interesting, and deviates from the common ground of using positional encoding only with transformer-based models;\n2. The paper is well written; \n3. The background (i.e., the functionality of positional encoding and the relevant works) is properly introduced; \n4. The experiments are well-designed, and the emerging observations are non-trivial.\n\nMy concerns on this paper mostly relate to the following factors:\n1. While I guess there would not be much difference, probably evaluating with metrics privileging the ordering would lead to a clearer picture. I see that, at the current stage, the paper employs \"token-wise accuracy\".\n2. I wonder whether some results may be a by-product of additional network parametrization in also having positional encoding, since, as per my understanding, the size of the RNNs of this study is relatively small. In particular, this doubt arises when looking at Appendix A.2, where positional encoding also appears to be a big help to the sorting task, where ordering of the input sequence should be completely irrelevant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the impact of positional encoding with Recurrent Neural Networks, to assess whether it would only constitute redundant information (given that the ordering of the input data is explicitly modeled by RNNs) or, conversely, be beneficial. \n\nThe authors do so with synthetic experiments: reverse-ordering, (+ delayed addition), and sorting of integers samples from \"vocabularies\" of various sizes. The experiments encompass both GRUs and LSTMs designs. \n\nUpon examination of the findings, the study highlights that, even though positional encoding could intuitively be seen as irrelevant for RNNs, it does indeed help them (in the scope of the tasks examined). The study further highlights a possible explanation by identifying a strong stabilization of the gradients of non-frequent tokens when positional encoding is included in the learning recipe.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I will follow up on the \"weaknesses\" bullet points above. Specifically:\n1. It'd be nice to pair the results of the paper with metrics accounting for the order when measuring the similarity between two sequences. The RBO [1] is one such example, but I would not necessarily champion this metric. \n2. Could the authors please highlight the difference, in terms of the raw number of learnable parameters, between RNNs with and without positional encoding? If this difference appears large, maybe scaling the total parametrization of \"vanilla\" RNNs to match those with positional encoding would shed some light on the effect that positional encoding has, especially for tasks where the order shall be deemed irrelevant. \n\n(PS: it may as well be that some underlying reason renders the second point unfeasible and I am not spotting it at the moment. I strongly encourage authors to point this out, if that's the case.)\n\n\nReferences:  \n[1] William Webber, Alistair Moffat, and Justin Zobel. A similarity measure for indefinite rankings. ACM\nTransactions on Information Systems.", "broader_impact_concerns": "I do not see unaddressed ethical / impact concerns about this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wPdX6T8pul", "reviewer": "Reviewer_7qMR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) Novel Perspective: The paper challenges the conventional view that positional encoding is exclusive to Transformers. It opens up a new avenue for improving RNN performance, which could be valuable for tasks involving large and sparse vocabularies.\n2)Clear Empirical Results: The experiments on synthetic benchmarks convincingly demonstrate the positive impact of positional encoding on RNNs, especially as the vocabulary size increases. The analysis of gradient stability provides further evidence supporting the authors' claims.\n3) Well-Written: The paper is generally well-written and organized, making the ideas and findings accessible.\n\nWeaknesses:\n\n1) Limited Task Scope: The primary weakness lies in the reliance on a single task (reverse ordering) for the majority of the experiments. While the findings are compelling, their generalizability to other tasks remains an open question. Additional experiments on diverse tasks would significantly strengthen the paper's conclusions.\n2) Lack of Real-World Application: Although the synthetic benchmarks provide a controlled environment for studying the phenomenon, the paper would benefit from demonstrating the practical implications of the findings on real-world tasks, such as natural language processing or time series prediction.\n3) Limited Exploration of Alternative Encodings: The paper briefly touches upon alternative positional encoding schemes but doesn't explore them in depth. A more thorough comparison of different encodings could provide insights into their relative merits and limitations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study investigates the effects of positional encoding on recurrent neural networks (RNNs). The authors report an interesting and somewhat counterintuitive finding: that positional encoding, typically associated with Transformer models, can also benefit RNNs, particularly when dealing with large vocabularies. The core argument is that low-frequency tokens in large vocabularies lead to gradient instability during RNN training, and positional encoding helps mitigate this issue.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The only thing I would really like to see is more tasks and experiments than just relying on one single toyish task. Otherwise the paper brings in nice ideas.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main concern of reviewers on acceptance of the paper was the experimental assessment performed only on synthetic data. The author, however, was able to improve this aspect adding more empirical evidence  in the revised version, also providing further discussion on the characterisation of the proposed approach. Overall, 3 out of 4 reviewers judged the current version of the paper sound and interesting for the TMLR audience.
The content of the paper does not fit any of the certifications.
The current version of the paper needs to fix some formatting issues, such as text overflow into right-hand margin.
Decision: Accept (poster)