SlowGuess/NAACL_2022_OCR huggingface dataset:
where  $(\pmb{q}_i',\pmb{k}_j')$  and  $(\pmb {q}_i,\pmb {k}_j)$  are two sets of querykey vectors computed in the previous and this text segment respectively for the same position pair  $(i,j)$ . Then we have:
$$
\begin{array}{l} \boldsymbol {c} _ {i} ^ {\leftrightarrow} = \frac {\sum_ {j \leq i} \operatorname {s i m} \left(\boldsymbol {q} _ {i} ^ {\prime} , \boldsymbol {k} _ {j} ^ {\prime}\right)}{\sum_ {j \leq i} \operatorname {s i m} \left(\boldsymbol {q} _ {i} ^ {\prime} , \boldsymbol {k} _ {j} ^ {\prime}\right) + \sum_ {j > i} \operatorname {s i m} \left(\boldsymbol {q} _ {i} , \boldsymbol {k} _ {j}\right)} \boldsymbol {c} _ {i} ^ {\rightarrow} \\ + \frac {\sum_ {j > i} \operatorname {s i m} \left(\boldsymbol {q} _ {i} , \boldsymbol {k} _ {j}\right)}{\sum_ {j \leq i} \operatorname {s i m} \left(\boldsymbol {q} _ {i} ^ {\prime} , \boldsymbol {k} _ {j} ^ {\prime}\right) + \sum_ {j > i} \operatorname {s i m} \left(\boldsymbol {q} _ {i} , \boldsymbol {k} _ {j}\right)} \boldsymbol {c} _ {i} ^ {\leftarrow} \\ = \frac {\sum_ {j \leq i} \operatorname {s i m} \left(\boldsymbol {q} _ {i} ^ {\prime} , \boldsymbol {k} _ {j} ^ {\prime}\right) \boldsymbol {v} _ {j} ^ {\prime} + \sum_ {j > i} \operatorname {s i m} \left(\boldsymbol {q} _ {i} , \boldsymbol {k} _ {j}\right) \boldsymbol {v} _ {j}}{\sum_ {j \leq i} \operatorname {s i m} \left(\boldsymbol {q} _ {i} ^ {\prime} , \boldsymbol {k} _ {j} ^ {\prime}\right) + \sum_ {j > i} \operatorname {s i m} \left(\boldsymbol {q} _ {i} , \boldsymbol {k} _ {j}\right)} \\ = \sum_ {j} \beta_ {j} \tilde {\boldsymbol {v}} _ {j}, \\ \end{array}
$$
where  $\sum_{j}\beta_{j} = 1$ . Finally, we derive  $c_{i}^{\leftrightarrow}$  as the weighted sum of the value vectors  $\tilde{v}_{j}$  from both the past  $(j\leq i)$  and the future  $(j > i)$  of the position  $i$ .
B Unstability Analysis of the RPE in Transformer-XL
We conjecture that the unstability of Eq. 6 stems from the terms involving the dot-product of  $\pmb{R}_{i-j}$  and another vector. So we start by considering the variance of  $\pmb{x}^{\top} \pmb{R}_{i-j}$  where  $\pmb{x} \in \mathbb{R}^{d}$  is a random vector. Without loss of generality, we assume that  $\pmb{x}$  has zero mean and a variance of  $\pmb{\sigma}$ :
$$
\begin{array}{l} \operatorname {E} \left(x _ {k}\right) = 0, \forall k \in [ 1, \dots , d ] \\ \operatorname {V a r} \left(x _ {k}\right) = \sigma_ {k, k}, \forall k \in [ 1, \dots , d ] \\ \operatorname {C o v} \left(x _ {k}, x _ {l}\right) = \sigma_ {k, l}, \forall l \neq k \in [ 1, \dots , d ] \\ \end{array}
$$
Let  $i - j = \Delta$ . According to Vaswani et al. (2017),  $R_{\Delta}$  takes the following form:
$$
\begin{array}{l} \boldsymbol {R} _ {\Delta} = \left[ \sin \left(\omega_ {1} \Delta\right), \cos \left(\omega_ {1} \Delta\right), \right. \\ \cdot \cdot \cdot , \sin (\omega_ {d / 2} \Delta), \cos (\omega_ {d / 2} \Delta) ], \\ \end{array}
$$
where  $w_{k} = 10000^{-2k / d}$ . Then the dot-product  $\pmb{x}^{\top}\pmb{R}_{\Delta}$  can be derived into the linear combination of sine and cosine functions:
$$
\boldsymbol {x} ^ {\top} \boldsymbol {R} _ {\Delta} = \sum_ {k = 1} ^ {d / 2} x _ {2 k - 1} \sin (\omega_ {k} \Delta) + x _ {2 k} \cos (\omega_ {k} \Delta),
$$
where we can easily derive that  $\operatorname{E}(\boldsymbol{x}^{\top}\boldsymbol{R}_{\Delta}) = 0$ . According to the variance expectation formula:  $\operatorname{Var}(x) = \operatorname{E}[x^2] - \operatorname{E}[x]^2$ , we can simplify the variance  $\operatorname{Var}(\boldsymbol{x}^{\top}\boldsymbol{R}_{\Delta})$  in the following:
$$
\begin{array}{l} \operatorname {V a r} \left(\boldsymbol {x} ^ {\top} \boldsymbol {R} _ {\Delta}\right) \\ = \operatorname {E} \left[ \left(\sum_ {k = 1} ^ {d / 2} x _ {2 k - 1} \sin \left(\omega_ {k} \Delta\right) + x _ {2 k} \cos \left(\omega_ {k} \Delta\right)\right) ^ {2} \right] \\ = \sum_ {k = 1} ^ {d / 2} \mathrm {E} \left[ x _ {2 k - 1} ^ {2} \right] \sin^ {2} \left(\omega_ {k} \Delta\right) + \mathrm {E} \left[ x _ {2 k} ^ {2} \right] \cos^ {2} \left(\omega_ {k} \Delta\right) \\ + 2 \sum_ {k = 1} ^ {d / 2} \sum_ {l = 1, l \neq k} ^ {d / 2} \operatorname {E} [ x _ {2 k - 1} x _ {2 l} ] \sin (\omega_ {k} \Delta) \cos (\omega_ {l} \Delta). \\ \end{array}
$$
We further simplify the above equation by assuming that all the elements have the same variance  $\sigma_{s}$ , and all pairs of distinct elements have the same covariance  $\sigma_{c}$ :
$$
\begin{array}{l} \operatorname {V a r} \left(\boldsymbol {x} ^ {\top} \boldsymbol {R} _ {\Delta}\right) = \sum_ {k = 1} ^ {d / 2} \sigma_ {s} \left[ \sin^ {2} \left(\omega_ {k} \Delta\right) + \cos^ {2} \left(\omega_ {k} \Delta\right) \right] \\ + 2 \sum_ {k = 1} ^ {d / 2} \sum_ {l = 1, l \neq k} ^ {d / 2} \sigma_ {c} \sin (\omega_ {k} \Delta) \cos (\omega_ {l} \Delta) \\ = \frac {d}{2} \sigma_ {s} + 2 \sigma_ {c} g (\Delta), \\ \end{array}
$$
where  $g(x) = \sum_{k=1}^{d/2} \sum_{l=1, l \neq k}^{d/2} \sin(\omega_k x) \cos(\omega_l x)$  is an odd function.
We consider the value of  $g(x)$  when  $x \approx 0$ .
Figure 7: The plot of  $g(x)$  when  $d = 64$ . We see that  $g(x)$  is symmetric with respect to the origin. The value of  $g(x)$  when  $x$  approaches zero from the left and right diverge greatly.
Since  $\sin (\omega_kx)\approx \omega_kx$ $\cos (\omega_kx)\approx 1$  , we have:
$$
\begin{array}{l} g (x) \approx \sum_ {k = 1} ^ {d / 2} \sum_ {l = 1} ^ {d / 2} \omega_ {k} x \\ = \frac {d}{2} \sum_ {k = 1} ^ {d / 2} w _ {k} x \\ = \frac {x d}{2} \sum_ {k = 1} ^ {d / 2} \left(\frac {1}{1 0 0 0 0 ^ {2 / d}}\right) ^ {k} \\ \approx \frac {d}{2 ((1 0 ^ {8}) ^ {1 / d} - 1)} \cdot x = \gamma_ {d} \cdot x. \\ \end{array}
$$
Since  $a^x \approx 1 + x \ln a$  when  $x \approx 0$ , we derive that  $\gamma_d \approx \frac{d^2}{2 \ln 10^8}$  with the grow of  $d$ . This causes  $g(x)$  to have a very steep slope near 0. Since  $g(x)$  is an odd function, the value of  $g(\Delta)$  and  $g(-\Delta)$  will have a huge gap ( $\Delta$  is a small positive value). To validate this, we plot the function of  $g(x)$  when  $d = 64$  in Figure 7.
Overall, the variance of  $\boldsymbol{x}^{\top}\boldsymbol{R}_{\Delta}$  is composed of two terms, the first being  $\sigma_{s}$  multiplied by a constant factor  $d / 2$ , and the second being  $\sigma_{c}$  multiplied by  $g(\Delta)$ . Note that  $\sigma_{s}$  is strictly positive, while  $\sigma_{c}$  does not have this restriction. Due the asymptotic behavior of  $g(\Delta)$  near 0, i.e.,  $\mathcal{O}(d^{2}\Delta)$ , we cannot find a proper  $\sigma_{c}$  that makes  $\operatorname{Var}(\boldsymbol{x}^{\top}\boldsymbol{R}_{\Delta})$  bounded by  $\mathcal{O}(d\sigma_s)$  for every  $\Delta$  that takes its value from both the positive and negative integers.
Finally, we plot the training curves of the two models using the RPE in Transformer-XL (xl-rpe) and our disentangled RPE (dis-rpe) in Figure 8 where we observed that the xl-rpe suffers from numerical unstability during training.
Figure 8: Comparison of the training dynamics using different encoding schemes: the disentangled RPE (dis-rpe) and the RPE of Transformer-XL (xl-rpe).
Dataset	train / dev / test
Wikitext-103	103,227,021 / 217,646 / 245,569
enwik8	88,982,818 / 4,945,742 / 4,943,417
text8	89,999,999 / 4,999,999 / 5,000,000
Table 4: Statistics of the datasets used in the experiments. For Wikitext-103, we use the official split from Merity et al. (2017) and present the number of tokens in each split. For enwik8 and text8, we use the split from Dai et al. (2019) and report the number of characters for each split.
Wikitext-103 dataset is extracted from the set of verified Good and Featured articles on English Wikipedia. The dataset retains the original case, punctuation and numbers, and covers a broad range of domains, e.g., science, culture, bibliography, and etc. The dataset is available under the Creative Commons Attribution-ShareAlike (CC BY-SA) License.
enwik8 dataset is the test set data of the Large Text Compression Benchmark which contains the first 100 million bytes of English Wikipedia dump on Mar. 3, 2006. All characters are encoded in UTF-8. This dataset is licensed under the CC BY-SA License.
text8 dataset contains the first 100 million bytes of the clean text of Wikipedia that retains only regular articles and image captions. All the letters are converted into lower case, and only letters in the 27 character alphabet, namely letters a-z and nonconsecutive spaces, are preserved. This dataset is licensed under the CC BY-SA License.
The statistics of the three datasets is shown in Table 4.
C.2 Model Configurations
We follow the base model configuration of Dai et al. (2019). On Wikitext-103, we use the Transformer model with 16 layers, 10 attention heads with a head dimension of 41. The inner dimension size of the feedforward layer is 2100. We use a dropout rate of 0.1 and no attention dropout. To cope with the large vocabulary, we use the adaptive embeddings (Baevski and Auli, 2019). We set the memory length to 150 and the target sequence length to 150 as well. On text8 and enwik8 datasets, we use the Transformer model with 12 layers, 8 attention heads with a head dimension of 64. The inner dimension size of the feedforward layer is 2048. We use a dropout rate of 0.1 and no attention dropout. We set the memory length to 512 and the target length to 512. Specifically, our LaMemo uses the disentangled relative positional encodings described in Sec. 3.3. The look-ahead attention shares the query, key and value projection matrices with those in the causal attention.
We trained the models using Adam (Kingma and Ba, 2015) optimizer, with no warmup. We used a learning rate of  $2.5 \times 10^{-4}$  which decayed to 0 at the end of training with a cosine schedule. On Wikitext-103, we trained the model with 250K steps using a batch size of 64. On enwik8 and text8, we trained the model with  $100\mathrm{K}^7$  steps using a batch size of 40. We conducted our experiments on 2 Tesla V100.
We present the hyperparameter search space in Table 5. The number of hyperparameter search trials was 10. We adopted a manual search to select the hyperparameters, and the selection criterion was ppl/bpc on the dev set. We did not use early stopping during training.
D Generated Examples
In this section, we present the examples generated by LaMemo and Transformer-XL trained on the Wikitext-103 dataset. Both models maintain a memory with a length of 512. We randomly select a piece of text from the test set as the context
Hyper-parameter	Search Space
Learning Rate	choice[1e-4, 2.5e-4, 5e-4]
Learning Rate Schedule	choise[linear, cosine]
Warmup Steps	choice[0, 1000, 2000]
Maximum Gradient Norm	choice[0.25, 0.5, 1.0]
Epsilon (Sec. 3.2)	choice[1e-6, 1e-5, 1e-4]
Optimizer	Adam
Epsilon (for Adam)	1e-8
Momentum (for Adam)	β1=0.9, β2=0.999
Table 5: Hyperparameter search space. choice indicates that the listed numbers will be chosen with the same probability. Best-found hyperparameters are in boldface.
and allow both models to generate 256 tokens following the context. We use top-p sampling with  $p = 0.95$  and detokenize the context and the generated texts to facilitate reading. We present the examples in Table 6 and 7. We present our major findings below:
- Both models are able to hallucinate imaginary contents fairly relevant to the limited contexts given as prompts.
- Transformer-XL sometimes generates topic-irrelevant contents without further elaboration (marked by underline), while LaMemo stays on topic more closely during the course of generation.
- Transformer-XL suffers more sever repetition issues (marked in **boldface**) than LaMemo both lexically and semantically.
<sup>7</sup>We used a smaller number of training steps compared to Dai et al. (2019), since it would take too long to train one model.
$= \mathrm{Shackleton}(\mathrm{crater}) =$
Shackleton is an impact crater that lies at the south pole of the Moon. The peaks along the crater's rim are exposed to almost continual sunlight, while the interior is perpetually in shadow (a Crater of eternal darkness). The low-temperature interior of this crater functions as a cold trap that may capture and freeze volatiles shed during comet impacts on the Moon. Measurements by the Lunar Prospector spacecraft showed higher than normal amounts of hydrogen within the crater, which may indicate the presence of water ice. The crater is named after Antarctic explorer Ernest Shackleton.
$= =$  Description  $= =$
The rotational axis of the Moon lies within Shackleton, only a few kilometers from its center. The crater is  $21\mathrm{km}$  in diameter and  $4.2\mathrm{km}$  deep. From the Earth, it is viewed edge-on in a region of rough, cratered terrain. It is located within the South Pole-Aitken basin on a massif. The rim is slightly raised about the surrounding surface and it has an outer rampart that has been only lightly impacted. No significant craters intersect the rim, and it is sloped about  $1.5^{\circ}$  toward the direction  $50 - 90^{\circ}$  from the Earth. The age of the crater is about 3.6 billion years and it has been in the proximity of the south lunar pole for at least the last two billion years.
Because the orbit of the Moon is tilted only  $5^{\circ}$  from the ecliptic, the interior of this crater lies in perpetual darkness. Estimates of the area in permanent shadow were obtained from Earth-based radar studies. Peaks along the rim of the crater are almost continually illuminated by sunlight, spending about  $80 - 90\%$  of each lunar orbit exposed to the Sun. Continuously illuminated mountains have been termed peaks of eternal light and have been predicted to exist since the 1900s.
The shadowed portion of the crater was imaged with the Terrain Camera of the Japanese SELENE spacecraft using the illumination of sunlight reflected off the rim. The interior of the crater consists of a symmetrical  $30^{\circ}$  slope that leads down to a  $6.6\mathrm{km}$  diameter floor. The handful of craters along the interior span no more than a few
LaMemo Generation:
kilometres of plains.
A slight rotation is observed in the crater that now bears significant elements like snow, ice, and water ice. In other areas of the crater there are indications that the nature of the alpine ice layer provides some protection to the craters by increasing their brightness. The craters seem to be cut by an ice thickness of  $10\mathrm{cm}$  (4 in). Images show their prominence, varying from a visible plateau to a multi-ring topography, indicating that they are present (a major component) in the craters. Their interior has a considerable surface area of dry ice.
As seen from Earth, the initial craters are marked by slow-moving metal-bearing mudstone-covered rock that has a surface of substantial amounts of coal, calcium, lead and sodium alloy, carbon monoxide, nitrogen oxides, and silicon dioxide. All are developed by processes such as heat transfer by the advancing radiation pressure. The least dense feature along the surface of the Moon is the large resurfacing event. There is no view that these rocks remain melting to about  $90^{\circ}\mathrm{C}$  (150°F), but that their surface shine is dark and their outer layers are light, reflecting vertical oil. This late flank and early 1970s discovery was announced in 1982, as a substantial amount of ice.
A parallel lava
Transformer-XL Generation:
During the hundreds of years after its formation, the crater now has significant seismic activity in the vicinity. Most craters are dome-shaped and are usually constructed of sedimentary materials that have been deposited during alpine ice times. The upper surface of the crater is covered with higher layers of rock that are cut by an extensive thickness of debris. The debris reaches the top of a mountain and comes in contact with visible ground planes. People are often observed wearing headgear of degrading materials such as clothing and boots, their shoes or hats, or even working on the surface. Below the crater. As the crater faces the crater it has thick, thin pipes or scarps. A total of more than 200 caves have been excavated, down to some  $40\mathrm{m}$  by  $20\mathrm{m}$ . This exceeding the margin of the crater where it actually passes through is considered to be very high. Other geologic features by the advancing magnetic field have been reported from the crater. However, in 1992, scientists announced they would study this area again.
The crater was once a common feature of the Post Lunar System. Its medieval boundaries were not fixed in the orbital plane of Mercury. An individual crater had been called "Discovery crater" and one referred to as "Bear crater", although it is likely that an additional crater was called
Table 6: Example 1 generated by LaMemo and Transformer-XL given a context prompt from the test set of Wikitext-103. Original Wikipedia page: https://en.wikipedia.org/wiki/Shackleton_(crater).
Nero was not expected to become Emperor because his maternal uncle, Caligula, had begun his reign at the age of 24 with enough time to produce his own heir. Nero's mother, Agrippina, lost favour with Caligula and was exiled in 39 after her husband's death. Caligula seized Nero's inheritance and sent him to be brought up by his less wealthy aunt, Domitia <unk>, who was the mother of Valeria <unk>, Claudius's third wife. Caligula, his wife <unk> and their infant daughter Julia Drusilla were murdered on 24 January 41. These events led Claudius, Caligula's uncle, to become emperor. Claudius allowed Agrippina to return from exile.
Claudius had married twice before marrying Valeria <unk>. His previous marriages produced three children including a son, Drusus, who died at a young age. He had two children with <unk> – Claudia Octavia (born 40) and Britannicus (born 41). <unk> was executed by Claudius in the year 48.
In 49 AD, Claudius married a fourth time, to Nero's mother Agrippina, despite her being his niece. To aid Claudius politically, young Nero was adopted in 50 and took the name Nero Claudius Caesar Drusus Germanicus (see adoption in Rome). Nero was older than his stepbrother Britannicus, and thus became heir to the throne. Nero was proclaimed an adult in 51 at the age of 14. He was appointed proconsul, entered and first addressed the Senate, made joint public appearances with Claudius, and was featured in coinage. In 53, he married his stepsister Claudia Octavia.
$= =$  Emperor (54-68 AD)  $= =$
$= = =$  Early rule  $= = =$
Claudius died in 54 and Nero, taking the name Nero Claudius Caesar Augustus Germanicus, was established as Emperor. Though accounts vary, many ancient historians state Agrippina poisoned Claudius. According to Pliny the Elder, she used poison mushrooms. It is not known how much Nero knew or if he was even involved in the death of Claudius.
Suetonius wrote "... for even if he was not the instigator of the emperor's death, he was at least privy to it, as he openly admitted; for he used afterwards to laud mushrooms, the vehicle in which the poison was administered to Claudius, as 'the food of the gods,' as the Greek proverb has it. At any rate, after Claudius' death he vented on him every kind of insult, in act and word, charging him now with folly and now with cruelty; for it was a favourite joke of his to say that Claudius had ceased 'to play the fool' among mortals, lengthening the first syllable of the word  $<\text{unk}>$ , and he disregarded many of his decrees and acts as the work of a madman and a  $<\text{unk}>$ . Finally, he neglected to enclose the place where his body was burned except with a low and mean wall."
According to Suetonius; Nero became Emperor at the age of 17 when the news of Claudius' death was made known, making him the youngest emperor at that time. Although, what Suetonius may have meant is that he was
killed as a result of the subsequent anti-Caligula propaganda.
At some time after 47 AD, Nero retired to his villa at Baiae. The combination of a curse that led to the death of Agrippa, and a duel between Claudius and his own son Drusus, may have encouraged the young Emperor to fire his own stela.
$= = = = =$  Germanicus  $= = = = =$
After becoming emperor, Nero received letters from Suetonius, which finally followed him on 28 May 44 AD with a letter to Claudius, completing the Emperor's plot. The letters were printed in Latin. Claudius' plot was offered to Caligula by Agrippinae the Elder, a woman from Kent, but she declined. Further letters from Gaius and Nero are known from the reigns of Tiberius and Caligul, though none of them mention Caligula.
Germanicus' conspiracy had developed into a protracted affair. His wife Antonia, his sister Agrippina's sister and grandmother, had refused to accept him because she believed they were connected for communication, but the Via Flavia was left to her husband and Agrippina the Elder, whom she had defeated in her husband's palace. Both had been sent through Gibraltar to compete for power. Gaius and Nero were outraged that Agrippina was supposed to be ready to support Caligula; Suetonius says that Nero insisted on making
killed as a result of poison and was deemed a good comedian. At now and then he had adopted Nero as his heir.
Although Nero's rule was as close as he was to his father to his daughter Agrippina, he was not to physically exhibit any ill degree of higher kingship but to deny it. The Augustan History describes him as having been strongly willed, possessing an excellent hand and often claiming the title "Augustus Caesar", and often referred to him as Caesar's "paternal heir". The Augustan History, however, asserts that he was no longer in China, and therefore he was raised as a Roman Hercules rather than a Roman citizen. Claudius Claudius was added as a junior emperor in 53 AD; he was crowned emperor in 61 AD.
$= = = =$  As Augustus  $\mathrm{I} = = = =$
Claudius had developed a keen interest in Claudius in divination, a practice which later developed into his incarnation as a youth figure in Julius Caesar. Claudius' grandfather, Leo I, ascended the throne in 23 AD and spent time in Rome, as did Claudius, who defeated Claudius in 42 AD. Claudius departed Rome after the death of Agrippa III in 65 AD. During the following years, Claudius was temporarily imprisoned in Rome, although possibly simply regulating the use of the captive dogs
Table 7: Example 2 generated by LaMemo and Transformer-XL given a context prompt from the test set of Wikitext-103. Original Wikipedia page: https://en.wikipedia.org/wiki/Nero.