SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Steven Morad
sm2558@cam.ac.uk
Stephan Liwicki
Toshiba Europe Ltd.
stephan.liwicki@toshiba.eu
Ryan Kortvelesy
rk627@cam.ac.uk
Amanda Prorok
asp45@cam.ac.uk
Nearly all real world tasks are inherently partially observable, necessitating the use of memory in Reinforcement Learning (RL). Most model-free approaches summarize the trajectory into a latent Markov state using memory models borrowed from Supervised Learning (SL), even though RL tends to exhibit different training and efficiency characteristics. Addressing this discrepancy, we introduce Fast and Forgetful Memory, an algorithm-agnostic memory model designed specifically for RL. Our approach constrains the model search space via strong structural priors inspired by computational psychology. It is a drop-in replacement for recurrent neural networks (RNNs) in recurrent RL algorithms, achieving greater reward than RNNs across various recurrent benchmarks and algorithms without changing any hyperparameters. Moreover, Fast and Forgetful Memory exhibits training speeds two orders of magnitude faster than RNNs, attributed to its logarithmic time and linear space complexity. Our implementation is available at https://github.com/proroklab/ffm.
Reinforcement Learning (RL) was originally designed to solve Markov Decision Processes (MDPs) [Sutton and Barto, 2018], but many real world tasks violate the Markov property, confining superhuman RL agents to simulators. When MDPs produce noisy or incomplete observations, they become Partially Observable MDPs (POMDPs). Using memory, we can summarize the trajectory into a Markov state estimate, extending convergence guarantees from traditional RL approaches to POMDPs [Kaelbling et al., 1998].
In model-free RL, there are two main approaches to modeling memory: (1) RL-specific architectures that explicitly model a probabilistic belief over Markov states [Kaelbling et al., 1998] and (2) general-purpose models such as RNNs or transformers that distill the trajectory into a fixed-size latent Markov state. Ni et al. [2022] and Yang and Nguyen [2021] reveal that with suitable hyperparameters, general-purpose memory often outperforms more specialized belief-based memory.
Most applications of memory to RL tend to follow Hausknecht and Stone [2015], using RNNs like Long Short-Term Memory (LSTM) or the Gated Recurrent Unit (GRU) to summarize the trajectory, with other works evaluating transformers, and finding them tricky and data-hungry to train [Parisotto et al., 2020]. Morad et al. [2023] evaluate a large collection of recent memory models across many
partially observable tasks, finding that the GRU outperformed all other models, including newer models like linear transformers. Interestingly, they show that there is little correlation between how well memory models perform in SL and RL. Even recent, high-profile RL work like Hafner et al. [2023], Kaptuowski et al. [2023] use RNNs over more modern alternatives, raising the question question: why do older memory models continue to overshadow their contemporary counterparts in modern RL?
Training model-free RL policies today is sample inefficient, alchemical, and prone to collapse [Schulman et al., 2015, Zhang et al., 2018], and reasoning over the entire trajectory magnifies these issues. SL can better utilize scale, compute, and dataset advancements than RL, largely removing the need for strong inductive biases. For example, transformers execute pairwise comparisons across all inputs, in a somewhat "brute force" approach to sequence learning. Similarly, State Space Models [Gu et al., 2021] or the Legendre Memory Unit [Voelker et al., 2019] are designed to retain information over tens or hundreds of thousands of timesteps, growing the model search space with each additional observation.
Through strong inductive biases, older recurrent memory models provide sample efficiency and stability in exchange for flexibility. For example, the GRU integrates inputs into the recurrent states in sequential order, has explicit forgetting mechanics, and keeps recurrent states bounded using saturating activation functions. These strong inductive biases curtail the model search space, improve training stability and sample efficiency, and provide a more "user-friendly" training experience. If inductive biases are indeed responsible for improved performance, can we better leverage them to improve memory in RL?
Contributions We introduce a memory model that summarizes a trajectory into a latent Markov state for a downstream policy. To enhance training stability and efficiency, we employ strong inductive priors inspired by computational psychology. Our model can replace RNNs in recurrent RL algorithms with a single line of code while training nearly two orders of magnitude faster than RNNs. Our experiments demonstrate that our model attains greater reward in both on-policy and off-policy settings and across various POMDP task suites, with similar sample efficiency to RNNs.
RL and SL models differ in their computational requirements. While in SL the model training duration is primarily influenced by the forward and backward passes, RL produces training data through numerous inference workers interacting with the environment step by step. Consequently, it's imperative for RL memory to be both fast and efficient during both training and inference. However, for memory models, there is often an efficiency trade-off between between the training and inference stages, as well as a trade-off between time and space complexity.
Recurrent Models Recurrent models like LSTM [Hochreiter and Schmidhuber, 1997], GRUs [Chung et al., 2014], Legendre Memory Units (LMUs) [Voelker et al., 2019], and Independent RNNs [Li et al., 2018] are slow to train but fast at inference. Each recurrent state  $S$  over a sequence must be computed sequentially
$$
\boldsymbol {y} _ {j}, \boldsymbol {S} _ {j} = f (\boldsymbol {x} _ {j}, \boldsymbol {S} _ {j - 1}), \quad j \in [ 1, \dots , n ] \tag {1}
$$
where  $x_{j}, y_{j}$  are the inputs and outputs respectively at time  $j$ , and  $f$  updates the state  $S$  incrementally. The best-case computational complexity of recurrent models scales linearly with the length of the sequence and such models cannot be parallelized over the time dimension, making them prohibitively slow to train over long sequences. On the other hand, such models are quick at inference, exhibiting constant-time complexity per inference timestep and constant memory usage over possibly infinite sequences.
Parallel Models Parallel or "batched" models like temporal CNNs [Bai et al., 2018] or transformers [Vaswani et al., 2017] do not rely on a recurrent state and can process an entire sequence in parallel i.e.,
$$
\boldsymbol {y} _ {j} = f \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {j}\right), \quad j \in [ 1, \dots , n ] \tag {2}
$$
Given the nature of GPUs, these models exhibit faster training than recurrent models. Unfortunately, such models require storing all or a portion of the trajectory, preventing their use on long or infinite
sequences encountered during inference. Furthermore, certain parallel models like transformers require quadratic space and  $n$  comparisons at each timestep during inference. This limits both the number of inference workers and their speed, resulting in inefficiencies, especially for on-policy algorithms.
Hybrid Models Linear transformers [Katharopoulos et al., 2020, Schlag et al., 2021, Su et al., 2021] or state space models [Gu et al., 2021, 2020] provide the best of both worlds by providing equivalent recurrent and closed-form (parallel) formulations
$$
\boldsymbol {S} _ {j} = f \left(\boldsymbol {x} _ {j}, \boldsymbol {S} _ {j - 1}\right) = g \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {j}\right), \quad j \in [ 1, \dots , n ] \tag {3}
$$
Training employs the batched formula to leverage GPU parallelism, while inference exploits recurrent models' low latency and small memory footprint. Thus, hybrid models are well-suited for RL because they provide both fast training and fast inference, which is critical given the poor sample efficiency of RL. However, recent findings show that common hybrid models typically underperform in comparison to RNNs on POMDPs [Morad et al., 2023].
We are given a sequence of actions and observations  $(\pmb{o}_1, \pmb{0}), (\pmb{o}_2, \pmb{a}_1), \dots (\pmb{o}_n, \pmb{a}_{n-1})$  up to time  $n$ . Let the trajectory  $\pmb{X}$  be some corresponding encoding  $\pmb{x} = \varepsilon(\pmb{o}, \pmb{a})$  of each action-observation pair
$$
\boldsymbol {X} _ {n} = \left[ \boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n} \right] = \left[ \varepsilon \left(\boldsymbol {0} _ {1}, \boldsymbol {0}\right), \dots , \varepsilon \left(\boldsymbol {0} _ {n}, \boldsymbol {a} _ {n - 1}\right) \right]. \tag {4}
$$
Our goal is to summarize  $X_{n}$  into a latent Markov state  $y_{n}$  using some function  $f$ . For the sake of both space and time efficiency, we restrict our model to the space of hybrid memory models. Then, our task is to find an  $f$ , such that
$$
\boldsymbol {y} _ {n}, \boldsymbol {S} _ {n} = f (\boldsymbol {X} _ {k: n}, \boldsymbol {S} _ {k - 1}), \tag {5}
$$
where  $\mathbf{X}_{k:n}$  is shorthand for  $\mathbf{x}_k, \mathbf{x}_{k+1}, \ldots, \mathbf{x}_n$ . Note that by setting  $k = n$ , we achieve a one-step recurrent formulation that one would see in an RNN, i.e.,  $\mathbf{y}_n, \mathbf{S}_n = f(\mathbf{X}_{n:n}, \mathbf{S}_{n-1}) = f(\mathbf{x}_n, \mathbf{S}_{n-1})$ .
4 Background
In the search for inductive priors to constrain our memory model search space, we turn to the field of computational psychology. In computational psychology, we use the term trace to refer to the physical embodiment of a memory – that is, the discernable alteration in the brain's structure before and after a memory's formation. Computational psychologists model long-term memory as a collection of traces, often in matrix form
$$
\boldsymbol {S} _ {n} = \left[ \boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n} \right], \tag {6}
$$
where  $\mathbf{x}_j$  are individual traces represented as column vectors and  $S_{n}$  is the memory at timestep  $n$  [Kahana, 2020].
Composite Memory Composite memory [Galton, 1883] approximates memory as a lossy blending of individual traces  $x$  via summation, providing an explanation for how a lifetime of experiences can fit within a fixed-volume brain. Murdock [1982] expresses this blending via the recurrent formula
$$
\boldsymbol {S} _ {n} = \gamma \boldsymbol {S} _ {n - 1} + \boldsymbol {B} _ {n} \boldsymbol {x} _ {n}, \tag {7}
$$
where  $\gamma \in (0,1)$  is the forgetting parameter and  $B_{n}$  is a diagonal matrix sampled from a Bernoulli distribution, determining a subset of  $\pmb{x}_n$  to add to memory. In Murdock's model,  $\pmb{S}$  is a vector, not a matrix. We can expand or "unroll" Murdock's recurrence relation, rewriting it in a closed form
$$
\boldsymbol {S} _ {n} = \gamma^ {n} \boldsymbol {S} _ {0} + \gamma^ {n - 1} \boldsymbol {B} _ {1} \boldsymbol {x} _ {1} + \dots \gamma^ {0} \boldsymbol {B} _ {n} \boldsymbol {x} _ {n}, \quad 0 <   \gamma <   1, \tag {8}
$$
making it clear that memory decays exponentially with time. Armed with equivalent recurrent and parallel formulations of composite memory, we can begin developing a hybrid memory model.
Contextual Drift It is vital to note that incoming traces  $\pmb{x}$  capture raw sensory data and lack context. Context, be it spatial, situational, or temporal, differentiates between seemingly identical raw sensory inputs, and is critical to memory and decision making. The prevailing theory among computational psychologists is that contextual and sensory information are mixed via an outer product [Howard and Kahana, 2002]
$$
\hat {\boldsymbol {x}} _ {n} = \boldsymbol {x} _ {n} \boldsymbol {\omega} _ {n} ^ {\top} \tag {9}
$$
$$
\boldsymbol {\omega} _ {n} = \rho \boldsymbol {\omega} _ {n - 1} + \boldsymbol {\eta} _ {n}, \quad 0 <   \rho <   1, \tag {10}
$$
where  $\eta_{n}$  is the contextual state at time  $n$ , and  $\rho$  ensures smooth or gradual changes between contexts. Although context spans many domains, we focus solely on temporal context.
5 Fast and Forgetful Memory
Fast and Forgetful Memory (FFM) is a hybrid memory model based on theories of composite memory and contextual drift. It is composed of two main components: a cell and an aggregator. The cell receives an input and recurrent state  $\mathbf{x}$ ,  $S$  and produces a corresponding output and updated recurrent state  $\mathbf{y}$ ,  $S$  (Figure 1). The aggregator resides within the cell and is responsible for updating  $S$  (Figure 2). We provide the complete equations for both the aggregator and cell and end with the reasoning behind their design.
Aggregator The aggregator computes a summary  $S_{n}$  of  $X_{1:n}$ , given a recurrent state  $S_{k-1}$  and inputs  $X_{k:n}$
$$
\boldsymbol {S} _ {n} = \boldsymbol {\gamma} ^ {n - k + 1} \odot \boldsymbol {S} _ {k - 1} + \sum_ {j = k} ^ {n} \boldsymbol {\gamma} ^ {n - j} \odot \left(\boldsymbol {x} _ {j} \mathbf {1} _ {c} ^ {\top}\right), \quad \boldsymbol {S} _ {n} \in \mathbb {C} ^ {m \times c} \tag {11}
$$
$$
\boldsymbol {\gamma} ^ {t} = \left(\exp (- \boldsymbol {\alpha}) \exp (- i \boldsymbol {\omega}) ^ {\top}\right) ^ {\odot t} = \left[ \begin{array}{c c c} \exp (- t \left(\alpha_ {1} + i \omega_ {1}\right)) & \dots & \exp (- t \left(\alpha_ {1} + i \omega_ {c}\right)) \\ \vdots & & \vdots \\ \exp (- t \left(\alpha_ {m} + i \omega_ {1}\right)) & \dots & \exp (- t \left(\alpha_ {m} + i \omega_ {c}\right)) \end{array} \right] \in \mathbb {C} ^ {m \times c}. \tag {12}
$$
where  $\odot$  is the Hadamard product (or power),  $m$  is the trace size,  $c$  is the context size, and  $\alpha \in \mathbb{R}_+^m, \omega \in \mathbb{R}^c$  are trainable parameters representing decay and context respectively. Multiplying column a vector by  $\mathbf{1}_c^\top$  "broadcasts" or repeats the column vector  $c$  times.
In Appendix C, we derive a memory-efficient and numerically stable closed form solution to compute all states  $S_{k:n}$  in parallel over the time dimension. Since our model operates over relative time, we map absolute time  $k, \ldots, n$  to relative time  $p \in 0, \ldots, t$ , where  $t = n - k$ . The closed form to compute a state  $S_{k + p}$  given  $S_{k - 1}$  is then
$$
\boldsymbol {S} _ {k + p} = \boldsymbol {\gamma} ^ {p + 1} \odot \boldsymbol {S} _ {k - 1} + \boldsymbol {\gamma} ^ {p - t} \odot \sum_ {j = 0} ^ {p} \boldsymbol {\gamma} ^ {t - j} \odot \left(\boldsymbol {x} _ {k + j} \mathbf {1} _ {c} ^ {\top}\right), \quad \boldsymbol {S} _ {k + p} \in \mathbb {C} ^ {m \times c}. \tag {13}
$$
We can rewrite the aggregator's closed form (Equation 13) in matrix notation to highlight its time-parallel nature, computing all states  $S_{k:n} = S_k, S_{k+1}, \ldots, S_{k+n}$  at once
$$
\boldsymbol {S} _ {k: n} = \left[ \begin{array}{c} \boldsymbol {\gamma} ^ {1} \\ \vdots \\ \boldsymbol {\gamma} ^ {t + 1} \end{array} \right] \odot \left[ \begin{array}{c} \boldsymbol {S} _ {k - 1} \\ \vdots \\ \boldsymbol {S} _ {k - 1} \end{array} \right] + \left[ \begin{array}{c} \boldsymbol {\gamma} ^ {- t} \\ \vdots \\ \boldsymbol {\gamma} ^ {0} \end{array} \right] \odot \left[ \begin{array}{c} \left(\sum_ {j = 0} ^ {0} \boldsymbol {\gamma} ^ {t - j} \odot \left(\boldsymbol {x} _ {k + j} \mathbf {1} _ {c} ^ {\top}\right)\right) \\ \vdots \\ \left(\sum_ {j = 0} ^ {t} \boldsymbol {\gamma} ^ {t - j} \odot \left(\boldsymbol {x} _ {k + j} \mathbf {1} _ {c} ^ {\top}\right)\right) \end{array} \right], \quad \boldsymbol {S} _ {k: n} \in \mathbb {C} ^ {(t + 1) \times m \times c}. \tag {14}
$$
The cumulative sum term in the rightmost matrix can be distributed across  $t$  processors, each requiring  $O(\log t)$  time using a prefix sum or scan [Harris et al., 2007].
Cell The aggregator alone is insufficient to produce a Markov state  $\pmb{y}$ .  $S$  is complex-valued and contains a large amount of information that need not be present in  $\pmb{y}$  at the current timestep, making
Figure 1: A detailed example of aggregator dynamics for  $m = 1, c = 2$ , showing how a one-dimensional input  $x_{1}$  contributes to the recurrent state  $S$  over time  $t = [1, n]$ . At the first timestep,  $S_{1}$  is simply  $x_{1}$ . Over time, the contribution of  $x_{1}$  to  $S$  undergoes exponential decay via the  $\alpha$  term (forgetting) and oscillations via the  $\omega$  term (temporal context). In this example, the contribution from  $x_{1}$  to  $S_{n}$  approaches zero at time  $n$ .
Figure 2: A visualization of an FFM cell running recurrently for a single timestep (i.e., inference mode). Inputs  $\boldsymbol{x}_k, \boldsymbol{S}_{k-1}$  go through various linear layers  $(\ell)$ , hadamard products  $(\odot)$ , addition  $(\oplus)$ , normalization (LN), and sigmoids  $(\sigma)$  to produce outputs  $\boldsymbol{y}_k, \boldsymbol{S}_k$ . The boxed region denotes the aggregator, which decays and shifts  $\boldsymbol{S}_{k-1}$  in time via  $\boldsymbol{\gamma}$ . During training, the aggregator computes  $\boldsymbol{S}_k, \ldots, \boldsymbol{S}_n$  in parallel so that we can compute all  $\boldsymbol{y}_k, \ldots, \boldsymbol{y}_n$  in a single forward pass.
it cumbersome to interpret for a downstream policy. Furthermore,  $X$  could benefit from additional preprocessing. The cell applies input gating to  $X$  and extracts a real-valued Markov state  $y$  from  $S$ .
$$
\tilde {\boldsymbol {x}} _ {k: n} = \ell_ {1} \left(\boldsymbol {x} _ {k: n}\right) \odot \sigma \left(\ell_ {2} \left(\boldsymbol {x} _ {k: n}\right)\right), \quad \tilde {\boldsymbol {x}} _ {k: n} \in \mathbb {R} ^ {(t + 1) \times m} \tag {15}
$$
$$
\boldsymbol {S} _ {k: n} = \operatorname {A g g} \left(\tilde {\boldsymbol {x}} _ {k: n}, \boldsymbol {S} _ {k - 1}\right), \quad \boldsymbol {S} _ {k: n} \in \mathbb {C} ^ {(t + 1) \times m \times c} \tag {16}
$$
$$
\boldsymbol {z} _ {k: n} = \ell_ {3} (\text {F l a t t e n} (\Re [ \boldsymbol {S} _ {k: n} ] | | \Im [ \boldsymbol {S} _ {k: n} ])), \quad \boldsymbol {z} _ {k: n} \in \mathbb {R} ^ {(t + 1) \times d} \tag {17}
$$
$$
\boldsymbol {y} _ {k: n} = \operatorname {L N} \left(\boldsymbol {z} _ {k: n}\right) \odot \sigma \left(\ell_ {4} \left(\boldsymbol {x} _ {k: n}\right)\right) + \ell_ {5} \left(\boldsymbol {x} _ {k: n}\right) \odot (1 - \sigma \left(\ell_ {4} \left(\boldsymbol {x} _ {k: n}\right)\right). \quad \boldsymbol {y} _ {k: n} \in \mathbb {R} ^ {(t + 1) \times d} \tag {18}
$$
Agg represents the aggregator (Equation 14) and  $\ell$  represents linear layers with mappings  $\ell_1, \ell_2: \mathbb{R}^d \to \mathbb{R}^m$ ,  $\ell_3: \mathbb{R}^{m \times 2c} \to \mathbb{R}^d$ ,  $\ell_4, \ell_5: \mathbb{R}^d \to \mathbb{R}^d$ .  $\Re, \Im$  extract the real and imaginary components of a complex number as reals, Flatten reshapes a matrix ( $m \times c \to mc$ ) and  $||$  is the concatenation operator. LN is nonparametric layer norm, and  $\sigma$  is sigmoid activation. Equation 15 applies input gating, Equation 16 computes the recurrent states, Equation 17 projects the state into the real domain, and Equation 18 applies output gating.
Modeling Inductive Biases To justify FFM's architecture, we detail its connections to computational psychology. Drawing from the theory of composite memory, we integrate sigmoidal gating to  $\pmb{x}$  in Equation 15, approximating the sparsity-driven Bernoulli term,  $B$ , presented in (Equation 7). This inductive bias represents that only a small subset of each sensory experience is stored in memory.
Following the blending constraint imposed by composite memory (Equation 8), we sum the traces together (Equation 11). We suspect this blending prior yields a model that is more robust to the bad policy updates that plague RL, compared to RNNs that apply nonlinear and destructive transforms (e.g. deletion) to the recurrent state.
In Equation 12, the exponential component  $\exp (-t\alpha)$  ensures that traces are forgotten according to composite memory decay from Equation 8. We choose an exponential decay  $e^{-|\alpha|}$ , approximating Murre and Dros [2015]. This inductive bias asymptotically decays traces, reducing the size of the model optimization space. This is in direct contrast to recent SL memory models which aim to store as much information as possible for as long as possible [Gu et al., 2021, Voelker et al., 2019, Schlag et al., 2021]. In our ablations, we show that forgetting is critical in RL.
Traditionally, contextual coding and composite memory are separate mechanisms, but we incorporate both through  $\pmb{\gamma}$ . The  $\exp(-i\omega t)$  component of  $\pmb{\gamma}$  in Equation 12 applies temporal context, enabling relative-time reasoning. Following contextual coding theory, we take the outer product of a trace vector, in our case decayed by  $\alpha$ , with a context vector produced by  $\omega$ . We can rewrite  $\pmb{\gamma}^t = \pmb{\gamma}^{t-1} \odot \pmb{\gamma}$ , mirroring the gradual recurrent context changes from Equation 10.
The recurrent state  $S$  is a complex-valued matrix, so we project it back to the real domain as  $z$  (Equation 17).  $z$  can be large for dimensions where  $\alpha$  is near zero (minimal decay), so we find it crucial to apply layer normalization in Equation 18. Finally, we apply a gated residual connection to improve convergence speed, letting gradient descent find the optimal mixture of input  $x$  to memory  $z$  for the Markov state  $y$ .
The Mathematics Underpinning FFM The goal of FFM is to produce a memory mechanism which can be parallelized across the time dimension. As discussed in section 4, we can achieve this through a summation of sensory information mixed by some arbitrary function  $\phi$  with a temporal context  $\omega_{j} = \psi (j)$ , generating an aggregator of the form
$$
\boldsymbol {S} _ {n} = \sum_ {j = 0} ^ {n} \phi (\psi (j), \boldsymbol {x} _ {j}). \tag {19}
$$
Any implementation under this formulation would provide memory with temporal context for absolute time. However, in order to maximize generalization, we wish to operate over relative time. This introduces a problem, as the relative temporal context  $\psi(n - j)$  associated with each  $x_{j}$  must be updated at each timestep. Therefore, we must find a function  $h$  that updates each temporal context with some offset:  $h(\psi(j), \psi(k)) = \psi(j + k)$ . Solving for  $h$ , we get:  $h(j, k) = \psi(\psi^{-1}(j) + \psi^{-1}(k))$ . If we also set  $\phi = h$ , then applying a temporal shift  $h(\cdot, k)$  to each term in the sum yields the same result as initially inserting each  $x_{j}$  with  $\psi(j + k)$ :
$$
h \left(\phi \left(\psi (j), x _ {j}\right), k\right) = \phi \left(\psi (j + k), x _ {j}\right) \tag {20}
$$
Consequently, it is possible to update the terms at each timestep to reflect the relative time  $n - j$ . However, it is intractable to recompute  $n$  relative temporal contexts at each timestep, as that would result in  $n^2$  time and space complexity for a sequence of length  $n$ . To apply temporal context in a batched manner in linear space, we can leverage the distributive property. If we select  $\phi(a, b) = a \cdot b$ , then the update distributes over the sum, updating all of the terms simultaneously. In other words, we can update the context of  $n$  terms in constant time and space (see Appendix C for the full derivation). As  $\psi$  is defined as a function of  $\phi$ , we can solve to find  $\psi(t) = e^{\xi t}$  (although any exponential constitutes a solution, we select the natural base). This yields an aggregator of the form
$$
\boldsymbol {S} _ {n} = \sum_ {j = 0} ^ {n} e ^ {\xi (n - j)} \boldsymbol {x} _ {j}. \tag {21}
$$