SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Recalling (3.2),  $\hat{h}_t = \sum_{n=1}^d R_n \lambda_n^{t-1}$ ,  $R_n, \lambda_n \in \mathbb{C}, t > 0$ , the  $\mathcal{O}(dL)$  complexity of the impulse response is apparent: for each of the  $t = 1, \dots, L$ ,  $\hat{h}_t$  can be computed in  $\mathcal{O}(d)$  time.
The  $\tilde{\mathcal{O}}(L)$  cost from the rational form follows by Lemma A.6.
Let  $h$  be a length- $L$  filter,  $\hat{h}$  a distilled filter of order  $d < L$  and let  $S_L, \hat{S}_L$  be the respective Hankel matrices. Then  $\inf_{\hat{S}_L} \| S_L - \hat{S}_L \|_2 = \sigma_d$ .
Proof. The theorem characterizes the best-case scenario in terms of approximation error of the distilled SSMs or a certain order  $d$  where it is clear that  $\mathrm{rank}\hat{\mathsf{S}}\leq d$ . This theorem is a direct application of the Adamyan-Arov-Krein (AAK) theory of infinite Hankel operators [7]. Let  $\hat{\mathsf{S}}_L^* = \arg \inf_{\hat{\mathsf{S}}_L}\| \mathsf{S}_L - \hat{\mathsf{S}}_L\| _2$ ; the AAK theorem says that every causal system can be optimally approximated by another causal system of lower dimension. Optimal here means
$$
\inf  \left\| \mathrm {S} _ {L} - \hat {\mathrm {S}} _ {L} ^ {*} \right\| = \inf  \left\| \mathrm {S} _ {L} - \mathrm {K} \right\|
$$
where the first infimum is taken over all Hankel matrices  $\mathbb{S}_L^*$  and the second over all arbitrary matrices  $\kappa$  (see [19, Chapter 8] and [54] for further details and references).
C.7 Proof of Proposition 3.3
The filter (3.2) has a state space matrices  $\mathsf{A} = \mathrm{diag}(\lambda_1, \ldots, \lambda_d) \in \mathbb{C}^{d \times d}$ ,  $\mathsf{B} = (1, \ldots, 1)^{\top} \in \mathbb{C}^{d \times 1}$ ,  $\mathsf{C} = (R_1, \ldots, R_d) \in \mathbb{C}^{1 \times d}$ ,  $\mathsf{D} = h_0$  whose step can be evaluated in  $\mathcal{O}(d)$  time and memory.
$\mathsf{D}$  is set to  $h_0$  by default. The result is proven showing that (3.2) can be written in the form  $\hat{h}_t = \mathsf{CA}^{t - 1}\mathsf{B}$  for  $t > 0$ . If we choose  $\mathsf{A} = \mathrm{diag}(\lambda)$  then the impulse response becomes
$$
\hat {h} _ {t} = \sum_ {n = 1} ^ {d} R _ {n} \lambda_ {n} ^ {t - 1} = \mathsf {C} [ \operatorname {d i a g} (\lambda) ] ^ {t - 1} \mathsf {B} = \sum_ {n = 1} ^ {d} \mathsf {C} _ {n} \mathsf {B} _ {n} \lambda_ {n} ^ {t - 1}
$$
The choice  $\mathsf{B}_n = 1$  for all  $n = 1,\ldots ,d$ ,  $\mathsf{B} = \mathbb{1}_d$  and  $\mathsf{C}_n = R_n$  finalizes a modal canonical state-space realization of the distilled filter. The  $\mathcal{O}(d)$  time complexity of the corresponding recurrent step is guaranteed by the decoupling of each state equation from another,
$$
x _ {t + 1} ^ {n} = \lambda_ {n} x _ {t} ^ {n} + u _ {t} \quad n = 1, \dots , d
$$
$$
y _ {t} = \sum_ {n = 1} ^ {d} R _ {n} x _ {t} ^ {n} + h _ {0} u _ {t}.
$$
Each of the  $d$  state equations can be computed (in parallel) in  $\mathcal{O}(1)$  time. The output equation is a dot product requiring  $d$  multiplications and  $d$  additions, hence the  $\mathcal{O}(d)$  time complexity of the recurrence.
C.8 Proof of Proposition 3.2
$x_{T} = (v_{T},\dots ,v_{T - d})$  where  $v = g*u$  and  $g$  is the filter whose transfer function is  $1 / \mathrm{den}(\hat{H})(z)$  and can be evaluated in  $\tilde{\mathcal{O}} (T)$ .
Without loss of generality, let us assume to have converted the distilled filter in canonical form (i.e. we have unrestricted access to the coefficients of the rational transfer function) and let  $D = 0$ . We use the notation of Section A.5. In  $z$ -domain, the state-to-input relation is given by
$$
Y (z) = \mathsf {C} X (z) = [ \beta_ {1} \quad \dots \quad \beta_ {d} ]   \bar {X} (z)
$$
On the other hand  $Y(z) = \hat{H}(z)U(z) = q(z) / p(z)U(z)$ . Therefore,
$$
\left[ \begin{array}{c c c} \beta_ {1} & \dots & \beta_ {d} \end{array} \right] X (z) = \frac {q (z)}{p (z)} U (z)
$$
$$
\Leftrightarrow \quad [ \beta_ {1} \quad \dots \quad \beta_ {d} ]   X (z) = [ \beta_ {1} \quad \dots \quad \beta_ {d} ] \left[ \begin{array}{c} z ^ {- 1} \\ \vdots \\ z ^ {- d} \end{array} \right] \frac {1}{p (z)} U (z)
$$
$$
\Leftrightarrow X (z) = \left[ \begin{array}{c} z ^ {- 1} \\ \vdots \\ z ^ {- d} \end{array} \right] \frac {1}{p (z)} U (z)
$$
Let  $V(z) = U(z) / p(z)$ . From the shift property of the  $z$ -transform it holds,
$$
\mathcal {Z} \{x \} (z) = X (z) = \left[ \begin{array}{c} z ^ {- 1} \\ \vdots \\ z ^ {- d} \end{array} \right] V (z) \quad \Leftrightarrow \quad x _ {t} = \left(v _ {t - 1}, v _ {t - 2}, \dots , v _ {t - d}\right)   \forall t > 0.
$$
$v$  can be obtained in  $\tilde{\mathcal{O}}(L)$  time via an FFT-convolution of the input  $u$  and  $g$ , the filter resulting from inverse transforming  $1/p(z)$ . The proof is convoluded setting  $t = L$
C.9 Proof of Theorem 4.1
Notation. We will be denoting the all 1 row vector of size  $k$ , given by  $[1 \quad 1 \quad \ldots \quad 1 \quad 1]$ , and the all 0 row vector of size  $k$ , given by  $[0 \quad 0 \quad \ldots \quad 0 \quad 0]$ , as  $\mathbf{1}^k$  and  $\mathbf{0}^k$ , respectively. We will also construe the standard basis vector  $e_i$  as a column vector in these notes. Next, we will adhere to the following matrix indexing convention:  $\mathsf{A}_{ij}$  is the entry in the  $i$ th row and the  $j$ th column,  $\mathsf{A}[i,:] \in \mathbb{F}^{1 \times n}$  denotes the  $i$ th row, and  $\mathsf{A}[:,j] \in \mathbb{F}^{m \times 1}$  denotes the  $j$ th column of  $\mathsf{A} \in \mathbb{F}^{m \times n}$ . Here, we also use  $\mathbb{0}^{m \times n} \in \mathbb{R}^{m \times n}$  and  $\mathsf{l}_n$  to denote the matrix of all zeros and the identity matrix of dimension  $n$ , respectively. Moreover, we extend the outer product between two vectors to a tensor product using the symbol  $\otimes$ , the computation of which is carried out batch-wise with some dimension of one or both of the input tensors. Finally, we express the binary encoding of  $i \in [n]$  in a row vector form, given by  $\mathsf{B}_i \in \mathbb{Z}_2^{P_n}$ , where  $P_n$  is the closest power of 2 to  $n$ .
Language and Model Description. The language  $\Lambda$  has  $s$  keys and  $s$  values:  $L_{K} := \{k_{1},\ldots ,k_{s}\}$ ,  $L_{V} := \{v_{1},\ldots ,v_{s}\}$ . Formally, the language  $\Lambda$  consists of sequences  $x\in (L_K\times L_V)^s\times L_K$ , where there is an associated mapping  $f_{x}:L_{K}\to L_{V}$ . For each sequence, the odd indices in  $[L]$  belong to  $L_{K}$ , for  $x_{1},x_{3},\ldots ,x_{L}$ , and we define
$$
x _ {2 \cdot i} = f _ {x} \left(x _ {2 \cdot i - 1}\right) \tag {C.3}
$$
The last item  $x_{L} \in \{x_{1}, x_{3}, \ldots, x_{L-1}\}$ , called the query, must be one of the keys that has appeared in  $x$  already. Our goal is to produce  $f_{x}(x_{L})$  at the end of the sequence, which we refer as the associated value. This problem is termed as the associative recall problem [55].
We will now outline the Hyena layer [2] with multiple heads as follows.
Algorithm 1 Hyena
Require: Input sequence  $u \in \mathbb{R}^{L \times D}$  from the previous layer, long convolution filter  $\mathsf{T}_h$ , number of heads  $M$ .
1:  $q^{m},k^{m},v^{m}\gets \mathsf{Projection}(u)$  for  $m\in [M]$
2: for  $m = 1,\dots ,M$  do
3: Perform the outer product  $z^m \gets k^m \otimes v^m \in \mathbb{R}^{L \times N \times N}$ , where  $N := D / M$ .
4: Apply the convolution independently and compute  $y_{t}^{m}\gets \mathsf{T}_{h}(z_{t}^{m})q_{t}^{m}\in \mathbb{R}^{L\times N}$
5: Average the output  $\overline{y} \gets \left( \sum_{m} y^{m} / M \right)$
6: Retrieve the value  $f(k_{L})$  of the key  $k_{L}$  from  $\overline{y}[L, :]$ .
In order to prove Theorem 4.1, we need the following technical statement concerning sparse recovery of a heavy-hitter.
Proposition C.1 (Heavy-Hitter Recovery). Let  $x \in \mathbb{R}^s$  be a vector with one entry bounded by  $1 \pm \frac{1}{3\sqrt[4]{s}}$  referred as the heavy-hitter—and the rest of the entries bounded by  $\pm \frac{1}{3\sqrt[4]{s}}$ . Then, there exists a matrix  $S^{(m)} \in \mathbb{R}^{s \times O(\sqrt{s} \log s)}$  such that the position of the heavy-hitter in  $x$  can be inferred from the average of  $M$  measurements with  $S^{(m)}$  given by  $\left( \sum_{m} x S^{(m)} \right) / M$  with probability of error  $\leq \frac{1}{s}$ .
Before presenting the proof of Proposition C.1, we use it to prove Theorem 4.1 as follows.
Proof of Theorem 4.1. We take  $D = O(\sqrt{s} \log^2 s)$  and  $M = 243 \cdot \log s$  so that  $N = O(\sqrt{s} \log s)$  and use the same projections and filters for each head. We will start by describing the projections of the input. To this end, let  $E: [L] \to 2s$  define a map from the row indices of  $u$  to the keys  $k_i$  and values  $f_x(k_i)$  given by
$$
E (t) = \left\{ \begin{array}{l l} i, & t \text {o d d}, x _ {t} = k _ {i}, \\ i + s, & t \text {e v e n}, x _ {t - 1} = k _ {i}, \end{array} \right. \tag {C.4}
$$
Here, we note that we also have
$$
E (t) = E (t - 1) + s, \quad t \text {e v e n} \tag {C.5}
$$
as the even indices are defined as  $x_{t} = f_{x}(x_{t - 1})$  for  $t$  even (C.3), whence  $x_{t - 1} \in L_K$  as  $t - 1$  is odd. Next, we can separate the keys  $q$ , queries  $q$  and values  $v$  from the input sequence  $u$ . For keys and queries, we will be using the Johnson-Lindenstrauss embedding [56]. We state its guarantee here.
For a set of points  $P \subseteq \mathbb{R}^s$ , let  $\epsilon, \delta > 0$  with  $k \geq 2\ln \left(\frac{2s}{\delta}\right) / \epsilon^2$ , and  $f: \mathbb{R}^s \to \mathbb{R}^k$  be the randomly constructed linear map from [56], then with probability of error  $\leq \delta$ , we have
$$
| \langle f (x), f (y) \rangle - \langle x, y \rangle | \leq \epsilon
$$
for all  $x,y\in P$
More precisely, we take  $\mathsf{R} \in \mathbb{R}^{O(\sqrt{s}\log s) \times s}$  to be the matrix representation of  $f$  with  $\epsilon := \frac{1}{3\sqrt[4]{s}}$  and  $\delta := \frac{1}{s^c}$  for some  $c > 1$  so that  $\mathsf{R}[:,i] = f(e_i)$ . Thus, we define
$$
q ^ {m} [ t,: ] = \left\{ \begin{array}{l l} \mathsf {R} [:, E (t - 1) ], & E (t - 1) \leq s \\ 0, & \text {o t h e r w i s e .} \end{array} \right. \tag {C.6}
$$
For values, we use the heavy-hitter recovery matrix as described in Proposition C.1 so that we have
$$
v ^ {m} [ t,: ] = \left\{ \begin{array}{l l} \mathsf {S} ^ {(m)} [ E (t),: ], & E (t) > s \\ 0, & \text {o t h e r w i s e}, \end{array} \right. \tag {C.7}
$$
Further, using 1DConv (equivalently, in terms of polynomials,  $h(X) \coloneqq X$ ), we can shift the queries to get the projection for keys  $k$  so that we have  $k^m[t, :] = q^m[t - 1, :]$
The Hyena filters, along with the specific convolution being performed by  $\mathsf{T}_h$ , are specifically described in terms of polynomial multiplications, for all  $m \in 1, \ldots, M$ , as follows.
$$
T _ {h} (X) := \sum_ {i = 0} ^ {L} X ^ {i}.
$$
Here, we note that  $\mathsf{T}_h(u)$  takes the cumulative sum over the input. That is, for all  $i$ , we have
$$
\mathrm {T} _ {h} (u) [ i,: ] = \sum_ {j = 0} ^ {i} u [ j,: ]
$$
We will now compute  $z^{m}$  as follows
$$
z ^ {m} = k ^ {m} \otimes v ^ {m}
$$
Further, applying the convolution, we get
$$
\mathrm {T} _ {h} \left(z _ {t} ^ {m}\right) = \sum_ {i = 0} ^ {t} k ^ {m} [ i,: ] \otimes v ^ {m} [ i,: ]
$$
For inference, it suffices to show that the last row of the output  $y$  recovers the output with high probability. Indeed, let  $t' \in [L]$  denote the row index of the value associated to the query such that the corresponding key has the following relation
$$
u _ {t ^ {\prime} - 1} = u _ {L}. \tag {C.8}
$$
Finally, we multiply by the query  $q$  across  $L$ . Specifically, we now look at the computation of the  $L$ th row of  $y$ :
$$
\begin{array}{l} y ^ {m} [ L,: ] = \left(\sum_ {t = 0} ^ {L} k ^ {m} [ t,: ] \otimes v ^ {m} [ t,: ]\right) q ^ {m} [ t,: ] \\ = \sum_ {t = 0} ^ {L} \left(q ^ {m} [ L,: ] ^ {\top} k ^ {m} [ t,: ]\right) v ^ {m} [ t,: ] \\ = \sum_ {t = 0} ^ {L} \left(q ^ {m} \left[ t ^ {\prime} - 1, : ] \right. ^ {\top} q ^ {m} [ t - 1, : ]\right) v ^ {m} [ t, : ] (C.9) \\ = \sum_ {\substack {t \in [ L ] \\ t \text{even}}}\left(\left(\mathrm {R} [:, E \left(t ^ {\prime} - 1\right])\right) ^ {\top} \mathrm {R} [:, E (t - 1)]\right) \mathrm {S} ^ {(m)}[ E (t), :] (C.10) \\ = \sum_ {\substack {t \in [ L ] \\ t \text{even}}} ^ {L} \left(\mathsf {R e} _ {E \left(t ^ {\prime} - 1\right)} ^ {\top} \mathsf {R e} _ {E (t - 1)}\right) \mathsf {S} ^ {(m)} [ E (t),: ] (C.11) \\ \end{array}
$$
Here, we are using the fact that  $k^m[t' - 1,:] = q^m[L,:]$  due to (C.8) in (C.9). We then change the indexing from (C.9) and (C.11) by observing that all the odd entries corresponding to values are zeroed out in  $\mathbf{K}$  (cf. C.6). Finally, we simply substitute (C.6) and (C.7) in (C.10) and (C.11), respectively.
Next, we define  $x \in \mathbb{R}^{s+1}$  with
$$
x _ {j} := \operatorname {R e} _ {E \left(t ^ {\prime} - 1\right)} ^ {\top} \operatorname {R e} _ {j} \tag {C.12}
$$
where  $j = t - 1$  with  $t \in [L]$  and  $t$  even. Here,  $x \in \mathbb{R}^{n + 1}$  is a vector of size  $s + 1$  as there are  $s + 1$  such even numbers in  $[L]$ . Note that  $x$  is the vector with a heavy-hitter from Proposition C.1. To see
this, observe that we have  $\left|x_{E(t' - 1)} - 1\right| \leq \frac{1}{3\sqrt[4]{s}}$  and  $|x_j| \leq \frac{1}{3\sqrt[4]{s}}$  for all  $j \neq E(t' - 1)$ . Using (C.11), with probability  $\geq 1 - \frac{1}{s^c}$ , we then have
$$
y ^ {m} [ L,: ] = x \mathbf {S} ^ {(m)}. \tag {C.13}
$$
By Proposition C.1, we can then infer the position of the key at  $t'$  with probability of error  $\frac{1}{s}$ . By the union bound, we can then retrieve the corresponding value with probability at least  $1 - \left(\frac{1}{s} + \frac{1}{s^c}\right)$ .
We will now prove C.1 as follows.
Proof of C.1. We will assume that  $s$  is a power of 2 for the sake of simplicity. We first specify how we will construct such an  $\mathsf{S}^{(m)}\in \mathbb{R}^{s\times O(\sqrt{s}\log s)}$ . Let  $h:[s]\to [\sqrt{s} ]$  be a hash function. We define  $\tilde{\mathbf{S}}\in \mathbb{R}^{s\times \sqrt{s}}$  to be
$$
\tilde {S} [:, i ] = \sum_ {j: h (j) = i} e _ {j}.
$$
That is, each column  $i$  of  $\tilde{S}$  is the sum of the standard basis vectors  $\mathbf{e}_j$  such that  $j$  is mapped by  $h$  to  $i$ . In other words, the locations of the non-zero entries in column  $i$  correspond to the preimage of  $i$  under  $h$ . We then multiply each non-zero entry of  $\tilde{S}$  independently at random by  $\pm 1$ . Next, we replace the  $k$ th row in  $\tilde{S}$  by multiplying all non-zero  $\pm 1$  entries at index  $i$  with the binary representation of  $i$  to get a matrix  $S^{(m)} \in \mathbb{R}^{s \times (\sqrt{s} \times \log s)}$ . That is, for a non-zero entry at index  $i$  in row  $k$ , we replace the  $i$ th entry with  $\pm 1 \cdot \mathbf{B}_i$ . Note here that each column still has at most  $\sqrt{s}$  non-zero entries. Finally, we stack  $243 \cdot \log s$ -many copies of  $S^{(m)}$  as heads so that each copy produces independent measurements  $x S^{(m)}$ . Here, we want to emphasize that each such copy uses fresh randomness for multiplying the non-zero entry of  $\tilde{S}$  independently at random by  $\pm 1$ .
Now, we will show that the average of the measurements with matrices  $\mathsf{S}^{(m)} \in \mathbb{R}^{s \times \sqrt{s} \log s}$  can locate the heavy-hitter in  $x$ , where  $x$  is the vector of inner products from (C.12). For this purpose, we first specify the algorithm for decoding the heavy-hitter.
Algorithm 2 Decoder
Require: The vector  $y$  such that  $y = x\mathbf{S}$
1: Split  $y$  into  $243 \cdot \log s$  blocks  $y^{(m)} \in \mathbb{R}^{\sqrt{s} \log s}$ , each of which is a result of multiplying  $x$  by  $S^{(m)}, m \in [243 \cdot \log n]$ .
2: Take the average  $\overline{y} \gets \frac{1}{243 \cdot \log s} \sum_{k} y^{(m)}$ .
3:  $\pmb{b} \gets I_r(|\overline{y}|) \in \mathbb{R}^{\sqrt{s}\log s}$ , cf. (C.14).
4: Retrieve  $\pmb{b}$  by isolating the binary representation of the position of the heavy-hitter in  $x$ .
Here, we define the function  $I_r: \mathbb{R}^{\sqrt{s}\log s} \to \mathbb{Z}_2^{\sqrt{s}\log s}$  to  $[x\mathsf{S}]_m$  that rounds each entry of its input to the nearest integer:
$$
I _ {r} ([ x \mathbf {S} ] _ {m}) = \mathbf {S} ^ {(m)} [ i, : ]. \tag {C.14}
$$
That is, in both cases, we retrieve the row in  $\mathsf{S}^{(m)}$  that corresponds to the heavy-hitter in  $x$ . Since the rows in  $\mathsf{S}^{(m)}$  are distinct, we can also infer the position of the heavy-hitter in  $x$  with probability 1.
We now show that  $y = x\mathsf{S}$ , which consists of  $243 \cdot \log n$  many independent copies of  $y^{(m)} = x\hat{\mathsf{S}}^{(m)}$ . Instead, notice that we can analyze  $\tilde{y} = x\tilde{\mathsf{S}}$  since each  $\hat{y}$  is the replacement of non-zero entries of  $\tilde{\mathsf{S}}$  with the binary representation of their indices times  $y$ . We will drop the superscript for now to avoid cluttering the notation. We can then make the following claim:
$$
\left| \tilde {y} _ {i} \right| = 1 \pm O \left(\epsilon \cdot \sqrt [ 4 ]{s}\right) \text {a n d , f o r} j \neq i, \left| \tilde {y} _ {j} \right| = O \left(\epsilon \cdot \sqrt [ 4 ]{s}\right). \tag {C.15}
$$
For the above claim, we note that the first part follows from the latter as it suffices to show that all the non-zero heavy-hitters contribute  $O(\epsilon \sqrt[4]{s})$  to the sum  $\tilde{y}_i = \langle x, \tilde{\mathbf{S}}[:, i] \rangle$ . Since each column in  $\tilde{\mathbf{S}}$  only interacts with  $\sqrt{s}$  sized sub-vector of  $x$ , each  $\tilde{y}_j$  for non-heavy hitters can be expressed as
$$
\tilde {y} _ {j} = \left\langle \bar {x}, \bar {\mathbf {S}} _ {j} \right\rangle \text {w i t h} \bar {x}, \bar {\mathbf {S}} _ {j} \in \mathbb {R} ^ {\sqrt {s}},
$$
where  $\overline{S}_j \in \mathbb{R}^{\sqrt{s}}$  contains the non-zero entries of  $\tilde{S}_j$  and  $\overline{x}$  is obtained by extracting the entries with corresponding indices from  $x$ . Here, we have  $\| \overline{x} \|_2 \leq \epsilon \sqrt[4]{s}$  since each entry associated with the non-heavy hitter is bounded as  $x_i \leq \epsilon$ , and thus,  $\| \overline{x} \|_2 = \sqrt{\sum_i x_i^2} \leq \sqrt{\sum_i \epsilon^2} = \sqrt{\sqrt{s} \cdot \epsilon^2} = \epsilon \sqrt[4]{s}$ .
Consequently, as  $\bar{S}_j$  is independently random  $\pm 1$ , we then must have
$$
\left| \langle \bar {x}, \bar {\mathrm {S}} _ {j} \rangle \right| \leq \frac {1}{3} \tag {C.16}
$$
with constant probability for  $j \neq i$ . To see this, note that
$$
\begin{array}{l} \mathbb {E} [ \langle \overline {{x}}, \overline {{\mathsf {S}}} _ {j} \rangle^ {2} ] = \sum_ {k, \ell} \mathbb {E} [ \overline {{\mathsf {S}}} _ {j k} \cdot \overline {{\mathsf {S}}} _ {j \ell} ] \cdot \overline {{x}} _ {i} \cdot \overline {{x}} _ {j} \\ = \| \bar {x} \| _ {2} ^ {2}, \\ \end{array}
$$
where the last equality follows since  $\mathbb{E}[\mathsf{S}_{jk}\cdot \mathsf{S}_{j\ell}] = \delta_{k,\ell}$  by the distribution on entries of  $\mathsf{S}_j$ . Now, we use Jensen's inequality [57] to get the following bound on the expectation of  $|\langle \overline{x},\overline{\mathsf{S}}_j\rangle |$
$$
\mathbb {E} \left[ | \langle \bar {x}, \bar {\mathrm {S}} _ {j} \rangle | \right] \leq \sqrt {\mathbb {E} [ \langle \bar {x} , \bar {\mathrm {S}} _ {j} \rangle^ {2} ]} \leq \epsilon \sqrt [ 4 ]{s}. \tag {C.17}
$$
We then use the expectation above to bound the relevant probability as follows:
$$
\begin{array}{l} \Pr \left[ | \langle \bar {x}, \bar {\mathbf {S}} _ {j} \rangle | \leq \frac {1}{3} \right] \geq 1 - \Pr \left[ | \langle \bar {x}, \bar {\mathbf {S}} _ {j} \rangle | \geq \frac {1}{3} \right] \\ \geq 1 - 3 \mathbb {E} \left[ | \langle \bar {x}, \bar {\mathrm {S}} _ {j} \rangle | \right] \tag {C.18} \\ \geq 1 - 3 \epsilon \cdot \sqrt [ 4 ]{s}, \\ \end{array}
$$
where we apply Markov's inequality [57] in (C.18). That is, we have shown that  $\tilde{y}_j$  is bounded by  $1/3$  with constant probability for  $j \neq i$ , and  $\tilde{y}_i$  is thus bounded by  $1 \pm \frac{1}{3}$ . Note here that each of the  $m$ -copies  $\tilde{y}_i^{(m)}$  will have identical guarantees.
Now, define the average  $\overline{\tilde{y}}_j \coloneqq \frac{1}{243\cdot\log s}\sum_m\tilde{y}^{(m)}$  so that  $\overline{y}_j$  (line 3 in Decoder) is the corresponding replacement of the non-zero entries with the binary representation of their indices. We now claim that this average  $\overline{\tilde{y}}_j \leq 4/9 < 1/2$  with high probability for  $j \neq i$ . To this end, we employ the multiplicative Chernoff bound [57] on the independent random variables  $\{\tilde{y}_j^{(h)}\}_{h}[0,1]$  with  $\mathbb{E}\left[\sum_{m}\tilde{y}_j^{(m)}\right] \leq 81\cdot \log s$  to get
$$
\begin{array}{l} \Pr \left[ \bar {\tilde {y}} _ {j} > \frac {4}{9} \right] = \Pr \left[ \sum_ {m} \tilde {y} _ {j} ^ {(m)} > \left(1 + \frac {1}{3}\right) \frac {1}{3} \cdot 2 4 3 \cdot \log s \right] \\ \leq \Pr \left[ \sum_ {m} \tilde {y} _ {j} ^ {(m)} \geq \left(1 + \frac {1}{3}\right) 8 1 \cdot \log s \right], \\ \leq \exp \left(- \left(\frac {1}{3 ^ {2}} \cdot 8 1 \cdot \log s\right) / 3\right), \\ = \frac {1}{s ^ {3}}. \\ \end{array}
$$
Therefore, we have shown that the average  $\overline{y}_j$  is less than  $1/2$  with probability at least  $1 - \frac{1}{s^3}$  for  $j \neq i$ . Consequently, we will have  $\overline{y}_i$  bounded by  $1 \pm \frac{1}{2}$ . Using the union bound over each  $j \neq i$  and the log  $s$  bits in the binary representation of  $j$ , we can then show that  $\overline{y}_{j+m} < 1/2$  for each  $j \neq i$ ,  $m \in [0, \log s]$  with probability  $1 - \frac{\log s}{s^2} \gg 1 - \frac{1}{s}$ .
D.1 Pre-training
To verify the effect of introducing heads to Hyena as described in Section 4, we train a series of models on THE PILE [11]. All MultiHyena models are set to 8 heads, and otherwise use the same hyperparameters of Hyena models of equivalent size. We set weight decay of Hyena filter parameters to 0, and lower the frequency of sine activations in the implicit MLP to 4. We follow the setup of [2], and first train models for 5, 10 and 15 billion tokens, adjusting the learning rate scheduler accordingly. Then, we train for 300 billion tokens. The results are reported in Tables 5.1 and 5.1.
D.2 Distillation Analysis
Distilling pre-trained long convolution sequence models (LCSM) with LaughingHyena can introduce errors on the convolution filter, which then propagate to the outputs.
Setup We perform a series of extensive experiments on all variants of LCSM, including pre-trained H3 models of sizes 125 million, 355 million, 1.3 billion and 2.7 billion parameters; Hyena of size 153 million parameters, and MultiHyena of size 153 million parameters. For H3 models, we report approximation errors on both shift as well as diagonal SSMs (reported as IIR and FIR). Each point corresponds to distillation carried out at a particular order, using LaughingHyena modal interpolation. To optimize the parameters of the modal form, we use gradient-based optimization and minimize the  $\ell_2$  discrepancy between filters in time domain. In particular, we use the ADAMW [58] optimizer with learning rate  $3\cdot 10^{-4}$ , and a cosine annealing decay schedule down to  $10^{-6}$  after 30 thousand iterations. Each individual filter of every layer is distilled in the same way.
Discussion The errors are shown in Figures D.1, D.1, D.2, D.3, D.4 and D.5. We observe H3 filters to be easier to distill into recurrences with small state without introducing significant errors, whereas Hyena variants learn filters with larger effective dimensions. This provides further evidence that training with implicit convolutions may yield in general more expressive filters.
Distillation Error of H3 125M
Figure D.1: Mean, lower and upper bounds across channels and layers of the distillation errors on 125M H3 model for both its IIR and FIR filters.
D.2.1 Pretrained Filters: Effective Dimension
Visualizations We qualitatively investigate LCSM filters at initialization and after pretraining. This visual inspection (Figures D.6, D.7 and D.8) complements the distillation error analysis of Section D.2.