SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Distribution of Hankel singular values We further compute the distribution of Hankel singular values of each long convolution filter in different models. The decay in the spectrum quantifies how easy it is to find a compact modal form with LaughingHyena, and serves as a proxy measure of effective dimension of the convolution. The results are shown in Figures D.9 and D.10.
Distillation Error of H3 355M
Figure D.2: Mean, lower and upper bounds across channels and layers of the distillation errors on 355M H3 model for both its IIR and FIR filters.
Distillation Error of H3 1.3B
Figure D.3: Mean, lower and upper bounds across channels and layers of the distillation errors on 1.3B H3 model for both its IIR and FIR filters.
D.3 Downstream Evaluation
We benchmark the downstream performance of MultiHyena and distilled MultiHyena on standard language modeling tasks from the LM-Eval-Harness [42] and HELM [41] suites. As a reference baseline, we evaluate Pythia [44] 160M.
Our objective is to quantify the absolute performance of MultiHyena and the downstream impact of distillation. We use the same procedure outlined in Section D.2 to distill MultiHyena.
D.4 Benchmarking
To demonstrate the superior performance of Laughing Hyena for autoregressive generation, we conduct a series of experiments to benchmark its latency, throughput, and memory usage for autoregressive generation with initial prompt length  $T$  and number of generated tokens  $K$ . For each experiment, we compare the performance of Laughing Hyena against a Transformer, a hybrid H3-attention model with 2 attention layers and a Hyena model. The latter two have been shown to match or achieve lower perplexity than Transformers on standard datasets (WIKITEXT103 and THE PILE). All experiments are carried out on a NVIDIA A100 with 80GB in float16 precision. Missing measurements for any model indicate Out of Memory (OOM) errors while doing autoregressive inference for that particular model.
Distillation Error of H3 2.7B
Figure D.4: Mean, lower and upper bounds across channels and layers of the distillation errors on 2.7B H3 model for both its IIR and FIR filters.
Distillation Error of MultiHyena 155M (300B tokens)
Distillation Error of Hyena 155M (150B tokens)
Figure D.5: Mean, lower and upper bounds across channels and layers of the distillation errors on Hyena and MultiHyena models.
Peak throughput We first evaluate the throughput (number of tokens generated per second) across different batch sizes, using a typical generation workload consisting of a prompt of length 512 and generating 256 tokens. Figure 1.1 measures peak throughput of different models. Since Laughing Hyena does not require caching intermediate kv-projections during generation, reduced memory requirements at a fixed model size allow it to process larger batch sizes.
Prompt length Autoregressive generation in Laughing Hyena is achieved through a two-step process: an initial prefetch step that uses the length-  $T$  prompt to initialize the state  $x_{T}$  and that generates all  $K$  tokens. In Figure 5.3 we demonstrate how the prefetch step scales for different prompt lengths, keeping batch sizes fixed at 64. Since prefetching in Laughing Hyena is carried out efficiently via convolutions (as described in Section 3.4), throughput scales more favorably than Transformers. Other models capable of prefetching via convolutions also achieve higher throughputs than Transformers but are ultimately slower than Laughing Hyena during the generation phase.
State throughput We measure the impact of SSM state dimension on the throughput of Laughing Hyena. Keeping batch sizes fixed reveals minimal impact for all dimensions smaller than
FIR Filters of H3 125M
IIR Filters of H3 125M
Figure D.6: Initialized and pre-trained convolution filters of H3.
100, which are sufficient to distill all models discussed in this work. All other measurements provided in this Section are carried out with a standard order 16. We note that it may be possible to further increase peak throughput by leveraging reduced memory footprints achieved by extremely small SSMs.
Latency over sequence length We benchmark the time taken to generate a variable number of tokens, starting from a prompt of length 512 tokens at batch size 1 (Figure D.11). Laughing Hyena tracks highly optimized Transformers. We note that Laughing Hyena is asymptotically more efficient than Transformers; however, this regime is bottlenecked by hardware-specific implementation details and optimizations. We expect optimized, platform-specific implementations of Laughing Hyena to outperform Transformers even at batch size 1. When the prompt is long, the prefilling step becomes the bottleneck, and all convolutional models outperform Transformers.
Parameter scaling To better understand how the performance of Laughing Hyena scales, we benchmark its latency, throughput, and peak memory utilization for autoregressive generation and 125M, 355M, 1.3B, 2.7B and 6.7B parameters. We compare the performance to that of Transformers, Hybrid-H3, and Hyena at the same number of parameters and report the results in Figure D.11. For the latency measurement, we use a batch size of 1 and benchmark the time taken to generate 128 tokens, starting from a prompt of length 512 tokens. For throughput and peak memory scaling against the number of parameters, we use a batch size of 64 and measure the throughput for generating 256 tokens starting with a prompt of length 512.
Filters of MultiHyena 155M (300B tokens)
Figure D.7: Initialized and pre-trained long convolution filters of MultiHyena.
Normalized Filters  $h_t / \| h \|_{\infty}$  Normalized Filters  $h_t / \| h \|_{\infty}$  Normalized Filters  $h_t / \| h \|_{\infty}$  Normalized Filters  $h_t / \| h \|_{\infty}$  Normalized Filters  $h_t / \| h \|_{\infty}$
Figure D.8: Initialized and pre-trained long convolution filters of Hyena  $(355M)$ .
Filters of Hyena 355M (200B tokens)
Hankel Singular Values of Hyena and MultiHyena Filters
Figure D.9: Distribution of Hankel singular values for Hyena and MultiHyena long convolution filters. MultiHyena filters have larger effective dimension, as evidenced by slower decay.
Hankel Singular Values of H3 (125M) FIR Filters
Hankel Singular Values of H3 (125M) IIR Filters
Figure D.10: Distribution of Hankel singular values for H3 long convolution filters. The values decay rapidly.
- - Laughing Hyena - - Hyena - - Hybrid Attn. H3 - - Transformer
Figure D.11: Generation latency, throughput and peak memory of Transformers, H3, Hyena and Laughing Hyena.
E.1 Associative Recall with MultiHyena
We follow the setup of [2] and train 2-layer Hyena and MultiHyena (with 8 heads) to solve associative recall via a standard next-token prediction objective. We focus on the sequence length 64k, high vocabulary size setting, and push vocabulary sizes past the maximum values considered in [2]. At vocabulary size 60, a difference between MultiHyena and Hyena can be observed (Table E.1), as experimental support for Theorem 4.1.
Model	Accuracy
Hyena	65
MultiHyena	98
Table E.1: Associative recall accuracy, sequence length 64k, vocabulary size 60.
E.2 Analysis of Hankel Singular Values of Pretrained Large Convolution Sequence Models
E.3 Model Order Reduction of H3
The H3 model is constructed with a combination of diagonal SSMs and shift SSMs. There exists various classical model order reduction techniques for these different types of SSMs. The following sections aim to present the formulation and effectiveness of two classical approaches on obtaining the compressed representation of a H3 model. More specifically, we study modal truncation and balanced truncation for compressing diagonal SSMs and shift SSMs respectively.
E.3.1 Modal Truncation
A discrete diagonal SSM  $(\mathsf{A} = \mathrm{diag}(\lambda_1,\dots ,\lambda_d),\mathsf{B}\in \mathbb{C}^{d\times 1}$  , and  $\mathsf{C}\in \mathbb{C}^{1\times d})$  can be directly converted into a residue-pole transfer function as follows:
$$
(\mathrm {A}, \mathrm {B}, \mathrm {C}) \rightarrow H (z) = \sum_ {i = 1} ^ {d} \frac {r _ {i}}{z - \lambda_ {i}}, \tag {E.1}
$$
where residue  $r_i = \mathsf{B}_i\mathsf{C}_i$ . Modal truncation aims to compress such a transfer function by essentially reducing the summation over  $d$  to  $n < d$ , of the  $n$  most influential modes. The influence from each node can be isolated by expressing it using the  $h_\infty$  norm of the system as follows:
$$
\| H (z) \| _ {\infty} = \sum_ {i = 1} ^ {d} \left\| \frac {r _ {i}}{z - \lambda_ {i}} \right\| _ {\infty} \leq \sum_ {i = 1} ^ {d} \frac {| r _ {i} |}{| 1 - | \lambda_ {i} | |}. \tag {E.2}
$$
Each mode  $i$  can be ranked using the bound formulated above. Subsequently, the  $d - n$  lowest modes could be discarded to form a reduced order model. Figure E.1 illustrates the monotonically decreasing  $l_{\infty}$  error with the increase in system order. However, this model reduction approach is only suitable for diagonalizable SSMs.
Modal Truncation Model Reduction Error of H3
Figure E.1: Modal truncation model reduction error  $(||h(t) - h_n(t)||_{\infty})$  across all diagonal SSM layers of the trained H3 125M model.
E.3.2 Balanced Truncation
A balanced SSM realization is one in which the observability  $P$  and controllability  $Q$  gramians are equal and diagonal. Such a realization can be formulated with the following Lyapunov equations [24]:
$$
\begin{array}{c} \mathrm {A} \Sigma \mathrm {A} ^ {\top} + \mathrm {B B} ^ {\top} = \Sigma , \\ \text {E . 3} \end{array}
$$
$$
\mathsf {A} ^ {\top} \Sigma \mathsf {A} + \mathsf {C} ^ {\top} \mathsf {C} = \Sigma ,
$$
where  $P = Q = \Sigma = \mathrm{diag}(\sigma_1,\dots ,\sigma_d)$  and  $\sigma_{i}\geq \sigma_{i + 1}$
Results from [59] shows that the  $n$ -order model reduction error is bounded by:
$$
E _ {\infty} \triangleq \| H (s) - H _ {n} (s) \| _ {\infty} \leq 2 \sum_ {i = d - n} ^ {d} \sigma_ {i}. \tag {E.4}
$$
Therefore, an  $n$ -order partition of the full balanced realization can be chosen, such that the discarded orders are the  $d - n$  lowest contributor to the error. The steps taken by [24], computes the  $n$ -order partition of the balanced realization as follows:
1. Form a Hankel matrix  $\mathbf{S}_d$  from the impulse response  $h_{1:n}$  of the shift SSM.
2. Obtain the eigenvector matrix  $V \in \mathbb{C}^{d \times d}$ , and the eigenvalues  $\lambda = \sigma^2$  via the eigendecomposition of  $S_d$ .
3. Choose the truncated model's order  $n < d$ , based on the bound in Equation E.4.
4. Compute the state-space matrices as follows:
$$
\mathsf {A} = V _ {2: d, 1: n} ^ {\top} V _ {1: d - 1, 1: n}, \quad \mathsf {B} = V _ {1, 1: n}, \quad \mathsf {C} = h _ {1: d} ^ {\top} V _ {1: d, 1: n}, \quad \mathsf {D} = h _ {0}. \tag {E.5}
$$
This model reduction technique was applied to a trained 125 million parameter H3, MultiHyena, and Hyena models as shown in Figures E.2, E.3, and E.4 respectively. It could be noted that the all models encountered an undesirable non-monotonic error reduction with the increase in order. Moreover, order reduction configurations such as the one in Figure E.3 layer 15 display signs of numerical instability.
Figure E.2: Balanced truncation model reduction error  $(||h(t) - h_n(t)||_{\infty})$  across all shift SSM layers of the trained H3 125M model. Note that Layer 6 is an Attention layer, therefore balanced truncation model order reduction is not possible.
Balanced Truncation Model Reduction Error of H3
Balanced Truncation Model Reduction Error of MultiHyena
Figure E.3: Balanced truncation model reduction error  $(||h(t) - h_n(t)||_{\infty})$  across all convolutional layers of the trained MultiHyena 155M model.
Balanced Truncation Model Reduction Error of Hyena
Figure E.4: Balanced truncation model reduction error  $(||h(t) - h_n(t)||_{\infty})$  across all convolutional layers of the trained Hyena 155M model.