Title: Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
Venue: NeurIPS 2023
Accepted: yes
Keywords: structured matrices

transformers

efficiency
Abstract: Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts and better performance. However, existing architectures such as Transformers scale quadratically along both these axes. We ask: are there performant architectures that can scale sub-quadratically along sequence length and model dimension? We introduce Monarch Mixer (M2), a new architecture that uses the same sub-quadratic primitive along both sequence length and model dimension: Monarch matrices, a simple class of expressive structured matrices that captures many linear transforms, achieves high hardware efficiency on GPUs, and scales sub-quadratically. As a proof of concept, we explore the performance of M2 in three domains: non-causal BERT-style language modeling, ViT-style image classification, and causal GPT-style language modeling. For non-causal BERT-style modeling, M2 matches BERT-base and BERT-large in downstream GLUE quality with up to 27% fewer parameters, and achieves up to 9.1$\times$ higher throughput at sequence length 4K. On ImageNet, M2 outperforms ViT-b by 1% in accuracy, with only half the parameters. Causal GPT-style models introduce a technical challenge: enforcing causality via masking introduces a quadratic bottleneck. To alleviate this bottleneck, we develop a novel theoretical view of Monarch matrices based on multivariate polynomial evaluation and interpolation, which lets us parameterize M2 to be causal while remaining sub-quadratic. Using this parameterization, M2 matches GPT-style Transformers at 360M parameters in pretraining perplexity on The PILE—showing for the first time that it may be possible to match Transformer quality without attention or MLPs.

MONARCH MIXER: A Simple Sub-Quadratic GEMM-Based Architecture

Abstract

Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts and better performance. However, existing architectures such as Transformers scale quadratically along both these axes. We ask: are there performant architectures that can scale sub-quadratically along sequence length and model dimension? We introduce MONARCH MIXER (M2), a new architecture that uses the same sub-quadratic primitive along both sequence length and model dimension: Monarch matrices, a simple class of expressive structured matrices that captures many linear transforms, achieves high hardware efficiency on GPUs, and scales sub-quadratically. As a proof of concept, we explore the performance of M2 in three domains: non-causal BERT-style language modeling, ViT-style image classification, and causal GPT-style language modeling. For non-causal BERT-style modeling, M2 matches BERT-base and BERT-large in downstream GLUE quality with up to 27% fewer parameters, and achieves up to 9.1 × higher throughput at sequence length 4K. On ImageNet, M2 outperforms ViT-b by 1% in accuracy, with only half the parameters. Causal GPT-style models introduce a technical challenge: enforcing causality via masking introduces a quadratic bottleneck. To alleviate this bottleneck, we develop a novel theoretical view of Monarch matrices based on multivariate polynomial evaluation and interpolation, which lets us parameterize M2 to be causal while remaining sub-quadratic. Using this parameterization, M2 matches GPT-style Transformers at 360M parameters in pretraining perplexity on The PILE-showing for the first time that it may be possible to match Transformer quality without attention or MLPs. 1

1 Introduction

Machine learning models in natural language processing and computer vision are being stretched to longer sequences and higher-dimensional representations to enable longer context and higher quality, respectively [6, 10, 62, 84]. However, existing architectures exhibit time and space complexities that grow quadratically in sequence length and/or model dimension-which limits context length and makes scaling expensive. For example, attention and MLP in Transformers scale quadratically in sequence length and model dimension [15]. In this paper, we explore a natural question: can we find a performant architecture that is sub-quadratic in both sequence length and model dimension?

P

0

P

i

M =

Order-

p

(

)

i

i = 1

p

def M2_layer(X):

# mix sequence

Z = M @ (k * (M @ X))

# mix channels

Y = M @

σ

(M @ Z.T))

return Y

Simple Layers

N

d

N

d

Efficient Mixing on

Sequence, Dimension

Π

Subquadratic:

)

Hardware-Efficient (GEMMs)

Expressive (generalizes FFT)

In our exploration, we seek a sub-quadratic primitive for both the sequence length and model dimension. Our framing takes inspiration from work such as MLP-mixer [73] and ConvMixer [74], which observed that many machine learning models operate by repeatedly mixing information along the sequence and model dimension axes, and used a single operator for both axes. Finding mixing operators that are expressive, sub-quadratic, and hardware-efficient is challenging. For example, the MLPs in MLP-mixer and convolutions in ConvMixer are expressive, but they both scale quadratically in their input dimension [73, 74]. Several recent studies have proposed sub-quadratic sequence mixing with long convolutions or state space models [27, 64, 77]-both computed using the FFT-but these models have poor FLOP utilization (3-5% [28]) and maintain quadratic scaling in model dimension. Meanwhile, there has been promising work in sparsifying dense MLP layers without losing quality, but some of the models can actually be slower than their dense counterparts, due to low hardware utilization [7, 8, 14, 26, 35].

We turn to an expressive class of sub-quadratic structured matrices called Monarch matrices [14] (Figure 1 left) to propose MONARCH MIXER (M2). Monarch matrices are a family of structured matrices that generalize the fast Fourier transform (FFT) and have been shown to capture a wide class of linear transforms including Hadamard transforms, Toeplitz matrices [32], AFDF matrices [57], and convolutions. They are parameterized as the products of block-diagonal matrices, called monarch factors , interleaved with permutation. Their compute scales sub-quadratically: setting the number of factors to p results in computational complexity of O ( pN ( p +1) /p ) in input length N , allowing the complexity to interpolate between O ( N log N ) at p = log N and O ( N 3 / 2 ) at p = 2 . 2

M2 uses Monarch matrices to mix information along the sequence and model dimension axes. It is both simple to implement and hardware-efficient: the block-diagonal Monarch factors can be computed efficiently on modern hardware using GEMMs (generalized matrix multiply algorithms). Our proof-of-concept implementation of an M2 layer, written in less than 40 lines of pure PyTorch (including imports), relies only on matrix multiplication, transpose, reshape, and elementwise products (see pseudocode in Figure 1 middle) and achieves 25.6% FLOP utilization 3 for inputs of size 64K on an A100 GPU. On newer architectures such as the RTX 4090, a simple CUDA implementation achieves 41.4% FLOP utilization at the same size.

Non-Causal Settings As a first proof of concept of M2, we evaluate how it compares to Transformers in terms of speed and quality in non-causal settings such as BERT-style masked language modeling [21] and ImageNet classification. We introduce M2-BERT, which replaces the attention blocks in BERT with bidirectional gated convolutions implemented using Monarch matrices and replaces the dense matrices in the MLP with Monarch matrices. M2-BERT reduces parameter count but maintains quality-matching BERT-base and BERT-large in downstream GLUE quality with 27% and 24% fewer parameters, respectively. Sub-quadratic scaling in sequence length enables high throughput at longer sequences-up to 9.1 × higher throughput at sequence length 4K than

HuggingFace BERT, and 3.1 × higher throughput at sequence length 8K than BERT optimized with FlashAttention [15].

For image classification, we adapt HyenaViT-b [64], an attention-free vision transformer based on gated convolutions. We replace the convolution operation with M2 primitives and replace the MLP layers with an M2 block as well. These changes reduce the parameter count compared to a ViT-b [22] model with the same model width and depth by a factor of 2. Surprisingly, despite this parameter reduction, we find that M2 slightly outperforms ViT-b and HyenaViT-b baselines, achieving 1% higher accuracy on ImageNet [18].

Causal Settings Causal settings such as GPT-style [65] auto-regressive language modeling present a technical challenge: masking out the upper triangular elements in an attention matrix (or equivalent structure) introduces a quadratic bottleneck. To alleviate this quadratic bottleneck with Monarch matrices, we develop new theory to characterize which parameterizations of Monarch matrices maintain causality. To do so, we take a view of p -order Monarch matrix multiplication as p -variate polynomial evaluation and interpolation (e.g., p = 2 factors corresponds to bivariate polynomials, Figure 2 left). Using this view, we show that the M2 convolution shown in Figure 1 (middle) can be viewed as manipulation of modular polynomial multiplication. This result allows us to develop conditions (Theorem 3) under which M2 is causal. We can use this causal parameterization to outperform GPT-style language models on causal language modeling by 0.2 PPL points on the PILE at model size 360M-without using either attention or MLP blocks.

Summary Overall, our results present a potential path to building machine learning models with sub-quadratic primitives. We hope our work can serve as a starting point to explore models that are more efficient in both sequence length and model dimension.

2 Preliminaries

In this section, we provide some background on the key components behind the cost of operations on GPUs, and then discuss the scaling characteristics of some common primitives used to mix information across the sequence dimension and model dimension in modern machine learning models.

GPU Accelerator Cost Model We provide a brief discussion of relevant factors affecting runtime performance of deep learning operations on GPUs. Depending on the balance of computation and memory accesses, operations can be classified as either compute-bound or memory-bound [44]. In compute-bound operations, the time accessing GPU memory is relatively small compared to the time spent doing arithmetic operations. Typical examples are matrix multiply with large inner dimension, and short convolution kernels with a large number of channels.

The speed of these operations is determined by the FLOP/s available on compute units, and the number of FLOPs necessary to complete the operation. In our paper, we exploit fast matrix multiply units such as tensor cores. On the A100, tensor cores can achieve 312 TFLOP/s in half-precision matrix multiply operations, while non-matrix multiply operations are limited to 19 TFLOP/s [59]. This trend began with tensor cores in the V100 [58], and is continuing into the next-generation H100 chips [60].

In memory-bound operations, the time taken by the operation is determined by the number of memory accesses, while time spent in computation is much smaller. Examples include most elementwise operations (e.g., activation, dropout) and reductions (e.g., sum, softmax, batch norm, layer norm).

Table 1: FLOP cost and utilization of various mixer layers, input dimension 64K on an RTX 4090.

FlashAttn FFT M2 Conv The runtime of memory-bound operations is determined by the memory bandwidth of different layers of the memory hierarchy . GPU memory is large but relatively slow-up to 80 GB on A100, but with bandwidth of 2 TB/s [59]. Higher levels of the memory hierarchy such as caches are much smaller (20 MB) but an order of magnitude faster (19 TB/s).

Common Mixer Primitives To help contextualize our work, we provide scaling and hardware utilization characteristics for a few common operations that are used to mix information in machine learning models, summarized in Table 1.

Transformers [75] use attention to mix information across the sequence dimension, and MLP blocks to mix information across the model dimension. Both of these blocks scale quadratically in input length. MLP layers are compute-bound, so they have high FLOP utilization out of the box. Attention blocks are memory-bound, so even the most optimized implementations such as FLASHATTENTION [15] have relatively lower FLOP utilization.

Recent work has made progress towards attention-free models by replacing attention layers with long convolution layers, interleaved with elementwise gating [27, 28, 36, 54, 64, 67-69]. These layers are computed using FFT operations using the FFT convolution theorem: y = K ∗ X = FFT -1 ( FFT ( X ) ∗ FFT ( K )) . While the FFT scales asymptotically well in O ( N log N ) , it is often memory-bound and thus has low FLOP utilization. In our work, we aim to construct a mixer that has both sub-quadratic scaling and high FLOP utilization.

3 MONARCH MIXER

In this section, we recall Monarch matrices, introduce how M2 uses Monarch matrices to mix along the sequence and model dimensions, and benchmark a M2 convolution in terms of hardware utilization.

3.1 Monarch Matrices

Monarch matrices [14] are a sub-quadratic class of structured matrices that are hardware-efficient and expressive. They can represent many linear transforms, including convolutions, Toeplitz-like transforms, low-displacement rank transforms, and orthogonal polynomials. Directly implementing these different structured transforms on GPUs as dense matrices can be inefficient. In contrast, their Monarch decompositions can be computed by interleaving matrix multiplications with tensor permutations.

A Monarch matrix M ∈ R N × N of order p is defined by the following:

where each P i is related to the 'base p √ N ' variant of the bit-reversal permutation, and B i is a block-diagonal matrix with block size b . Setting b = p √ N achieves sub-quadratic compute cost. For example, for p = 2 , b = √ N , Monarch matrices require O ( N 3 / 2 ) compute in sequence length N .

In this paper, we use Monarch matrices to construct architectures that are sub-quadratic in both sequence length N and model dimension d . We will often parameterize order2 Monarch matrices, written as M = PLPRP , where L and R are block-diagonal matrices (for 'left' and 'right'), and P = P 2 = P 1 = P 0 is a permutation that reshapes the input to 2D, transposes it, and flattens it to 1D. A common case is to set L = R = ( I √ N ⊗ F √ N ) , where F √ N is a √ N DFT matrix, and ⊗ is the Kronecker product.

3.2 MONARCH MIXER Architecture

We describe how MONARCH MIXER uses Monarch matrices and elementwise operations to construct sub-quadratic architectures (Figure 1 middle). We take a mixer view of model architectures, where each layer is a sequence of mixing operations across the sequence and the model dimension axes. Each layer takes as input a sequence of embeddings X ∈ R N × d , and outputs a sequence Y ∈ R N × d , where N is the sequence length, and d is the model dimension. For simplicity, we show the order-2 case here, though we can use higher-order blocks to scale to longer sequences and larger model dimensions.

Let M 1 , M 2 ∈ R N × N and M 3 , M 4 ∈ R d × d be order-2 Monarch matrices, let K 1 ∈ R N × d , let σ be an optional point-wise non-linearity ( e.g. ReLU), and let glyph[circledot] be elementwise multiplication. M2

uses Monarch matrices to construct expressive architectures. For example, a convolutional block with a sparse MLP can be expressed as follows:

1. Mix along sequence axis:

2. Mix along embedding axis:

When M 1 is set to the DFT and M 2 is set to the inverse DFT, Equation 2 exactly corresponds to a convolution with kernel K 1 parameterized in frequency space. Equation 3 corresponds to an MLP with the dense matrices replaced by Monarch matrices. More expressive layers are also easily expressible; for example, replacing Equation 2 with V glyph[circledot] M 2 ( K 1 glyph[circledot] M 1 ( Q glyph[circledot] K )) , where Q , K , V are linear projections of X , reproduces a gated convolution block, as in [27, 28, 64].

The basic M2 layer is simple to implement; pseudocode is shown in Figure 1 (middle), and the Appendix gives an efficient implementation of M2 in under 40 lines of pure PyTorch (including imports). The convolution case with Monarch matrices fixed to DFT and inverse DFT matrices also admits implementations based on FFT algorithms [11].

3.3 Architecture Benchmarks

We benchmark the efficiency of the M ( K glyph[circledot] MX ) convolution operator (Equation 2) implemented in a simple CUDA kernel (calling standard cuBLAS sub-routines [61]), as the dimension N increases. Equation 3 scales similarly, as dimension d increases. We keep the block size b fixed to √ N .

Table 2 shows the FLOP cost and utilization of a M2 operator as a function of the input size on an A100 as well as on an RTX 4090. On the A100, the operator is more dominated by the data movement costs of the permutation operations (see the Appendix for a roofline analysis). For longer inputs, the sub-quadratic scaling allows MONARCH MIXER to outperform dense matrix multiplication. On the RTX 4090, which has a larger and faster L2 cache than the A100, we can manually optimize an implementation to amortize data movement costs.

4 Theoretical Analysis: M2 as Polynomial Multiplication

In this section, we develop theory to make the M2 layer causal in the input X -e.g., ensure that an output Y i of the M2 should only depend on X 1 , ..., X i . Our approach involves interpreting Monarch matrix multiplication as multivariate polynomial evaluation and interpolation. We then show that an M2 convolution is equivalent to modular polynomial manipulation in a univariate basis.

The challenge is controlling the degrees of the resulting univariate polynomials, to prevent 'underflow' under modular multiplication (see Figure 2 for an overview). Our key result is deriving sufficient conditions on the degrees of the bivariate polynomials defining the Monarch factors to prevent such underflow. We focus on the bivariate case (order p = 2) in the body, and give the general multivariate case in the Appendix. We present proof sketches in the main body, and leave proofs and additional results for the Appendix.

P

P

P

M -1 (Mu    Mk)

Causal

Parameterization

r(Y)

Conditions on Bivariate

Polynomial Degrees

Univariate Multiplication

deg(f), deg(g) < N / 2

f(Z) g(Z) mod Z N

{

{

{

Monarch Multiplication as Polynomial Evaluation First, we show that order-2 Monarch matrixvector multiplication M · u is equivalent to bivariate polynomial evaluation.

Fix a Monarch matrix M ∈ R N × N = PLPRP , for two block-diagonal matrices L and R with blocks of size b = √ N . We can interpret Monarch matrices as bivariate polynomial evaluation by setting A = { ω 0 , . . . , ω b -1 } as a set of evaluation points (e.g., the b th roots of unity), and letting { glyph[lscript] 0 ( X,Y ) , . . . , glyph[lscript] b -1 ( X,Y ) } , { r 0 ( Y ) , . . . , r N -1 ( Y ) } be sets of basis polynomials with individual degrees of X,Y being < √ N . The values of { glyph[lscript] 0 ( X,Y ) , . . . , glyph[lscript] b -1 ( X,Y ) } evaluated on A 2 determine the entries of L , and the values of { r 0 ( Y ) , . . . , r N -1 ( Y ) } evaluated on A determine the entries of R . We give the mapping from glyph[lscript], r, and A to L and R in the Appendix.

Then, matrix-vector multiplication between M and a vector u is equivalent to polynomial evaluation of the basis functions glyph[lscript], r on the evaluation points A 2 :

Theorem 1. Let m ( j ) = j mod √ N . For any vector u ∈ R N , Mu is a bivariate polynomial u ( X,Y ) evaluated at A 2 , with u ( X,Y ) = ∑ N -1 j =0 u j f j ( X,Y ) , where f j ( X,Y ) = glyph[lscript] m ( j ) ( X,Y ) r j ( Y ) .

Monarch Inverse as Polynomial Interpolation Next, we exploit the fact that Monarch inverse multiplication M -1 · u is equivalent to polynomial interpolation in the basis polynomials of M . √

Theorem 2. Let M 0 , M 1 , M 2 be Monarch matrices, and let A be the set of N roots of unity. Then, the operation is equivalent to representing the polynomial

in terms of the basis polynomials glyph[lscript], r corresponding to M 0 , and where k ( X,Y ) and u ( X,Y ) are the polynomials corresponding to M 1 k and M 2 u , respectively.

The above follows from Theorem 1 and the fact that Monarch matrix-vector multiplication with an inverse Monarch matrix is equivalent to polynomial interpolation in a given basis. The mod part comes from the fact that A is the set of roots of the polynomial Z √ N -1 .

Causal Monarch Maps Now, we give a class of Monarch matrices from which we can build a causal map. First, we define a polynomial with minimum degree j :

Definition 1. A polynomial of minimum degree j (and maximum degree N -1 ) is defined as ¯ q j ( Z ) = ∑ N -1 a = j ¯ q j [ a ] Z a .

To ensure causality, we first convert the bivariate polynomial basis into a univariate basis, and then we expand the degree of the univariate polynomial. The resulting univariate polynomial multiplication is naturally causal (exploiting similar properties as the causal FFT convolution).

We use the Kronecker substitution ( X ← Z, Y ← Z √ N ) to convert the bivariate polynomial basis into a univariate basis: √

where m ( j ) is defined as in Theorem 1.

Then, the following class of Monarch matrices (with the conversion to univariate polynomial basis as above) forms a causal map:

√ .

Theorem 3. Let u , k ∈ R n , where n < N/ 2 . Let m ( j ) be as in Theorem 1, and k ( j ) = ⌊ j/ N ⌋ Then define the basis polynomials glyph[lscript] m ( j ) to have minimum degree m ( j ) , basis polynomials r j to have minimum degree k ( j ) , and all polynomials q j ( Z ) to have maximum degree < N/ 2 for all j < N/ 2 and for N/ 2 ≤ j < N have maximum degree N -1 . Let M N be defined by such basis polynomials via (5) where the evaluation points are now the N th roots of unity. Then, we have that

gives a causal map in u .

Theorem 3 gives a causal map that can be computed entirely using Monarch matrices - enforcing causality with sub-quadratic scaling. The main technical ingredient in proving the above result is that the product q j ( Z ) q j ′ ( Z ) can be written as a linear combination of q a ( Z ) for j + j ′ ≤ a < N (this uses the above specified properties on the minimum and maximum degrees of q j ( Z ) ). This in turn implies that the term k j ′ u j q j ( Z ) q j ′ ( Z ) only contributes to the coefficients of 'higher order' basis polynomials q a ( Z ) for a ≥ j + j ′ in the product k ( Z ) u ( Z ) , which is needed for causality. Figure 2 gives an example of restricted polynomials generating a causal map.

5 Experiments

We compare MONARCH MIXER to Transformers on three tasks where Transformers have been dominant: BERTstyle non-causal masked language modeling, ViT-style image classification, and GPT-style causal language modeling. In each, we show that we can match Transformers in quality using neither attention nor MLPs. We additionally evaluate wall-clock speedups against strong Transformer baselines in the BERT setting. Additional experiments on speech and alternative architectures are given in Appendix B, and experimental details are given in Appendix C.

5.1 Non-Causal Language Modeling

We introduce M2-BERT, an M2-based architecture for non-causal language modeling. M2-BERT acts as a dropin replacement for BERT-style language models [21], which are a workhorse application of the Transformer architecture [1, 39, 40, 45, 48, 49, 52, 56, 85, 89]. We train M2-BERT using masked language modeling over C4 [66] with the bert-base-uncased tokenizer.

x

q

k

v

short

conv

short

conv

short

conv

Monarch

long conv

Monarch

long conv

y

Sequence

Mixer

Dimension

Mixer

x

GeLU

y

GLU

M2-BERT starts with a Transformer backbone and replaces the attention and MLPs with M2 layers, shown in Figure 3. In the sequence mixer, we replace attention with bidirectional gated convolutions with a residual convolution (Figure 3 left). To recover convolutions, we set the Monarch matrices to DFT and inverse DFT matrices. Following [27, 64], we also add short depthwise convolutions after the projections. In the dimension mixer, we replace the two dense matrices in MLPs with learned block-diagonal matrices (Monarch matrix of order 1, b = 4 ). We pretrain two M2-BERT-base models, at 80M and 110M, and two M2-BERT-large models, at 260M and 341M. These are equivalent to BERT-base and BERT-large, respectively.

Downstream GLUE Scores First, we evaluate M2-BERT models on downstream fine-tuning compared to BERT-base and BERT-large from [20]. We take the pretrained models and fine-tune them on BERT, following the procedure in [38]. Table 3 shows performance for BERT-base equivalent models, and Table 4 shows performance for BERT-large equivalent models. M2-BERT-base can

match BERT-base in GLUE quality with 27% fewer parameters-or outperform BERT-base in quality by 1.3 points when parameter matched. M2-BERT-large matches BERT-large with 24% fewer parameters, and outperforms by 0.7 points when parameter matched.

GPU Throughput by Sequence Length Next, we evaluate throughput of M2-BERT models by sequence length, compared to HuggingFace implementations of BERT, as well as optimized implementations of BERT running FlashAttention [15]. Table 5 shows forward throughput for BERTbase equivalent models, and the appendix shows throughput for BERT-large (where the performance trends are similar). Inference times are reported in tokens/ms on an A100-40GB GPU. M2-BERTbase achieves higher throughput than even highly-optimized BERT models, and up to 9.1 × faster throughput than a standard HuggingFace implementation at sequence length 4K.

CPU Inference Latency Finally, we report CPU inference latency for M2-BERT-base (80M) compared to BERT-base, running direct PyTorch implementations for both. In short sequences, the impacts of data locality still dominate the FLOP reduction, and operations such as filter generation (which are not present in BERT) pay a higher cost. Starting at sequences 1K and longer, M2-BERTbase starts to have speedup over BERT-base, up to 6.5 × at sequence length 8K. We believe further optimization and applying IO-aware principles can further improve CPU performance.

5.2 Image Classification

To validate that our methods generalize to images as well as language for non-causal modeling, we next evaluate M2 on image classification. We compare M2 to ViT-style models and recent work, HyenaViT-b [64], which uses gated long convolutions to replace the attention layers in ViT-b. In our work, M2-ViT builds off HyenaViT-b and replaces the long convolutions with the M2 operator in Equation 2 (again setting the Monarch matrices to the DFT and inverse DFT). We replace the MLP blocks in HyenaViT-b with block-diagonal matrices, similarly to M2-BERT. Appendix B additionally compares M2 to the Swin-family of architectures [50, 51].

Table 7 shows the performance of MONARCH MIXER against ViT-b, HyenaViT-b, and ViT-bMonarch (which replaces the MLP blocks of standard ViT-b with Monarch matrices) on ImageNet-1k. MONARCH MIXER outperforms the other models with only half the parameters of the original ViT-s model. Surprisingly, MONARCH MIXER also outperforms ResNet-152, with fewer parameters-even though the latter was explicitly designed for ImageNet performance.

5.3 Causal Language Modeling

GPT-style causal language modeling is a critical application for Transformers [6, 31, 43]. We introduce M2-GPT, a M2-based architecture for causal language modeling. For the sequence mixer, M2-GPT combines the convolutional filter from Hyena [64], the state-of-the-art attention-free

language model, with parameter sharing across multiple heads from H3 [27]. We use the causal parameterization of Equation 2 to replace the FFT in these architectures, and we remove the MLP layers entirely. The resulting architecture is entirely attention- and MLP-free.

We pretrain M2-GPT on the PILE, a standard dataset for causal language modeling. Following prior work [28, 64], we train models at two model sizes, with varying amounts of training data-decaying the learning rate appropriately for each experiment. Table 8 shows the results. Even though our model is attention- and MLP-free, it outperforms both Transformers and Hyena in perplexity on pretraining. These results suggest that radically different architectures than Transformers may be performant on causal language modeling.

6 Related Work

Long Convolutions Recent work proposes to use long convolution layers as a replacement for the Transformer attention layers in sequence modeling [28, 64, 67-69]. Many of these models rely on the FFT convolution theorem to compute the long convolutions. We build on the insights in many of these architectures in constructing our M2 architectures, and additionally replaces the FFT operations with Monarch matrices.

Our work is also related to a rich literature in convolutions in other bases, such as Chebyshev bases [82] or orthogonal polynomial bases [34]. These approaches have analogues in our multivariate analysis; replacing the basis polynomials of the Monarch matrices in MONARCH MIXER may be able to approximate some of these operations. An interesting question for future work would be to study how well our techniques and concerns about causality and hardware utilization translate to these alternative convolution bases.

Optimization of deep learning primitives There is a rich history of the optimization of deep learning primitives, as accelerating their performance can yield substantial savings in compute and cost for large models. There are many approaches to speed up these operations, but they usually either reduce data movement or compute.

Reducing data movement: In many applications, the major bottleneck is the storage and movement of large amounts of memory. One popular approach to reducing data movement is checkpointing, wherein one stores fewer intermediate results and recomputes the others on-the-fly where they are needed, trading additional compute for memory [46, 78]. Another approach is kernel fusion, wherein algorithms initially described as sequential steps can often be fused in ways that improve their properties. For example, it is generally faster to implement a dot-product through a multiply-

accumulate rather than first multiplying and then accumulating. Recently, libraries such as PyTorch 2.0 [63] have added kernel fusion capabilities, although the very best performance usually still arises from handwritten kernels. Third, in order to better exploit memory locality, it is often fastest to load small blocks of memory, do intensive computation on them, and then write the results a tile at a time [83]. Finally, many algorithms also have hand-optimizations that can remove unnecessary computation or memory accesses [55].

Efficient algorithms usually make use of a combination of these techniques. For example, FlashAttention [15] uses all four to dramatically decrease both the latency and memory consumption of multi-head attention. Though we have made a modest effort to implement MONARCH MIXER efficiently, we think it likely that MONARCH MIXER could be further optimized by these techniques.

Reducing flops: A first target for optimization is the multi-layer perceptron (MLP), owing to its ubiquity. A variety of structured sparse factorizations exist, many of which we draw on in this work [7, 11, 14, 16, 17, 19, 26, 91]. Attention is also a popular target for optimization. Recently, a plethora of sub-quadratic approximations of attention have emerged, that aim to approximate attention to reduce its quadratic complexity. Some methods rely on sparsification, relying on the fact that the attention matrix is extremely sparse at long sequence lengths [2, 23, 24, 42, 53]. Others use low-rank approximations of the attention matrix [13, 79, 91] or kernel methods instead [9, 41]. A subset use a combination of these techniques, such as [8, 72]. Finally, a third category of methods [27, 64] aim to replace attention entirely, relying on state-space models [33].

7 Discussion and Conclusion

We explore MONARCH MIXER (M2), a new architecture that is sub-quadratic in both sequence length and model dimension and is hardware-efficient on modern accelerators. We motivate M2 from both theoretical and systems performance perspectives and conduct a preliminary proof-ofconcept investigation into performance on masked language modeling, image classification, and causal language modeling.

While our initial results are promising, our work is only a first step in this direction. The M2 layer can likely be further optimized with systems optimization techniques such as kernel fusion. Our work has also not been optimized for inference like more well-established models such as Transformers, or even more recent models such as state space models. It also remains to be seen whether M2 layers can have as widespread applicability as Transformers. We hope that these can be fruitful directions for future work.

A discussion of broader impacts can be found in the Appendix.

Acknowledgments

We gratefully acknowledge the support of DARPA under Nos. FA86501827865 (SDH) and FA86501827882 (ASED); NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); ONR under No. N000141712266 (Unifying Weak Supervision); the Moore Foundation, NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, the Okawa Foundation, American Family Insurance, Google Cloud, Microsoft Azure, Swiss Re, Brown Institute for Media Innovation, Department of Defense (DoD) through the National Defense Science and Engineering Graduate Fellowship (NDSEG) Program, Fannie and John Hertz Foundation, National Science Foundation Graduate Research Fellowship Program, Texas Instruments Stanford Graduate Fellowship in Science and Engineering, and members of the Stanford DAWN project: Teradata, Facebook, Google, Ant Financial, NEC, VMWare, and Infosys. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of DARPA, NIH, ONR, or the U.S. Government. JG and AR's work is supported by NSF grant# CCF-2247014. IJ's work is supported by an NSF Graduate Fellowship.

References

Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 , 2019.

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020.

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery.

Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet1k. arXiv preprint arXiv:2205.01580 , 2022.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher R´ e. Pixelated butterfly: Simple and efficient sparse training for neural network models. 2021.

Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher R´ e. Scatterbrain: Unifying sparse and low-rank attention. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 , 2020.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

James W Cooley and John W Tukey. An algorithm for the machine calculation of complex fourier series. Mathematics of computation , 19(90):297-301, 1965.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 702-703, 2020.

Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. Advances in neural information processing systems , 33:4271-4282, 2020.

Tri Dao, Beidi Chen, Nimit S Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher R´ e. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning . PMLR, 2022.

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R´ e. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems , 2022.

Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, and Christopher R´ e. Learning fast algorithms for linear transforms using butterfly factorizations, 2020.

Tri Dao, Nimit S. Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, and Christopher R´ e. Kaleidoscope: An efficient, learnable representation for all structured linear maps, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840 , 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In arXiv:1810.04805 , 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning , pages 5547-5569. PMLR, 2022.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research , 23(1):5232-5270, 2022.

Wikimedia Foundation. Wikimedia downloads.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 , 2018.

Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher R´ e. Hungry hungry hippos: Towards language modeling with state space models. International Conference on Learning Representations , 2023.

Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher R´ e. Simple hardware-efficient long convolutions for sequence modeling. International Conference on Machine Learning , 2023.

Morgan Funtowicz. Scaling up bert-like model inference on modern cpu - part 1, 2021.

Jonas Geiping and Tom Goldstein. Cramming: Training a language model on a single gpu in one day. arXiv:2212.14034v1 , 2022.

Google. Bard, https://bard.google.com/ . 2023.

Robert M Gray et al. Toeplitz and circulant matrices: A review. Foundations and Trends® in Communications and Information Theory , 2(3):155-239, 2006.

Albert Gu, Karan Goel, and Christopher R´ e. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 , 2021.

Nicholas Hale and Alex Townsend. An algorithm for the convolution of legendre series. SIAM Journal on Scientific Computing , 36(3):A1207-A1220, 2014.

Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 , 2015.

Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. arXiv preprint arXiv:2209.12951 , 2022.

Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781 , 2019.

Peter Izsak, Moshe Berchansky, and Omer Levy. How to train bert with an academic budget. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 10644-10652, 2021.

Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 8018-8025, 2020.

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics , 8:64-77, 2020.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning , pages 5156-5165. PMLR, 2020.

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020.

Jan Koco´ n, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. Chatgpt: Jack of all trades, master of none. arXiv preprint arXiv:2302.10724 , 2023.

Elias Konstantinidis and Yiannis Cotronis. A practical performance model for compute and memory bound gpu kernels. In 2015 23rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing , pages 651-658. IEEE, 2015.

MVKoroteev. Bert: a review of applications in natural language processing and understanding. arXiv preprint arXiv:2103.11943 , 2021.

Mitsuru Kusumoto, Takuya Inoue, Gentaro Watanabe, Takuya Akiba, and Masanori Koyama. A graph theoretic framework of recomputation algorithms for memory-efficient backpropagation. Advances in Neural Information Processing Systems , 32, 2019.

Lagrange polynomial. Lagrange polynomial - Wikipedia, the free encyclopedia, 2005. https: //en.wikipedia.org/wiki/Lagrange_polynomial .

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics , 36(4):1234-1240, 2020.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In International Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Xiaofei Ma, Zhiguo Wang, Patrick Ng, Ramesh Nallapati, and Bing Xiang. Universal text representation from bert: An empirical study. arXiv preprint arXiv:1910.07973 , 2019.

Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. Luna: Linear unified nested attention. Advances in Neural Information Processing Systems , 34:2441-2453, 2021.

Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: moving average equipped gated attention. arXiv preprint arXiv:2209.10655 , 2022.

Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867 , 2018.

Derek Miller. Leveraging bert for extractive text summarization on lectures. arXiv preprint arXiv:1906.04165 , 2019.

Marcin Moczulski, Misha Denil, Jeremy Appleyard, and Nando de Freitas. Acdc: A structured efficient linear layer. arXiv preprint arXiv:1511.05946 , 2015.

NVIDIA. Nvidia Tesla V100 GPU architecture, 2017.

NVIDIA. Nvidia A100 tensor core GPU architecture, 2020.

NVIDIA. Nvidia H100 tensor core GPU architecture, 2022.

NVIDIA. cuBLAS, 2023.

OpenAI. Gpt-4 technical report, 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher R´ e. Hyena hierarchy: Towards larger convolutional language models. International Conference on Machine Learning , 2023.

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 , 2019.

David W Romero, R Bruintjes, Erik J Bekkers, Jakub M Tomczak, Mark Hoogendoorn, and JC van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. In 10th International Conference on Learning Representations , 2022.

David W Romero, David M Knigge, Albert Gu, Erik J Bekkers, Efstratios Gavves, Jakub M Tomczak, and Mark Hoogendoorn. Towards a general purpose cnn for long range dependencies in { N } d. arXiv preprint arXiv:2206.03398 , 2022.

David W Romero, Anna Kuzina, Erik J Bekkers, Jakub Mikolaj Tomczak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. In International Conference on Learning Representations , 2021.

Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270 , 2021.

G. Szeg¨ o. Orthogonal Polynomials . Number v.23 in American Mathematical Society colloquium publications. American Mathematical Society, 1967.

Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys , 55(6):1-28, 2022.

Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlpmixer: An all-mlp architecture for vision. Advances in neural information processing systems , 34:24261-24272, 2021.

Asher Trockman and J Zico Kolter. Patches are all you need? Transactions on Machine Learning Research , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. volume 30, 2017.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv:1804.07461 , 2018.

Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander M Rush. Pretraining without attention. arXiv preprint arXiv:2212.10544 , 2022.

Qipeng Wang, Mengwei Xu, Chao Jin, Xinran Dong, Jinliang Yuan, Xin Jin, Gang Huang, Yunxin Liu, and Xuanzhe Liu. Melon: Breaking the memory wall for resource-efficient ondevice machine learning. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services , pages 450-463, 2022.

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020.

Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 , 2021.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-theart natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 2020.

Kuan Xu and Ana F. Loureiro. Spectral approximation of convolution operators. SIAM Journal on Scientific Computing , 40(4):A2336-A2355, 2018.

Yufan Xu, Saurabh Raje, Atanas Rountev, Gerald Sabin, Aravind Sukumaran-Rajam, and P Sadayappan. Training of deep learning pipelines on memory-constrained gpus via segmented fused-tiled execution. In Proceedings of the 31st ACM SIGPLAN International Conference on Compiler Construction , pages 104-116, 2022.

Lili Yu, D´ aniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers, 2023.

Shanshan Yu, Jindian Su, and Da Luo. Improving bert-based text classification with auxiliary sentence and domain knowledge. IEEE Access , 7:176600-176612, 2019.

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision , pages 558-567, 2021.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023-6032, 2019.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 , 2019.

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 13001-13008, 2020.

Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. Advances in Neural Information Processing Systems , 34:17723-17736, 2021.

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV) , December 2015.

Author Contributions

D.Y.F. Conceptualized the research; coordinated collaborations; developed M2 architectures; led experimental and implementation efforts; assisted in development of theoretical results; coordinated writing.

S.A. Worked on developing M2 architectures; worked on implementing and conducting BERT experiments; worked on implementing and performing CPU experiments; assisted in writing and framing the work.

J.G. Led development of theory and causal algorithms; wrote Appendix D.

I.J. Led development of theory and causal algorithms; wrote Appendix D.

S.E. Worked on writing and framing the work; assisted in development of M2 architectures; assisted in the optimized M2 implementation; conducted mixer benchmarks; assisted with BERT experiments; conducted Swin ImageNet experiments.

A.W.T.

Conducted ViT experiments; assisted in writing.

B.S. Assisted in optimized M2 implementation; conducted mixer benchmarks; assisted in writing.

M.P. Assisted in development of M2-GPT architecture.

A.R. Supervised theory development; developed proofs; reviewed manuscript.

C.R. Supervised research; reviewed manuscript.

Simran Arora, Jessica Grogan, Isys Johnson, Sabri Eyuboglu, and Armin Thomas contributed equally to this work.

Appendix

Appendix A discusses broader impacts of our work. Appendix B presents additional experiments. Appendix C gives details for the experiments, including model architectures and hyperparameters. Appendix D gives missing details and proofs for the theoretical analysis, as well as generalizations to broader results. Appendix E gives a PyTorch code listing of an M2 layer.

A Broader Impacts

Our work seeks to understand the fundamental capabilities and limitations of newly-emerging model architectures. As the amount of data and model size grows, we also seek to understand how to make training these models more efficient, both in terms of the amount of training context and the model size. This potentially connects to energy savings during model development and deployment, as well as making machine learning models accessible to a larger population of people.

However, as with any machine learning models, developing new techniques may impact a wide range of applications, each with potential benefits and harms. Making language model training cheaper and longer context may make it cheaper to spread disinformation. Similarly, improving the efficiency of model training may not reduce the overall environmental footprint of training, since the same resources may be used to train more models, or train the same models for longer. While our work makes partial progress on the fronts of efficiency and understanding, it does not explicitly address the issues of fairness and bias in language models. In addition, our work demonstrates a proof-of-concept; it has error modes, and we recognize the inherent risks of training and using machine learning models, including language models. Detailed discussions of these risks are in [3, 5, 80].

B Additional Experiments

B.1 Per-Task GLUE Numbers

We report full GLUE numbers for M2-BERT-base and M2-BERT-large in Table 9.

B.2 Additional Throughput Results

We report the throughput of M2-BERT-base (80M) compared to BERT models of the same size (BERT-base with fewer parameters), as well as the throughput of M2-BERT-large (260M) compared to BERT-large.

Table 10 compares the performance of M2-BERT-base (80M) to BERT models parameter-matched to 80M parameters. M2 is slower than FlashAttention for sequence lengths 512 and 1K, but outperforms FlashAttention starting at sequence length 2K. We believe further optimization of the M2 kernel can close the gap to FlashAttention for short sequences.

Table 11 compares M2-BERT-large (260M) to BERT-large. Trends are mostly similar to comparisons against BERT-base; M2 nearly matches FlashAttention at sequence length 512, and outperforms it for sequence length 1K and longer. We also see up to 4.3 × speedup over HuggingFace BERT-large at sequence length 2K.

B.3 ImageNet Comparison against Swin

Table 12 reports the results of replacing attention and MLP in Swin-V2 using M2 as a drop-in replacement. Surprisingly, Swin-M2 outperforms Swin-MLP-B, is competitive with Swin-V1-B, and comes within 1 point of Swin-V2-B, even without any hyperparameter tuning or architecture adjustment from the ViT formula. We expect that performance may improve further with hyperparameter tuning specific to M2.

B.4 Speech Applications

Table 13 presents the performance of M2 on Speech Commands-10, a speech classification task over raw 1-second clips sampled at 16 kHz. M2 is competitive with state-of-the-art architectures on this task.

B.5 CIFAR10

Table 14 shows the performance of MONARCH MIXER on CIFAR10. The trends are largely the same as on ImageNet.

B.6 Learnable Monarch Matrices in Sequence Mixer

In most of our models, we have used fixed Monarch matrices for the sequence mixer, and learnable Monarch matrices for the dimension mixer. Table 15 presents an experiment evaluating using learnable Monarch matrices for the sequence mixer on the sequential CIFAR task. We use a nongated convolutional architecture based off long convolutions, as presented in [28]. Learning the Monarch matrices in the sequence mixer yields 1.5 points of lift.

B.7 Roofline Analysis

Figure 4 shows a Roofline analysis of a simple PyTorch implementation of a single M2 operator M -1 ( Mu glyph[circledot] Mk on an A100 GPU, with 4K input length. The operation is more dominated by the data movement operations, which helps explain why performance is higher on newer architectures like RTX 4090 (which have faster and larger L2 cache).

B.8 Associative Recall

In Table 16, we present a simple experiment demonstrating the causal parameterization of M2 on associative recall, a synthetic language designed to test in-context learning. The model demonstrates in-context learning abilities in sequences up to 128K tokens, but Transformers do not scale past 8K.

B.9 BERT Experiments with Alternative Architecture

Here, we report results using an older version of the M2-BERT architecture, that uses non-gated convolutions and is trained on English Wikipedia [25] and English Bookcorpus [92]. For clarity, we refer to this model as M1-BERT.

We found that M1-BERT could match Transformers on MLM quality, but underperformed on downstream fine-tuning. We attribute this gap in performance to sub-optimal training hyperparameters (optimized for throughput using NVIDIA MLPerf hyperparameters) as well as a sub-optimal architecture. We report results here for completeness, but refer to the gated convolution architecture in the main body as the proper M2-BERT model.

These models followed the reference implementations and hyperparameters from Hugging Face Transformers examples [81] and Nvidia Deep Learning examples ( https://github.com/NVIDIA/ DeepLearningExamples ). In particular, we use the LAMB optimizer with a learning rate of 5 e -3 . For each sequence length, we use as large a minibatch size as possible that fits on the GPU (A10080GB in Table 17 and V100 in Table 18). We set the gradient accumulation to reach a global batch size of 65 , 536 sequences. To investigate the effect of sequence length, each model is trained for a fixed sequence length in a single phase of training (in contrast to some training protocols, which train the model in multiple phases, each at different sequence lengths).

Time to a Fixed Pretraining Quality on 8xA100 We compare time to a fixed pretraining quality, training M1-BERT-base on English Wikipedia [25] and English Bookcorpus [92]. We compare against BERT-base trained with FLASHATTENTION [15], as well as the Monarch-BERT-base implementation from the original Monarch paper [14]. We measure wall-clock time for M1-BERT and the base Transformer to reach 50% in masked language modeling accuracy on 8xA100 Nvidia GPUs with 80GB memory each. Table 17 summarizes results. In short sequence lengths, M1-BERT is

comparable to FLASHATTENTION, even without using a heavily-optimized fused kernel. In longer sequence lengths, the FLOP savings make M1-BERT more efficient-up to 2.4 × faster than BERT with FLASHATTENTION at sequence length 4096 .

BERTinHalf a Day Inspired by recent work focusing on training under limited resource constraints [30], we measure how far we can get when training on a single V100 GPU in 12 hours. In Table 18, we report the masked language modeling accuracy achieved by the same set of models and sequence lengths (except for the FLASHATTENTION baseline, which is not supported on V100). We observe M1-BERT both achieves higher accuracy within the time limit and can be trained at longer sequence lengths than the baseline architectures.

Downstream Fine-Tuning We evaluate the quality of M1-BERT-base models on the GLUE benchmark [76]. Table 19 shows fine-tuning performance on the GLUE tasks, using the same hyperparameters and 5 epochs for all tasks and both models. M1-BERT-base is competitive with Transformers trained using MLPerf hyperparameters on Bookcorpus and Wikitext, but underperforms fully-trained transformers and M2-BERT-base.

C Experiment Details

C.1 Model Architectures

In this section, we describe the exact model architectures we used for each task, including the design of the block (residuals and gating). We additionally release our code for reproducibility,

BERT Language Modeling The M2-BERT architectures use a standard BERT backbone, but replace the attention with bidirectional gated convolutions and replace the linear layers in the MLPs with block-diagonal matrices. All the M2-BERT architectures use an expansion factor of four. M2-BERT-base (80M) has a model width of 768 and 12 layers; M2-BERT-base (110M) has a model width of 960 and 12 layers; M2-BERT-large (260M) has a model width of 1536 and 12 layers; and M2-BERT-large (341M) has a model width of 1792 and 12 layers. We train all these models on C4 for 70,000 steps, with sequence length 128, and global batch size 4096 sequences. For all the models, we use decoupled AdamW with learning rate 8e-4 and decoupled weight decay 1e-5. We use linear learning rate decay with a warmup of 6% of the steps, and we use MLM masking percentage of 30%.

For GLUE fine-tuning, we do a small search of learning rate, weight decay, and number of epochs. Following [38], we fine-tune RTE, MRPC, and STS-B from the MNLI checkpoint. We fine-tune all tasks with sequence length 128. For some tasks, we also pool the embeddings of all the non-padding tokens instead of using the CLS token.

The final hyperparameters for M2-BERT-base (80M) are decoupled AdamW with learning rate 5e-5 and weight decay 5e-6 for 3 epochs for MNLI; AdamW with learning rate 5e-5 and weight decay 0.01 for 6 epochs for RTE; AdamW with learning rate 3e-5 and weight decay 0.01 for 10 epochs on QQP; AdamW with learning rate 5e-5 and weight decay 1e-5 for 10 epochs with average pooling for QNLI; decoupled AdamW with learning rate 3e-5 and weight decay 3ed-6 for 3 epochs for SST-2; AdamW

M2PyTorchperformancevsRoofline (A100 GPU withFP16TensorCores)

A100Roofline

M2 PyTorch Implementation

1014

Performance (FLOPs/s)

1013

1012

100

101

102

103

104

Arithmetic Intensity (FLOPs/Byte)

with learning rate 7e-5 and weight decay 0.01 for 10 epochs for STS-B; AdamW with learning rate 5e-5 and weight decay 0.01 for 10 epochs for MRPC; and decoupled AdamW with learning rate 5e-5 and weight decay 5e-6 for 10 epochs for COLA.

For M2-BERT-base (110M), the hyperparameters are decoupled AdamW with learning rate 5e-5 and weight decay 5e-6 for 3 epochs for MNLI; decoupled AdamW with learning rate 1e-5 and weight decay 1e-6 for 3 epochs for RTE; decoupled AdamW with learning rate 3e-5 and weight decay 3e-6 for 5 epochs on QQP; decoupled AdamW with learning rate 5e-5 and weight decay 1e-5 for 10 epochs with average pooling for QNLI; decoupled AdamW with learning rate 3e-5 and weight decay 3ed-6 for 3 epochs for SST-2; decoupled AdamW with learning rate 8e-5 and weight decay 3e-6 for 10 epochs for STS-B; decoupled AdamW with learning rate 8e-5 and weight decay 8e-5 for 10 epochs for MRPC; and AdamW with learning rate 8e-5 and weight decay 5e-6 for 10 epochs for COLA.

For M2-BERT-large (260M), the hyperparameters are decoupled AdamW with learning rate 5e-5 and weight decay 5e-6 for 3 epochs for MNLI; decoupled AdamW with learning rate 1e-5 and weight decay 1e-6 for 3 epochs for RTE; decoupled AdamW with learning rate 3e-5 and weight decay 3e-6 for 5 epochs on QQP; decoupled AdamW with learning rate 5e-5 and weight decay 1e-5 for 10 epochs for QNLI; decoupled AdamW with learning rate 3e-5 and weight decay 3ed-6 for 3 epochs for SST-2; decoupled AdamW with learning rate 7e-5 and weight decay 3e-6 for 10 epochs for STS-B; decoupled AdamW with learning rate 8e-5 and weight decay 8e-6 for 10 epochs for MRPC; and AdamW with learning rate 5e-5 and weight decay 5e-6 for 10 epochs for COLA.

For M2-BERT-large (341M), the hyperparameters are decoupled AdamW with learning rate 5e-5 and weight decay 5e-6 for 3 epochs for MNLI; AdamW with learning rate 5e-5 and weight decay 1e-6 for 2 epochs for RTE; decoupled AdamW with learning rate 3e-5 and weight decay 3e-6 for 5 epochs on QQP; decoupled AdamW with learning rate 5e-5 and weight decay 1e-6 for 10 epochs for QNLI; decoupled AdamW with learning rate 3e-5 and weight decay 3ed-6 for 3 epochs for SST-2; decoupled AdamW with learning rate 8e-5 and weight decay 3e-5 for 8 epochs for STS-B; decoupled AdamW with learning rate 8e-5 and weight decay 8e-6 for 10 epochs for MRPC; and decoupled AdamW with learning rate 5e-5 and weight decay 1e-6 for 10 epochs for COLA.

ViT We use a standard ViT model architecture as base [22]. In line with recent improvements to the ViT architecture [4, 70, 86], we use sinusoidal position embeddings and global average-pooling (GAP) instead of a class token.

We adapt the ViT architecture by replacing its MLP and/or attention components with Monarch Matrices (similar to our adaptation of BERT):

We replace the MLP with randomly initialized Monarch Matrices of the same dimension as the dense matrices of the MLP and learn those matrices during training, setting the number of blocks in the block-diagonal matrices to 4 .

We replace attention with the recently introduced Hyena operator [64]. The Hyena operator represents a recurrence of two efficient sub-quadratic primitives, an implicit long convolution and multiplicative element-wise gating of the projected input. Hyena operators apply the FFT algorithm to achieve fast long convolutions in sub-quadratic time. We further adapt the Hyena operator by replacing its long convolutions with the M2 operator and setting the Monarch Matrices to the DFT and inverse DFT.

ViT for ImageNet-1k In line with other work [4, 14, 64, 70], we use a ViT-base architecture with 12 layers, a hidden size of 768 , 12 attention heads per layer, an intermediate size of the MLP projection of 3 , 072 , and a patch size of 16 × 16 pixels. For optimization, we follow the training procedure of T2T-ViT [86], including augmentations such as RandAugment [12] (magnitude = 9 , magnitude-std = 0 . 5 , layers = 2 ), Mixup [88] ( α = 0 . 8 ), CutMix [87] ( α = 1 . 0 ), Random erasing [90] (rate = 0 . 25 ), and AugMix [37]. See Table 20 for all other training settings.

ViT for CIFAR-10 We use a ViT architecture with 6 layers, a hidden size of 128 , 8 attention heads per layer, an intermediate size of the MLP projection of 512 , and a patch size of 4 × 4 pixels. We further tune weight decay ( 0 or 0 . 1 ), stochastic depth rate ( 0 or 0 . 1 ), and base learning rate ( 1 e -4 or 3 e -4 or 1 e -3 ) and report the test performance for the model variant that achieved the highest accuracy in a separate held-out validation dataset (randomly selected 10% of training data). We also apply an early stopping rule such that training is stopped if the model's validation loss does not improve for 10 training epochs. See Table 20 for all other training settings.

GPT Causal Language Modeling Similarly to our ViT approach, we also replace attention with the Hyena operator, using the same architecture as in [64] as a starting point. The Hyena architecture has two convolutions, which can be computed using the FFT convolution theorem. In our architecture, we additionally replace these FFT operations with causal Monarch matrices.

In addition, we re-use the heads extension from the H3 architecture [27]. The heads extension groups the model dimension into heads, ties together the long convolution parameters in each head, and then computes the outer product between different input projections. An algorithmic listing adapted from the H3 paper [27] is provided in Listing 1, with updates to replace the SSM layers with Hyena convolutions. We use a head dimension of 16. Setting the head dimension to be 1 and replacing the Monarch matrices with FFT is equivalent to the Hyena layer.

Algorithm 1 M2 Hyena Layer with Heads

Input: Input sequence u ∈ R N × d from the previous layer, weight matrices W X 1 , W X 2 , W V , W O ∈ R d × d , causal Monarch matrix M , short convolution kernels K 1 , K 2 , K 3 , a Hyena convolution kernel K long , head dimension d h .

Output:

Output sequence

y

∈

R

N

×

d

Compute X 1 = u W X 1 , X 2 = u W X 2 , V = u W V ∈ R N × d .

Pass X 1 , X 2 , V each through the short convolution using the causal Monarch matrices: X 1 , X 2 , V = M -1 ( MX 1 glyph[circledot] MK 1 ) , M -1 ( MX 2 glyph[circledot] MK 2 ) , M -1 ( MV glyph[circledot] MK 3 ) .

Split X 1 , X 2 , V into H 'heads' ( X 1 ( h ) , X 2 ( h ) , V ( h ) for h = 1 , . . . , H ), each a sequence of N vectors of size d h = d/H .

for 1 ≤ h ≤ H do

Take the batched outer product X 2 ( h ) ( V ( h ) ) glyph[latticetop] ∈ R N × d h × d h (batched in the N -dimension) and pass it through the long convolution using the causal Monarch: XV ( h ) = M -1 ( MX 2 ( h ) ( V ( h ) ) glyph[latticetop] glyph[circledot] MK long ) ∈ R N × d h × d h .

Batch-multiply by X 1 : O ( h ) = [ X 1 ( h ) 1 XV ( h ) 1 , . . . , X 1 ( h ) N XV ( h ) N ] ∈ R N × d h (batched in the N -dimension).

Concatenate the output O ( h ) of each head, and multiply by the output projection matrix W O ∈ R d × d .

Finally, we remove the MLP layers entirely (equivalent to replacing the layer with an identity), and make the model wider to compensate (the depths match the equivalent Hyena models). The small model has a model width of 1160 with 18 layers and uses a learning rate of 0.0006, and the medium model has model width of 1344 with 40 layers and uses a learning rate of 0.0008. All other hyperparameters match the Hyena models [64].

D Missing details from Section 4

This section contains all the missing details (including proofs) from Section 4.

In Appendix D.1, we review some definitions and results on multi-variate polynomials and set some notation needed for this section. In Appendix D.2, we explicitly connect Monarch matrices for p = 2 and bivariate polynomial evaluation. Specifically, we prove Theorem 1 and Theorem 2. Then in Appendix D.3 we show how to instantiate the bivariate basis polynomials so that we get a causal map. This includes converting the bivariate polynomials to univariate polynomials (with evaluations over the N th roots of unity) and this proves Theorem 3. We then show how this causal map can be implemented only using GEMMs (and O ( N 3 / 2 ) FLOPs) in Appendix D.4.

Next, we note that while our evaluations points are over complex numbers, our input and output to the Monarch convolution layers are over reals. Hence, it is natural to wonder if we can implement the entire layer just with operations over real numbers. One potential advantage of this is that we theoretically only have to keep N real numbers for intermediate results (instead of 2 N reals

numbers when we keep track of vectors in C N ). This can reduce the data movement costs. Further, multiplication of two complex numbers requires six operations over real numbers (four multiplication and two addition). Thus, moving to an implementation that only uses real numbers could potentially lead to wall clock time speedup. We propose one such scheme in Appendix D.5 that proves a version of Theorem 3 just over reals by moving to the Chebyshev basis (instead of the standard monomial basis). This creates new technical challenges, which we also address.

Finally, we generalize our results to arbitrary p ≥ 2 in Appendix D.6. We would like to point out that to get a causal map (in Theorem 17) we need to 'embed' input vectors of size n into vectors of size N = 2 p · n + O ( n 1 -1 /p ) . For p = 2 , we avoided the blowup of 2 2 = 4 with a blowup of 2 instead (via Theorem 3). Whether this is possible to do (i.e. have a blowup of 2 instead of 2 p ) for p > 2 is an interesting direction for future work. Further, the matrices that lead to causal map can be represented with O ( pN 2 /p ) parameters while the matrices in Theorem 3 use more parameters. Extending the causal map for p > 2 that uses O ( N 1+ 1 p ) parameters is an exciting direction for future work.

D.1 Background and Notation

We collect known facts and definitions about multi-variate polynomials in Appendix D.1.1 and recall some notation from [14] in Appendix D.1.2. These will be needed throughout this appendix section.

D.1.1 Multi-variate Polynomials

Basic Definitions Let p ≥ 1 be an integer. We recollect some definitions on p -variate polynomials (over R ) in variables X 0 , . . . , X p -1 . When p ∈ { 1 , 2 } , we will use variables in { X,Y,Z } for notational simplicity.

We will use X to denote the vector of variables ( X 0 , . . . , X p -1 ) . Further for j ∈ Z p ≥ 0 , we use the notation

X j is a (standard basis) monomial , where j = ( j 0 , . . . , j p -1 ) .

A generic p -variate polynomial is defined as (with standard monomial representation)

where the coefficient q j ∈ R .

We will need the following notion of degrees:

glyph[negationslash]

Definition 2 (Degree) . Let 0 ≤ a < p . The degree of X a in X j (with j = ( j 0 , . . . , j p -1 ) ) is j a . The degree of X a of q ( X ) , denoted by deg X a ( q ) is the maximum degree of X a over all monomials X j with q j = 0 .

Note that for p = 1 the above coincides with the usual notion of degree of a univariate polynomial q ( Z ) , in which case we just use deg( q ( Z )) to denote deg Z ( q ( Z )) .

We will need the notion of taking mod of a p -variate polynomial with p -tuple of polynomials. The notion of mod is well defined for a univariate polynomial (which we will assume as a given below) but in general for arbitrary p -variate polynomials q ( X ) and q ′ ( X ) , the operation q ( X ) mod q ′ ( X ) is not well defined. However, we will only need the following restricted operation:

Definition 3. Let p ≥ 1 . Fix a p -tuple of polynomials R 0 ( X 0 ) , . . . , R p -1 ( X p -1 ) . Then for any j ∈ Z p ≥ 0 , we define

For a general polynomial p ( X ) ,

is defined by extending the definition for X j by linearity.

Polynomial Evaluation Given a p -variate polynomial q ( X ) and an point a ∈ R p , the evaluation of q at a denoted by q ( a ) is evaluation of q as a function at a .

Given subsets S a ⊆ C , we define q ( X ) evaluated at × p -1 a =0 S a as the vector of values q ( a ) overall a ∈ × p -1 a =0 S a .

In this paper, we will in many cases evaluate polynomials at the appropriate roots of unity. Specifically for an integer N , we will define

and note that the N th roots of unity is the set { ω i N | 0 ≤ i < N } .

Polynomial Interpolation We now recall univariate and bivariate polynomial interpolation results (proved via the Lagrange basis), which we will use in later subsections.

Theorem 4. Let D ≥ 1 be an integer. Given y i for 0 ≤ i < D and α i for 0 ≤ i < D there exists a unique univariate polynomial P ( X ) with deg( P ) < D , such that for all 0 ≤ i < D ,

Proof. This proof is based on the Wikipedia entry for Lagrange polynomials [47].

glyph[negationslash]

Given a sequence of values α i for 0 ≤ i < D s.t. α i = α j , i = j , the Lagrange basis for polynomials of degree < D for these values is the set of each polynomials { p 0 ( X ) , p 1 ( X ) , . . . p D -1 ( X ) } each of degree D -1 . Each basis polynomial are defined as:

glyph[negationslash]

The Lagrange interpolating polynomial for those nodes through the corresponding values y i for 0 ≤ i < D is the linear combination:

glyph[negationslash]

By definition,

By (9), for all 0 ≤ i < D :

Proof. Define and for 0 ≤ j, glyph[lscript] < D Y

From above, we have for all i, j, k, glyph[lscript]

Then, for all 0 ≤ i < D X , 0 ≤ j < D Y :

Finally, the interpolating polynomial is unique. Assume there is another polynomial M ( X ) of degree < D such that M ( α i ) = y i for all 0 ≤ i < D . Then the difference M ( X ) -P ( X ) is 0 at D distinct points α i for 0 ≤ i < D . And the only polynomials of degree < D with more than D -1 roots is the 0 polynomial. So, M ( X ) = P ( X ) .

Theorem 5. Let D X , D Y ≥ 1 be integers. Given values y ij for 0 ≤ i < D X , 0 ≤ j < D Y and D X distinct points ( α 0 , . . . , α D X -1 ) , D Y distinct points ( β 0 , . . . , β D Y -1 ) there exists a unique bivariate polynomial P ( X,Y ) with deg X ( P ) < D X , deg Y ( P ) < D Y , such that for all 0 ≤ i < D X , 0 ≤ j < D Y :

where p i and ¯ p j are Lagrange basis polynomials defined in the proof of Theorem 4 such that for 0 ≤ i, k < D X ,

By definition of Lagrange basis polynomials, deg X ( P ) < D X and deg Y ( P ) < D Y .

Finally, the interpolating polynomial is unique. Assume there is another polynomial M ( X,Y ) with deg X ( M ) < D X and deg Y ( M ) < D Y such that M ( α i , β j ) = y ij for all 0 ≤ i < D X and 0 ≤ j < D Y . Then the difference M ( X,Y ) -P ( X,Y ) is 0 at D X · D Y distinct points, ( α i , β j ) for 0 ≤ i < D X , 0 ≤ j < D Y . And the only polynomial with deg X < D X and deg Y < D Y that has D X · D Y roots is the 0 polynomial.

D.1.2 Notation

Here we recall notation we will use from [14].

The class of Monarch matrices is defined in appendix C of [14] as M ( b,N ) which are N × N matrices with block size b for any integer 0 ≤ b ≤ N that divides N . When b = √ N we drop b from the notation giving ( i 1 , i 0 ) and ( j 1 , j 0 ) . For example, this is used in Proof of Corollary 2.

Row index i can be represented as ( i 1 , i 0 ) b . Which gives i = i 1 b + i 0 .

Similarly, column index j can be represented as ( j 1 , j 0 ) b . Which gives j = j 1 b + j 0 . Note that when b = √ N , j 1 = k ( j ) and j 0 = m ( j ) . We choose to use the ( j 1 , j 0 ) notation here since that notation is easier to generalize for p > 2 .

L ∈ DB ( b,N ) is an N × N matrix with b × b blocks that are all diagonal matrices.

R ∈ BD ( b,N ) meaning it's a block diagonal N × N matrix with block size b × b .

We have a class of permutation matrices defined as σ ( b,N ) ( i ) = i 0 · N b + i 1 . This can be denoted by an N × N matrix, P ( b,N ) , where the i th row is e σ ( b,N )( i ) .

We'll use i or pair notation ( i 1 , i 0 ) b to denote the rows, and j or pair notation ( j 1 , j 0 ) b to denote columns. It should be clear from context which one we're using.

For any 0 ≤ j 1 < √ N , let glyph[lscript] j 1 ( X,Y ) be an arbitrary bivariate polynomial with deg X ( glyph[lscript] j 1 ) , deg Y ( glyph[lscript] j 1 ) < √ N .

For any 0 ≤ j 1 , j 0 < N , let r j 1 ,j 0 ( Y ) be an arbitrary univariate polynomial of degree < N . Let A = ( α 0 , . . . , α √ N -1 ) , B = ( β 0 , . . . , β √ N -1 ) each be a sequence of distinct eval points. Note that A and B need not be disjoint.

From the proof of Theorem 3 in the Appendix C of the Monarch paper [14] we get,

Define DB and BD as set of all such L and R matrices over R N × N where if i 0 = k 0

glyph[negationslash]

Pictorially, L and R look as follows:

k1

k=j1

k→

L =PLP

R

We note that our definition of Monarch matrix M in Section 3 is slightly different in that M = M ′ P with M ′ as defined in [14].

Therefore, glyph[negationslash]

and if k 1 = j 1

D.2 Monarch Matrices and Bivariate Polynomial Evaluation

Given polynomials glyph[lscript] j 1 ( X,Y ) for 0 ≤ j 1 < √ N , polynomials r j 1 ,j 0 ( Y ) for 0 ≤ j 1 , j 0 < √ N , evaluation points A = ( α 0 , ..., α √ N -1 ) B = ( β 0 , · · · , β √ N -1 ) (as in Appendix D.1.2), define the matrices L ∈ DB √ N,N and R ∈ BD √ N,N as:

For every 0 ≤ j 1 , i 1 , i 0 < √ N :

For every 0 ≤ j 1 , j 0 , i 0 < √ N :

Note that all entries of L and R not specified above are set to 0 .

Let f be the above function that maps coefficients of glyph[lscript] j 1 ( X,Y ) (which are coefficients of monomials X i 1 Y i 0 for all 0 ≤ i 1 , i 0 < √ N and hence represented by a matrix in R √ N × √ N ) and coefficients of r j 1 ,j 0 ( Y ) (which are coefficients of monomials Y i 0 for all 0 ≤ i 0 < √ N and hence represented by a vector in R √ N ) for all 0 ≤ j 1 , j 0 < √ N to pairs of matrices in DB √ N,N ×BD √ N,N .

Theorem 6. Let f be as defined above. Then f is a bijection.

Proof. To prove f is bijection we must show f is one-to-one and f -1 is one-to-one (and exists).

To show f is one to one means each set of polynomials' coefficients given to f , will output a unique set of matrices ( L , R ) ∈ DB √ N,N ×BD √ N,N . This follows from (21), (22) and the known fact that polynomial evaluation is a function.

Now, to show f -1 exists and is one-to-one, we must show that there is a map from any pair ( L , R ) ∈ DB √ N,N ×BD √ N,N to unique sets of polynomials, ¯ glyph[lscript], ¯ r , with parameters as defined in Appendix D.1.2. Further, we need

We will use Theorems 5 and 4 to show the existence of ¯ glyph[lscript] j 1 and ¯ r j 1 ,j 0 polynomials, giving us the mapping from the matrices to unique polynomials.

We first show the existence of the unique polynomials in (24). Fix 0 ≤ j 1 , j 0 < √ N . Then consider the values 0 ≤ i 0 < √ N :

Then by Theorem 4, there exists a unique polynomial of degree < √ N (call it ¯ r j 1 ,j 0 ( Y ) ) such that for all 0 ≤ i 0 < √ N ,

and which by (25) shows (24).

Next we show the existence of the unique polynomials in (23). Fix 0 ≤ j 1 < √ N . Consider the values 0 ≤ i 1 , i 0 < √ N :

Then by Theorem 5, there exists a unique bi-variate polynomial of deg X < √ N and deg Y < √ N (call it ¯ glyph[lscript] j 1 ( X,Y ) ) such that for all 0 ≤ i 1 , i 0 < √ N ,

which by (26) shows (23).

Therefore f is a bijection.

We can now conclude:

Corollary 1. For every matrix M ′ as defined in (20) , there exists unique polynomials glyph[lscript] j 1 ( X,Y ) and r j 1 ,j 0 ( Y ) , such that for all 0 ≤ i 1 , i 0 , j 1 , j 0 < √ N ,

Proof. Follows from (20), (21), (22) and Theorem 6.

D.2.1 Proof of Theorem 1

We begin with an immediate consequence of Corollary 1:

Corollary 2. Let A,B ⊂ C such that | A | = | B | = √ N . Then the j th column of M ′ is the evaluation of the polynomial glyph[lscript] j 1 ( X,Y ) · r j 1 ,j 0 ( Y ) over A × B .

Proof. Observe that for fixed j 0 , j 1 the right hand side of (28) is glyph[lscript] j 1 ( X,Y ) · r j 1 ,j 0 ( Y ) evaluated at all ( α, β ) ∈ A × B . Thus, ( j 1 , j 0 ) column is evaluation of glyph[lscript] j 1 ( X,Y ) · r j 1 ,j 0 ( Y ) over points in A × B , as desired.

Next, we state a generalization of Theorem 1 that follows from Corollary 2:

Corollary 3. Let A and B be as in Corollary 2. For any vector u , M ′ · u is ¯ u ( X,Y ) evaluated at A × B . Further,

where glyph[lscript] and r are defined by M ′ as in Corollary 1.

Proof. Follows from Corollary 2 and definition of matrix vector multiplication.

In Theorem 1 and the following sections, we consider the polynomial evaluated over the basis polynomials defined by

.

Corollary 4. Let A and B be as in Corollary 2. For any vector u , and M · u is u ( X,Y ) evaluated at A × B . Further,

where glyph[lscript] and r are defined by M ′ as in Corollary 1.

Proof. Follows from Corollary 3 and definition of M .

Specifically, Theorem 1 is a special case of Corollary 4 where glyph[lscript] j 0 ( X,Y ) = glyph[lscript] m ( j ) ( X,Y ) and r j ( Y ) = r j 0 ,j 1 ( Y ) .

D.2.2 Proof of Theorem 2

By Corollary 4, for all 0 ≤ j 1 , i 1 < √ N ,

are the basis polynomials corresponding to M 1 and M 2 . For the coefficient vector k = ( k j 1 ,j 0 ) 0 ≤ j 1 ,j 0 < √ N and similarly for u = ( u j 1 ,j 0 ) 0 ≤ j 1 ,j 0 < √ N , we can construct two polynomials

whose evaluation over ( α i 1 , β i 0 ) = ( ω i 1 , ω i 0 ) where recall as in Appendix D.1.1 ω = e 2 πι √ N , by Theorem 1 is equivalent to the products M 1 · k and M 2 · u , respectively. Taking the component-wise product, y= ( M 1 · k ) glyph[circledot] ( M 2 · u ) , the entry at i = ( i 1 , i 0 ) is given by

Noting that the element of A , i.e. the √ N -th roots of unity, satisfy Z √ N = 1 means that the above are evaluations of

at A × A . Finally, Theorem 1 and the fact that M -1 0 exists implies M 0 · y is polynomial interpolation into basis polynomials corresponding to M 0 . (Here we use the well known fact polynomial interpolation is the inverse of polynomial evaluation).

D.3 Proof of Theorem 3

We review some concepts in Appendix D.3.1. In Appendix D.3.2, we discuss square matrices and causality in terms of operations on univariate polynomials. This allows us to define a general class of operators for causal 1D convolution. In Appendix D.3.3, we give a class of matrices suitable for perform causal Monarch convolution. Specifically, we prove Theorem 3.

D.3.1 Review

Consider the linear operation on an input vector u :

We say that the map is causal to mean the entry y [ i ] only depends on u [0] , u [1] , . . . u [ i ] . This will be the case when A is a lower triangular matrix (we index the top left entry of A as (0 , 0) ). When A is a lower triangular Toeplitz matrix with entries corresponding to some coefficient vector k , this operation is exactly the 1D convolution

where k ′ = ( k , 0 n ) , u ′ = ( u , 0 n ) , and F n is the n × n DFT matrix.

Definition 4. For a matrix M ∈ R n × n , let us define the map

as matrix convolution . When M is a Monarch matrix, (31) is called Monarch convolution .

In this section, we are interested in determining large subclasses of matrices M such that for any coefficient vector k , (31) is causal in u . Weprovide a class of matrices for which Monarch convolution is causal.

We note that for general Monarch matrix M , (31) is not causal in u . By Theorem 2, we have

This is not causal because the mod ( X √ N -1 , Y √ N -1 ) term condenses higher order terms into lower order terms, hence the y [ i ] wouldn't just depend on input information up to value i .

D.3.2 Univariate Matrix Convolutions

We start with a couple of notation assumptions.

Assumption 1. N is a perfect square.

Assumption 2. We will not use pair notation for this subsection since throughout we have i = i 1 √ N + i 0 and j = j +1 √ N + j 0 .

In order to discuss square matrices in terms of univariate polynomials, we give univariate analogs of Theorem 1 and Theorem 2 for general univariate basis. With an eye toward towards performing causal convolution, we restrict our analysis to certain classes of univariate polynomials.

We first define matrices whose j th columns are the evaluation of a minimum degree j (and maximum degree N -1 ) polynomial (recall Definition 1). We generalize Theorem 3 to such matrices.

Lemma 1. For sequence of points A = { 1 , ω N , · · · ω N -1 N } where ω N is the N th root of unity, let M be defined as

where ¯ q j ( Z ) is defined as in Definition 1. Then for any vector v ∈ R N , M · v is equivalent to evaluating the polynomial

at { 1 , ω N , · · · ω N -1 N } .

Proof. By our definition of M , the column M [: , j ] is exactly the evaluation of the polynomial q j ( Z ) at each point in A . The claimed result comes from the definition of matrix vector multiplication and (33).

Note that M or any M 's in this sub-section are not necessarily Monarch matrices.

Next, we state the following intermediate result:

Proposition 1. Let A be the set of the N -th roots of unity. Then for M 1 , M 2 defined as in (32)

is the same as evaluating the polynomial

over A where k ( Z ) , u ( Z ) are of the form (33) , corresponding to M 1 and M 2 , respectively. In other words, for any 0 ≤ i < N ,

Proof. This result follows from Lemma 1 and the definition of the Hadamard product.

Next, we state a re-interpretation of M -1 y :

Proposition 2. Let M be a full rank matrix whose columns are the evaluations of the basis polynomials ¯ q j ( Z ) from Definition 1 for 0 ≤ j < N , and let y ∈ R N be an arbitrary vector. If u = M -1 y , then for all 0 ≤ i < N

where u ( Z ) is the same as in Lemma 1 for M . In other words, M -1 y is the polynomial interpolaton problem for the polynomial basis ¯ q j ( Z ) for 0 ≤ j < N .

Proof. This follows from Lemma 1 and the fact that M is invertible.

From Propositions 1 and 2, we get the following generalization of Theorem 2:

Theorem 7. For matrices M 0 , M 1 , M 2 as defined above, the operation

is equivalent to representing the polynomial

in terms of the basis polynomials where

where k ( Z ) , u ( Z ) are defined as in Lemma 1 in terms of the basis polynomials corresponding to M 1 and M 2 , respectively, and (ˆ q j ( Z )) 0 ≤ j<N corresponds to M 0 .

Proof. Follows from Propositions 1 and 2.

Now we give the class of matrices from which we can build a causal map. Specifically we prove a generalization of Theorem 3:

Theorem 8. Let n ≥ 1 , let N = ⌈ √ 2 n ⌉ 2 . Then define the basis polynomial ¯ q j ( Z ) to have minimum degree j and maximum degree n -1 for 0 ≤ j < n , and for n ≤ j < N , ¯ q j ( Z ) has minimum degree j and maximum degree N -1 .

For all M N with basis columns defined by (¯ q j ( Z )) 0 ≤ j<N as above, the operation

gives a causal map.

Proof. To prove this is causal means each entry, f [ i ] is dependent only on u [0] , u [1] , . . . u [ i ] , where f = ( M -1 N ( M N · ( k , 0 N -n ) ◦ M N · ( u , 0 N -n )) ) . By Theorem 7 , we have

Since deg( k ( Z ) · u ( Z )) ≤ 2 n -2 ≤ N -2 , this is equivalent to

By our choice of ¯ q j , we ensure that ¯ q j · ¯ q j ′ has minimum degree j + j ′ and deg(¯ q j · ¯ q j ′ ) ≤ 2 n -2 < N for any 0 ≤ j, j ′ < n . Then by Lemma 2 (see below), there exists coefficients α j + j ′ ,i ′ such that,

If we define

then for 0 ≤ i < n , we get:

, as desired.

Note that f i only depends on u [0] , u [1] , . . . u [ i ]

We used the following lemma in the proof of Theorem 8.

Lemma 2. Let ¯ q j ( Z ) be defined as in Theorem 8. Then for any 0 ≤ j, j ′ < n ,

for some set of coefficients α j + j ′ ,i .

Proof. We first note that by our choice of ¯ q j , the minimum degree of ¯ q j ( Z ) · ¯ q j ′ ( Z ) is j + j ′ , and deg(¯ q j ( Z ) · ¯ q j ′ ( Z )) ≤ 2 n -2 ≤ N -1 . Our claim follows from that fact that any polynomial p d ( Z ) of minimum degree d and deg( p d ) < N can be expressed as a linear combination of ¯ q d ( Z ) , ¯ q d +1 ( Z ) , . . . , ¯ q N -1 ( Z ) . 4

D.3.3 Causal Monarch Convolutions

In this section we will prove Theorem 3. We will do so by showing that the basis polynomials q j ( Z ) as defined in Theorem 3 are a special case of the basis polynomials ¯ q j ( Z ) as defined in Theorem 8.

We start with a couple of notation assumptions.

Assumption 3. In this sub-section we will using block size b = √ N therefore, we are dropping the block size from index notation. For example, ( i 1 , i 0 ) √ N becomes ( i 1 , i 0 ) for this section.

Assumption 4. Permutation matrices in this subsection are all the same P ( √ N,N ) so we drop the subscript and just use P .

Definition 5. Define

glyph[lscript] j 1 ( Z ) has minimum degree j 1 , and r j 1 ,j 0 ( Z ) has minimum degree j 0 . All polynomials q ′ ( j 1 ,j 0 ) ( Z ) have maximum degree ≤ N -1 .

Next we argue that the above basis polynomials have a specific minimum degree.

Lemma 3. Polynomial q ′ ( j 1 ,j 0 ) ( Z ) as defined in equation (36) has minimum degree j 0 √ N + j 1 .

Proof. We need to show that q ′ ( j 1 ,j 0 ) ( Z ) is a minimum degree j 0 √ N + j 1 polynomial. Note that

where ˜ glyph[lscript] √ N -1 -j 1 ( · ) has degree √ N -1 -j 1 and ˜ r √ N -1 -j 0 ( · ) has degree √ N -1 -j 0 . Simplifying we get

Note that the polynomial q ′ ( j 1 ,j 0 ) ( Z ) has minimum degree j 0 √ N + j 1 . This is not j 1 √ N + j 0 as in defined in equation (1), we will talk more about this soon.

Next, we observe that the polynomials in (36) define a matrix that satisfies (20).

Lemma 4. Let q ′ ( j 1 ,j 0 ) ( Z ) for 0 ≤ j 1 √ N + j 0 < n be as in (36). Define

Then M ′ satisfies (20) .

Proof. If we evaluate the polynomials q ′ ( j 1 ,j 0 ) ( Z ) at ω i N for 0 ≤ i < N , we get

The above corresponds to how we define (20) since we have with

and

Recall from Lemma 3 that q ′ ( j 1 ,j 0 ) has minimum degree j 0 √ N + j 1 . For causality we need q ( j 1 ,j 0 ) ( Z ) to have degree minimum j 1 √ N + j 0 (then the polynomials will satisfy the minimum degree requirements from (1)). Therefore, we permute the columns of M ′ ,

Note that the above M = PLPRP and is indeed a Monarch matrix as defined in Section 3.

Note that the basis polynomials of M are defined as,

We note that the above is same as q j ( Z ) defined in (5) where j = j 1 N + j 0 with the correction in (5) that q j ( Z ) = glyph[lscript] m ( j ) · r j ( Z √ N ) where r j ( Y ) = ˜ r j 0 ,j 1 ( Y ) .

We are finally ready to prove Theorem 3.

Corollary 5 (Theorem 3 restated) . Let N = ⌈ √ 2 n ⌉ 2 . Define M N by q ( j 1 ,j 0 ) as in (38). Then,

gives a causal map.

Proof. Due to using q polynomials as in (38), by Lemma 3 the degree of the ( j 1 , j 0 ) th column is a polynomial with minimum degree j 1 √ N + j 0 . 5 This implies that these basis polynomials are a subset of the more general causal maps in Theorem 8, which proves the claim.

D.4 Block Algorithms for Complex Numbers and Block Size √ N

In the following subsections we restate the results in Section D.3.3 in terms of block operations. As mentioned earlier, this is so that we can do computations on Monarch matrices using only GEMM operations (and simple data movement operations like permutations).

In Appendix D.4.1 we consider arbitrary Monarch matrices and in Appendix D.4.2 we consider the sub-class of Monarch matrices corresponding to Theorem 3.

D.4.1 General Block Monarch Convolution

In this subsection we re-state general Monarch convolutions in terms of block operations.

Recall from equation (22) we defined the block diagonal matrix R as follows ( 0 ≤ i 0 , j 1 , j 0 < √ N ):

where deg(˜ r j 1 ,j 0 ) < √ N .

To do so, we will first work with M ′ N such that M N = M ′ N · P . We want to express Monarch matrices, M N , as univariate polynomial evaluation over { 1 , ω N , . . . , ω N -1 N } . Towards that end, define

By simple observation that ω ( i 1 √ N + i 0 ) √ N N = ω i 0 √ N N = ω i 0 √ N , we have

Fix 0 ≤ j 1 , j 0 < √ N so we're looking at the j th 0 column of block R j 1 ,j 1 , going down the column we evaluate the polynomial ˜ r j 1 ,j 0 at points ( 1 , ω √ N , . . . , ω √ N -1 √ N ) . Which is equivalent to a matrix multiplication of Fourier matrix of size √ N and a matrix of the coefficients of the ˜ r polynomials.

So we can think of the blocks of R as a Fourier matrix times a coefficient matrix. In other words, let us define a matrix ˜ R ∈ R N × √ N which will hold √ N coefficient blocks ˜ R 0 , ˜ R 1 , . . . ˜ R √ N -1 ∈ R √ N × √ N such that,

In other words we have, where

Then we define

Next, we restate the above as product of two block diagonal matrices:

FN

Ro

R0,0

=R

FN

In other words, we have

Equation (21) defined L as evaluation of bivariate polynomials. We now wish to do the same with univariate polynomials.

Recall that L = PLP . We define the i th 0 diagonal block ( 0 ≤ i 1 , i 0 , j 1 < √ N ) for L as:

where deg( glyph[lscript] j 1 ) < N .

Let us define a matrix ˜ L ∈ R N × √ N that will hold the coefficients of polynomials glyph[lscript] j 1 ( Z ) i.e.

We can multiply this matrix with the Fourier matrix of size N × N to get the blocks of L (which we will need to diagonalize). Specifically define

The rows of L ′′ and the rows of M ′ are both indexed by i . Meaning they're ordered in lexicographic ordering ( i 1 , i 0 ) , which is a problem for the following reason. The block diagonal matrix L made from L ′′ needs to be in lexicographic ordering ( i 0 , i 1 ) (see (41)) since it gets permuted on the left M = PLP and right allowing M to be ordered by ( i 1 , i 0 ) . Therefore, when composing L from L ′′ we must permute the rows by P . So we get,

where

Let

Then

Pictorially:

L'o

P

FN

=L

L

L'

L'

Let f be the function that maps coefficient matrices ˜ L , ˜ R ∈ R N × √ N to ( L , R ) where L and R are defined by (42) and (40).

Theorem 9. Let f be as defined above. Then f is a bijection.

Proof. To prove f is a bijection we must show f is one-to-one and f -1 is one-to-one (and exists).

To show f is one-to-one means both ˜ L and ˜ R given to f will output a unique pair ( L , R ) . This follows from (42) and (40), and the fact that polynomial evaluation is a function.

Now to show f -1 exists and is one-to-one, we must show that there's a map for any ( L , R ) to unique ˜ L , ˜ R ∈ R N × √ N . Then for any pair ( L , R ) where both L and R are block diagonal matrices, there's a map to unique sets of polynomials, glyph[lscript], ˜ r where each coefficient is an entry of ˜ L , ˜ R (thus giving a unique mapping from ( L , R ) 's to ( ˜ L , ˜ R ) 's).

We need to show the existence of glyph[lscript] j 1 ( Z ) and ˜ r j 1 ,j 0 ( Y ) such that:

and

Fix 0 ≤ j 1 , j 0 < √ N . Then consider the values 0 ≤ i 0 < √ N :

Then by Theorem 4 there exists a unique polynomial of degree < √ N (call it ˜ r j 1 ,j 0 ) such that for all 0 ≤ i 0 < √ N :

There will be N polynomials with √ N coefficients each, meaning there's unique sets of coefficients to make up the indices of ˜ R .

Now to get the entries of ˜ L from L , fix 0 ≤ j 1 < √ N . Then consider the values 0 ≤ i 1 , i 0 < √ N :

Then by Theorem 4 there exists a unique polynomial of degree < N (call it glyph[lscript] j 1 ) such that for all 0 ≤ i 1 , i 0 < √ N :

There will be √ N polynomials with N coefficients each, meaning there's unique sets of coefficients to make up the indices of ˜ L .

Algorithm 2 is pseudo code for the map from ˜ L , ˜ R ∈ R N × √ N to block diagonal matrices L , R .

Algorithm 2 BLOCKY MONARCH ( ˜ L , ˜ R )

Input:

Output: Block diagonal matrices L , R ∈ C N × N

˜ L , ˜ R ∈ R N × √ N

1: Let F N be the Fourier transform Monarch matrix PL F PR F P

2: L ′ ← P · F N · ˜ L √

4: L ′ a ← L ′ [ a N : a N + √ N -1 , :]

3: for a ← 0 to N -1 do √ √

5: L ← diag ( L ′ 0 , . . . L ′ √ N -1 )

6: for a ← 0 to √ N -1 do 7: R a ← F √ N · ˜ R [ a √ N : a √ N + √ N -1 , :]

8: R ← diag ( R 0 , . . . , R √ N -1 )

9: return L , R

glyph[triangleright] First, compute L from ˜ L glyph[triangleright] See Corollary 6

glyph[triangleright] Now compute R from ˜ R

Lemma 5. Algorithm 2 uses O ( N 3 / 2 ) FLOPs and 3 √ N GEMMs of two √ N × √ N matrices.

Proof. Lines 1 and 2 are multiplying a monarch matrix representing the Fourier transform times a N × √ N matrix in block fashion again, giving O ( N 3 / 2 ) FLOPs.

Similarly, lines 6 and 7 have two √ N matrices multiplied √ N times. Giving O ( N 3 / 2 ) FLOPs. Lines 3, 4 , 5, 8, and 9 don't count towards FLOPs

Now that we have the operations in terms of blocks, we will make them causal in the following sub-section.

D.4.2 Causal Block Monarch Convolution

Recall that a Monarch matrix is defined as

then per equation (37) we have

Then if q ( j 1 ,j 0 ) ( Z ) is the basis polynomial corresponding to the ( j 1 , j 0 ) th column of M , by (38) the basis polynomial corresponding to M is

with the minimum degree of j 1 √ N + j 0 (recall that we pick glyph[lscript] j 0 and ˜ r j 0 ,j 1 to have minimum degree j 0 and j 1 respectively) as desired.

Theorem 10. Let L , R ← BLOCKY MONARCH ( ˜ L , ˜ R ) . Let M be as in (37) . Then for every 0 ≤ i , j < N , we have: √

where q ( j 1 ,j 0 ) is as in (44).

Proof. To prove this we need to show the ( j 1 , j 0 ) th column of M is the basis polynomial q ( j 1 ,j 0 ) evaluated at the N th roots of unity, where q ( j 1 ,j 0 ) is as defined in (44). Note we have

By definition we have,

By (39) we have

and by (41) and the fact that we have

Thus, we have

Since

we have

where the last equality follows from (45).

Corollary 6. The DFT is M as in Theorem 10 when all blocks of ˜ R and the top block of ˜ L are the identity matrix of size √ N × √ N (the rest of ˜ L is all 0 's).

Proof. Since only the diagonal of the top block of ˜ L will contain any non-zero values we only index the first 0 , . . . , √ N -1 rows. And since all blocks of ˜ R are the identity matrix we get

All other entries of ˜ L and ˜ R are 0. Thus, we have

Then by Theorem 10 note and

and

As per Theorem 10,

which implies M is F N as desired.

Algorithm 3 is pseudo code for the Monarch convolution algorithm. It maps an input space of ˜ L , ˜ R matrices, a kernel vector k , and input vector u to a vector f .

We next outline how to make the map in Algorithm 3 causal. Towards that end, we observe:

Lemma 6. For any fixed n ≥ 1 , let N = ⌈ √ 2 n ⌉ 2 . Define ˜ L , ˜ R ∈ R N × √ N such that for every 0 ≤ j 1 < √ N ,

Define ˜ L ′ , ˜ R ′ ∈ R N × √ N with corresponding coefficient blocks ( ˜ L ′ k ) 0 ≤ k< √ N , ( ˜ R ′ k ) 0 ≤ k< √ N as

where

Further, we require that ˜ R ′ j 1 [ j 0 , j 0 ] , ˜ L ′ 0 [ j 1 , j 1 ] are all non-zero entries for all 0 ≤ j 0 , j 1 < √ N . Then for j 0 √ N + j 1 < √ N ⌊ √ N 2 ⌋ , the basis polynomial q ′ ( j 1 ,j 0 ) ( Z ) = glyph[lscript] ′ j 1 ( Z ) r ′ j 1 ,j 0 ( Z √ N ) of M ′ = PLPR has minimum degree j 0 √ N + j 1 and maximum degree ≤ N 2 -1 , and for √ N ⌊ √ N 2 ⌋ ≤ j 0 √ N + j 1 < N , the basis polynomial q ′ ( j 1 ,j 0 ) ( Z ) has minimum degree j 0 √ N + j 1 and maximum degree ≤ N -1 .

Note that in M = M ′ P the ( j 1 , j 0 ) basis polynomial q j 1 ,j 0 ( Z ) = q ′ j 0 ,j 1 ( Z ) has the required degree bound.

Proof. Let L ′ , R ′ ← BLOCKY MONARCH ( ˜ L ′ , ˜ R ′ ) , and denote their corresponding polynomials as glyph[lscript] ′ j 1 ( Z ) , r ′ j 1 ,j 0 ( Z √ N ) , respectively for every 0 ≤ j 1 , j 0 < √ N . By our definition, for all 0 ≤ j 0 , j 1 < √ N , r ′ j 1 ,j 0 ( Z √ N ) has minimum degree j 0 √ N and glyph[lscript] ′ j 1 ( Z ) has minimum degree j 1 . Then it follows that the basis polynomial q ′ ( j 1 ,j 0 ) ( Z ) = glyph[lscript] ′ j 1 ( Z ) r ′ j 1 ,j 0 ( Z √ N ) has minimum degree j 1 + j 0 √ N .

We now look at the degrees of each basis polynomial. From our definition of R ′ j 1 , all entries R ′ j 1 [ i 0 , j 0 ] = 0 for j 0 < ⌊ √ N 2 ⌋ and i 0 ≥ ⌊ √ N 2 ⌋ . This implies that for 0 ≤ j 0 < ⌊ √ N 2 ⌋ , and 0 ≤ j 1 < √ N , we have deg( r ′ j 1 ,j 0 ) < ⌊ √ N 2 ⌋ . Since we are only looking at L ′ 0 ( Z ) , note that degree

deg( glyph[lscript] ′ j 1 ) ≤ √ N -1 . Then it follows that

(Note that in the above j 0 √ N + j 1 ≤ √ N ⌊ √ N 2 ⌋ -1 by same calculations above as needed.)

For ⌊ √ N 2 ⌋ ≤ j 0 < √ N and 0 ≤ j 1 < √ N , deg( r ′ j 1 ,j 0 ) = √ N -1 , and glyph[lscript] ′ j 1 ( Z ) degree √ N -1 . Then it follows that

as desired. (Note that j 0 √ N + j 1 ≥ ⌊ √ N 2 ⌋ √ N

as needed.)

Finally, we use Lemma 6 and Theorem 8 to conclude the following:

Theorem 11. For any fixed n ≥ 1 , let N = ⌈ √ 2 n ⌉ 2 . Let ˜ L ′ , ˜ R ′ ∈ R N × N with corresponding coefficient blocks ( ˜ L ′ k ) 0 ≤ k< √ N , ( ˜ R ′ k ) 0 ≤ k< √ N be defined as in Lemma 6. Then for any k , u ∈ R n , BLOCKMONARCHCONV ( ˜ L ′ , ˜ R ′ , k ′ , u ′ ) [0 : n -1] , where k ′ = ( k , 0 N -n ) , u ′ = ( u , 0 N -n ) , is causal in u .

Proof. Note that this is the same setting as Theorem 8. If N = ⌈ √ 2 n ⌉ 2 , then ⌊ N 2 ⌋ ≥ n . Then by Lemma 6, the basis polynomials q j 1 ,j 0 ( Z ) of M = PLPR have deg( q ( j 1 ,j 0 ) ) < N 2 for 0 ≤ j 1 √ N + j 0 < √ N ⌊ √ N 2 ⌋ ≤ ⌊ N 2 ⌋ . 6

Then (34) computes BLOCKMONARCHCONV ( ˜ L ′ , ˜ R ′ , k ′ , u ′ ) . Since Algorithm 3 performs the same operation as (34), the result follows from Lemma 6 and Theorem 8.

D.5 Bivariate Polynomials with Kronecker Substitution Over Reals

Our earlier results pertain to complex evaluation points. In this section we define causal convolution over real evaluation points. To do this, we redefine our basis polynomials in terms of the Chebyshev polynomials of the first kind.

In Appendix D.5.1 we recover Theorem 8 for univariate polynomials defined over real evaluation points. This identifies a class of matrices that we use to define to define a structured subclass in Appendix D.5.2. We show that these matrices can be used to perform (31). Finally, in Appendix D.5.3 we give the analog of Theorem 11 over real evaluation points. However, the resulting matrices are not Monarch matrices but they are close. In Appendix D.5.4 and Appendix D.5.5 we discuss how we can exploit this closeness, and compute (31) more efficiently.

D.5.1 Univariate Evaluation over Real Numbers

For any integer a ≥ 0 , the Chebyshev polynomial of the first kind with degree a is denoted as T a ( Z ) and is defined as

To parallel Appendix D.3.2, we consider the class of basis polynomials with the form

evaluated over ( ω N,i ) 0 ≤ i<N where and has the property

at each point in A .

Proof. By our definition of M , the column M [: , j ] is exactly the evaluation of the polynomial ¯ q N j ( Z ) at each point in A . The claimed result comes from the definition of matrix vector multiplication and (51).

This leads to the following analog of Theorem 7.

Theorem 12. For matrices M 0 , M 1 , M 2 , each of form as in Lemma 7, the operation

is equivalent to representing the polynomial

in terms of the basis polynomials

Then (52) follows since for

Let us consider the class of matrices defined over (48) and (49).

Lemma 7. For sequence of points A = { ω N, 0 , . . . , ω N,N -1 } , let M be defined as

where ¯ q N j ( Z ) and ω N,i are defined as in (48) and (49) , respectively. Then for any vector u , M · u is equivalent to evaluating the polynomial

where k ( Z ) , u ( Z ) are defined in terms of the respective basis polynomials corresponding to M 1 and M 2 as in Lemma 7, and ( ˆ q N j ( Z ) ) 0 ≤ j<N corresponds to M 0 .

Proof. For A = { ω N,i } 0 ≤ i<N , define

we have the following for any α ∈ A :

The claim follows from Lemma 7, (52), the invertibility of M 0 , and the known fact that T N ( Z ) = q A ( Z ) .

We also utilize the following result:

Lemma 8. Let ¯ q glyph[floorleft] N 2 glyph[floorright] j ( Z ) be defined as in (48) . Then for any 0 ≤ j, j ′ < N ,

for some set of coefficients α j + j ′ ,i .

Proof. From (48), we have

Recall that within a m dimensional vector space of polynomials, we can define a basis by choosing any set of polynomials with degrees 0 , . . . , m -1 . Because deg ( ¯ q glyph[floorleft] N 2 glyph[floorright] j · ¯ q glyph[floorleft] N 2 glyph[floorright] j ′ ) ≤ 2 ⌊ N 2 ⌋ -( j + j ′ ) -2 , it can be written as a linear combination of any set of 2 ⌊ N 2 ⌋ -( j + j ′ ) -1 polynomials where for 0 ≤ a ≤ 2 ⌊ N 2 ⌋ -( j + j ′ ) -2 , the a -th polynomial has degree a . In other words we can choose the a -th polynomial as q N N -a -1 ( Z ) . Thus we have

for some set of coefficients α j + j ′ ,a . Then after reindexing i ← N -a -1 we have

The claim follows by setting α j + j ′ ,j + j ′ +1 = 0 , and if N is odd, α j + j ′ +1 ,j + j ′ +2 = 0 , and α j + j ′ ,i = ¯ α j + j ′ ,N -i -1 for other i .

This allows us to prove the following causality result for convolutions over real evaluation points.

Theorem 13. Fix a family of basis polynomials ¯ q N 0 , ¯ q N 1 . . . , ¯ q N N -1 as defined in (48) . Let N ≥ 1 be a perfect square, n ≤ ⌊ N 2 ⌋ , k , u ∈ R n and M N defined by basis ( ¯ q N j ( Z ) ) 0 ≤ j<N . Let k ′′ = ( k , 0 glyph[floorleft] N 2 glyph[floorright] -n ) and u ′′ = ( u , 0 glyph[floorleft] N 2 glyph[floorright] -n ) . Then the operation

defines a causal map in u .

Proof. Let

In order to prove that (55) is actually causal in the input u ∈ R n , we must show that for all 0 ≤ i < N , f [ i ] is dependent only on u [0] , u [1] , . . . u [ i ] . Let k ′ = ( 0 glyph[ceilingleft] N 2 glyph[ceilingright] , k ′′ ) and u ′ = ( 0 glyph[ceilingleft] N 2 glyph[ceilingright] , u ′′ ) . By Lemma 7, M N · k ′ and M N · u ′ correspond to the evaluations of the polynomials

Let us define

Note that for 0 ≤ j < ⌈ N 2 ⌉ , the coefficients k ′ j = u ′ j = 0 . Then (57) becomes

which is equivalent to

For 0 ≤ j < ⌊ N 2 ⌋ , deg ( ¯ q glyph[floorleft] N 2 glyph[floorright] j ) = ⌊ N 2 ⌋ -j -1 , and deg ( ¯ q N j + glyph[ceilingleft] N 2 glyph[ceilingright] ) = N -⌈ N 2 ⌉ -j -1 = ⌊ N 2 ⌋ -j -1 . This implies that ¯ q glyph[floorleft] N 2 glyph[floorright] j ( Z ) and ¯ q N j + glyph[ceilingleft] N 2 glyph[ceilingright] ( Z ) are both linear combinations of ⌊ N 2 ⌋ -j -1 Chebychev polynomials. Then we can set ¯ q glyph[floorleft] N 2 glyph[floorright] j ( Z ) = ¯ q N j + glyph[ceilingleft] N 2 glyph[ceilingright] ( Z ) . Similarly, note that for 0 ≤ j < n , k ′ j + glyph[ceilingleft] N 2 glyph[ceilingright] = k j . Then it follows that k ( Z ) = k ′ ( Z ) , and by a similar argument, u ( Z ) = u ′ ( Z ) . Then by Theorem 12 we have

where the last statement follows since deg ( k ( Z )) , deg ( u ( Z )) ≤ n -1 < ⌊ N 2 ⌋ , implying that their product has deg ( ¯ k ( Z ) · ¯ u ( Z ) ) < deg ( T N ( Z )) = N . We want to write f ( Z ) in the form

for some set of coefficients set of coefficients f i . From (58), (59) becomes

We show that for all 0 ≤ i < N , f i is a function of ( u i ′ ) 0 ≤ i ′ ≤ i . Then note from the above that each k j and u j ′ appears in terms of ¯ q N i where i ≥ j + j ′ . Then we have

as desired.

D.5.2 Structured Causal Matrices

In this section we narrow our scope from the general class of matrices defined in Appendix D.5.1 to a particular structured subclass. Now let us define the class of structured causal matrices over the real numbers.

Definition 6. Define

where

We define structured causal (SC) matrix polynomials as

A N × N SC matrix is defined over the set of real evaluation points as

We note that in (63) we deviate from the usual ordering of indices ( j 1 , j 0 ) to ( j 0 , j 1 ) . We show that the SC matrix falls under the category of matrices defined by (50).

Lemma 9. Let M C denote the set of all matrices defined by of (50) , and M SC denote the set of all matrices defined by Definition 6. Then M SC ⊂ M C .

Proof. To show that M SC ⊂ M C , then it is sufficient to show that for j = j 0 √ N + j 1 , any q ′ N ( j 1 ,j 0 ) √ N ( Z ) is equivalent to some ¯ q N j ( Z ) as in (48). From (63) we have

Note that

From the fact that deg ( T a ) = a , we can use T a as a polynomial basis. Then q ′ N ( j 1 ,j 0 ) √ N can be represented as a linear combination of T a ( Z ) like so:

which is exactly the form of (48).

Lemma 9 allows us to apply the causality result from Appendix D.5.1.

Corollary 7. Fix a family of basis polynomials q N 0 , 0 , q N 0 , 1 , . . . , q N √ N -1 , √ N -1 as defined in (62) . For any perfect square N ≥ 1 , n ≤ ⌊ N 2 ⌋ , k , u ∈ R n and M ′ N defined by basis ( q ′ N ( j 1 ,j 0 ) √ N ( Z ) ) 0 ≤ j 1 ,j 0 < √ N as in (63) . Then the operation (55) with M N ← M ′ N defines a causal map in u .

Proof. Follows from Theorem 13 and Lemma 9.

D.5.3 Block Operations on Structured Causal Matrices

In this section we show how to build structured causal matrices through block operations.

Constructing M

Recall that in Appendix D.4.1, we defined L , R in terms of coefficient matrices ˜ R , ˜ L ∈ R N × √ N with blocks ˜ R k , ˜ L k for 0 ≤ k < √ N , noting that for each block R j 1 ,j 1 ,

and for ˜ L ∈ R N × √ N

These matrices are then diagonalized into L , R . We use Definition 6 to similarly define L , R , ∈ R N × N with blocks { L j 1 ,j 0 } 0 ≤ j 1 ,j 0 < √ N , { R j 1 ,j 0 } 0 ≤ j 1 ,j 0 < √ N where:

and all other entries are zero. Let the coefficient matrices ˜ L , ˜ R ∈ R N × √ N be defined with respect to blocks as follows:

where the entries of glyph[lscript] √ N -j 1 -1 and ˜ r √ N -j 1 -1 , √ N -j 0 -1 are defined as in (60) and (61). Now let C N ∈ R N × N where

be the Chebyshev transform. Then analogous to Appendix D.4.1, we define

This allows us to give an algorithm the following construction algorithm for L and R .

Algorithm 4 BLOCKSC ( ˜ L , ˜ R )

Input:

˜ ˜ Output: Block diagonal matrices L , R ∈ R N × N

L

,

R

∈

R

N

× √

N

We use Algorithm 4 to specify another type of matrix M ∈ R N × N .

Lemma 10. Let

where L and R are outputs from BLOCKSC . Then each entry in M is defined as

glyph[triangleright] First, compute L from ˜ L

glyph[triangleright] Now compute R from ˜ R

Proof. Let M 0 = PLPR , then M = M 0 P . Then we have

Then we get

This gives us as desired.

Next we will discuss the relation between matrices that we get from Lemma 10 and SC matrices.

D.5.4 Constructing M ′ from M

Since C N / ∈ M SC 7 , Algorithm 4 is not a proper analog of Algorithm 2. In this section, we show how to convert M produced from Algorithm 4 into a matrix with a block decomposible form. Recall from (63) that

We note the distinction between the above and (67). Specifically, we note that M is evaluated on two sets of evaluation points- the glyph[lscript] √ N -j 1 -1 ( Z ) and ˜ r √ N -j 1 -1 , √ N -j 0 -1 polynomials are evaluated over the N roots and √ N roots of unity while M ′ is only evaluated the N -th roots of unity. However, they are close due to the following property:

Lemma 11. Let T a be a Chebyshev polynomial of the first kind of degree a , and define ω N,i as in (49) . Then

In the above the first equality follows from (46).

Analogous to how we utilized roots of unity, Lemma 11 allows us to express our basis polynomials in terms of two related sets of evaluation points, ω √ N,i 0 and ω N,i . The following lemma demonstrates how to translate between M ′ and M .

Lemma 12. For all i, j ,

Proof.

Proof. By (68) we have

where the second statement follows from Lemma 11. Then from (60) and (47) we get

Note from (61) that ˜ r √ N -j 0 -1 , √ N -j 1 -1 [ a ] = 0 if ( √ N -j 1 -1 -a ) is odd. Then equivalently we get

Then from (67) we have

Block Matrix Multiplication for Structured Causal Matrices Lemma 12 shows us how to compute M ′ from M . However, this does not mean that the matrix vector multiplication problem for M ′ can be implemented efficiently. In this section we show how to perform matrix vector multiplication of M ′ with any u ∈ R N from two matrix-vector multiplications of M (and so these operations are indeed efficient).

The key to our approach involves considering the parity of each block index 0 ≤ i 1 < √ N . Let us define a map MIX : R glyph[floorleft] N 2 glyph[floorright] × R glyph[ceilingleft] N 2 glyph[ceilingright] ↦→ R N such that MIX ( u 0 , u 1 ) = u where

We use this map to show that M ′ u and u glyph[latticetop] M ′ can be computed efficiently.

Lemma 13. For any u ∈ R N and M ∈ R N × N , M ′ u can be computed via two matrix-vector multiplications: M · MIX ( u 0 , 0 ) and M · MIX ( 0 , u 1 ) .

Proof. For 0 ≤ i = i 1 √ N + i 0 < N , 0 ≤ j = j 1 √ N + j 0 < N , let D ∈ R N × N be the diagonal matrix defined such that D [ i, i ] = ( -1) i 1 ( √ N -1 ) . Lemma 12 implies that

We want to shift the ( -1) to u . Compute z 0 = M · ( MIX ( u 0 , 0 ) + MIX ( 0 , u 1 )) , z 1 = M · ( MIX ( u 0 , 0 ) -MIX ( 0 , u 1 )) . Define y ′ such that

It can be verified that D · y = M u

′ ′ , which completes the proof.

We now give the analogous result for u glyph[latticetop] M ′ .

Lemma 14. For any u ∈ R N and M ∈ R N × N , u glyph[latticetop] M ′ can be computed via two matrix-vector multiplications: MIX ( u 0 , 0 ) glyph[latticetop] M and MIX ( 0 , u 1 ) glyph[latticetop] M .

Proof. For 0 ≤ i = i 1 √ N + i 0 < N , 0 ≤ j = j 1 √ N + j 0 < N , let D ∈ R N × N be the diagonal matrix defined such that D [ i, i ] = ( -1) i 1 ( √ N -1 ) , and u ′ = u glyph[latticetop] D = MIX ( u ′ 0 , u ′ 1 ) . Lemma 12 implies that

We want to shift the ( -1) to u ′ . Compute z 0 = ( MIX ( u ′ 0 , 0 ) glyph[latticetop] + MIX ( 0 , u ′ 1 ) glyph[latticetop] ) · M , z 1 = ( MIX ( u ′ 0 , 0 ) glyph[latticetop] -MIX ( 0 , u ′ 1 ) glyph[latticetop] ) · M . If y = M ′ u , then one can check that

which completes the proof.

Lemma 13 implies that computing M N k and M N u in (55) can be done efficiently. However, we do not know how to compute M -1 N y for any arbitrary y . We address this partially in the next section.

D.5.5 Inverse of Chebyschev Transform

In this subsection we show how computing C -1 N can be done efficiently.

Lemma 15. Let C N ∈ R N × N be defined as (66) . Then

Given the above, the goal then is to compute C glyph[latticetop] N · u or equivalently u glyph[latticetop] C N for any u ∈ R N . Note that is sufficient to show that

for C N , S N ∈ R N × N such that u glyph[latticetop] C N and u glyph[latticetop] S N can be computed with two invocations of Lemma 14. Indeed we have

We use this to define C N and S N , along with two additional matrices ˜ C N , ˜ S N ∈ R N × N such that for C N and ˜ C N :

and similarly for S N and ˜ S N :

We summarize these results into the following lemma.

Lemma 16. Define C N , ˜ C N , S N and ˜ S N as in (71) and (72) , respectively. Then

where ˜ C N and ˜ S N are of the form (67) .

Finally Lemma 14 and Lemma 16 imply the following:

Theorem 14. For any u ∈ R N ,

can be computed from four calls matrix vector multiplication with Monarch matrices.

Proof Sketch for Theorem 14 Lemma 16 tells us that u glyph[latticetop] C N and u glyph[latticetop] S N are sufficient to compute u glyph[latticetop] C N , and (72) shows us that ˜ C N , ˜ S N are Monarch matrices that allows u glyph[latticetop] C N and u glyph[latticetop] S N to be computed from two matrix- vector multiplications with ˜ C N and ˜ S N , respectively. Then the claim follows from applying Lemma 14 twice.

D.6 Multivariate MONARCH MIXER with Block Size = p √ N

In this section, we generalize Theorem 1, Theorem 2, and Theorem 3 to arbitrary p ≥ 2 . In Appendix D.6.1, we set up notation. We then generalize Theorem 1 and Theorem 2 to general p in Appendix D.6.2 and then generalize Theorem 3 in Appendix D.6.3. In Appendix D.6.4 we connect definition of p -variate Monarch in Appendix D.6.2 to the one defined in Section 3. Finally in Appendix D.6.5 we discuss some extensions and generalizations.

D.6.1 Notation

We will use notation from Appendix D.5.1.

Fix an integer p ≥ 2 . We will be working with indices j , i where j = ( j 0 , . . . , j p -1 ) and i = ( i 0 , . . . , i p -1 ) with 0 ≤ i a , j a < p √ N for every 0 ≤ a < p . We will denote the set of all sub-indices as [0 , p √ N ) p , and the operator glyph[precedesequal] to denote the lexicographical ordering of vectors in [0 , p √ N ) p .

For any 0 ≤ b ′ ≤ M ≤ N such that b ′ divides M and M divides N , define the following permutation matrices:

Note that P b ′ ,N,N is exactly the same as P b ′ ,N from earlier.

If we use σ ( b, M, N ) to denote the corresponding permutation then it takes the input ( i p -1 , . . . , i 0 ) b and maps it to ( i p -1 , . . . , i p ′ , i p ′ -2 , . . . , i 0 , i p ′ ) b (where p = log b N and p ′ = log b M ). I.e. σ ( b, M, N ) does not change the first p -p ′ sub-indices and then does a left rotation on the remaining p ′ sub-indices.

For the rest of the section, assume b = p √ N and then consider the following 'sub-index reversal' permutation matrix 8 :

If σ R ( b, N ) is the permutation corresponding to the permutation matrix above, then σ R ( b, N ) maps ( i p -1 , . . . , i 0 ) ↦→ ( i 0 , . . . , i p -1 ) .

D.6.2 Generalizing Theorem 1 and Theorem 2

For a specific 0 ≤ a < p , let

be an arbitrary polynomial of degree < p √ N in the Chebyshev polynomial basis (see (47)).

We will be interested in the evaluations of the above polynomials over the set

where ω √ N,i is defined as in ( ?? ).

The p -variate version of Monarch matrices M ′ ∈ R N × N as follows. (See Appendix D.6.4 to see how these are exactly related to the definition in (1)). For every row index i ∈ [0 , p √ N ) p and column index j ∈ [0 , p √ N ) p , we have

To express the above in terms of a polynomial basis we will need the following definition. For any 0 ≤ a < p and i define the Lagrange basis polynomial ∆ ( a ) i ( X 0 , . . . , X a -1 ) such that for any 0 ≤ m 0 , . . . , m a -1 < p √ N , we have

We use the above to convert the polynomials in (73) to not depend on the sub-indices in i (at least for the definition in (75)). For every j and 0 ≤ a < p , define:

Note that the summation fixes the last p -a sub-indices in i since the definition of glyph[lscript] ( a ) j , i ( X a ) only depends on ( i 0 , . . . , i a -1 ) . This implies that for any 0 ≤ i 0 , . . . , i a < p √ N , we have

We are now ready to define our basis p -variate polynomials. For any index j ∈ [0 , p √ N ) p , define

Then (75) can be re-written as

The above leads to the following result, which generalizes Theorem 1 to general p :

Theorem 15. Let M ′ , A and q N j ( X 0 , . . . , X p -1 ) be as defined in (75) , (74) and (77) . Then for any vector u , M · u is equivalent to evaluating the polynomial

at each point in A p .

Proof. By (78), the column M ′ [: , j ] is exactly the evaluation of the polynomial (77) at each point in A p . Then the claim follows from the definition of matrix vector multiplication and (79).

This then leads to the following result, (which generalizes Theorem 2):

Theorem 16. For matrices M 0 , M 1 , M 2 , each of form as in Theorem 15, the operation

is equivalent to representing the polynomial

in terms of the basis polynomials

where k ( X ) , u ( X ) are defined in terms of the respective basis polynomials corresponding to M 1 and M 2 as in Theorem 15, and ˆ q N j ( X ) s corresponds to M 0 .

Proof. Define

Then (80) follows since for

we have the following for any a ∈ A p we have:

Using the known fact that T p √ N ( Z ) = ∏ p √ N -1 c =0 ( Z -ω p √ N,c ) = q A ( Z ) , the claim follows from Theorem 15, (80), and the invertibility of M 0 .

D.6.3 Generalizing Theorem 3 for p ≥ 2

To convert Theorem 16 into a causal map we basically have to blow up n → 2 p · n . Further, paralleling (48) we need to change the definition in (73) to have degree p √ N -j a -1 instead of the earlier p √ N -1 :

Note that now the RHS only depends on a and j a (let us call the RHS ˜ glyph[lscript] ( a, p √ N ) j a ( X a ) ), so the next definition becomes easier: √

We are now ready to define our causal basis polynomials. For any index j , define

These polynomials form a structured subclass of the polynomials defined in (77). Lemma 17. The class of polynomials ˜ q N j ( X 0 , . . . , X p -1 ) defined in (83) are a special case of (77) .

Proof. This follows from the fact that (82) is a special case of (73) for every i , j , 0 ≤ a < p .

We show that the product of two ˜ q ⌊ p √ N 2 ⌋ p j ( X 0 , . . . , X p -1 ) type polynomials can be written has a linear combination of ˜ q N m ( X 0 , . . . , X p -1 ) with the indices of m being lexicographically larger than the original indices.

for some set of coefficients α j + j ′ , m .

Proof. From (83) we have,

Let us fix 0 ≤ a < p .

Because (82) is of the same form as in (48), we can apply Lemma 8 to each product ˜ glyph[lscript] ( a, ⌊ p √ N 2 ⌋) j a ( X a ) · ˜ glyph[lscript] ( a, ⌊ p √ N 2 ⌋) j ′ a ( X a ) , which gives us

Going back to (85), we get

Let α j + j ′ , m = ∏ p -1 a =0 α ( a ) j a + j ′ a ,m a . Then we get

as desired.

We now define the following padding scheme, PAD ( k ) .

Algorithm 5 PAD ( k )

The above basis and padding scheme allows us to extend Theorem 3 to general p .

Theorem 17. Fix a family of basis polynomials ˜ q N j ( X ) as defined in (83) . Let N ≥ 1 be a perfect power of p , n ≤ ⌊ p √ N 2 ⌋ p , k , u ∈ R n and M ′ N defined by basis ˜ q N j ( X ) . Then the operation

defines a causal map in u .

Proof. Let k = PAD ( k ) , u = PAD ( u )

In order to prove that (86) is causal in the input u ∈ R n , we must show that for all i ∈ [ 0 , p √ N ) p , f i is dependent only on u i ′ for i ′ glyph[precedesequal] i .

By Theorem 15, M ′ N · k ′ and M ′ N · u ′ correspond to the evaluations of the polynomials

respectively. Let us define

Let us define deg ( ˜ q N j ) = ( p √ N -j a -1 ) p -1 a =0 as the vector of length p consisting of the degrees of the component univariate polynomials ˜ glyph[lscript] ( a ) j , i ( X a ) defined as in (82). Then we have deg ( ˜ q ⌊ p √ N 2 ⌋ p j ) =

Since deg ( ˜ q ⌊ p √ N 2 ⌋ p j ) = deg ( ˜ q N j ′ ) , we can set ˜ glyph[lscript] ( a ) j , i ( X a ) = ˜ glyph[lscript] ( a ) j ′ , i ( X a ) . Similarly, note that for j , j ′ as above, k ′ j ′ = k j and u ′ j ′ = u j . Then it follows that k ( X 0 , . . . , X p -1 ) = k ′ ( X 0 , . . . , X p -1 ) , and by a similar argument, u ( X 0 , . . . , X p -1 ) = u ′ ( X 0 , . . . , X p -1 ) . Then by Theorem 16 we have

where the second line follows by observing that each 0 ≤ a < p , we have deg X a ( k ( X ) · u ( X )) < 2 ⌊ p √ N 2 ⌋ and observing that 2 ⌊ p √ N 2 ⌋ ≤ p √ N . We want to write f ( X 0 , . . . , X p -1 ) in the form

for a set of coefficients f m . From (89) and (90) we get

Thus, for any m ∈ [ 0 , p √ N ) p , we have

implying that f m depends only on u j for j glyph[precedesequal] m , as desired.

D.6.4 p -variate Monarch

Recall that we have fixed b = p √ N (and hence N = b p ). Define the p -variate Monarch matrix as follows:

Where each B a is block diagonal with b × b blocks for every 0 ≤ a < p . Recall that Equation (1) has a permutation P 0 at the end while the above definition does not have any permutation at the end. One trivial way to show the equivalence of above to Equation (1) is to define P 0 = I . In Appendix D.6.5,

we show that exists other non-trivial choices for P 0 . Further for 1 ≤ i ≤ p , the P i in Equation (1) connect to the above definition as follows:

Finally we connect the above definition of p -variate Monarch to the earlier definition based on polynomial evaluation:

Lemma 19. Equation (75) can be written as Equation (91) .

Proof. In (91), B a would correspond to the evaluations glyph[lscript] ( a ) j , i ( ω p √ N,i a ) from earlier. Specifically the following holds for any 0 ≤ a < p . We index the q p -1 blocks of B a by ( i 0 , . . . , i a -1 ) , ( j p -1 , . . . , j a +1 ) and the row and column 'offsets' within each such block are indexed by i a and j a respectively. Connecting back to glyph[lscript] ( a ) j , i ( · ) we set

We will prove the claim as follows. Let e j ∈ R N have a 1 in the j th location and 0 's elsewhere. Our goal is to multiply this vector on the right of Equation (91) and show that we get the j th column of M ′ as defined in Equation (75). We will do so by induction.

Define y 0 = e j and y 1 = B 0 · y 0 . Then for every 1 ≤ a < p , define

Note that we have

Next, we claim that for every 1 ≤ a ≤ p , we have for every ( i 0 , . . . , i a -1 ) ∈ [0 , p √ N ) a -1 ,

where for a = p , we think of (( i 0 , . . . , i a -2 ) , ( j p -1 , . . . , j a ) , i a -1 ) = ( i 0 , . . . , i p -1 ) . We first note that Equation (94) is enough to prove the claim. Indeed we have that

where the RHS in the above is the same as RHS in Equation (75). To see the claim note that by definition of P R b,N , we have ( P R b,N · y p ) [( i p -1 , . . . , i 0 )] = y p [( i 0 , . . . , i p -1 )] . Equation (93) then established the claim.

To complete the proof we prove Equation (94) by induction on a .

We next consider the base case of a = 1 . Note that y 1 is just the j th column of B 0 (as y 0 = e j ) and in that case Equation (94) follows from Equation (92).

For the inductive hypothesis, assume that Equation (94) is true for a for some a ≥ 1 . We now want to argue Equation (94) for y a +1 . Towards that end define

Note that

Now by definition of P b,b p -a +1 ,N , we have

We claim that Equation (94) is true for a + 1 from the above along with Equation (95) and Equation (92). Indeed, fix any ( i 0 , . . . , i a -1 ) . Then in Equation (95) the entry z a [(( i 0 , . . . , i a -1 ) , ( j p -1 , . . . , j a ))] gets multiplied by the entries B (( i 0 ,...,i a -1 ) , ( j p -1 ,...,j a +1 )) a [ i a , j a ] for all values of i a ∈ [0 , p √ N ) . The inductive hypothesis and Equation (92) then proves the inductive step, as desired.

Finally, we note that we can generalize Algorithm 4 for constructing B a for 0 ≤ a < p (since this is the multivariate case some of the steps are a bit simplified):

Algorithm 6 BLOCKY MULTIVAR MONARCH ( ˜ B (0) , . . . ˜ B ( p -1) , N, p ) Input: ˜ B (0) , . . . , ˜ B ( p -1) ∈ R N × b where b = p √ N Output: Block diagonal matrices B 0 , . . . , B p -1 ∈ R N × N glyph[triangleright] Compose each output matrix from corresponding input matrix 1: for y ← 0 to p -1 do 2: for a ← 0 to N b -1 do 3: B ( a ) y ← C b · ˜ B ( a ) ( y ) [ ab : ab + b -1 , :] 4: B y ← diag ( B (0) y , . . . B ( N b -1) y ) 5: return B 0 , . . . , B p -1

D.6.5 Extensions and Open Questions

In this sub-section we outline certain (fairly) straightforward extension of our theoretical results and conclude with some open questions.

Comparing Equation (91) to Equation (1) We note that we can post multiply M in Equation (91) with a large class of permutations for which Theorem 17 still holds. We outline the technical reason why this is true. At the heart of the argument for why Equation (86) gives a causal map is Lemma 18. Specifically note that the sum in RHS in Equation (84), is over all j + j ′ glyph[precedesequal] m . The main observation is that this partial order still holds if we permute the b -variate representation of j , j ′ and m in the same way. In other words, for any permutation σ : [0 , p ) → [0 , p ) if we define σ ( j ) = ( j σ (0) , . . . , j σ ( p -1) ) and similarly σ ( j ′ ) , σ ( m ) . Then we still have σ ( j ) + σ ( j ′ ) glyph[precedesequal] σ ( m ) . This in turn implies the following. Let P σ be a permutation that maps j ∈ [0 , p √ N ) p to σ ( j ) ∈ [0 , p √ N ) p . Then Theorem 17 holds if we replace M ′ by M · P σ with M as in Equation (91).

Evaluation points Our results as presented are for specific classes of evaluation points. A natural question to ask is if our results can be extended to more general set of evaluation points. It turns out that our results for p -variate Monarch matrices can be extended to a wider class of evaluation points. Specifically, for each 0 ≤ a < p , let S a ⊂ C with | S a | = p √ N . Then our results in this sub-section hold if we replace the evaluation points from A p to × p -1 a =0 S a . The only thing that changes in our proofs is that in Theorem 16, we replace mod ( T p √ N ( X 0 ) , . . . , T p √ N ( X p -1 ) ) by mod ( q S 0 ( X 0 ) , . . . , q S p -1 ( X p -1 ) ) , where q A ( Z ) is as defined in Equation (81). This result can then be propagated throughout the rest of our proofs.

On the other hand, our results in Appendix D.3 and Appendix D.5 do exploit specific properties of the evaluation points (specifically ( ω i N ) √ N = ω i 0 √ N for Appendix D.3 and T √ N ( ω N,i ) = ( -1) i 1 ω √ N,i 0 for Appendix D.5). To generalize these results to other sets of evaluation points, we need the existence of degree √ N polynomial that maps (in a √ N -to1 fashion) A to a set of √ N elements. Another interesting open question is to avoid the blowup n → 2 p · n in Theorem 17 and ideally only pay a blowup n → 2 n for every p ≥ 2 as we were able to do in Appendix D.3 and Appendix D.5 (with p = 2 ).

E Implementation

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new neural network layer which runs efficiently on modern GPUs, and exhibits strong performance against state-of-the art on several benchmarks. The layer is based on Monarch matrices, introduced in [7]. Monarch matrices use permutation matrices, and block diagonal matrices to represent dependancies across features and temporal dimensions. Inspired by butterfly matrices in FFT, this matrix parameterization can represent anything from convolutions to fully connected matrices, depending on their order. The Monarch Mixer layer introduced in this paper uses two monarch matrices as well as a matrix (K) that is used in point-wise multiplication. Performance is compared on language and image classification, replacing transformer and fully connected layers with the M2 layer. Finally, the paper presents a theoretical derivation of how this layer can be modified for use in causal language tasks, while maintaining its computational efficiency.\n\nSTRENGTHS:\nTechnically, this paper makes a strong contribution by proposing a computationally efficient layer. The computational performance is benchmarked on modern GPUs. I really liked the discussion on factors affecting runtime performance on modern GPUs -- this is a really valuable introduction to this topic. The proposed layer achieves superior performance, with fewer trainable parameters, in less time. Like MLPMixer, it also does away with the attention mechanism of transformers -- which suggests M2 is also a good architectural inductive bias -- perhaps competitive with attention? Comparing against [7], I would say the largest technical contribution of this paper is on how to modify the M2 layer perform in the causal setting.\n\nWEAKNESSES:\nThe main weakness of the paper is that it is somewhat hard to read/understand without having to read [7]. Section 4 in particular is quite dense. It seems that the authors struggled to fit the paper within the NeurIPS page limit.\n\nQUESTIONS:\n-Does the expressivity of M increase with the order p? Perhaps this is obvious, but it should be stated explicitly in the paper. Can M be used to express any dense matrix? \n\n-Why do you think your model exceeds the performance of other state-of-the art models without using attention? I always thought one of the most important aspects of attention was that it afforded permutation invariance, does M2 serve a similar purpose?\n\nLIMITATIONS:\nLimitations were addressed in second paragraph of Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper takes a fresh approach by addressing the issue of high complexity in current neural networks. It points out that the computational complexity of Transformers is quadratic with respect to both the sequence length and the feature dimension. Previous papers primarily focused on reducing the complexity related to sequence length, but this paper is the first to propose a method that reduces complexity for both sequence length and feature dimension. The specific approach used is the utilization of second-order Monarch Matrices, with the model structure referred to as the Monarch Mixer. Additionally, the authors introduce a novel initialization method for the Monarch Matrices, enabling them to handle causal l language modeling. The effectiveness of this approach is validated in the areas of non-causal language modeling, causal language modeling, and image classification.\n\nSTRENGTHS:\nIndeed, it is novel to consider the optimization of complexity from both the sequence dimension and the feature dimension. Furthermore, initializing the model for the causal scenario poses a definite challenge, and the authors have successfully accomplished this task.\n\nWEAKNESSES:\nThere aren't many weaknesses, for specific questions, please refer to the **Questions** section.\n\nQUESTIONS:\n1. I'm not very familiar with the Monarch Matrix, but is its core idea to use the product of block-diagonal matrices and permutation matrices as a replacement for the dense matrix?\n\n2. What is the motivation behind the Monarch Matrix? Despite having significantly fewer parameters, it seems to perform comparably to dense matrices in small-scale models. Can this conclusion be extended to models larger than 10 billion parameters?\n\n3. In Line 257, \"We set the Monarch matrices to DFT and inverse DFT matrices, to simulate long convolutions [15, 37], and do not learn them.\" Does this refer to setting the block-diagonal matrices of the Monarch Matrix as DFT matrices?\n\n4. The experiments for Non-Causal Language Modeling and Image Classification should be more comprehensive, such as considering the Monarch matrices as learnable.\n\n5. Could you provide a more intuitive explanation of the initialization for the causal scenario? From examining the code, it seems like the Monarch Matrix is initialized as a lower triangular matrix?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe Monarch Mixer (M2) combines MLP mixer and Conv mixer and yields a new family of mixers that is formalized in terms of Monarch matrices. the approach is novel and reminds me of an extension of [15] in their reference. \n\nThe main advantage of M2 is in its sub-quadratic computation capability. The authors evaluate their method on a set of large language models and prove their framework functions properly on meaningful and challenging tasks. The paper is well-written and easy to follow.\n\nSTRENGTHS:\nThe paper stands out in its clear and proper presentation. The idea is well motivated from a hardware perspective, explained intuitively, and studied theoretically. Evaluation of the method on causal and non-causal large language models is an asset and demonstrates a clear potential of the method.\n\nWEAKNESSES:\n- The new approach is only benchmarked on Transformer tasks, while speech applications look relevant but are ignored.\n- It would be fair to see a comparison of inference latency and metric performance of the M2-based BERT with a configuration of BERT that has a smaller number of parameters, but the same number of FLOPs.\n\nQUESTIONS:\n- Table 3 mentions a \"Transformer\" model. What is the configuration of the model?\n- Tables 4 to 7 mention BERT-s. What is BERT-s? What are the details of its architecture?\n- Table 2 seems to be incomplete. The FLOP utilization for dense matmul is missing from Table 2.\n- Caption of Table 4 mentions GPU, however the body of the Table contains no inference results on GPU. Did the authors perform inference on CPU only? Why GPU inference is not reported in Table 4?\n- A comparison with SwinMLP-B [1] and other SWIN-v2 models is more informative than the relatively older ViT model. \n- What is BERT-s? What is ViT-s? please provide the model details.\n\n[1]: Zheng, Hao, Guohui Wang, and Xuchen Li. \"Swin-MLP: a strawberry appearance quality identification method by Swin Transformer and multi-layer perceptron.\" Journal of Food Measurement and Characterization 16.4 (2022): 2789-2800.\n\nLIMITATIONS:\nWhile the method clearly shows that the method works well on Transformer models, convolutional models, and speech applications are ignored.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}