SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our work seeks to understand the fundamental capabilities and limitations of newly-emerging model architectures. As the amount of data and model size grows, we also seek to understand how to make training these models more efficient, both in terms of the amount of training context and the model size. This potentially connects to energy savings during model development and deployment, as well as making machine learning models accessible to a larger population of people.
However, as with any machine learning models, developing new techniques may impact a wide range of applications, each with potential benefits and harms. Making language model training cheaper and longer context may make it cheaper to spread disinformation. Similarly, improving the efficiency of model training may not reduce the overall environmental footprint of training, since the same resources may be used to train more models, or train the same models for longer. While our work makes partial progress on the fronts of efficiency and understanding, it does not explicitly address the issues of fairness and bias in language models. In addition, our work demonstrates a proof-of-concept; it has error modes, and we recognize the inherent risks of training and using machine learning models, including language models. Detailed discussions of these risks are in [3, 5, 80].
B.1 Per-Task GLUE Numbers
We report full GLUE numbers for M2-BERT-base and M2-BERT-large in Table 9.
B.2 Additional Throughput Results
We report the throughput of M2-BERT-base (80M) compared to BERT models of the same size (BERT-base with fewer parameters), as well as the throughput of M2-BERT-large (260M) compared to BERT-large.
Table 9: Fine-tuning performance on GLUE [76]. We report the standard metrics - F1 scores for QQP and MRPC, Matthew's correlation for CoLA, Spearman's correlation for STS-B, and accuracy for the remaining tasks, following the procedure from [38].
Model	MNLI (m / mm)	RTE	QNLI	QQP	SST2	STS-B	CoLA	MRPC	Average
M2-BERT-base (80M)	78.4 / 78.6	68.5	84.6	86.7	92.0	86.3	53.0	89.8	79.9
M2-BERT-base (110M)	79.6 / 80.5	69.3	86.0	87.0	92.3	86.9	56.0	89.2	80.9
M2-BERT-large (260M)	81.7 / 81.9	72.8	84.7	87.8	93.3	88.0	59.2	90.0	82.2
M2-BERT-large (341M)	82.2 / 82.3	75.0	87.0	87.7	92.4	88.3	59.6	90.1	82.8
Table 10: Throughput in tokens/ms by context length for M2-BERT-base (80M) compared to 80M BERT models.
Model	512	1024	2048	4096	8192
HF BERT (79M)	248.4	157.3	86.0	46.8	OOM
FlashAttention BERT (79M)	433.3	425.1	335.2	217.4	122.6
M2-BERT-base (80M)	386.3	380.7	378.9	353.9	320.1
M2 Speedup over HF BERT (80M)	1.6×	2.4×	4.4×	7.5×	-
Table 10 compares the performance of M2-BERT-base (80M) to BERT models parameter-matched to 80M parameters. M2 is slower than FlashAttention for sequence lengths 512 and 1K, but outperforms FlashAttention starting at sequence length 2K. We believe further optimization of the M2 kernel can close the gap to FlashAttention for short sequences.
Table 11 compares M2-BERT-large (260M) to BERT-large. Trends are mostly similar to comparisons against BERT-base; M2 nearly matches FlashAttention at sequence length 512, and outperforms it for sequence length 1K and longer. We also see up to  $4.3 \times$  speedup over HuggingFace BERT-large at sequence length 2K.
B.3 ImageNet Comparison against Swin
Table 12 reports the results of replacing attention and MLP in Swin-V2 using M2 as a drop-in replacement. Surprisingly, Swin-M2 outperforms Swin-MLP-B, is competitive with Swin-V1-B, and comes within 1 point of Swin-V2-B, even without any hyperparameter tuning or architecture adjustment from the ViT formula. We expect that performance may improve further with hyperparameter tuning specific to M2.
B.4 Speech Applications
Table 13 presents the performance of M2 on Speech Commands-10, a speech classification task over raw 1-second clips sampled at  $16\mathrm{kHz}$ . M2 is competitive with state-of-the-art architectures on this task.
B.5 CIFAR10
Table 14 shows the performance of MONARCH MIXER on CIFAR10. The trends are largely the same as on ImageNet.
B.6 Learnable Monarch Matrices in Sequence Mixer
In most of our models, we have used fixed Monarch matrices for the sequence mixer, and learnable Monarch matrices for the dimension mixer. Table 15 presents an experiment evaluating using learnable Monarch matrices for the sequence mixer on the sequential CIFAR task. We use a nongated convolutional architecture based off long convolutions, as presented in [28]. Learning the Monarch matrices in the sequence mixer yields 1.5 points of lift.
Table 11: Throughput in tokens/ms by context length for M2-BERT-large (260M) compared to BERT-large.
Model	512	1024	2048	4096	8192
HF BERT-large (340M)	75.4	47.1	25.2	OOM	OOM
FlashAttention BERT-large (340M)	125.0	111.9	91.6	54.5	OOM
M2-BERT-large (260M)	122.5	118.6	109.4	94.5	75.0
M2 Speedup over HF BERT-large (340M)	1.6×	2.5×	4.3×	-	-
Table 12: ImageNet accuracy of Swin models.
Model	ImageNet (acc@1)	ImageNet (acc@5)
Swin-MLP-B	81.3	95.3
Swin-V1-B	83.5	96.5
Swin-V2-B	84.2	96.9
M2-Swin-B	83.5	96.7
B.7 Roofline Analysis
Figure 4 shows a Roofline analysis of a simple PyTorch implementation of a single M2 operator  $\mathbf{M}^{-1}(\mathbf{Mu}\odot \mathbf{Mk}$  on an A100 GPU, with 4K input length. The operation is more dominated by the data movement operations, which helps explain why performance is higher on newer architectures like RTX 4090 (which have faster and larger L2 cache).
B.8 Associative Recall
In Table 16, we present a simple experiment demonstrating the causal parameterization of M2 on associative recall, a synthetic language designed to test in-context learning. The model demonstrates in-context learning abilities in sequences up to 128K tokens, but Transformers do not scale past 8K.
B.9 BERT Experiments with Alternative Architecture
Here, we report results using an older version of the M2-BERT architecture, that uses non-gated convolutions and is trained on English Wikipedia [25] and English Bookcorpus [92]. For clarity, we refer to this model as M1-BERT.
We found that M1-BERT could match Transformers on MLM quality, but underperformed on downstream fine-tuning. We attribute this gap in performance to sub-optimal training hyperparameters (optimized for throughput using NVIDIA MLPerf hyperparameters) as well as a sub-optimal architecture. We report results here for completeness, but refer to the gated convolution architecture in the main body as the proper M2-BERT model.
These models followed the reference implementations and hyperparameters from Hugging Face Transformers examples [81] and Nvidia Deep Learning examples (https://github.com/NVIDIA/DeepLearningExamples). In particular, we use the LAMB optimizer with a learning rate of  $5e - 3$ . For each sequence length, we use as large a minibatch size as possible that fits on the GPU (A100-80GB in Table 17 and V100 in Table 18). We set the gradient accumulation to reach a global batch size of 65, 536 sequences. To investigate the effect of sequence length, each model is trained for a fixed sequence length in a single phase of training (in contrast to some training protocols, which train the model in multiple phases, each at different sequence lengths).
Time to a Fixed Pretraining Quality on 8xA100 We compare time to a fixed pretraining quality, training M1-BERT-base on English Wikipedia [25] and English Bookcorpus [92]. We compare against BERT-base trained with FLASHATTENTION [15], as well as the Monarch-BERT-base implementation from the original Monarch paper [14]. We measure wall-clock time for M1-BERT and the base Transformer to reach  $50\%$  in masked language modeling accuracy on 8xA100 Nvidia GPUs with 80GB memory each. Table 17 summarizes results. In short sequence lengths, M1-BERT is
Table 13: Accuracy on Speech-Commands 10. An "x" means that the model did not fit in memory.
M2	S4	WaveGan-D	Transformer	Performer	CKConv
97.9	97.5	96.3	x	30.8	71.7
Table 14: Accuracy on CIFAR-10.
Model	Top-1%	Description
ViT (1.2M)	78.6	Attention + MLP
ViT + Monarch (607K)	79.0	Attention, MLP-Free
HyenaViT (1.3M)	80.6	Attention-Free + MLP
HyenaViT-M2 (741K)	80.8	Attention-Free + MLP Free
comparable to FLASHATTENTION, even without using a heavily-optimized fused kernel. In longer sequence lengths, the FLOP savings make M1-BERT more efficient—up to  $2.4 \times$  faster than BERT with FLASHATTENTION at sequence length 4096.
BERT in Half a Day Inspired by recent work focusing on training under limited resource constraints [30], we measure how far we can get when training on a single V100 GPU in 12 hours. In Table 18, we report the masked language modeling accuracy achieved by the same set of models and sequence lengths (except for the FLASHATTENTION baseline, which is not supported on V100). We observe M1-BERT both achieves higher accuracy within the time limit and can be trained at longer sequence lengths than the baseline architectures.
Downstream Fine-Tuning We evaluate the quality of M1-BERT-base models on the GLUE benchmark [76]. Table 19 shows fine-tuning performance on the GLUE tasks, using the same hyperparameters and 5 epochs for all tasks and both models. M1-BERT-base is competitive with Transformers trained using MLP Perf hyperparameters on Bookcorpus and Wikitext, but underperforms fully-trained transformers and M2-BERT-base.
C.1 Model Architectures
In this section, we describe the exact model architectures we used for each task, including the design of the block (residuals and gating). We additionally release our code for reproducibility,
BERT Language Modeling The M2-BERT architectures use a standard BERT backbone, but replace the attention with bidirectional gated convolutions and replace the linear layers in the MLPs with block-diagonal matrices. All the M2-BERT architectures use an expansion factor of four. M2-BERT-base (80M) has a model width of 768 and 12 layers; M2-BERT-base (110M) has a model width of 960 and 12 layers; M2-BERT-large (260M) has a model width of 1536 and 12 layers; and M2-BERT-large (341M) has a model width of 1792 and 12 layers. We train all these models on C4 for 70,000 steps, with sequence length 128, and global batch size 4096 sequences. For all the models, we use decoupled AdamW with learning rate 8e-4 and decoupled weight decay 1e-5. We use linear learning rate decay with a warmup of  $6\%$  of the steps, and we use MLM masking percentage of  $30\%$ .
For GLUE fine-tuning, we do a small search of learning rate, weight decay, and number of epochs. Following [38], we fine-tune RTE, MRPC, and STS-B from the MNLI checkpoint. We fine-tune all tasks with sequence length 128. For some tasks, we also pool the embeddings of all the non-padding tokens instead of using the CLS token.
The final hyperparameters for M2-BERT-base (80M) are decoupled AdamW with learning rate 5e-5 and weight decay 5e-6 for 3 epochs for MNLI; AdamW with learning rate 5e-5 and weight decay 0.01 for 6 epochs for RTE; AdamW with learning rate 3e-5 and weight decay 0.01 for 10 epochs on QQP; AdamW with learning rate 5e-5 and weight decay 1e-5 for 10 epochs with average pooling for QNLI; decoupled AdamW with learning rate 3e-5 and weight decay 3ed-6 for 3 epochs for SST-2; AdamW
Table 15: Accuracy on sequential CIFAR for fixed vs. learnable Monarch in the sequence mixer.
Model	sCIFAR Accuracy
M2, Fixed Monarch	91.0
M2, Learnable Monarch	92.5
Figure 4: Roofline plot of a PyTorch implementation of a single M2 operator  $\mathbf{M}^{-1}(\mathbf{M}\mathbf{u}\odot \mathbf{M}\mathbf{k})$
with learning rate 7e-5 and weight decay 0.01 for 10 epochs for STS-B; AdamW with learning rate 5e-5 and weight decay 0.01 for 10 epochs for MRPC; and decoupled AdamW with learning rate 5e-5 and weight decay 5e-6 for 10 epochs for COLA.
For M2-BERT-base (110M), the hyperparameters are decoupled AdamW with learning rate 5e-5 and weight decay 5e-6 for 3 epochs for MNLI; decoupled AdamW with learning rate 1e-5 and weight decay 1e-6 for 3 epochs for RTE; decoupled AdamW with learning rate 3e-5 and weight decay 3e-6 for 5 epochs on QQP; decoupled AdamW with learning rate 5e-5 and weight decay 1e-5 for 10 epochs with average pooling for QNLI; decoupled AdamW with learning rate 3e-5 and weight decay 3ed-6 for 3 epochs for SST-2; decoupled AdamW with learning rate 8e-5 and weight decay 3e-6 for 10 epochs for STS-B; decoupled AdamW with learning rate 8e-5 and weight decay 8e-5 for 10 epochs for MRPC; and AdamW with learning rate 8e-5 and weight decay 5e-6 for 10 epochs for COLA.
For M2-BERT-large (260M), the hyperparameters are decoupled AdamW with learning rate 5e-5 and weight decay 5e-6 for 3 epochs for MNLI; decoupled AdamW with learning rate 1e-5 and weight decay 1e-6 for 3 epochs for RTE; decoupled AdamW with learning rate 3e-5 and weight decay 3e-6 for 5 epochs on QQP; decoupled AdamW with learning rate 5e-5 and weight decay 1e-5 for 10 epochs for QNLI; decoupled AdamW with learning rate 3e-5 and weight decay 3ed-6 for 3 epochs for SST-2; decoupled AdamW with learning rate 7e-5 and weight decay 3e-6 for 10 epochs for STS-B; decoupled AdamW with learning rate 8e-5 and weight decay 8e-6 for 10 epochs for MRPC; and AdamW with learning rate 5e-5 and weight decay 5e-6 for 10 epochs for COLA.
For M2-BERT-large (341M), the hyperparameters are decoupled AdamW with learning rate 5e-5 and weight decay 5e-6 for 3 epochs for MNLI; AdamW with learning rate 5e-5 and weight decay 1e-6 for 2 epochs for RTE; decoupled AdamW with learning rate 3e-5 and weight decay 3e-6 for 5 epochs on QQP; decoupled AdamW with learning rate 5e-5 and weight decay 1e-6 for 10 epochs for QNLI; decoupled AdamW with learning rate 3e-5 and weight decay 3ed-6 for 3 epochs for SST-2; decoupled AdamW with learning rate 8e-5 and weight decay 3e-5 for 8 epochs for STS-B; decoupled AdamW with learning rate 8e-5 and weight decay 8e-6 for 10 epochs for MRPC; and decoupled AdamW with learning rate 5e-5 and weight decay 1e-6 for 10 epochs for COLA.
ViT We use a standard ViT model architecture as base [22]. In line with recent improvements to the ViT architecture [4, 70, 86], we use sinusoidal position embeddings and global average-pooling (GAP) instead of a class token.
Table 16: In-context learning performance on associative recall at various sequence lengths, vocab size 20.  $\times$  indicates the Transformer did not finish in a week.
Model	0.5K	2K	8K	32K	128K
Transformer	100.0	100.0	100.0	X	X
MONARCH MIXER	98.7	99.4	99.4	99.4	99.4
Table 17: Time in hours to reach  $50\%$  masked language modeling validation accuracy on 8xA100 with different sequence lengths.
Model	512	1024	2048	4096	Architecture Details
BERT-base-FLASHATTENTION (110M)	2.7	3.8	5.7	13.2	Attention, MLP
BERT-base-HuggingFace (110M)	3.3	5.6	13.1	26.7	Attention, MLP
BERT-Monarch-base (80M)	3.1	4.7	10.3	22.1	Attention, MLP-free
M1-BERT-base (55M)	2.5	3.5	4.0	5.5	Attention-Free, MLP-free
Speedup	1.1×	1.1×	1.3×	2.4×
We adapt the ViT architecture by replacing its MLP and/or attention components with Monarch Matrices (similar to our adaptation of BERT):
We replace the MLP with randomly initialized Monarch Matrices of the same dimension as the dense matrices of the MLP and learn those matrices during training, setting the number of blocks in the block-diagonal matrices to 4.
We replace attention with the recently introduced Hyena operator [64]. The Hyena operator represents a recurrence of two efficient sub-quadratic primitives, an implicit long convolution and multiplicative element-wise gating of the projected input. Hyena operators apply the FFT algorithm to achieve fast long convolutions in sub-quadratic time. We further adapt the Hyena operator by replacing its long convolutions with the M2 operator and setting the Monarch Matrices to the DFT and inverse DFT.
ViT for ImageNet-1k In line with other work [4, 14, 64, 70], we use a ViT-base architecture with 12 layers, a hidden size of 768, 12 attention heads per layer, an intermediate size of the MLP projection of 3, 072, and a patch size of  $16 \times 16$  pixels. For optimization, we follow the training procedure of T2T-ViT [86], including augmentations such as RandAugment [12] (magnitude = 9, magnitude-standard = 0.5, layers = 2), Mixup [88] ( $\alpha = 0.8$ ), CutMix [87] ( $\alpha = 1.0$ ), Random erasing [90] (rate = 0.25), and AugMix [37]. See Table 20 for all other training settings.
ViT for CIFAR-10 We use a ViT architecture with 6 layers, a hidden size of 128, 8 attention heads per layer, an intermediate size of the MLP projection of 512, and a patch size of  $4 \times 4$  pixels. We further tune weight decay (0 or 0.1), stochastic depth rate (0 or 0.1), and base learning rate ( $1e - 4$  or  $3e - 4$  or  $1e - 3$ ) and report the test performance for the model variant that achieved the highest accuracy in a separate held-out validation dataset (randomly selected  $10\%$  of training data). We also apply an early stopping rule such that training is stopped if the model's validation loss does not improve for 10 training epochs. See Table 20 for all other training settings.
GPT Causal Language Modeling Similarly to our ViT approach, we also replace attention with the Hyena operator, using the same architecture as in [64] as a starting point. The Hyena architecture has two convolutions, which can be computed using the FFT convolution theorem. In our architecture, we additionally replace these FFT operations with causal Monarch matrices.
In addition, we re-use the heads extension from the H3 architecture [27]. The heads extension groups the model dimension into heads, ties together the long convolution parameters in each head, and then computes the outer product between different input projections. An algorithmic listing adapted from the H3 paper [27] is provided in Listing 1, with updates to replace the SSM layers with Hyena convolutions. We use a head dimension of 16. Setting the head dimension to be 1 and replacing the Monarch matrices with FFT is equivalent to the Hyena layer.
Table 18: Masked language modeling validation accuracy achieved on a single V100 in 12 hours with different sequence lengths.  $\pmb{x}$  indicates the model does not fit on device with a batch size of 1.
Model	512	1024	2048	4096	8192	Architecture Details
BERT-base (110M)	11.5	7.8	6.8	✘	✘	Attention, MLP
BERT-Monarch-base	6.9	8.5	6.8	✘	✘	Attention, MLP-Free
M1-BERT-base	20.2	20.2	20.1	17.1	12.9	Attention-Free, MLP-Free
Table 19: Fine-tuning performance on the GLUE benchmark [76], after pretraining on Wikipedia and Bookcorpus. We report the standard metrics - F1 scores for QQP and MRPC, Matthew's correlation for CoLA, Spearman's correlation for STS-B, and accuracy for the remaining tasks [21].
Model	MNLI (m / mm)	RTE	QNLI	QQP	SST2	STS-B	CoLA	MRPC	Architecture Details
BERT no pretrain	34.1 / 34.1	47.3	50.0	68.6	79.9	17.8	0.0	77.9	Attention, MLP
BERT-base	74.5 / 74.7	55.6	69.3	81.8	83.9	19.8	12.1	74.2	Attention, MLP
M1-BERT-base	69.9 / 70.5	53.1	73.2	81.4	85.2	68.1	33.6	75.4	Attention-free, MLP-free
Algorithm 1 M2 Hyena Layer with Heads
Input: Input sequence  $u \in \mathbb{R}^{N \times d}$  from the previous layer, weight matrices  $\mathbf{W}_{X1}, \mathbf{W}_{X2}, \mathbf{W}_V, \mathbf{W}_O \in \mathbb{R}^{d \times d}$ , causal Monarch matrix  $\mathbf{M}$ , short convolution kernels  $\mathbf{K}_1, \mathbf{K}_2, \mathbf{K}_3$ , a Hyena convolution kernel  $\mathbf{K}_{\mathrm{long}}$ , head dimension  $d_h$ .
Output: Output sequence  $y \in \mathbb{R}^{N \times d}$
Compute  $\bar{\mathbf{X}}_1 = \bar{u}\mathbf{W}_{X1},\bar{\mathbf{X}}_2 = u\mathbf{W}_{X2},\mathbf{V} = u\mathbf{W}_V\in \mathbb{R}^{N\times d}$
Pass  $\mathbf{X}_1, \mathbf{X}_2, \mathbf{V}$  each through the short convolution using the causal Monarch matrices:  $\overline{\mathbf{X}_1}, \overline{\mathbf{X}_2}, \overline{\mathbf{V}} = \mathbf{M}^{-1}(\mathbf{M}\mathbf{X}_1 \odot \mathbf{M}\mathbf{K}_1), \mathbf{M}^{-1}(\mathbf{M}\mathbf{X}_2 \odot \mathbf{M}\mathbf{K}_2), \mathbf{M}^{-1}(\mathbf{M}\mathbf{V} \odot \mathbf{M}\mathbf{K}_3)$ .
Split  $\overline{\mathbf{X}_1},\overline{\mathbf{X}_2},\overline{\mathbf{V}}$  into  $H$  "heads"  $(\overline{\mathbf{X}}_1^{(h)},\overline{\mathbf{X}}_2^{(h)},\overline{\mathbf{V}}^{(h)}$  for  $h = 1,\dots ,H)$  , each a sequence of  $N$  vectors of size  $d_{h} = d / H$
for  $1\leq h\leq H$  do
Take the batched outer product  $\overline{\mathbf{X}}_2^{(h)}(\overline{\mathbf{V}}^{(h)})^\top \in \mathbb{R}^{N\times d_h\times d_h}$  (batched in the  $N$ -dimension) and pass it through the long convolution using the causal Monarch:  $\mathbf{X}\mathbf{V}^{(h)} = \mathbf{M}^{-1}(\mathbf{M}\overline{\mathbf{X}}_2^{(h)}(\overline{\mathbf{V}}^{(h)})^\top \odot \mathbf{M}\mathbf{K}_{\mathrm{long}})\in$ $\mathbb{R}^{N\times d_h\times d_h}$ .
Batch-multiply by  $\overline{\mathbf{X}}_1$ :  $\mathbf{O}^{(h)} = [\overline{\mathbf{X}}_{11}^{(h)}\mathbf{X}\mathbf{V}_1^{(h)},\dots,\overline{\mathbf{X}}_{1N}^{(h)}\mathbf{X}\mathbf{V}_N^{(h)}]\in \mathbb{R}^{N\times d_h}$  (batched in the  $N$  dimension).
Concatenate the output  $\mathbf{O}^{(h)}$  of each head, and multiply by the output projection matrix  $\mathbf{W}_O\in \mathbb{R}^{d\times d}$ .
Finally, we remove the MLP layers entirely (equivalent to replacing the layer with an identity), and make the model wider to compensate (the depths match the equivalent Hyena models). The small model has a model width of 1160 with 18 layers and uses a learning rate of 0.0006, and the medium model has model width of 1344 with 40 layers and uses a learning rate of 0.0008. All other hyperparameters match the Hyena models [64].
D Missing details from Section 4
This section contains all the missing details (including proofs) from Section 4.
In Appendix D.1, we review some definitions and results on multi-variate polynomials and set some notation needed for this section. In Appendix D.2, we explicitly connect Monarch matrices for  $p = 2$  and bivariate polynomial evaluation. Specifically, we prove Theorem 1 and Theorem 2. Then in Appendix D.3 we show how to instantiate the bivariate basis polynomials so that we get a causal map. This includes converting the bivariate polynomials to univariate polynomials (with evaluations over the  $N$ th roots of unity) and this proves Theorem 3. We then show how this causal map can be implemented only using GEMMs (and  $O(N^{3/2})$  FLOPs) in Appendix D.4.
Next, we note that while our evaluations points are over complex numbers, our input and output to the Monarch convolution layers are over reals. Hence, it is natural to wonder if we can implement the entire layer just with operations over real numbers. One potential advantage of this is that we theoretically only have to keep  $N$  real numbers for intermediate results (instead of  $2N$  reals
Table 20: ViT training settings.
ImageNet-1k	CIFAR-10
Optimizer	AdamW
Optimizer momentum	β1, β2 = 0.9, 0.999
Learning rate schedule	Cosine decay w/ linear warmup
Dropout rate	0
Label smoothing	0.1
Image size	224 x 224	32 x 32
Base learning rate	1e-3	{1e-4, 3e-4, 1e-3}
Batch size	1024	512
Training epochs	300	up to 500
Warmup epochs	10	5
Stochastic depth rate	0.1	{0, 0.1}
Weight decay	0.05	{0, 0.1}
numbers when we keep track of vectors in  $\mathbb{C}^N$ . This can reduce the data movement costs. Further, multiplication of two complex numbers requires six operations over real numbers (four multiplication and two addition). Thus, moving to an implementation that only uses real numbers could potentially lead to wall clock time speedup. We propose one such scheme in Appendix D.5 that proves a version of Theorem 3 just over reals by moving to the Chebyshev basis (instead of the standard monomial basis). This creates new technical challenges, which we also address.