SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 9. Let  $\mathcal{M}^C$  denote the set of all matrices defined by of (50), and  $\mathcal{M}^{SC}$  denote the set of all matrices defined by Definition 6. Then  $\mathcal{M}^{SC} \subset \mathcal{M}^C$ .
Proof. To show that  $\mathcal{M}^{SC} \subset \mathcal{M}^C$ , then it is sufficient to show that for  $j = j_0\sqrt{N} + j_1$ , any  $q_{(j_1,j_0)\sqrt{N}}^N (Z)$  is equivalent to some  $\bar{q}_j^N (Z)$  as in (48). From (63) we have
$$
q _ {(j _ {1}, j _ {0}) _ {\sqrt {N}}} ^ {N} (Z) = \ell_ {\sqrt {N} - j _ {0} - 1} (Z) \cdot \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} \left(T _ {\sqrt {N}} (Z)\right). \tag {64}
$$
$$
\begin{array}{l} \deg \left(q ^ {\prime} _ {(j _ {1}, j _ {0}) _ {\sqrt {N}}} ^ {N}\right) = \sqrt {N} - j _ {0} - 1 + (\sqrt {N} - j _ {1} - 1) \sqrt {N} \\ = N - \sqrt {N} j _ {1} - j _ {0} - 1 \\ = N - j - 1. \\ \end{array}
$$
From the fact that  $\deg (T_a) = a$ , we can use  $T_{a}$  as a polynomial basis. Then  $q^{\prime}_{(j_1,j_0)\sqrt{N}}^N$  can be represented as a linear combination of  $T_{a}(Z)$  like so:
$$
q _ {(j _ {1}, j _ {0}) \sqrt {N}} ^ {N} (Z) = \sum_ {a = 0} ^ {N - j - 1} q _ {(j _ {1}, j _ {0})} [ a ] T _ {a} (Z),
$$
which is exactly the form of (48).
Lemma 9 allows us to apply the causality result from Appendix D.5.1.
Corollary 7. Fix a family of basis polynomials  $q_{0,0}^{N}, q_{0,1}^{N}, \ldots, q_{\sqrt{N} - 1, \sqrt{N} - 1}^{N}$  as defined in (62). For any perfect square  $N \geq 1$ ,  $n \leq \left\lfloor \frac{N}{2} \right\rfloor$ ,  $\mathbf{k}, \mathbf{u} \in \mathbb{R}^n$  and  $\mathbf{M}_N'$  defined by basis  $\left(q_{(j_1, j_0) \sqrt{N}}^{N}(Z)\right)_{0 \leq j_1, j_0 < \sqrt{N}}$  as in (63). Then the operation (55) with  $\mathbf{M}_N \gets \mathbf{M}_N'$  defines a causal map in  $\mathbf{u}$ .
Proof. Follows from Theorem 13 and Lemma 9.  $\square$
D.5.3 Block Operations on Structured Causal Matrices
In this section we show how to build structured causal matrices through block operations.
Constructing M
Recall that in Appendix D.4.1, we defined  $\mathbf{L},\mathbf{R}$  in terms of coefficient matrices  $\widetilde{\mathbf{R}},\widetilde{\mathbf{L}}\in \mathbb{R}^{N\times \sqrt{N}}$  with blocks  $\widetilde{\mathbf{R}}_k,\widetilde{\mathbf{L}}_k$  for  $0\leq k < \sqrt{N}$ , noting that for each block  $\mathbf{R}_{j_1,j_1}$
$$
\mathbf {R} _ {j _ {1}, j _ {1}} = \mathbf {F} _ {\sqrt {N}} \cdot \widetilde {\mathbf {R}} _ {j _ {1}, j _ {1}},
$$
and for  $\widetilde{\mathbf{L}}\in \mathbb{R}^{N\times \sqrt{N}}$
$$
\mathbf {L} ^ {\prime} = \mathbf {P F} _ {N} \cdot \tilde {\mathbf {L}}.
$$
These matrices are then diagonalized into  $\mathbf{L},\mathbf{R}$ . We use Definition 6 to similarly define  $\mathbf{L},\mathbf{R},\in \mathbb{R}^{N\times N}$  with blocks  $\{\mathbf{L}_{j_1,j_0}\}_{0\leq j_1,j_0 < \sqrt{N}},\{\mathbf{R}_{j_1,j_0}\}_{0\leq j_1,j_0 < \sqrt{N}}$  where:
$$
\mathbf {L} _ {i _ {1}, j _ {1}} \left[ i _ {0}, i _ {0} \right] = \ell_ {\sqrt {N} - j _ {1} - 1} \left(\omega_ {N, i}\right) \quad \mathbf {R} _ {j _ {1}, j _ {1}} \left[ i _ {0}, j _ {0} \right] \leftarrow \tilde {r} _ {\sqrt {N} - j _ {1} - 1, \sqrt {N} - j _ {0} - 1} \left(\omega_ {N, i}\right), \tag {65}
$$
and all other entries are zero. Let the coefficient matrices  $\widetilde{\mathbf{L}},\widetilde{\mathbf{R}}\in \mathbb{R}^{N\times \sqrt{N}}$  be defined with respect to blocks as follows:
$$
\widetilde {\mathbf {L}} [ a, j _ {1} ] = \ell_ {\sqrt {N} - j _ {1} - 1} [ a ] \qquad \widetilde {\mathbf {R}} _ {j _ {1}} [ a, j _ {0} ] = \tilde {r} _ {\sqrt {N} - j _ {1} - 1, \sqrt {N} - j _ {0} - 1} [ a ],
$$
where the entries of  $\ell_{\sqrt{N} -j_1 - 1}$  and  $\tilde{r}_{\sqrt{N} -j_1 - 1,\sqrt{N} -j_0 - 1}$  are defined as in (60) and (61). Now let  $\mathbf{C}_N\in \mathbb{R}^{N\times N}$  where
$$
\mathbf {C} _ {N} [ i, j ] = T _ {j} \left(\omega_ {N, i}\right) \tag {66}
$$
be the Chebyshev transform. Then analogous to Appendix D.4.1, we define
$$
\mathbf {R} _ {j _ {1}, j _ {1}} = \mathbf {C} _ {\sqrt {N}} \cdot \widetilde {\mathbf {R}} _ {j _ {1}, j _ {1}} \qquad \mathbf {L} ^ {\prime} = \mathbf {P C} _ {N} \cdot \widetilde {\mathbf {L}}.
$$
This allows us to give an algorithm the following construction algorithm for  $\mathbf{L}$  and  $\mathbf{R}$ .
Algorithm 4 BLOCKSC(  $\tilde{\mathbf{L}},\tilde{\mathbf{R}}$
Input:  $\widetilde{\mathbf{L}},\widetilde{\mathbf{R}}\in \mathbb{R}^{N\times \sqrt{N}}$  Output: Block diagonal matrices  $\mathbf{L},\mathbf{R}\in \mathbb{R}^{N\times N}$
1:  $\mathbf{L}^{\prime}\gets \mathbf{P}\cdot \mathbf{C}_{N}\cdot \tilde{\mathbf{L}}$
2: for  $a \gets 0$  to  $\sqrt{N} - 1$  do
3: \(\mathbf{L}_a^\prime \gets \overline{\mathbf{L}}^\prime [a\sqrt{N}:a\sqrt{N} +\sqrt{N} -1,:\]
4:  $\mathbf{L} \gets \mathrm{diag}(\mathbf{L}_0', \ldots, \mathbf{L}_{\sqrt{N} - 1}')$
5: for  $a \gets 0$  to  $\sqrt{N} \sim 1$  do
6: \(\mathbf{R}_a\gets \mathbf{C}_{\sqrt{N}}\cdot \mathbf{R}[a\sqrt{N}:a\sqrt{N} +\sqrt{N} -1,:\]
7:  $\mathbf{R} \gets \mathrm{diag}\left(\mathbf{R}_0, \dots, \mathbf{R}_{\sqrt{N} - 1}\right)$
8: return L, R
We use Algorithm 4 to specify another type of matrix  $\mathbf{M} \in \mathbb{R}^{N \times N}$ .
Lemma 10. Let
$$
\mathbf {M} = \mathbf {P L P R P}
$$
where  $\mathbf{L}$  and  $\mathbf{R}$  are outputs from BLOCKSC. Then each entry in  $\mathbf{M}$  is defined as
$$
\mathbf {M} [ i, j ] = \ell_ {\sqrt {N} - j _ {0} - 1} \left(\omega_ {N, i}\right) \cdot \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} \left(\omega_ {\sqrt {N}, i _ {0}}\right) \tag {67}
$$
Proof. Let  $\mathbf{M}_0 = \mathbf{PLPR}$ , then  $\mathbf{M} = \mathbf{M}_0\mathbf{P}$ . Then we have
$$
\mathbf {M} _ {0} [ (i _ {1}, i _ {0}), (j _ {1}, j _ {0}) ] = \ell_ {\sqrt {N} - j _ {1} - 1} (\omega_ {N, i}) \cdot \tilde {r} _ {\sqrt {N} - j _ {1} - 1, \sqrt {N} - j _ {0} - 1} (\omega_ {N, i _ {0}}).
$$
$$
\mathbf {M} [ i, j ] = \mathbf {M} _ {0} \mathbf {P} [ (i _ {1}, i _ {0}), (j _ {1}, j _ {0}) ] = \mathbf {M} _ {0} [ (i _ {1}, i _ {0}), (j _ {0}, j _ {1}) ].
$$
This gives us
$$
\mathbf {M} [ (i _ {1}, i _ {0}), (j _ {1}, j _ {0}) ] = \ell_ {\sqrt {N} - j _ {0} - 1} (\omega_ {N, i}) \cdot \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} (\omega_ {N, i _ {0}}).
$$
Next we will discuss the relation between matrices that we get from Lemma 10 and SC matrices.
D.5.4 Constructing  $\mathbf{M}'$  from M
Since  $\mathbf{C}_N \notin \mathcal{M}^{SC7}$ , Algorithm 4 is not a proper analog of Algorithm 2. In this section, we show how to convert  $\mathbf{M}$  produced from Algorithm 4 into a matrix with a block decomposable form. Recall from (63) that
$$
\mathbf {M} ^ {\prime} [ i, j ] = q _ {(j _ {1}, j _ {0}) _ {\sqrt {N}}} ^ {N} (\omega_ {N, i}) = \ell_ {\sqrt {N} - j _ {0} - 1} (Z) \cdot \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} \left(T _ {\sqrt {N}} (Z)\right). \tag {68}
$$
We note the distinction between the above and (67). Specifically, we note that  $\mathbf{M}$  is evaluated on two sets of evaluation points—the  $\ell_{\sqrt{N} - j_1 - 1}(Z)$  and  $\tilde{r}_{\sqrt{N} - j_1 - 1, \sqrt{N} - j_0 - 1}$  polynomials are evaluated over the  $N$  roots and  $\sqrt{N}$  roots of unity while  $\mathbf{M}'$  is only evaluated the  $N$ -th roots of unity. However, they are close due to the following property:
Lemma 11. Let  $T_{a}$  be a Chebyshev polynomial of the first kind of degree  $a$ , and define  $\omega_{N,i}$  as in (49). Then
$$
T _ {\sqrt {N}} \left(\omega_ {N, i}\right) = (- 1) ^ {i _ {1}} \cdot \omega_ {\sqrt {N}, i _ {0}}.
$$
$$
\begin{array}{l} T _ {\sqrt {N}} \left(\omega_ {N, i}\right) = \cos \left(\frac {\sqrt {N} \left(i _ {1} \sqrt {N} + i _ {0} + \frac {1}{2}\right) \pi}{N}\right) \\ = \cos \left(i _ {1} \pi + \frac {\pi \left(i _ {0} + \frac {1}{2}\right)}{\sqrt {N}}\right) \\ = (- 1) ^ {i _ {1}} \cos \left(\frac {\pi \left(i _ {0} + \frac {1}{2}\right)}{\sqrt {N}}\right) \\ = (- 1) ^ {i _ {1}} \cdot \omega_ {\sqrt {N}, i _ {0}}. \\ \end{array}
$$
In the above the first equality follows from (46).
Analogous to how we utilized roots of unity, Lemma 11 allows us to express our basis polynomials in terms of two related sets of evaluation points,  $\omega_{\sqrt{N},i_0}$  and  $\omega_{N,i}$ . The following lemma demonstrates how to translate between  $\mathbf{M}'$  and  $\mathbf{M}$ .
Lemma 12. For all  $i,j$
$$
\mathbf {M} ^ {\prime} [ i, j ] = (- 1) ^ {i _ {i} (\sqrt {N} - 1)} (- 1) ^ {i _ {i} j _ {1}} \cdot \mathbf {M} [ i, j ]. \tag {69}
$$
<sup>7</sup>We do not have a proof of this claim, but to us it seems unlikely that  $\mathbf{C}_N\in \mathcal{M}^{SC}$
Proof. By (68) we have
$$
\begin{array}{l} \mathbf {M} ^ {\prime} [ i, j ] = \ell_ {\sqrt {N} - j _ {0} - 1} (\omega_ {N, i}) \cdot \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} \left(T _ {\sqrt {N}} (\omega_ {N, i})\right) \\ = \ell_ {\sqrt {N} - j _ {0} - 1} \left(\omega_ {N, i}\right) \cdot \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} \left((- 1) ^ {i _ {1}} \omega_ {\sqrt {N}, i _ {0}}\right) \\ \end{array}
$$
where the second statement follows from Lemma 11. Then from (60) and (47) we get
$$
\begin{array}{l} \mathbf {M} ^ {\prime} [ i, j ] = \ell_ {\sqrt {N} - j _ {0} - 1} (\omega_ {N, i}) \cdot \sum_ {a = 0} ^ {\sqrt {N} - j _ {1} - 1} \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} [ a ] T _ {a} \left((- 1) ^ {i _ {1}} \omega_ {\sqrt {N}, i _ {0}}\right) \\ = \ell_ {\sqrt {N} - j _ {0} - 1} \left(\omega_ {N, i}\right) \cdot \sum_ {a = 0} ^ {\sqrt {N} - j _ {1} - 1} \cdot \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} [ a ] (- 1) ^ {i _ {1} a} T _ {a} \left(\omega_ {\sqrt {N}, i _ {0}}\right). \\ \end{array}
$$
Note from (61) that  $\tilde{r}_{\sqrt{N} -j_0 - 1,\sqrt{N} -j_1 - 1}[a] = 0$  if  $(\sqrt{N} -j_{1} - 1 - a)$  is odd. Then equivalently we get
$$
\begin{array}{l} \mathbf {M} ^ {\prime} [ i, j ] = \ell_ {\sqrt {N} - j _ {0} - 1} (\omega_ {N, i}) \cdot (- 1) ^ {i _ {1} (\sqrt {N} - j _ {1} - 1)} \\ \cdot \sum_ {a = 0} ^ {\sqrt {N} - j _ {1} - 1} \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} [ a ] (- 1) ^ {i _ {1} (\sqrt {N} - j _ {1} - 1 - a)} T _ {a} \left(\omega_ {\sqrt {N}, i _ {0}}\right) \\ = \ell_ {\sqrt {N} - j _ {0} - 1} (\omega_ {N, i}) \cdot (- 1) ^ {i _ {1} (\sqrt {N} - j _ {1} - 1)} \left(\sum_ {a = 0} ^ {\sqrt {N} - j _ {1} - 1} \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} [ a ] T _ {a} (\omega_ {\sqrt {N}, i _ {0}})\right) \\ = (- 1) ^ {i _ {1} (\sqrt {N} - j _ {1} - 1)} \ell_ {\sqrt {N} - j _ {0} - 1} (\omega_ {N, i}) \cdot \tilde {r} _ {\sqrt {N} - j _ {0} - 1, \sqrt {N} - j _ {1} - 1} \left(\omega_ {\sqrt {N}, i _ {0}}\right). \\ \end{array}
$$
Then from (67) we have
$$
\begin{array}{l} \mathbf {M} ^ {\prime} [ i, j ] = (- 1) ^ {i _ {1} (\sqrt {N} - j _ {1} - 1)} \mathbf {M} [ i, j ] \\ = (- 1) ^ {i _ {1} (\sqrt {N} - 1)} (- 1) ^ {i _ {1} j _ {1}} \mathbf {M} [ i, j ]. \\ \end{array}
$$
Block Matrix Multiplication for Structured Causal Matrices Lemma 12 shows us how to compute  $\mathbf{M}'$  from  $\mathbf{M}$ . However, this does not mean that the matrix vector multiplication problem for  $\mathbf{M}'$  can be implemented efficiently. In this section we show how to perform matrix vector multiplication of  $\mathbf{M}'$  with any  $\mathbf{u} \in \mathbb{R}^{N}$  from two matrix-vector multiplications of  $\mathbf{M}$  (and so these operations are indeed efficient).
The key to our approach involves considering the parity of each block index  $0 \leq i_1 < \sqrt{N}$ . Let us define a map  $\mathrm{MIX} : \mathbb{R}^{\left\lfloor \frac{N}{2} \right\rfloor} \times \mathbb{R}^{\left\lceil \frac{N}{2} \right\rceil} \mapsto \mathbb{R}^{N}$  such that  $\mathrm{MIX}(\mathbf{u}_0, \mathbf{u}_1) = \mathbf{u}$  where
$$
\mathbf {u} [ i ] = \mathbf {u} _ {i _ {1}} \mod 2 [ i / 2 ].
$$
We use this map to show that  $\mathbf{M}'\mathbf{u}$  and  $\mathbf{u}^\top \mathbf{M}'$  can be computed efficiently.
Lemma 13. For any  $\mathbf{u} \in \mathbb{R}^N$  and  $\mathbf{M} \in \mathbb{R}^{N \times N}$ ,  $\mathbf{M}'\mathbf{u}$  can be computed via two matrix-vector multiplications:  $\mathbf{M} \cdot \mathrm{MIX}(\mathbf{u}_0, \mathbf{0})$  and  $\mathbf{M} \cdot \mathrm{MIX}(\mathbf{0}, \mathbf{u}_1)$ .
Proof. For  $0 \leq i = i_1\sqrt{N} + i_0 < N$ ,  $0 \leq j = j_1\sqrt{N} + j_0 < N$ , let  $\mathbf{D} \in \mathbb{R}^{N \times N}$  be the diagonal matrix defined such that  $\mathbf{D}[i, i] = (-1)^{i_1(\sqrt{N} - 1)}$ . Lemma 12 implies that
$$
\mathbf {M} ^ {\prime} [ i, j ] = \left\{ \begin{array}{l l} \mathbf {D M} [ i, j ] & \text {i f i _ {1} i s e v e n} \\ \mathbf {D M} [ i, j ] & \text {i f i _ {1} i s o d d , j _ {1} i s e v e n} \\ - (\mathbf {D M} [ i, j ]) & \text {o t h e r w i s e .} \end{array} \right.
$$
We want to shift the  $(-1)$  to  $\mathbf{u}$ . Compute  $\mathbf{z}_0 = \mathbf{M} \cdot (\mathrm{MIX}(\mathbf{u}_0, \mathbf{0}) + \mathrm{MIX}(\mathbf{0}, \mathbf{u}_1))$ ,  $\mathbf{z}_1 = \mathbf{M} \cdot (\mathrm{MIX}(\mathbf{u}_0, \mathbf{0}) - \mathrm{MIX}(\mathbf{0}, \mathbf{u}_1))$ . Define  $\mathbf{y}'$  such that
$$
\mathbf {y} ^ {\prime} [ j ] = \left\{ \begin{array}{l l} \mathbf {z} _ {0} [ j ] & \text {i f} j _ {1} \text {i s e v e n} \\ \mathbf {z} _ {1} [ j ] & \text {i f} j _ {1} \text {i s o d d} \end{array} \right..
$$
It can be verified that  $\mathbf{D} \cdot \mathbf{y}' = \mathbf{M}'\mathbf{u}$ , which completes the proof.
We now give the analogous result for  $\mathbf{u}^{\top}\mathbf{M}^{\prime}$
Lemma 14. For any  $\mathbf{u} \in \mathbb{R}^N$  and  $\mathbf{M} \in \mathbb{R}^{N \times N}$ ,  $\mathbf{u}^\top \mathbf{M}'$  can be computed via two matrix-vector multiplications:  $\mathrm{MIX}(\mathbf{u}_0, \mathbf{0})^\top \mathbf{M}$  and  $\mathrm{MIX}(\mathbf{0}, \mathbf{u}_1)^\top \mathbf{M}$ .
Proof. For  $0 \leq i = i_1\sqrt{N} + i_0 < N$ ,  $0 \leq j = j_1\sqrt{N} + j_0 < N$ , let  $\mathbf{D} \in \mathbb{R}^{N \times N}$  be the diagonal matrix defined such that  $\mathbf{D}[i, i] = (-1)^{i_1(\sqrt{N} - 1)}$ , and  $\mathbf{u}' = \mathbf{u}^\top \mathbf{D} = \mathrm{MIX}(\mathbf{u}_0', \mathbf{u}_1')$ . Lemma 12 implies that
We want to shift the  $(-1)$  to  $\mathbf{u}'$ . Compute  $\mathbf{z}_0 = \left(\mathrm{MIX}(\mathbf{u}_0', \mathbf{0})^\top + \mathrm{MIX}(\mathbf{0}, \mathbf{u}_1')^\top\right) \cdot \mathbf{M}$ ,  $\mathbf{z}_1 = \left(\mathrm{MIX}(\mathbf{u}_0', \mathbf{0})^\top - \mathrm{MIX}(\mathbf{0}, \mathbf{u}_1')^\top\right) \cdot \mathbf{M}$ . If  $\mathbf{y} = \mathbf{M}'\mathbf{u}$ , then one can check that
$$
\mathbf {y} [ j ] = \left\{ \begin{array}{l l} \mathbf {z} _ {0} [ j ] & \text {i f} j _ {1} \text {i s e v e n} \\ \mathbf {z} _ {1} [ j ] & \text {i f} j _ {1} \text {i s o d d} \end{array} \right.,
$$
Lemma 13 implies that computing  $\mathbf{M}_N\mathbf{k}$  and  $\mathbf{M}_N\mathbf{u}$  in (55) can be done efficiently. However, we do not know how to compute  $\mathbf{M}_N^{-1}\mathbf{y}$  for any arbitrary  $\mathbf{y}$ . We address this partially in the next section.
D.5.5 Inverse of Chebyshev Transform
In this subsection we show how computing  $\mathbf{C}_N^{-1}$  can be done efficiently.
Lemma 15. Let  $\mathbf{C}_N\in \mathbb{R}^{N\times N}$  be defined as (66). Then
$$
\mathbf {C} _ {N} ^ {- 1} = \mathbf {C} _ {N} ^ {\top} \cdot d i a g (1 / N, 2 / N, \dots , 2 / N).
$$
Proof. Follows since  $\mathbf{C}_N(\mathbf{C}_N)^\top = \mathrm{diag}(N,N / 2,N / 2,\dots ,N / 2)$  [71].
Given the above, the goal then is to compute  $\mathbf{C}_N^\top \cdot \mathbf{u}$  or equivalently  $\mathbf{u}^\top \mathbf{C}_N$  for any  $\mathbf{u} \in \mathbb{R}^N$ . Note that is sufficient to show that
$$
\mathbf {C} _ {N} = \overline {{\mathbf {C}}} _ {N} + \overline {{\mathbf {S}}} _ {N}
$$
for  $\overline{\mathbf{C}}_N, \overline{\mathbf{S}}_N \in \mathbb{R}^{N \times N}$  such that  $\mathbf{u}^\top \overline{\mathbf{C}}_N$  and  $\mathbf{u}^\top \overline{\mathbf{S}}_N$  can be computed with two invocations of Lemma 14. Indeed we have
$$
\begin{array}{l} \mathbf {C} _ {N} [ i, j ] = \cos \left(\frac {(i + \frac {1}{2}) (j _ {1} \sqrt {N} + j _ {0}) \pi}{N}\right) \\ = \cos \left(\frac {\pi (i + \frac {1}{2}) j _ {1}}{\sqrt {N}} + \frac {\pi (i + \frac {1}{2}) j _ {0}}{N}\right), \\ = \cos \left(\frac {\pi (i + \frac {1}{2}) j _ {1}}{\sqrt {N}}\right) \cos \left(\frac {\pi (i + \frac {1}{2}) j _ {0}}{N}\right) - \sin \left(\frac {\pi (i + \frac {1}{2}) j _ {1}}{\sqrt {N}}\right) \sin \left(\frac {\pi (i + \frac {1}{2}) j _ {0}}{N}\right). \tag {70} \\ \end{array}
$$
We use this to define  $\overline{\mathbf{C}}_N$  and  $\widetilde{\mathbf{S}}_N$ , along with two additional matrices  $\widetilde{\mathbf{C}}_N, \widetilde{\mathbf{S}}_N \in \mathbb{R}^{N \times N}$  such that for  $\overline{\mathbf{C}}_N$  and  $\widetilde{\mathbf{C}}_N$ :
$$
\begin{array}{l} \overline {{\mathbf {C}}} _ {N} [ i, j ] = \cos \left(\frac {\pi (i + \frac {1}{2}) j _ {1}}{\sqrt {N}}\right) \cos \left(\frac {\pi (i + \frac {1}{2}) j _ {0}}{N}\right) \\ = \cos \left(\pi i _ {1} j _ {1} + \frac {\pi (i _ {0} + \frac {1}{2}) j _ {1}}{\sqrt {N}}\right) \cos \left(\frac {\pi (i + \frac {1}{2}) j _ {0}}{N}\right) \\ = (- 1) ^ {i _ {1} j _ {1}} \cos \left(\frac {\pi \left(i _ {0} + \frac {1}{2}\right) j _ {1}}{\sqrt {N}}\right) \cos \left(\frac {\pi \left(i + \frac {1}{2}\right) j _ {0}}{N}\right) \stackrel {\text {d e f}} {=} (- 1) ^ {i _ {1} j _ {1}} \widetilde {\mathbf {C}} _ {N} [ i, j ], \tag {71} \\ \end{array}
$$
and similarly for  $\overline{\mathbf{S}}_N$  and  $\widetilde{\mathbf{S}}_N$ :
$$
\begin{array}{l} \overline {{\mathbf {S}}} _ {N} [ i, j ] = \sin \left(\frac {\pi (i + \frac {1}{2}) j _ {1}}{\sqrt {N}}\right) \sin \left(\frac {\pi (i + \frac {1}{2}) j _ {0}}{N}\right) \\ = (- 1) ^ {i _ {1} j _ {1}} \sin \left(\frac {\pi \left(i _ {0} + \frac {1}{2}\right) j _ {1}}{\sqrt {N}}\right) \sin \left(\frac {\pi \left(i + \frac {1}{2}\right) j _ {0}}{N}\right) \stackrel {\text {d e f}} {=} (- 1) ^ {i _ {1} j _ {1}} \widetilde {\mathbf {S}} _ {N} [ i, j ]. \tag {72} \\ \end{array}
$$
We summarize these results into the following lemma.
Lemma 16. Define  $\overline{\mathbf{C}}_N$ ,  $\widetilde{\mathbf{C}}_N$ ,  $\overline{\mathbf{S}}_N$  and  $\widetilde{\mathbf{S}}_N$  as in (71) and (72), respectively. Then
$$
\mathbf {C} _ {N} = \overline {{\mathbf {C}}} _ {N} - \overline {{\mathbf {S}}} _ {N}
$$
where  $\widetilde{\mathbf{C}}_N$  and  $\widetilde{\mathbf{S}}_N$  are of the form (67).
Finally Lemma 14 and Lemma 16 imply the following:
Theorem 14. For any  $\mathbf{u} \in \mathbb{R}^N$ ,
$$
\mathbf {y} = \mathbf {u} ^ {\top} \mathbf {C} _ {N}
$$
can be computed from four calls matrix vector multiplication with Monarch matrices.
Proof Sketch for Theorem 14 Lemma 16 tells us that  $\mathbf{u}^{\top}\overline{\mathbf{C}}_N$  and  $\mathbf{u}^{\top}\overline{\mathbf{S}}_N$  are sufficient to compute  $\mathbf{u}^{\top}\mathbf{C}_N$ , and (72) shows us that  $\widetilde{\mathbf{C}}_N, \widetilde{\mathbf{S}}_N$  are Monarch matrices that allows  $\mathbf{u}^{\top}\overline{\mathbf{C}}_N$  and  $\mathbf{u}^{\top}\overline{\mathbf{S}}_N$  to be computed from two matrix-vector multiplications with  $\widetilde{\mathbf{C}}_N$  and  $\widetilde{\mathbf{S}}_N$ , respectively. Then the claim follows from applying Lemma 14 twice.
D.6 Multivariate MONARCH MIXER with Block Size  $= \sqrt[p]{N}$
In this section, we generalize Theorem 1, Theorem 2, and Theorem 3 to arbitrary  $p \geq 2$ . In Appendix D.6.1, we set up notation. We then generalize Theorem 1 and Theorem 2 to general  $p$  in Appendix D.6.2 and then generalize Theorem 3 in Appendix D.6.3. In Appendix D.6.4 we connect definition of  $p$ -variate Monarch in Appendix D.6.2 to the one defined in Section 3. Finally in Appendix D.6.5 we discuss some extensions and generalizations.
D.6.1 Notation
We will use notation from Appendix D.5.1.
Fix an integer  $p \geq 2$ . We will be working with indices  $\mathbf{j}, \mathbf{i}$  where  $\mathbf{j} = (j_0, \ldots, j_{p-1})$  and  $\mathbf{i} = (i_0, \ldots, i_{p-1})$  with  $0 \leq i_a, j_a < \sqrt[p]{N}$  for every  $0 \leq a < p$ . We will denote the set of all sub-indices as  $[0, \sqrt[p]{N})^p$ , and the operator  $\preceq$  to denote the lexicographical ordering of vectors in  $[0, \sqrt[p]{N})^p$ .
For any  $0 \leq b' \leq M \leq N$  such that  $b'$  divides  $M$  and  $M$  divides  $N$ , define the following permutation matrices:
$$
\mathbf{P}_{b^{\prime},M,N} = \operatorname {diag}\left(\underbrace{\mathbf{P}_{b^{\prime},M,\ldots,\mathbf{P}_{b^{\prime},M}}}_{\substack{\frac{N}{M}\text{times}}}\right).
$$
Note that  $\mathbf{P}_{b',N,N}$  is exactly the same as  $\mathbf{P}_{b',N}$  from earlier.
If we use  $\sigma(b, M, N)$  to denote the corresponding permutation then it takes the input  $(i_{p-1}, \ldots, i_0)_b$  and maps it to  $(i_{p-1}, \ldots, i_{p'}, i_{p'-2}, \ldots, i_0, i_{p'})_b$  (where  $p = \log_b N$  and  $p' = \log_b M$ ). I.e.  $\sigma(b, M, N)$  does not change the first  $p - p'$  sub-indices and then does a left rotation on the remaining  $p'$  sub-indices.
For the rest of the section, assume  $b = \sqrt[p]{N}$  and then consider the following 'sub-index reversal' permutation matrix:
$$
\mathbf {P} _ {b, b ^ {p}} ^ {R} = \prod_ {a = 0} ^ {p - 2} \mathbf {P} _ {b ^ {p - a - 1}, b ^ {p - a}, b ^ {p}}.
$$
If  $\sigma^R (b,N)$  is the permutation corresponding to the permutation matrix above, then  $\sigma^R (b,N)$  maps  $(i_{p - 1},\ldots ,i_0)\mapsto (i_0,\ldots ,i_{p - 1})$
D.6.2 Generalizing Theorem 1 and Theorem 2
For a specific  $0 \leq a < p$ , let
$$
\ell_ {\mathbf {j}, \mathbf {i}} ^ {(a)} (X _ {a}) = \sum_ {m = 0} ^ {\sqrt [ p ]{N} - 1} \ell_ {(j _ {a + 1}, \dots , j _ {p - 1}), (i _ {0}, \dots , i _ {a - 1})} ^ {(a)} [ m ] \cdot T _ {m} (X _ {a}) \tag {73}
$$
be an arbitrary polynomial of degree  $< \sqrt[p]{N}$  in the Chebyshev polynomial basis (see (47)).
We will be interested in the evaluations of the above polynomials over the set
$$
A \stackrel {\text {d e f}} {=} \left(\omega_ {\bar {\mathcal {V}} \overline {{N}}, 0}, \dots , \omega_ {\bar {\mathcal {V}} \overline {{N}}, \bar {\mathcal {V}} \overline {{N}} - 1}\right), \tag {74}
$$
where  $\omega_{\sqrt{N},i}$  is defined as in  $(\ref{eq:1})$
The  $p$ -variate version of Monarch matrices  $\mathbf{M}' \in \mathbb{R}^{N \times N}$  as follows. (See Appendix D.6.4 to see how these are exactly related to the definition in (1)). For every row index  $\mathbf{i} \in [0, \sqrt[p]{N})^p$  and column index  $\mathbf{j} \in [0, \sqrt[p]{N})^p$ , we have
$$
\mathbf {M} ^ {\prime} [ \mathbf {i}, \mathbf {j} ] = \prod_ {a = 0} ^ {p - 1} \ell_ {\mathbf {j}, \mathbf {i}} ^ {(a)} \left(\omega_ {\bar {V} \bar {N}, i _ {a}}\right). \tag {75}
$$
To express the above in terms of a polynomial basis we will need the following definition. For any  $0 \leq a < p$  and  $\mathbf{i}$  define the Lagrange basis polynomial  $\Delta_{\mathbf{i}}^{(a)}(X_0, \ldots, X_{a-1})$  such that for any  $0 \leq m_0, \ldots, m_{a-1} < \sqrt[p]{N}$ , we have
$$
\Delta_ {\mathbf {i}} ^ {(a)} \left(\omega_ {\sqrt {N}, m _ {0}}, \ldots , \omega_ {\sqrt {N}, m _ {a - 1}}\right) = \left\{ \begin{array}{l l} 1 & \text {i f} i _ {0} = m _ {0}, i _ {1} = m _ {1}, \ldots , i _ {a - 1} = m _ {a - 1} \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
We use the above to convert the polynomials in (73) to not depend on the sub-indices in  $\mathbf{i}$  (at least for the definition in (75)). For every  $\mathbf{j}$  and  $0\leq a < p$ , define:
$$
\ell_{\mathbf{j}}^{(a)}\left(X_{0},\ldots ,X_{a}\right) = \sum_{\substack{\mathbf{i} = (i_{0},\ldots ,i_{a - 1},\mathbf{0}_{p - a}),i_{0},\ldots ,i_{a - 1}\in [0,\overline{\mathbb{V} / N})}}\Delta_{\mathbf{i}}^{(a)}\left(X_{0},\ldots ,X_{a - 1}\right)\cdot \ell_{\mathbf{j},\mathbf{i}}^{(a)}\left(X_{a}\right).
$$
Note that the summation fixes the last  $p - a$  sub-indices in  $\mathbf{i}$  since the definition of  $\ell_{\mathbf{j},\mathbf{i}}^{(a)}(X_a)$  only depends on  $(i_0,\ldots ,i_{a - 1})$ . This implies that for any  $0\leq i_0,\dots ,i_a < \sqrt[p]{N}$ , we have