SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, we generalize our results to arbitrary  $p \geq 2$  in Appendix D.6. We would like to point out that to get a causal map (in Theorem 17) we need to 'embed' input vectors of size  $n$  into vectors of size  $N = 2^p \cdot n + O\left(n^{1 - 1 / p}\right)$ . For  $p = 2$ , we avoided the blowup of  $2^2 = 4$  with a blowup of 2 instead (via Theorem 3). Whether this is possible to do (i.e. have a blowup of 2 instead of  $2^p$ ) for  $p > 2$  is an interesting direction for future work. Further, the matrices that lead to causal map can be represented with  $O\left(pN^{2 / p}\right)$  parameters while the matrices in Theorem 3 use more parameters. Extending the causal map for  $p > 2$  that uses  $O\left(N^{1 + \frac{1}{p}}\right)$  parameters is an exciting direction for future work.
D.1 Background and Notation
We collect known facts and definitions about multi-variate polynomials in Appendix D.1.1 and recall some notation from [14] in Appendix D.1.2. These will be needed throughout this appendix section.
D.1.1 Multi-variate Polynomials
Basic Definitions Let  $p \geq 1$  be an integer. We recollect some definitions on  $p$ -variate polynomials (over  $\mathbb{R}$ ) in variables  $X_0, \ldots, X_{p-1}$ . When  $p \in \{1,2\}$ , we will use variables in  $\{X, Y, Z\}$  for notational simplicity.
We will use  $\mathbf{X}$  to denote the vector of variables  $(X_0, \ldots, X_{p-1})$ . Further for  $\mathbf{j} \in \mathbb{Z}_{\geq 0}^p$ , we use the notation
$$
\mathbf {X} ^ {\mathbf {j}} = \prod_ {a = 0} ^ {p - 1} X _ {a} ^ {j _ {a}}.
$$
$\mathbf{X}^{\mathrm{j}}$  is a (standard basis) monomial, where  $\mathbf{j} = (j_0,\dots ,j_{p - 1})$
A generic  $p$ -variate polynomial is defined as (with standard monomial representation)
$$
q(\mathbf{X}) = \sum_{\mathbf{j}\in \mathbb{Z}_{\geq 0}^{p}}q_{\mathbf{j}}\cdot \mathbf{X}^{\mathbf{j}},
$$
where the coefficient  $q_{\mathrm{j}} \in \mathbb{R}$ .
We will need the following notion of degrees:
Definition 2 (Degree). Let  $0 \leq a < p$ . The degree of  $X_{a}$  in  $\mathbf{X}^{\mathbf{j}}$  (with  $\mathbf{j} = (j_0, \dots, j_{p-1})$ ) is  $j_a$ . The degree of  $X_{a}$  of  $q(\mathbf{X})$ , denoted by  $\deg_{X_a}(q)$  is the maximum degree of  $X_{a}$  over all monomials  $\mathbf{X}^{\mathbf{j}}$  with  $q_{\mathbf{j}} \neq 0$ .
Note that for  $p = 1$  the above coincides with the usual notion of degree of a univariate polynomial  $q(Z)$ , in which case we just use  $\deg(q(Z))$  to denote  $\deg_Z(q(Z))$ .
We will need the notion of taking mod of a  $p$ -variate polynomial with  $p$ -tuple of polynomials. The notion of mod is well defined for a univariate polynomial (which we will assume as a given below) but in general for arbitrary  $p$ -variate polynomials  $q(\mathbf{X})$  and  $q^{\prime}(\mathbf{X})$ , the operation  $q(\mathbf{X}) \mod q^{\prime}(\mathbf{X})$  is not well defined. However, we will only need the following restricted operation:
Definition 3. Let  $p \geq 1$ . Fix a  $p$ -tuple of polynomials  $R_0(X_0), \ldots, R_{p-1}(X_{p-1})$ . Then for any  $\mathbf{j} \in \mathbb{Z}_{\geq 0}^p$ , we define
$$
\mathbf {X} ^ {\mathrm {j}} \mod \left(R _ {0} \left(X _ {0}\right), \dots , R _ {p - 1} \left(X _ {p - 1}\right)\right) = \prod_ {a = 0} ^ {p - 1} \left(X ^ {j _ {a}} \mod \left(R _ {a} \left(X _ {a}\right)\right)\right).
$$
For a general polynomial  $p(\mathbf{X})$
$$
p (\mathbf {X}) \mod \left(R _ {0} \left(X _ {0}\right), \dots , R _ {p - 1} \left(X _ {p - 1}\right)\right)
$$
is defined by extending the definition for  $\mathbf{X}^{\mathbf{j}}$  by linearity.
Polynomial Evaluation Given a  $p$ -variate polynomial  $q(\mathbf{X})$  and an point  $\mathbf{a} \in \mathbb{R}^p$ , the evaluation of  $q$  at  $\mathbf{a}$  denoted by  $q(\mathbf{a})$  is evaluation of  $q$  as a function at  $\mathbf{a}$ .
Given subsets  $S_{a} \subseteq \mathbb{C}$ , we define  $q(\mathbf{X})$  evaluated at  $\times_{a=0}^{p-1} S_{a}$  as the vector of values  $q(\mathbf{a})$  over all  $\mathbf{a} \in \times_{a=0}^{p-1} S_{a}$ .
In this paper, we will in many cases evaluate polynomials at the appropriate roots of unity. Specifically for an integer  $N$ , we will define
$$
\omega_ {N} = e ^ {2 \pi \iota / N}
$$
and note that the  $N$ th roots of unity is the set  $\{\omega_N^i | 0 \leq i < N\}$ .
Polynomial Interpolation We now recall univariate and bivariate polynomial interpolation results (proved via the Lagrange basis), which we will use in later subsections.
Theorem 4. Let  $D \geq 1$  be an integer. Given  $y_{i}$  for  $0 \leq i < D$  and  $\alpha_{i}$  for  $0 \leq i < D$  there exists a unique univariate polynomial  $P(X)$  with  $\deg(P) < D$ , such that for all  $0 \leq i < D$ ,
$$
P \left(\alpha_ {i}\right) = y _ {i}. \tag {7}
$$
Proof. This proof is based on the Wikipedia entry for Lagrange polynomials [47].
Given a sequence of values  $\alpha_{i}$  for  $0\leq i < D$  s.t.  $\alpha_{i}\neq \alpha_{j},i\neq j$  , the Lagrange basis for polynomials of degree  $<  D$  for these values is the set of each polynomials  $\{p_0(X),p_1(X),\ldots p_{D - 1}(X)\}$  each of degree  $D - 1$  . Each basis polynomial are defined as:
$$
p _ {i} (X) = \frac {X - \alpha_ {0}}{\alpha_ {i} - \alpha_ {0}} \dots \frac {X - \alpha_ {i - 1}}{\alpha_ {i} - \alpha_ {i - 1}} \cdot \frac {X - \alpha_ {i + 1}}{\alpha_ {i} - \alpha_ {i + 1}} \dots \frac {X - \alpha_ {D - 1}}{\alpha_ {i} - \alpha_ {D - 1}} = \prod_ {\substack {0 \leq j <   D \\ j \neq i}} \frac {X - \alpha_ {j}}{\alpha_ {i} - \alpha_ {j}}. \tag{8}
$$
$$
p _ {i} \left(\alpha_ {j}\right) = \left\{ \begin{array}{l l} 1 & \text {f o r} j = i \\ 0 & \text {o t h e r w i s e} \end{array} . \right. \tag {9}
$$
The Lagrange interpolating polynomial for those nodes through the corresponding values  $y_{i}$  for  $0 \leq i < D$  is the linear combination:
$$
P (X) = \sum_ {i = 0} ^ {D - 1} y _ {i} \cdot p _ {i} (X). \tag {10}
$$
By (9), for all  $0 \leq i < D$ :
$$
P \left(\alpha_ {i}\right) = y _ {i}. \tag {11}
$$
Finally, the interpolating polynomial is unique. Assume there is another polynomial  $M(X)$  of degree  $< D$  such that  $M(\alpha_i) = y_i$  for all  $0 \leq i < D$ . Then the difference  $M(X) - P(X)$  is 0 at  $D$  distinct points  $\alpha_i$  for  $0 \leq i < D$ . And the only polynomials of degree  $< D$  with more than  $D - 1$  roots is the 0 polynomial. So,  $M(X) = P(X)$ .
Theorem 5. Let  $D_X, D_Y \geq 1$  be integers. Given values  $y_{ij}$  for  $0 \leq i < D_X, 0 \leq j < D_Y$  and  $D_X$  distinct points  $(\alpha_0, \ldots, \alpha_{D_X - 1})$ ,  $D_Y$  distinct points  $(\beta_0, \ldots, \beta_{D_Y - 1})$  there exists a unique bivariate polynomial  $P(X,Y)$  with  $\deg_X(P) < D_X$ ,  $\deg_Y(P) < D_Y$ , such that for all  $0 \leq i < D_X, 0 \leq j < D_Y$ :
$$
P \left(\alpha_ {i}, \beta_ {j}\right) = y _ {i j}. \tag {12}
$$
$$
P (X, Y) = \sum_ {j = 0} ^ {m} \sum_ {i = 0} ^ {n} y _ {i j} \cdot p _ {i} (X) \cdot \bar {p} _ {j} (Y), \tag {13}
$$
where  $p_i$  and  $\bar{p}_j$  are Lagrange basis polynomials defined in the proof of Theorem 4 such that for  $0 \leq i, k < D_X$ ,
$$
p _ {i} \left(\alpha_ {k}\right) = \left\{ \begin{array}{l l} 1 & \text {f o r} i = k \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {14}
$$
and for  $0 \leq j, \ell < D_{Y}$
$$
\bar {p} \left(\beta_ {\ell}\right) = \left\{ \begin{array}{l l} 1 & \text {f o r} k = \ell \\ 0 & \text {o t h e r w i s e} \end{array} . \right. \tag {15}
$$
From above, we have for all  $i,j,k,\ell$
$$
p _ {i} \left(\alpha_ {k}\right) \cdot \bar {p} _ {j} \left(\beta_ {\ell}\right) = \left\{ \begin{array}{l l} 1 & \text {f o r} i = k \text {a n d} j = \ell \\ 0 & \text {o t h e r w i s e} \end{array} . \right. \tag {16}
$$
Then, for all  $0 \leq i < D_X$ ,  $0 \leq j < D_Y$ :
$$
P \left(\alpha_ {i}, \beta_ {j}\right) = y _ {i j}. \tag {17}
$$
By definition of Lagrange basis polynomials,  $\deg_X(P) < D_X$  and  $\deg_Y(P) < D_Y$ .
Finally, the interpolating polynomial is unique. Assume there is another polynomial  $M(X,Y)$  with  $\deg_X(M) < D_X$  and  $\deg_Y(M) < D_Y$  such that  $M(\alpha_i,\beta_j) = y_{ij}$  for all  $0\leq i < D_X$  and  $0\le j < D_Y$ . Then the difference  $M(X,Y) - P(X,Y)$  is 0 at  $D_X\cdot D_Y$  distinct points,  $(\alpha_{i},\beta_{j})$  for  $0\leq i < D_X,0\leq j < D_Y$ . And the only polynomial with  $\deg_X < D_X$  and  $\deg_Y < D_Y$  that has  $D_X\cdot D_Y$  roots is the 0 polynomial.
D.1.2 Notation
Here we recall notation we will use from [14].
1. The class of Monarch matrices is defined in appendix C of [14] as  $\mathcal{M}^{(b,N)}$  which are  $N\times N$  matrices with block size  $b$  for any integer  $0\leq b\leq N$  that divides  $N$ . When  $b = \sqrt{N}$  we drop  $b$  from the notation giving  $(i_1,i_0)$  and  $(j_{1},j_{0})$ . For example, this is used in Proof of Corollary 2.
2. Row index  $i$  can be represented as  $(i_1, i_0)_b$ . Which gives  $i = i_1b + i_0$ .
3. Similarly, column index  $j$  can be represented as  $(j_{1}, j_{0})_{b}$ . Which gives  $j = j_{1}b + j_{0}$ . Note that when  $b = \sqrt{N}$ ,  $j_{1} = k(j)$  and  $j_{0} = m(j)$ . We choose to use the  $(j_{1}, j_{0})$  notation here since that notation is easier to generalize for  $p > 2$ .
4.  $\overline{\mathbf{L}}\in \mathcal{DB}^{(b,N)}$  is an  $N\times N$  matrix with  $b\times b$  blocks that are all diagonal matrices.
5.  $\mathbf{R} \in \mathcal{BD}^{(b,N)}$  meaning it's a block diagonal  $N \times N$  matrix with block size  $b \times b$ .
6. We have a class of permutation matrices defined as  $\sigma_{(b,N)}(i) = i_0\cdot \frac{N}{b} +i_1$ . This can be denoted by an  $N\times N$  matrix,  $P_{(b,N)}$ , where the  $i^{\mathrm{th}}$  row is  $\mathbf{e}_{\sigma (b,N)(i)}$ .
7. We'll use  $i$  or pair notation  $(i_1, i_0)_b$  to denote the rows, and  $j$  or pair notation  $(j_1, j_0)_b$  to denote columns. It should be clear from context which one we're using.
For any  $0 \leq j_1 < \sqrt{N}$ , let  $\ell_{j_1}(X,Y)$  be an arbitrary bivariate polynomial with  $\deg_X(\ell_{j_1})$ ,  $\deg_Y(\ell_{j_1}) < \sqrt{N}$ .
For any  $0 \leq j_1, j_0 < \sqrt{N}$ , let  $r_{j_1,j_0}(Y)$  be an arbitrary univariate polynomial of degree  $< \sqrt{N}$ .
Let  $A = (\alpha_0, \ldots, \alpha_{\sqrt{N} - 1})$ ,  $B = (\beta_0, \ldots, \beta_{\sqrt{N} - 1})$  each be a sequence of distinct eval points. Note that  $A$  and  $B$  need not be disjoint.
From the proof of Theorem 3 in the Appendix C of the Monarch paper [14] we get,
$$
\begin{array}{l} \mathbf {L} = \mathbf {P} _ {(b, N)} \cdot \overline {{\mathbf {L}}} \cdot \mathbf {P} _ {(b, N)} ^ {\top} \\ = \mathbf {P} _ {(b, N)} \cdot \overline {{\mathbf {L}}} \cdot \mathbf {P} _ {(\frac {N}{b}, N)}. \\ \end{array}
$$
$$
\begin{array}{l} \overline {{\mathbf {L}}} = \mathbf {P} _ {(b, N)} ^ {\top} \cdot \mathbf {L} \cdot \mathbf {P} _ {(b, N)} \\ = \mathbf {P} _ {\left(\frac {N}{b}, N\right)} \cdot \mathbf {L} \cdot \mathbf {P} _ {(b, N)}. \\ \end{array}
$$
Define  $\mathcal{DB}$  and  $\mathcal{BD}$  as set of all such  $\overline{\mathbf{L}}$  and  $\mathbf{R}$  matrices over  $\mathbb{R}^{N\times N}$  where if  $i_0\neq k_0$
$$
\overline {{\mathbf {L}}} _ {i _ {1}, k _ {1}} \left[ i _ {0}, k _ {0} \right] \stackrel {\text {d e f}} {=} \overline {{\mathbf {L}}} \left[ \left(i _ {1}, i _ {0}\right) _ {\sqrt {N}}, \left(k _ {1}, k _ {0}\right) _ {\sqrt {N}} \right] = 0 \tag {18}
$$
and if  $k_{1}\neq j_{1}$
$$
\mathbf {R} _ {k _ {1}, j _ {1}} \left[ k _ {0}, j _ {0} \right] \stackrel {\text {d e f}} {=} \mathbf {R} \left[ \left(k _ {1}, k _ {0}\right) _ {\sqrt {N}}, \left(j _ {1}, j _ {0}\right) _ {\sqrt {N}} \right] = 0. \tag {19}
$$
Pictorially,  $\overline{\mathbf{L}}$  and  $\mathbf{R}$  look as follows:
In [14], Monarch matrices with block size  $b = \sqrt{N}$ ,  $\mathbf{M}' = \overline{\mathbf{L}} \cdot \mathbf{R}$ , and thus for all  $0 \leq i_1, i_0, j_1, j_0 < \sqrt{N}$ :
$$
\mathbf {M} ^ {\prime} \left[ \left(i _ {1}, i _ {0}\right) _ {\sqrt {N}}, \left(j _ {1}, j _ {0}\right) _ {\sqrt {N}} \right] = \overline {{\mathbf {L}}} _ {i _ {1}, j _ {1}} \left[ i _ {0}, i _ {0} \right] \cdot \mathbf {R} _ {j _ {1}, j _ {1}} \left[ i _ {0}, j _ {0} \right]. \tag {20}
$$
We note that our definition of Monarch matrix  $\mathbf{M}$  in Section 3 is slightly different in that  $\mathbf{M} = \mathbf{M}'\mathbf{P}$  with  $\mathbf{M}'$  as defined in [14].
D.2 Monarch Matrices and Bivariate Polynomial Evaluation
Given polynomials  $\ell_{j_1}(X,Y)$  for  $0\leq j_{1} < \sqrt{N}$ , polynomials  $r_{j_1,j_0}(Y)$  for  $0\leq j_{1},j_{0} < \sqrt{N}$ , evaluation points  $A = (\alpha_{0},\dots,\alpha_{\sqrt{N} -1})B = (\beta_{0},\dots,\beta_{\sqrt{N} -1})$  (as in Appendix D.1.2), define the matrices  $\overline{\mathbf{L}}\in \mathcal{DB}^{\sqrt{N},N}$  and  $\mathbf{R}\in \mathcal{BD}^{\sqrt{N},N}$  as:
- For every  $0 \leq j_{1}, i_{1}, i_{0} < \sqrt{N}$ :
$$
\overline {{\mathbf {L}}} _ {i _ {1}, j _ {1}} \left[ i _ {0}, i _ {0} \right] \leftarrow \ell_ {j _ {1}} \left(\alpha_ {i _ {1}}, \beta_ {i _ {0}}\right). \tag {21}
$$
- For every  $0 \leq j_{1}, j_{0}, i_{0} < \sqrt{N}$ :
$$
\mathbf {R} _ {j _ {1}, j _ {1}} \left[ i _ {0}, j _ {0} \right] \leftarrow r _ {j _ {1}, j _ {0}} \left(\beta_ {i _ {0}}\right). \tag {22}
$$
Note that all entries of  $\overline{\mathbf{L}}$  and  $\mathbf{R}$  not specified above are set to 0.
Let  $f$  be the above function that maps coefficients of  $\ell_{j_1}(X,Y)$  (which are coefficients of monomials  $X^{i_1}Y^{i_0}$  for all  $0\leq i_1,i_0 < \sqrt{N}$  and hence represented by a matrix in  $\mathbb{R}^{\sqrt{N}\times \sqrt{N}}$ ) and coefficients of  $r_{j_1,j_0}(Y)$  (which are coefficients of monomials  $Y^{i_0}$  for all  $0\leq i_0 < \sqrt{N}$  and hence represented by a vector in  $\mathbb{R}^{\sqrt{N}}$ ) for all  $0\leq j_1,j_0 < \sqrt{N}$  to pairs of matrices in  $\mathcal{D}\mathcal{B}^{\sqrt{N},N}\times \mathcal{B}\mathcal{D}^{\sqrt{N},N}$ .
Theorem 6. Let  $f$  be as defined above. Then  $f$  is a bijection.
Proof. To prove  $f$  is bijection we must show  $f$  is one-to-one and  $f^{-1}$  is one-to-one (and exists).
To show  $f$  is one to one means each set of polynomials' coefficients given to  $f$ , will output a unique set of matrices  $(\overline{\mathbf{L}}, \mathbf{R}) \in \mathcal{D}\mathcal{B}^{\sqrt{N}, N} \times \mathcal{B}\mathcal{D}^{\sqrt{N}, N}$ . This follows from (21), (22) and the known fact that polynomial evaluation is a function.
Now, to show  $f^{-1}$  exists and is one-to-one, we must show that there is a map from any pair  $(\overline{\mathbf{L}},\mathbf{R})\in \mathcal{DB}^{\sqrt{N},N}\times \mathcal{BD}^{\sqrt{N},N}$  to unique sets of polynomials,  $\bar{\ell},\bar{r}$ , with parameters as defined in Appendix D.1.2. Further, we need
$$
\overline {{\mathbf {L}}} _ {i _ {1}, j _ {1}} \left[ i _ {0}, i _ {0} \right] = \bar {\ell} _ {j _ {1}} \left(\alpha_ {i _ {1}}, \beta_ {i _ {0}}\right) \tag {23}
$$
$$
\mathbf {R} _ {j _ {1}, j _ {1}} \left[ i _ {0}, j _ {0} \right] = \bar {r} _ {j _ {1}, j _ {0}} \left(\beta_ {i _ {0}}\right). \tag {24}
$$
We will use Theorems 5 and 4 to show the existence of  $\bar{\ell}_{j_1}$  and  $\bar{r}_{j_1,j_0}$  polynomials, giving us the mapping from the matrices to unique polynomials.
We first show the existence of the unique polynomials in (24). Fix  $0 \leq j_1, j_0 < \sqrt{N}$ . Then consider the values  $0 \leq i_0 < \sqrt{N}$ :
$$
y _ {i _ {0}} \leftarrow \mathbf {R} _ {j _ {1}, j _ {1}} \left[ i _ {0}, j _ {0} \right]. \tag {25}
$$
Then by Theorem 4, there exists a unique polynomial of degree  $< \sqrt{N}$  (call it  $\bar{r}_{j_1,j_0}(Y)$ ) such that for all  $0 \leq i_0 < \sqrt{N}$ ,
$$
\bar {r} _ {j _ {1}, j _ {0}} \left(\beta_ {i _ {0}}\right) = y _ {i _ {0}},
$$
which by (25) shows (24).
Next we show the existence of the unique polynomials in (23). Fix  $0 \leq j_1 < \sqrt{N}$ . Consider the values  $0 \leq i_1, i_0 < \sqrt{N}$ :
$$
y _ {i _ {1}, i _ {0}} \leftarrow \overline {{\mathbf {L}}} _ {i _ {1}, j _ {1}} \left[ i _ {0}, i _ {0} \right]. \tag {26}
$$
Then by Theorem 5, there exists a unique bi-variate polynomial of  $\deg_X < \sqrt{N}$  and  $\deg_Y < \sqrt{N}$  (call it  $\bar{\ell}_{j_1}(X,Y)$ ) such that for all  $0 \leq i_1, i_0 < \sqrt{N}$ ,
$$
\bar {\ell} _ {j _ {1}} \left(\alpha_ {i _ {1}}, \beta_ {i _ {0}}\right) = y _ {i _ {1}, i _ {0}}, \tag {27}
$$
which by (26) shows (23).
Therefore  $f$  is a bijection.
We can now conclude:
Corollary 1. For every matrix  $\mathbf{M}'$  as defined in (20), there exists unique polynomials  $\ell_{j_1}(X,Y)$  and  $r_{j_1,j_0}(Y)$ , such that for all  $0\leq i_1,i_0,j_1,j_0 < \sqrt{N}$
$$
\mathbf {M} ^ {\prime} \left[ \left(i _ {1}, i _ {0}\right) _ {\sqrt {N}}, \left(j _ {1}, j _ {0}\right) _ {\sqrt {N}} \right] = \ell_ {j _ {1}} \left(\alpha_ {i _ {1}}, \beta_ {i _ {0}}\right) \cdot r _ {j _ {1}, j _ {0}} \left(\beta_ {i _ {0}}\right). \tag {28}
$$
Proof. Follows from (20), (21), (22) and Theorem 6.  $\square$
D.2.1 Proof of Theorem 1
We begin with an immediate consequence of Corollary 1:
Corollary 2. Let  $A, B \subset \mathbb{C}$  such that  $|A| = |B| = \sqrt{N}$ . Then the jth column of  $\mathbf{M}'$  is the evaluation of the polynomial  $\ell_{j_1}(X,Y) \cdot r_{j_1,j_0}(Y)$  over  $A \times B$ .
Proof. Observe that for fixed  $j_0, j_1$  the right hand side of (28) is  $\ell_{j_1}(X,Y)\cdot r_{j_1,j_0}(Y)$  evaluated at all  $(\alpha ,\beta)\in A\times B$ . Thus,  $(j_{1},j_{0})$  column is evaluation of  $\ell_{j_1}(X,Y)\cdot r_{j_1,j_0}(Y)$  over points in  $A\times B$ , as desired.
Next, we state a generalization of Theorem 1 that follows from Corollary 2:
Corollary 3. Let  $A$  and  $B$  be as in Corollary 2. For any vector  $\mathbf{u}$ ,  $\mathbf{M}' \cdot \mathbf{u}$  is  $\bar{u}(X, Y)$  evaluated at  $A \times B$ . Further,
$$
\bar {u} (X, Y) = \sum_ {0 \leq j _ {1}, j _ {0} <   \sqrt {N}} u _ {j _ {1}, j _ {0}} \cdot \ell_ {j _ {1}} (X, Y) \cdot r _ {j _ {1}, j _ {0}} (Y), \tag {29}
$$
where  $\ell$  and  $r$  are defined by  $\mathbf{M}'$  as in Corollary 1.
Proof. Follows from Corollary 2 and definition of matrix vector multiplication.
In Theorem 1 and the following sections, we consider the polynomial evaluated over the basis polynomials defined by
$$
\mathbf {M} = \mathbf {M} ^ {\prime} \mathbf {P}
$$
Corollary 4. Let  $A$  and  $B$  be as in Corollary 2. For any vector  $\mathbf{u}$ , and  $\mathbf{M} \cdot \mathbf{u}$  is  $u(X, Y)$  evaluated at  $A \times B$ . Further,
$$
u (X, Y) = \sum_ {0 \leq j _ {1}, j _ {0} <   \sqrt {N}} u _ {j _ {0}, j _ {1}} \cdot \ell_ {j _ {0}} (X, Y) \cdot r _ {j _ {0}, j _ {1}} (Y), \tag {30}
$$
Proof. Follows from Corollary 3 and definition of  $\mathbf{M}$ .
Specifically, Theorem 1 is a special case of Corollary 4 where  $\ell_{j_0}(X,Y) = \ell_{m(j)}(X,Y)$  and  $r_j(Y) = r_{j_0,j_1}(Y)$ .
D.2.2 Proof of Theorem 2
By Corollary 4, for all  $0 \leq j_1, i_1 < \sqrt{N}$ ,
$$
\ell_ {j _ {0}} (X, Y), r _ {j _ {0}, j _ {1}} (Y), \bar {\ell} _ {j _ {0}} (X, Y), \bar {r} _ {j _ {0}, j _ {1}} (Y)
$$
are the basis polynomials corresponding to  $\mathbf{M}_1$  and  $\mathbf{M}_2$ . For the coefficient vector  $\mathbf{k} = (k_{j_1,j_0})_{0\leq j_1,j_0 < \sqrt{N}}$  and similarly for  $\mathbf{u} = (u_{j_1,j_0})_{0\leq j_1,j_0 < \sqrt{N}}$ , we can construct two polynomials
$$
k (X, Y) = \sum_ {0 \leq j _ {1}, j _ {0} <   \sqrt {N}} k _ {j _ {1}, j _ {0}} \cdot \ell_ {j _ {0}} (X, Y) \cdot r _ {j _ {0}, j _ {1}} (Y)
$$
$$
u (X, Y) = \sum_ {0 \leq j _ {1}, j _ {0} <   \sqrt {N}} u _ {j _ {1}, j _ {0}} \cdot \bar {\ell} _ {j _ {0}} (X, Y) \cdot \bar {r} _ {j _ {0}, j _ {1}} (Y)
$$
whose evaluation over  $(\alpha_{i_1},\beta_{i_0}) = (\omega^{i_1},\omega^{i_0})$  where recall as in Appendix D.1.1  $\omega = e^{\frac{2\pi t}{\sqrt{N}}}$ , by Theorem 1 is equivalent to the products  $\mathbf{M}_1\cdot \mathbf{k}$  and  $\mathbf{M}_2\cdot \mathbf{u}$ , respectively. Taking the component-wise product,  $\mathbf{y} = (\mathbf{M}_1\cdot \mathbf{k})\odot (\mathbf{M}_2\cdot \mathbf{u})$ , the entry at  $i = (i_1,i_0)$  is given by
$$
\mathbf {y} \left[ \left(i _ {1}, i _ {0}\right) \right] = k \left(\omega^ {i _ {1}}, \omega^ {i _ {0}}\right) \cdot u \left(\omega^ {i _ {1}}, \omega^ {i _ {0}}\right).
$$
Noting that the element of  $A$ , i.e. the  $\sqrt{N}$ -th roots of unity, satisfy  $Z^{\sqrt{N}} = 1$  means that the above are evaluations of
$$
h (X, Y) = k (X, Y) \cdot u (X, Y) \mod \left(X ^ {\sqrt {N}} - 1, Y ^ {\sqrt {N}} - 1\right)
$$
at  $A \times A$ . Finally, Theorem 1 and the fact that  $\mathbf{M}_0^{-1}$  exists implies  $\mathbf{M}_0 \cdot \mathbf{y}$  is polynomial interpolation into basis polynomials corresponding to  $\mathbf{M}_0$ . (Here we use the well-known fact polynomial interpolation is the inverse of polynomial evaluation).
We review some concepts in Appendix D.3.1. In Appendix D.3.2, we discuss square matrices and causality in terms of operations on univariate polynomials. This allows us to define a general class of operators for causal 1D convolution. In Appendix D.3.3, we give a class of matrices suitable for perform causal Monarch convolution. Specifically, we prove Theorem 3.
D.3.1 Review
Consider the linear operation on an input vector  $\mathbf{u}$ :
$$
\mathbf {y} = \mathbf {A} \cdot \mathbf {u}.
$$
We say that the map is causal to mean the entry  $\mathbf{y}[i]$  only depends on  $\mathbf{u}[0],\mathbf{u}[1],\ldots \mathbf{u}[i]$ . This will be the case when  $\mathbf{A}$  is a lower triangular matrix (we index the top left entry of  $\mathbf{A}$  as  $(0,0)$ ). When  $\mathbf{A}$  is a lower triangular Toeplitz matrix with entries corresponding to some coefficient vector  $\mathbf{k}$ , this operation is exactly the 1D convolution
$$
\mathbf {y} = \mathbf {k} * \mathbf {u} = \left(\mathbf {F} _ {2 n} ^ {- 1} \cdot \left(\left(\mathbf {F} _ {2 n} \cdot \mathbf {k} ^ {\prime}\right) \circ \left(\mathbf {F} _ {2 n} \cdot \mathbf {u} ^ {\prime}\right)\right)\right) [ 0: n - 1 ],
$$
where  $\mathbf{k}' = (\mathbf{k},\mathbf{0}_n)$ ,  $\mathbf{u}' = (\mathbf{u},\mathbf{0}_n)$ , and  $\mathbf{F}_n$  is the  $n\times n$  DFT matrix.
Definition 4. For a matrix  $\overline{\mathbf{M}}\in \mathbb{R}^{n\times n}$ , let us define the map
$$
\mathbf {y} = \mathbf {M} ^ {- 1} (\overline {{\mathbf {M}}} \cdot \mathbf {k} \odot \overline {{\mathbf {M}}} \cdot \mathbf {u}) \tag {31}
$$
as matrix convolution. When  $\overline{\mathbf{M}}$  is a Monarch matrix, (31) is called Monarch convolution.
In this section, we are interested in determining large subclasses of matrices  $\overline{\mathbf{M}}$  such that for any coefficient vector  $\mathbf{k}$ , (31) is causal in  $\mathbf{u}$ . We provide a class of matrices for which Monarch convolution is causal.
We note that for general Monarch matrix  $\mathbf{M}$ , (31) is not causal in  $\mathbf{u}$ . By Theorem 2, we have
$$
y (X, Y) = k (X, Y) \cdot u (X, Y) \mod (X ^ {\sqrt {N} - 1}, Y ^ {\sqrt {N} - 1}).
$$
This is not causal because the mod  $(X^{\sqrt{N} -1},Y^{\sqrt{N} -1})$  term condenses higher order terms into lower order terms, hence the  $\mathbf{y}[i]$  wouldn't just depend on input information up to value  $i$ .
D.3.2 Univariate Matrix Convolutions
We start with a couple of notation assumptions.
Assumption 1.  $N$  is a perfect square.
Assumption 2. We will not use pair notation for this subsection since throughout we have  $i = i_1\sqrt{N} + i_0$  and  $j = j + 1\sqrt{N} + j_0$ .
In order to discuss square matrices in terms of univariate polynomials, we give univariate analogs of Theorem 1 and Theorem 2 for general univariate basis. With an eye toward towards performing causal convolution, we restrict our analysis to certain classes of univariate polynomials.
We first define matrices whose  $j^{\mathrm{th}}$  columns are the evaluation of a minimum degree  $j$  (and maximum degree  $N - 1$ ) polynomial (recall Definition 1). We generalize Theorem 3 to such matrices.
Lemma 1. For sequence of points  $A = \{1, \omega_N, \dots, \omega_N^{N-1}\}$  where  $\omega_N$  is the  $N^{th}$  root of unity, let  $\overline{\mathbf{M}}$  be defined as
$$
\overline {{\mathbf {M}}} [ i, j ] = \bar {q} _ {j} \left(\omega_ {N} ^ {i}\right) \tag {32}
$$