SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f _ {h} (X) = S _ {h, a} + \frac {\sum_ {i = \frac {N + 3}{2}} ^ {N} \exp (Q _ {h} (\phi (x _ {1})) ^ {\top} K _ {h} (\phi (x _ {i}))) V _ {h} (\phi (x _ {i}))}{\exp (L _ {h})} \in \mathbb {R} ^ {m}.
$$
Bob concludes the protocol by computing and outputting  $f(X)_1$ , using his knowledge of each  $f_h(X)$  and of  $\psi$ .
By the equivalences previously established, Bob returns 1 if and only if  $\mathrm{DISJ}(a,b) = 1$ . Because the protocol requires  $O(mpH\log \log N)$  bits of communication, we can only avoid contradicting Fact 5 if  $mpH \geq \Omega(n / \log \log N) = \Omega(N / \log \log N)$ .
Remark 1. The domain restrictions to Match3 stipulated in Equation (3) make the Match3 problem substantially easier to solve than the full-domain case. Indeed, under the domain restrictions,
$$
\operatorname {M a t c h} 3 (X) _ {1} = \max  _ {i \in \{2, \dots , \frac {N + 1}{2} \}} \operatorname {M a t c h} 2 (X) _ {i},
$$
which is computable by a two-layer single-headed transformer network with constant embedding dimension. The first layer computes each Match2  $(X)_i$  with the construction in the proof of Theorem 6, and the second computes the maximum of the previous outputs by using those outputs as key vectors.
While Informal Conjecture 1 suggests that two layers are insufficient to compute the full-domain version of Match3, this restricted variant introduces a concise depth separation (see Eldan and Shamir [2016], Telgarsky [2016], Daniely [2017]) between one- and two-layer transformer models.
C.3 Higher-order tensor attention
We introduce a novel category of higher-order tensor-based transformer models in order to show that problems like Match3 that are hard to compute with standard transformer models can be made solvable. An  $s$ -order transformer is designed to efficiently compute dense  $s$ -wise interactions among input elements in an analogous manner to how standard transformers compute pairwise interactions. (We think of a standard transformer as second-order.) Before defining the new type of attention, we introduce notation to express the needed tensor products.
For vectors  $v^1 \in \mathbb{R}^{N_1}$  and  $v^2 \in \mathbb{R}^{N_2}$ , let  $v^1 \otimes v^2 \in \mathbb{R}^{N_1N_2}$  denote their Kronecker product by  $(v^1 \otimes v^2)_{(i_1 - 1)N_2 + i_2} = v_{i_1}^1 v_{i_2}^2$ . The column-wise Kronecker product of matrices  $A^1 \in \mathbb{R}^{N_1 \times m}$  and  $A^2 \in \mathbb{R}^{N_2 \times m}$  is
$$
A ^ {1} \star A ^ {2} = \left[ A _ {1} ^ {1} \mid \dots \mid A _ {m} ^ {1} \right] \star \left[ A _ {1} ^ {2} \mid \dots \mid A _ {m} ^ {2} \right] = \left[ A _ {1} ^ {1} \otimes A _ {1} ^ {2} \mid \dots \mid A _ {m} ^ {1} \otimes A _ {m} ^ {2} \right] \in \mathbb {R} ^ {N _ {1} N _ {2} \times m}.
$$
The following generalizes the definition of self-attention.
Definition 7. For order  $s \geq 2$ , input dimension  $d$ , output dimension  $d'$ , embedding dimension  $m$ , bit complexity  $p$ , and matrices  $Q, K^1, \ldots, K^{s-1} \in \mathbb{R}^{d \times m}$  and  $V^1, \ldots, V^{s-1} \in \mathbb{R}^{d \times d'}$  (encoded with  $p$ -bit fixed-point numbers), an  $s$ -order self-attention unit is a function  $f_{Q, K, V}: \mathbb{R}^{N \times d} \to \mathbb{R}^{N \times d'}$  with
$$
f_{Q,K,V}(X) = \operatorname {softmax}(\underbrace{XQ}_{\in \mathbb{R}^{N\times m}}\underbrace{((XK^{1})\star\cdots\star(XK^{s - 1}))^{\intercal}}_{\in \mathbb{R}^{m\times N^{s - 1}}})\underbrace{((XV^{1})\star\cdots\star(XV^{s - 1}))}_{\in \mathbb{R}^{N^{s - 1}\times d^{\prime}}}).
$$
The input to the row-wise softmax is an  $N \times N^{s-1}$  matrix. Let  $\mathcal{A}_{d,m,d',p}^{\otimes s}$  denote the set containing all such attention units.
Note that  $\mathcal{A}_{d,m,d',p}^{\otimes 2} = \mathcal{A}_{d,m,d',p}$ . Because  $s$ -order self-attention units have the same domain and codomain as standard self-attention, multiple units can be analogous combined to construct multi-headed attention units and full transformer models. We define  $\mathcal{A}_{d,m,d',p}^{M,\otimes s}$  and  $\mathcal{T}_{d,m,d',p}^{D,H,\otimes s}$  accordingly.
The purpose of the  $s$ -order transformer model as a theoretical construct is to posit how strictly generalizing the architecture in order to permit higher order outer products transfers the expressive powers of standard transformer architectures to more sophisticated interactions among elements of the input sequence  $X$ . The model is not defined to be immediately practical, due to its steep computational cost of evaluation.
However, the trade-offs involved in using such architectures resemble those already made by using transformer models instead of fully-connected networks. Transformers are already computationally wasteful relative to the number of the parameters, and these models likely succeed only because extremely efficient factorized parameterization exist. Likewise, third-order transformers could indeed be practical if even more factorization proves useful, since the computational costs may prove mild if the embedding dimension  $m$ , number of heads  $H$ , and depth  $D$  necessary to succeed on a task exceed the sequence length  $N$  for standard second-order transformers.
C.4 Efficient representation of Match3 with third-order self-attention
Theorem 18 (Match3 construction with third-order self-attention). For any sequence length  $N$ , input range  $M = N^{O(1)}$ , and fixed-precision bit complexity  $p = O(\log M)$ , there exists a third-order transformer architecture  $f \in \mathcal{T}_{1,m,1,p}^{1,1,\otimes 3}$  with a single self-attention unit with embedding dimension  $m = 5$  such that for all  $X \in [M]^N$ ,  $f(X) = \mathrm{Match3}(X)$ .
Proof of Theorem 18. The proof is almost identical to that of Theorem 6, except that we instead use a different key and query transforms to express a different trigonometric function:
$$
Q \phi (x _ {i}) = c \left(\cos \left(\frac {2 \pi x _ {i}}{M}\right), - \cos \left(\frac {2 \pi x _ {i}}{M}\right), \sin \left(\frac {2 \pi x _ {i}}{M}\right), \sin \left(\frac {2 \pi x _ {i}}{M}\right), 1\right),
$$
$$
K ^ {1} \phi (x _ {i}) = \left(\cos \left(\frac {2 \pi x _ {i}}{M}\right), \sin \left(\frac {2 \pi x _ {i}}{M}\right), - \cos \left(\frac {2 \pi x _ {i}}{M}\right), \sin \left(\frac {2 \pi x _ {i}}{M}\right), 0\right),
$$
$$
K ^ {2} \phi (x _ {i}) = \left(\cos \left(\frac {2 \pi x _ {i}}{M}\right), \sin \left(\frac {2 \pi x _ {i}}{M}\right), \sin \left(\frac {2 \pi x _ {i}}{M}\right), - \cos \left(\frac {2 \pi x _ {i}}{M}\right), 0\right).
$$
Together, these ensure that the resulting tensor products reduce to a trigonometric expression that is maximized when  $x_{i} + x_{j_{1}} + x_{j_{2}} = 0$  (mod  $M$ ). That is,
$$
\left(\phi (X) Q \left(\left(\phi (X) K ^ {1}\right) \star \left(\phi (X) K ^ {2}\right)\right) ^ {\top}\right) _ {i, \left(j _ {1} - 1\right) + j _ {2}} = c \cos \left(\frac {2 \pi \left(x _ {i} + x _ {j _ {1}} + x _ {j _ {2}}\right)}{M}\right).
$$
We similarly let  $V^{1}\phi(x_{i}) = V^{2}\phi(x_{i}) = \vec{1}$  and  $V^{1}\phi(x') = V^{2}\phi(x') = \vec{0}$ . The remaining choice of  $c$  and the output MLP, and the analysis of the softmax proceeds identically to the previous proof.
C.5 Heuristic argument for Informal Conjecture 1
Conjecture 19 (Formal version of Informal Conjecture 1). For sufficiently large  $N$  and any  $d \geq 1$ , for all  $M \geq N + 1$  and  $mpHD \leq N^{\Omega(1)}$ , there is no  $f \in T_{1,m,1,p}^{D,H}$  satisfying  $f(X) = \operatorname{Match3}(X)$  for all  $X \in [M]^N$ .
We believe that the conjecture holds due to a heuristic information-theoretic argument. Define the distribution  $\mathcal{D}$  over inputs  $X\in \mathbb{R}^N$  that will be used to show that the model cannot compute Match3 for  $M = N^4$  with high probability. We draw  $\mathbf{X}$  from  $\mathcal{D}$  as follows:
$(E_{1})$  With probability  $\frac{1}{2}$ , draw each  $\mathbf{x}_i$  iid from Unif([M]).
$(E_2)$  With probability  $\frac{1}{2}$ , draw  $j_1, j_2, j_3$  iid from  $\mathrm{Unif}(\binom{[N]}{3})$ . For all  $i \neq j_3$ , draw each  $\mathbf{x}_i$  iid from  $\mathrm{Unif}([M])$ . Let  $\mathbf{x}_{j_3} = -\mathbf{x}_{j_1} - \mathbf{x}_{j_2} \pmod{M}$ .
Note that under event  $E_{1}$ , a three matching elements exist with probability at most  $\frac{1}{N}$ , and
$$
\Pr \left[ \operatorname {M a t c h} 3 (\mathbf {X}) = \vec {0} \mid E _ {1} \right] \geq 1 - \frac {1}{N}.
$$
Under event  $E_{2}$ , a triple of matching elements is always planted, so Match3(X) ≠ 0. It would suffice to prove that—unless a transformer is sufficiently large—it is impossible to determine whether Match3(X) = 0 with probability at least 0.9.
Under  $\mathcal{D}$ , any subset of  $\{\mathbf{x}_1, \ldots, \mathbf{x}_N\}$  consists of iid integers drawn uniformly from  $[M]$ , unless all of  $\mathbf{x}_{j_1}, \mathbf{x}_{j_2}, \mathbf{x}_{j_3}$  appear in the subset. Consider a transformer architecture with  $p$ -bit precision,  $m$ -dimensional embeddings,  $H$  heads per layer, and  $D$  layers. We argue informally that a single-element output of a self-attention unit can take into account information about  $mp$  more inputs  $\mathbf{x}_1, \ldots, \mathbf{x}_N$  than that it had in the previous layer. By induction, after  $D$  layers of  $H$ -headed self-attention with interleaved MLPs, each element is a function of at most  $mpHD$  inputs. Until an element exists that is a function of at least two of the three of  $\mathbf{x}_{j_1}, \mathbf{x}_{j_2}, \mathbf{x}_{j_3}$ , we assume that the elements "known" by each output are chosen independently of the indices  $j_1, j_2, j_3$ . (Given two elements of the triple, the third element can be identified with a single self-attention unit.) Hence, we argue that it suffices to show that the probability any two elements of the triple  $j_1, j_2, j_3$  occurring within any of the  $N$  sets of  $mpHD$  inputs is vanishingly small for sufficiently large transformer parameters. The probability of single collection having any of two of the three inputs is at most
$$
\frac {3 \binom {m p H D} {2}}{\binom {N} {2}} \leq 3 \left(\frac {e m p H D}{N}\right) ^ {2}.
$$
Thus, the probability that any collection has all three inputs is no more than  $3(empHD)^2 / N$ . If  $mpHD = O(\sqrt{N})$ , then the randomly chosen triple will not jointly appear as the outcome of a single
element of a self-attention unit with probability at least 0.9, and the transformer will be unexpected to successfully distinguish between the two cases.
Should the conjecture hold, it would represented a tight lower bound on the size of the smallest standard transformer architecture necessary to compute Match3.
Theorem 20 (Tightness of Conjecture 19). For any sequence length  $N$ , if the input range satisfies  $M = N^{O(1)}$  and the transformer size parameters satisfy  $p \geq \log(M)$ ,  $H = 1$ ,  $m \geq 4$ , and  $mD \geq CN^2$  for some universal constant  $C$ , then there exists a transformer architecture  $f \in \mathcal{T}_{1,m,1,p}^{D,H}$  such that  $f(X) = \operatorname{Match3}(X)$ .
Proof. We construct an architecture that collects a group of candidate pairs in each layer of single-headed self-attention and verifies whether there exists a triple incorporating each pair that satisfies the summation property. Then, all candidate triples are disposed of, and the subsequent layer collects a new family of candidates.
To do so, we first let  $\ell := \left\lfloor \frac{m}{2} \right\rfloor - 1 \geq 1$  represent the total number of pairs shared in each layer of attention. We let  $P = \binom{[N]}{2}$  represent a collection of all pairs of indices and partition it into  $D$  subsets  $P_1, \ldots, P_D$ , each containing  $\ell$  distinct pairs. (Since  $|P| = \frac{N(N + 1)}{2}$ , any  $D$  satisfying the theorem's preconditions is sufficiently large for this to be a proper partition.) Our construction ensures that there exist  $x_i + x_{j_1} + x_{j_2} = 0 \pmod{M}$  for  $(j_1, j_2) \in P_k$ , then the  $k$ th layer of self attention will verify its existence and mark  $x_i$  as belonging to the match. Throughout the network, we maintain that the first two dimensions of any embedding of the  $i$ th element correspond to  $x_i \in [M]$  and a bit indicating whether a match has been found yet containing  $x_i$ .
Consider the first layer of self-attention, and let  $P_{1} = \{(i_{1},j_{1}),\ldots ,(i_{\ell},j_{\ell})\}$ . We set the input MLP  $\phi_1:\mathbb{R}^d\to \mathbb{R}^m$  and respective matrices  $Q^{1},K^{1}\in \mathbb{R}^{m\times m}$  such that
$$
Q ^ {1} \phi_ {1} (x _ {i}) = c e _ {1} \text {a n d} K ^ {1} \phi_ {1} (x _ {i}) = \left\{ \begin{array}{l l} e _ {1} & \text {i f} i \in P _ {1} \\ \vec {0} & \text {o t h e r w i s e}, \end{array} \right.
$$
for sufficiently large  $c$ . We additionally let
$$
V ^ {1} \phi_ {1} (x _ {i}) = \left\{ \begin{array}{l l} (2 \ell + 1) \cdot (x _ {i}; 0; \vec {0}) & i \not \in P _ {1}, \\ (2 \ell + 1) \cdot (x _ {i}; 0; x _ {i} e _ {2 \iota - 1}) & i = i _ {\iota}, \\ (2 \ell + 1) \cdot (x _ {i}; 0; x _ {i} e _ {2 \iota}) & i = j _ {\iota}. \end{array} \right.
$$
By making use of a residual connection, we ensure that the  $i$ th outcome of the self-attention is  $(x_{i},0,x_{i_{1}},x_{j_{1}},\ldots ,x_{i_{\ell}},x_{j_{\ell}})$ . We encode an MLP to compute
$$
\left(x _ {i}, 0, x _ {i _ {1}}, x _ {j _ {1}}, \dots , x _ {i _ {\ell}}, x _ {j _ {\ell}}\right) \mapsto \left(x _ {i}, \mathbb {1} \left\{\exists \iota \in [ \ell ] \text {s . t .} x _ {i} + x _ {i _ {\iota}} + x _ {j _ {\iota}} = \vec {0} \quad (\operatorname {m o d} M) \right\}; \vec {0}\right).
$$
We repeat this construction  $D$  times, with the only modifications being the replacement of  $P_{1}$  and the fact that the second dimension of the embedding remains 1 after being set to that value. After  $D$  layers, the final MLP outputs the value of the second dimension, which will be 1 if and only if the respective  $x_{i}$  belongs to a three-way match.
C.6 Sharper separations for embedded subgraph detection problems
In pursuit of proving separations analogous to the one between Theorem 18 and Conjecture 19, we draw techniques for proving lower bounds for graph problems in the CONGEST model of distributed computation with restricted bandwidth [Peleg, 2000].<sup>5</sup>
5At a high level, the CONGEST model features  $N$  players that communicate in synchronous rounds over a network (an undirected graph with  $[N]$  as its vertices) to solve a computational problem Peleg [2000]. In each round, each player can send a message to each of its neighbors. The computation that each player does with the messages received from its neighbors is unrestricted; the primary resources considered in CONGEST is the number of rounds of communication and the message sizes. Although CONGEST is often studied for solving computational problems on input graphs with vertices  $[N]$ , the input graph need not be the same as the communication network.
The problems we consider take, as input, the adjacency matrix  $X \in \{0,1\}^{N \times N}$  of an  $N$ -vertex graph  $G = (\mathcal{V}, \mathcal{E})$  with  $\mathcal{V} = [N]$ , so  $x_{i,j} = \mathbb{1}\left\{\left(i,j\right) \in \mathcal{E}\right\}$ . We may regard each row of  $X$  as a high-dimensional  $(d = N)$  embedding of the  $i$ -th vertex containing information about which (outgoing) edges are incident to the  $i$ -th vertex. We consider the following problems:
$$
\begin{array}{l} \operatorname {D i r e c t e d C y c l e} 3 (X) = \left(\mathbb {1} \left\{\exists j _ {1}, j _ {2} \in [ N ] \text {s . t .} x _ {i, j _ {1}} x _ {j _ {1}, j _ {2}} x _ {j _ {2}, i} = 1 \right\}\right) _ {i \in [ N ]}; \\ \operatorname {C y c l e} 5 (X) = \left(\mathbb {1} \left\{\exists j _ {1}, j _ {2}, j _ {3}, j _ {4} \in [ N ] \text {s . t .} x _ {i, j _ {1}} x _ {j _ {1}, j _ {2}} x _ {j _ {2}, j _ {3}} x _ {j _ {3}, j _ {4}} x _ {j _ {4}, i} = 1 \right\}\right) _ {i \in [ N ]}, \\ \text {w i t h} \operatorname {d o m} (\text {C y c l e 5}) = \{X: X = X ^ {\top} \}. \\ \end{array}
$$
The former treats  $X$  as a directed graph (where  $X$  need not be symmetric) and asks whether each input belongs to a directed 3-cycle. The latter insists that  $X$  be an undirected graph by enforcing symmetry and determines membership in (undirected) 5-cycles.
However, solving these problems with any transformer model of constant order trivially requires having the product of the precision  $p$ , embedding dimension  $m$ , heads per layer  $H$ , and depth  $D$  grow polynomially with  $N$ , since each attention unit is limited to considering at most  $pm$  bits of information from each input. Such a lower bound is not interesting for dense graphs, where every vertex may have  $\Omega(N)$  incident edges; the bottleneck is not due to any feature of standard attention units (and would persist with higher-order attention).
To circumvent this issue, we consider an augmented self-attention unit, which permits each element of the self-attention tensor to depend on both its respective inner product and on the presence of edges among corresponding inputs.
Definition 8. For order  $s \geq 2$ , input dimension  $d$ , output dimension  $d'$ , embedding dimension  $m$ , bit complexity  $p$ , matrices  $Q, K^1, \ldots, K^{s-1} \in \mathbb{R}^{d \times m}$  and  $V^1, \ldots, V^{s-1} \in \mathbb{R}^{d \times d'}$  (encoded with  $p$ -bit fixed-point numbers), and cell-wise attention tensor function  $\kappa: \{0,1\}^{s(s-1)} \times \mathbb{R} \to \mathbb{R}$ , an  $s$ -order graph self-attention unit is a function  $f_{Q,K,V}: \mathbb{R}^{N \times d} \to \mathbb{R}^{N \times d'}$  with
$$
f _ {Q, K, V} (X) = \operatorname {s o f t m a x} (\kappa (X, X Q ((X K ^ {1}) \star \dots \star (X K ^ {s - 1})) ^ {\intercal})) ((X V ^ {1}) \star \dots \star (X V ^ {s - 1})).
$$
For attention tensor  $A \in \mathbb{R}^{N^{\otimes s}}$ , we abuse notation by writing  $\kappa(X, A)$  as short-hand for the particular cell-wise application of a fixed function, incorporating information about all relevant edges:
$$
\kappa (X, A) _ {i _ {1}, \dots , i _ {s}} = \kappa (x _ {i _ {1}, i _ {2}}, x _ {i _ {1}, i _ {3}}, \dots , x _ {i _ {s}, i _ {s - 1}}, x _ {i _ {s}, i _ {s - 2}}, A _ {i _ {1}, \dots , i _ {s}}).
$$
Let  $\mathcal{A}\mathcal{G}_{d,m,d',p}^{\otimes s}$  and  $\mathcal{T}\mathcal{G}_{d,m,d',p}^{D,H,\otimes s}$  denote all such attention units and all such transformers respectively.
Now, we provide four results that exhibit separations between orders of graph self-attention.
Theorem 21 (Hardness of representing Cycle5 with standard graph transformer). For sufficiently large  $N$ , any  $f \in \mathcal{T}\mathcal{G}_{N,m,1,p}^{D,H}$  satisfying  $f(X) = \mathrm{Cycle5}(X)$  for all  $X \in \{0,1\}^{N \times N}$  with  $X = X^{\top}$  requires  $mpHD = \Omega(N / \log^2 N)$ .
Theorem 22 (Efficient construction of Cycle5 with fifth-order graph transformer). For sequence length  $N$  and bit-complexity  $p = O(\log N)$ , there exists a fourth-order graph transformer architecture  $f \in \mathcal{T}\mathcal{G}_{N,1,1,p}^{1,1,\otimes 5}$  with a single graph self-attention unit such that for all  $X \in \{0,1\}^{N \times N}$  with  $X = X^{\top}$ ,  $f(X) = \mathrm{Cycle5}(X)$ .
Theorem 23 (Hardness of representing DirectedCycle3 with standard graph transformer). For sufficiently large  $N$ , any  $f \in \mathcal{TG}_{N,m,1,p}^{D,H}$  satisfying  $f(X) = \operatorname{DirectedCycle3}(X)$  for all  $X \in \{0,1\}^{N \times N}$  requires  $mpHD = \Omega(N / \log^2 N)$ .
Theorem 24 (Efficient construction of DirectedCycle3 with fourth-order graph transformer). For sequence length  $N$  and bit-complexity  $p = O(\log N)$ , there exists a third-order graph transformer architecture  $f \in \mathcal{T}G_{N,1,1,p}^{1,1,\otimes 3}$  with a single graph self-attention unit such that for all  $X \in \{0,1\}^{N \times N}$ ,  $f(X) = \text{DirectedCycle3}(X)$ .
The proofs of Theorems 22 and 24 are immediate from the construction. Because each cell of the self-attention tensor has explicit access the existence of all relevant edges,  $\kappa$  can be configured to ensure that cell's value is large if and only if the requisite edges for the desired structure all exist. Taking a softmax with a blank element (like in Theorem 6) ensures that the outcome of the
Figure 4: The CONGEST graph  $G^{N}$  visualized for  $N = 6$  with root nodes  $\{u_{i}\}_{i\in [N]}$  in blue, leaf nodes  $\{v_{i,j}\}_{i,j\in [N]}$  in green, and the nodes  $V_{1}$  of the binary tree  $B_{1}$  shaded red and edges  $E_{1}$  colored red.
self-attention unit for a given element distinguishes between whether or not it belongs to a 5-cycle or a directed 3-cycle. The output MLP ensure that the proper output is returned.
We prove Theorems 21 and 23 by introducing a particular CONGEST communication graph that can be used to simulate any model in  $\mathcal{T}\mathcal{G}_{d,m,d',p}^{D,H}$  (and hence, also any model in  $\mathcal{T}_{d,m,d',p}^{D,H}$ ) in  $O(mHD\log N)$  rounds of communication. Then, we show for each problem that we can encode each instance of the set disjointness communication problem as an instance of Cycle5 (or DirectedCycle3) and derive a contradiction from the communication graph.
C.6.1 A CONGEST communication graph that generalizes standard graph transformer computation
The key principle of our analysis is that the predominant limitation of a transformer model is in its communication bandwidth and not its computational abilities. We model transformers as having element-wise multi-layer perceptron units with unbounded computational ability (but bounded precision inputs and outputs) and self-attention units, which compute linear combinations of inputs in a carefully regimented way that limits the ability of individual elements to share information with one another. Here, we introduce a specific CONGEST graph for each sequence length  $N$  and show that every transformer has a communication protocol that simulates its computation in this graph.
For fixed  $N$ , we design an undirected CONGEST graph  $G^{N} = (V^{N}, E^{N})$  with  $O(N^{2})$  nodes, each having degree at most 3. (Note that this graph is not the same as the graph provided as input  $X$  to a transformer; this graph is consistent across all transformers taking input of sequence size  $N$ .) Let  $u_{1}, \ldots, u_{N}$  be nodes in  $V^{N}$  corresponding to each input. For every pair  $i, j \in [N]$ , let  $v_{i,j}$  be a node as well. For each  $i \in [N]$ , let  $B_{i} = (V_{i}, E_{i})$  be a balanced binary tree having root  $u_{i}$  and leaves  $v_{i,1}, \ldots, v_{i,N}, v_{1,i}, \ldots, v_{N,i}$ . Hence, each  $B_{i}$  has  $O(N)$  vertices of degree 3 and is of depth  $O(\log N)$ . Let  $V^{N} = V_{1} \cup \dots \cup V_{N}$  and  $E^{N} = E_{1} \cup \dots \cup E_{N}$ . Noting that  $E_{1}, \ldots, E_{N}$  are disjoint and that  $V_{1}, \ldots, V_{N}$  are disjoint, except for leaves  $v_{i,j}$ , we ascertain that  $G^{N}$  contains  $O(N^{2})$  vertices of degree at most 3 and has diameter  $O(\log N)$ . We visualize the graph  $G^{N}$  with a highlighted tree  $B_{1}$  in Figure 4.
Lemma 25. For any transformer  $f \in \mathcal{T}\mathcal{G}_{d,m,d',p}^{D,H}$  and any  $X \in \mathbb{R}^{N \times d}$  with  $p$ -bit fixed-precision numbers, there exists a CONGEST communication protocol on the graph  $G^{N}$  that shares  $p$  bits of information between adjacent vertices per round satisfying the following characteristics:
- Before any communication begins, each node  $u_{i}$  is provided with  $x_{i}$  and each node  $v_{i,j}$  is provided with  $x_{i,j}$  and  $x_{j,i}$ .
- After  $T = O(HD(m + \log N))$  rounds of communication, each node  $u_{i}$  outputs  $f(X)_{i}$ .