SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1a visualizes the sparse averaging task as a bipartite graph between subsets  $y_{i}$  and elements  $z_{i}$  with corresponding averages. Theorems 2 and 4 jointly show that the minimum embedding dimension  $m$  of single self-attention units  $\mathcal{A}_{d,m,d',p}'$  that  $O\left(\frac{1}{q}\right)$ -approximate qSA scales linearly with  $q$ . We believe that the sparse averaging problem is thus a canonical problem establishing the representational capabilities and inductive biases of self-attention units.
3.1 Self-attention can approximate qSA when  $m\gtrsim q$
Our principle positive result shows that the sparse averaging task  $q\mathrm{SA}$  can be approximately solved using fixed-precision arithmetic self-attention units with embedding dimension  $m$  growing with  $q\log N$ .
Theorem 2 (Fixed-precision). For any  $N$ , any  $m \geq \Omega(d' + q \log N)$ , any  $\epsilon \in (0,1)$ , and  $p = \Omega(\log(\frac{q}{\epsilon} \log N))$ , there exists some  $f \in \mathcal{A}_{d,m,d',p}'$  that  $\epsilon$ -approximates qSA.
While the full proof appears in Appendix B.1, we briefly sketch the argument here. Because the output of a self-attention unit is a convex combination of rows of the value matrix  $\phi(X)V \in \mathbb{R}^{N \times d'}$ , a natural way to approximate  $q$  SA with a unit of self-attention is to let each value be the corresponding
2A simple construction involves letting  $XQ = XK$  with iid Gaussian columns fixed for every index  $i$ . Then, the diagonals of  $XQK^{\top}X^{\top}$  are far larger than all other entries and its softmax is approximately  $I_N$ .
<sup>3</sup>We may encode a  $q$  element subset of  $[N]$  as a vector in  $[N]^q$  constrained to have distinct components.
(a) Bipartite graph relating  $y_{i}$  and  $z_{i}$  in  $q\mathrm{SA}(X)$ .
(b) Attention and value matrices used for the self-attention construction of  $q\mathrm{SA}(X)$  in Theorem 2.
(c) Key and query embeddings that produce the self-attention matrix in (b).
(a)  $T = 0$
Figure 2: Attention matrix  $\mathrm{softmax}(\phi (X)QK^{\top}\phi (X)^{\top})\in \mathbb{R}^{20\times 20}$  for a fixed example after  $T$  epochs of training a self-attention unit to solve qSA for  $q = 3$ . Each row  $i$  corresponds to subset  $y_{i}$ , and each cell  $j\in y_{i}$  is outlined in red. See Appendix D for experimental details.
(b)  $T = 1000$
Figure 1: A visualization of the  $q$  SA function outputs given a sequence of inputs  $(z_{i};y_{i};i)_{i\in [N]}$  as a bipartite graph between subsets  $y_{i}$  and vectors  $z_{i}$  (a), and of the attention matrix (b) and underlying embeddings (c) that produce the self-attention construction in Theorem 2.
(c)  $T = 40000$
vector in the average (i.e.  $V^{\mathrm{T}}\phi(x_i) = z_i$ ) and choose the key and query functions in order to ensure that the attention matrix satisfies
$$
\operatorname {s o f t m a x} (\phi (X) Q K ^ {\intercal} \phi (X) ^ {\intercal}) _ {i, j} \approx \left\{ \begin{array}{l l} \frac {1}{q} & \text {i f} j \in y _ {i}, \\ 0 & \text {o t h e r w i s e}. \end{array} \right.
$$
To do so, let each key  $K^{\top}\phi (x_{i})$  represent a fixed vertex on a convex polytope, which depends only on index  $i$  and is constructed from random binary vectors. We select each query  $Q^{\top}\phi (x_{i})$  to ensure that  $\phi (x_{i})^{\top}QK^{\top}\phi (x_{j})$  is a fixed large value if  $j\in y_{i}$  and a slightly smaller value otherwise. We obtain the precise query, key, and value embeddings by employing tools from dual certificate analysis from the theory of compressed sensing.
We visualize this construction in Figure 1b and 1c for  $q = 3$  and  $d' = 4$ , which presents the associated attention and value matrices necessary for the construction, and plots a polytope of keys (red dots) with each face corresponding to each subset  $y_i$  (green dots). The construction is empirically relevant; Figure 2 shows that a unit of self-attention trained on data generated by the  $q$ SA task recovers a similar attention matrix to the one stipulated in our construction and visualized in Figure 1b.
The logarithmic dependence of the embedding dimension  $m$  on the sequence length  $N$  can be eliminated by considering self-attention units with real-valued arithmetic with infinite bit complexity.
Theorem 3 (Infinite-precision). For fixed  $N$ ,  $m \geq \Omega(d' + q)$  and  $\epsilon > 0$ , there exists some  $f \in \mathcal{A}_{d,m,d'}'$  that  $\epsilon$ -approximates  $q$  SA.
Figure 3: The  $mp$ -bit communication protocol used to reduce the hardness of computing  $q$ SA with a single unit of self-attention to the hardness of solving the DISJ communication problem for the proof of Theorem 4 for  $q = 4$ .
The proof of Theorem 3 employs a similar polytope-based construction in Appendix B.2, relying on a cyclic polytope rather than one drawn from discrete boolean vectors. Theorem 16 proves the near-optimality of that bound by employing a geometric argument to show that a variant of  $q\mathrm{SA}$  can only be approximated by a restricted family of self-attention units with a sufficiently high-dimensional embedding.
3.2 Self-attention cannot approximate  $q$  SA when  $m \lesssim q$
We show that the construction used to prove Theorem 2 is nearly optimal.
Theorem 4. For any sufficiently large  $q$ , any  $N \geq 2q + 1$ , and any  $d' \geq 1$ , there exists a universal constant  $c$  such that if  $mp \leq cq$ , then no  $f \in T_{d,m,d',p}^{1,1}$  exists that  $\frac{1}{2q}$ -approximates qSA.
(By choosing  $p = O(\log (q\log N))$ , Theorem 2 is shown to be optimal up to logarithmic factors of  $q$  and doubly-logarithmic factors of  $N$ .)
The proof of Theorem 4 employs a standard communication complexity argument based on a reduction from the following set disjointness problem in the two-party communication model, in which each party possesses a subset of an  $n$  element domain (coded as  $n$ -bit strings), and they wish to jointly determine whether their subsets are disjoint. We note that communication complexity is commonly-used technique for proving lower bounds on the representational power of circuits and feedforward neural networks [see, e.g., Karchmer and Wigderson, 1988, Ben-David et al., 2002, Martens et al., 2013, Vardi et al., 2021].
Fact 5 (Set disjointness communication lower bound [Yao, 1979]). Suppose Alice and Bob are given inputs  $a, b \in \{0,1\}^n$ , respectively, with the goal of jointly computing  $\mathrm{DISJ}(a,b) = \max_i a_i b_i$  by alternately sending a single bit message to the other party over a sequence of communication rounds. Any deterministic protocol for computing  $\mathrm{DISJ}(a,b)$  requires at least  $n$  rounds of communication.
Our proof designs a communication protocol that Alice and Bob use to jointly compute  $\mathrm{DISJ}(a,b)$  when  $n = q$  in  $O(mp)$  rounds of communication, under the assumption that such an  $f$  exists that closely approximates  $q\mathrm{SA}$ .
- Alice encodes her input  $a$  in a single subset by letting  $y_{2q + 1} = \{2i + a_i - 1 : i \in [q]\}$ .
- Bob uses his input  $b$  to assign  $z_{2i - 1}$  to  $2b_{i} - 1$  and  $z_{2i} = -1$  for all  $i \in [q]$ .
- All other input components are set to constant values known by both parties.
Alice sends her  $mp$ -bit query embedding  $Q^{\top} \phi(x_{2q+1})$  bit-by-bit to Bob, who approximately computes  $q \mathrm{SA}$  by determining the outcome of  $f$ . The crux of the reduction shows that  $q \mathrm{SA}(X)_{2q+1} = -1$  if and only if  $a_i b_i = 0$  for all  $i \in [q]$ , which allows Bob to determine  $\mathrm{DISJ}(a, b)$ .
We visualize the protocol in Figure 3 and give the proof in Appendix B.3. The proofs of Theorems 7, 11, 21, and 23 employ similar communication complexity reductions to DISJ.
4 Standard transformer models can only efficiently represent intrinsically pairwise functions
In this section, we argue that the standard transformer architecture is unable to efficiently represent functions that do not decompose into a small number of pairwise-symmetric functions. We do this by contrasting the (in)approximability of intrinsically pairwise and triple-wise functions, respectively Match2 and Match3 (defined in (1) and (2)), and their variants.
4.1 Efficient computation of Match2 with standard self-attention
We first show that Match2 can be efficiently approximated by a single standard (pairwise) self-attention unit.
Theorem 6. For any input size  $N$ , input range  $M = N^{O(1)}$ , and fixed-precision bit complexity  $p = O(\log M)$ , there exists a transformer architecture  $f \in \mathcal{T}_{1,m,1,p}^{1,1}$  with a single self-attention unit with embedding dimension  $m = 3$  such that for all  $X \in [M]^N$ ,  $f(X) = \mathrm{Match2}(X)$ .
The proof, given in Appendix C.1 uses both a "blank token" and a trigonometric positional embedding, which ensures that
$$
\phi (x _ {i}) ^ {\intercal} Q K ^ {\intercal} \phi (x _ {j}) = c \sum_ {k = 1} ^ {d} \cos \left(\frac {2 \pi (x _ {i , k} + x _ {j , k})}{M}\right)
$$
for some sufficiently large constant  $c$ . This embedding ensures that a cell of the attention matrix  $\operatorname{softmax}(\phi(X)QK^{\top}\phi(X)^{\top})_{i,j}$  is extremely close to zero, unless  $x_i = -x_j$  (mod  $M$ ).
4.2 Hardness of computing Match3 with a multi-headed self-attention layer
Although Match2 can be efficiently represented using a single unit of standard self-attention, representing Match3 using an entire layer of multi-headed attention units is impossible unless either the number of heads  $H$ , the embedding dimension  $m$ , or the precision  $p$  grows as  $N^{\Omega(1)}$ .
Theorem 7. There is universal constant  $c > 0$  such that for sufficiently large  $N$ , and any  $M \geq N + 1$ , if  $mpH \leq cN / \log \log N$ , then there is no  $f \in T_{1,m,1,p}^{1,H}$  satisfying  $f(X) = \operatorname{Match3}(X)$  for all  $X \in [M]^N$ .
We give the proof in Appendix C.2. Like that of Theorem 4, the proof relies on a reduction from set disjointness in two-party communication. The proof of the lower bound applies a domain-restricted variant of Match3, which actually makes the problem substantially simpler to solve. In Remark 1, we show how this variant of Match3 introduces a depth separation between the representational powers of single-layer and two-layer transformer models.
As mentioned in the introduction, we also conjecture that multiple layers of multi-headed attention are subject to the same impossibility (Conjecture 19). The impossibility is specific to standard (pairwise) attention; in Appendix C.4, we show that Match3 can be efficiently computed with a single unit of third-order self-attention.
4.3 More efficient constructions for simplified Match3 computations
While the previous sections suggest that no efficient construction exists to compute Match3 with standard transformer models, practical examples of triple detection abound. For example, a transformer-based language model will likely succeed in linking a subject/verb/object triple because all three tokens likely inhabit the same local region and because the model could agglomerate the triple by first identifying a pair and then adding the third. Here, we introduce two variants on the Match3 problem that have additional structure to serve as hints. The first variant specifies triple sums comprising the input element and a neighboring pair elsewhere in the sequence: for each  $i \in [N]$ ,
$$
\operatorname {M a t c h 3 B i g r a m} (X) _ {i} = \mathbb {1} \left\{\exists j \text {s . t .} x _ {i} + x _ {j} + x _ {j + 1} = 0 (\mathrm {m o d} M) \right\}.
$$
The second focuses on localized sums, where are all components of a triple must be within a fixed range of constant width  $K \ll N$ : for each  $i \in [N]$ ,
$$
\operatorname {M a t c h 3 L o c a l} (X) _ {i} = \mathbb {1} \left\{\exists j _ {1}, j _ {2} \text {s . t .} x _ {i} + x _ {j _ {1}} + x _ {j _ {2}} = 0 (\mathrm {m o d} M), | i - j _ {1} |, | i - j _ {2} | \leq K \right\}.
$$
We show that the two can be efficiently represented using compact standard transformer models.
Theorem 8. For any  $N$ ,  $M = N^{O(1)}$ , and  $p = O(\log M)$ , there exists a transformer architecture  $f \in \mathcal{T}_{1,m,1,p}^{D,1}$  with embedding dimension  $m = 3$  and depth  $D = 2$  such that for all  $X \in [M]^{N \times d}$ ,  $f(X) = \operatorname{Match3Bigram}(X)$ .
Informally, the first layer of the construction uses a sinusoidal positional encoding to compute each bigram sum  $x_{j} + x_{j+1}$  in the  $j$ th element of the sequence. The second layer applies the Match2 construction provided by Theorem 6 to determine whether there exists a  $j$  for each  $i$  such that  $x_{i} + x_{j} + x_{j+1} = 0 \pmod{M}$ .
Theorem 9. For any  $d$ ,  $N$ ,  $M = N^{O(1)}$ ,  $p = O(\log M)$ , and  $K \leq N$ , there exists a transformer architecture  $f \in \mathcal{T}_{1,m,1,p}^{1,1}$  with embedding dimension  $m = O(K \log N)$  and bit-complexity  $p = O(\log (K \log N))$  such that for all  $X \in [M]^{N \times d}$ ,  $f(X) = \mathrm{Match3Local}(X)$ .
Proof. We implement the localized construction by using Theorem 2 to construct a specific sparse simultaneous average of the inputs with  $q \coloneqq 2K + 1$  and  $d' \coloneqq 2K + 1$ . To do so, we use the input MLP to convert  $x_{i}$  to the embedding  $(z_{i};y_{i};i)$ , for zero-padded input
$$
z _ {i} = x _ {i} e _ {\bar {i}} \in \mathbb {R} ^ {2 K + 1}
$$
for  $\bar{i} = i$  (mod  $2K + 1$ ) and subset
$$
y _ {i} = \{i - K, i - K + 1, \ldots , i + K \} \in \left( \begin{array}{c} [ N ] \\ 2 K + 1 \end{array} \right).
$$
This construction ensures that the  $i$ th element of self-attention output computes (a rotation of)  $(x_{i - K}, x_{i - K + 1}, \ldots, x_{i + K})$ . An output MLP can then verify whether any matching triples involving  $x_i$  exist among those vectors.
We are grateful for many discussions with and feedback from Navid Ardeshir, Peter Bartlett, Alberto Bietti, Yuval Efron, Christos Papadimitriou, Shivam Nadimpalli, Rocco Servedio, Yusu Wang, and Cyril Zhang. This work was supported in part by NSF grants CCF-1740833 and IIS-1563785, a JP Morgan Faculty Award, and an NSF Graduate Research Fellowship.
Anders Aamand, Justin Chen, Piotr Indyk, Shyam Narayanan, Ronitt Rubinfeld, Nicholas Schiefer, Sandeep Silwal, and Tal Wagner. Exponentially improving the complexity of simulating the Weisfeiler-Lehman test with graph neural networks. In Advances in Neural Information Processing Systems 35, 2022.
Dana Angluin. Local and global properties in networks of processors. In Proceedings of the Twelfth Annual ACM Symposium on Theory of Computing, 1980.
Shai Ben-David, Nadav Eiron, and Hans Ulrich Simon. Limitations of learning via embeddings in euclidean half spaces. Journal of Machine Learning Research, 3(Nov):441-461, 2002.
Satwik Bhattachamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transformers to recognize formal languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020.
Satwik Bhattachamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in transformers and their ability to learn sparse boolean functions. arXiv preprint arXiv:2211.12316, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
Colin Wei, Yining Chen, and Tengyu Ma. Statistically meaningful approximation: a case study on approximating tuning machines with transformers. Advances in Neural Information Processing Systems, 35:12071-12083, 2022.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.
Andrew Chi-Chih Yao. Some complexity questions related to distributive computing (preliminary report). In Proceedings of the Eleventh Annual ACM Symposium on Theory of Computing, 1979.
Shunyu Yao, Binghui Peng, Christos H. Papadimitriou, and Karthik Narasimhan. Self-attention networks can process bounded hierarchical languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021.
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems 30, 2017.
Günter M Ziegler. Lectures on polytopes. Graduate Texts in Mathematics, 152, 2006.
Aaron Zweig and Joan Bruna. Exponential separations in symmetric neural networks. CoRR, abs/2206.01266, 2022. doi: 10.48550/arXiv.2206.01266.
A Fully-connected neural networks and recurrent neural networks cannot efficiently approximate qSA
A.1 Only wide fully-connected neural networks can approximate  $q$ SA
In this section, we show that any fully-connected neural network that approximates  $q\mathrm{SA}:\mathbb{R}^{Nd}\to \mathbb{R}^{Nd^{\prime}}$  must have width  $m = \Omega (N)$ . We consider networks of the form  $f(x) = g(Wx)$  for some weight matrix  $W\in \mathbb{R}^{m\times Nd}$  (the first layer weights) and arbitrary function  $g:\mathbb{R}^m\rightarrow \mathbb{R}^{Nd'}$  (computed by subsequent layers of a neural network).
Theorem 10. Suppose  $q \leq \frac{N}{2}$ . Any fully-connected neural network  $f$  defined as above that  $\frac{1}{2q}$ -approximates qSA satisfies  $m \geq \operatorname{rank}(W) \geq \frac{Nd'}{2}$ .
Proof. For simplicity, we arrange the input as
$$
x = (1; \dots ; N; y _ {1}; \dots ; y _ {N}; z _ {1}; \dots ; z _ {N})
$$
and  $W = [\tilde{W};V_1;\ldots ;V_N]$  with  $z_{1},\ldots ,z_{N}\in \mathbb{B}^{d^{\prime}}$ $\tilde{W}\in \mathbb{R}^{m\times N(d - d^{\prime})}$ , and  $V_{1},\dots,V_{N}\in \mathbb{R}^{m\times d^{\prime}}$ . If  $\mathrm{rank}(W)\leq \frac{Nd'}{2} -1$ , then so too is  $\mathrm{rank}([V_q;\ldots ;V_N])\leq \frac{Nd'}{2} -1$ , and  $[V_q;\ldots ;V_N]$  has a nontrivial null space containing a nonzero vector  $u = (u_{q};\ldots ;u_{N})\in \mathbb{R}^{(N - q)d^{\prime}}$ . Let
$$
\xi = \frac {1}{\max _ {j \in \{q , . . . , N \}} \| u _ {j} \| _ {2}} (u _ {q}; \ldots ; u _ {N}),
$$
$z = (\vec{0};\ldots ;\vec{0};\xi_q;\ldots ;\xi_N)$  , and  $z^{\prime} = (\vec{0};\dots ;\vec{0}; - \xi_{q};\dots ; - \xi_{N})$  . Then,
1.  $z_{j},z_{j}^{\prime}\in \mathbb{B}^{d^{\prime}}$  for all  $j\in [N]$
2.  $V_{j}z_{j} = V_{j}z_{j}^{\prime} = 0$  for all  $j\in [N]$ ; and
3.  $\| z_{j^{*}} - z_{j^{*}}^{\prime} \|_{2} = 2$  for some  $j^{*} \in \{q, \dots, N\}$ .
Therefore, for any  $y_{1}, \ldots, y_{N} \in \binom{[N]}{q}$ , respective  $x = (1; \ldots; N; y_{1}; \ldots; y_{N}; z_{1}; \ldots; z_{N})$  and  $x' = (1; \ldots; N; y_{1}; \ldots; y_{N}; z_{1}'; \ldots; z_{N}')$  satisfy  $f(x) = f(x')$ . Consider  $y$  with  $y_{j} = (1, \ldots, q - 1, j)$  for each  $j \in \{q, \ldots, N\}$ . Then,
$$
q \mathrm {S A} (x) _ {j} = \frac {1}{q} \xi_ {j} \mathrm {a n d} q \mathrm {S A} (x ^ {\prime}) _ {j} = - \frac {1}{q} \xi_ {j}.
$$
Hence,  $\| q\mathrm{SA}(x)_{j^*} - q\mathrm{SA}(x')_{j^*}\| _2\geq \frac{2}{q}$ . Because  $f(x) = f(x')$
$$
\max \left(\| f (x) - q \mathrm {S A} (x) _ {j ^ {*}} \| _ {2}, \| f (x ^ {\prime}) - q \mathrm {S A} (x ^ {\prime}) _ {j ^ {*}} \| _ {2}\right) \geq \frac {1}{q},
$$
so  $f$  can approximate  $q\mathrm{SA}$  to accuracy no better than  $\frac{1}{q}$ .
A.2 Only high-memory recurrent neural networks can approximate  $q\mathrm{SA}$
In this section, we show that any memory-bounded algorithm that approximates  $q\mathrm{SA}:\mathbb{R}^{N\times d}\to \mathbb{R}^{N\times d'}$  must use a large "hidden state" (memory) as it processes the input elements. This lower bound applies to various recurrent neural network (RNN) architectures.
A memory-bounded algorithm with an  $m$ -bit memory processes input  $X \in \mathbb{R}^{N \times d}$  sequentially as follows. There is an initial memory state  $h_0 \in \{0,1\}^m$ . For  $i = 1,2,\dots,N$ , the algorithm computes the  $i$ -th output  $f(X)_i \in \mathbb{R}^{d'}$  and the updated memory state  $h_i$  as a function of the input  $x_i \in \mathbb{R}^d$  and previous memory state  $h_{i-1}$ :
$$
(f (X) _ {i}, h _ {i}) = g _ {i} (x _ {i}, h _ {i - 1}),
$$
4We regard inputs as  $Nd$  -dimensional vectors rather than  $N\times d$  matrices.
where  $g_{i} \colon \mathbb{R}^{d} \times \{0,1\}^{m} \to \mathbb{R}^{d^{\prime}} \times \{0,1\}^{m}$  is permitted to be an arbitrary function, and  $f \colon \mathbb{R}^{N \times d} \to \mathbb{R}^{N \times d^{\prime}}$  is the function computed by the algorithm.
Our lower bound applies to algorithms that only need to solve the subclass of "causal" instances of  $q\mathrm{SA}$  in which the input  $X = ((z_{i},y_{i},i))_{i\in [N]}\in \mathbb{R}^{N\times d}$  is promised to satisfy  $y_{i} = \emptyset$  for all  $i\leq N / 2 + 1$ , and  $y_{i}\subseteq \{1,\ldots ,N / 2 + 1\}$  for all  $i > N / 2 + 1$ .
Theorem 11. For any  $\varepsilon \in (0,1)$ , any memory-bounded algorithm that  $\varepsilon$ -approximates qSA (for  $q = 1$  and  $d' = 1$ ) on the subclass of "causal" instances must have memory  $m \geq (N - 1)/2$ .
Proof. Consider an  $m$ -bit memory-bounded algorithm computing a function  $f\colon \mathbb{R}^{N\times d}\to \mathbb{R}^N$  that  $\varepsilon$ -approximates  $q\mathrm{SA}$  (for  $q = 1$  and  $d^{\prime} = 1$ ). We construct, from this algorithm, a communication protocol for DISJ (with  $N = 2n + 1$ ) that uses  $m$  bits of communication.
Let  $a, b \in \{0, 1\}^n$  be the input for DISJ provided to Alice and Bob, respectively. The protocol is as follows.
1. Alice constructs inputs  $x_{i} = (z_{i},\emptyset ,i)$  for  $i = 1,\dots ,n + 1$ , where for each  $i = 1,\ldots ,n$
$$
z _ {i} = \left\{ \begin{array}{l l} + 1 & \text {i f} a _ {i} = 0, \\ - 1 & \text {i f} a _ {i} = 1, \end{array} \right.
$$
$$
z _ {n + 1} = + 1.
$$
Bob constructs inputs  $x_{n+1+i} = (0, y_{n+1+i}, n+1+i)$  for  $i = 1, \ldots, n$ , where
$$
y _ {n + 1 + i} = \left\{ \begin{array}{l l} \{n + 1 \} & \text {i f} b _ {i} = 0, \\ \{i \} & \text {i f} b _ {i} = 1. \end{array} \right.
$$
Observe that, for this input  $X = (x_{1},\ldots ,x_{2n + 1})$ , we have
$$
q \mathrm {S A} (X) _ {n + 1 + i} = \left\{ \begin{array}{l l} + 1 & \text {i f} a _ {i} b _ {i} = 0, \\ - 1 & \text {i f} a _ {i} b _ {i} = 1. \end{array} \right.
$$
2. Alice simulates the memory-bounded algorithm on the first  $n + 1$  inputs  $x_{1},\ldots ,x_{n + 1}$ , and sends Bob the  $m$ -bit memory state  $h_{n + 1}$ . This requires  $m$  bits of communication.
3. Starting with  $h_{n+1}$ , Bob continues the simulation of the memory-bounded algorithm on these  $n$  additional inputs  $x_{n+2}, \ldots, x_{2n+1}$ .
4. If any output  $f(X)_{n + 1 + i}$  for  $i = 1, \dots, n$  satisfies
$$
f (X) _ {n + 1 + i} <   0,
$$
then Bob outputs 1 (not disjoint); otherwise Bob outputs 0 (disjoint).
The approximation guarantee of  $f$  implies that  $\mathrm{sign}(f(X)_{n+1+i}) = q\mathrm{SA}(X)_{n+1+i}$  for all  $i = 1, \ldots, n$ , so Bob outputs 1 if and only if  $a$  and  $b$  are not disjoint. Because this protocol for DISJ uses  $m$  bits of communication, by Fact 5, it must be that  $m \geq n = (N-1)/2$ .