SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\phi (\cdot ;\cdot):\mathcal{X}\times \mathcal{W}\to [-B,B]$  is  $L$  -Lipschitz (in  $d_{\mathcal{X}}$  ) in the first argument, and  $\mu$  is a signed measure over  $\mathcal{W}$  with finite total measure  $A = \mathrm{TV}(\mu) < \infty$  . Then for any  $\varepsilon >0$  , there exists  $\alpha_{1},\dots ,\alpha_{K}\in \{\pm 1\}$ $\mathbf{w}_1,\dots ,\mathbf{w}_K\in \mathcal{W}$  with  $K = \mathcal{O}(A^{2}B^{2}\log \mathcal{N}(\mathcal{X},d_{\mathcal{X}},\frac{\varepsilon}{3AL}) / \varepsilon^2)$  , such that
$$
\sup  _ {\mathbf {x} \in \mathcal {X}} \left| g (\mathbf {x}) - \frac {A}{K} \sum_ {i = 1} ^ {K} \alpha_ {i} \phi (\mathbf {x}; \mathbf {w} _ {i}) \right| \leq \varepsilon ,
$$
where  $\mathcal{N}(\mathcal{X},d_{\mathcal{X}},\frac{\varepsilon}{3AL})$  denotes the  $(\frac{\varepsilon}{3AL})$ -covering number of  $\mathcal{X}$  in  $d_{\mathcal{X}}$ .
Proof. Let  $\alpha(\mathbf{w}) \coloneqq \mathrm{sign}(d\mu(\mathbf{w})) \in \{\pm 1\}$  denote the sign of the density  $d\mu(\mathbf{w})$ . We have
$$
g (\mathbf {x}) = A \int_ {\mathcal {W}} \alpha (\mathbf {w}) \phi (\mathbf {x}; \mathbf {w}) \times \frac {| d \mu (\mathbf {w}) |}{A}. \tag {9}
$$
Note that  $|d\mu(\mathbf{w})| / A$  is the density of a probability distribution over  $\mathcal{W}$ . Thus for any  $\mathbf{x} \in \mathcal{X}$ , as long as  $K \geq \mathcal{O}(A^2 B^2 \log(1/\delta) / \varepsilon^2)$ , we can sample  $\mathbf{w}_1, \ldots, \mathbf{w}_K \stackrel{\mathrm{iid}}{\sim} |d\mu(\cdot)| / A$ , and obtain by Hoeffding's inequality that with probability at least  $1 - \delta$ ,
$$
\left| g (\mathbf {x}) - \frac {A}{K} \sum_ {i = 1} ^ {K} \alpha \left(\mathbf {w} _ {i}\right) \phi \left(\mathbf {x}; \mathbf {w} _ {i}\right) \right| \leq \varepsilon .
$$
Let  $\mathcal{N}(\frac{\varepsilon}{3AL})\coloneqq \mathcal{N}(\mathcal{X},d_{\mathcal{X}},\frac{\varepsilon}{3AL})$  for shorthand. By union bound, as long as  $K\geq \mathcal{O}(A^{2}B^{2}\log (\mathcal{N}(\frac{\varepsilon}{3AL}) / \delta) / \varepsilon^{2})$  , we have with probability at least  $1 - \delta$  that for every  $\widehat{\mathbf{x}}$  in the covering set corresponding to  $\mathcal{N}(\frac{\varepsilon}{3AL})$
$$
\left| g (\widehat {\mathbf {x}}) - \frac {A}{K} \sum_ {i = 1} ^ {K} \alpha \left(\mathbf {w} _ {i}\right) \phi \left(\widehat {\mathbf {x}}; \mathbf {w} _ {i}\right) \right| \leq \varepsilon / 3.
$$
Taking  $\delta = 1/2$  (for which  $K = \mathcal{O}(A^2 B^2 \log \mathcal{N}(\frac{\varepsilon}{3AL}) / \varepsilon^2)$ ), by the probabilistic method, there exists a deterministic set  $\{\mathbf{w}_i\}_{i \in [K]} \subset \mathcal{W}$  and  $\{\alpha_i := \alpha(\mathbf{w}_i)\}_{i \in [K]} \in \{\pm 1\}$  such that the above holds.
Next, note that both  $g$  (by (9)) and the function  $\mathbf{x} \mapsto \frac{A}{K} \sum_{i=1}^{K} \alpha(\mathbf{w}_i) \phi(\mathbf{x}; \mathbf{w}_i)$  are  $(AL)$ -Lipschitz. Therefore, for any  $\mathbf{x} \in \mathcal{X}$ , taking  $\widehat{\mathbf{x}}$  to be the point in the covering set with  $d_{\mathcal{X}}(\mathbf{x}, \widehat{\mathbf{x}}) \leq \frac{\varepsilon}{3AL}$ , we have
$$
\begin{array}{l} \left| g (\mathbf {x}) - \frac {A}{K} \sum_ {i = 1} ^ {K} \alpha \left(\mathbf {w} _ {i}\right) \phi \left(\mathbf {x}; \mathbf {w} _ {i}\right) \right| \\ \leq | g (\mathbf {x}) - g (\widehat {\mathbf {x}}) | + \left| g (\widehat {\mathbf {x}}) - \frac {A}{K} \sum_ {i = 1} ^ {K} \alpha (\mathbf {w} _ {i}) \phi (\widehat {\mathbf {x}}; \mathbf {w} _ {i}) \right| + \left| \frac {A}{K} \sum_ {i = 1} ^ {K} \alpha (\mathbf {w} _ {i}) \phi (\widehat {\mathbf {x}}; \mathbf {w} _ {i}) - \frac {A}{K} \sum_ {i = 1} ^ {K} \alpha (\mathbf {w} _ {i}) \phi (\mathbf {x}; \mathbf {w} _ {i}) \right| \\ \leq A L \cdot \frac {\varepsilon}{3 A L} + \frac {\varepsilon}{3} + A L \cdot \frac {\varepsilon}{3 A L} = \varepsilon . \\ \end{array}
$$
This proves the lemma.
Proposition B.1 (Approximating smooth  $k$ -variable functions). For any  $\varepsilon_{\mathrm{approx}} > 0$ ,  $R \geq 1$ ,  $C_{\ell} > 0$ , we have the following: Any  $(R, C_{\ell})$ -smooth function (Definition B.1)  $g: \mathbb{R}^k \to \mathbb{R}$  is  $(\varepsilon_{\mathrm{approx}}, R, M, C)$ -approximable by sum of relus (Definition D.1) with  $M \leq C(k)C_{\ell}^2 \log(1 + C_{\ell} / \varepsilon_{\mathrm{approx}}) / \varepsilon_{\mathrm{approx}}^2$  and  $C \leq C(k)C_{\ell}$ , where  $C(k) > 0$  is a constant that depends only on  $k$ . In other words, there exists
$$
f (\mathbf {z}) = \sum_ {m = 1} ^ {M} c _ {m} \sigma \left(\mathbf {a} _ {m} ^ {\top} [ \mathbf {z}; 1 ]\right) \quad \text {w i t h} \quad \sum_ {m = 1} ^ {M} | c _ {m} | \leq C, \quad \max  _ {m \in [ M ]} \| \mathbf {a} _ {m} \| _ {1} \leq 1,
$$
such that  $\sup_{\mathbf{z}\in [-R,R]^k}|f(\mathbf{z}) - g(\mathbf{z})|\leq \varepsilon_{\mathrm{approx}}$
Proof. As function  $g: \mathsf{B}_{\infty}^{k}(R) \to \mathbb{R}$  is  $(R, C_{\ell})$ -smooth, we can apply Lemma B.5 to obtain that there exists a signed measure  $\mu$  over  $\mathcal{W} := \{\mathbf{w} \in \mathbb{R}^{k+1}: \| \mathbf{w} \|_1 \leq 1\}$  such that
$$
g (\mathbf {z}) = \int_ {\mathcal {W}} \frac {1}{R} \sigma (\mathbf {w} ^ {\top} [ \mathbf {z}; R ]) d \mu (\mathbf {w}), \quad \forall \mathbf {z} \in [ - R, R ] ^ {k},
$$
and  $A = \mathrm{TV}(\mu) \leq C(k)C_{\ell}$  where  $C(k) > 0$  denotes a constant depending only on  $k$ .
We now apply Lemma B.6 to approximate the above random feature by finitely many neurons. Let  $\mathbf{x} := [\mathbf{z}; R] \in \mathcal{X} := [-R, R]^k \times \{R\}$ . Then, the function  $\phi(\mathbf{x}; \mathbf{w}) := \frac{1}{R} \sigma(\mathbf{w}^\top \mathbf{x}) = \sigma(\frac{1}{R} \mathbf{w}^\top [\mathbf{z}; R])$  is bounded by  $B = 1$  and  $(1/R)$ -Lipschitz in  $\mathbf{x}$  (in the standard  $\ell_{\infty}$ -distance). Further, we have  $\log \mathcal{N}(\mathcal{X}, \| \cdot - \cdot \|_{\infty}, \frac{\varepsilon_{\mathrm{approx}}}{3A / R}) \leq \mathcal{O}(k \log(1 + A / \varepsilon_{\mathrm{approx}}))$ . We can thus apply Lemma B.6 to obtain that, for
$$
M = \mathcal {O} \big (k A ^ {2} \log (1 + A / \varepsilon_ {\mathrm {a p p r o x}}) / \varepsilon_ {\mathrm {a p p r o x}} ^ {2} \big) = C (k) C _ {\ell} ^ {2} \log (1 + C _ {\ell} / \varepsilon_ {\mathrm {a p p r o x}}) / \varepsilon_ {\mathrm {a p p r o x}} ^ {2},
$$
there exists  $\pmb{\alpha} = \{\alpha_{m}\}_{m\in [M]}\subset \{\pm 1\}$  and  $\mathbf{W} = \{\mathbf{w}_m\}_{m\in [M]}\subset \mathcal{W} = \{\mathbf{w}\in \mathbb{R}^{k + 1}:lone\mathbf{w} = 1\}$  such that
$$
\sup  _ {\mathbf {z} \in [ - R, R ] ^ {2}} | g (\mathbf {z}) - f _ {\boldsymbol {\alpha}, \mathbf {W}} (\mathbf {z}) | \leq \varepsilon_ {\text {a p p r o x}},
$$
where (recalling  $\mathbf{z} = [s;t])$
$$
f _ {\boldsymbol {\alpha}, \mathbf {W}} (\mathbf {z}) = \frac {A}{M} \sum_ {m = 1} ^ {M} \alpha_ {m} \sigma \bigg (\frac {1}{R} \mathbf {w} _ {m} ^ {\top} [ \mathbf {z}; R ] \bigg) = \sum_ {m = 1} ^ {M} \underbrace {\frac {A \alpha_ {m}}{M}} _ {c _ {m}} \sigma \bigg (\underbrace {\left[ \frac {1}{R} \mathbf {w} _ {m , 1 : k} ; w _ {m , k + 1} \right] ^ {\top}} _ {\mathbf {a} _ {m} ^ {\top}} [ \mathbf {z}; 1 ]).
$$
Note that we have  $\sum_{m=1}^{M}|c_m| = A \leq C(k)C_{\ell}$ , and  $\|\mathbf{a}_m\|_1 \leq \|\mathbf{w}_m\|_1 = 1$ . This is the desired result.
B.3 Optimization
The following convergence result for minimizing a smooth and strongly convex function is standard from the convex optimization literature, see e.g. Bubeck [13, Theorem 3.10].
Proposition B.2 (Gradient descent for smooth and strongly convex functions). Suppose  $L: \mathbb{R}^d \to \mathbb{R}$  is  $\alpha$ -strongly convex and  $\beta$ -smooth for some  $0 < \alpha \leq \beta$ . Then, the gradient descent iterates  $\mathbf{w}_{\mathrm{GD}}^{t+1} := \mathbf{w}_{\mathrm{GD}}^t - \eta \nabla L(\mathbf{w}_{\mathrm{GD}}^t)$  with learning rate  $\eta = 1/\beta$  and initialization  $\mathbf{w}_{\mathrm{GD}}^0 \in \mathbb{R}^d$  satisfies for any  $t \geq 1$ ,
$$
\left\| \mathbf {w} _ {\mathrm {G D}} ^ {t} - \mathbf {w} ^ {\star} \right\| _ {2} ^ {2} \leq \exp (- t / \kappa) \cdot \left\| \mathbf {w} _ {\mathrm {G D}} ^ {0} - \mathbf {w} ^ {\star} \right\| _ {2} ^ {2},
$$
$$
L (\mathbf {w} _ {\mathrm {G D}} ^ {t}) - L (\mathbf {w} ^ {\star}) \leq \frac {\beta}{2} \exp (- t / \kappa) \cdot \left\| \mathbf {w} _ {\mathrm {G D}} ^ {0} - \mathbf {w} ^ {\star} \right\| _ {2} ^ {2},
$$
where  $\kappa \coloneqq \beta /\alpha$  is the condition number of  $L$ , and  $\mathbf{w}^{\star}\coloneqq \arg \min_{\mathbf{w}\in \mathbb{R}^{d}}L(\mathbf{w})$  is the minimizer of  $L$ .
The following convergence result of proximal gradient descent (PGD) on convex composite minimization problem is also standard, see e.g. [8].
Proposition B.3 (Proximal gradient descent for convex function). Suppose  $L = f + h$ ,  $f: \mathbb{R}^d \to \mathbb{R}$  is convex and  $\beta$ -smooth for some  $\beta > 0$ ,  $h: \mathbb{R}^d \to \mathbb{R}$  is a simple convex function. Then, the proximal gradient descent iterates  $\mathbf{w}_{\mathrm{PGD}}^{t+1} := \mathbf{prox}_{\eta h}(\mathbf{w}_{\mathrm{PGD}}^t - \eta \nabla f(\mathbf{w}_{\mathrm{PGD}}^t))$  with learning rate  $\eta = 1/\beta$  and initialization  $\mathbf{w}_{\mathrm{GD}}^0 \in \mathbb{R}^d$  satisfies the following for any  $t \geq 1$ :
1.  $\{L(\mathbf{w}_{\mathrm{PGD}}^t)\}$  is a decreasing sequence.
2. For any minimizer  $\mathbf{w}^{\star}\in \arg \min_{\mathbf{w}\in \mathbb{R}^{d}}L(\mathbf{w})$
$$
L \left(\mathbf {w} _ {\mathrm {G D}} ^ {t + 1}\right) - L \left(\mathbf {w} ^ {\star}\right) \leq \frac {\beta}{2} \left(\left\| \mathbf {w} _ {\mathrm {P G D}} ^ {t} - \mathbf {w} ^ {\star} \right\| _ {2} ^ {2} - \left\| \mathbf {w} _ {\mathrm {P G D}} ^ {t + 1} - \mathbf {w} ^ {\star} \right\| _ {2} ^ {2}\right),
$$
and hence  $\left\{\| \mathbf{w}_{\mathrm{PGD}}^t -\mathbf{w}^\star \| _2^2\right\}$  is also a decreasing sequence.
3. For  $k \geq 1, t \geq 0$ , it holds that
$$
L \left(\mathbf {w} _ {\mathrm {G D}} ^ {t + k}\right) - L \left(\mathbf {w} ^ {\star}\right) \leq \frac {\beta}{2 k} \left\| \mathbf {w} _ {\mathrm {P G D}} ^ {t} - \mathbf {w} ^ {\star} \right\| _ {2} ^ {2}.
$$
B.4 Uniform convergence
The following result is shown in [87, Section 5.6].
Theorem B.1. Suppose that  $\psi :[0, + \infty)\to [0, + \infty)$  is a convex, non-decreasing function that satisfies  $\psi (x + y)\geq \psi (x)\psi (y)$ . For any random variable  $X$ , we consider the Orlicz norm induced by  $\psi$ :  $\| X\|_{\psi}\coloneqq \inf \left\{K > 0:\mathbb{E}\psi (|X| / K)\right\} \leq 1$
Suppose that  $\{X_{\theta}\}_{\theta}$  is a zero-mean random process indexed by  $\theta \in \Theta$  such that  $\| X_{\theta} - X_{\theta'}\|_{\psi} \leq \rho(\theta, \theta')$  for some metric  $\rho$  on the space  $\Theta$ . Then it holds that
$$
\mathbb {P} \left(\sup  _ {\theta , \theta^ {\prime} \in \Theta} | X _ {\theta} - X _ {\theta^ {\prime}} | \leq 8 (J + t)\right) \leq \frac {1}{\psi (t / D)} \forall t \geq 0,
$$
where  $D$  is the diameter of the metric space  $(\Theta, \rho)$ , and the generalized Dudley entropy integral  $J$  is given by
$$
J := \int_ {0} ^ {D} \psi^ {- 1} (N (\delta ; \Theta , \rho)) d \delta ,
$$
where  $N(\delta ;\Theta ,\rho)$  is the  $\delta$  -covering number of  $(\Theta ,\rho)$
As a corollary of Theorem B.1, we have the following result.
Proposition B.4 (Uniform concentration bound by chaining). Suppose that  $\{X_{\theta}\}_{\theta \in \Theta}$  is a zero-mean random process given by
$$
X _ {\theta} := \frac {1}{N} \sum_ {i = 1} ^ {N} f (z _ {i}; \theta) - \mathbb {E} _ {z} [ f (z; \theta) ],
$$
where  $z_{1},\dots ,z_{N}$  are i.i.d samples from a distribution  $\mathbb{P}_z$  such that the following assumption holds:
(a) The index set  $\Theta$  is equipped with a distance  $\rho$  and diameter  $D$ . Further, assume that for some constant  $A$ , for any ball  $\Theta'$  of radius  $r$  in  $\Theta$ , the covering number admits upper bound  $\log N(\delta; \Theta', \rho) \leq d \log (2Ar / \delta)$  for all  $0 < \delta \leq 2r$ .
(b) For any fixed  $\theta \in \Theta$  and  $z$  sampled from  $\mathbb{P}_z$ , the random variable  $f(z;\theta)$  is a  $\mathrm{SG}(B^0)$ -sub-Gaussian random variable.
(c) For any  $\theta, \theta' \in \Theta$  and  $z$  sampled from  $\mathbb{P}_z$ , the random variable  $f(z; \theta) - f(z; \theta')$  is a  $\mathrm{SG}(B^1 \rho(\theta, \theta'))$ -sub-Gaussian random variable.
Then with probability at least  $1 - \delta$ , it holds that
$$
\sup _ {\theta \in \Theta} | X _ {\theta} | \leq C B ^ {0} \sqrt {\frac {d \log (2 A \kappa) + \log (1 / \delta)}{N}},
$$
where  $C$  is a universal constant, and we denote  $\kappa = 1 + B^{1}D / B^{0}$
Furthermore, if we replace the SG in assumption (b) and (c) by SE, then with probability at least  $1 - \delta$ , it holds that
$$
\sup _ {\theta \in \Theta} | X _ {\theta} | \leq C B ^ {0} \left[ \sqrt {\frac {d \log (2 A \kappa) + \log (1 / \delta)}{N}} + \frac {d \log (2 A \kappa) + \log (1 / \delta)}{N} \right].
$$
Proof. Fix a  $D_0 \in (0, D]$  to be specified later. We pick a  $(D_0/2)$ -covering  $\Theta_0$  of  $\Theta$  so that  $\log |\Theta_0| \leq d\log(2AD / D_0)$ . Then, by the standard uniform covering of independent sub-Gaussian random variables, we have with probability at least  $1 - \delta/2$ ,
$$
\sup _ {\theta \in \Theta_ {0}} | X _ {\theta} | \leq C B ^ {0} \sqrt {\frac {d \log (2 A D / D _ {0}) + \log (2 / \delta)}{N}}.
$$
Assume that  $\Theta_0 = \{\theta_1, \dots, \theta_n\}$ . For each  $j \in [n]$ , we consider  $\Theta_j$  is the ball centered at  $\theta_j$  of radius  $D_0$  in  $(\Theta, \rho)$ . Then  $\theta \in \Theta_j$  has diameter  $D_0$  and admits covering number bound  $\log \mathcal{N}(\Theta_j, \delta) \leq d\log(AD_0 / \delta)$ . Hence, we can apply Theorem B.1 with the process  $\{X_\theta\}_{\theta \in \Theta_j}$ , then
$$
\psi = \psi_ {2}, \quad \| X _ {\theta} - X _ {\theta^ {\prime}} \| _ {\psi} \leq \frac {B ^ {1}}{\sqrt {N}} \rho (\theta , \theta^ {\prime}),
$$
and a simple calculation yields
$$
\mathbb {P} \left(\sup  _ {\theta , \theta^ {\prime} \in \Theta_ {j}} | X _ {\theta} - X _ {\theta^ {\prime}} | \leq C ^ {\prime} B ^ {1} D _ {0} \left(\sqrt {\frac {d \log (2 A)}{N}} + t\right)\right) \leq 2 \exp (- N t ^ {2}) \forall t \geq 0.
$$
Therefore, we can let  $t \leq \sqrt{\log(2n / \delta) / N}$  in the above inequality and taking the union bound over  $j \in [n]$ , and hence with probability at least  $1 - \delta / 2$ , it holds that for all  $j \in [n]$ ,
$$
\sup _ {\theta , \theta^ {\prime} \in \Theta_ {j}} | X _ {\theta} - X _ {\theta^ {\prime}} | \leq C ^ {\prime} B ^ {1} D _ {0} \sqrt {\frac {2 d \log (2 A D / D _ {0}) + \log (4 / \delta)}{N}}.
$$
Notice that for each  $\theta \in \Theta$ , there exists  $j \in [n]$  such that  $\theta \in \Theta_{j}$ , and hence
$$
\left| X _ {\theta} \right| \leq \left| X _ {\theta_ {j}} \right| + \left| X _ {\theta} - X _ {\theta_ {j}} \right|.
$$
Thus, with probability at least  $1 - \delta$ , it holds
$$
\sup_{\theta \in \Theta}\left|X_{\theta}\right|\leq \sup_{\theta \in \Theta_{0}}\left|X_{\theta}\right| + \sup_{j}\sup_{\theta \in \Theta_{j}}\left|X_{\theta} - X_{\theta_{j}}\right|\leq C^{\prime \prime}(B_{0} + B^{1}D_{0})\sqrt{\frac{d\log(2AD / D_{0}) + \log(2 / \delta)}{N}}.
$$
Taking  $D_0 = D / \kappa$  completes the proof of SG case.
We next consider the SE case. The idea is the same as the SG case, but in this case we need to consider the following Orlicz-norm:
$$
\psi_ {N} (t) := \exp \left(\frac {N t ^ {2}}{t + 1}\right) - 1.
$$
Then Bernstein's inequality of SE random variables yields
$$
\left\| X _ {\theta} - X _ {\theta^ {\prime}} \right\| _ {\psi_ {N}} \leq C _ {0} B ^ {1} \rho (\theta , \theta^ {\prime})
$$
for some universal constant  $C_0$ . Therefore, we can repeat the argument above to deduce that with probability at least  $1 - \delta$ , it holds
$$
\sup _ {\theta \in \Theta} | X _ {\theta} | \leq C ^ {\prime \prime} (B _ {0} + B ^ {1} D _ {0}) \left[ \sqrt {\frac {d \log (2 A D / D _ {0}) + \log (2 / \delta)}{N}} + \frac {d \log (2 A D / D _ {0}) + \log (2 / \delta)}{N} \right].
$$
Taking  $D_0 = D / \kappa$  completes the proof.
B.5 Useful properties of transformers
The following result can be obtained immediately by "joining" the attention heads and MLP layers of two single-layer transformers.
Proposition B.5 (Joining parallel single-layer transformers). Suppose that  $P_{1}:\mathbb{R}^{(D_{0} + D_{1})\times N}\to \mathbb{R}^{D_{1}\times N},P_{2}:\mathbb{R}^{(D_{0} + D_{2})\times N}\to \mathbb{R}^{D_{2}\times N}$  are two sequence-to-sequence functions that are implemented by single-layer transformers, i.e. there exists  $\pmb{\theta}_{1},\pmb{\theta}_{2}$  such that
$$
\mathrm {T F} _ {\boldsymbol {\theta} _ {1}}: \mathbf {H} _ {1} = \left[ \begin{array}{c} \mathbf {h} _ {i} ^ {(0)} \\ \mathbf {h} _ {i} ^ {(1)} \end{array} \right] _ {1 \leq i \leq N} \in \mathbb {R} ^ {(D _ {0} + D _ {1}) \times N} \mapsto \left[ \begin{array}{c} \mathbf {H} ^ {(0)} \\ P _ {1} (\mathbf {H} _ {1}) \end{array} \right],
$$
$$
\mathrm {T F} _ {\boldsymbol {\theta} _ {2}}: \mathbf {H} _ {2} = \left[ \begin{array}{c} \mathbf {h} _ {i} ^ {(0)} \\ \mathbf {h} _ {i} ^ {(2)} \end{array} \right] _ {1 \leq i \leq N} \in \mathbb {R} ^ {(D _ {0} + D _ {2}) \times N} \mapsto \left[ \begin{array}{c} \mathbf {H} ^ {(0)} \\ P _ {2} (\mathbf {H} _ {2}) \end{array} \right].
$$
Then, there exists  $\pmb{\theta}$  such that for  $\mathbf{H}'$  that takes form  $\mathbf{h}_i' = [\mathbf{h}_i^{(0)};\mathbf{h}_i^{(1)};\mathbf{h}_i^{(2)}]$ , with  $\mathbf{h}_i^{(0)}\in \mathbb{R}^{D_0},\mathbf{h}_i^{(1)}\in \mathbb{R}^{D_1},\mathbf{h}_i^{(2)}\in \mathbb{R}^{D_2}$ , we have
$$
\mathrm {T F} _ {\boldsymbol {\theta}}: \mathbf {H} ^ {\prime} = \left[ \begin{array}{c} \mathbf {h} _ {i} ^ {(0)} \\ \mathbf {h} _ {i} ^ {(1)} \\ \mathbf {h} _ {i} ^ {(2)} \end{array} \right] _ {1 \leq i \leq N} \in \mathbb {R} ^ {(D _ {0} + D _ {1} + D _ {2}) \times N} \mapsto \left[ \begin{array}{c} \mathbf {H} ^ {(0)} \\ P _ {1} (\mathbf {H} _ {1}) \\ P _ {2} (\mathbf {H} _ {2}) \end{array} \right].
$$
Further,  $\pmb{\theta}$  has at most  $M\leq M_1 + M_2$  heads,  $D^{\prime}\leq D_{1}^{\prime} + D_{2}^{\prime}$  hidden dimension in its MLP layer, and norm bound  $\| \pmb {\theta}\| \leq \| \pmb {\theta}_1\| +\| \pmb {\theta}_2\|$
Proposition B.6 (Joining parallel multi-layer transformers). Suppose that  $P_{1}:\mathbb{R}^{(D_{0} + D_{1})\times N}\to \mathbb{R}^{D_{1}\times N},P_{2}:\mathbb{R}^{(D_{0} + D_{2})\times N}\to \mathbb{R}^{D_{2}\times N}$  are two sequence-to-sequence functions that are implemented by multi-layer transformers, i.e. there exists  $\theta_{1},\theta_{2}$  such that
Further,  $\pmb{\theta}$  has at most  $L\leq \max \{L_1,L_2\}$  layers,  $\max_{\ell \in [L]}M^{(\ell)}\leq \max_{\ell \in [L]}\left(M_1^{(\ell)} + M_2^{(\ell)}\right)$  heads,  $\max_{\ell \in [L]}D^{(\ell)}\leq \max_{\ell \in [L]}\left(D_1^{(\ell)} + D_2^{(\ell)}\right)$  hidden dimension in its MLP layer (understanding the size of the empty layers as 0), and norm bound  $\| \pmb {\theta}\| \leq \| \pmb {\theta}_1\| +\| \pmb {\theta}_2\|$
Proof. When  $L_{1} = L_{2}$  ( $\theta_{1}$  and  $\theta_{2}$  have the same number of layers), the result follows directly by applying Proposition B.5 repeatedly for all  $L_{1}$  layers and the definition of the norm (2).
If (without loss of generality)  $L_{1} < L_{2}$ , we can augment  $\pmb{\theta}_{1}$  to  $L_{2}$  layers by adding  $(L_{2} - L_{1})$  layers with zero attention heads, and zero MLP hidden dimension (note that this does not change  $M_{1}, D_{1}^{\prime}$  and  $\| \pmb{\theta}_1\|$ ). Due to the residual structure, the transformer maintains the output  $P_{1}(\mathbf{H}_{1})$  throughout layer  $L_{1} + 1,\dots ,L_{2}$ , and it reduces to the case  $L_{1} = L_{2}$ .
C Extension to decoder-based architecture
Here we briefly discuss how our theoretical results can be adapted to decoder-based architectures (henceforth decoder TFs). Adopting the setting as in Section 2, we consider a sequence of  $N$  input vectors  $\{\mathbf{h}_i\}_{i=1}^N \subset \mathbb{R}^D$ , written compactly as an input matrix  $\mathbf{H} = [\mathbf{h}_1, \ldots, \mathbf{h}_N] \in \mathbb{R}^{D \times N}$ . Recall that  $\sigma(t) \coloneqq \mathrm{ReLU}(t) = \max \{t, 0\}$  denotes the standard relu activation.
C.1 Decoder-based transformers
Decoder TFs are the same as encoder TFs, except that the attention layers are replaced by masked attention layers with a specific decoder-based (causal) attention mask.
Definition C.1 (Masked attention layer). A masked attention layer with  $M$  heads is denoted as  $\mathrm{MAttn}_{\pmb{\theta}}(\cdot)$  with parameters  $\pmb{\theta} = \{(\mathbf{V}_m, \mathbf{Q}_m, \mathbf{K}_m)\}_{m \in [M]} \subset \mathbb{R}^{D \times D}$ . On any input sequence  $\mathbf{H} \in \mathbb{R}^{D \times N'}$  with  $N' \leq N$ ,
$$
\widetilde {\mathbf {H}} = \operatorname {M A t t n} _ {\boldsymbol {\theta}} (\mathbf {H}) := \mathbf {H} + \sum_ {m = 1} ^ {M} \left(\mathbf {V} _ {m} \mathbf {H}\right) \times \left(\left(\operatorname {M S K} _ {1: N ^ {\prime}, 1: N ^ {\prime}}\right) \circ \sigma \left(\left(\mathbf {Q} _ {m} \mathbf {H}\right) ^ {\top} \left(\mathbf {K} _ {m} \mathbf {H}\right)\right)\right) \in \mathbb {R} ^ {D \times N ^ {\prime}}, \tag {10}
$$
where  $\circ$  denotes the entry-wise (Hadamard) product of two matrices, and  $\mathrm{MSK} \in \mathbb{R}^{N \times N}$  is the mask matrix given by
$$
\mathrm {M S K} = \left[ \begin{array}{c c c c c} 1 & 1 / 2 & 1 / 3 & \dots & 1 / N \\ 0 & 1 / 2 & 1 / 3 & \dots & 1 / N \\ 0 & 0 & 1 / 3 & \dots & 1 / N \\ \dots & \dots & \dots & \dots & \dots \\ 0 & 0 & 0 & \dots & 1 / N \end{array} \right].
$$
In vector form, we have
$$
\widetilde {\mathbf {h}} _ {i} = [ \mathrm {A t t n} _ {\pmb {\theta}} (\mathbf {H}) ] _ {i} = \mathbf {h} _ {i} + \sum_ {m = 1} ^ {M} \frac {1}{i} \sum_ {j = 1} ^ {i} \sigma (\langle \mathbf {Q} _ {m} \mathbf {h} _ {i}, \mathbf {K} _ {m} \mathbf {h} _ {j} \rangle) \cdot \mathbf {V} _ {m} \mathbf {h} _ {j}.
$$
Notice that standard masked attention definitions use the pre-activation additive masks (with mask value  $-\infty$ ) [84]. The post-activation multiplicative masks we use is equivalent to the pre-activation additive masks, and the modified presentation is for notational convenience. We also use a normalized ReLU activation  $t \mapsto \sigma(t)/i$  in place of the standard softmax activation to be consistent with Definition 1. Note that the normalization  $1/i$  is to ensure that the attention weights  $\{\sigma(\langle \mathbf{Q}_m \mathbf{h}_i, \mathbf{K}_m \mathbf{h}_j \rangle)/i\}_{j \in [i]}$  is a set of non-negative weights that sum to  $O(1)$ . The motivation of masked attention layer is to ensure that, when processing a sequence of tokens, the computations at any token do not see any later token.
We next define the decoder-based transformers with  $L \geq 1$  transformer layers, each consisting of a masked attention layer (c.f. Definition C.1) followed by an MLP layer (c.f. Definition 2). This definition is similar to the definition of encoder-based transformers (c.f., Definition 3), except that we replace the attention layers by masked attention layers.
Definition C.2 (Decoder-based Transformer). An  $L$ -layer decoder-based transformer, denoted as  $\mathrm{DTF}_{\theta}(\cdot)$ , is a composition of  $L$  self-attention layers each followed by an MLP layer:  $\mathbf{H}^{(L)} = \mathrm{DTF}_{\theta}(\mathbf{H}^{(0)})$ , where  $\mathbf{H}^{(0)} \in \mathbb{R}^{D \times N}$  is the input sequence, and
$$
\mathbf {H} ^ {(\ell)} = \mathrm {M L P} _ {\boldsymbol {\theta} _ {\mathrm {m l p}} ^ {(\ell)}} \Big (\mathrm {M A t t n} _ {\boldsymbol {\theta} _ {\mathrm {m a t t n}} ^ {(\ell)}} \big (\mathbf {H} ^ {(\ell - 1)} \big) \Big), \quad \ell \in \{1, \ldots , L \}.
$$