SlowGuess/NeurIPS_2023_OCR huggingface dataset:
According to the formulation (7):
$$
X _ {\cdot i} ^ {(t + 1)} = \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} W _ {j _ {k} j _ {k + 1}} ^ {(k)}\right) D _ {j _ {t + 1}} ^ {(t)} P ^ {(t)}... D _ {j _ {1}} ^ {(0)} P ^ {(0)} X _ {\cdot j _ {0}} ^ {(0)},
$$
we thus obtain that
$$
\begin{array}{l} \| X _ {\cdot i} ^ {(t + 1)} \| _ {\infty} = \left\| \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} W _ {j _ {k} j _ {k + 1}} ^ {(k)}\right) D _ {j _ {t + 1}} ^ {(t)} P ^ {(t)}... D _ {j _ {1}} ^ {(0)} P ^ {(0)} X _ {\cdot j _ {0}} ^ {(0)} \right\| _ {\infty} \\ \leq \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} \left| W _ {j _ {k} j _ {k + 1}} ^ {(k)} \right|\right) \left\| D _ {j _ {t + 1}} ^ {(t)} P ^ {(t)}... D _ {j _ {1}} ^ {(0)} P ^ {(0)} \right\| _ {\infty} \left\| X _ {. j _ {0}} ^ {(0)} \right\| _ {\infty} \\ \leq \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} \left| W _ {j _ {k} j _ {k + 1}} ^ {(k)} \right|\right) \left\| X _ {\cdot j _ {0}} ^ {(0)} \right\| _ {\infty} \\ \leq C _ {0} \left(\sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} \left| W _ {j _ {k} j _ {k + 1}} ^ {(k)} \right|\right)\right) \\ = C _ {0} \| \left(\left| W ^ {(0)} \right| \dots \left| W ^ {(t)} \right|\right) _ {\cdot i} \| _ {1}, \\ \end{array}
$$
where  $C_0$  equals the maximal entry in  $|X^{(0)}|$ .
The assumption A3 implies that there exists  $C' > 0$  such that for all  $t \in \mathbb{N}_{\geq 0}$  and  $i \in [d]$ ,
$$
\left\| \left(\left| W ^ {(0)} \right| \dots \left| W ^ {(t)} \right|\right) _ {\cdot i} \right\| _ {1} \leq C ^ {\prime} N.
$$
Hence there exists  $C'' > 0$  such that for all  $t \in \mathbb{N}_{\geq 0}$  and  $i \in [d]$ , we have
$$
\left\| X _ {\cdot i} ^ {(t)} \right\| _ {\infty} \leq C ^ {\prime \prime},
$$
proving the existence of  $C > 0$  such that  $\| X^{(t)}\|_{\max}\leq C$  for all  $t\in \mathbb{N}_{\geq 0}$
Lemma 2 is a direct corollary of Lemma 1 and the assumption that  $\Psi(\cdot, \cdot)$  assigns bounded attention scores to bounded inputs.
E Proof of Lemma 3
E.1 Auxiliary results
We make use of the following sufficient condition for the ergodicity of the infinite products of row-stochastic matrices.
Lemma 8 (Corollary 5.1 [14]). Consider a sequence of row-stochastic matrices  $\{S^{(t)}\}_{t = 0}^{\infty}$ . Let  $a_{t}$  and  $b_{t}$  be the smallest and largest entries in  $S^{(t)}$ , respectively. If  $\sum_{t = 0}^{\infty}\frac{a_t}{b_t} = \infty$ , then  $\{S^{(t)}\}_{t = 0}^{\infty}$  is ergodic.
In order to make use of the above result, we first show that long products of  $P^{(t)}$ 's from  $\mathcal{P}_{\mathcal{G},\epsilon}$  will eventually become strictly positive. For  $t_0 \leq t_1$ , we denote
$$
P ^ {\left(t _ {1}: t _ {0}\right)} = P ^ {\left(t _ {1}\right)} \dots P ^ {\left(t _ {0}\right)}.
$$
Lemma 9. Under the assumption A1, there exist  $T \in \mathbb{N}$  and  $c > 0$  such that for all  $t_0 \geq 0$ ,
$$
c \leq P _ {i j} ^ {\left(t _ {0} + T: t _ {0}\right)} \leq 1, \forall 1 \leq i, j \leq N.
$$
Proof. Fix any  $T \in \mathbb{N}_{\geq 0}$ . Since  $\| P^{(t)} \|_{\infty} \leq 1$  for any  $P^{(t)} \in \mathcal{P}_{\mathcal{G},\epsilon}$ , it follows that  $\| P^{(t_0 + T:t_0)} \|_{\infty} \leq 1$  and hence  $P_{ij}^{(t_0 + T:t_0)} \leq 1$ , for all  $1 \leq i,j \leq N$ .
To show the lower bound, without loss of generality, we will show that there exist  $T \in \mathbb{N}$  and  $c > 0$  such that
$$
P _ {i j} ^ {(T: 0)} \geq c, \forall 1 \leq i, j \leq N.
$$
Since each  $P^{(t)}$  has the same connectivity pattern as the original graph  $\mathcal{G}$ , it follows from the assumption A1 that there exists  $T \in \mathbb{N}$  such that  $P^{(T:0)}$  is a positive matrix, following a similar argument as the one for Proposition 1.7 in [23]: For each pair of nodes  $i,j$ , since we assume that the graph  $\mathcal{G}$  is connected, there exists  $r(i,j)$  such that  $P_{ij}^{(r(i,j):0)} > 0$ . On the other hand, since we also assume each node has a self-loop,  $P_{ii}^{(t:0)} > 0$  for all  $t \geq 0$  and hence for  $t \geq r(i,j)$ ,
$$
P _ {i j} ^ {(t: 0)} \geq P _ {i i} ^ {(t - r (i, j))} P _ {i j} ^ {(r (i, j): 0)} > 0.
$$
For  $t \geq t(i) := \max_{j \in \mathcal{G}} r(i, j)$ , we have  $P_{ij}^{(t:0)} > 0$  for all node  $j$  in  $\mathcal{G}$ . Finally, if  $t \geq T := \max_{i \in \mathcal{G}} t(i)$ , then  $P_{ij}^{(t:0)} > 0$  for all pairs of nodes  $i, j$  in  $\mathcal{G}$ . Notice that  $P_{ij}^{(T:0)}$  is a weighted sum of walks of length  $T$  between nodes  $i$  and  $j$ , and hence  $P_{ij}^{(T:0)} > 0$  if and only if there exists a walk of length  $T$  between nodes  $i$  and  $j$ . Since for all  $t \in \mathbb{N}_{\geq 0}$ ,  $P_{ij}^{(t)} \geq \epsilon$  if  $(i, j) \in E(\mathcal{G})$ , we conclude that  $P_{ij}^{(T:0)} \geq \epsilon^T := c$ .
E.2 Proof of Lemma 3
Given the sequence  $\{P^{(t)}\}_{t = 0}^{\infty}$ , we use  $T\in \mathbb{N}$  from Lemma 9 and define
$$
\bar {P} ^ {(k)} := P ^ {((k + 1) T: k T)}.
$$
Then  $\{P^{(t)}\}_{t = 0}^{\infty}$  is ergodic if and only if  $\{\bar{P}^{(k)}\}_{k = 0}^{\infty}$  is ergodic. Notice that by Lemma 9, for all  $k\in \mathbb{N}_{\geq 0}$ , there exists  $c > 0$  such that  $c\leq \bar{P}_{ij}^{(k)}\leq 1,\forall 1\leq i,j\leq N$ . Then Lemma 3 is a direct consequence of Lemma 8.
F Proof of Lemma 5
F.1 Notations and auxiliary results
Consider a sequence  $\{D^{(t)}P^{(t)}\}_{t = 0}^{\infty}$  in  $\mathcal{M}_{\mathcal{G},\epsilon}$ . For  $t_0\leq t_1$ , define
$$
Q _ {t _ {0}, t _ {1}} := D ^ {(t _ {1})} P ^ {(t _ {1})} \dots D ^ {(t _ {0})} P ^ {(t _ {0})}
$$
$$
\delta_ {t} = \| D ^ {(t)} - I _ {N} \| _ {\infty},
$$
where  $I_N$  denotes the  $N \times N$  identity matrix. It is also useful to define
$$
\begin{array}{l} \hat {Q} _ {t _ {0}, t _ {1}} := P ^ {(t _ {1})} Q _ {t _ {0}, t _ {1} - 1} \\ := P ^ {\left(t _ {1}\right)} D ^ {\left(t _ {1} - 1\right)} P ^ {\left(t _ {1} - 1\right)} \dots D ^ {\left(t _ {0}\right)} P ^ {\left(t _ {0}\right)}. \\ \end{array}
$$
We start by proving the following key lemma, which states that long products of matrices in  $\mathcal{M}_{\mathcal{G},\epsilon}$  eventually become a contraction in  $\infty$ -norm.
Lemma 10. There exist  $0 < c < 1$  and  $T \in \mathbb{N}$  such that for all  $t_0 \leq t_1$ ,
$$
\| \hat {Q} _ {t _ {0}, t _ {1} + T} \| _ {\infty} \leq (1 - c \delta_ {t _ {1}}) \| \hat {Q} _ {t _ {0}, t _ {1}} \| _ {\infty}.
$$
Proof. First observe that for every  $T \geq 0$ ,
$$
\begin{array}{l} \| \hat {Q} _ {t _ {0}, t _ {1} + T} \| _ {\infty} \leq \| P ^ {(t _ {1} + T)} D ^ {(t _ {1} + T - 1)} P ^ {(t _ {1} + T - 1)} \dots D ^ {(t _ {1} + 1)} P ^ {(t _ {1} + 1)} D ^ {(t _ {1})} \| _ {\infty} \| \hat {Q} _ {t _ {0}, t _ {1}} \| _ {\infty} \\ \leq \left\| P ^ {\left(t _ {1} + T\right)} P ^ {\left(t _ {1} + T - 1\right)} \dots P ^ {\left(t _ {1} + 1\right)} D ^ {\left(t _ {1}\right)} \right\| _ {\infty} \left\| \hat {Q} _ {t _ {0}, t _ {1}} \right\| _ {\infty}, \\ \end{array}
$$
where the second inequality is based on the following element-wise inequality:
$$
P ^ {(t _ {1} + T)} D ^ {(t _ {1} + T - 1)} P ^ {(t _ {1} + T - 1)} \dots D ^ {(t _ {1} + 1)} P ^ {(t _ {1} + 1)} \le_ {\mathrm {e w}} P ^ {(t _ {1} + T)} P ^ {(t _ {1} + T - 1)} \dots P ^ {(t _ {1} + 1)}.
$$
By Lemma 9, there exist  $T \in \mathbb{N}$  and  $0 < c < 1$  such that
$$
\left(P ^ {\left(t _ {1} + T\right)} \dots P ^ {\left(t _ {1} + 1\right)}\right) _ {i j} \geq c, \forall 1 \leq i, j \leq N.
$$
Since the matrix product  $P^{(t_1 + T)}P^{(t_1 + T - 1)}\ldots P^{(t_1 + 1)}$  is row-stochastic, multiplying it with the diagonal matrix  $D^{(t_1)}$  from right decreases the row sums by at least  $c(1 - D_{\min}^{(t_1)}) = c\delta_{t_1}$ , where  $D_{\min}^{(t_1)}$  here denotes the smallest diagonal entry of the diagonal matrix  $D^{(t_1)}$ . Hence,
$$
\left\| P ^ {\left(t _ {1} + T\right)} P ^ {\left(t _ {1} + T - 1\right)} \dots P ^ {\left(t _ {1} + 1\right)} D ^ {\left(t _ {1}\right)} \right\| _ {\infty} \leq 1 - c \delta_ {t _ {1}}.
$$
F.2 Proof of Lemma 4
Now define  $\beta_{k} := \prod_{t=0}^{k}(1 - c\delta_{t})$  and let  $\beta := \lim_{k \to \infty} \beta_{k}$ . Note that  $\beta$  is well-defined because the partial product is non-increasing and bounded from below. Then we present the following result, which is stated as Lemma 4 in the main paper and from which the ergodicity of any sequence in  $\mathcal{M}_{\mathcal{G},\epsilon}$  is an immediate result.
Proof. We will prove the two cases separately.
[Case  $\beta = 0$ ] We will show that  $\beta = 0$  implies  $\lim_{k\to \infty}\| \hat{Q}_{0,k}\|_{\infty} = 0$ , and as a result,  $\lim_{k\to \infty}\| Q_{0,k}\|_{\infty} = 0$ . For  $0\leq j\leq T - 1$ , let us define
$$
\beta^ {j} := \prod_ {k = 0} ^ {\infty} (1 - \delta_ {j + k T}).
$$
Then by Lemma 10, we get that
$$
\lim  _ {k \rightarrow \infty} \| \hat {Q} _ {0, k T} \| _ {\infty} \leq \beta^ {j} \| \hat {Q} _ {0, j} \| _ {\infty}.
$$
By construction,  $\beta = \Pi_{j=0}^{T-1} \beta^j$ . Hence, if  $\beta = 0$  then  $\beta^{j_0} = 0$  for some  $0 \leq j_0 \leq T-1$ , which yields  $\lim_{k \to \infty} \| \hat{Q}_{0,k} \|_{\infty} = 0$ . Consequently,  $\lim_{k \to \infty} \| Q_{0,k} \|_{\infty} = 0$  implies that  $\lim_{k \to \infty} Q_{0,k} = 0$ .
[Case  $\beta > 0$ ] First observe that if  $\beta > 0$ , then  $\forall 0 < \eta < 1$ , there exist  $m \in \mathbb{N}_{\geq 0}$  such that
$$
\prod_ {t = m} ^ {\infty} \left(1 - c \delta_ {t}\right) > 1 - \eta . \tag {9}
$$
Using  $1 - x \leq e^{-x}$  for all  $x \in \mathbb{R}$ , we deduce
$$
\prod_ {t = m} ^ {\infty} e ^ {- c \delta_ {t}} > 1 - \eta .
$$
It also follows from (9) that  $1 - c\delta_{t} > 1 - \eta$ , or equivalently  $\delta_t < \frac{\eta}{c}$  for  $t \geq m$ . Choosing  $\eta < \frac{c}{2}$  thus ensures that  $\delta_t < \frac{1}{2}$  for  $t \geq m$ . Putting this together with the fact that, there exists  $b > 0$  such that  $1 - x \geq e^{-bx}$  for all  $x \in [0, \frac{1}{2}]$ , we obtain
$$
\prod_ {t = m} ^ {\infty} \left(1 - \delta_ {t}\right) \geq \prod_ {t = m} ^ {\infty} e ^ {- b \delta_ {t}} > \left(1 - \eta\right) ^ {\frac {b}{c}} := 1 - \eta^ {\prime}. \tag {10}
$$
Define the product of row-stochastic matrices  $P^{(M:m)} \coloneqq P^{(M)} \ldots P^{(m)}$ . It is easy to verify the following element-wise inequality:
$$
\left(\prod_ {t = m} ^ {M} \left(1 - \delta_ {t}\right)\right) P ^ {(M: m)} \leq_ {\mathrm {e w}} Q _ {m, M} \leq_ {\mathrm {e w}} P ^ {(M: m)},
$$
which together with (10) leads to
$$
\left(1 - \eta^ {\prime}\right) P ^ {(M: m)} \le_ {\mathrm {e w}} Q _ {m, M} \le_ {\mathrm {e w}} P ^ {(M: m)}. \tag {11}
$$
$$
\begin{array}{l} \| B Q _ {m, M} \| _ {\infty} = \| B (Q _ {m, M} - P ^ {(M: m)}) + B P ^ {(M: m)} \| _ {\infty} \\ \leq \| B \left(Q _ {m, M} - P ^ {(M: m)}\right) \| _ {\infty} + \| B P ^ {(M: m)} \| _ {\infty} \\ = \| B \left(Q _ {m, M} - P ^ {(M: m)}\right) \| _ {\infty} \\ \leq \| B \| _ {\infty} \| Q _ {m, M} - P ^ {(M: m)} \| _ {\infty} \\ \leq \eta^ {\prime} \| B \| _ {\infty} \\ \leq \eta^ {\prime} \sqrt {N}, \\ \end{array}
$$
where the last inequality is due to the fact that  $\| B\| _2 = 1$ . By definition,  $Q_{0,M} = Q_{m,M}Q_{0,m - 1}$ , and hence
$$
\left\| B Q _ {0, M} \right\| _ {\infty} \leq \left\| B Q _ {m, M} \right\| _ {\infty} \left\| Q _ {0, m - 1} \right\| _ {\infty} \leq \left\| B Q _ {m, M} \right\| _ {\infty} \leq \eta^ {\prime} \sqrt {N}. \tag {12}
$$
The above inequality (12) holds when taking  $M \to \infty$ . Then taking  $\eta \to 0$  implies  $\eta' \to 0$  and together with (12), we conclude that
$$
\lim_{M\to \infty}\| BQ_{0,M}\|_{\infty} = 0  ,
$$
4 Choose, e.g.,  $b = 2\log 2$ .
$$
\lim_{M\to \infty}BQ_{0,M} = 0  .
$$
F.3 Proof of Lemma 5
Notice that both cases  $\beta = 0$  and  $\beta > 0$  in Lemma 4 imply the ergodicity of  $\{D^{(t)}P^{(t)}\}_{t=0}^{\infty}$ . Hence the statement is a direct corollary of Lemma 4.
G Proof of Lemma 6
In order to show that  $\mathrm{JSR}(\tilde{\mathcal{P}}_{\mathcal{G},\epsilon}) < 1$ , we start by making the following observation.
Lemma 11. A sequence  $\{P^{(n)}\}_{n = 0}^{\infty}$  is ergodic if and only if  $\prod_{n = 0}^{t}\tilde{P}^{(n)}$  converges to the zero matrix.
Proof. For any  $t \in \mathbb{N}_{\geq 0}$ , it follows from the third property of the orthogonal projection  $B$  (see, Page 6 of the main paper) that
$$
B \prod_ {n = 0} ^ {t} P ^ {(n)} = \prod_ {n = 0} ^ {t} \tilde {M} ^ {(n)} P.
$$
$$
\begin{array}{l} \{P ^ {(n)} \} _ {n = 0} ^ {\infty} \text {i s e r g o d i c} \Longleftrightarrow \lim  _ {t \rightarrow \infty} B \prod_ {n = 0} ^ {t} P ^ {(n)} = 0 \\ \Longleftrightarrow \lim  _ {t \rightarrow \infty} \prod_ {n = 0} ^ {t} \tilde {P} ^ {(n)} B = 0 \\ \Longleftrightarrow \lim _ {t \to \infty} \prod_ {n = 0} ^ {t} \tilde {P} ^ {(n)} = 0. \\ \end{array}
$$
Next, we utilize the following result, as a means to ensure a joint spectral radius strictly less than 1 for a bounded set of matrices.
Lemma 12 (Proposition 3.2 in [35]). For any bounded set of matrices  $\mathcal{M}$ ,  $\mathrm{JSR}(\mathcal{M}) < 1$  if and only if for any sequence  $\{M^{(n)}\}_{n=0}^{\infty}$  in  $\mathcal{M}$ ,  $\prod_{n=0}^{t} M^{(n)}$  converges to the zero matrix.
Here, "bounded" means that there exists an upper bound on the norms of the matrices in the set. Note that  $\mathcal{P}_{\mathcal{G},\epsilon}$  is bounded because  $\| P\|_{\infty} = 1$ , for all  $P\in \mathcal{M}_{\mathcal{G},\epsilon}$ . To show that  $\tilde{\mathcal{P}}_{\mathcal{G},\epsilon}$  is also bounded, let  $\tilde{P}\in \tilde{\mathcal{P}}_{\mathcal{G},\epsilon}$ , then by definition, we have
$$
\tilde {P} B = B P, P \in \mathcal {M} _ {\mathcal {G}, \epsilon} \Rightarrow \tilde {P} = B P B ^ {T},
$$
since  $BB^{T} = I_{N - 1}$ . As a result,
$$
\| \tilde {P} \| _ {2} = \| B P B ^ {T} \| _ {2} \leq \| P \| _ {2} \leq \sqrt {N},
$$
where the first inequality is due to  $\| B\| _2 = \| B^\top \| _2 = 1$ , and the second inequality follows from  $\| P\|_{\infty} = 1$ .
Combining Lemma 5, Lemma 11 and Lemma 12, we conclude that  $\mathrm{JSR}(\tilde{\mathcal{P}}_{\mathcal{G},\epsilon}) < 1$
H Proof of Lemma 7
Note that any sequence  $\{M^{(n)}\}_{n = 0}^{\infty}$  in  $\mathcal{M}_{\mathcal{G},\epsilon ,\delta}$  satisfies  $\beta = 0$  , where  $\beta$  is defined in Lemma 4. This implies that for any sequence  $\{M^{(n)}\}_{n = 0}^{\infty}$  in  $\mathcal{M}_{\mathcal{G},\epsilon ,\delta}$  , we have
$$
\lim  _ {t \rightarrow \infty} \prod_ {n = 0} ^ {t} M ^ {(n)} = 0.
$$
Since  $\| M \|_{\infty} \leq \delta$  for all  $M \in \mathcal{M}_{\mathcal{G}, \epsilon, \delta}$ , again by Lemma 12, we conclude that  $\mathrm{JSR}(\mathcal{M}_{\mathcal{G}, \epsilon, \delta}) < 1$ .
I Proof of Theorem 1
Recall the formulation of  $X_{\cdot i}^{(t + 1)}$  in (7):
$$
X _ {\cdot i} ^ {(t + 1)} = \sigma (P ^ {(t)} (X ^ {(t)} W ^ {(t)}) _ {\cdot i}) = \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} W _ {j _ {k} j _ {k + 1}} ^ {(k)}\right) D _ {j _ {t + 1}} ^ {(t)} P ^ {(t)} \dots D _ {j _ {1}} ^ {(0)} P ^ {(0)} X _ {\cdot j _ {0}} ^ {(0)}.
$$
Then the first part of the theorem directly follows from Lemma 5.
To derive the exponential convergence rate, consider the linear and nonlinear cases separately:
I.1 Bounds for the two cases
[Case: linear] In the linear case where all  $D = I_N$ , it follows from Lemma 6 that
$$
\begin{array}{l} \| B X _ {\cdot i} ^ {(t + 1)} \| _ {2} = \left\| \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} W _ {j _ {k} j _ {k + 1}} ^ {(k)}\right) B P ^ {(t)}... P ^ {(0)} X _ {\cdot j _ {0}} ^ {(0)} \right\| _ {2} \\ \leq \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} \left| W _ {j _ {k} j _ {k + 1}} ^ {(k)} \right|\right) \left\| B P ^ {(t)}... P ^ {(0)} X _ {. j _ {0}} ^ {(0)} \right\| _ {2} \\ = \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} \left| W _ {j _ {k} j _ {k + 1}} ^ {(k)} \right|\right) \left\| \tilde {P} ^ {(t)}... \tilde {P} ^ {(0)} B X _ {. j _ {0}} ^ {(0)} \right\| _ {2} \\ \leq \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} \left| W _ {j _ {k} j _ {k + 1}} ^ {(k)} \right|\right) C q ^ {t + 1} \left\| B X _ {\cdot j _ {0}} ^ {(0)} \right\| _ {2} \\ \leq C ^ {\prime} q ^ {t + 1} \left(\sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} \left| W _ {j _ {k} j _ {k + 1}} ^ {(k)} \right|\right)\right) \\ = C ^ {\prime} q ^ {t + 1} \| \left(\left| W ^ {(0)} \right| \dots \left| W ^ {(t)} \right|\right). _ {i} \| _ {1}, \tag {13} \\ \end{array}
$$
where  $C' = C \max_{j \in [d]} \| BX_{\cdot j}^{(0)} \|_2$  and  $\| \cdot \|_1$  denotes the 1-norm. Specifically, the first inequality follows from the triangle inequality, and the second inequality is due to the property of the joint spectral radius in (8), where  $\mathrm{JSR}(\tilde{\mathcal{P}}_{\mathcal{G},\epsilon}) < q < 1$ .
[Case: nonlinear] Consider  $T \in \mathbb{N}$  defined in Lemma 10, where there exists  $a_1 \in \mathbb{N}_{\geq 0}$  and  $a_2 \in \{0\} \cup [K - 1]$  such that  $T = a_1 K + a_2$ . Given the condition  $(\star)$ , and Lemma 10, there exists  $0 < c < 1$  such that for all  $m \in \mathbb{N}_{\geq 0}$ ,
$$
\left\| \hat {Q} _ {m K + n _ {m}, m K + n _ {m} + T} \right\| _ {\infty} \leq \left(1 - c \delta^ {\prime}\right),
$$
where  $\delta' = 1 - \delta$ . Note that since for all  $n_m, m \in \mathbb{N}_{\geq 0}$ ,  $a_2 + n_m \leq 2K$ , we get that
$$
n _ {m} + T \leq (a _ {1} + 2) K.
$$
Since for all  $m\in \mathbb{N}_{\geq 0}$
$$
\| \hat {Q} _ {m K + n _ {m}: (m + a _ {1} + 2) K + n _ {m + a _ {1} + 2}} \| _ {\infty} \leq \left(1 - c \delta^ {\prime}\right),
$$
it implies that for all  $n\in \mathbb{N}_{\geq 0}$
$$
\begin{array}{l} \left\| Q _ {0: n (a _ {1} + 2) K} \right\| _ {\infty} \leq (1 - c \delta^ {\prime}) ^ {n} = \left((1 - c \delta^ {\prime}) ^ {\frac {n}{n (a _ {1} + 2) K}}\right) ^ {n (a _ {1} + 2) K} \\ = \left((1 - c \delta^ {\prime}) ^ {\frac {1}{(a _ {1} + 2) K}}\right) ^ {n (a _ {1} + 2) K}. \\ \end{array}
$$
Denote  $\underline{q} \coloneqq (1 - c\delta')^{\frac{1}{(a_1 + 2)K}}$ . By the equivalence of norms, we get that for all  $n \in \mathbb{N}_{\geq 0}$ ,
$$
\left\| Q _ {0: n (a _ {1} + 2) K} \right\| _ {2} \leq \sqrt {N} \underline {{q}} ^ {n (a _ {1} + 2) K}.
$$
Then for any  $j\in \{0\} \cup [(a_1 + 2)K - 1]$
$$
\| Q _ {0: n (a _ {1} + 2) K + j} \| _ {2} \leq \sqrt {N} \underline {{q}} ^ {- j} \underline {{q}} ^ {n (a _ {1} + 2) K + j} \leq C \underline {{q}} ^ {n (a _ {1} + 2) K + j}
$$
where  $C \coloneqq \sqrt{N} \underline{q}^{-(a_1 + 2)K + 1}$ . Rewriting the indices, we conclude that
$$
\left\| Q _ {0: t} \right\| _ {2} \leq C \underline {{q}} ^ {t}, \forall t \geq 0.
$$
The above bound implies that for any  $q$  satisfies  $\underline{q} \leq q < 1$ ,
$$
\begin{array}{l} \| B X _ {\cdot i} ^ {(t + 1)} \| _ {2} = \left\| \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} W _ {j _ {k} j _ {k + 1}} ^ {(k)}\right) B D _ {j _ {t + 1}} ^ {(t)} P ^ {(t)}... D _ {j _ {1}} ^ {(0)} P ^ {(0)} X _ {\cdot j _ {0}} ^ {(0)} \right\| _ {2} \\ \leq \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} \left| W _ {j _ {k} j _ {k + 1}} ^ {(k)} \right|\right) \left\| B D _ {j _ {t + 1}} ^ {(t)} P ^ {(t)} \dots D _ {j _ {1}} ^ {(0)} P ^ {(0)} X _ {\cdot j _ {0}} ^ {(0)} \right\| _ {2} \\ \leq \sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} \left| W _ {j _ {k} j _ {k + 1}} ^ {(k)} \right|\right) \left\| D _ {j _ {t + 1}} ^ {(t)} P ^ {(t)} \dots D _ {j _ {1}} ^ {(0)} P ^ {(0)} \right\| \left\| X _ {. j _ {0}} ^ {(0)} \right\| _ {2} \\ \leq C ^ {\prime} q ^ {t + 1} \left(\sum_ {j _ {t + 1} = i, (j _ {t}, \dots , j _ {0}) \in [ d ] ^ {t + 1}} \left(\prod_ {k = 0} ^ {t} \left| W _ {j _ {k} j _ {k + 1}} ^ {(k)} \right|\right)\right) \\ = C ^ {\prime} q ^ {t + 1} \| \left(\left| W ^ {(0)} \right| \dots \left| W ^ {(t)} \right|\right). _ {i} \| _ {1}, \tag {14} \\ \end{array}
$$
where again,  $C' = C_{j \in [d]} \| BX_{\cdot j}^{(0)}\|_2$ .
I.2 Proof of the exponential convergence
Based on the inequality established for both linear and nonlinear case in (13) and (14), we derive the rest of the proof. Since  $\| Bx\| _2 = \| x\| _2$  if  $x^{\top}\mathbf{1} = 0$  for  $x\in \mathbb{R}^N$ , we also have that if  $X^{\top}\mathbf{1} = 0$  for  $X\in \mathbb{R}^{N\times d}$ , then
$$
\| B X \| _ {F} = \| X \| _ {F},
$$
using which we obtain that
$$
\begin{array}{l} \mu \left(X ^ {(t + 1)}\right) = \| X ^ {(t + 1)} - \mathbf {1} \gamma_ {X ^ {(t + 1)}} \| _ {F} = \| B X ^ {(t + 1)} \| _ {F} = \sqrt {\sum_ {i = 1} ^ {d} \| B X _ {i} ^ {(t + 1)} \| _ {2} ^ {2}} \\ \leq C ^ {\prime} q ^ {t + 1} \sqrt {\sum_ {i = 1} ^ {d} \left\| \left(\left| W ^ {(0)} \right| \dots \left| W ^ {(t)} \right|\right) _ {. i} \right\| _ {1} ^ {2}} \\ \leq C ^ {\prime} q ^ {t + 1} \sqrt {\left(\sum_ {i = 1} ^ {d} \| | (W ^ {(0)} | \dots | W ^ {(t)} |) _ {\cdot i} \| _ {1}\right) ^ {2}} \\ = C ^ {\prime} q ^ {t + 1} \| \left| \left(W ^ {(0)} \right| \dots \left| W ^ {(t)} \right| \right| \| _ {1, 1}, \\ \end{array}
$$
where  $\| \cdot \|_{1,1}$  denotes the matrix  $(1,1)$ -norm (recall from Section A.2 that for a matrix  $M \in \mathbb{R}^{m \times n}$ , we have  $\| M \|_{1,1} = \sum_{i=1}^{m} \sum_{j=1}^{n} |M_{ij}|$ ). The assumption A3 implies that there exists  $C''$  such that for all  $t \in \mathbb{N}_{\geq 0}$ ,
$$
\left\| \left(\left| W ^ {(0)} \right| \dots \left| W ^ {(t)} \right|\right) \right\| _ {1, 1} \leq C ^ {\prime \prime} d ^ {2}.
$$
Thus we conclude that there exists  $C_1$  such that for all  $t \in \mathbb{N}_{\geq 0}$ ,
$$
\mu (X ^ {(t)}) \leq C _ {1} q ^ {t}.
$$
Remark 3. Similar to the linear case, one can also use Lemma 7 to establish exponential rate for oversmoothing when dealing with nonlinearities for which Assumption 4 holds in the strict sense, i.e.  $0 \leq \frac{\sigma(x)}{x} < 1$  (e.g., GELU and SiLU nonlinearities). Here, we presented an alternative proof requiring weaker conditions, making the result directly applicable to nonlinearities such as ReLU and Leaky ReLU.
J Proof of Proposition 2
Since  $D_{\mathrm{deg}}^{-1}A$  is similar to  $D_{\mathrm{deg}}^{-1/2}AD_{\mathrm{deg}}^{-1/2}$ , they have the same spectrum. For  $D_{\mathrm{deg}}^{-1}A$ , the smallest nonzero entry has value  $1/d_{\mathrm{max}}$ , where  $d_{\mathrm{max}}$  is the maximum node degree in  $\mathcal{G}$ . On the other hand, it follows from the definition of  $\mathcal{P}_{\mathcal{G},\epsilon}$  that
$$
\epsilon d _ {\max } \leq 1.
$$
Therefore,  $\epsilon \leq 1 / d_{\mathrm{max}}$  and thus  $D_{\mathrm{deg}}^{-1}A\in \mathcal{P}_{\mathcal{G},\epsilon}$
We proceed by proving the following result.
Lemma 13. For any  $M$  in  $\mathcal{M}$ , the spectral radius of  $M$  denoted by  $\rho(M)$ , satisfies
$$
\rho (M) \leq \operatorname {J S R} (\mathcal {M}).
$$
Proof. Gelfand's formula states that  $\rho(M) = \lim_{k \to \infty} \| M^k \|^{\frac{1}{k}}$ , where the quantity is independent of the norm used [22]. Then comparing with the definition of the joint spectral radius, we can immediately conclude the statement.
Let  $B(D_{\deg}^{-1}A) = \tilde{P} B$ . By definition,  $\tilde{P}\in \tilde{\mathcal{P}}_{\mathcal{G},\epsilon}$  since  $D_{\mathrm{deg}}^{-1}A\in \mathcal{P}_{\mathcal{G},\epsilon}$  as shown before the lemma. Moreover, the spectrum of  $\tilde{P}$  is the spectrum of  $D_{\mathrm{deg}}^{-1}A$  after reducing the multiplicity of eigenvalue 1 by one. Under the assumption A1, the eigenvalue 1 of  $D_{\mathrm{deg}}^{-1}A$  has multiplicity 1, and hence  $\rho (\tilde{P}) = \lambda$  where  $\lambda$  is the second largest eigenvalue of  $D_{\mathrm{deg}}^{-1}A$ . Putting this together with Lemma 13, we conclude that
$$
\lambda \leq \operatorname {J S R} (\tilde {\mathcal {P}} _ {\mathcal {G}, \epsilon})
$$