SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L _ {A, B} \left(\hat {\theta} _ {n}\right) - \hat {L} _ {A, B} \left(\hat {\theta} _ {n}\right) \lesssim B ^ {2} \sqrt {\frac {\log (4 / \delta)}{n}} + B ^ {2} \sqrt {\frac {p _ {\mathcal {H}} \log \left(B \gamma \left(U _ {\lambda} / \alpha_ {\mathcal {H}}\right) + 3\right)}{n}}. \tag {31}
$$
Step 3. By Lemma B.13,
$$
\hat {L} _ {A, B} \left(\hat {\theta} _ {n}\right) \leqslant \frac {1}{2} \left(\epsilon_ {*} + \sigma^ {2}\right) + \frac {1}{2} B ^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}} + \lambda M _ {*}. \tag {32}
$$
Combining (30), (31) and (32) gives
$$
L _ {A, B} \left(\hat {\theta} _ {n}\right) - \frac {1}{2} \left(\epsilon_ {*} + \sigma^ {2}\right) \lesssim B ^ {2} \sqrt {\frac {\log (4 / \delta)}{n}} + B ^ {2} \sqrt {\frac {p _ {\mathcal {H}} \log \left(B \gamma \left(U _ {\lambda} / \alpha_ {\mathcal {H}}\right) + 3\right)}{n}} + \lambda M _ {*}. \tag {33}
$$
Then we complete the proof by noting  $L_{A,B}(\hat{\theta}_n) = \frac{1}{2}\left\| \pi_A\circ h_{\hat{\theta}_n} - h^*\right\|_{L^2 (P)}^2 +\frac{1}{2}\sigma^2$
B.3 Fano's Method for Random Estimators
In this paper, the lower bounds for LCNs and FCNs are established using Fano's method from minimax theory (Wainwright, 2019, Section 15). However, different from the traditional statistical setup where the estimator is deterministic, the estimators produced by stochastic optimizers are random. To address this issue, we develop a variant of Fano's method for random estimators, which might be of independent interest.
We first recall the definition of Kullback-Leibler (KL) divergence.
Definition B.16 (KL Divergence). Given two distributions  $Q$  and  $P$ , the KL divergence between them is given by
$$
D _ {\mathrm {K L}} (Q \| P) := \left\{ \begin{array}{l l} \mathbb {E} _ {Q} \left[ \log \frac {\mathrm {d} Q}{\mathrm {d} P} \right] & \text {w h e n Q i s a b s o l u t e l y c o n t i n u o u s w i t h r e s p e c t t o P}, \\ + \infty & \text {o t h e r w i s e}. \end{array} \right.
$$
If the distributions have densities with respect to some underlying measure  $\nu$ :  $\mathrm{d}Q = q\mathrm{d}\nu$ ,  $\mathrm{d}P = p\mathrm{d}\nu$ , then then the KL divergence can be written in the form
$$
D _ {\mathrm {K L}} (Q \| P) = \int q (x) \log \frac {q (x)}{p (x)} \mathrm {d} v (x).
$$
In particular, we will use the fact: for  $P = \mathcal{N}(\mu_1, \sigma^2)$  and  $Q = \mathcal{N}(\mu_2, \sigma^2)$ , we have
$$
D _ {\mathrm {K L}} (P \| Q) = \frac {\left| \mu_ {1} - \mu_ {2} \right| ^ {2}}{2 \sigma^ {2}} \tag {34}
$$
and the following lemma.
Lemma B.17. Denote by  $Q_{n}$ ,  $P_{n}$  the  $n$ -fold product distribution of  $Q$  and  $P$ , respectively. When  $Q$  is absolutely continuous with respect to  $P$ ,  $D_{\mathrm{KL}}(Q_n \| P_n) = nD_{\mathrm{KL}}(Q \| P)$
Proof. This follows trivially from the fact that  $\frac{\mathrm{d}Q_n}{\mathrm{d}P_n} = \frac{\mathrm{d}(Q\times\cdots\times Q)}{\mathrm{d}(P\times\cdots\times P)} = \left(\frac{\mathrm{d}Q}{\mathrm{d}P}\right)^n$
Minimax Risk. Let  $\{P_{\theta} : \theta \in \Theta\}$  be a set of probability distributions indexed by  $\theta \in \Theta$  and supported on  $\Omega$ . Let  $Z_{\theta}$  be a sample drawn from  $P_{\theta}$ . A random estimator  $\mathbb{A} : \Omega \mapsto \mathcal{M}(\Theta)$  returns a random variable taking values in  $\Theta$ , which is a random estimate of  $\theta$  by using the sample  $Z_{\theta}$ . Note that in this definition, the estimator is a random variable and it can model the one produced by stochastic optimizers such as SGD and Adam. In addition, when it is deterministic, this definition recovers the one commonly used in the statistical literature.
Let  $d:\Theta \times \Theta \mapsto \mathbb{R}_{\geqslant 0}$  be a metric over  $\Theta$ . The minimax risk of learning distributions in  $\{P_{\theta}:\theta \in \Theta \}$  is defined by
$$
\inf  _ {\mathbb {A} \in \mathcal {A}} \sup  _ {\theta \in \Theta} \mathbb {E} _ {\theta , \mathbb {A}} \left[ d \left(\theta , \mathbb {A} \left(Z _ {\theta}\right)\right) ^ {2} \right], \tag {35}
$$
where  $\mathbb{E}_{\theta, \mathbb{A}}$  means the expectation with respect to  $Z_{\theta}$  and  $\mathbb{A}$ , and the outer infimum is taken over  $\mathcal{A}$ , i.e., the set of all the mappings from  $\Omega$  to  $\mathcal{M}(\Theta)$ .
We next present a modification of Fano's method for random estimators. The proof process resembles that of the original Fano's method, but with some necessary modifications. This extension, although not difficult, can be very useful as random estimators have become ubiquitous nowadays.
Theorem B.18 (Fano's method for random estimators). Suppose that for  $M \in \mathbb{N}$  and  $A > 0$ , there exists a packing  $\theta_1, \ldots, \theta_M \in \Theta$  such that  $d(\theta_i, \theta_j)^2 \geqslant 4A$  for any  $i \neq j$ , then
$$
\inf  _ {\mathbb {A} \in \mathcal {A}} \sup  _ {\theta \in \Theta} \mathbb {E} _ {\theta , \mathbb {A}} \left[ d \left(\theta , \mathbb {A} \left(Z _ {\theta}\right)\right) ^ {2} \right] \geqslant A \cdot \left(1 - \frac {1}{M ^ {2} \log M} \sum_ {i, j} ^ {M} D _ {\mathrm {K L}} \left(P _ {\theta_ {i}} \| P _ {\theta_ {j}}\right) - \frac {\log 2}{\log M}\right) \tag {36}
$$
Proof. By Markov's inequality,  $\mathbb{E}_{\theta}\left[\mathbb{E}_{\mathbb{A}}\left[d\left(\theta ,\mathbb{A}(Z_{\theta})\right)^{2}\right]\right] \geqslant A\cdot \mathbb{P}_{\theta}\left(\mathbb{E}_{\mathbb{A}}\left[d\left(\theta ,\mathbb{A}(Z_{\theta})\right)^{2}\right] > A\right)$ , it is sufficient to lower bound
$$
\inf  _ {\mathbb {A} \in \mathcal {A}} \sup  _ {\theta \in \Theta} \mathbb {P} _ {\theta} \left(\mathbb {E} _ {\mathbb {A}} \left[ d (\theta , \mathbb {A} (Z _ {\theta})) ^ {2} \right] > A\right) \tag {37}
$$
for some  $A > 0$ . This will be useful for techniques based on information theory.
The principle is simple: pack the index set  $\Theta$  with balls of some radius  $4A$ , that is find  $\theta_{1},\ldots ,\theta_{M}\in \Theta$  such that
$$
\forall i \neq j, d \left(\theta_ {i}, \theta_ {j}\right) ^ {2} \geqslant 4 A,
$$
and transform the estimation problem into a hypothesis test, that is, an algorithm going from the data  $Z_{\theta}$  to one out of  $M$  potential outcomes.
To this end, we take the supremum over a smaller set:
$$
\sup  _ {\theta \in \Theta} \mathbb {P} _ {\theta} \left(\mathbb {E} _ {\mathbb {A}} \left[ d \left(\theta , \mathbb {A} \left(Z _ {\theta}\right)\right) ^ {2} \right] > A\right) \geqslant \max  _ {j \in [ M ]} \mathbb {P} _ {\theta_ {j}} \left(\mathbb {E} _ {\mathbb {A}} \left[ d \left(\theta_ {j}, \mathbb {A} \left(Z _ {\theta_ {j}}\right)\right) ^ {2} \right] > A\right) \tag {38}
$$
Any algorithm  $\mathbb{A}$  gives a test
$$
g \left(\mathbb {A} \left(Z _ {\theta}\right)\right) = \arg \min  _ {j \in [ M ]} \mathbb {E} _ {\mathbb {A}} \left[ d \left(\theta_ {j}, \mathbb {A} \left(Z _ {\theta}\right)\right) ^ {2} \right] \in [ M ], \tag {39}
$$
where ties are broken arbitrarily (e.g., by selecting the minimal index).
If, for some  $j \in [M]$ ,  $g(\mathbb{A}(Z_{\theta_j})) \neq j$ , there exists  $k \neq j$ , such that  $\mathbb{E}_{\mathbb{A}}\left[d\left(\theta_k,\mathbb{A}(Z_{\theta_j})\right)^2\right] \leqslant \mathbb{E}_{\mathbb{A}}\left[d\left(\theta_j,\mathbb{A}(Z_{\theta_j})\right)^2\right]$ . Moreover, using the triangle inequality for  $d(\cdot ,\cdot)$ , we get:
$$
d \left(\theta_ {j}, \theta_ {k}\right) ^ {2} \leqslant 2 \mathbb {E} _ {\mathbb {A}} \left[ d \left(\theta_ {j}, \mathbb {A} \left(Z _ {\theta_ {j}}\right)\right) ^ {2} + d \left(\mathbb {A} \left(Z _ {\theta_ {j}}\right), \theta_ {k}\right) ^ {2} \right], \tag {40}
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathbb {A}} \left[ d \left(\theta_ {j}, \mathbb {A} \left(Z _ {\theta_ {j}}\right)\right) ^ {2} \right] \geqslant \frac {1}{2} d \left(\theta_ {j}, \theta_ {k}\right) ^ {2} - \mathbb {E} _ {\mathbb {A}} \left[ d \left(\mathbb {A} \left(Z _ {\theta_ {j}}\right), \theta_ {k}\right) ^ {2} \right] \tag {41} \\ \geqslant \frac {1}{2} d \left(\theta_ {j}, \theta_ {k}\right) ^ {2} - \mathbb {E} _ {\mathbb {A}} \left[ d \left(\mathbb {A} \left(Z _ {\theta_ {j}}\right), \theta_ {j}\right) ^ {2} \right] \quad (\text {u s i n g t h e o p t i m a l} k), \\ \end{array}
$$
which implies  $\mathbb{E}_{\mathbb{A}}\left[d\left(\theta_j,\mathbb{A}(Z_{\theta_j})\right)^2\right]\geqslant \frac{1}{4} d\left(\theta_j,\theta_k\right)^2\geqslant A.$  Thus, we have
$$
\mathbb {P} _ {\theta_ {j}} \left(\mathbb {E} _ {\mathbb {A}} \left[ d \left(\theta_ {j}, \mathbb {A} \left(Z _ {\theta_ {j}}\right)\right) ^ {2} \right] > A\right) \geqslant \mathbb {P} _ {\theta_ {j}} \left(g \left(\mathbb {A} \left(Z _ {\theta_ {j}}\right)\right) \neq j\right), \tag {42}
$$
$$
\begin{array}{l} \inf  _ {\mathbb {A} \in \mathcal {A}} \sup  _ {\theta \in \Theta} \mathbb {E} _ {\theta , \mathbb {A}} \left[ d \left(\theta , \mathbb {A} \left(Z _ {\theta}\right)\right) ^ {2} \right] \geqslant A \cdot \inf  _ {g} \max  _ {j \in \{1, \dots , M \}} \mathbb {P} _ {\theta_ {j}} \left(g \left(Z _ {\theta_ {j}}\right) \neq j\right) \\ \geqslant A \cdot \inf  _ {g} \frac {1}{M} \sum_ {j = 1} ^ {M} \mathbb {P} _ {\theta_ {j}} (g \left(Z _ {\theta_ {j}}\right) \neq j), \tag {43} \\ \end{array}
$$
which can be further lower bounded using the following lemma (Bach, 2023, Corollary 12.1).
Lemma B.19 (Fano's inequality for multiple hypothesis testing). Given  $M$  distributions  $P_{1}, P_{2}, \ldots, P_{M}$  over  $\Omega$ , let  $Z_{\theta_{j}} \sim P_{\theta_{j}}$  for  $j = 1, 2, \ldots, M$ . Then
$$
\inf  _ {g} \frac {1}{M} \sum_ {j = 1} ^ {M} \mathbb {P} _ {\theta_ {j}} (g \left(Z _ {\theta_ {j}}\right) \neq j) \geqslant 1 - \frac {1}{M ^ {2} \log M} \sum_ {j, j ^ {\prime} = 1} ^ {M} D _ {\mathrm {K L}} \left(P _ {\theta_ {j}} \| P _ {\theta_ {j ^ {\prime}}}\right) - \frac {\log 2}{\log M}, \tag {44}
$$
where the infimum is taken over all testing methods, i.e., all maps from  $\Omega$  to  $[M]$ .
Combining Lemma B.19 and equation (43), we complete the proof.
In this paper, we will mostly use a variant of the above Fano's inequality for regression problems. Recall that in regression, the data are generated by  $y = h(x) + \xi$  with  $x \sim P$  and  $\xi \sim \mathcal{N}(0, \sigma^2)$ . Let  $Q_h$  be the joint distribution of  $(\mathbf{x}, y)$  and  $S_n = \{(\mathbf{x}_i, y_i)\}_{i=1}^n$  be  $n$  i.i.d. samples drawn from  $Q_h$ . Denote by  $Q_{n,h}$  the  $n$ -fold product distribution of  $Q_h$ . Let  $\mathcal{F}$  be the class of target functions. In such a case, the set of probability distributions is indexed by  $h \in \mathcal{F}$ :
$$
\left\{Q _ {n, h}: h \in \mathcal {F} \right\}.
$$
Proposition B.20 (Fano's method for regression). Let  $\mathcal{F}$  be a target function class. If there exists a packing  $h_1, \ldots, h_M \in \mathcal{F}$  such that for any  $i \neq j$ ,  $\|h_i - h_j\|_{L^2(P)}^2 \geqslant 4A$ , we have
$$
\inf  _ {\mathbb {A} \in \mathcal {A}} \sup  _ {h ^ {*} \in \mathcal {F}} \mathbb {E} [ \| h _ {\mathbb {A} (S _ {n})} - h ^ {*} \| _ {L ^ {2} (P)} ^ {2} ] \geqslant A \cdot \left(1 - \frac {n}{2 \sigma^ {2} M ^ {2} \log M} \sum_ {j, j ^ {\prime} = 1} ^ {M} \| h _ {j} - h _ {j ^ {\prime}} \| _ {L ^ {2} (P)} ^ {2} - \frac {\log 2}{\log M}\right), \tag {45}
$$
where the expectation is taken over both the sampling of  $S_{n}$  and randomness of  $\mathbb{A}$ .
Proof. Define  $d(h, h')^2 = \| h - h' \|_{L^2(P)}^2$ . WLOG, assume that  $P$  has a density function  $p$  and denote by  $\phi_{h,x}$  the density function of  $\mathcal{N}(h(x), \sigma^2)$ . Then, the density function of  $Q_h$  is  $p(x) \phi_{h,x}(y)$ . Then, we have
$$
\begin{array}{l} D _ {\mathrm {K L}} \left(Q _ {n, h} \| Q _ {n, h ^ {\prime}}\right) = n D _ {\mathrm {K L}} \left(Q _ {h} \| Q _ {h ^ {\prime}}\right) \\ = n \int p (x) \phi_ {x, h} (y) \log \left(\frac {p (x) \phi_ {x , h} (y)}{p (x) \phi_ {x , h ^ {\prime}} (y)}\right) d x d y \\ = n \int p (x) \left(\int \phi_ {x, h} (y) \log \left(\frac {\phi_ {x , h} (y)}{\phi_ {x , h ^ {\prime}} (y)}\right) d y\right) d x \\ = \frac {n}{2 \sigma^ {2}} \int | h (x) - h ^ {\prime} (x) | ^ {2} p (x) d x = \frac {n}{2 \sigma^ {2}} \| h - h ^ {\prime} \| _ {L ^ {2} (P)} ^ {2}, \\ \end{array}
$$
where the first equality follows from Lemma B.17 and the third equality is due to equation (34). We remark that this equality is also right when  $P$  does not admit a density. In this case, we can construct and verify the Radon-Nikodym derivative, and then compute the KL divergence by the definition. Lastly, by plugging the above estimate into (36) in Theorem B.18, we complete the proof.
Remark B.21. When we use Fano's method (Proposition B.20) to lower-bound the minimax risk, the most important step is to construct a "proper" packing. Take a look at the lower bound in (45). For fixed  $M \in \mathbb{N}$  and  $A > 0$ , a proper packing should ensure  $\sum_{i,j=1}^{M} \| h_i - h_j \|_{L^2(P)}^2$  to be as small as possible; otherwise the packing may yield a suboptimal lower bound. In fact, in most applications, we will simply attempt to bound the worst-case resolution:  $\sup_{i,j} \| h_i - h_j \|_{L^2(P)}^2$ .
C Capacity Control for CNNs and LCNs
We first define a norm on the parameter space for controlling the model capacity effectively. Let  $\theta = \{\theta_1, \theta_2, \dots, \theta_L, W_o\}$  denote all the parameters of our network, where  $\theta_l \in \Theta_l$  and  $W_o \in \Theta_o$  denote the parameters used to parameterize  $\mathcal{T}_l$  and  $\mathcal{M}_o$ , respectively.
Denote by  $V_{l}$  the space of  $l$ -th layer hidden state. Let  $\widetilde{V}_l \in \mathbb{R}^{p_l}$  be the vectorized version of  $V_{l}$  armed with standard  $\ell_2$  norm and  $\widetilde{\mathcal{T}}_l: \widetilde{V}_{l-1} \mapsto \widetilde{V}_l$  be the corresponding representation of  $\mathcal{T}_l$ . Since  $\widetilde{\mathcal{T}}_l$  is linear, there must exist unique  $A: \Theta_l \mapsto \mathbb{R}^{p_l \times p_{l-1}}$  and  $B: \Theta_l \mapsto \mathbb{R}^{p_l}$  such that  $\widetilde{\mathcal{T}}_l \operatorname{vec}(\mathbf{z}^{(l)}) = A(\theta_l) \operatorname{vec}(\mathbf{z}^{(l)}) + B(\theta_l)$ . Then we define
$$
\left\| \theta_ {l} \right\| = \left\| \widetilde {\mathcal {T}} _ {l} \right\| := \left\| A \left(\theta_ {l}\right) \right\| _ {2} + \left\| B \left(\theta_ {l}\right) \right\| _ {2}, \tag {46}
$$
to control the "magnitude" of  $\widetilde{\mathcal{T}}_l$ , which ensures that for any  $\widetilde{\mathbf{z}} \in \widetilde{V}_{l-1}$  we have  $\| \widetilde{\mathcal{T}}_l \widetilde{\mathbf{z}} \|_2 \leqslant \| \widetilde{\mathcal{T}}_l \| (\| \widetilde{\mathbf{z}} \|_2 + 1)$ . Furthermore, we consider a new norm over the whole parameter space  $\Theta$  by
$$
\left\| \theta \right\| := \left\| W _ {o} \right\| _ {2} + \sum_ {l = 1} ^ {L} \left\| \theta_ {l} \right\|. \tag {47}
$$
It is equivalent to say  $\| \theta \| = \| \mathcal{M}_o\| +\sum_{l = 1}^L\| \widetilde{\mathcal{T}}_l\|$ , which is the sum of operator norms of all linear transforms.
We remark that we will show the norm (47) provides tight control of covering number of CNNs and LCNs, but is less informative and interpretable. Thus, we introduce the  $\| \theta \|_{\mathcal{P}}$  norm in (3) for CNNs and LCNs, which is an upper bound of  $\| \theta \|$  as demonstrated below. As opposed to  $\| \theta \|$ ,  $\| \theta \|_{\mathcal{P}}$  is much more explicit, interpretable, and easier to compute. In the following, for both CNNs and LCNs, we will first derive the matrix form of  $\widetilde{\mathcal{T}}_l$ , thereby obtaining  $A(\cdot)$  and  $B(\cdot)$ . Abusing notation, we will use  $\mathcal{T}_l$  to denote  $\widetilde{\mathcal{T}}_l$  for simplicity.
C.1 CNNs
Recall that for  $l \in [L]$ ,  $\mathcal{T}_l: \mathbb{R}^{D_{l-1} \times C_{l-1}} \mapsto \mathbb{R}^{D_l \times C_l}$  is parameterized by a kernel  $W^{(l)} \in \mathbb{R}^{C_l \times C_{l-1} \times s}$  and bias  $\mathbf{b}^{(l)} \in \mathbb{R}^{C_l}$  as follows
$$
\left(\mathcal {T} _ {l} (\mathbf {z})\right) _ {:, j} = \sum_ {i = 1} ^ {C _ {l - 1}} \mathbf {z} _ {:, i} * _ {s} W _ {j, i,:} ^ {(l)} + b _ {j} ^ {(l)} \mathbf {1}, \quad \text {f o r} j = 1, \dots , C _ {l}. \tag {48}
$$
We further point out that the number of parameters in our CNN is
$$
N _ {\mathrm {c n n}} = C _ {L} D _ {L} + \sum_ {l = 0} ^ {L - 1} \left(2 C _ {l} + 1\right) C _ {l + 1} \tag {49}
$$
A patch-based reformulation of CNNs. For technical simplicity, it is easier to take a patch-based viewpoint of  $\mathcal{T}_l$ :  $\mathcal{T}_l$  operates on each patch with an identical local linear transform. Note that  $D_{l} = D_{l - 1} / s$  and we can divide  $\mathbf{z}^{(l - 1)}$  into  $D_{l}$  patches of the shape  $s\times C_{l - 1}$ :
$$
\mathbf {z} ^ {(l - 1)} = \left( \begin{array}{c} P _ {1} ^ {(l - 1)} \\ P _ {2} ^ {(l - 1)} \\ \vdots \\ P _ {D _ {l}} ^ {(l - 1)} \end{array} \right) \in \mathbb {R} ^ {D _ {l - 1} \times C _ {l - 1}}, \quad P _ {j} ^ {(l - 1)} = \mathbf {z} _ {I _ {j,:}} ^ {(l - 1)} \in \mathbb {R} ^ {s \times C _ {l - 1}} \text {f o r} j \in [ D _ {l} ],
$$
where  $I_{j} \coloneqq [(j - 1)s + 1, js]$  denotes the  $j$ -th patch. Then  $\mathcal{T}_l$  essentially makes the following transform
$$
\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)} = \left( \begin{array}{c} \mathcal {A} ^ {(l)} P _ {1} ^ {(l - 1)} \\ \mathcal {A} ^ {(l)} P _ {2} ^ {(l - 1)} \\ \vdots \\ \mathcal {A} ^ {(l)} P _ {D _ {l}} ^ {(l - 1)} \end{array} \right),
$$
where  $\mathcal{A}^{(l)}:\mathbb{R}^{s\times C_{l - 1}}\to \mathbb{R}^{1\times C_l}$  is the local patch transform given by
$$
\left(\mathcal {A} ^ {(l)} P\right) _ {i} = \sum_ {j = 1} ^ {C _ {l - 1}} \sum_ {k = 1} ^ {s} W _ {i, j, k} P _ {j, k} + b _ {i} ^ {(l)} \quad \text {f o r} i = 1, 2, \dots , C _ {l}.
$$
In a matrix form,  $\mathcal{A}^{(l)}$  is given by
$$
\mathcal {A} ^ {(l)} P = \widetilde {W} ^ {(l)} \operatorname {v e c} (P) + \mathbf {b} ^ {(l)},
$$
$$
\widetilde {W} ^ {(l)} = \left( \begin{array}{c} \operatorname {v e c} \left(W _ {1,:,:}\right) ^ {\top} \\ \operatorname {v e c} \left(W _ {2,:,:}\right) ^ {\top} \\ \vdots \\ \operatorname {v e c} \left(W _ {C _ {l},:,:}\right) ^ {\top} \end{array} \right) \in \mathbb {R} ^ {C _ {l} \times \left(s C _ {l - 1}\right)}. \tag {50}
$$
The matrix form of  $\mathcal{T}_l$ . Let
$$
\widetilde {\mathbf {z}} ^ {(l)} = \left(\operatorname {v e c} \left(P _ {1} ^ {(l)}\right) ^ {\top}, \operatorname {v e c} \left(P _ {2} ^ {(l)}\right) ^ {\top}, \dots , \operatorname {v e c} \left(P _ {D _ {l}} ^ {(l)}\right) ^ {\top}\right) ^ {\top} \in \mathbb {R} ^ {D _ {l} C _ {l}}.
$$
Abusing the notation, we still use  $\mathcal{T}_l: \mathbb{R}^{D_{l-1}C_{l-1}} \to \mathbb{R}^{D_lC_l}$  to denote the matrix form of the one defined in (48). Then, we have
$$
\mathcal {T} _ {l} \widetilde {\mathbf {z}} ^ {(l - 1)} = K ^ {(l)} \widetilde {\mathbf {z}} ^ {(l - 1)} + \mathbf {s} ^ {(l)} \tag {51}
$$
$$
K ^ {(l)} = \left( \begin{array}{c c c c} \widetilde {W} ^ {(l)} & 0 & \ldots & 0 \\ 0 & \widetilde {W} ^ {(l)} & \ldots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \ldots & \widetilde {W} ^ {(l)} \end{array} \right) \in \mathbb {R} ^ {D _ {l} C _ {l} \times D _ {l - 1} C _ {l - 1}}, \quad \mathbf {s} ^ {(l)} = \left( \begin{array}{c} \mathbf {b} ^ {(l)} \\ \mathbf {b} ^ {(l)} \\ \vdots \\ \mathbf {b} ^ {(l)} \end{array} \right) \in \mathbb {R} ^ {D _ {l} C _ {l}},
$$
where the block matrix has  $D_{l}$  blocks and  $\widetilde{W}^{(l)}$  is given by (50).
The parameter norm. By (51), it is easy to see that
$$
\left\| \mathcal {T} _ {l} \right\| \leqslant \left\| K ^ {(l)} \right\| _ {2} + \left\| \mathbf {s} ^ {(l)} \right\| _ {2} = \left\| \widetilde {W} ^ {(l)} \right\| _ {2} + \sqrt {\frac {d}{2 ^ {l - 2}}} \left\| \mathbf {b} ^ {(l)} \right\| _ {2} \leqslant \left\| W ^ {(l)} \right\| _ {F} + \sqrt {D _ {l}} \left\| \mathbf {b} ^ {(l)} \right\| _ {2} \tag {52}
$$
$$
\left\| \theta \right\| = \left\| W _ {o} \right\| _ {2} + \sum_ {l = 1} ^ {L} \left\| \mathcal {T} _ {l} \right\| \leqslant \left\| W _ {o} \right\| _ {2} + \sum_ {l = 1} ^ {L} \left(\left\| W ^ {(l)} \right\| _ {F} + \sqrt {D _ {l}} \left\| \mathbf {b} ^ {(l)} \right\| _ {2}\right) = \| \theta \| _ {\mathcal {P}}. \tag {53}
$$
Lemma C.1. For any  $\theta, \theta'$  satisfy that  $\| \theta \| \leqslant J$  and  $\| \theta' \| \leqslant J$  and any  $\mathbf{x} \in \mathcal{X}$ , assume  $\sigma_l$  satisfies  $\sigma_l(0) = 0$  and locally Lipschitz condition  $\| \sigma_l(\mathbf{y}) - \sigma_l(\mathbf{z}) \|_2 \leqslant Q_l(\mathbf{x}) \| \mathbf{y} - \mathbf{z} \|_2$  for any  $\mathbf{y}, \mathbf{z} \in \{\mathcal{T}_l \circ \dots \circ \sigma_1 \circ \mathcal{T}_1(\mathbf{x}) : \| \theta \| \leqslant J\}$ . We further denote  $\bar{Q}_{\sigma}(\mathbf{x}) = \prod_{l=1}^{L} (Q_l(\mathbf{x}) + 1)$ . Then, we have
$$
\left| h _ {\theta} (\mathbf {x}) - h _ {\theta^ {\prime}} (\mathbf {x}) \right| \leqslant \bar {Q} _ {\sigma} (\mathbf {x}) \left(\| \mathbf {x} \| _ {2} + 1\right) (1 + J) ^ {L} \| \theta - \theta^ {\prime} \| \tag {54}
$$
Proof. First assume that  $\theta, \theta'$  only differ in the  $l$ -th layer,  $l = 1, \ldots, L$ .
$$
\begin{array}{l} \left| h _ {\theta} (\mathbf {x}) - h _ {\theta^ {\prime}} (\mathbf {x}) \right| = \left| \mathcal {M} _ {o} \circ \dots \circ \sigma_ {l} \circ \mathcal {T} _ {l} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x}) - \mathcal {M} _ {o} \circ \dots \circ \sigma_ {l} \circ \widetilde {\mathcal {T}} _ {l} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x}) \right| \\ \leqslant \operatorname {L i p} \left(\mathcal {M} _ {o} \circ \dots \circ \sigma_ {l}\right) \| \mathcal {T} _ {l} - \widetilde {\mathcal {T}} _ {l} \| \left(\| \sigma_ {l - 1} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x}) \| _ {2} + 1\right) \\ \leqslant \operatorname {L i p} \left(\mathcal {M} _ {o} \circ \dots \circ \sigma_ {l}\right) \left\| \mathcal {T} _ {l} - \widetilde {\mathcal {T}} _ {l} \right\| \left(\| \mathbf {x} \| _ {2} + 1\right) (1 + J) ^ {l - 1} \left(\prod_ {j = 1} ^ {l - 1} \left(Q _ {j} (\mathbf {x}) + 1\right)\right) \\ \leqslant \left(\prod_ {j = l} ^ {L} \left(Q _ {j} (\mathbf {x}) + 1\right)\right) (1 + J) ^ {L - l + 1} \left\| \mathcal {T} _ {l} - \widetilde {\mathcal {T}} _ {l} \right\| \left(\| \mathbf {x} \| _ {2} + 1\right) (1 + J) ^ {l - 1} \left(\prod_ {j = 1} ^ {l - 1} \left(Q _ {j} (\mathbf {x}) + 1\right)\right) \\ = \bar {Q} _ {\sigma} (\mathbf {x}) \left(\| \mathbf {x} \| _ {2} + 1\right) (1 + J) ^ {L} \left\| \mathcal {T} _ {l} - \widetilde {\mathcal {T}} _ {l} \right\| \tag {55} \\ \end{array}
$$
Second, assume  $\theta, \theta'$  only differ in the final layer. We have
$$
\begin{array}{l} \left| h _ {\theta} (\mathbf {x}) - h _ {\theta^ {\prime}} (\mathbf {x}) \right| = \left| \mathcal {M} _ {o} \circ \dots \circ \sigma_ {l} \circ \mathcal {T} _ {l} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x}) - \widetilde {\mathcal {M}} _ {o} \circ \dots \circ \sigma_ {l} \circ \mathcal {T} _ {l} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x}) \right| \\ \leqslant \left\| W _ {o} - \widetilde {W _ {o}} \right\| _ {2} \| (\sigma_ {L} \circ \dots \circ \sigma_ {l} \circ \mathcal {T} _ {l} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x})) \| _ {2} \\ \leqslant \left\| W _ {o} - \widetilde {W _ {o}} \right\| _ {2} \bar {Q} _ {\sigma} (\mathbf {x}) \left(\| \mathbf {x} \| _ {2} + 1\right) (1 + J) ^ {L} \tag {56} \\ \end{array}
$$
Lastly, we consider general  $\theta, \theta'$  with  $\| \theta \| \leqslant J$  and  $\| \theta' \| \leqslant J$ . We claim that one can find  $\theta_1, \ldots, \theta_{L+1}$  such that
-  $\| \theta_l\| \leqslant J$  for all  $l\in [L + 1]$
-  $\theta' = \theta_{L+1}$ ,  $\theta$  and  $\theta_1$  only differ in one layer;
for any  $l$ ,  $\theta_l$  and  $\theta_{l + 1}$  only differ in one layer.
It is easy to check that this can be done by first replacing the maximum norm layer in  $\theta$  by the minimum norm layer in  $\theta'$ , and so on.
Then, by applying the telescoping sum, we complete the proof.