SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \underline {{\mathbf {T}}} \right\| _ {\mathrm {s p}} := \left\| \widetilde {\mathbf {T}} _ {M} \right\|.
$$
Lemma 16. For any  $t$ -matrix  $\mathbf{W} \in \mathbb{R}^{m \times n \times c}$  and  $t$ -vector  $\underline{\mathbf{x}} \in \mathbb{R}^{n \times 1 \times c}$ , the  $t$ -product  $\mathbf{W} *_{M} \underline{\mathbf{x}}$  defined under  $M$  transform in Eq. (1) is equivalent to a linear operator  $\mathrm{op}(\underline{\mathbf{W}})$  on  $\mathrm{unfold}(\underline{\mathbf{x}})$  in the original domain defined as follows
$$
\mathrm {o p} (\underline {{\mathbf {W}}}) (\underline {{\mathbf {x}}}) = \left(\mathbf {M} ^ {- 1} \otimes \mathbf {I} _ {m}\right) \left[ \operatorname {b d i a g} \left((\mathbf {M} \otimes \mathbf {I} _ {m}) \text {u n f o l d} (\underline {{\mathbf {W}}})\right) \left(\mathbf {M} \otimes \mathbf {I} _ {n}\right) \right] \text {u n f o l d} (\underline {{\mathbf {x}}}), \tag {17}
$$
where  $\otimes$  denotes the Kronecker product, and the operations of  $\mathrm{unfold}(\underline{\mathbf{W}})$  and  $\mathrm{unfold}(\underline{\mathbf{x}})$  are given explicitly as follows
$$
\operatorname {u n f o l d} (\underline {{\mathbf {W}}}) = \left[ \begin{array}{c} \underline {{\mathbf {W}}} _ {:,: 1} \\ \underline {{\mathbf {W}}} _ {:,: 2} \\ \vdots \\ \underline {{\mathbf {W}}} _ {:,: c} \end{array} \right] \in \mathbb {R} ^ {m c \times n}, \quad \operatorname {u n f o l d} (\underline {{\mathbf {x}}}) = \left[ \begin{array}{c} \underline {{\mathbf {x}}} _ {:, 1, 1} \\ \underline {{\mathbf {x}}} _ {:, 1, 2} \\ \vdots \\ \underline {{\mathbf {x}}} _ {:, 1, c} \end{array} \right] \in \mathbb {R} ^ {n c}.
$$
Since Eq. (17) is a straightforward reformulation of the definition of t-product in [36, Definition 6.3], the proof is simply omitted.
According to Lemma 16, we have the following remark on the relationship between t-NNs and fully connected neural networks (FNNs).
Remark (Connnection with FNNs). The t-NNs and FNNs can be treated as special cases of each other.
(I) When the channel number  $c = 1$ , the t-product becomes to standard matrix multi-lication and the proposed t-NN predictor Eq. (5) degenerates to an  $(L + 1)$ -layer FNN, which means the FNN is a special case of the t-NN.
(II) On the other hand, by the definition of  $t$ -product, the  $t$ -NN  $f(\cdot; \underline{\mathbf{W}})$  in Eq. (5) has the compounding representation as an FNN:
$$
f _ {\underline {{\mathbf {W}}}} = \mathbf {w} \circ \sigma \circ \mathrm {o p} (\underline {{\mathbf {W}}} ^ {(L)}) \circ \sigma \circ \mathrm {o p} (\underline {{\mathbf {W}}} ^ {(L - 1)}) \circ \dots \circ \sigma \circ \mathrm {o p} (\underline {{\mathbf {W}}} ^ {(1)}).
$$
Thus,  $t$ -NN can also be seen as a special case of FNN.
C Standard and Adversarial Generalization Bounds for t-NNs
C.1 Standardarad Generalization Bound for t-NNs
Lemma 17. Consider the ReLU activation. For any  $t$ -vector-valued function set  $\mathcal{H}$  and any convex and monotonically increasing function  $g: \mathbb{R} \to [0, \infty)$ ,
$$
\mathbb {E} _ {\varepsilon} \left[ \sup  _ {\mathbf {h} \in \mathcal {H}, \underline {{\mathbf {W}}} \colon \| \underline {{\mathbf {W}}} \| _ {\mathrm {F}} \leq R} g \left(\left\| \sum_ {i = 1} ^ {N} \varepsilon_ {i} \sigma (\underline {{\mathbf {W}}} * _ {M} \mathbf {h} (\underline {{\mathbf {x}}} _ {i})) \right\| _ {\mathrm {F}}\right) \right] \leq 2 \mathbb {E} _ {\varepsilon} \left[ \sup  _ {\mathbf {h} \in \mathcal {H}} g \left(R \left\| \sum_ {i = 1} ^ {N} \varepsilon_ {i} \mathbf {h} (\underline {{\mathbf {x}}} _ {i}) \right\| _ {\mathrm {F}}\right) \right],
$$
where  $R > 0$  is a constant.
Proof. This lemma is a direct corollary of Lemma 1 in Ref. [8] by using Eq. (17).
Proof of Lemma 3. According to Lemma 29, we can upper bound the generalization error of  $\ell \circ f$  for any  $f\in \mathfrak{F}$  through the (empirical) Rademacher complexity  $\hat{R}_S(\ell \circ \mathfrak{F})$  where  $\ell \circ \mathfrak{F} := \{\ell \circ f\mid f\in \mathfrak{F}\}$ . Further regarding the  $L_{\ell}$ -Lipschitzness<sup>8</sup> of the loss function  $\ell$ , we have  $\hat{R}_S(\ell \circ \mathfrak{F})\leq L_\ell \hat{R}_S(\mathfrak{F})$  by the Talagrand's contraction lemma (Lemma 30). Then, it remains to bound  $\hat{R}_S(\mathfrak{F})$ .
To upper bound  $\hat{R}_S(\mathfrak{F})$ , we follow the proof of [8, Theorem 1]. By Jensen's inequality, the (scaled) Rademacher complexity  $N\hat{R}_S(\mathfrak{F}) = \mathbb{E}_{\varepsilon}\sup_{f\in \mathfrak{F}}\sum_{i = 1}^{N}\varepsilon_if(\underline{\mathbf{x}}_i)$  satisfies
$$
\frac {1}{\lambda} \log \exp \left(\lambda \cdot \mathbb {E} _ {\varepsilon} \sup  _ {f \in \mathfrak {F}} \sum_ {i = 1} ^ {N} \varepsilon_ {i} f (\underline {{\mathbf {x}}} _ {i})\right) \leq \frac {1}{\lambda} \log \left(\mathbb {E} _ {\varepsilon} \sup  _ {f \in \mathfrak {F}} \exp \lambda \sum_ {i = 1} ^ {N} \varepsilon_ {i} f (\underline {{\mathbf {x}}} _ {i})\right), \tag {18}
$$
where  $\lambda > 0$  is an arbitrary parameter. Then, we can use a "peeling" argument [8, 37] as follows.
The Rademacher complexity can be upper bounded as
$$
\begin{array}{l} N \hat {R} _ {S} (\mathfrak {F}) = \mathbb {E} _ {\boldsymbol {\varepsilon}} \sup  _ {\mathbf {f} ^ {(L)}, \| \mathbf {w} \| _ {2} \leq B _ {w}} \sum_ {i = 1} ^ {N} \varepsilon_ {i} \mathbf {w} ^ {\top} \mathbf {f} ^ {(L)} (\underline {{\mathbf {x}}} _ {i}) \\ \leq \frac{1}{\lambda}\log \mathbb{E}_{\boldsymbol{\varepsilon}}\sup_{\substack{\| \mathbf{w}\|_{2}\leq B_{w}\\ \mathbf{f}^{(L - 1)},\left\| \underline{\mathbf{W}}^{(L)}\right\|_{F}\leq B_{L}}}\exp \left(\lambda \sum_{i = 1}^{N}\varepsilon_{i}\mathbf{w}^{\top}\sigma \Bigg(\underline{\mathbf{W}}^{(L)}*_{M}\mathbf{f}^{(L - 1)}(\underline{\mathbf{x}}_{i})\Bigg)\right) \\ \leq \frac{1}{\lambda}\log \mathbb{E}_{\boldsymbol{\varepsilon}}\sup_{\substack{\mathbf{f}^{(L - 1)}\\ \left\| \underline{\mathbf{W}}^{(L)}\right\|_{\mathrm{F}}\leq B_{L}}}\exp \left(B_{w}\lambda \left\| \sum_{i = 1}^{N}\varepsilon_{i}\sigma \Bigg(\underline{\mathbf{W}}^{(L)}*_{M}\mathbf{f}^{(L - 1)}(\underline{\mathbf{x}}_{i})\Bigg)\right\|_{\mathrm{F}}\right) \\ \leq \frac{1}{\lambda}\log \left(2\cdot \mathbb{E}_{\boldsymbol {\varepsilon}}\sup_{\substack{\mathbf{f}^{(L - 2)}\\ \| \underline{\mathbf{W}}^{(L - 1)}\|_{\mathrm{F}}\leq B_{L - 1}}}\exp \left(B_{w}B_{L}\lambda \left\| \sum_{i = 1}^{N}\varepsilon_{i}\Big(\underline{\mathbf{W}}^{(L - 1)}*_{M}\mathbf{f}^{(L - 2)}(\underline{\mathbf{x}}_{i})\Big)\right\|_{\mathrm{F}}\right)\right) \\ \leq \dots \\ \leq \frac {1}{\lambda} \log \left(2 ^ {L - 1} \cdot \mathbb {E} _ {\boldsymbol {\varepsilon}} \sup  _ {\| \underline {{\mathbf {W}}} ^ {(1)} \| _ {\mathrm {F}} \leq B _ {1}} \exp \left(B _ {w} \prod_ {l = 2} ^ {L} B _ {l} \lambda \left\| \sum_ {i = 1} ^ {N} \varepsilon_ {i} \left(\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \mathbf {f} ^ {(0)} (\underline {{\mathbf {x}}} _ {i})\right) \right\| _ {\mathrm {F}}\right)\right) \\ \leq \frac {1}{\lambda} \log \left(2 ^ {L} \cdot \mathbb {E} _ {\varepsilon} \exp \left(B _ {w} \prod_ {l = 1} ^ {L} B _ {l} \lambda \left\| \sum_ {i = 1} ^ {N} \varepsilon_ {i} \underline {{\mathbf {X}}} _ {i} \right\| _ {\mathrm {F}}\right)\right). \tag {19} \\ \end{array}
$$
Letting  $B_{\mathbf{W}} = B_w \prod_{l=1}^L B_l$ , define a random variable
$$
Z = B _ {\underline {{\mathbf {W}}}} \cdot \left\| \sum_ {i = 1} ^ {N} \varepsilon_ {i} \underline {{\mathbf {X}}} _ {i} \right\| _ {\mathrm {F}}
$$
This is a natural consequence of (A.2) in Assumption 2. See Eq. (7).
as a function of random variables  $\{\varepsilon_i\}_i$ . Then
$$
\frac {1}{\lambda} \log \left(2 ^ {L} \cdot \mathbb {E} \exp \lambda Z\right) = \frac {L \log 2}{\lambda} + \frac {1}{\lambda} \log (\mathbb {E} \exp \lambda (Z - \mathbb {E} Z)) + \mathbb {E} Z. \tag {20}
$$
By Jensen's inequality,  $\mathbb{E}Z$  can be upper bounded by
$$
\begin{array}{l} \mathbb {E} Z = B _ {\underline {{\mathbf {W}}}} \mathbb {E} \left\| \sum_ {i = 1} ^ {N} \varepsilon_ {i} \underline {{\mathbf {x}}} _ {i} \right\| _ {\mathrm {F}} \leq B _ {\underline {{\mathbf {W}}}} \sqrt {\mathbb {E} \left\| \sum_ {i = 1} ^ {N} \varepsilon_ {i} \underline {{\mathbf {x}}} _ {i} \right\| _ {\mathrm {F}} ^ {2}} = B _ {\underline {{\mathbf {W}}}} \sqrt {\mathbb {E} _ {\boldsymbol {\epsilon}} \left[ \sum_ {i , j = 1} ^ {N} \varepsilon_ {i} \varepsilon_ {j} (\operatorname {v e c} (\underline {{\mathbf {x}}} _ {i})) ^ {\top} (\operatorname {v e c} (\underline {{\mathbf {x}}} _ {j})) \right]} \\ = B _ {\underline {{\mathbf {W}}}} \sqrt {\sum_ {i = 1} ^ {N} \left\| \underline {{\mathbf {x}}} _ {i} \right\| _ {\mathbf {F}} ^ {2}}. \\ \end{array}
$$
To handle the log  $\left(\mathbb{E}\exp \lambda (Z - \mathbb{E}Z)\right)$  term in Eq. (20), note that  $Z$  is a deterministic function of the i.i.d. random variables  $\{\varepsilon_i\}_{i}$ , and satisfies
$$
Z \left(\varepsilon_ {1}, \dots , \varepsilon_ {i}, \dots , \varepsilon_ {m}\right) - Z \left(\varepsilon_ {1}, \dots , - \varepsilon_ {i}, \dots , \varepsilon_ {m}\right) \leq 2 B _ {\mathbf {W}} \| \underline {{\mathbf {x}}} _ {i} \| _ {\mathbf {F}}.
$$
This means that  $Z$  satisfies a bounded-difference condition, which by the proof of [5, Theorem 6.2], implies that  $Z$  is sub-Gaussian, with the variance factor
$$
v = \frac {1}{4} \left(2 B _ {\underline {{\mathbf {W}}}} \| \underline {{\mathbf {x}}} _ {i} \| _ {\mathrm {F}}\right) ^ {2} = B _ {\underline {{\mathbf {W}}}} ^ {2} \sum_ {i = 1} ^ {N} \| \underline {{\mathbf {x}}} _ {i} \| _ {\mathrm {F}} ^ {2}.
$$
$$
\frac {1}{\lambda} \log \left(\mathbb {E} \exp \lambda (Z - \mathbb {E} Z)\right) \leq \frac {1}{\lambda} \frac {\lambda^ {2} B _ {\underline {{\mathbf {W}}}} ^ {2} \sum_ {i = 1} ^ {N} \| \underline {{\mathbf {x}}} _ {i} \| _ {\mathrm {F}} ^ {2}}{2} = \frac {\lambda B _ {\underline {{\mathbf {W}}}} ^ {2} \sum_ {i = 1} ^ {N} \| \underline {{\mathbf {x}}} _ {i} \| _ {\mathrm {F}} ^ {2}}{2}.
$$
Choosing  $\lambda = \frac{\sqrt{2L\log 2}}{B_{\mathbf{W}}\sqrt{\sum_{i = 1}^{N}\left\|\underline{\mathbf{x}}_i\right\|_{\mathrm{F}}^2}}$  and using the above inequality, we get that Eq. (19) can be upper bounded as follows
$$
\begin{array}{l} \frac {1}{\lambda} \log \left(2 ^ {L} \cdot \mathbb {E} \exp \lambda Z\right) \leq \mathbb {E} Z + \sqrt {2 L \log 2} B _ {\underline {{\mathbf {W}}}} \sqrt {\sum_ {i = 1} ^ {N} \| \underline {{\mathbf {x}}} _ {i} \| _ {\mathrm {F}} ^ {2}} \\ \leq (\sqrt {2 L \log 2} + 1) B _ {\underline {{\mathbf {W}}}} \sqrt {\sum_ {i = 1} ^ {N} \left\| \underline {{\mathbf {x}}} _ {i} \right\| _ {\mathbf {F}} ^ {2}}. \\ \end{array}
$$
Further applying Lemma 29 completes the proof.
C.2 Adversarial Generalization Bound for t-NNs
Proof of Theorem 5. According to Theorem 2 and Eq. (4) in Ref. [3], the adversarial generalization gap of  $\ell \circ f$  for any  $f\in \mathfrak{F}$  with  $L_{\ell}$ -Lipschitz continuous loss function  $\ell$  satisfying Assumption 2 can be upper bounded by  $L_{\ell}\hat{R}_{S}(\mathfrak{F}^{\mathrm{adv}})$ , where  $\hat{R}_S(\mathfrak{F}^{\mathrm{adv}})$  is the empirical Rademacher complexity of the adversarial version  $\mathfrak{F}^{\mathrm{adv}}$  of the function set  $\mathfrak{F}$  defined as follows
$$
\mathfrak {F} ^ {\mathrm {a d v}} := \left\{\tilde {f}: (\underline {{\mathbf {x}}}, y) \mapsto \min  _ {R _ {\mathrm {a}} (\underline {{\mathbf {x}}} - \underline {{\mathbf {x}}} ^ {\prime}) \leq \xi} y f \left(\underline {{\mathbf {x}}} ^ {\prime}\right) \mid f \in \mathfrak {F} \right\}. \tag {21}
$$
To bound  $\hat{R}_S(\mathfrak{F}^{\mathrm{adv}})$ , we use the Dudley's inequality (Lemma 31) which requires to compute the covering number of  $\mathfrak{F}^{\mathrm{adv}}$ .
Let  $\mathbf{C}_l$  be the  $\delta_l$ -covering of  $\{\underline{\mathbf{W}}^{(l)} \mid \left\| \underline{\mathbf{W}}^{(l)} \right\|_{\mathrm{F}} \leq B_l\}$ ,  $\forall l = 1, \dots, L$ . Consider the following subset of  $\mathfrak{F}$  whose t-matrix weights are all in  $\mathbf{C}_l$ :
$$
\mathfrak {F} _ {c} := \left\{f _ {c}: \underline {{\mathbf {x}}} \mapsto f _ {\mathcal {W} _ {c}} (\underline {{\mathbf {x}}}) \mid \mathcal {W} _ {c} = \left(\mathbf {w}, \underline {{\mathbf {W}}} _ {c} ^ {(1)}, \dots , \underline {{\mathbf {W}}} _ {c} ^ {(L)}\right), \underline {{\mathbf {W}}} _ {c} ^ {(l)} \in \mathbf {C} _ {l} \right\}
$$
with adversarial version
$$
\mathfrak {F} _ {c} ^ {\mathrm {a d v}} := \left\{\tilde {f} _ {c}: (\underline {{\mathbf {x}}}, y) \mapsto \inf  _ {R _ {\mathrm {a}} (\underline {{\mathbf {x}}} - \underline {{\mathbf {x}}} ^ {\prime}) \leq \xi} y f _ {c} (\underline {{\mathbf {x}}} ^ {\prime})   |   f _ {c} \in \mathfrak {F} _ {c} \right\}.
$$
For all  $\tilde{f} \in \mathfrak{F}^{\mathrm{adv}}$ , we need to find the smallest distance to  $\mathfrak{F}_c^{\mathrm{adv}}$ , i.e. we need to calculate
$$
\sup  _ {\tilde {f} \in \mathfrak {F} ^ {\text {a d v}}} \inf  _ {\tilde {f} _ {c} \in \mathfrak {F} _ {c} ^ {\text {a d v}}} \left\| \tilde {f} - \tilde {f} _ {c} \right\| _ {S}.
$$
For all  $(\underline{\mathbf{x}}_i, y_i) \in S$ , given  $\tilde{f}$  and  $\tilde{f}_c$  with  $\left\| \underline{\mathbf{W}}^{(l)} - \underline{\mathbf{W}}_c^{(l)} \right\|_{\mathrm{F}} \leq \delta_l$ ,  $l = 1, \dots, L$ , consider
$$
| \tilde {f} (\underline {{\mathbf {x}}} _ {i}, y _ {i}) - \tilde {f} _ {c} (\underline {{\mathbf {x}}} _ {i}, y _ {i}) | = \left| \inf  _ {R _ {\mathrm {a}} (\underline {{\mathbf {x}}} - \underline {{\mathbf {x}}} ^ {\prime}) \leq \xi} y _ {i} f (\underline {{\mathbf {x}}} _ {i} ^ {\prime}) - \inf  _ {R _ {\mathrm {a}} (\underline {{\mathbf {x}}} - \underline {{\mathbf {x}}} ^ {\prime}) \leq \xi} y _ {i} f _ {c} (\underline {{\mathbf {x}}} _ {i} ^ {\prime}) \right|.
$$
Letting  $\underline{\mathbf{x}}_i^f = \mathrm{arginf}_{R_{\mathrm{a}}(\underline{\mathbf{x}}_i - \underline{\mathbf{x}}_i') \leq \xi} y_i f(\underline{\mathbf{x}}_i')$  and  $\underline{\mathbf{x}}_i^c = \mathrm{arginf}_{R_{\mathrm{a}}(\underline{\mathbf{x}}_i - \underline{\mathbf{x}}_i') \leq \xi} y_i f_c(\underline{\mathbf{x}}_i')$ , we have
$$
| \tilde {f} (\underline {{\mathbf {x}}} _ {i}, y _ {i}) - \tilde {f} _ {c} (\underline {{\mathbf {x}}} _ {i}, y _ {i}) | = | y _ {i} f (\underline {{\mathbf {x}}} _ {i} ^ {f}) - y _ {i} f _ {c} (\underline {{\mathbf {x}}} _ {i} ^ {c}) |.
$$
$$
\underline {{\mathbf {x}}} _ {i} ^ {\xi} = \left\{ \begin{array}{l l} \underline {{\mathbf {x}}} _ {i} ^ {c} & \text {i f} \quad y _ {i} f (\underline {{\mathbf {x}}} _ {i} ^ {f}) \geq y _ {i} f _ {c} (\underline {{\mathbf {x}}} _ {i} ^ {c}) \\ \underline {{\mathbf {x}}} _ {i} ^ {f} & \text {i f} \quad y _ {i} f (\underline {{\mathbf {x}}} _ {i} ^ {c}) <   y _ {i} f _ {c} (\underline {{\mathbf {x}}} _ {i} ^ {c}) \end{array} \right..
$$
$$
| \tilde {f} (\underline {{\mathbf {x}}} _ {i}, y _ {i}) - \tilde {f} _ {c} (\underline {{\mathbf {x}}} _ {i}, y _ {i}) | = | y _ {i} f (\underline {{\mathbf {x}}} _ {i} ^ {f}) - y _ {i} f _ {c} (\underline {{\mathbf {x}}} _ {i} ^ {g}) | \leq | y _ {i} f (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) - y _ {i} f _ {c} (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) | = | f (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) - f _ {c} (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) |.
$$
Let  $g_{l}(\underline{\mathbf{x}}^{\delta}) = \mathbf{w}^{\top}\mathrm{vec}\left(\sigma (\underline{\mathbf{W}}_{c}^{(L)}*_{M}\sigma (\underline{\mathbf{W}}_{c}^{(L - 1)}*_{M}\dots *_{M}\sigma (\underline{\mathbf{W}}_{c}^{(l + 1)}*_{M}\sigma (\underline{\mathbf{W}}_{c}^{(l)}*_{M}\dots *_{M}\sigma (\underline{\mathbf{W}}_{c}^{(1)}*_{M}\right.$
$\left.\overline{\mathbf{x}}^{\delta}\right)\cdot \cdot \cdot \left.\right)$  and  $g_0(\underline{\mathbf{x}}^\delta) = f_c(\underline{\mathbf{x}}^\delta)$  . Then, we have
$$
| f (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) - f _ {c} (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) | \leq \sum_ {l = 1} ^ {L} | g _ {l} (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) - g _ {l - 1} (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) |.
$$
We can see that
$$
\left\| \sigma (\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}) \dots)\right) \| _ {\mathrm {F}} \leq \prod_ {l ^ {\prime} = 1} ^ {l} \left\| \underline {{\mathbf {W}}} ^ {(l ^ {\prime})} \right\| _ {\mathrm {F}} \left\| \underline {{\mathbf {x}}} _ {i} ^ {\xi} \right\| _ {\mathrm {F}} \leq \prod_ {l ^ {\prime} = 1} ^ {l} B _ {l} B _ {x, R _ {\mathrm {a}}, \xi},
$$
$$
\begin{array}{l} \left\| \sigma (\underline {{\mathbf {W}}} _ {c} ^ {(l + 1)} * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}) \dots)) - \sigma (\underline {{\mathbf {W}}} ^ {(l + 1)} * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}) \dots)) \right\| _ {\mathrm {F}} \\ \leq \left\| \underline {{\mathbf {W}}} _ {c} ^ {(l + 1)} * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}) \dots) - \underline {{\mathbf {W}}} ^ {(l + 1)} * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}) \dots) \right\| _ {\mathrm {F}} \\ \leq \left\| \left(\underline {{\mathbf {W}}} _ {c} ^ {(l + 1)} - \underline {{\mathbf {W}}} ^ {(l + 1)}\right) * _ {M} \sigma \left(\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma \left(\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}\right) \dots\right) \right\| _ {\mathrm {F}} \\ \leq \left\| \underline {{\mathbf {W}}} _ {c} ^ {(l + 1)} - \underline {{\mathbf {W}}} ^ {(l + 1)} \right\| _ {\mathrm {F}} \left\| \sigma \left(\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma \left(\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}\right) \dots\right) \right\| _ {\mathrm {F}} \\ \leq \delta_ {l} \cdot \prod_ {l ^ {\prime} = 1} ^ {l - 1} B _ {l ^ {\prime}} B _ {x, R _ {\mathrm {a}}, \xi}. \tag {22} \\ \end{array}
$$
$$
\left| g _ {l} \left(\underline {{\mathbf {x}}} _ {i} ^ {\xi}\right) - g _ {l - 1} \left(\underline {{\mathbf {x}}} _ {i} ^ {\xi}\right) \right| \leq \delta_ {l} \cdot B _ {\mathbf {w}} \prod_ {l ^ {\prime} \neq l} B _ {l ^ {\prime}} B _ {x, R _ {\mathrm {a}}, \xi} = \frac {\delta_ {l} B _ {\tilde {f}}}{B _ {l}},
$$
where  $B_{\bar{f}}$  is given in Lemma 39.
Letting  $\frac{\delta_lB_{\tilde{f}}}{B_l} = \frac{\epsilon}{L}$  it gives
$$
\sup  _ {\tilde {f} \in \mathfrak {F} ^ {\text {a d v}}} \inf  _ {\tilde {f} _ {c} \in \mathfrak {F} _ {c} ^ {\text {a d v}}} \left\| \tilde {f} - \tilde {f} _ {c} \right\| _ {S} \leq \sum_ {l = 1} ^ {L - 1} | g _ {l} (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) - g _ {l - 1} (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) | \leq \epsilon .
$$
Then,  $\mathfrak{F}_c^{\mathrm{adv}}$  is an  $\epsilon$ -covering of  $\mathfrak{F}^{\mathrm{adv}}$ . We further proceed by computing the  $\epsilon$ -covering number of  $\mathfrak{F}^{\mathrm{adv}}$  as follows:
$$
\mathsf {N} \left(\mathfrak {F} ^ {\text {a d v}}, \| \cdot \| _ {S}, \epsilon\right) \leq \left| \mathfrak {F} _ {c} ^ {\text {a d v}} \right| = \prod_ {l = 1} ^ {L} | \mathbf {C} _ {l} | \leq \prod_ {l = 1} ^ {L} \left(\frac {3 B _ {l}}{\delta_ {l}}\right) ^ {c d _ {l - 1} d _ {l}} \leq \left(\frac {3 L B _ {\tilde {f}}}{\epsilon}\right) ^ {\sum_ {l = 1} ^ {L} c d _ {l - 1} d _ {l}}.
$$
where the second inequality is due to Lemma 32.
Then, we use Dudley's integral in Lemma 31 to upper bound  $\hat{R}_S(\mathfrak{F}^{\mathrm{adv}})$  as follows
$$
\begin{array}{l} \hat {R} _ {S} \left(\mathfrak {F} ^ {\mathrm {a d v}}\right) \leq \inf  _ {\delta > 0} \left(8 \delta + \frac {1 2}{\sqrt {N}} \int_ {\delta} ^ {D _ {\bar {f}} / 2} \sqrt {\log N \left(\mathfrak {F} ^ {\mathrm {a d v}} , \| \cdot \| _ {S} , \epsilon\right)} \mathrm {d} \epsilon\right) \\ \leq \inf  _ {\delta > 0} \left(8 \delta + \frac {1 2}{\sqrt {N}} \int_ {\delta} ^ {D _ {\tilde {f}} / 2} \sqrt {\left(\sum_ {l = 1} ^ {L} c d _ {l - 1} d _ {l}\right) \log \left(3 L B _ {\tilde {f}} / (\epsilon)\right)} \mathrm {d} \epsilon\right) \\ = \inf  _ {\delta > 0} \left(8 \delta + \frac {1 2 D _ {\tilde {f}} \sqrt {\sum_ {l = 1} ^ {L} c d _ {l - 1} d _ {l}}}{\sqrt {N}} \int_ {\delta / D _ {\tilde {f}}} ^ {1 / 2} \sqrt {\log (3 L / (2 t))} \mathrm {d} t\right), \\ \end{array}
$$
where the diameter  $D_{\tilde{f}}$  of  $\mathfrak{F}^{\mathrm{adv}}$  is given in Lemma 40 and we can find from Lemma 40 that  $D_{\tilde{f}} = 2B_{\tilde{f}}$  in our setting.
Following Ref. [55], let  $\delta \to 0$ , and use integration by part, we obtain  $\int_0^{1/2} \sqrt{\log(3L/(2t))} \, \mathrm{d}t \leq \sqrt{\log 3L}$ . Hence, we have
$$
\hat {R} _ {S} (\mathfrak {F} ^ {\mathrm {a d v}}) \leq \frac {2 4 B _ {\mathbf {w}} \prod_ {l = 1} ^ {L} B _ {l} \left(\sqrt {\sum_ {l = 1} ^ {L} c d _ {l - 1} d _ {l}}\right) B _ {x , R _ {\mathrm {a}} , \xi}}{\sqrt {N}},
$$
and the proof can be completed by using Lemma 29.
C.3 Generalization Bound under Exact Low-tubal-rank Parameterization
Proof of Theorem 6. The idea is similar to the proof of Theorem 5. According to Theorem 2 and Eq. (4) in Ref. [3], the adversarial generalization gap of  $\ell \circ f$  for any  $f\in \mathfrak{F}_{\mathbf{r}}$  with  $L_{\ell}$ -Lipschitz continuous loss function  $\ell$  satisfying Assumption 2 can be upper bounded by  $L_{\ell}\hat{R}_{S}(\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}})$ , where  $\hat{R}_S(\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}})$  is the empirical Rademacher complexity of the adversarial version  $\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$  of function set  $\mathfrak{F}_{\mathbf{r}}$  defined as follows
$$
\mathfrak {F} _ {\mathbf {r}} ^ {\mathrm {a d v}} := \left\{\tilde {f}: (\underline {{\mathbf {x}}}, y) \mapsto \min  _ {R _ {\mathrm {a}} (\underline {{\mathbf {x}}} - \underline {{\mathbf {x}}} ^ {\prime}) \leq \xi} y f \left(\underline {{\mathbf {x}}} ^ {\prime}\right) \mid f \in \mathfrak {F} _ {\mathbf {r}} \right\}. \tag {23}
$$
To bound  $\hat{R}_S(\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}})$ , we first use the Dudley's inequality (Lemma 31) and compute the covering number of  $\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$ .
Let  $\mathbf{C}_l$  be the  $\delta_l$ -covering of  $\{\underline{\mathbf{W}}^{(l)} \mid \left\| \underline{\mathbf{W}}^{(l)} \right\|_{\mathrm{F}} \leq B_l$  and  $r_{\mathrm{t}}(\underline{\mathbf{W}}^{(l)}) \leq r_l\}$ ,  $\forall l = 1, \dots, L$ . Consider the following subset of  $\mathfrak{F}_{\mathbf{r}}$  whose t-matrix weights are all in  $\mathbf{C}_l$ :
$$
\mathfrak {F} _ {c} := \left\{f _ {c}: \underline {{\mathbf {x}}} \mapsto f (\underline {{\mathbf {x}}}; \underline {{\mathbf {W}}} _ {c}) \mid \underline {{\mathbf {W}}} _ {c} = \left(\mathbf {w}, \underline {{\mathbf {W}}} _ {c} ^ {(1)}, \dots , \underline {{\mathbf {W}}} _ {c} ^ {(L)}\right), \underline {{\mathbf {W}}} _ {c} ^ {(l)} \in \mathbf {C} _ {l} \right\}
$$
For all  $\tilde{f} \in \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$ , we need to find the smallest distance to  $\mathfrak{F}_c^{\mathrm{adv}}$ , i.e. we need to calculate
$$
\sup  _ {\tilde {f} \in \mathfrak {F} _ {\mathbf {r}} ^ {\mathrm {a d v}}} \inf  _ {\tilde {f} _ {c} \in \mathfrak {F} _ {c} ^ {\mathrm {a d v}}} \left\| \tilde {f} - \tilde {f} _ {c} \right\| _ {S}.
$$
Letting  $\underline{\mathbf{x}}_i^f = \mathrm{arginf}_{R_{\mathrm{a}}(\underline{\mathbf{x}}_i - \underline{\mathbf{x}}_i')\leq \xi}y_i f(\underline{\mathbf{x}}_i')$  and  $\underline{\mathbf{x}}_i^c = \mathrm{arginf}_{R_{\mathrm{a}}(\underline{\mathbf{x}}_i - \underline{\mathbf{x}}_i')\leq \xi}y_i f_c(\underline{\mathbf{x}}_i')$ , we have