SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparing Theorem 6 with Theorem 5, we observe that the adversarial generalization bound under transformed low-rank parameterization has a better scaling, specifically  $O(\sqrt{c\sum_{l}r_{l}(d_{l-1} + d_{l}) / N})$ . This also implies that a smaller number  $N$  of training examples is required to achieve the desired accuracy, as well as reduced energy consumption, storage requirements, and computational cost. Please refer to Sec. A.1 in the appendix for numerical evidence.
4.2 Implicit Bias of Gradient Flow for Adversarial Training of Over-parameterized t-NNs
Although Theorem 6 shows exactly transformed low-rank parameterization leads to lower bounds, the well trained t-NNs on real data rarely have exactly transformed low-rank weights. In this section, we prove that the highly over-parameterized t-NNs, trained by adversarial training with gradient flow (GF), are approximately of transformed low-rank parameterization under certain conditions.
First, the proposed t-NN  $f(\underline{\mathbf{x}}; \underline{\mathbf{W}})$  is said to be (positively) homogeneous as the condition  $f(\underline{\mathbf{x}}; a\underline{\mathbf{W}}) = a^{L + 1}f(\underline{\mathbf{x}}; \underline{\mathbf{W}})$  holds for any positive constant  $a$ . Motivated by Ref. [29], we focus on the scale invariant adversarial perturbations defined as follows.
Definition 4 (Scale invariant adversarial perturbation [29]). An adversarial perturbation  $\underline{\delta}_i(\underline{\mathbf{W}})$  is said to be scale invariant for  $f(\underline{\mathbf{x}};\underline{\mathbf{W}})$  at any given example  $\underline{\mathbf{x}}_i$  if it satisfies  $\underline{\delta}_i(a\underline{\mathbf{W}}) = \underline{\delta}_i(\underline{\mathbf{W}})$  for any positive constant  $a$ .
Lemma 7. The  $l_{2}$ -FGM [34], FGSM [9],  $l_{2}$ -PGD and  $l_{\infty}$ -PGD [31] perturbations for the t-NNs are all scale invariant.
Then, we consider adversarial training of t-NNs with scale invariant adversarial perturbations by GF, which can be seen as gradient descent with infinitesimal step size. When using GF for the ReLU t-NNs,  $\underline{\mathbf{W}}$  changes continuously with time, and the trajectory of parameter  $\underline{\mathbf{W}}$  during training is an arc  $\underline{\mathbf{W}}: [0, \infty) \to \mathbb{R}^{\dim(\underline{\mathbf{W}})}, t \mapsto \underline{\mathbf{W}}(t)$  that satisfies the differential inclusion [7, 30]
$$
\frac {\mathrm {d} \underline {{\mathbf {W}}} (t)}{\mathrm {d} t} \in - \partial^ {\circ} \hat {\mathcal {L}} ^ {\mathrm {a d v}} (\underline {{\mathbf {W}}} (t)) \tag {12}
$$
for  $t \geq 0$  a.e., where  $\partial^{\circ} \hat{\mathcal{L}}^{\mathrm{adv}}$  denotes the Clarke's subdifferential [7] with respect to  $\underline{\mathbf{W}}(t)$ . If  $\hat{\mathcal{L}}^{\mathrm{adv}}(\underline{\mathbf{W}})$  is actually a  $C^{1}$ -smooth function, the above differential inclusion reduces to
$$
\frac {\mathrm {d} \underline {{\mathbf {W}}} (t)}{\mathrm {d} t} = - \frac {\partial \hat {\mathcal {L}} ^ {\mathrm {a d v}} (\underline {{\mathbf {W}}} (t))}{\partial \underline {{\mathbf {W}}} (t)} \tag {13}
$$
for any  $t \geq 0$ , which corresponds to the GF with differential in the usual sense. However, for simplicity, we follow Refs. [45, 46] and still use Eq. (13) to denote Eq. (12) with a slight abuse of notation, even if  $\hat{\mathcal{L}}^{\mathrm{adv}}$  does not satisfy differentiability but only local Lipschitzness<sup>6</sup>.
We also make an assumption on the training data as follows.
Assumption 8 (Existence of a separability of adversarial examples during training). There exists a time  $t_0$  such that  $\hat{\mathcal{L}}^{\mathrm{adv}}(t_0) \leq N^{-1} \ell(b_{\mathfrak{f}})$ .
This assumption is a generalization of the separability condition in Refs. [29, 30]. Adversarial training can typically achieve this separability in practice, i.e., the model can fit adversarial examples of the training dataset, making the above assumption reasonable. Then, we obtain the following lemma.
Lemma 9 (Convergence to the direction of a KKT point). Consider the hypothesis class  $\mathfrak{F}$  in Eq. (6). Under Assumptions 2 and 8, the limit point of normalized weights  $\{\underline{\mathbf{W}}(t) / \|\underline{\mathbf{W}}(t)\|_{\mathrm{F}} : t \geq 0\}$  of the  $GF$  for Eq. (13), i.e., the empirical adversarial risk with scale invariant adversarial perturbations  $\underline{\delta}_i(\underline{\mathbf{W}})$ , is aligned with the direction of a KKT point of the minimization problem:
$$
\min  _ {\underline {{\mathbf {W}}}} \frac {1}{2} \| \underline {{\mathbf {W}}} \| _ {\mathrm {F}} ^ {2}, \quad \text {s . t .} y _ {i} f (\underline {{\mathbf {x}}} _ {i} + \underline {{\boldsymbol {\delta}}} _ {i} (\underline {{\mathbf {W}}}); \underline {{\mathbf {W}}}) \geq 1, \quad i = 1, \dots , N. \tag {14}
$$
Building upon Lemma 9, we can establish that highly over-parameterized t-NNs undergoing adversarial training with GF will exhibit an implicit bias towards transformed low-rank weights.
Note that the ReLU function is not differentiable at 0. Practical implementations of gradient methods define the derivative  $\sigma'(0)$  to be a constant in  $[0,1]$ . In this work we assume for convenience that  $\sigma'(0) = 0$ .
Theorem 10 (Implicit low-rankness for t-NNs induced by GF). Suppose there is an example  $\underline{\mathbf{x}}_i$  satisfying  $\| \underline{\mathbf{x}}_i\|_{\mathrm{F}}\leq 1$  in the training set  $S = \{(\underline{\mathbf{x}}_i,y_i)\}_{i = 1}^N$ . Suppose there is a  $(J + 1)$ -layer  $(J\geq 2)$  ReLU  $t$ -NN, denoted by  $g(\underline{\mathbf{x}};\underline{\mathbf{V}})$  with parameters  $\underline{\mathbf{V}} = (\underline{\mathbf{V}}^{(1)},\dots ,\underline{\mathbf{V}}^{(J)},\mathbf{v})$ , satisfying the conditions:
(C.1) the dimensionality of the weight tensor  $\underline{\mathbf{V}}^{(j)}\in \mathbb{R}^{m_j\times m_{j - 1}\times c}$  of the  $j$  -th t-product layer satisfies  $m_j\geq 2,j = 1,\dots ,J;$
(C.2) there is a constant  $B_v > 0$ , such that the Euclidean norm of the weights  $\underline{\mathbf{V}} = (\underline{\mathbf{V}}^{(1)}, \dots, \underline{\mathbf{V}}^{(L)}, \mathbf{v})$  satisfy  $\| \underline{\mathbf{V}}^{(j)} \|_{\mathrm{F}} \leq B_v$  for any  $j = 1, \dots, J$  and  $\| \mathbf{v} \|_2 \leq B_v$ ;
(C.3) for all  $i\in \{1,\dots ,N\}$ , we have  $y_{i}g(\underline{\mathbf{x}}_{i} + \underline{\boldsymbol{\delta}}_{i}(\underline{\mathbf{V}});\underline{\mathbf{V}})\geq 1$
Then, we consider the class of over-parameterized  $t$ -NNs  $\mathfrak{F} = \{f(\underline{\mathbf{x}};\underline{\mathbf{W}})\}$  defined in Eq. (5) satisfying
(C.4) the number  $L$  of t-product layers is much greater than  $J$ ;
(C.5) the dimensionality of weight  $\underline{\mathbf{W}}^{(l)}\in \mathbb{R}^{d_l\times d_{l - 1}\times c}$  satisfies  $d_{l}\gg \max_{j\leq J}\{m_{j}\}$  for any  $l\leq L$
Let  $\underline{\mathbf{W}}^{*} = (\underline{\mathbf{W}}^{*(1)},\dots ,\underline{\mathbf{W}}^{*(L)},\mathbf{w}^{*})$  be a global optimum of Problem (14). Namely,  $\underline{\mathbf{W}}^*$  parameterizes a minimum-norm  $t$ -NN  $f(\underline{\mathbf{x}};\underline{\mathbf{W}}^{*})\in \mathfrak{F}$  that labels the perturbed training set correctly with margin 1 under scale invariant adversarial perturbations. Then, we have
$$
\frac {L}{\sum_ {l = 1} ^ {L} \left(r _ {\mathrm {s t b}} \left(\widetilde {\mathbf {W}} _ {M} ^ {* (l)}\right)\right) ^ {- 1 / 2}} \leq \frac {1}{\left(1 + \frac {1}{L}\right) \left(\frac {1}{B _ {v}}\right) ^ {\frac {J + 1}{L + 1}} \sqrt {\frac {L + 1}{(J + 1) + (\operatorname {c m} _ {J}) (L - J)}} - \frac {1}{L}},
$$
where  $\widetilde{\mathbf{W}}_M^{*(l)}$  denotes the  $M$ -block-diagonal matrix of weight tensor  $\underline{\mathbf{W}}^{*(l)}$  for any  $l = 1, \dots, L$ .
By the above theorem, when  $L$  is sufficiently large, the harmonic mean of the square root of the stable rank of  $\widetilde{\mathbf{W}}_M^{*(l)}$ , i.e., the  $M$ -block-diagonal matrix of weight tensor  $\underline{\mathbf{W}}^{*(l)}$ , is approximately bounded by  $\sqrt{cm_J}$ , which is significantly smaller than the square root of the dimensionality  $\sqrt{\min\{cd_l,cd_{l-1}\}}$  according to condition (C.5) in Theorem 10. Thus,  $f(\underline{\mathbf{x}}; \underline{\mathbf{W}}^*)$  has a nearly low-rank parameterization in the transformed domain. In our case, the weights  $\underline{\mathbf{W}}(t)$  generated by GF tend to have an infinite norm and to converge in direction to a transformed low-rank solution. Moreover, note that the ratio between the spectral norm and the F-norm is invariant to scaling, and hence it suggests that after a sufficiently long time, GF tends to reach a t-NN with transformed low-rank weight tensors. Refer to Sec. A.2 for numerical evidence supporting Theorem 10.
4.3 Robust Generalization with Approximate Transformed Low-rank Parameterization
Theorem 10 establishes that for highly over-parameterized adversarial training with GF, well-trained t-NNs exhibit approximately transformed low-rank parameters under specific conditions. In this section, we analyze the AGP of t-NNs that possess an approximately transformed low-rank parameterization<sup>7</sup>.
Initially, by employing low-tubal-rank tensor approximation [20], one can always compress an approximately low-tubal-rank parameterized t-NN  $f$  by a t-NN  $g \in \mathfrak{F}_{\mathbf{r}}$  with an exact low-tubal-rank parameterization, ensuring a small distance between  $g$  and  $f$  in the parameter space. Now, the question is: Can the small parametric distance between  $f$  and  $g$  also indicate a small difference in their adversarial generalization behaviors? To answer this question, we first define the  $(\delta, \mathbf{r})$ -approximate low-tubal-rank parameterized functions.
Definition 5  $((\delta, \mathbf{r})$ -approximate low-tubal-rank parameterization). A  $t$ -NN  $f(\underline{\mathbf{x}}; \underline{\mathbf{W}}) \in \mathfrak{F}$  with weights  $\underline{\mathbf{W}} = (\mathbf{w}, \underline{\mathbf{W}}^{(1)}, \dots, \underline{\mathbf{W}}^{(L)})$  is said to satisfy the  $(\delta, \mathbf{r})$ -approximate low-tubal parameterization with tolerance  $\delta > 0$  and rank  $\mathbf{r} = (r_1, \dots, r_L)^\top \in \mathbb{N}^L$ , if there is a  $t$ -NN  $g(\underline{\mathbf{x}}; \underline{\mathbf{W}}_{\mathbf{r}}) \in \mathfrak{F}_{\mathbf{r}}$  whose weights  $\underline{\mathbf{W}}_g = (\mathbf{w}, \underline{\mathbf{W}}_{r_1}^{(1)}, \dots, \underline{\mathbf{W}}_{r_L}^{(L)})$  satisfy  $\| \underline{\mathbf{W}}_{r_l}^{(l)} - \underline{\mathbf{W}}^{(l)} \|_{\mathrm{F}} \leq \delta$  for any  $l = 1, \dots, L$ .
Furthermore, let's consider the collection of t-NNs with approximately low-tubal-rank weights
$$
\mathfrak {F} _ {\delta , \mathbf {r}} := \{f \in \mathfrak {F} \mid f \text {s a t i s f i e s t h e} (\delta , \mathbf {r}) \text {- a p p r o x i m a t e l o w - t u b a l - r a n k p a r a m e t e r i z a t i o n} \}. \tag {15}
$$
<sup>7</sup>We use the tubal rank as a measure of low-rankness in the transformed domain for notation simplicity. One can also consider the average rank [52] or multi-rank [50] for more refined bounds with quite similar techniques.
Subsequently, we analyze the AGP for any  $f \in \mathfrak{F}_{\delta,\mathbf{r}}$  in terms of its low-tubal-rank compression  $g \in \mathfrak{F}_{\mathbf{r}}$ . The idea is motivated by the work on compressed bounds for non-compressed but compressible models [43], originally developed for generalization analysis of NNs for standard training.
Under Assumption 2, we first define  $\mathfrak{F}_{\delta,\mathbf{r}}^{\mathrm{adv}} := \{\tilde{f} : (\underline{\mathbf{x}}, y) \mapsto \min_{R_{\mathrm{a}}(\underline{\mathbf{x}}' - \underline{\mathbf{x}}) \leq \xi} y f(\underline{\mathbf{x}}') \mid f \in \mathfrak{F}_{\delta,\mathbf{r}}\}$  as the adversarial version of  $\mathfrak{F}_{\delta,\mathbf{r}}$ . To analyze the AGP of  $f \in \mathfrak{F}_{\delta,\mathbf{r}}$  through  $g \in \mathfrak{F}_{\mathbf{r}}$ , we instead consider their adversarial counterparts  $\tilde{f} \in \mathfrak{F}_{\delta,\mathbf{r}}^{\mathrm{adv}}$  and  $\tilde{g} \in \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$ , where  $\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$  is defined as  $\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}} := \{\tilde{g} : (\underline{\mathbf{x}}, y) \mapsto \min_{R_{\mathrm{a}}(\underline{\mathbf{x}} - \underline{\mathbf{x}}') \leq \xi} y g(\underline{\mathbf{x}}') \mid g \in \mathfrak{F}_{\mathbf{r}}\}$ . Define the Minkowski difference of  $\mathfrak{F}_{\delta,\mathbf{r}}^{\mathrm{adv}}$  and  $\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$  as  $\mathfrak{F}_{\delta,\mathbf{r}}^{\mathrm{adv}} - \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}} := \{\tilde{f} - \tilde{g} \mid \tilde{f} \in \mathfrak{F}_{\delta,\mathbf{r}}^{\mathrm{adv}}, \tilde{g} \in \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}\}$ . The empirical  $L_2$ -norm of a t-NN  $h \in \mathfrak{F}$  on the training data  $S = \{(\underline{\mathbf{x}}_i, y_i)\}_{i=1}^N$  is defined as  $\|h\|_S := \sqrt{N^{-1} \sum_{i=1}^N h^2(\underline{\mathbf{x}}_i, y_i)}$ , and the population  $L_2$ -norm is  $\|h\|_{L_2} := \sqrt{\mathbb{E}_P(\underline{\mathbf{x}}, y)[h^2(\underline{\mathbf{x}}, y)]}$ . Define the local Rademacher complexity of  $\mathfrak{F}_{\delta,\mathbf{r}}^{\mathrm{adv}} - \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$  of radius  $t > 0$  as  $\dot{R}_{\mathbf{r}}(\mathfrak{F}_{\delta,\mathbf{r}}^{\mathrm{adv}} - \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}) := \bar{R}_N(\{h \in \mathfrak{F}_{\delta,\mathbf{r}}^{\mathrm{adv}} - \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}} | \|h\|_{L_2} \leq t\})$ , where  $\bar{R}_N(\mathcal{H})$  denotes the average Rademacher complexity of a function class  $\mathcal{H}$  [4].
The first part of the upcoming Theorem 12 shows that a small parametric distance between  $f$  and  $g$  leads to a small empirical  $L_{2}$ -distance in the adversarial output space. Specifically, for any  $f(\underline{\mathbf{x}}; \underline{\mathbf{W}}) \in \mathfrak{F}_{\delta, \mathbf{r}}$  with compression  $g(\underline{\mathbf{x}}; \underline{\mathbf{W}}_{\mathbf{r}})$ , their (adversarial) empirical  $L_{2}$ -distance  $\| \tilde{f}(\underline{\mathbf{x}}; \underline{\mathbf{W}}) - \tilde{g}(\underline{\mathbf{x}}; \underline{\mathbf{W}}_{\mathbf{r}}) \|_{S}$  can be bounded by a small constant  $\hat{\tau} > 0$  in linearity of  $\delta$ . We also aim for a small population  $L_{2}$ -distance by first assuming the local Rademacher complexity  $\dot{R}_{\mathfrak{r}}(\mathfrak{F}_{\delta, \mathbf{r}}^{\mathrm{adv}} - \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}})$  can be bounded by a concave function of  $\mathfrak{r}$ , following common practice in Rademacher complexity analysis [4, 43].
Assumption 11. For any  $\mathfrak{r} > 0$ , there exists a function  $\phi(\mathfrak{r}): [0, \infty) \to [0, \infty)$  such that  $\dot{R}_{\mathfrak{r}}(\mathfrak{F}_{\delta, \mathbf{r}}^{\mathrm{adv}} - \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}) \leq \phi(\mathfrak{r})$  and  $\phi(2\mathfrak{r}) \leq 2\phi(\mathfrak{r})$ .
We further define  $\mathfrak{r}_* = \mathfrak{r}_*(t) \coloneqq \inf \left\{\mathfrak{r} > 0 \mid 16B_{\hat{f}}\mathfrak{r}^{-2}\phi(\mathfrak{r}) + B_{\hat{f}}\mathfrak{r}^{-1}\sqrt{2t / N} + 2tB_{\hat{f}}^2\mathfrak{r}^{-2} / N \leq 1 / 2\right\}$  for any  $t > 0$ , such that the population  $L_2$ -norm of any  $h \in \mathfrak{F}_{\delta, \mathbf{r}}^{\mathrm{adv}} - \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$  can be bounded by  $\| h \|_{L_2}^2 \leq 2(\| h \|_S^2 + \mathfrak{r}_*^2)$  using the peeling argument [42, Theorem 7.7]. We then establish an adversarial generalization bound for approximately low-tubal-rank t-NNs as follows.
Theorem 12 (Adversarial generalization bound for general approximately low-tubal-rank t-NNs). (I). For any  $f \in \mathfrak{F}_{\delta, \mathbf{r}}$  with adversarial proxy  $\tilde{f} \in \mathfrak{F}_{\delta, \mathbf{r}}^{\mathrm{adv}}$ , there exists a function  $g \in \mathfrak{F}_{\mathbf{r}}$  with adversarial proxy  $\tilde{g} \in \mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$ , such that the empirical  $L_{2}$ -distance  $\| \tilde{f} - \tilde{g} \|_{S} \leq \delta B_{\tilde{f}} \sum_{l=1}^{L} B_{l}^{-1} =: \hat{\mathfrak{r}}$ .
(II). Let  $\mathfrak{r} := \sqrt{2(\mathfrak{r}^2 + \mathfrak{r}_*^2)}$ . Under Assumptions 1, 2, 4, 11, there exist constants  $C_1, C_2 > 0$  satisfying
$$
\begin{array}{l} \mathcal {L} ^ {\mathrm {a d v}} (f) - \hat {\mathcal {L}} ^ {\mathrm {a d v}} (f) \leq \underbrace {\frac {C _ {1} L _ {\ell} B _ {\tilde {f}}}{\sqrt {N}} \sqrt {\mathsf {c} \sum_ {l = 1} ^ {L} r _ {l} \left(d _ {l - 1} + d _ {l}\right) \log \left(9 (L + 1)\right)} + B \sqrt {\frac {t}{2 N}}} _ {\text {m a i n t e r m}} \tag {16} \\ + \underbrace {C _ {2} \left(\Phi (\mathfrak {r}) + L _ {\ell} \dot {\mathfrak {r}} \sqrt {\frac {t}{N}} + \frac {t L _ {\ell} B _ {\tilde {f}}}{N}\right)} _ {\text {b i a s t e r m}}, \\ \end{array}
$$
for any  $f \in \mathfrak{F}_{\delta, \mathbf{r}}$  with probability at least  $1 - 4e^{-t}$  for any  $t > 0$ , where  $\Phi(\mathfrak{r})$  is defined as
$$
\Phi (\mathfrak {r}) := \bar {R} _ {N} \left(\left\{\ell \circ \tilde {f} - \ell \circ \tilde {g} \mid \tilde {f} \in \mathfrak {F} _ {\delta , \mathbf {r}} ^ {\mathrm {a d v}}, \tilde {g} \in \mathfrak {F} _ {\mathbf {r}} ^ {\mathrm {a d v}}, \| \tilde {f} - \tilde {g} \| _ {L _ {2}} \leq \mathfrak {r} \right\}\right).
$$
The main term of the bound quantifies the complexity of functions in  $\mathfrak{F}_{\mathbf{r}}$  with exact low-tubal-rank parameterization in adversarial settings, which can be significantly smaller than that of  $\mathfrak{F}_{\delta,\mathbf{r}}$ . On the other hand, the bias term captures the sample complexity required to bridge the gap between approximately low-tubal-rank parameterized  $\mathfrak{F}_{\delta,\mathbf{r}}$  and exactly low-tubal-rank parameterized  $\mathfrak{F}_{\mathbf{r}}$ . As we usually observe  $\mathfrak{r}_*^2 = o(1/\sqrt{N})$ , setting  $\hat{\mathfrak{r}} = o_p(1)$  allows the bias term to decay faster than the main term, which is  $O(1/\sqrt{N})$ . Theorem 12 suggests that a small parametric distance between  $f \in \mathfrak{F}_{\delta,\mathbf{r}}$  and  $g \in \mathfrak{F}_{\mathbf{r}}$  also implies a small difference in their adversarial generalization behaviors.
A special case. We also showcase a specific scenario where the weights of t-product layers exhibit a polynomial spectral decay in the transformed domain, leading to a considerably small AGP bound.
Assumption 13. Consider the setting where any  $t$ -NN  $f(\underline{\mathbf{x}};\underline{\mathbf{W}})\in \mathfrak{F}_{\delta ,\mathbf{r}}$  has tensor weights  $\underline{\mathbf{W}}^{(l)}$  ( $l = 1,\dots ,L$ ) whose singular values in the transformed domain satisfy  $\sigma_{j}(M(\underline{\mathbf{W}}^{(l)})_{\therefore ,k})\leq V_{0}\cdot j^{-\alpha}$ , where  $V_{0} > 0$  is a constant, and  $\sigma_{j}(\cdot)$  is the  $j$ -th largest singular value of a matrix.
Under Assumption 13, the weight tensor  $\underline{\mathbf{W}}^{(l)}$  can be approximated by its optimal tubal-rank- $r_l$  approximation  $\underline{\mathbf{W}}_{r_l}^{(l)}$  for any  $1 \leq r_l \leq \min\{d_l, d_{l-1}\}$  with error  $\| \underline{\mathbf{W}}^{(l)} - \underline{\mathbf{W}}_{r_l}^{(l)} \|_{\mathrm{F}} \leq \sqrt{\mathrm{c} / (2\alpha - 1)} V_0(r_l - 1)^{(1 - 2\alpha)/2}$  [20], which can be much smaller than  $\| \underline{\mathbf{W}}_{r_l}^{(l)} \|_{\mathrm{F}}$  when  $\alpha > 1/2$  is sufficiently large. Thus, we can find an exactly low-tubal-rank parameterized  $g \in \mathfrak{F}_{\mathbf{r}}$  for any  $f \in \mathfrak{F}_{\delta,\mathbf{r}}$  satisfying Assumption 13, such that the parametric distance between  $g$  and  $f$  is quite small. The following theorem shows that the small parametric distance also leads to a small AGP.
Theorem 14. Under Assumptions 1, 2, 4, and 13, if we let  $\hat{\mathfrak{r}} = V_0B_{\tilde{f}}\sum_{l=1}^{L}(r_l + 1)^{-\alpha}B_l^{-1}$ , then for any  $t-NN$ $f \in \mathfrak{F}_{\delta,\mathbf{r}}$ , there exists a function  $g \in \mathfrak{F}_{\mathbf{r}}$  whose  $t$ -product layer weights have tubal-rank exactly no greater than  $r_l$ , satisfying  $\| \tilde{f} - \tilde{g} \|_S \leq \hat{\mathfrak{r}}$ . Further, there is a constant  $C_\alpha$  only depending on  $\alpha$  such that the AGP, i.e.,  $\mathcal{L}^{\mathrm{adv}}(f) - \hat{\mathcal{L}}^{\mathrm{adv}}(f)$ , of any  $f \in \mathfrak{F}_{\delta,\mathbf{r}}$  can be upper bounded by
$$
C _ {\alpha} L _ {\ell} \bigg \{B _ {\tilde {f}} E _ {1} + \hat {\mathfrak {r}} \sqrt {E _ {1}} + E _ {2} ^ {\frac {2 \alpha}{2 \alpha + 1}} \left(B _ {\tilde {f}} ^ {\frac {2 \alpha - 1}{2 \alpha + 1}} + 1\right) + \hat {\mathfrak {r}} ^ {\frac {2 \alpha}{2 \alpha + 1}} \sqrt {E _ {2}} + (\hat {\mathfrak {r}} + \frac {B}{L _ {\ell}}) \sqrt {\frac {t}{N}} + \frac {1 + t B _ {\tilde {f}}}{N} \bigg \},
$$
for any  $t > 0$  with probability at least  $1 - 4e^{-t}$ , where  $E_1 = N^{-1} \mathsf{c} \sum_{l=1}^{L} r_l(d_l + d_{l-1}) \log(9NLB_{\tilde{f}}/\sqrt{\mathsf{c}})$  and  $E_2 = N^{-1} \mathsf{c} \sum_{l=1}^{L} \left(LV_0B_{\tilde{f}}B_l^{-1}\right)^{1/\alpha}(d_l + d_{l-1}) \log(9NLB_{\tilde{f}}/\sqrt{\mathsf{c}})$ .
This suggests that by choosing a sufficiently large  $\alpha > 1/2$ , where each weight tensor has a tubal-rank close to 1, we can attain a superior generalization error bound. It is important to note that the rank  $r_l$  can be arbitrarily chosen, and there exists a trade-off relationship between  $\hat{\tau}$  and  $E_1$ . Therefore, by selecting the rank appropriately for a balanced trade-off, we can obtain an optimal bound as follows.
Corollary 15. Under the same assumption to Theorem 14, if we choose the parameter  $\mathbf{r}$  of tubal ranks in  $\mathfrak{F}_{\mathbf{r}}$  by  $r_l = \min \{\left[\left(LV_0B_{\bar{f}}B_l^{-1}\right)^{1 / \alpha}\right], d_l, d_{l - 1}\}$ , then there is a constant  $C_{\alpha}$  only depending on  $\alpha$  such that the AGP of any  $f \in \mathfrak{F}_{\delta,\mathbf{r}}$  can be upper bounded as
$$
\begin{array}{l} \mathcal {L} ^ {\mathrm {a d v}} (f) - \hat {\mathcal {L}} ^ {\mathrm {a d v}} (f) \leq C _ {\alpha} L _ {\ell} \left\{B _ {\tilde {f}} ^ {1 - 1 / (2 \alpha)} \sqrt {\frac {\mathrm {c} \sum_ {l = 1} ^ {L} \left(L V _ {0} B _ {l} ^ {- 1}\right) ^ {1 / \alpha} \left(d _ {l} + d _ {l - 1}\right) \log \left(9 N L B _ {\tilde {f}} / \sqrt {\mathrm {c}}\right)}{N}} \right. \\ + E _ {2} ^ {\frac {2 \alpha}{2 \alpha + 1}} \left(B _ {\tilde {f}} ^ {\frac {2 \alpha - 1}{2 \alpha + 1}} + 1\right) + \sqrt {E _ {2}} + \frac {B}{L _ {\ell}} \sqrt {\frac {t}{N}} + \frac {1 + t B _ {\tilde {f}}}{N} \}, \\ \end{array}
$$
with probability at least  $1 - 4e^{-t}$  for any  $t > 0$ .
It is worth highlighting that the bound exhibits a linear dependency on the number of neurons in the t-product layers, represented as  $O(\sqrt{\mathsf{c}\sum_{l}(d_{l} + d_{l - 1}) / N})$ . In contrast, Theorem 5 demonstrates a dependency on the total number of parameters, denoted as  $O(\sqrt{\mathsf{c}\sum_{l}d_{l}d_{l - 1} / N})$ . This observation suggests that employing the low-tubal-rank parameterization can potentially enhance adversarial generalization for t-NNs.
T-SVD-based data and function representation. The unique feature of t-SVD-based data representation, in contrast to classical low-rank decomposition methods, is the presence of low-rankness in the transformed domain. This transformed low-rankness is crucial for effectively modeling real multi-channel data with both smoothness and low-rankness [24, 49, 50]. Utilized in t-product layers in DNNs [32, 36, 53], t-SVD has also been a workhorse for function representation and achieves impressive empirical performance. While t-SVD-based signal processing models have been extensively studied theoretically [13, 24, 27, 40, 50, 60], the t-SVD-based learning model itself has not been thoroughly scrutinized until this paper. Hence, this study represents the first theoretical analysis of t-SVD-based learning models, contributing to the understanding of their theoretical foundations.