SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andong Wang
andong.wang@riken.jp
chao.li@riken.jp
Mingyuan Bai
mingyuan.bai@riken.jp
Zhong Jin
China University of Petroleum-Beijing at Karamay
zhongjin@cupk.edu.cn
Guoxu Zhou*
Guangdong University of Technology
gx.zhou@gdut.edu.cn
Qibin Zhao*
qibin.zhao@riken.jp
Multi-channel learning has gained significant attention in recent applications, where neural networks with t-product layers (t-NNs) have shown promising performance through novel feature mapping in the transformed domain. However, despite the practical success of t-NNs, the theoretical analysis of their generalization remains unexplored. We address this gap by deriving upper bounds on the generalization error of t-NNs in both standard and adversarial settings. Notably, it reveals that t-NNs compressed with exact transformed low-rank parameterization can achieve tighter adversarial generalization bounds compared to non-compressed models. While exact transformed low-rank weights are rare in practice, the analysis demonstrates that through adversarial training with gradient flow, highly over-parameterized t-NNs with the ReLU activation can be implicitly regularized towards a transformed low-rank parameterization under certain conditions. Moreover, this paper establishes sharp adversarial generalization bounds for t-NNs with approximately transformed low-rank weights. Our analysis highlights the potential of transformed low-rank parameterization in enhancing the robust generalization of t-NNs, offering valuable insights for further research and development.
Multi-channel learning is a task to extract representations from the data with multiple channels, such as multispectral images, time series, and multi-view videos, in an efficient and robust manner [24, 39, 60, 61]. Among the methods tackling this task, neural networks with t-product layers (t-NNs, see Eq. (5) for a typical example) [12, 36] came to the stage very recently with remarkable efficiency and robustness in various applications such as graph learning, remote sensing, and more [1, 11, 14, 32, 38, 39, 53, 58]. What distinguishes t-NNs from other networks is the inclusion of t-product layers, founded on the algebraic framework of tensor singular value decomposition (t-SVD) [19, 40, 60, 61]. Unlike traditional tensor decompositions, t-SVD explores the transformed low-rankness, i.e., the low-rank structure of a tensor in the transformed domain under an invertible linear transform [18]. The imposed transform in t-product layers provides additional expressivity to neural networks, while
*Qibin Zhao and Guoxu Zhou are the corresponding authors.
the controllable transformed low-rank structure in t-NNs enables a flexible balance between model accuracy and robustness [36, 38, 53].
Despite the impressive empirical performance of t-NNs, the theoretical foundations behind their success remain unclear. The lack of systematic theoretical analysis hinders deeper comprehension and exploration of more effective applications and robust performance of t-NNs. Furthermore, the inclusion of the additional transform in t-NNs renders the theoretical analysis more technically challenging compared to existing work on general neural networks [29, 37, 43, 55]. To address this challenge, we establish for the first time a theoretical framework for t-NNs to understand both the standard and robust generalization behaviors, providing both theoretical insights and practical guidance for the efficient and robust utilization of t-NNs. Specifically, we address the following fundamental questions:
- Can we theoretically characterize the generalization behavior of general t-NNs? Yes. We derive the upper bounds on the generalization error for general t-NNs in both standard and adversarial settings in Sec. 3.
- How does exact transformed low-rankness influence the robust generalization of t-NNs? In Sec. 4.1, our analysis shows that t-NNs with exactly transformed low-rank weights exhibit lower adversarial generalization bounds and require fewer samples, highlighting the benefits of transformed low-rank weights in t-NNs for improved robustness and efficiency.
- How does adversarial learning of t-NNs affect the transformed ranks of their weight tensors? In Sec. 4.2, we deduce that weight tensors tend to be of transformed low-rankness approximately for highly over-parameterized t-NNs with ReLU activation under adversarial training with gradient flow.
- How is robust generalization impacted by approximately transformed low-rank weight tensors in t-NNs? In Sec. 4.3, we establish sharp adversarial generalization bounds for t-NNs with approximately transformed low-rank weights by carefully bridging the gap with exact transformed low-rank parameterization. This finding again underscores the importance of incorporating transformed low-rank weights as a means to enhance the robustness of t-NNs.
In this section, we introduce the notations and provide a concise overview of t-SVD and t-NNs, which play a central role in the subsequent analysis.
Notations. We use lowercase, lowercase boldface, and uppercase boldface letters to denote scalars, e.g.,  $a \in \mathbb{R}$ , vectors, e.g.,  $\mathbf{a} \in \mathbb{R}^m$ , and matrices, e.g.,  $\mathbf{A} \in \mathbb{R}^{m \times n}$ , respectively. Following the standard notations in Ref. [19], a 3-way tensor of size  $d \times 1 \times c$  is also called a  $t$ -vector and denoted by underlined lowercase, e.g.,  $\underline{\mathbf{x}}$ , whereas a 3-way tensor of size  $m \times n \times c$  is also called a  $t$ -matrix and denoted by underlined uppercase, e.g.,  $\underline{\mathbf{X}}$ . We use a  $t$ -vector  $\underline{\mathbf{x}} \in \mathbb{R}^{d \times 1 \times c}$  to represent a multi-channel example, where  $c$  denotes the number of channels and  $d$  is the number of features for each channel.
Given a matrix  $\mathbf{A} \in \mathbb{R}^{m \times n}$ , its Frobenius norm (F-norm) and spectral norm are defined as  $\| \mathbf{A} \|_{\mathrm{F}} \coloneqq \sqrt{\sum_{i=1}^{\min\{m,n\}} \sigma_i^2}$  and  $\| \mathbf{A} \| \coloneqq \max_i \sigma_i$ , respectively, where  $\sigma_i, i = 1, \dots, \min\{m,n\}$  are its singular values. The stable rank of a non-zero matrix  $\mathbf{A}$  is defined as the squared ratio of its F-norm and spectral norm  $r_{\mathrm{stb}}(\mathbf{A}) \coloneqq \| \mathbf{A} \|_{\mathrm{F}}^2 / \| \mathbf{A} \|^2$ . Given a tensor  $\underline{\mathbf{T}}$ , define its  $l_p$ -norm and F-norm respectively as  $\| \underline{\mathbf{T}} \|_{l_p} \coloneqq \| \operatorname{vec}(\underline{\mathbf{T}}) \|_{l_p}$ , and  $\| \underline{\mathbf{T}} \|_{\mathrm{F}} \coloneqq \| \operatorname{vec}(\underline{\mathbf{T}}) \|_{l_2}$ , where  $\operatorname{vec}(\cdot)$  denotes the vectorization operation of a tensor [21]. Given  $\underline{\mathbf{T}} \in \mathbb{R}^{m \times n \times c}$ , let  $\underline{\mathbf{T}}_{:,i}$  denote its  $i$ th frontal slice. The inner product between two tensors  $\underline{\mathbf{A}}, \underline{\mathbf{B}}$  is defined as  $\langle \underline{\mathbf{A}}, \underline{\mathbf{B}} \rangle \coloneqq \operatorname{vec}(\underline{\mathbf{A}})^{\top} \operatorname{vec}(\underline{\mathbf{B}})$ . The frontal-slice-wise product of two tensors  $\underline{\mathbf{A}}, \underline{\mathbf{B}}$ , denoted by  $\underline{\mathbf{A}} \odot \underline{\mathbf{B}}$ , equals a tensor  $\underline{\mathbf{T}}$  such that  $\underline{\mathbf{T}}_{:,i} = \underline{\mathbf{A}}_{:,i}, \underline{\mathbf{B}}_{:,i}$ ,  $i = 1, \dots, c$  [19]. We use  $|\cdot|$  as the absolute value for a scalar and cardinality for a set. We use  $\circ$  to denote the function composition operation. Additional notations will be introduced upon their first occurrence.
2.1 Tensor Singular Value Decomposition
The framework of tensor singular value decomposition (t-SVD) is based on the t-product under an invertible linear transform  $M$  [18]. In recent studies, the transformation matrix  $\mathbf{M}$  defining the transform  $M$  is restricted to be orthogonal [50] for better properties, which is also followed in this
paper. Given any orthogonal matrix  $\mathbf{M} \in \mathbb{R}^{c \times c}$ , define the associated linear transform  $M(\cdot)$  with its inverse  $M^{-1}(\cdot)$  on any  $\underline{\mathbf{T}} \in \mathbb{R}^{m \times n \times c}$  as
$$
M (\underline {{\mathbf {T}}}) := \underline {{\mathbf {T}}} \times_ {3} \mathbf {M}, \text {a n d} M ^ {- 1} (\underline {{\mathbf {T}}}) := \underline {{\mathbf {T}}} \times_ {3} \mathbf {M} ^ {- 1}, \tag {1}
$$
where  $\times_{3}$  denotes the tensor matrix product on mode-3 [18].
Definition 1 (t-product [18]). The  $t$ -product of any  $\underline{\mathbf{A}} \in \mathbb{R}^{m \times n \times c}$  and  $\underline{\mathbf{B}} \in \mathbb{R}^{n \times k \times c}$  under transform  $M$  in Eq. (1) is denoted and defined as  $\underline{\mathbf{A}} *_{M} \underline{\mathbf{B}} = \underline{\mathbf{C}} \in \mathbb{R}^{m \times k \times c}$  such that  $M(\underline{\mathbf{C}}) = M(\underline{\mathbf{A}}) \odot M(\underline{\mathbf{B}})$  in the transformed domain. Equivalently, we have  $\underline{\mathbf{C}} = M^{-1}(M(\underline{\mathbf{A}}) \odot M(\underline{\mathbf{B}}))$  in the original domain.
Definition 2 (M-block-diagonal matrix). The  $M$ -block-diagonal matrix of any  $\underline{\mathbf{T}} \in \mathbb{R}^{m \times n \times c}$ , denoted by  $\widetilde{\mathbf{T}}_M$ , is the block diagonal matrix whose diagonal blocks are the frontal slices of  $M(\underline{\mathbf{T}})$ :
$$
\widetilde {\mathbf {T}} _ {M} := \mathrm {b d i a g} (M (\underline {{\mathbf {T}}})) := \left[ \begin{array}{c c c c} M (\underline {{\mathbf {T}}}) _ {:,: 1} & & & \\ & M (\underline {{\mathbf {T}}}) _ {:,: 2} & & \\ & & \ddots & \\ & & & M (\underline {{\mathbf {T}}}) _ {:,: c} \end{array} \right] \in \mathbb {R} ^ {m c \times n c}.
$$
In this paper, we also follow the definition of t-transpose, t-identity tensor, t-orthogonal tensor, and f-diagonal tensor given by Ref. [18], and thus the t-SVD is introduced as follows.
Definition 3 (t-SVD, tubal rank [18]). Tensor Singular Value Decomposition  $(t\text{-SVD})$  of  $\mathbf{T} \in \mathbb{R}^{m \times n \times c}$  under the invertible linear transform  $M$  in Eq. (1) is given as follows
$$
\underline {{\mathbf {T}}} = \underline {{\mathbf {U}}} * _ {M} \underline {{\mathbf {S}}} * _ {M} \underline {{\mathbf {V}}} ^ {\top}, \tag {2}
$$
where  $\underline{\mathbf{U}}\in \mathbb{R}^{m\times m\times c}$  and  $\underline{\mathbf{V}}\in \mathbb{R}^{n\times n\times c}$  are  $t$ -orthogonal, and  $\underline{\mathbf{S}}\in \mathbb{R}^{m\times n\times c}$  is  $f$ -diagonal. The tubal rank of  $\underline{\mathbf{T}}$  is defined as the number of non-zero tubes of  $\underline{\mathbf{S}}$  in its  $t$ -SVD in Eq. (2), i.e.,  $r_{\mathrm{t}}(\underline{\mathbf{T}}):= |\{i|\underline{\mathbf{S}} (i,i,:)\neq \mathbf{0},i\leq \min \{m,n\} \} |.$
For any  $\underline{\mathbf{T}}\in \mathbb{R}^{m\times n\times c}$  with the tubal rank  $r_t(\underline{\mathbf{T}})$ , we have following relationship between its t-SVD and the matrix SVD of its  $M$ -block-diagonal matrix [26, 50]:
$$
\underline {{\mathbf {T}}} = \underline {{\mathbf {U}}} * _ {M} \underline {{\mathbf {S}}} * _ {M} \underline {{\mathbf {V}}} ^ {\top} \Leftrightarrow \widetilde {\mathbf {T}} _ {M} = \widetilde {\mathbf {U}} _ {M} \cdot \widetilde {\mathbf {S}} _ {M} \cdot \widetilde {\mathbf {V}} _ {M} ^ {\top}, \quad \text {a n d} \quad \mathrm {c} \cdot r _ {\mathrm {t}} (\underline {{\mathbf {T}}}) \geq \operatorname {r a n k} (\widetilde {\mathbf {T}} _ {M}). \tag {3}
$$
As the  $M$ -block-diagonal matrix  $\widetilde{\mathbf{T}}_M$  is defined after transforming tensor  $\underline{\mathbf{T}}$  from the original domain to the transformed domain, the relationship  $c \cdot r_{\mathrm{t}}(\underline{\mathbf{T}}) \geq \mathrm{rank}(\widetilde{\mathbf{T}}_M)$  indicates that the tubal rank can be chosen as a measure of transformed low-rankness [26, 50].
2.2 Neural Networks with t-Product Layer (t-NNs)
In this subsection, we will introduce the formulation of the t-product layer in t-NNs, which is designed for multi-channel feature learning.
Multi-channel feature learning via t-product. Suppose we have a multi-channel example represented by a t-vector  $\underline{\mathbf{x}}\in \mathbb{R}^{d\times 1\times c}$ , where  $c$  is the number of channels and  $d$  is the number of features. We define an  $L$ -layer t-NN feature extractor  $\mathbf{f}(\underline{\mathbf{x}})$ , to extract  $d_{L}$  features for each channel of  $\underline{\mathbf{x}}$ :
$$
\mathbf {f} (\underline {{\mathbf {x}}}) = \mathbf {f} ^ {(L)} (\underline {{\mathbf {x}}}); \quad \mathbf {f} ^ {(l)} (\underline {{\mathbf {x}}}) = \sigma \left(\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \mathbf {f} ^ {(l - 1)} (\underline {{\mathbf {x}}})\right), l = 1, \dots , L; \quad \mathbf {f} ^ {(0)} (\underline {{\mathbf {x}}}) = \underline {{\mathbf {x}}}, \tag {4}
$$
where the  $l$ -th layer  $\mathbf{f}^{(l)}$  first conducts t-product with weight tensor (t-matrix)  $\underline{\mathbf{W}}^{(l)} \in \mathbb{R}^{d_l \times d_{l-1} \times c}$  on the output of the  $(l-1)$ -th layer as multi-channel features $^2$ $\mathbf{f}^{(l-1)}(\underline{\mathbf{x}}) \in \mathbb{R}^{d_{l-1} \times 1 \times c}$  to obtain a  $(d_l \times 1 \times c)$ -dimensional representation and then uses the entry-wisely ReLU activation $^3$ $\sigma(x) = \max\{x, 0\}$  for nonlinearity.
Remark. Unlike Refs. [36], [32] and [53] whose nonlinear activation is performed in the transformed domain, the t-NN model in Eq. (4) considers the nonlinear activation in the original domain and hence is consistent with traditional neural networks.
2For simplicity, let  $d_0 = d$  by treating the input example  $\mathbf{x}$  as the 0-th layer  $\mathbf{f}^{(0)}$ .
<sup>3</sup>Although we consider ReLU activation in this paper, most of the main theoretical results (e.g., Theorems 3, 5, 6, 12, and 14) can be generalized to general Lipschitz activations with slight modifications in the proof.
By adding a linear classification module with weight  $\mathbf{w} \in \mathbb{R}^{cd_L}$  after the feature exaction module in Eq. (4), we consider the following t-NN predictor whose sign can be utilized for binary classification:
$$
f (\underline {{\mathbf {x}}}; \underline {{\mathbf {W}}}) := \mathbf {w} ^ {\top} _ {\text {V E C}} \left(\mathbf {f} ^ {(L)} (\underline {{\mathbf {x}}})\right) \in \mathbb {R}. \tag {5}
$$
Let  $\underline{\mathbf{W}} := \{\underline{\mathbf{W}}^{(1)}, \dots, \underline{\mathbf{W}}^{(L)}, \mathbf{w}\}$  be the collection of all the weights. With a slight abuse of notation, let  $\| \underline{\mathbf{W}} \|_{\mathrm{F}} := \sqrt{\|\mathbf{w}\|_2^2 + \sum_{l=1}^{L} \|\underline{\mathbf{W}}^{(l)}\|_{\mathrm{F}}^2}$  denote the Euclidean norm of all the weights. The function class of general t-NNs whose weights are bounded in the Euclidean norm is defined as
$$
\mathfrak {F} := \left\{f (\mathbf {x}; \underline {{\mathbf {W}}}) \mid \| \mathbf {w} \| _ {2} \leq B _ {w}, \| \underline {{\mathbf {W}}} ^ {(l)} \| _ {\mathrm {F}} \leq B _ {l}, l = 1, \dots , L \right\}, \tag {6}
$$
with positive constants  $B_w$  and  $B_l$ ,  $l = 1, \dots, L$ . Let  $B_{\mathbf{W}} \coloneqq B_w \prod_{l=1}^{L} B_l$  for simplicity.
3 Standard and Robust Generalization Bounds for t-NNs
This section establishes both the standard and robust generalization bounds for any t-NN  $f \in \mathfrak{F}$ .
3.1 Standard Generalization for General t-NNs
Suppose we are given a training multi-channel dataset  $S$  consisting of  $N$  example-label pairs  $\{(\underline{\mathbf{x}}_i, y_i)\}_{i=1}^N \subset \mathbb{R}^{d \times 1 \times c} \times \{\pm 1\}$  i.i.d. drawn from an underlying data distribution  $P_{\underline{\mathbf{x}},y}$ .
Assumption 1. Every input example  $\underline{\mathbf{x}}\in \mathbb{R}^{d\times 1\times c}$  has an upper bounded  $F$  -norm, i.e.,  $\| \underline{\mathbf{x}}\|_{\mathrm{F}}\leq B_x$  where  $B_{x}$  is a positive constant.
When a loss function  $\ell(f(\underline{\mathbf{x}}_i), y_i)$  is considered as the measure of the classification quality, we define the empirical and population risk for any  $f \in \mathfrak{F}$  as  $\hat{\mathcal{L}}(f) \coloneqq N^{-1} \sum_{i=1}^{N} \ell(f(\underline{\mathbf{x}}_i), y_i)$  and  $\mathcal{L}(f) \coloneqq \mathbb{E}_{P_{(\underline{\mathbf{x}}, y)}}[\ell(f(\underline{\mathbf{x}}), y)]$ , respectively. Similar to Ref. [30], we make assumptions on the loss as follows.
Assumption 2. The loss  $\ell(h(\underline{\mathbf{x}}), y)$  can be expressed as  $\ell(h(\underline{\mathbf{x}}), y) = \exp(-\mathfrak{f}(yh(\underline{\mathbf{x}})))$  for any  $t$ -NN  $h \in \mathfrak{F}$ , such that:
(A.1) the range of loss  $\ell (\cdot ,\cdot)$  is  $[0,B]$ , where  $B$  is a positive constant;
(A.2) function  $\mathfrak{f}:\mathbb{R}\to \mathbb{R}$  is  $C^1$ -smooth;
(A.3)  $f'(x) \geq 0$  for any  $x \in \mathbb{R}$ ;
(A.4) there exists  $b_{\mathfrak{f}} \geq 0$  such that  $x \mathfrak{f}'(x)$  is non-decreasing for  $x \in (b_{\mathfrak{f}}, +\infty)$ , and the derivative  $x \mathfrak{f}'(x) \to +\infty$  as  $x \to +\infty$ ;
(A.5) let  $\mathfrak{g} : [\mathfrak{f}(b_{\mathfrak{f}}), +\infty) \to [b_{\mathfrak{f}}, +\infty)$  be the inverse function of  $\mathfrak{f}$  on the domain  $[b_{\mathfrak{f}}, +\infty)$ . There exist  $b_{\mathfrak{g}} \geq \max \{2\mathfrak{f}(b_{\mathfrak{f}}), \mathfrak{f}(2b_{\mathfrak{f}})\}$  and  $K \geq 1$ , such that  $\mathfrak{g}'(x) \leq K\mathfrak{g}'(\theta x)$  and  $\mathfrak{f}'(y) \leq K\mathfrak{f}'(\theta y)$  for any  $x \in (b_{\mathfrak{g}}, +\infty)$ ,  $y \in (\mathfrak{g}(b_{\mathfrak{g}}), +\infty)$  and  $\theta \in [1/2, 1)$ .
Assumption (A.1) is a natural assumption in generalization analysis [3, 59], and Assumptions (A.2)-(A.5) are the same as Assumption (B3) in Ref. [30]. According to Assumption (A.2), the loss function  $\ell(\cdot, \cdot)$  satisfies the  $L_{\ell}$ -Lipschitz continuity
$$
\left| \ell \left(h (\underline {{\mathbf {x}}}} _ {1}), y _ {1}\right) - \ell \left(h (\underline {{\mathbf {x}}}} _ {2}), y _ {2}\right) \right| \leq L _ {\ell} \left| y _ {1} h \left(\underline {{\mathbf {x}}} _ {1}\right) - y _ {2} \left(\underline {{\mathbf {x}}} _ {2}\right) \right|, \quad \text {w i t h} L _ {\ell} = \sup  _ {| q | \leq B _ {\hat {f}}} f ^ {\prime} (q) e ^ {- f (q)}, \tag {7}
$$
where  $B_{\tilde{f}}$  is an upper bound on the output of any t-NN  $h \in \mathfrak{F}$ . The Lipschitz continuity is also widely assumed for generalization analysis of DNNs [55, 59]. Assumption 2 is satisfied by commonly used loss functions such as the logistic loss and the exponential loss.
The generalization gap  $\mathcal{L}(f) - \hat{\mathcal{L}}(f)$  of any function  $f \in \mathfrak{F}$  can be bounded as follows.
Lemma 3 (Generalization bound for t-NNs). Under Assumptions 1 and 2, it holds for any  $f \in \mathfrak{F}$  that
$$
\mathcal {L} (f) - \hat {\mathcal {L}} (f) \leq \frac {L _ {\ell} B _ {x} B _ {\mathbf {W}}}{\sqrt {N}} (\sqrt {2 \log (2 (L + 1))} + 1) + 3 B \sqrt {\frac {t}{2 N}}, \tag {8}
$$
with probability at least  $1 - 2e^{-t}$  for any  $t > 0$ .
Here for the ease of notation presentation, we use the tensor notation  $\mathbf{W}$  instead of the set notation  $\mathcal{W}$ .
Remark. When the input example has channel number  $c = 1$ , the generalization bound in Theorem 3 is consistent with the  $F$ -norm-based bound in Ref. [8].
3.2 Robust Generalization for General t-NNs
We study the adversarial generalization behavior of t-NNs in this section. We first make the following assumption on the adversarial perturbations.
Assumption 4. Given an input example  $\underline{\mathbf{x}}$ , the adversarial perturbation is chosen within a radius- $\xi$  ball of norm  $R_{\mathrm{a}}(\cdot)$  with compatibility constant [35] defined as  $\mathsf{C}_{R_{\mathrm{a}}} \coloneqq \sup_{\underline{\mathbf{x}} \neq \mathbf{0}} R_{\mathrm{a}}(\underline{\mathbf{x}}) / \| \underline{\mathbf{x}} \|_{\mathrm{F}}$ .
The assumption allows for much broader adversary classes than the commonly considered  $l_{p}$ -attacks [54, 55]. For example, if one treats the multi-channel data  $\underline{\mathbf{x}} \in \mathbb{R}^{d \times 1 \times c}$  as a matrix of dimensionality  $d \times c$  and attacks it with nuclear norm attacks [17], then the constant  $C_{R_{\mathrm{a}}} = \sqrt{\min\{d, c\}}$ .
Given an example-label pair  $(\mathbf{x},y)$ , the adversarial loss for any predictor  $f$  is defined as  $\tilde{\ell}(f(\mathbf{x}),y) = \max_{R_{\mathrm{a}}(\mathbf{x}^{\prime} - \mathbf{x}) \leq \xi} \ell(f(\mathbf{x}^{\prime}),y)$ . The empirical and population adversarial risks are thus defined as  $\hat{\mathcal{L}}^{\mathrm{adv}}(f) := N^{-1} \sum_{i=1}^{N} \tilde{\ell}(f(\mathbf{x}_{i}),y_{i})$  and  $\mathcal{L}^{\mathrm{adv}}(f) := \mathbb{E}_{P_{(\mathbf{x},y)}}[\tilde{\ell}(f(\mathbf{x}),y)]$ , respectively. The adversarial generalization performance is measured by the adversarial generalization gap (AGP) defined as  $\mathcal{L}^{\mathrm{adv}}(f) - \hat{\mathcal{L}}^{\mathrm{adv}}(f)$ . Let  $B_{\tilde{f}} := (B_{x} + \xi C_{R_{\mathrm{a}}}) B_{\underline{\mathbf{W}}}$ . For any  $f \in \mathfrak{F}$ , its AGP is bounded as follows.
Theorem 5 (Adversarial generalization bound for t-NNs). Under Assumptions 1, 2, and 4, there exists a constant  $C$  such that for any  $f \in \mathfrak{F}$ , it holds with probability at least  $1 - 2e^{-t} (\forall t > 0)$ :
$$
\mathcal {L} ^ {\mathrm {a d v}} (f) - \hat {\mathcal {L}} ^ {\mathrm {a d v}} (f) \leq \frac {C L _ {\ell} B _ {\tilde {f}}}{\sqrt {N}} \sqrt {\mathrm {c} \sum_ {l = 1} ^ {L} d _ {l - 1} d _ {l} \log \left(3 (L + 1)\right)} + 3 B \sqrt {\frac {t}{2 N}}. \tag {9}
$$
Remark. When the input example has channel number  $c = 1$  and the attacker uses  $l_p$ -attack, the adversarial generalization bound in Theorem 5 degenerates to the one in Theorem 4 of Ref. [55].
4 Transformed Low-rank Parameterization for Robust Generalization
4.1 Robust Generalization with Exact Transformed Low-rank Parameterization
According to Theorem 5, the AGP bound scales with the square root of the parameter complexity, specifically as  $O(\sqrt{c(\sum_l d_{l-1}d_l) / N})$ . This implies that achieving the desired adversarial accuracy may require a large number  $N$  of training examples. Furthermore, high parameter complexity leads to increased energy consumption, storage requirements, and computational cost when deploying large t-NN models, particularly on resource-constrained embedded and mobile devices.
To this end, we propose a transformed low-rank parameterization scheme to compress the original t-NN models  $\mathfrak{F}$ . Specifically, given a vector of pre-set ranks  $\mathbf{r} = (r_1,\dots ,r_L)^\top \in \mathbb{R}^L$  where  $r_l\leq \min \{d_l,d_{l - 1}\}$ , we consider the following subset of the original t-NNs:
$$
\mathfrak {F} _ {\mathbf {r}} := \left\{f \mid f \in \mathfrak {F}, \text {a n d} r _ {\mathrm {t}} \left(\underline {{\mathbf {W}}} ^ {(l)}\right) \leq r _ {l}, l = 1, \dots , L \right\}. \tag {10}
$$
In the function set  $\mathfrak{F}_{\mathbf{r}}$ , the weight tensor  $\underline{\mathbf{W}}^{(l)}$  of the  $l$ -th layer has the upper bounded tubal rank, which means low-rankness in the transformed domain<sup>5</sup>. We bound the AGP for any  $f \in \mathfrak{F}_{\mathbf{r}}$  as follows.
Theorem 6 (Adversarial generalization bound for t-NNs with transformed low-rank weights). Under Assumptions 1, 2, and 4, there exists a constant  $C'$  such that
$$
\mathcal {L} ^ {\mathrm {a d v}} \left(f _ {\mathbf {r}}\right) - \hat {\mathcal {L}} ^ {\mathrm {a d v}} \left(f _ {\mathbf {r}}\right) \leq \frac {C ^ {\prime} L _ {\ell} B _ {\tilde {f}}}{\sqrt {N}} \sqrt {\mathsf {c} \sum_ {l = 1} ^ {L} r _ {l} \left(d _ {l - 1} + d _ {l}\right) \log (9 (L + 1))} + 3 B \sqrt {\frac {t}{2 N}}, \tag {11}
$$
holds for any  $f_{\mathbf{r}} \in \mathfrak{F}_{\mathbf{r}}$  with probability at least  $1 - 2e^{-t} (\forall t > 0)$ .
For empirical implementations, one can adopt similar rank learning strategy to Ref. [15] to select a suitable rank parameter  $\mathbf{r}$ . Due to the scope of this paper, we leave this for future work.