Title: Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Tensor SVD; Tensor Neural Networks; Transformed Low-rankness; Adversarial Generalization; Implicit Bias.
Abstract: Multi-channel learning has gained significant attention in recent applications, where neural networks with t-product layers (t-NNs) have shown promising performance through novel feature mapping in the transformed domain. 
However, despite the practical success of t-NNs, the theoretical analysis of their generalization remains unexplored. We address this gap by deriving upper bounds on the generalization error of t-NNs in both standard and adversarial settings. Notably, it reveals that t-NNs compressed with exact transformed low-rank parameterization can achieve tighter adversarial generalization bounds compared to non-compressed models. While exact transformed low-rank weights are rare in practice, the analysis demonstrates that through adversarial training with gradient flow, highly over-parameterized t-NNs with the ReLU activation can be implicitly regularized towards a transformed low-rank parameterization under certain conditions. Moreover, this paper establishes sharp adversarial generalization bounds for t-NNs with approximately transformed low-rank weights. Our analysis highlights the potential of transformed low-rank parameterization in enhancing the robust generalization of t-NNs, offering valuable insights for further research and development.

Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks

Abstract

Multi-channel learning has gained significant attention in recent applications, where neural networks with t-product layers (t-NNs) have shown promising performance through novel feature mapping in the transformed domain. However, despite the practical success of t-NNs, the theoretical analysis of their generalization remains unexplored. We address this gap by deriving upper bounds on the generalization error of t-NNs in both standard and adversarial settings. Notably, it reveals that t-NNs compressed with exact transformed low-rank parameterization can achieve tighter adversarial generalization bounds compared to non-compressed models. While exact transformed low-rank weights are rare in practice, the analysis demonstrates that through adversarial training with gradient flow, highly over-parameterized t-NNs with the ReLU activation can be implicitly regularized towards a transformed low-rank parameterization under certain conditions. Moreover, this paper establishes sharp adversarial generalization bounds for t-NNs with approximately transformed low-rank weights. Our analysis highlights the potential of transformed low-rank parameterization in enhancing the robust generalization of t-NNs, offering valuable insights for further research and development.

1 Introduction

Multi-channel learning is a task to extract representations from the data with multiple channels, such as multispectral images, time series, and multi-view videos, in an efficient and robust manner [24, 39, 60, 61]. Among the methods tackling this task, neural networks with t-product layers (t-NNs, see Eq. (5) for a typical example) [12, 36] came to the stage very recently with remarkable efficiency and robustness in various applications such as graph learning, remote sensing, and more [1, 11, 14, 32, 38, 39, 53, 58]. What distinguishes t-NNs from other networks is the inclusion of t-product layers, founded on the algebraic framework of tensor singular value decomposition (t-SVD) [19, 40, 60, 61]. Unlike traditional tensor decompositions, t-SVD explores the transformed low-rankness, i.e., the low-rank structure of a tensor in the transformed domain under an invertible linear transform [18]. The imposed transform in t-product layers provides additional expressivity to neural networks, while

Mingyuan Bai RIKEN AIP

Guoxu Zhou ∗

the controllable transformed low-rank structure in t-NNs enables a flexible balance between model accuracy and robustness [36, 38, 53].

Despite the impressive empirical performance of t-NNs, the theoretical foundations behind their success remain unclear. The lack of systematic theoretical analysis hinders deeper comprehension and exploration of more effective applications and robust performance of t-NNs. Furthermore, the inclusion of the additional transform in t-NNs renders the theoretical analysis more technically challenging compared to existing work on general neural networks [29, 37, 43, 55]. To address this challenge, we establish for the first time a theoretical framework for t-NNs to understand both the standard and robust generalization behaviors, providing both theoretical insights and practical guidance for the efficient and robust utilization of t-NNs. Specifically, we address the following fundamental questions:

Can we theoretically characterize the generalization behavior of general t-NNs? Yes. We derive the upper bounds on the generalization error for general t-NNs in both standard and adversarial settings in Sec. 3.

How does exact transformed low-rankness influence the robust generalization of t-NNs? In Sec. 4.1, our analysis shows that t-NNs with exactly transformed low-rank weights exhibit lower adversarial generalization bounds and require fewer samples, highlighting the benefits of transformed low-rank weights in t-NNs for improved robustness and efficiency.

How does adversarial learning of t-NNs affect the transformed ranks of their weight tensors? In Sec. 4.2, we deduce that weight tensors tend to be of transformed low-rankness approximately for highly over-parameterized t-NNs with ReLU activation under adversarial training with gradient flow.

How is robust generalization impacted by approximately transformed low-rank weight tensors in t-NNs? In Sec. 4.3, we establish sharp adversarial generalization bounds for t-NNs with approximately transformed low-rank weights by carefully bridging the gap with exact transformed low-rank parameterization. This finding again underscores the importance of incorporating transformed low-rank weights as a means to enhance the robustness of t-NNs.

2 Notations and Preliminaries

In this section, we introduce the notations and provide a concise overview of t-SVD and t-NNs, which play a central role in the subsequent analysis.

Notations. We use lowercase, lowercase boldface, and uppercase boldface letters to denote scalars, e.g. , a ∈ R , vectors, e.g. , a ∈ R m , and matrices, e.g. , A ∈ R m × n , respectively. Following the standard notations in Ref. [19], a 3-way tensor of size d × 1 × c is also called a t-vector and denoted by underlined lowercase, e.g. , x , whereas a 3-way tensor of size m × n × c is also called a t-matrix and denoted by underlined uppercase, e.g. , X . We use a t-vector x ∈ R d × 1 × c to represent a multi-channel example, where c denotes the number of channels and d is the number of features for each channel.

Given a matrix A ∈ R m × n , its Frobenius norm (F-norm) and spectral norm are defined as ∥ A ∥ F := √ ∑ min { m,n } i =1 σ 2 i and ∥ A ∥ := max i σ i , respectively, where σ i , i = 1 , · · · , min { m,n } are its singular values. The stable rank of a non-zero matrix A is defined as the squared ratio of its F-norm and spectral norm r stb ( A ) := ∥ A ∥ 2 F / ∥ A ∥ 2 . Given a tensor T , define its l p -norm and F-norm respectively as ∥ T ∥ l p := ∥ vec ( T ) ∥ l p , and ∥ T ∥ F := ∥ vec ( T ) ∥ l 2 , where vec ( · ) denotes the vectorization operation of a tensor [21]. Given T ∈ R m × n × c , let T : , : ,i denote its i th frontal slice. The inner product between two tensors A , B is defined as ⟨ A , B ⟩ := vec ( A ) ⊤ vec ( B ) . The frontal-slice-wise product of two tensors A , B , denoted by A ⊙ B , equals a tensor T such that T : , : ,i = A : , : ,i B : , : ,i , i = 1 , · · · , c [19]. We use | · | as the absolute value for a scalar and cardinality for a set. We use ◦ to denote the function composition operation. Additional notations will be introduced upon their first occurrence.

2.1 Tensor Singular Value Decomposition

The framework of tensor singular value decomposition (t-SVD) is based on the t-product under an invertible linear transform M [18]. In recent studies, the transformation matrix M defining the transform M is restricted to be orthogonal [50] for better properties, which is also followed in this

paper. Given any orthogonal matrix M ∈ R c × c , define the associated linear transform M ( · ) with its inverse M -1 ( · ) on any T ∈ R m × n × c as

where × 3 denotes the tensor matrix product on mode3 [18].

Definition 1 (t-product [18]) . The t-product of any A ∈ R m × n × c and B ∈ R n × k × c under transform M in Eq. (1) is denoted and defined as A ∗ M B = C ∈ R m × k × c such that M ( C ) = M ( A ) ⊙ M ( B ) in the transformed domain. Equivalently, we have C = M -1 ( M ( A ) ⊙ M ( B )) in the original domain.

Definition 2 ( M -block-diagonal matrix) . The M -block-diagonal matrix of any T ∈ R m × n × c , denoted by ˜ T M , is the block diagonal matrix whose diagonal blocks are the frontal slices of M ( T ) :

In this paper, we also follow the definition of t-transpose, t-identity tensor, t-orthogonal tensor, and f-diagonal tensor given by Ref. [18], and thus the t-SVD is introduced as follows.

Definition 3 (t-SVD, tubal rank [18]) . Tensor Singular Value Decomposition (t-SVD) of T ∈ R m × n × c under the invertible linear transform M in Eq. (1) is given as follows

where U ∈ R m × m × c and V ∈ R n × n × c are t-orthogonal, and S ∈ R m × n × c is f-diagonal. The tubal rank of T is defined as the number of non-zero tubes of S in its t-SVD in Eq. (2), i.e., r t ( T ) := |{ i | S ( i, i, :) = 0 , i ≤ min { m,n }}| .

̸

For any T ∈ R m × n × c with the tubal rank r t ( T ) , we have following relationship between its t-SVD and the matrix SVD of its M -block-diagonal matrix [26, 50]:

As the M -block-diagonal matrix ˜ T M is defined after transforming tensor T from the original domain to the transformed domain, the relationship c · r t ( T ) ≥ rank ( ˜ T M ) indicates that the tubal rank can be chosen as a measure of transformed low-rankness [26, 50].

2.2 Neural Networks with t-Product Layer (t-NNs)

In this subsection, we will introduce the formulation of the t-product layer in t-NNs, which is designed for multi-channel feature learning.

Multi-channel feature learning via t-product. Suppose we have a multi-channel example represented by a t-vector x ∈ R d × 1 × c , where c is the number of channels and d is the number of features. We define an L -layer t-NN feature extractor f ( x ) , to extract d L features for each channel of x :

where the l -th layer f ( l ) first conducts t-product with weight tensor (t-matrix) W ( l ) ∈ R d l × d l -1 × c on the output of the ( l -1) -th layer as multi-channel features 2 f ( l -1) ( x ) ∈ R d l -1 × 1 × c to obtain a ( d l × 1 × c )-dimensional representation and then uses the entry-wisely ReLU activation 3 σ ( x ) = max { x, 0 } for nonlinearity.

Remark. Unlike Refs. [36], [32] and [53] whose nonlinear activation is performed in the transformed domain, the t-NN model in Eq. (4) considers the nonlinear activation in the original domain and hence is consistent with traditional neural networks.

By adding a linear classification module with weight w ∈ R cd L after the feature exaction module in Eq. (4), we consider the following t-NN predictor whose sign can be utilized for binary classification:

Let W := { W (1) , · · · , W ( L ) , w } be the collection of all the weights 4 . With a slight abuse of notation, let ∥ W ∥ F := √ ∥ w ∥ 2 2 + ∑ L l =1 ∥ W ( l ) ∥ 2 F denote the Euclidean norm of all the weights. The function class of general t-NNs whose weights are bounded in the Euclidean norm is defined as

with positive constants B w and B l , l = 1 , · · · , L . Let B W := B w ∏ L l =1 B l for simplicity.

3 Standard and Robust Generalization Bounds for t-NNs

This section establishes both the standard and robust generalization bounds for any t-NN f ∈ F .

3.1 Standard Generalization for General t-NNs

Suppose we are given a training multi-channel dataset S consisting of N example-label pairs { ( x i , y i ) } N i =1 ⊂ R d × 1 × c ×{± 1 } i.i.d. drawn from an underlying data distribution P x ,y .

Assumption 1. Every input example x ∈ R d × 1 × c has an upper bounded F-norm, i.e., ∥ x ∥ F ≤ B x , where B x is a positive constant.

When a loss function ℓ ( f ( x i ) , y i ) is considered as the measure of the classification quality, we define the empirical and population risk for any f ∈ F as ˆ L ( f ) := N -1 ∑ N i =1 ℓ ( f ( x i ) , y i ) and L ( f ) := E P ( x ,y ) [ ℓ ( f ( x ) , y )] , respectively. Similar to Ref. [30], we make assumptions on the loss as follows.

Assumption 2. The loss ℓ ( h ( x ) , y ) can be expressed as ℓ ( h ( x ) , y ) = exp( -f ( yh ( x )) for any t-NN h ∈ F , such that:

(A.1) the range of loss ℓ ( · , · ) is [0 , B ] , where B is a positive constant;

(A.2) function f : R → R is C 1 -smooth;

(A.3) f ′ ( x ) ≥ 0 for any x ∈ R ;

(A.4) there exists b f ≥ 0 such that x f ′ ( x ) is non-decreasing for x ∈ ( b f , + ∞ ) , and the derivative x f ′ ( x ) → + ∞ as x → + ∞ ;

(A.5) let g : [ f ( b f ) , + ∞ ) → [ b f , + ∞ ) be the inverse function of f on the domain [ b f , + ∞ ) . There exist b g ≥ max { 2 f ( b f ) , f (2 b f ) } and K ≥ 1 , such that g ′ ( x ) ≤ K g ′ ( θx ) and f ′ ( y ) ≤ K f ′ ( θy ) for any x ∈ ( b g , + ∞ ) , y ∈ ( g ( b g ) , + ∞ ) and θ ∈ [1 / 2 , 1) .

Assumption (A.1) is a natural assumption in generalization analysis [3, 59], and Assumptions (A.2)(A.5) are the same as Assumption (B3) in Ref. [30]. According to Assumption (A.2) , the loss function ℓ ( · , · ) satisfies the L ℓ -Lipschitz continuity

where B ˜ f is an upper bound on the output of any t-NN h ∈ F . The Lipschitz continuity is also widely assumed for generalization analysis of DNNs [55, 59]. Assumption 2 is satisfied by commonly used loss functions such as the logistic loss and the exponential loss.

The generalization gap L ( f ) -ˆ L ( f ) of any function f ∈ F can be bounded as follows.

Lemma 3 (Generalization bound for t-NNs) . Under Assumptions 1 and 2, it holds for any f ∈ F that

with probability at least 1 -2 e -t for any t > 0 .

Remark. When the input example has channel number c = 1 , the generalization bound in Theorem 3 is consistent with the F-norm-based bound in Ref. [8].

3.2 Robust Generalization for General t-NNs

We study the adversarial generalization behavior of t-NNs in this section. We first make the following assumption on the adversarial perturbations.

̸

Assumption 4. Given an input example x , the adversarial perturbation is chosen within a radiusξ ball of norm R a ( · ) with compatibility constant [35] defined as C R a := sup x = 0 R a ( x ) / ∥ x ∥ F .

The assumption allows for much broader adversary classes than the commonly considered l p -attacks [54, 55]. For example, if one treats the multi-channel data x ∈ R d × 1 × c as a matrix of dimensionality d × c and attacks it with nuclear norm attacks [17], then the constant C R a = √ min { d, c } .

Given an example-label pair ( x , y ) , the adversarial loss for any predictor f is defined as ˜ ℓ ( f ( x ) , y ) = max R a ( x ′ -x ) ≤ ξ ℓ ( f ( x ′ ) , y ) . The empirical and population adversarial risks are thus defined as ˆ L adv ( f ) := N -1 ∑ N i =1 ˜ ℓ ( f ( x i ) , y i ) and L adv ( f ) := E P ( x ,y ) [ ˜ ℓ ( f ( x ) , y )] , respectively. The adversarial generalization performance is measured by the adversarial generalization gap (AGP) defined as L adv ( f ) -ˆ L adv ( f ) . Let B ˜ f := ( B x + ξ C R a ) B W . For any f ∈ F , its AGP is bounded as follows.

Theorem 5 (Adversarial generalization bound for t-NNs) . Under Assumptions 1, 2, and 4, there exists a constant C such that for any f ∈ F , it holds with probability at least 1 -2 e -t ( ∀ t > 0) :

Remark. When the input example has channel number c = 1 and the attacker uses l p -attack, the adversarial generalization bound in Theorem 5 degenerates to the one in Theorem 4 of Ref. [55].

4 Transformed Low-rank Parameterization for Robust Generalization

4.1 Robust Generalization with Exact Transformed Low-rank Parameterization

According to Theorem 5, the AGP bound scales with the square root of the parameter complexity, specifically as O ( √ c ( ∑ l d l -1 d l ) /N ) . This implies that achieving the desired adversarial accuracy may require a large number N of training examples. Furthermore, high parameter complexity leads to increased energy consumption, storage requirements, and computational cost when deploying large t-NN models, particularly on resource-constrained embedded and mobile devices.

To this end, we propose a transformed low-rank parameterization scheme to compress the original t-NN models F . Specifically, given a vector of pre-set ranks r = ( r 1 , · · · , r L ) ⊤ ∈ R L where r l ≤ min { d l , d l -1 } , we consider the following subset of the original t-NNs:

In the function set F r , the weight tensor W ( l ) of the l -th layer has the upper bounded tubal rank, which means low-rankness in the transformed domain 5 . We bound the AGP for any f ∈ F r as follows.

Theorem 6 (Adversarial generalization bound for t-NNs with transformed low-rank weights) . Under Assumptions 1, 2, and 4, there exists a constant C ′ such that

holds for any f r ∈ F r with probability at least 1 -2 e -t ( ∀ t > 0) .

Comparing Theorem 6 with Theorem 5, we observe that the adversarial generalization bound under transformed low-rank parameterization has a better scaling, specifically O ( √ c ∑ l r l ( d l -1 + d l ) /N ) . This also implies that a smaller number N of training examples is required to achieve the desired accuracy, as well as reduced energy consumption, storage requirements, and computational cost. Please refer to Sec. A.1 in the appendix for numerical evidence.

4.2 Implicit Bias of Gradient Flow for Adversarial Training of Over-parameterized t-NNs

Although Theorem 6 shows exactly transformed low-rank parameterization leads to lower bounds, the well trained t-NNs on real data rarely have exactly transformed low-rank weights. In this section, we prove that the highly over-parameterized t-NNs, trained by adversarial training with gradient flow (GF), are approximately of transformed low-rank parameterization under certain conditions.

First, the proposed t-NN f ( x ; W ) is said to be (positively) homogeneous as the condition f ( x ; a W ) = a L +1 f ( x ; W ) holds for any positive constant a . Motivated by Ref. [29], we focus on the scale invariant adversarial perturbations defined as follows.

Definition 4 (Scale invariant adversarial perturbation [29]) . An adversarial perturbation δ i ( W ) is said to be scale invariant for f ( x ; W ) at any given example x i if it satisfies δ i ( a W ) = δ i ( W ) for any positive constant a .

Lemma 7. The l 2 -FGM [34], FGSM [9], l 2 -PGD and l ∞ -PGD [31] perturbations for the t-NNs are all scale invariant.

Then, we consider adversarial training of t-NNs with scale invariant adversarial perturbations by GF, which can be seen as gradient descent with infinitesimal step size. When using GF for the ReLU t-NNs, W changes continuously with time, and the trajectory of parameter W during training is an arc W : [0 , ∞ ) → R dim ( W ) , t ↦→ W ( t ) that satisfies the differential inclusion [7, 30]

for t ≥ 0 a.e. , where ∂ ◦ ˆ L adv denotes the Clarke's subdifferential [7] with respect to W ( t ) . If ˆ L adv ( W ) is actually a C 1 -smooth function, the above differential inclusion reduces to

for any t ≥ 0 , which corresponds to the GF with differential in the usual sense. However, for simplicity, we follow Refs. [45, 46] and still use Eq. (13) to denote Eq. (12) with a slight abuse of notation, even if ˆ L adv does not satisfy differentiability but only local Lipschitzness 6 .

We also make an assumption on the training data as follows.

Assumption 8 (Existence of a separability of adversarial examples during training) . There exists a time t 0 such that ˆ L adv ( t 0 ) ≤ N -1 ℓ ( b f ) .

This assumption is a generalization of the separability condition in Refs. [29, 30]. Adversarial training can typically achieve this separability in practice, i.e. , the model can fit adversarial examples of the training dataset, making the above assumption reasonable. Then, we obtain the following lemma.

Lemma 9 (Convergence to the direction of a KKT point) . Consider the hypothesis class F in Eq. (6). Under Assumptions 2 and 8, the limit point of normalized weights { W ( t ) / ∥ W ( t ) ∥ F : t ≥ 0 } of the GF for Eq. (13), i.e., the empirical adversarial risk with scale invariant adversarial perturbations δ i ( W ) , is aligned with the direction of a KKT point of the minimization problem:

Building upon Lemma 9, we can establish that highly over-parameterized t-NNs undergoing adversarial training with GF will exhibit an implicit bias towards transformed low-rank weights.

Theorem 10 (Implicit low-rankness for t-NNs induced by GF) . Suppose there is an example x i satisfying ∥ x i ∥ F ≤ 1 in the training set S = { ( x i , y i ) } N i =1 . Suppose there is a ( J +1) -layer ( J ≥ 2 ) ReLU t-NN, denoted by g ( x ; V ) with parameters V = ( V (1) , · · · , V ( J ) , v ) , satisfying the conditions:

(C.1) the dimensionality of the weight tensor V ( j ) ∈ R m j × m j -1 × c of the j -th t-product layer satisfies m j ≥ 2 , j = 1 , · · · , J ;

(C.2) there is a constant B v > 0 , such that the Euclidean norm of the weights V = ( V (1) , · · · , V ( L ) , v ) satisfy ∥ V ( j ) ∥ F ≤ B v for any j = 1 , · · · , J and ∥ v ∥ 2 ≤ B v ;

(C.3) for all i ∈ { 1 , · · · , N } , we have y i g ( x i + δ i ( V ); V ) ≥ 1 .

Then, we consider the class of over-parameterized t-NNs F = { f ( x ; W ) } defined in Eq. (5) satisfying

(C.4) the number L of t-product layers is much greater than J ;

(C.5) the dimensionality of weight W ( l ) ∈ R d l × d l -1 × c satisfies d l ≫ max j ≤ J { m j } for any l ≤ L .

Let W ∗ = ( W ∗ (1) , · · · , W ∗ ( L ) , w ∗ ) be a global optimum of Problem (14). Namely, W ∗ parameterizes a minimum-norm t-NN f ( x ; W ∗ ) ∈ F that labels the perturbed training set correctly with margin 1 under scale invariant adversarial perturbations. Then, we have

where ˜ W ∗ ( l ) M denotes the M -block-diagonal matrix of weight tensor W ∗ ( l ) for any l = 1 , · · · , L .

By the above theorem, when L is sufficiently large, the harmonic mean of the square root of the stable rank of ˜ W ∗ ( l ) M , i.e. , the M -block-diagonal matrix of weight tensor W ∗ ( l ) , is approximately bounded by √ c m J , which is significantly smaller than the square root of the dimensionality √ min { c d l , c d l -1 } according to condition (C.5) in Theorem 10. Thus, f ( x ; W ∗ ) has a nearly low-rank parameterization in the transformed domain. In our case, the weights W ( t ) generated by GF tend to have an infinite norm and to converge in direction to a transformed low-rank solution. Moreover, note that the ratio between the spectral norm and the F-norm is invariant to scaling, and hence it suggests that after a sufficiently long time, GF tends to reach a t-NN with transformed low-rank weight tensors. Refer to Sec. A.2 for numerical evidence supporting Theorem 10.

4.3 Robust Generalization with Approximate Transformed Low-rank Parameterization

Theorem 10 establishes that for highly over-parameterized adversarial training with GF, well-trained tNNs exhibit approximately transformed low-rank parameters under specific conditions. In this section, we analyze the AGP of t-NNs that possess an approximately transformed low-rank parameterization 7 .

Initially, by employing low-tubal-rank tensor approximation [20], one can always compress an approximately low-tubal-rank parameterized t-NN f by a t-NN g ∈ F r with an exact low-tubal-rank parameterization, ensuring a small distance between g and f in the parameter space. Now, the question is: Can the small parametric distance between f and g also indicate a small difference in their adversarial generalization behaviors? To answer this question, we first define the ( δ, r ) -approximate low-tubal-rank parameterized functions.

Definition 5 ( ( δ, r ) -approximate low-tubal-rank parameterization) . A t-NN f ( x ; W ) ∈ F with weights W = ( w , W (1) , · · · , W ( L ) ) is said to satisfy the ( δ, r ) -approximate low-tubal parameterization with tolerance δ > 0 and rank r = ( r 1 , · · · , r L ) ⊤ ∈ N L , if there is a t-NN g ( x ; Wr ) ∈ F r whose weights W g = ( w , W (1) r 1 , · · · , W ( L ) r L ) satisfy ∥ W ( l ) r l -W ( l ) ∥ F ≤ δ for any l = 1 , · · · , L .

Furthermore, let's consider the collection of t-NNs with approximately low-tubal-rank weights

F δ, r := { f ∈ F | f satisfies the ( δ, r ) -approximate low-tubal-rank parameterization } . (15)

Subsequently, we analyze the AGP for any f ∈ F δ, r in terms of its low-tubal-rank compression g ∈ F r . The idea is motivated by the work on compressed bounds for non-compressed but compressible models [43], originally developed for generalization analysis of NNs for standard training.

adv ˜ ′

Under Assumption 2, we first define F δ, r := { f : ( x , y ) ↦→ min R a ( x ′ -x ) ≤ ξ yf ( x ) | f ∈ F δ, r } as the adversarial version of F δ, r . To analyze the AGP of f ∈ F δ, r through g ∈ F r , we instead consider their adversarial counterparts ˜ f ∈ F adv δ, r and ˜ g ∈ F adv r , where F adv r is defined as F adv r := { ˜ g : ( x , y ) ↦→ min R a ( x -x ′ ) ≤ ξ yg ( x ′ ) ∣ ∣ g ∈ F r } . Define the Minkowski difference of F adv δ, r and F adv r as F adv δ, r -F adv r := { ˜ f -˜ g | ˜ f ∈ F adv δ, r , ˜ g ∈ F adv r } . The empirical L 2 -norm of a t-NN h ∈ F on the training data S = { ( x i , y i ) } N i =1 is defined as ∥ h ∥ S := √ N -1 ∑ N i =1 h 2 ( x i , y i ) , and the population L 2 -norm is ∥ h ∥ L 2 := √ E P ( x ,y ) [ h 2 ( x , y )] . Define the local Rademacher complexity of F adv δ, r -F adv r of radius r > 0 as ˙ R r ( F adv δ, r -F adv r ) := ¯ R N ( { h ∈ F adv δ, r -F adv r | ∥ h ∥ L 2 ≤ r } ) , where ¯ R N ( H ) denotes the average Rademacher complexity of a function class H [4].

The first part of the upcoming Theorem 12 shows that a small parametric distance between f and g leads to a small empirical L 2 -distance in the adversarial output space. Specifically, for any f ( x ; W ) ∈ F δ, r with compression g ( x ; Wr ) , their (adversarial) empirical L 2 -distance ∥ ˜ f ( x ; W ) -˜ g ( x ; Wr ) ∥ S can be bounded by a small constant ˆ r > 0 in linearity of δ . We also aim for a small population L 2 -distance by first assuming the local Rademacher complexity ˙ R r ( F adv δ, r -F adv r ) can be bounded by a concave function of r , following common practice in Rademacher complexity analysis [4, 43].

Assumption 11. For any r > 0 , there exists a function ϕ ( r ) : [0 , ∞ ) → [0 , ∞ ) such that ˙ R r ( F adv δ, r -F adv r ) ≤ ϕ ( r ) and ϕ (2 r ) ≤ 2 ϕ ( r ) .

We further define r ∗ = r ∗ ( t ) := inf { r > 0 ∣ ∣ 16 B ˜ f r -2 ϕ ( r ) + B ˜ f r -1 √ 2 t/N +2 tB 2 ˜ f r -2 /N ≤ 1 / 2 } for any t > 0 , such that the population L 2 -norm of any h ∈ F adv δ, r -F adv r can be bounded by ∥ h ∥ 2 L 2 ≤ 2( ∥ h ∥ 2 S + r 2 ∗ ) using the peeling argument [42, Theorem 7.7]. We then establish an adversarial generalization bound for approximately low-tubal-rank t-NNs as follows.

Theorem 12 (Adversarial generalization bound for general approximately low-tubal-rank t-NNs) . (I). For any f ∈ F δ, r with adversarial proxy ˜ f ∈ F adv δ, r , there exists a function g ∈ F r with adversarial proxy ˜ g ∈ F adv r , such that the empirical L 2 -distance ∥ ˜ f -˜ g ∥ S ≤ δB ˜ f ∑ L l =1 B -1 l =: ˆ r . (II). Let ˙ r := √ 2(ˆ r 2 + r 2 ∗ ) . Under Assumptions 1, 2, 4, 11, there exist constants C 1 , C 2 > 0 satisfying

for any f ∈ F δ, r with probability at least 1 -4 e -t for any t > 0 , where Φ( r ) is defined as

The main term of the bound quantifies the complexity of functions in F r with exact low-tubal-rank parameterization in adversarial settings, which can be significantly smaller than that of F δ, r . On the other hand, the bias term captures the sample complexity required to bridge the gap between approximately low-tubal-rank parameterized F δ, r and exactly low-tubal-rank parameterized F r . As we usually observe r 2 ∗ = o (1 / √ N ) , setting ˆ r = o p (1) allows the bias term to decay faster than the main term, which is O (1 / √ N ) . Theorem 12 suggests that a small parametric distance between f ∈ F δ, r and g ∈ F r also implies a small difference in their adversarial generalization behaviors .

A special case. We also showcase a specific scenario where the weights of t-product layers exhibit a polynomial spectral decay in the transformed domain, leading to a considerably small AGP bound.

Assumption 13. Consider the setting where any t-NN f ( x ; W ) ∈ F δ, r has tensor weights W ( l ) ( l = 1 , · · · , L ) whose singular values in the transformed domain satisfy σ j ( M ( W ( l ) ) : , : ,k ) ≤ V 0 · j -α , where V 0 > 0 is a constant, and σ j ( · ) is the j -th largest singular value of a matrix.

Under Assumption 13, the weight tensor W ( l ) can be approximated by its optimal tubal-rankr l approximation W ( l ) r l for any 1 ≤ r l ≤ min { d l , d l -1 } with error ∥ W ( l ) -W ( l ) r l ∥ F ≤ √ c / (2 α -1) V 0 ( r l -1) (1 -2 α ) / 2 [20], which can be much smaller than ∥ W ( l ) r l ∥ F when α > 1 / 2 is sufficiently large. Thus, we can find an exactly low-tubal-rank parameterized g ∈ F r for any f ∈ F δ, r satisfying Assumption 13, such that the parametric distance between g and f is quite small. The following theorem shows that the small parametric distance also leads to a small AGP.

Theorem 14. Under Assumptions 1, 2, 4, and 13, if we let ˆ r = V 0 B ˜ f ∑ L l =1 ( r l +1) -α B -1 l , then for any t-NN f ∈ F δ, r , there exists a function g ∈ F r whose t-product layer weights have tubal-rank exactly no greater than r l , satisfying ∥ ˜ f -˜ g ∥ S ≤ ˆ r . Further, there is a constant C α only depending on α such that the AGP, i.e., L adv ( f ) -ˆ L adv ( f ) , of any f ∈ F δ, r can be upper bounded by

for any t > 0 with probability at least 1 -4 e -t , where E 1 = N -1 c ∑ L l =1 r l ( d l + d l -1 ) log(9 NLB ˜ f / √ c ) and E 2 = N -1 c ∑ L l =1 ( LV 0 B ˜ f B -1 l ) 1 /α ( d l + d l -1 ) log(9 NLB ˜ f / √ c ) .

This suggests that by choosing a sufficiently large α > 1 / 2 , where each weight tensor has a tubal-rank close to 1, we can attain a superior generalization error bound. It is important to note that the rank r l can be arbitrarily chosen, and there exists a trade-off relationship between ˆ r and E 1 . Therefore, by selecting the rank appropriately for a balanced trade-off, we can obtain an optimal bound as follows.

Corollary 15. Under the same assumption to Theorem 14, if we choose the parameter r of tubal ranks in F r by r l = min {⌈ ( LV 0 B ˜ f B -1 l ) 1 /α ⌉ , d l , d l -1 } , then there is a constant C α only depending on α such that the AGP of any f ∈ F δ, r can be upper bounded as

with probability at least 1 -4 e -t for any t > 0 .

It is worth highlighting that the bound exhibits a linear dependency on the number of neurons in the t-product layers, represented as O ( √ c ∑ l ( d l + d l -1 ) /N ) . In contrast, Theorem 5 demonstrates a dependency on the total number of parameters, denoted as O ( √ c ∑ l d l d l -1 /N ) . This observation suggests that employing the low-tubal-rank parameterization can potentially enhance adversarial generalization for t-NNs.

5 Related Works

T-SVD-based data and function representation. The unique feature of t-SVD-based data representation, in contrast to classical low-rank decomposition methods, is the presence of low-rankness in the transformed domain. This transformed low-rankness is crucial for effectively modeling real multi-channel data with both smoothness and low-rankness [24, 49, 50]. Utilized in t-product layers in DNNs [32, 36, 53], t-SVD has also been a workhorse for function representation and achieves impressive empirical performance. While t-SVD-based signal processing models have been extensively studied theoretically [13, 24, 27, 40, 50, 60], the t-SVD-based learning model itself has not been thoroughly scrutinized until this paper. Hence, this study represents the first theoretical analysis of t-SVD-based learning models, contributing to the understanding of their theoretical foundations.

Theoretical analysis methods. Our analysis draws on norm-based generalization analysis [37] and implicit regularization of gradient descent-based learning [46] as related theoretical analysis methods. Norm-based generalization analysis plays a crucial role in theoretical analysis across various domains, including standard generalization analysis of DNNs [8], compressed models [22], non-compressed models [43], and adversarial generalization analysis [3, 55, 59]. Our work extends norm-based tools to analyze both standard and adversarial generalization in t-NNs, going beyond the traditional use of matrix products. For implicit regularization of gradient descent based learning, extensive past research has been conducted on implicit bias of GF for both standard and adversarial training of homogeneous networks building on matrix product layers, respectively [16, 30, 45]. We non-trivially extend these methods to analyze t-NNs and reveals that GF for over-parameterized ReLU t-NNs produces nearly transformed low-rank weights under scale invariant adversarial perturbations.

Our theoretical results notably deviate from the standard error bounds for fully connected neural networks (FNNs) in several ways:

The generalization bounds in Lemma 3 and Theorem 5 for t-NNs diverge from their counterparts for FNNs in Refs. [8, 55, 59] due to the channel number c in t-NNs. Moreover, Theorem 5 encompasses a wider range of adversary classes than the l p -attacks in the aforementioned references.

The uniqueness of Theorem 6, compared to Refs. [55, 59], stems from its consideration of weight low-rankness in the adversarial generalization bound, suggesting possible robustness improvements in generalization.

Our exploration of the implicit bias in GF for adversarial training presents a novel angle: the bias towards approximate transformed low-rankness in t-NNs. While Ref. [29] focuses on the implicit bias in adversarial training for FNNs, centered on KKT point convergence with exponential loss, our work delves deeper, considering a wider array of loss functions in adversarial training for t-NNs.

A crucial distinction in our adversarial generalization bounds, detailed in Section 4.3, from nonadversarial bounds for FNNs [43] is the integration of the localized Rademacher complexity. This encompasses the Minkowski difference between adversarial counterparts of both approximately and exactly low-tubal-rank t-NNs as seen in Theorem 12.

6 Concluding Remarks

A thorough investigation of the generalization behavior of t-NNs is conducted for the first time. We derive upper bounds for the generalization gaps of standard and adversarially trained t-NNs and propose compressing t-NNs with a transformed low-rank structure for more efficient adversarial learning and tighter bounds on the adversarial generalization gap. Our analysis shows that adversarial training with GF in highly over-parameterized settings results in t-NNs with approximately transformed low-rank weights. We further establish sharp adversarial generalization bounds for t-NNs with approximately transformed low-rank weights. Our findings demonstrate that utilizing the transformed low-rank parameterization can significantly enhance the robust generalization of t-NNs, carrying both theoretical and empirical significance.

Limitations. While this paper adheres to the norm-based framework for capacity control [8, 37], it is worth noting that the obtained generalization bounds may be somewhat conservative. However, this limitation can be mitigated by employing more sophisticated analysis techniques, as evidenced by recent studies [2, 25, 56, 57].

Discussions. The inclination of adversarial training towards low-rank/sparse weights, and the reciprocal effects of parameter reduction on robustness, are currently at the forefront of ongoing research. This domain has witnessed a spectrum of observations and results [6, 23, 41, 51]. In this study, we propose that employing low-rank parameterization can enhance the adversarial robustness of t-NNs, as evidenced by our analysis of uniform adversarial generalization error bounds. However, despite these promising results, it is crucial to emphasize the necessity of a more exhaustive exploration of low-rank parameterization. Its implications, particularly when considered in the context of approximation, estimation, and optimization, are profound and warrant further dedicated research efforts. Such a comprehensive investigation will undoubtedly enhance our understanding and fully unlock the potential of low-rank parameterization in neural networks.

Acknowledgments

We extend our deepest gratitude to Linfeng Sui and Xuyang Zhao for their indispensable support in implementing the Python code for t-NNs during the rebuttal phase. Our sincere appreciation also goes to both the area chair and reviewers for their unwavering dedication and meticulous attention given to this paper. This research was supported by RIKEN Incentive Research Project 100847-202301062011, by JSPS KAKENHI Grant Numbers JP20H04249 and JP23H03419, and in part by National Natural Science Foundation of China under Grants 62103110 and 62073087.

References

J. An, W. Liu, Q. Liu, L. Guo, P. Ren, and T. Li. Dginet: Dynamic graph and interaction-aware convolutional network for vehicle trajectory prediction. Neural Networks , 151:336-348, 2022. 1

S. Arora, R. Ge, B. Neyshabur, and Y. Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning , pages 254-263. PMLR, 2018. 6

P. Awasthi, N. Frank, and M. Mohri. Adversarial learning guarantees for linear hypotheses and neural networks. In International Conference on Machine Learning , pages 431-441. PMLR, 2020. 3.1, 5, C.2, C.3

P. L. Bartlett, O. Bousquet, and S. Mendelson. Localized rademacher complexities. In Computational Learning Theory: 15th Annual Conference on Computational Learning Theory, COLT 2002 Sydney, Australia, July 8-10, 2002 Proceedings 15 , pages 44-58. Springer, 2002. 4.3, 29

S. Boucheron, G. Lugosi, and P. Massart. Concentration inequalities: A nonasymptotic theory of independence . Oxford university press, 2013. C.1

J. Cosentino, F. Zaiter, D. Pei, and J. Zhu. The search for sparse, robust neural networks. arXiv preprint arXiv:1912.02386 , 2019. 6

J. Dutta, K. Deb, R. Tulshyan, and R. Arora. Approximate KKT points and a proximity measure for termination. Journal of Global Optimization , 56(4):1463-1499, 2013. 4.2, 4.2, D.1, 21

N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory , pages 297-299. PMLR, 2018. 3.1, 5, 6, 6, C.1

I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations , 2015. 7, A.1, D, D

F. Graf, S. Zeng, M. Niethammer, and R. Kwitt. On measuring excess capacity in neural networks. arXiv preprint arXiv:2202.08070 , 2022. 34

F. Han, Y. Miao, Z. Sun, and Y. Wei. T-ADAF: Adaptive data augmentation framework for image classification network based on tensor t-product operator. Neural Processing Letters , 2023. 1

L. Horesh, E. Newman, M. E. Kilmer, and H. Avron. Generating and managing deep tensor neural networks, Dec. 20 2022. US Patent 11,531,902. 1

J. Hou, F. Zhang, H. Qiu, J. Wang, Y. Wang, and D. Meng. Robust low-tubal-rank tensor recovery from binary measurements. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021. 5

Z. Huang, X. Li, Y. Ye, and M. K. Ng. Mr-gcn: Multi-relational graph convolutional networks based on generalized tensor product. In IJCAI , volume 20, pages 1258-1264, 2020. 1

Y. Idelbayev and M. A. Carreira-Perpinán. Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8049-8059, 2020. 5

Z. Ji and M. Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems , 33:17176-17186, 2020. 5, D

E. Kazemi, T. Kerdreux, and L. Wang. Trace-norm adversarial examples. arXiv preprint arXiv:2007.01855 , 2020. 3.2

E. Kernfeld, M. Kilmer, and S. Aeron. Tensor-tensor products with invertible linear transforms. Linear Algebra and its Applications , 485:545-570, 2015. 1, 2.1, 2.1, 1, 2.1, 3, 6, 7, 8

M. E. Kilmer, K. Braman, et al. Third-order tensors as operators on matrices: A theoretical and computational framework with applications in imaging. SIAM J MATRIX ANAL A , 34(1):148172, 2013. 1, 2, 9

M. E. Kilmer, L. Horesh, H. Avron, and E. Newman. Tensor-tensor algebra for optimal representation and compression of multiway data. Proceedings of the National Academy of Sciences , 118(28):e2015851118, 2021. 4.3, 4.3, E.2, E.2, F.1, F.1

T. G. Kolda and B. W. Bader. Tensor decompositions and applications. SIAM Review , 51(3):455500, 2009. 2

J. Li, Y . Sun, J. Su, T. Suzuki, and F. Huang. Understanding generalization in deep learning via tensor methods. In International Conference on Artificial Intelligence and Statistics , pages 504-515. PMLR, 2020. 5

N. Liao, S. Wang, L. Xiang, N. Ye, S. Shao, and P. Chu. Achieving adversarial robustness via sparsity. Machine Learning , pages 1-27, 2022. 6

X. Liu, S. Aeron, V. Aggarwal, and X. Wang. Low-tubal-rank tensor completion using alternating minimization. IEEE TIT , 66(3):1714-1737, 2020. 1, 5

S. Lotfi, M. Finzi, S. Kapoor, A. Potapczynski, M. Goldblum, and A. G. Wilson. PACBayes compression bounds so tight that they can explain generalization. Advances in Neural Information Processing Systems , 35:31459-31473, 2022. 6

C. Lu. Transforms based tensor robust pca: Corrupted low-rank tensors recovery via convex optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1145-1152, 2021. 2.1, 2.1

C. Lu, J. Feng, W. Liu, Z. Lin, S. Yan, et al. Tensor robust principal component analysis with a new tensor nuclear norm. IEEE TPAMI , 2019. 5

C. Lu, X. Peng, and Y. Wei. Low-rank tensor completion with a new tensor nuclear norm induced by invertible linear transforms. In CVPR , pages 5996-6004, 2019. 10

B. Lv and Z. Zhu. Implicit bias of adversarial training for deep neural networks. In International Conference on Learning Representations , 2022. 1, 4.2, 4, 4.2, 5, 6, A.2, D, D, D.1

K. Lyu and J. Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations , 2020. 3.1, 3.1, 4.2, 4.2, 5, A.2, D, D, 19, D.1, 20, 10, D.1, 21, D.2, 11, D.2, 23, 24

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018. 7, D, D, D

O. A. Malik, S. Ubaru, L. Horesh, M. E. Kilmer, and H. Avron. Dynamic graph convolutional networks using the tensor M -product. In Proceedings of the 2021 SIAM international conference on data mining (SDM) , pages 729-737. SIAM, 2021. 1, 2.2, 5

O. L. Mangasarian and S. Fromovitz. The fritz john necessary optimality conditions in the presence of equality and inequality constraints. Journal of Mathematical Analysis and applications , 17(1):37-47, 1967. D.1

T. Miyato, A. M. Dai, and I. Goodfellow. Adversarial training methods for semi-supervised text classification. In International Conference on Learning Representations , 2017. 7, D

S. Negahban, B. Yu, M. J. Wainwright, and P. K. Ravikumar. A unified framework for highdimensional analysis of M-estimators with decomposable regularizers. In Proceedings of Advances in Neural Information Processing Systems , pages 1348-1356, 2009. 4

E. Newman, L. Horesh, H. Avron, and M. Kilmer. Stable tensor neural networks for rapid deep learning. arXiv preprint arXiv:1811.06569 , 2018. 1, 2.2, 5, B.2

B. Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory , pages 1376-1401. PMLR, 2015. 1, 5, 6, C.1

F. Qian, Z. Liu, Y. Wang, S. Liao, S. Pan, and G. Hu. Dtae: Deep tensor autoencoder for 3-D seismic data interpolation. IEEE Transactions on Geoscience and Remote Sensing , 60:1-19, 2021. 1

F. Qian, Z. Liu, Y. Wang, Y. Zhou, and G. Hu. Ground truth-free 3-D seismic random noise attenuation via deep tensor convolutional neural networks in the time-frequency domain. IEEE Transactions on Geoscience and Remote Sensing , 60:1-17, 2022. 1

H. Qiu, Y. Wang, S. Tang, D. Meng, and Q. Yao. Fast and provable nonconvex tensor RPCA. In International Conference on Machine Learning , pages 18211-18249. PMLR, 2022. 1, 5

D. Savostianova, E. Zangrando, G. Ceruti, and F. Tudisco. Robust low-rank training via approximate orthonormal constraints. arXiv preprint arXiv:2306.01485 , 2023. 6

I. Steinwart and A. Christmann. Support vector machines . Springer Science & Business Media, 2008. 4.3, 9, E, 35

T. Suzuki, H. Abe, and T. Nishimura. Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network. In International Conference on Learning Representations , 2020. 1, 4.3, 5, 6, E, 27

M. Talagrand. A new look at independence. The Annals of Probability , pages 1-34, 1996. 30

N. Timor, G. Vardi, and O. Shamir. Implicit regularization towards rank minimization in ReLU networks. In The 34th International Conference on Algorithmic Learning Theory , 2023. 4.2, 5

G. Vardi and O. Shamir. Implicit regularization in ReLU networks with the square loss. In Conference on Learning Theory , pages 4224-4258. PMLR, 2021. 4.2, 5

R. Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018. E, E.1, 31, 32

A. Wang, Z. Jin, and G. Tang. Robust tensor decomposition via t-SVD: Near-optimal statistical guarantee and scalable algorithms. Signal Processing , 167:107319, 2020. A.3

A. Wang, C. Li, Z. Jin, and Q. Zhao. Robust tensor decomposition via orientation invariant tubal nuclear norms. In AAAI , pages 6102-6109, 2020. 5

A. Wang, G. Zhou, Z. Jin, and Q. Zhao. Tensor recovery via ∗ L -spectral k -support norm. IEEE Journal of Selected Topics in Signal Processing , 15(3):522-534, 2021. 2.1, 2.1, 2.1, 7, 5

L. Wang, G. W. Ding, R. Huang, Y. Cao, and Y. C. Lui. Adversarial robustness of pruned neural networks. 2018. 6

Z. Wang, J. Dong, X. Liu, and X. Zeng. Low-rank tensor completion by approximating the tensor average rank. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4612-4620, 2021. 7

Z. Wu, L. Shu, Z. Xu, Y. Chang, C. Chen, and Z. Zheng. Robust tensor graph convolutional networks via t-SVD based graph augmentation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2090-2099, 2022. 1, 2.2, 5

D. Xia and M. Yuan. On polynomial time methods for exact low rank tensor completion. arXiv preprint arXiv:1702.06980 , 2017. 3.2

J. Xiao, Y. Fan, R. Sun, and Z.-Q. Luo. Adversarial Rademacher complexity of deep neural networks. arXiv preprint arXiv:2211.14966 , 2022. 1, 3.1, 3.2, 3.2, 5, 6, C.2, C.3

J. Xiao, Y. Fan, R. Sun, J. Wang, and Z.-Q. Luo. Stability analysis and generalization bounds of adversarial training. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. 6

J. Xiao, R. Sun, and Z.-Q. Luo. PAC-bayesian spectrally-normalized bounds for adversarially robust generalization. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. 6

J. Yang, C. Fu, F. Deng, M. Wen, X. Guo, and C. Wan. Toward interpretable graph tensor convolution neural network for code semantics embedding. ACM Transactions on Software Engineering and Methodology , 2023. 1

D. Yin, R. Kannan, and P. Bartlett. Rademacher complexity for adversarially robust generalization. In International Conference on Machine Learning , pages 7085-7094. PMLR, 2019. 3.1, 3.1, 5, 6

X. Zhang and M. K. Ng. Sparse nonnegative tensor factorization and completion with noisy observations. IEEE Transactions on Information Theory , 68(4):2551-2572, 2022. 1, 5

X. Zhang and M. K.-P. Ng. Low rank tensor completion with poisson observations. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021. 1

Appendix

Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks

In the appendix, we begin by presenting numerical evaluations of our theoretical findings. Subsequently, we introduce the additional notations and preliminaries related to t-SVD, followed by the proofs of the propositions mentioned in the main text.

Our analysis and proofs pertaining to t-NNs differ from those for FNNs as follows:

Firstly, unlike the analysis for FNNs' generalization bounds based on Rademacher complexity in Refs. [8, 55, 59], we derive specific lemmas for standard and adversarial generalization bounds in t-NNs. This is due to the unique structure of (low-rank) t-product layers. We reformulate the t-product through an operator-like expression in Lemma 16, paving the way for pivotal Lemma 17, supporting the t-product-based 'peeling argument.' Additionally, we introduce Lemmas 37, 38, and Lemma 33 to handle t-product layer output norms and covering low-tubal-rank tensors.

Secondly, proving the implicit bias of GF for adversarial training of t-NNs, specifically the approximately transformed low-rankness, is nontrivial in comparison to the proof in Ref. [29] for the implicit bias of adversarial training for FNNs. As we consider more general loss functions for t-NNs in contrast to the exponential loss for FNNs in Ref. [29], we first derive a more general convergence result to the direction of a KKT point for t-NNs Lemma 9, and then goes deeper by using a constructive approach to establish the approximately transformed low-rankness in Theorem 10.

Thirdly, differing from Ref. [43] which focuses on standard FNN generalization, our approach delves into t-NNs' adversarial generalization. We achieve this by introducing the ( δ, r ) -parameterization, bounding localized Rademacher complexity for a Minkowski set in adversarial settings, and using low-tubal-rank approximations for tensor weights.

Contents

A Numerical Evaluations of the Theoretical Results

This section presents numerical evaluations for our theoretical results. All training process is conducted on nVidia A100 GPU. For additional information and access to the demo code, please visit the following URL: https://github.com/pingzaiwang/Analysis4TNN/ .

A.1 Effects of Exact Transformed Low-rank Weights on the Adversarial Generalization Gap

To validate the adversarial generalization bound in Theorem 6, we have conducted experiments on the MNIST dataset to explore the relationship between adversarial generalization gaps (AGP), weight tensor low-rankness, and training sample size. We consider binary classification of 3 and 7, with FSGM [9] attacks of strength 20 / 255 . The t-NN consists of three t-product layers and one FC layer, with weight tensor dimensions of 28 × 28 × 28 for W (1) , W (2) , and W (3) , and 784 for the FC weight w . As an input to the t-NN, each MNIST image of size 28 × 28 is treated as a t-vector of size 28 × 1 × 28 .

(a)

(b)

0.014

t-NN without low-rankweights

0.014

t-NN without low-rankweights

GeneralizationGap

t-NN with low-rank weights (r=4)

Adversarial GeneralizationGap

t-NN with low-rank weights (r=4)

0.012

0.012

0.010

0.010

0.008

0.008

Adversarial

0.006

0.006

0.004

0.004

0.002

0.002

0.000

0.000

1000

2000

3000

4000

5000

0.015

0.020

0.025

0.030

0.035

0.040

0.045

Sample size (N)

1/√N

Theorem 6 emphasizes: ( i ) lower weight tensor rank leads to smaller bounds on the adversarial generalization gaps, and ( ii ) the bound diminishes at a rate of O (1 / √ N ) as N increases. We explored this by conducting experiments, controlling the upper bounds of the tubal-rank to 4 and 28 for low and full tubal-rank cases, and systematically increasing the number of training samples.

Fig. 1 presents the results. The curves indicate that t-NNs with lower rank weight tensors have smaller robust generalization errors. Interestingly, the adversarial generalization errors seem to follow a linear relationship with 1 / √ N , approximately validating the generalization error bound in Theorem 6 by approximating the scaling behavior of the empirical errors.

A.2 Implicit Bias of GF-based Adversarial Training to Approximately Transformed Low-rank Weight Tensors

We carried out experiments to confirm two theoretical statements related to the analysis of GF-based adversarial training.

Statement A.2.1 Theorem 10 reveals that, under specific conditions, well-trained t-NNs with highly over-parameterized adversarial training using GF show nearly transformed low-rank parameters.

Statement A.2.2 Lemma 22 asserts that the empirical adversarial risk approaches zero, and the F-norm of the weights grows infinitely as t approaches infinity.

In continuation of the experimental settings in Sec. A.1, we focus on binary classification on MNIST under FGSM attacks. The t-NN is structured with three t-product layers and one FC layer, with weight dimensions set to D × 28 × 28 for W (1) , D × D × 28 for W (2) and W (3) , and 28 D for the FC weight w . Our experiments involve setting values of D to 128 and 256 , respectively, and we track

the effective rank of each weight tensor, the empirical adversarial risk, and the F-norm of the weights as the number of epochs progresses. Since implementing gradient flow with infinitely small step size is impractical in real experiments, we opt for SGD with a constant learning rate and batch-size of 80 , following the setting on fully connected layers in Ref. [29].

For Statement A.2.1, we present preliminary results illustrating the progression of the stable ranks of the M -block-diagonal matrix of tensor weights in Fig. 2 for the settings D ∈ { 128 , 256 } . Notably, these results show that the effective ranks decrease as more epochs are executed, thereby confirming the influence of implicit bias on transformed low-rankness, as described in Statement A.2.1.

For Statement A.2.2, we present initial numerical findings depicting the progress of the empirical adversarial risk and the F-norm of the weights in Figs. 3 and 4, respectively. These results exhibit a consistent pattern with the theoretical descriptions outlined in Statement A.2.2 and the numerical results reported in Ref. [29] for adversarial training and Ref. [30] for standard training of FNNs. Specifically, we observe a decreasing trend in the empirical risk function and an increasing trend in the weight tensor's F-norm, which align with the expected behavior based on our theoretical framework and corroborate the numerical results presented in Refs. [29, 30].

(a)

(b)

D=128

D=256

t-productlayer1

t-productlayer1

60

Stablerankof weighttensors

t-productlayer2

Stablerankofweighttensors

120

t-productlayer2

50

t-productlayer3

t-productlayer3

100

40

80

30

60

20

40

10

20

0

0

0

2000

4000

6000

8000

10000

12000

0

2500

5000

7500

10000125001500017500

Epoch

Epoch

(a)

(b)

D=128

D=256

0.008

0.008

empirica

0.006

empirica

0.006

Adversarial

0.004

Adversarial

0.004

0.002

0.002

0.000

0.000

0

2000

4000

6000

8000

10000

12000

0

2500

5000

750010000125001500017500

Epoch

Epoch

(a)

(b)

D=128

D=256

t-productlayer1

6.80

t-productlayer2

9.45

t-productlayer3

6.75

fweights

F-Norm of weights

9.40

6.70

F-Norm of

9.35

6.65

6.60

9.30

t-productlayer1

t-productlayer2

6.55

9.25

t-productlayer3

0

2000

4000

6000

8000

10000

12000

0

2500

5000

7500

10000125001500017500

Epoch

Epoch

A.3 Additional Regularization for a Better Low-rank Parameterized t-NN

It is natural to ask: instead of using adversarial training with GF in highly over-parameterized settings to train a approximately transformed low-rank t-NN, is it possible to apply some extra regularizations in training to achieve a better low-rank parameterization ?

Yes, it is possible to apply additional regularizations during training to achieve a better low-rank representation in t-NNs. Instead of relying solely on adversarial training with gradient flow in highly over-parameterized settings, these extra regularizations can potentially promote and enforce low-rankness in the network.

To validate the concern regarding the addition of an extra regularization term, we performed a preliminary experiment. In this experiment, we incorporated the tubal nuclear norm [48] as an explicit regularizer to induce low-rankness in the transformed domain. Specifically, we add the tubal nuclear norm regularization to the t-NN with three t-product layer D = 128 in Sec. A.2 with a regularization parameter 0 . 01 , and keep the other settings the same as Sec. A.2. We explore how the stable ranks of tensor weights evolve with the epoch number with/without tubal nuclear norm regularization.

The experimental results are depicted in Fig. 5. According to Fig. 5, it becomes evident that the introduction of the explicit low-rank regularizer significantly enforced low-rankness in the transform domain of the weight tensors.

D=128

t-productlayer1

ofweighttensors

60

t-product layer2

t-productlayer3

50

t-product layer1(regularized)

t-productlayer2(regularized)

40

t-product layer3(regularized)

30

Stablerank

20

10

0

0

50

100

150

200

250

300

350

400

Epoch

B Notations and Preliminaries of t-SVD

B.1 Notations

For simplicity, we use c , c 0 , C etc. to denote constants whose values can vary from line to line. We use We first give the most commonly used notations in Table 1.

B.2 Additional Preliminaries of t-SVD

We give additional notions and propositions about t-SVD omitted in the main body of the paper.

Definition 6 ([18]) . The t-transpose of T ∈ R m × n × c under the M transform in Eq. (1), denoted by T ⊤ , satisfies

Definition 7 ([18]) . The t-identity tensor I ∈ R m × m × c under M transform in Eq. (1) is the tensor such that each frontal slice of M ( I ) is a c × c identity matrix,i.e,

Given the appropriate dimensions, it is trivial to verify that T ∗ M I = T and I ∗ M T = T .

Definition 8 ([18]) . A tensor Q ∈ R d × d × d 3 is t-orthogonal under M transform in Eq. (1) if it satisfies

Definition 9 ([19]) . A tensor is called f-diagonal if all its frontal slices are diagonal matrices.

Definition 10 (Tensor t-spectral norm [28]) . The tensor t-spectral norm of any tensor T under M transform in Eq. (1) is defined as the matrix spectral norm of its M -block-diagonal matrix ˜ T M , i.e.,

Lemma 16. For any t-matrix W ∈ R m × n × c and t-vector x ∈ R n × 1 × c , the t-product W ∗ M x defined under M transform in Eq. (1) is equivalent to a linear operator op ( W ) on unfold ( x ) in the orginal domain defined as follows

where ⊗ denotes the Kronecker product, and the operations of unfold ( W ) and unfold ( x ) are given explicitly as follows

Since Eq. (17) is a straightforward reformulation of the definition of t-product in [36, Definition 6.3], the proof is simply omitted.

According to Lemma 16, we have the following remark on the relationship between t-NNs and fully connected neural networks (FNNs).

Remark (Connnection with FNNs) . The t-NNs and FNNs can be treated as special cases of each other.

(I) When the channel number c = 1 , the t-product becomes to standard matrix multi-lication and the proposed t-NN predictor Eq. (5) degenerates to an ( L +1) -layer FNN, which means the FNN is a special case of the t-NN.

(II) On the other hand, by the definition of t-product, the t-NN f ( · ; W ) in Eq. (5) has the compounding representation as an FNN:

Thus, t-NN can also be seen as a special case of FNN.

C Standarad and Adversarial Generalization Bounds for t-NNs

C.1 Standarad Generalization Bound for t-NNs

Lemma 17. Consider the ReLU activation. For any t-vector-valued function set H and any convex and monotonically increasing function g : R → [0 , ∞ ) ,

where R > 0 is a constant.

Proof. This lemma is a direct corollary of Lemma 1 in Ref. [8] by using Eq. (17).

Proof of Lemma 3. According to Lemma 29, we can upper bound the generalization error of ℓ ◦ f for any f ∈ F through the (empirical) Rademacher complexity ˆ R S ( ℓ ◦ F ) where ℓ ◦ F := { ℓ ◦ f | f ∈ F } . Further regarding the L ℓ -Lipschitzness 8 of the loss function ℓ , we have ˆ R S ( ℓ ◦ F ) ≤ L ℓ ˆ R S ( F ) by the Talagrand's contraction lemma (Lemma 30). Then, it remains to bound ˆ R S ( F ) .

To upper bound ˆ R S ( F ) , we follow the proof of [8, Theorem 1]. By Jensen's inequality, the (scaled) Rademacher complexity N ˆ R S ( F ) = E ε sup f ∈ F ∑ N i =1 ε i f ( x i ) satisfies

where λ > 0 is an arbitrary parameter. Then, we can use a 'peeling' argument [8, 37] as follows. The Rademacher complexity can be upper bounded as

Letting B W = B w ∏ L l =1 B l , define a random variable

as a function of random variables { ε i } i . Then

By Jensen's inequlity, E Z can be upper bounded by

To handle the log ( E exp λ ( Z -E Z ) ) term in Eq. (20) , note that Z is a deterministic function of the i.i.d. random variables { ε i } i , and satisfies

This means that Z satisfies a bounded-difference condition , which by the proof of [5, Theorem 6.2], implies that Z is sub-Gaussian, with the variance factor

and satisfies

√

Choosing λ = 2 L log 2 B W √ ∑ N i =1 ∥ x i ∥ 2 F and using the above inequality, we get that Eq. (19) can be upper bounded as follows

Further applying Lemma 29 completes the proof.

C.2 Adversarial Generalization Bound for t-NNs

Proof of Theorem 5. According to Theorem 2 and Eq. (4) in Ref. [3], the adversarial generalization gap of ℓ ◦ f for any f ∈ F with L ℓ -Lipschitz continuous loss function ℓ satisfying Assumption 2 can be upper bounded by L ℓ ˆ R S ( F adv ) , where ˆ R S ( F adv ) is the empirical Rademacher complexity of the adversarial version F adv of the function set F defined as follows

To bound ˆ R S ( F adv ) , we use the Dudley's inequality (Lemma 31) which requires to compute the covering number of F adv .

Let C l be the δ l -covering of { W ( l ) | ∥ ∥ ∥ W ( l ) ∥ ∥ ∥ F ≤ B l } , ∀ l = 1 , · · · , L . Consider the following subset of F whose t-matrix weights are all in C l :

with adversarial version

For all ˜ f ∈ F adv , we need to find the smallest distance to F adv c , i.e. we need to calculate

For all ( x i , y i ) ∈ S , given ˜ f and ˜ f c with ∥ ∥ ∥ W ( l ) -W ( l ) c ∥ ∥ ∥ F ≤ δ l , l = 1 , · · · , L , consider

Letting x f i = arginf R a ( x i -x ′ i ) ≤ ξ y i f ( x ′ i ) and x c i = arginf R a ( x i -x ′ i ) ≤ ξ y i f c ( x ′ i ) , we have | ˜ f ( x i , y i ) -˜ f c ( x i , y i ) | = | y i f ( x f i ) -y i f c ( x c i ) | .

Let

Then,

| ˜ f ( x i , y i ) -˜ f c ( x i , y i ) | = | y i f ( x f i ) -y i f c ( x g i ) | ≤ | y i f ( x ξ i ) -y i f c ( x ξ i ) | = | f ( x ξ i ) -f c ( x ξ i ) | . Let g l ( x δ ) = w ⊤ vec ( σ ( W ( L ) c ∗ M σ ( W ( L -1) c ∗ M · · · ∗ M σ ( W ( l +1) c ∗ M σ ( W ( l ) ∗ M · · · ∗ M σ ( W (1) ∗ M x δ ) · · · ))) ) and g 0 ( x δ ) = f c ( x δ ) . Then, we have

We can see that

and

Then, we have

where B ˜ f is given in Lemma 39.

̸

Letting δ l B ˜ f B l = ϵ L it gives

Then, F adv c is an ϵ -covering of F adv . We further proceed by computing the ϵ -covering number of F adv as follows:

where the second inequality is due to Lemma 32.

Then, we use Dudley's integral in Lemma 31 to upper bound ˆ R S ( F adv ) as follows

where the diameter D ˜ f of F adv is given in Lemma 40 and we can find from Lemma 40 that D ˜ f = 2 B ˜ f in our setting.

Following Ref. [55], let δ → 0 , and use integration by part, we obtain ∫ 1 / 2 0 √ log ( 3 L/ (2 t ) ) d t ≤ √ log 3 L . Hence, we have

and the proof can be completed by using Lemma 29.

C.3 Generalization Bound under Exact Low-tubal-rank Parameterization

Proof of Theorem 6. The idea is similar to the proof of Theorem 5. According to Theorem 2 and Eq. (4) in Ref. [3], the adversarial generalization gap of ℓ ◦ f for any f ∈ F r with L ℓ -Lipschitz continuous loss function ℓ satisfying Assumption 2 can be upper bounded by L ℓ ˆ R S ( F adv r ) , where ˆ R S ( F adv r ) is the empirical Rademacher complexity of the adversarial version F adv r of function set F r defined as follows

To bound ˆ R S ( F adv r ) , we first use the Dudley's inequality (Lemma 31) and compute the covering number of F adv r .

Let C l be the δ l -covering of { W ( l ) | ∥ ∥ ∥ W ( l ) ∥ ∥ ∥ F ≤ B l and r t ( W ( l ) ) ≤ r l } , ∀ l = 1 , · · · , L . Consider the following subset of F r whose t-matrix weights are all in C l :

with adversarial version

For all ˜ f ∈ F adv r , we need to find the smallest distance to F adv c , i.e. we need to calculate

For all ( x i , y i ) ∈ S , given ˜ f and ˜ f c with ∥ ∥ ∥ W ( l ) -W ( l ) c ∥ ∥ ∥ F ≤ δ l , l = 1 , · · · , L , consider

Letting x f i = arginf R a ( x i -x ′ i ) ≤ ξ y i f ( x ′ i ) and x c i = arginf R a ( x i -x ′ i ) ≤ ξ y i f c ( x ′ i ) , we have | ˜ f ( x i , y i ) -˜ f c ( x i , y i ) | = | y i f ( x f i ) -y i f c ( x c i ) | .

Let

Then,

| ˜ f ( x i , y i ) -˜ f c ( x i , y i ) | = | y i f ( x f i ) -y i f c ( x g i ) | ≤ | y i f ( x ξ i ) -y i f c ( x ξ i ) | = | f ( x ξ i ) -f c ( x ξ i ) | . Let g l ( x δ ) = w ⊤ vec ( σ ( W ( L ) c ∗ M σ ( W ( L -1) c ∗ M · · · ∗ M σ ( W ( l +1) c ∗ M σ ( W ( l ) ∗ M · · · ∗ M σ ( W (1) ∗ M x δ ) · · · ))) ) and g 0 ( x δ ) = f c ( x δ ) . Then, we have

We can see that

and

Then, we have

Letting δ l B ˜ f B l = ϵ L gives

̸

Then, F adv c is an ϵ -covering of F adv r . We further proceed by computing the ϵ -covering number of F adv r as follows:

where the inequality ( i ) holds due to Lemma 33.

Then, we use Dudley's integral to upper bound ˆ R S ( F adv r ) as follows

where the diameter D ˜ f of F adv is given in Lemma 40 and we have D ˜ f = 2 B ˜ f . Following Ref. [55], let δ → 0 , and use interation by part, we obtain ∫ 1 / 2 0 √ log ( 9 L/ (2 t ) ) d t ≤ √ log 9 L. Further applying Lemma 29 completes the proof.

D Implicit bias towards low-rankness in the transformed domain

Recent research has shown that GF maximizes the margin of homogeneous networks during standard training, which leads to an implicit bias towards margin maximization [16, 30]. Moreover, it has been demonstrated that this implicit bias also extends to adversarial margin maximization during adversarial training of multi-homogeneous fully connected neural networks with exponential loss [29]. Our analysis builds on these findings by showing that this implicit bias also holds for adversarial training of t-NNs when the adversarial perturbation is scale invariant [29].

First, it is straightforward to see that any t-NN f ∈ F is homogeneous as follows

for any positive constant a .

Lemma 18 (Euler's theorem on t-NNs) . For any t-NN f ∈ F , we have

Proof. By taking derivatives with respect to a on both sides of Eq. (24), we obtain

which immediately results in Eq. (25).

In this section, we follow the setting of Ref. [29] where the adversarial perturbation δ i is scale invariant. As Lemma 7 shows, l 2 -FGM [34], FGSM [9], l 2 -PGD and l ∞ -PGD [31] perturbations for the t-NNs are all scale invariant.

Proof of Lemma 7. Note that by taking derivatives with respect to x on both sides of Eq. (24), we have

Therefore, any ∂f ( x ; W ) ∂ x is positive homogeneous. Then, for any non-zero z = ∂f ( x ; W ) ∂ x , we prove Lemma 7 in the following cases:

l 2 -FGM perturbtion [34]. The l 2 -FGM perturbtion is defined as δ FGM ( W ) = ξy ˜ ℓ ′ z ∥ ∥ ∥ y ˜ ℓ ′ z ∥ ∥ ∥ -1 F = -ξy z ∥ z ∥ -1 F , because ˜ ℓ ′ ≤ 0 . Using Eq. (26), we have

FGSM perturbations [9]. The FGSM perturbtion is taken as δ FSGM ( W ) = ξ sgn ( ξy ˜ ℓ ′ z ) . Using Eq. (26), we have

l 2 -PGD perturbation [31]. The l 2 -PGD perturbtion is taken as

where j is the attack step, P B 2 ( ξ ) is the projector onto l 2 -norm ball of radius ξ , and ρ is the learning rate. We prove by induction. For j = 0 , we have

If we have δ j PGD ( a W ) = δ j PGD ( W ) , then for j +1 , we have

l ∞ -PGD perturbation [31]. Since the scale invariance of this pertubation can be proved very similarly to that of l 2 -PGD perturbations, we just omit it.

For an original example x i , the margin for its adversarial example x i + δ i ( W ) is defined as ˜ q i ( W ) := y i f ( x i + δ i ( W ); W ) ; for sample S = { ( x i , y i ) } N i =1 , the margin for the N corresponding examples is denoted by ˜ q m ( W ) where m ∈ argmin m =1 , ··· ,N y i f ( x i + δ i ( W ) : W ) .

Let ρ = ∥ W ∥ F for simplicity. We use the normalized parameter ˆ W := W /ρ to denote the direction of the weights W . Weintroduce the normalized margin of ( x i , y i ) as ˆ q i ( W ) := ˜ q i ( ˆ W ) = ˜ q i ( W ) ρ -( L +1) , and similarly define the normalized robust margin of the sample S as ˆ q m ( W ) := ˜ q m ( ˆ W ) = ˜ q m ( W ) ρ -( L +1) .

Note that the adversarial empirical risk can be written as ˆ L adv = 1 N ∑ N i =1 e -f (˜ q i ( W )) . Motivated by [30] which uses the LogSumExp function to smoothly approximate the normalized standard margin, we define the smoothed normalized margin ˜ γ as follows.

Definition 11 (Smoothed normalized robust margin) . For a loss function 9 ℓ satisfying Assumption 2, the smoothed normalized robust margin is defined as

To better understand the relation between the normalized sample robust margin ˆ q m and the smoothed normalized robust margin ˜ γ , we provide the following lemma.

Lemma 19 (Adapted from Lemma A.5 of Ref. [30]) . Under Assumption 2, we have the following properties about the robust marin ˜ q m :

(b) If log 1 ˆ adv > f ( b f ) , then there exists ξ ∈ ( f (˜ q m ) -log N, f (˜ q m )) ∩ ( b f , ∞ ) such that

which shows the smoothed normalized margin ˜ γ is a rough approximation of the normalized robust margin ˆ q m .

D.1 Convergence to KKT points of Euclidean norm minimization in direction

The KKT condition for the optimization problem Eq. (14) are

where the dual variables λ i ≥ 0 . We define the approximate KKT point in a similar manner to Ref. [29] as follows.

Definition 12 (Approximate KKT points) . The ( κ, ι ) -approximate KKT points of the optimization problem are those feasible points which satisfy the following two conditions:

where κ, ι > 0 and λ i ≥ 0 .

Proof of Lemma 9. Let ˇ W := W / ˜ q 1 L +1 m denote the scaled version of W ( t ) such that the sample robust margin ˜ q m = 1 . Thus we have f ( x ; W ) = ˜ q m f ( x ; ˇ W ) by homogeneity of t-NNs. According to Lemma 18, we further have

leading to

We will prove that ˇ W is a ( κ, ι ) -KKT point of Problem (14) with ( κ, ι ) → 0 .

Let ˙ W ( t ) := d W ( t ) d t for simplicity. By the chain rule and GF update rule, we have

Using the homogeneity of t-NNs, we obtain

By letting ν ( t ) = ∑ N i =1 e -f (˜ q i ) f ′ (˜ q i )˜ q i , we obtain 〈 ˙ W , W 〉 = ( L +1) ν/N .

We construct the dual variables λ i in Problem (31) in terms of ˙ W as follows

To prove ˇ W is a ( κ, ι ) -KKT point of Problem (14), we need to check the conditions in Problem (31).

Step 1: Check Condition (I) of the ( κ, ι ) -approximate KKT conditions . We check the Condition (I) in Problem (31) for all t > t 0 as follows

where equality ( i ) is obtained by using the definition that ˇ W = W / ˜ q 1 L +1 m , the fact ˜ q i ( W ) = y i f ( x i + δ i ( W ); W ) = y i · ˜ q m f ( x i + δ i ( ˇ W ); ˇ W ) = ˜ q i ( ˇ W ) due to the scale invariance of the adversarial perturbation δ i ( W ) and the homogeneity of t-NN, and the chain rule in computing ∂ ˜ q i ∂ W as follows

In Eq. (33), equality ( ii ) holds by property ( b ) in Lemma 19; ( iii ) holds by the non-decreasing property of ˜ γ ( t ) for all t ∈ [ t 0 , ∞ ) in Lemma 22.

Note that Eq. (33) indicates that κ ( t ) is in terms of the cosine of the angle between W ( t ) and ˙ W ( t ) . We can further obtain that κ ( t ) → 0 as t → ∞ by showing the angle between W ( t ) and ˙ W ( t ) approximates 0 which was orignially observed by Ref. [30] for standard training 10 on a fixed training sample S .

Lemma 20 (Adapted from Lemma C.12 in Ref. [30]) . Under Assumption 2 and Assumption 8 for t-NNs, the angle between W ( t ) and ˙ W ( t ) approximates 0 as t → ∞ along the trajectory of adversarial training with scale invariant adversarial pertubations, i.e.,

Note that according to Assumption 8 we have ˜ γ ( t 0 ) ≥ b f ρ ( t 0 ) -( L +1) which means ˜ γ ( t 0 ) cannot be arbitrarily close to 0 . Thus, by invoking Lemma 20 , we obtain that

Step 2: Check Condition (II) of the ( κ, ι ) -approximate KKT conditions . We check the Condition (II) in Problem (31) for all t > t 0 as follows

where ( i ) is due to the definition that ˇ W = W / ˜ q 1 L +1 m and the fact that ˜ q i ( W ) = y i f ( x i + δ i ( W ); W ) = y i · ˜ q m f ( x i + δ i ( ˇ W ); ˇ W ) = ˜ q i ( ˇ W ) due to the scale invariance of the adversarial pertabation δ i ( W ) and the homogenouty of t-NN.

To upper bound Eq. (35), first note that ∥ ∥ ∥ ˙ W ∥ ∥ ∥ F ≥ 〈 ˙ W , ˆ W 〉 = 〈 1 N ∑ N i =1 e -f (˜ q i ) f ′ (˜ q i ) ∂ ˜ q i ∂ W , ˆ W 〉 = ρ -1 ( L +1) ν/N , in which ν can be further lower bounded as

where ( i ) is due to Lemma 23, ( ii ) holds because of Lemma 24, and ( iii ) uses N ˆ L adv = ∑ i e -f (˜ q i ) ≥ e -f (˜ q m ) . Combing Eq. (35) and Eq. (36) yields

where ( i ) uses ˜ q -2 / ( L +1) m ρ 2 ≤ ˜ γ -2 / ( L +1) by Lemma 19.

In Eq. (37), if ˜ q i > ˜ q m , then there exists an ξ i ∈ (˜ q m , ˜ q i ) such that f (˜ q m ) -f (˜ q i ) = f ′ ( ξ i )(˜ q i -˜ q m ) by the mean value theorem. Further, we know that f ′ (˜ q i ) ≤ K ⌈ log 2 (˜ q i /ξ i ) ⌉ f ′ ( ξ i ) by Assumption 2. Note that ⌈ log 2 (˜ q i /ξ i ) ⌉ ≤ log 2 (2 B 0 ρ L +1 / ˜ q m ) ≤ log 2 (2 B 0 / ˜ γ ) , where

̸

Then, for all t > t 0 , we have

̸

where ( i ) holds because the function x ↦→ e -z z on (0 , ∞ ) has maximum e at z = 1 ; ( ii ) is due to a log c b = b log c a ; ( iii ) holds by the non-decreasing property of ˜ γ ( t ) for all t ∈ [ t 0 , ∞ ) in Lemma 22. Note that by Lemma 22, we have lim t →∞ ˆ L adv ( t ) = 0 , which further yields

Step 3: Check the condition for convergence to KKT point . According to Eq. (34) and Eq. (39), the limit point of ˇ W ( t ) satisfy the ( κ, ι ) -approximate KKT conditions of Problem 14 along the

trajectory of adversarial training of t-NN with scale invariant adversarial perturbations where lim t →∞ ( κ ( t ) , ι ( t )) = 0 . Then, we need to check the condition between ( κ, ι ) -approximate points and KKT points.

According to Ref. [30], the KKT condition becomes a necessary condition for global optimality of Problem (14) when the Mangasarian-Fromovitz Constraint Qualification (MFCQ) [33] is satisfied. It is straightforward to see that Problem (14) satisfies the MFCQ condition, i.e.,

at every feasible point W . Then restating the theorem in Ref. [7] regarding the relation between ( κ, ι ) -approximate KKT point and KKT point in our setting yields the following result.

Theorem 21 (Theorem 3.6 in Ref. [7] and Theorem C.4 in Ref. [30]) . Let

{ ˇ W ( j ) | j ∈ N } be a sequence of feasible point of Problem (14), and ˇ W ( j ) is a ( κ ( j ) , ι ( j )) -approximate KKT point for all j with two sequences 11 { κ ( j ) > 0 | j ∈ N } and { ι ( j ) > 0 | j ∈ N } and lim j →∞ ( κ ( j ) , ι ( j )) = 0 . If lim j →∞ ˇ W ( j ) = ˇ W ∗ , and MFCQ holds at ˇ W ∗ , then ˇ W ∗ is a KKT point of Problem (14).

Recall that ˇ W = W / ˜ q 1 L +1 m . Then, it can be concluded that the limit point of { W ( t ) / ∥ W ( t ) ∥ F : t > 0 } of GF for empirical adversarial risk ˆ L adv ( W ) := 1 N ∑ N i =1 e -f ( y i f ( x i + δ i ( W ); W )) with scale invariant perturbations δ i is aligned with the direction of a KKT point of Problem (14).

D.2 Technical Lemmas for Proving Lemma 9

Lemma 22. Under Assumption 2, we have the following statements for GF-based adversarial training in Eq. (13) with scale invariant perturbations:

( I ) . For a.e. t ∈ ( t 0 , ∞ ) , the smoothed normalized robust margin ˜ γ ( W ( t )) defined in Eq. (29) is non-decreasing, i.e.,

( II ) . The adversarial objective ˆ L adv ( W ) := 1 N ∑ N i =1 e -f ( y i f ( x i + δ i ( W ); W )) with scale invariant pertubations δ i converges to zero as t →∞ , i.e.,

and the Euclidean norm of the t-NN weights diverges, i.e.,

Proof of Lemma 22. We follow the idea of Ref. [30] to prove Lemma 22 as follows.

Step 1: Prove Part (I) . We prove (I) by showing the following results for all t ≥ t 0 ,

Recalling the quantity ν ( t ) = ∑ N i =1 e -f (˜ q i ) f (˜ q i )˜ q i , by chain rule we obtain

where ( i ) holds due to Lemma 23.

By the chaining rule, we also have

Note that according to Eq. (13), we have

for t ≥ 0 almost everywhere. One the other hand, we have

Thus, we obtain

By the chain rule,

for t > 0 allmost everywhere. So, we have

Step 2: Prove Part (II) . Motivated by [30, Lemma B.8], we prove (II) as follows. First, note that

By lower bounding ν with Lemma 23 and replacing ρ with ( g (log 1 N ˆ L adv ) / ˜ γ ) 1 / ( L +1) by the definition smoothed normalized robust margin of ˜ γ in Eq. (29), we obtain

where the last inequality holds due to the non-decreasing property of ˜ γ in Part ( I ) . Then, we obtain

By integrating on both sides from t 0 to t , we obtain

where

We use proof by contradiction to show the empirical training risk ˆ L adv converges to zero. Note that 1 ( N ˆ L adv ) is non-decreasing. If 1 ( N ˆ L adv ) does not grow to ∞ , then neither does G ( 1 ( N ˆ L adv ) ) . But the RHS of Eq. (42) grows to ∞ , which is a contradiction. Therefore, lim t →∞ N ˆ L adv = 0 . Hence lim t →∞ ˆ L adv ( t ) = 0 and lim t →∞ ρ ( t ) = ∞ .

Lemma 23 (Adapted from Lemma B.5 of Ref. [30]) . The quantity ν = ∑ i e -f (˜ q i ) f (˜ q i )˜ q i has a lower bound for all t ∈ ( t 0 , ∞ ) ,

Lemma 24 (Lemma D.1 of Ref. [30]) . For f ( · ) and g ( · ) in Assumption 2, we have

D.3 Proof of Theorem 10

Note that the t-NN g ( x ; V ) with weights V = ( V (1) , · · · , V ( J ) , v ) in Theorem 6 has the following structure

Let α = ( 1 B v ) L -J L +1 and β = ( 1 B v ) -J +1 L +1 , where L is a sufficiently large integer greater than J . We then construct a t-NN h ( x ; H ) of L t-product layers which perfectly realizes g ( x ; V ) . Specifically, we construct h with weights H = ( H (1) , · · · , H ( L ) , h ) satisfying the following equation

or more clearly

where I is the t-identity tensor.

It is easy to prove that for any input x , the input and output of g ( x ; V ) = h ( x ; H ) , it can also be proved that

Therefore, h ( x ; H ) can also robustly classify ( x i , y i ) N i =1 because g ( x ; V ) can robustly classify ( x i , y i ) N i =1 .

Then, we consider the class of over-parameterized t-NNs F = { f ( x ; W ) } defined in Eq. (14) with dimensionality of weight W ( l ) ∈ R d l × d l -1 × c safisfying d l ≫ max j ≤ J { m j } for all l = 1 , · · · , L . Specifically, we construct f with weights

and structure

Note that according to our construction there is a function f ( x ; W h ) ∈ F with weights W h = ( W (1) h , · · · , W ( L ) h , w h ) satisfying

and

We can also see that h ′ ( x + δ ) = h ( x + δ ) for any x ∈ R d × 1 × c and any δ satisfying R a ( δ ) ≤ ξ . Thus, we can say that the weight W h of f ( x ; W h ) is a feasible solution to Problem (14), i.e.,

Now consider the optimal solution W ∗ = ( W ∗ (1) , · · · , W ∗ ( L ) , w ∗ ) to Problem (14). Then according to the optimility of W ∗ and the feasibility of W h to Problem 14, we have

and

As there is an example ( x ∗ , y ∗ ) satisfying ∥ x ∗ ∥ F ≤ 1 in the training set S = { ( x i , y i ) } N i =1 ⊆ R d × 1 × c ×{± 1 } . Then according to Eq. (45), we have

which means indicating that

On the other hand, according to Eq. (44) and Lemma 25, we have

Therefore, we obtain

Note that

where ˜ W ∗ ( l ) M denotes the M -block-diagonal matrix of weight tensor W ∗ ( l ) . Then, we have

Taking the reciprocal of both sides gives

Proof. Let 1 ≤ i < j ≤ L . For µ > 0 , we define a t-NN f µ ( x ; V ) with weights V = ( V (1) , · · · , V L , v ) which are constructed from f ( x ; W ∗ ) whose weights W ∗ = ( W ∗ (1) , · · · , W ∗ ( L ) , w ∗ ) is an optimal solution to Problem (14). Specifically, the construction of V = ( V (1) , · · · , V L , v ) is given as follows:

̸

̸

Note that for every input example x and perturbation δ , f µ ( x + δ ; V ) = f ( x + δ ; W ∗ ) , then V is also feasible to Problem (14). Note that we have

2 2

When µ = 1 the above expression equals 2 ∥ ∥ ∥ W ∗ ( i ) ∥ ∥ ∥ F -2 ∥ ∥ ∥ W ∗ ( j ) ∥ ∥ ∥ F . Hence, if ∥ ∥ ∥ W ∗ ( i ) ∥ ∥ ∥ F = ∥ ∥ ∥ W ∗ ( j ) ∥ ∥ ∥ F , then the derivative at µ is non-zero, which leads to a contradiction to the optimality of W ∗ to Problem (14). Note that if we consider changing norms of W ∗ ( i ) and w instead of W ∗ ( i ) and W ∗ ( j ) , the same conclusion also holds. Thus, the optimality of W ∗ strictly leads to ∥ ∥ ∥ W ∗ ( i ) ∥ ∥ ∥ F = ∥ ∥ ∥ W ∗ ( j ) ∥ ∥ ∥ F = ∥ w ∗ ∥ 2 .

̸

E Generalization bound of approximately low-tubal-rank t-NNs

Proof of Theorem 12. Given a ( δ, r ) -approximately low-tubal-rank parameterized t-NN f ( x ; W ) ∈ F δ, r ⊂ F δ, r , let g ( x ; Wr ) ∈ F r be its compressed version whose t-product layer weight tensors have tubal-ranks upper bounded by r .

Step 1: Upper bound the adversarial empirical L 2 -distance between f and g . Consider function g ( x ) = g ( x ; Wr ) parameterized by Wr = ( W (1) r 1 , · · · , W ( L ) r L , w ) as the function whose t-product layer weights are low-tubal-rank approximations of f ( x ; W ) . Let ˜ f ( x , y ) = inf R a ( x -x ′ ) ≤ ξ yf ( x ′ ) and ˜ g ( x , y ) = inf R a ( x -x ′ ) ≤ ξ yg ( x ′ ) denote the adversarial versions of f and g , respectively.

We first bound the adversarial empirical L 2 -distance between f and g as follows.

Letting x f i = arginf R a ( x i -x ′ i ) ≤ ξ y i f ( x ′ i ) and x g i = arginf R a ( x i -x ′ i ) ≤ ξ y i g ( x ′ i ) , we have | ˜ f ( x i , y i ) -˜ g ( x i , y i ) | = | y i f ( x f i ) -y i g ( x g i ) | . By letting

we obtain

We can see that for any l = 1 , · · · , L :

and

Thus, we have

This gives

Then, we can set

Step 2: Divide and conquer the adversarial gap. To upper bound the adversarial gap L adv ( f ) -ˆ L adv ( f ) of f by using the properties of its compressed version g , we first decompose the adversarial gap into three terms as follows

̸

with high probability 1 -2 e -t .

Step 2.2 : Upper bound I . Note that term I can be written as

Step (2.2.1) : Characterize the concentration behavior of ℓ ◦ ˜ f -ℓ ◦ ˜ g . Given a constant r > 0 , consider the event E 2 ( r ) in which ∥ ∥ ∥ ˜ f -˜ g ∥ ∥ ∥ L 2 ≤ r already holds with high probability. Then, conditioned on Event E 2 ( r ) , by using the L ℓ -Lipschitz continuity of the loss function ℓ ( · , · ) derived from Assumption 2, it can be proved that ℓ ( ˜ f ( x ) , y ) -ℓ (˜ g ( x ) , y ) also has a small population L 2 -norm with high probability.

Regarding Eq. (49) , it is natural to characterize the concentration behavior of centered random variable ℓ ( ˜ f ( x ) , y ) -ℓ (˜ g ( x ) , y ) -E [ ℓ ( ˜ f ( x ) , y ) -ℓ (˜ g ( x ) , y )] .

First, its variance under event E 2 can be upper bounded by

Second, we upper bound its L ∞ -norm. First, Lemma 39 indicates that for any h ∈ F with adversarial version ˜ h ∈ F adv , we have ∥ ∥ ∥ ˜ h ∥ ∥ ∥ L ∞ ≤ B ˜ f := B W B x,R a ,ξ . Then, by F r ⊂ F δ, r ⊂ F , we have ∥ ∥ ∥ ˜ f ∥ ∥ ∥ L ∞ ≤ B ˜ f and ∥ ˜ g ∥ L ∞ ≤ B ˜ f . Therefore, we can upper bound

Then, the Talagrand's concentration inequality (Lemma 35) yields that with probability at least 1 -e -t :

Then, by the the standard symmetrization argument [47], we obtain an upper bound on term ( I ) as follows:

where Φ( r ) is defined as

Thus, there is a constant C > 0 such that for any ˜ f -˜ g ∈ ( F adv δ, r -F adv r ) satisfying ∥ ∥ ∥ ˜ f -˜ g ∥ ∥ ∥ L 2 ≤ r , it holds with probability at least 1 -e -t that

We denote the above event by E 3 ( r ) . Note that Event E 3 ( r ) is conditioned on Event E 2 ( r ) .

Step (2.2.2) : Upper bound the probability of Event E 2 ( r ) := { ∥ ∥ ∥ ˜ f -˜ g ∥ ∥ ∥ L 2 ≤ r } . We further bound the probability of Event E 2 ( r ) in which ∥ ∥ ∥ ˜ f -˜ g ∥ ∥ ∥ L 2 ≤ r holds. Generally speaking, ˜ f and ˜ g are date dependent and we can only bound the empirical L 2 -distance between them. However, the local Rademacher complexity is characterized by the population L 2 -norm. Thus, we need to bound the population L 2 -distance between ˜ f and ˜ g . Motivated by [43], we use the ratio type empirical process to bound the bound the population L 2 -distance.

According to Assumption 11, their exists a function ϕ : [0 , ∞ ) → [0 , ∞ ) such that

Define the quantity Γ( r ) := E [ sup h ( 1 N ∑ N i =1 ε i h 2 ( x i , y i ) ∣ ∣ h ∈ ( F adv δ, r -F adv r ) : ∥ h ∥ L 2 ≤ r )] . Then, we have

where ( i ) is by the Talagrand's contraction lemma (Lemma 30).

We can verify that the square h 2 ( · ) of any function h ∈ F adv δ, r -F adv r satisfies

(i) its L ∞ -norm is upper bouned by B 2 ˜ f , i.e., ∥ ∥ h 2 ∥ ∥ L ∞ = sup ( x ,y ) | h 2 ( x , y ) | ≤ B 2 ˜ f .

(ii) its second-order moment satisfies E x ,y [ ( h 2 ( x , y ) 2 ) ] ≤ E x ,y [ B 2 ˜ f ( h 2 ( x , y )) ] = B 2 ˜ f E x ,y [ h 2 ( x , y ) ] .

Thus, h 2 satisfy the conditions in Eq. (7.6) and Eq. (7.7) of [42] with parameters B = B 2 ˜ f , V = B 2 ˜ f and ϑ = 1 . Noting that we have upper bounded Γ( r ) by 2 B ˜ f ϕ ( r ) , then by the peeling trick [42, Eq. (7.17)], we can show for any r > inf { √ E [ h 2 ] : h ∈ ( F adv δ, r -F adv r ) : ∥ h ∥ L 2 ≤ r } and t > 0 that

We further define a function r ∗ = r ∗ ( t ) as

which is useful to bound the ratio of the empirical L 2 -norm and the population L 2 -norm of an elements h ∈ F adv δ, r -F adv r with probability at least 1 -e t :

Recalling that ∥ ∥ ∥ ˜ f -˜ g ∥ ∥ ∥ S ≤ ˆ r , we obtain that the probability of Event E 2 (˙ r ) with ˙ r = √ 2(ˆ r 2 + r 2 ∗ ( t )) is at least 1 -e -t .

Step 2.3 : Combining Events E 1 , E 2 (˙ r ) , E 3 (˙ r ) . By combining Eqs. (46), (47), (51) and (48) along with their underlying events E 1 , E 2 (˙ r ) , E 3 (˙ r ) , we obtain

with probability at least 1 -4 e -t .

E.1 Several Useful Results

According to Theorem 12, it remains to upper bound Φ(˙ r ) . In this subsection, we derive upper bounds on Φ( r ) in terms of covering numbers of the considered function sets F adv r and F adv δ, r .

Consider the supremum of the empirical L 2 -norm of any function ( ˜ f -˜ g ) ∈ ( F adv δ, r -F adv r ) on sample S = { ( x i , y i ) } N i =1 when the population L 2 -norm is bounded by a given radius r > 0 as follows

Recall that we have assumed ˙ R r ( F adv δ, r -F adv r ) ≤ ϕ ( r ) . We now give an explict example of ϕ ( r ) in terms of the sum of covering entropy of F adv δ, r and F adv r

where ( i ) is the definition of localized Rademacher complexity; ( ii ) is due to Dudley's inequality (Lemma 31); ( iii ) is obtained by letting a = 1 /N ; ( iv ) holds by Lemma 34.

Lemma 26. We can upper bound Φ( r ) by using ϕ ( r ) as follows

Proof. Recall that Φ( r ) is the average Rademacher complexity of the function set

where ( i ) holds because the loss function ℓ is L ℓ -Lischitz continous.

To bound Φ( r ) , we first bound the its empirical version using the Dudley's inequlity (Lemma 31) up to a constant as follows

where in ( i ) we let a = L ℓ /N ; ( ii ) holds by the Lipschitzness of ℓ and the definition of covering number; we use change of variable t = ϵ/L ℓ in ( iii ) .

By taking expectations on the RHS of Eq. (56) with respect to the sample S , we obtain Eq. (55).

To determine an appropriate radius of the population L 2 -norm ˙ r = 2 √ ˆ r 2 + r 2 ∗ , we need to compute the value of r ∗ of satisfying Eq. (52). Using Eq. (54), we show how to compute ϕ ( r ) when the covering numbers of F adv δ, r and F adv r satisfy a special bound.

Lemma 27 (Adapted from Lemma 3 in Ref. [43]) . Suppose that the covering numbers of F adv δ, r and F adv r satisfy

for some q ≤ 1 . Then, it holds that

(I) The bound ϕ ( r ) of the local Rademacher complexity ˙ R r ( F adv δ, r -F adv r ) of radius r can be upper bounded as

for a universal constant C > 0 and a constant C q > 0 which only depends on q ≤ 1 .

(II) In particular, the quantity r ∗ ( t ) satisying Eq. (52) can be upper bounded as

Proof of Lemma 27. According to Eq. (54) which gives ϕ ( r ) and the definition of β S in Eq. (53), we need to upper bound

Hence, if I ≥ II , then

which leads to

If I < II , then by using Young's inequality we obtain

for any c 1 > 0 . Thus, by taking c 1 = 4 C (1 -q ) B ˜ f , we obtain

where C q > 0 is a universal constant only depending on q .

Then, we obtain the bound on ϕ ( r ) in Eq. (58). The bound on r 2 ∗ can be obtained by simple calculations based on Eqs. (52) and (58).

Lemma 28. When the population L 2 -norm of any function h = ˜ f -˜ g ∈ ( F adv δ, r -F adv r ) is upper bounded by r , its squared empirical L 2 -norm can be upper bouned as follows:

Proof. The squared empirical L 2 -norm can be upper bouned based on the population L 2 -norm as follows

(60)

where ( i ) is due to the symmertrization argument [47] and ε = { ε i } N i =1 are i.i.d. Rademacher variables, and ( ii ) holds because of the contraction inequality (Lemma 30).

E.2 Adversarial Generalization Gap under Assumption 13

Proof of Theorem 14. In this situation, we can see that for any 1 ≤ r l ≤ min { d l , d l -1 } , we can approximate W ( l ) with its optimal tubal-rankr l approximation tensor W ( l ) r l according to [20, Theorem 3.7] and achieve the following approximation error bound on F-norm

where ( i ) holds because of Lemma 41.

We also have a specral norm bound for W ( l ) -W ( l ) r l according to [20, Theorem 3.7] under Assumption 13 as follows and we also have

Consider function g ( x ) = g ( x ; Wr ) parameterized by Wr = ( W (1) r 1 , · · · , W ( L ) r L , w ) as the function whose t-product layer weights are low-tubal-rank approximations of f ( x ; W ) . Let ˜ f ( x , y ) = inf R a ( x -x ′ ) ≤ ξ yf ( x ′ ) and ˜ g ( x , y ) = inf R a ( x -x ′ ) ≤ ξ yg ( x ′ ) denote the adversarial versions of f and g , respectively.

This helps us bounding ˆ r as follows

Letting x f i = arginf R a ( x i -x ′ i ) ≤ ξ y i f ( x ′ i ) and x g i = arginf R a ( x i -x ′ i ) ≤ ξ y i g ( x ′ i ) , we have | ˜ f ( x i , y i ) -˜ g ( x i , y i ) | = | y i f ( x f i ) -y i g ( x g i ) | . By letting

we obtain

| ˜ f ( x i , y i ) -˜ g ( x i , y i ) | = | y i f ( x f i ) -y i g ( x g i ) | ≤ | y i f ( x ξ i ) -y i g ( x ξ i ) | = | f ( x ξ i ) -g ( x ξ i ) | . Let h l ( x ξ ) = w ⊤ vec ( σ ( W ( L ) r l ∗ M σ ( W ( L -1) r -1 ∗ M · · · ∗ M σ ( W ( l +1) r ∗ M σ ( W ( l ) ∗ M · · · ∗ M σ ( W (1) ∗ M x ξ ) · · · ))) ) and h 0 ( x ξ ) = g ( x ξ ) . Then, we have

We can see that for any l = 1 , · · · , L :

and

where in ( i ) we relax the inequality by using the tensor spectral norm of insted of the F-norm of W ( l ) r l -W ( l ) which leads to a smaller upper bound. Thus, we have

This gives

Then, we can set

Then we can construct an ϵ -cover of F adv δ, r composed of ˜ g ( x ; Wr ) by carefully setting the value of rank parameter r = ( r 1 , · · · , r L ) ⊤ according to the covering accuracy ϵ . Directily setting δ sp l D ˜ f 2 B l = ϵ/L , we obtain where

̸

Then, we obtain the covering entropy of F adv δ, r by

where ( ii ) holds by Eq. (61). Since F adv r ⊂ F adv δ, r , we have

To use Lemma 27, we bound log N ( F adv δ, r , ∥·∥ S , ϵ/ 2) + N ( F adv r , ∥·∥ S , ϵ/ 2) when ϵ ≥ 2 /N as follows:

where

For simplicity, further let

Then, according to Lemma 27, we have

which further leads to

Note that

Then by simple calculation, we have

Proof of Corollary 15. The bound in Corollary 15 can be directly obtained if we choose the parameter r of tubal ranks in F r by r l = min {⌈ ( LV 0 B ˜ f B -1 l ) 1 /α ⌉ , d l , d l -1 } .

F Useful Notions and Lemmas

In this section, we provide several notions and lemmas which are used in the previous analysis.

F.1 Tools for Analyzing General DNNs

We briefly list the tools used in this paper for analyzing the generalization error of general DNNs, including Rademacher complexity, covering number, and concentration inequalities, etc. .

Definition 13 (Rademacher complexity) . Given an i.i.d. sample S := { ( x i , y i ) } N i =1 of size N and a function class H , the empirical Rademacher complexity of H is defined as

where ε 1 , · · · , ε N are i.i.d. Rademacher variables, i.e., ε i equals to 1 or -1 with equal probability. The average Rademacher complexity is further defined as

Lemma 29 ([4]) . Given an i.i.d. sample S := { ( x i , y i ) } N i =1 of size N , a loss function ℓ ( h ( · ) , y ) taking values in [0 , B ] , the generalization error of any function f in hypothesis set F satisfies

with probability at least 1 -e -t for all t ≥ 0 .

Lemma 30 (Talagrand's contraction lemma [44]) . Given function set F and L ℓ -Lipschtz function ℓ , for a function sets defined as l F := { ℓ ◦ f ∣ ∣ f ∈ F} , we have

Definition 14 ( ϵ -covering net) . Let ϵ > 0 and ( X , d ( · , · )) be a metric space, where d ( · , · ) is a (pseudo)-metric. We say Z ⊂ X is an ϵ -covering net of X , if for any x ∈ X , there exists z ∈ Z such that d ( x, z ) ≤ ϵ . Define the smallest |Z| as the ϵ -covering number of X and denote as N ( X , d ( · , · ) , ϵ ) .

Given a traning dataset S = { x i , y i } N i =1 and a function set F . Consider the output space of the space of F restricted on S , i.e., F| S = { ( f ( x 1 , y 1 ) , · · · , f ( x N , y N ) ) ⊤ ∣ ∣ f ∈ F} . Then, define a pseudo-norm of F| S as: ∥ f ∥ S := N -1 √ ∑ N i =1 f ( x i , y i ) 2 . Then, the Rademacher complexity of F could be upper bounded by the ϵ -covering number of F under the empirical l 2 -pseudo-metric by the Dudley's inequality as follows:

Lemma 31 (Dudley's integral inequality [47]) . The Rademacher complexity ˆ R S ( F ) satisfies

Lemma 32 (Covering number of norm balls [47]) . Let B be a l p -norm ball with radius W . Let d ( x 1 , x 2 ) = ∥ x 1 -x 2 ∥ p . Define the ϵ -covering number of B as N ( B , d ( · , · ) , ϵ ) , we have

Lemma 33 (Covering number of low-tubal-rank tensors) . For the set of tensors T r := { T ∈ R m × n × c ∣ ∣ r t ( T ) ≤ r, ∥ T ∥ F ≤ 1 } with r ≤ min { m,n } , its ϵ -covering number can be upper bounded by

Proof. Consider the reduced t-SVD [20] of a tensor X = U ∗ M S ∗ M V ⊤ ∈ T r , where ∗ M denotes the t-product induced by the linear transform M ( · ) defined in Eq. (1) , U ∈ R m × r × c and V ∈ R n × r × c are (semi)-t-orthogonal tensors, and S ∈ R r × r × c is an f-diagonal tensor. As T ∈ T r , we have ∥ S ∥ F = ∥ X ∥ F ≤ 1 . The idea is to cover T r by covering the set of factor tensors U , V and S .

Let D ⊂ R r × r × c be the set of f-diagonal tensors with F-norm equal to one. We take D c to be an ϵ/ 3 -covering net for D . Then by Lemma 32, we have | D c | ≤ (9 /ϵ ) r c . Next, let O m,r = { U ∈ R m × r × c | U ∗ M U ⊤ = I } . To cover O m,r , we consider the ∥·∥ ∞ , 2 , 2 -norm defined as

Let Q m,r := { X ∈ R m × r × c | ∥ X ∥ ∞ , 2 , 2 ≤ 1 } . Then, we have O m,r ⊂ Q m,r by the definition of torthogonal tensors. Letting Q c m,r be an ϵ/ 3 -covering net of Q m,r , then we obtain | Q c m,r | ≤ (9 /ϵ ) mr c by Theorem 32. Similarly, an ϵ/ 3 -covering net of Q n,r satisfies | Q c n,r | ≤ (9 /ϵ ) nr c .

Now construct a set T c r = { U c ∗ M S c ∗ M ( V c ) ⊤ | U c ∈ Q c m,r , S c ∈ D c , V c ∈ Q c n,r } . Then, we have

Net, we will show that T c r is an ϵ -covering net of T r , i.e., for any X ∈ T r , there is an X c ∈ T c r satisfying ∥ X -X c ∥ F ≤ ϵ .

Given X ∈ T r , consider its reduced t-SVD as X = U ∗ M S ∗ M V ⊤ ∈ T r . Then, there exists X c = U c ∗ M S c ∗ M ( V c ) ⊤ with U c ∈ Q c m,r , S c ∈ D c , V c ∈ Q c n,r satisying ∥ U -U c ∥ ∞ , 2 , 2 ≤ ϵ/ 3 , ∥ S -S c ∥ F ≤ ϵ/ 3 , and ∥ V -V c ∥ ∞ , 2 , 2 ≤ ϵ/ 3 . This gives

For the first term, note that since V is a t-orthogonal tensor,

and

Hence, ∥ ∥ ∥ ( U -U c ) ∗ M S ∗ M V ⊤ ∥ ∥ ∥ F ≤ ϵ/ 3 . Similarly, we have ∥ ∥ U c ∗ M S c ∗ M ( V -V c ) ⊤ ∥ ∥ F ≤ ϵ/ 3 .

The middle term can be bounded ∥ ∥ ∥ U c ∗ M ( S -S c ) ∗ M V ⊤ ∥ ∥ ∥ F ≤ ∥ S -S c ∥ F ≤ ϵ/ 3 due to the property of t-orthognal tensors U c and V c [20]. Therefore, for any X ∈ T r , there is an X c ∈ T c r satisfying ∥ X -X c ∥ F ≤ ϵ .

Lemma 34 (Covering number bounds for composition and addtion [10]) . Let F 1 and F 2 be classes of functions on normed space ( X , ∥·∥ X ) → ( Y , ∥·∥ Y ) and let F be a class of c -Lipschitz functions ( Y , ∥·∥ Y ) → ( Z , ∥·∥ Z ) . Then for any X ∈ X N and ϵ F 1 , ϵ F 2 , ϵ F > 0 , it holds that

and

Specifically, if F = { f } is a singleton, we have

Lemma 35 (Simplifided Talagrand's concentration inequality [42]) . Let F be a function class on X that is separable with respect to L ∞ -norm, and { x i } N i =1 be i.i.d. random variables in X . Furthermore, suppose there exist constants V ≥ and U ≥ 0 such that V = sup f ∈F E [( f -E [ f ]) 2 ] and U = sup f ∈F ∥ f ∥ L ∞ . Letting Z := sup f ∈F | N -1 ∑ N i =1 f ( x i ) -E [ f ] | , then it holds for all t > 0 that

F.2 Some Results for Analyzing t-NNs

In this subsection, we present several fundamental statements described as lemmas for analyzing t-NNs. First, we give Lemmas 36-38, which are used in the analysis of the t-product layers.

Lemma36. Let σ ( · ) : R → R be a L σ -Lipschitz function, i.e., | σ ( x ) -σ ( y ) | ≤ L σ | x -y | , ∀ x, y ∈ R . If it is applied element-wisely to any two real vectors x and y , then it holds that

Lemma 37. The following inequalities hold:

Proof. According to the definition of the M transform in Eq. (1) and the orthogonality of M , we have

and

where inequality ( i ) holds because ∥ AB ∥ F ≤ ∥ A ∥ ∥ B ∥ F for any matrices A , B with appropriate dimensions.

Lemma 38 (The t-product layer is Lipschitz continuous) . Suppose the activation function is L σ -Lipschitz, then a layer of t-product layer h ( x ) = σ ( W ∗ M x ) is at most L σ ∥ W ∥ F -Lipschitz.

Proof. According to the Lipschitzness of the activation function, we have

We then present Lemma 39 and Lemma 40 which are used in upper bounding the input and output of t-NNs in adversarial settings.

Lemma 39. Given a fixed example x ∈ R d × 1 × c , if an adversary x ′ satisfies R a ( x -x ′ ) ≤ ξ , then it holds that

∥

x

∥

F

B

x

+

ξ

C

R

a

.

Lemma 40. The L ∞ -norm of any f ( x ; W ) ∈ F adv defined on the set of input examples X is upper bounded by

The diameter of F adv is can be upper bounded as follows

Proof. For any f ∈ F , given an example ( x , y ) ∈ X × {± 1 } , let x ∗ ∈ arginf R a ( x -x ′ ) ≤ ξ yf ( x ′ ) be one adversarial example. Then, we have

′

which also implies

Lemma 41 helps upper bounding the F-norm of residuals after low-tubal-rank approximation of the weight tensors of t-NNs under Assumption 13.

Lemma 41. Given constants a > 0 , α > 1 , suppose a sequence { z j } ∞ j =1 satisfying polynomial decay z j ≤ aj -α , then for any positive integer n , we have

Proof. We compute the sum of the sequence using integration as follows:

Peer reviews:
comments: SUMMARY:
The paper provides a thorough investigation of the generalization behavior of t-NNs for the first time, which closes the gap between the practical success of t-NNs and their theoretical analysis. To be specific, the authors derived the upper bounds for the generalization gaps. The authors also propose that compressing t-NNs with a transformed low-rank structure will result in more efficient adversarial training and tighter bounds. In addition to that, the authors show that adversarial training with GF in highly over-parameterized settings results in t-NNs with approximately transformed low-rank weights, and derive the sharp adversarial generalization bounds in this scenario as well.

STRENGTHS:
- The paper provides a thorough investigation of the generalization behavior of t-NNs for the first time.
- The authors provide the bound for t-NNs generalization gaps with firm proof.
- The authors further explore the generalization bound of t-NNs with a transformed low-rank structure and show that there exists a training framework that will result in approximately transformed low-rank weights.
- The work is well motivated and with firm theoretical evaluation.

WEAKNESSES:
- Only theoretical evaluation is provided. Some empirical evaluations regarding the approximately transformed low-rank weights are expected.

QUESTIONS:
- Instead of using adversarial training with GF in highly over-parameterized settings to train a approximately transformed low-rank t-NN, is it possible to apply some extra regularizations in training to achieve a better low-rank transformation, for example, LoRA?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks\" considers neural networks with t-product layers, that encode the weights in tensor format equipped with the t-product as the corresponding tensor operation and ReLU activations.\nThe paper is an analytical work that derives upper bounds for the generalization error in t-NNs, using norm bound estimates. Further, the influence of low-rankedness of the Tensor structures on robust generalization is inspected in the setting of exact low-rankedness and  approximately transformed low-rankedness. Results show that low-rank t-NNs improve robust generalization properties. In particular, the Adverserial generalization bound for low-rank tNNs is shown to scale with the tubal rank of the tensors, instead of their dimension as in the full-rank setting. \nAdditional findings show that in an overparametrized setting t-NNs yield approximately low-rank weights during adversarial learning.\n\nSTRENGTHS:\n- Although this is a rather technical paper, the authors manage to present the findings comprehensively by rigorously introducing the relevant concepts with a precise notation.\n- An extensive Appendix is provided to give an introduction to t-SVDs, and overview of previous work and technical details.\n- The theoretical results of the work are discussed and compared with previous works.\n\nWEAKNESSES:\n- No code and no numerical examples are provided to support the presented results.\n\nAlthough this is a rather theoretical work, a small test case can improve the presentation of this paper tremendously. \nI think this is a strong paper that gives a multitude of insights. \n\nA brief explanation of the intuition behind t-NN layers would also be helpful.\n\nQUESTIONS:\n- What is the intuition of the transformed t-product, i.e. what is the reason that the M matrix is needed? What is the special case of M being the identiy? \n- Theorem 6: Can this statement be related to \"standard\" matrix valued neural networks with low-rank factorizations? \n- Theorem 10: Are you able to validate the statement in a numerical example. It would be interesting to see, how the rank of a layer changes as J gets larger.\n\nLIMITATIONS:\nThe authors discuss the limitations of their work, which manifest in rather loose bounds on the generalization error. They provide a suggestion on how to mitigate this issue in future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper analyzes the generalization ability of t-product layers (t-NNs) by deriving upper bounds on generalization error in standard and adversarial settings.

STRENGTHS:
The paper advances the theoretical understanding of t-NNs and derives their generalization behavior in two practical settings.

WEAKNESSES:
The paper focuses on and is written about t-product layers, referenced as [7, 28] in the first lines of the introduction. The first reference is a patent, and the second is a paper from 2018 (from the same authors as the patent), introducing the concept more or less academically. The submission assumes the reader's familiarity with these concepts right from the start, which is a mistake, and does not attempt to clarify or recap the importance of these concepts up until Sec. 3. From this standpoint, the writing should be improved. 

The importance of the usage of t-product layers in the Deep Learning community is not supported with sufficient evidence, leaving the question of the practical utility of the studied framework unanswered.

QUESTIONS:
Is Definition-1 (t-product) equivalent to the batch matrix-multiplication operation, where `c` corresponds to the batch dimension?
Is Definition-3 (t-SVD) equivalent to the SVD of a block-diagonal representation from Definition 2?
Why do these concepts need special terms? What is the advantage of such complex structures over the traditional deep learning layers?

LIMITATIONS:
Discussed in the Conclusion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the generalization error in the standard and the adversarial settings of t-NNs, neural networks with layers and features parametrized via t-vectors and t-products.\n\nSTRENGTHS:\n- The provided generalization bounds for t-NNs show that neural networks with low-rank parameters have the potential for a better generalization.\n- This type of generalization analysis for tNNs seems new\n\nWEAKNESSES:\n1. Although I appreciate this type of analysis for t-NN did not appear before, the t-product is essentially a composition of operations obtained through a suitable block reshaping of the parameter kernel, thus extending the results from standard fully connected linear layers seems like a natural and relatively direct result. In light of this, the novelty of this paper seems limited\n\n2. I would have liked to see a few numerical experiments validating the theoretical findings and the tightness of upper bounds -- also to address the claimed limitations at the end of the paper\n\nQUESTIONS:\n- the authors should clarify in the main text how the obtained results differentiate with respect the corresponding bounds for linear fully connected nets (for example the bound in Thm 6 and Thm 10) and should highlight what the main differences in the proof/analysis of these bounds are with respect to the linear case\n\n- I think the rank-adaptive approach for selecting the ranks of the network mentioned on line 186 deserves more attention - in particular, i do not really understand in what way rank-adaptivity can be exploited in the analysis provided by Thm 6. See also [x],[y] for alternative recent rank-adaptive strategies for linear layers.\n\n- The fact that adversarial training leads to low-rank weights and vice-versa reducing parameters has a positive/negative effect of robustness (as mentioned at line 250) is subject to ongoing active research and there is a variety of different observations/results on this subject (see eg [a-d]). I would rephrase here in the light of these papers. In any case, these empirical studies are done for networks in the standard form not for t-NNs. Using that evidence to comment on findings for t-NN seems to support my weakness no. 1\n\n[x] H. Yang, M. Tang, W. Wen, F. Yan, D. Hu, A. Li, H. Li, and Y. Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification (IEEE/CVF 2020)  \n[y] S. Schotthöfer, E. Zangrando, J. Kusch, G. Ceruti, and F. Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations (NeurIPS 2022)   \n[a] https://openreview.net/pdf?id=SJGrAisIz\n[b] https://arxiv.org/abs/1912.02386\n[c] https://arxiv.org/abs/2306.01485\n[d] https://link.springer.com/content/pdf/10.1007/s10994-021-06049-9.pdf?pdf=button\n\nLIMITATIONS:\nThe authors have added a \"limitations\" paragraph stating that the obtained generalization bounds may be somewhat conservative, which I agree with. I am not sure I understand the proposed solution to the limitations though.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-30"}