SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theoretical analysis methods. Our analysis draws on norm-based generalization analysis [37] and implicit regularization of gradient descent-based learning [46] as related theoretical analysis methods. Norm-based generalization analysis plays a crucial role in theoretical analysis across various domains, including standard generalization analysis of DNNs [8], compressed models [22], non-compressed models [43], and adversarial generalization analysis [3, 55, 59]. Our work extends norm-based tools to analyze both standard and adversarial generalization in t-NNs, going beyond the traditional use of matrix products. For implicit regularization of gradient descent based learning, extensive past research has been conducted on implicit bias of GF for both standard and adversarial training of homogeneous networks building on matrix product layers, respectively [16, 30, 45]. We non-trivially extend these methods to analyze t-NNs and reveals that GF for over-parameterized ReLU t-NNs produces nearly transformed low-rank weights under scale invariant adversarial perturbations.
Our theoretical results notably deviate from the standard error bounds for fully connected neural networks (FNNs) in several ways:
- The generalization bounds in Lemma 3 and Theorem 5 for t-NNs diverge from their counterparts for FNNs in Refs. [8, 55, 59] due to the channel number c in t-NNs. Moreover, Theorem 5 encompasses a wider range of adversary classes than the  $l_{p}$ -attacks in the aforementioned references.
- The uniqueness of Theorem 6, compared to Refs. [55, 59], stems from its consideration of weight low-rankness in the adversarial generalization bound, suggesting possible robustness improvements in generalization.
- Our exploration of the implicit bias in GF for adversarial training presents a novel angle: the bias towards approximate transformed low-rankness in t-NNs. While Ref. [29] focuses on the implicit bias in adversarial training for FNNs, centered on KKT point convergence with exponential loss, our work delves deeper, considering a wider array of loss functions in adversarial training for t-NNs.
- A crucial distinction in our adversarial generalization bounds, detailed in Section 4.3, from non-adversarial bounds for FNNs [43] is the integration of the localized Rademacher complexity. This encompasses the Minkowski difference between adversarial counterparts of both approximately and exactly low-tubal-rank t-NNs as seen in Theorem 12.
A thorough investigation of the generalization behavior of t-NNs is conducted for the first time. We derive upper bounds for the generalization gaps of standard and adversarially trained t-NNs and propose compressing t-NNs with a transformed low-rank structure for more efficient adversarial learning and tighter bounds on the adversarial generalization gap. Our analysis shows that adversarial training with GF in highly over-parameterized settings results in t-NNs with approximately transformed low-rank weights. We further establish sharp adversarial generalization bounds for t-NNs with approximately transformed low-rank weights. Our findings demonstrate that utilizing the transformed low-rank parameterization can significantly enhance the robust generalization of t-NNs, carrying both theoretical and empirical significance.
Limitations. While this paper adheres to the norm-based framework for capacity control [8, 37], it is worth noting that the obtained generalization bounds may be somewhat conservative. However, this limitation can be mitigated by employing more sophisticated analysis techniques, as evidenced by recent studies [2, 25, 56, 57].
**Discussions.** The inclination of adversarial training towards low-rank/sparse weights, and the reciprocal effects of parameter reduction on robustness, are currently at the forefront of ongoing research. This domain has witnessed a spectrum of observations and results [6, 23, 41, 51]. In this study, we propose that employing low-rank parameterization can enhance the adversarial robustness of t-NNs, as evidenced by our analysis of uniform adversarial generalization error bounds. However, despite these promising results, it is crucial to emphasize the necessity of a more exhaustive exploration of low-rank parameterization. Its implications, particularly when considered in the context of approximation, estimation, and optimization, are profound and warrant further dedicated research efforts. Such a comprehensive investigation will undoubtedly enhance our understanding and fully unlock the potential of low-rank parameterization in neural networks.
We extend our deepest gratitude to Linfeng Sui and Xuyang Zhao for their indispensable support in implementing the Python code for t-NNs during the rebuttal phase. Our sincere appreciation also goes to both the area chair and reviewers for their unwavering dedication and meticulous attention given to this paper. This research was supported by RIKEN Incentive Research Project 100847-202301062011, by JSPS KAKENHI Grant Numbers JP20H04249 and JP23H03419, and in part by National Natural Science Foundation of China under Grants 62103110 and 62073087.
[1] J. An, W. Liu, Q. Liu, L. Guo, P. Ren, and T. Li. Dginet: Dynamic graph and interaction-aware convolutional network for vehicle trajectory prediction. Neural Networks, 151:336-348, 2022. 1
[2] S. Arora, R. Ge, B. Neyshabur, and Y. Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pages 254-263. PMLR, 2018. 6
[3] P. Awasthi, N. Frank, and M. Mohri. Adversarial learning guarantees for linear hypotheses and neural networks. In International Conference on Machine Learning, pages 431-441. PMLR, 2020. 3.1, 5, C.2, C.3
[4] P. L. Bartlett, O. Bousquet, and S. Mendelson. Localized rademacher complexities. In Computational Learning Theory: 15th Annual Conference on Computational Learning Theory, COLT 2002 Sydney, Australia, July 8–10, 2002 Proceedings 15, pages 44–58. Springer, 2002. 4.3, 29
[5] S. Boucheron, G. Lugosi, and P. Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013. C.1
[6] J. Cosentino, F. Zaiter, D. Pei, and J. Zhu. The search for sparse, robust neural networks. arXiv preprint arXiv:1912.02386, 2019. 6
[7] J. Dutta, K. Deb, R. Tulshyan, and R. Arora. Approximate KKT points and a proximity measure for termination. Journal of Global Optimization, 56(4):1463-1499, 2013. 4.2, 4.2, D.1, 21
[8] N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pages 297-299. PMLR, 2018. 3.1, 5, 6, 6, C.1
[9] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. 7, A.1, D, D
[10] F. Graf, S. Zeng, M. Niethammer, and R. Kwitt. On measuring excess capacity in neural networks. arXiv preprint arXiv:2202.08070, 2022. 34
[11] F. Han, Y. Miao, Z. Sun, and Y. Wei. T-ADAF: Adaptive data augmentation framework for image classification network based on tensor t-product operator. Neural Processing Letters, 2023. 1
[12] L. Horesh, E. Newman, M. E. Kilmer, and H. Avron. Generating and managing deep tensor neural networks, Dec. 20 2022. US Patent 11,531,902. 1
[13] J. Hou, F. Zhang, H. Qiu, J. Wang, Y. Wang, and D. Meng. Robust low-tubal-rank tensor recovery from binary measurements. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. 5
[14] Z. Huang, X. Li, Y. Ye, and M. K. Ng. Mr-gcn: Multi-relational graph convolutional networks based on generalized tensor product. In *IJCAI*, volume 20, pages 1258–1264, 2020. 1
[15] Y. Idelbayev and M. A. Carreira-Perpinan. Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8049–8059, 2020. 5
[55] J. Xiao, Y. Fan, R. Sun, and Z.-Q. Luo. Adversarial Rademacher complexity of deep neural networks. arXiv preprint arXiv:2211.14966, 2022. 1, 3.1, 3.2, 3.2, 5, 6, C.2, C.3
[56] J. Xiao, Y. Fan, R. Sun, J. Wang, and Z.-Q. Luo. Stability analysis and generalization bounds of adversarial training. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. 6
[57] J. Xiao, R. Sun, and Z.-Q. Luo. PAC-bayesian spectrally-normalized bounds for adversarially robust generalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 6
[58] J. Yang, C. Fu, F. Deng, M. Wen, X. Guo, and C. Wan. Toward interpretable graph tensor convolution neural network for code semantics embedding. ACM Transactions on Software Engineering and Methodology, 2023. 1
[59] D. Yin, R. Kannan, and P. Bartlett. Rademacher complexity for adversarially robust generalization. In International Conference on Machine Learning, pages 7085-7094. PMLR, 2019. 3.1, 3.1, 5, 6
[60] X. Zhang and M. K. Ng. Sparse nonnegative tensor factorization and completion with noisy observations. IEEE Transactions on Information Theory, 68(4):2551-2572, 2022. 1, 5
[61] X. Zhang and M. K.-P. Ng. Low rank tensor completion with poisson observations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. 1
Appendix Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks
In the appendix, we begin by presenting numerical evaluations of our theoretical findings. Subsequently, we introduce the additional notations and preliminaries related to t-SVD, followed by the proofs of the propositions mentioned in the main text.
Our analysis and proofs pertaining to t-NNs differ from those for FNNs as follows:
- Firstly, unlike the analysis for FNNs' generalization bounds based on Rademacher complexity in Refs. [8, 55, 59], we derive specific lemmas for standard and adversarial generalization bounds in t-NNs. This is due to the unique structure of (low-rank) t-product layers. We reformulate the t-product through an operator-like expression in Lemma 16, paving the way for pivotal Lemma 17, supporting the t-product-based "peeling argument." Additionally, we introduce Lemmas 37, 38, and Lemma 33 to handle t-product layer output norms and covering low-tubal-rank tensors.
- Secondly, proving the implicit bias of GF for adversarial training of t-NNs, specifically the approximately transformed low-rankness, is nontrivial in comparison to the proof in Ref. [29] for the implicit bias of adversarial training for FNNs. As we consider more general loss functions for t-NNs in contrast to the exponential loss for FNNs in Ref. [29], we first derive a more general convergence result to the direction of a KKT point for t-NNs Lemma 9, and then goes deeper by using a constructive approach to establish the approximately transformed low-rankness in Theorem 10.
- Thirdly, differing from Ref. [43] which focuses on standard FNN generalization, our approach delves into t-NNs' adversarial generalization. We achieve this by introducing the  $(\delta, \mathbf{r})$ -parameterization, bounding localized Rademacher complexity for a Minkowski set in adversarial settings, and using low-tubal-rank approximations for tensor weights.
1 Introduction 1
2 Notations and Preliminaries 2
2.1 Tensor Singular Value Decomposition 2
2.2 Neural Networks with t-Product Layer (t-NNs) 3
3 Standard and Robust Generalization Bounds for t-NNs 4
3.1 Standard Generalization for General t-NNs 4
3.2 Robust Generalization for General t-NNs 5
4 Transformed Low-rank Parameterization for Robust Generalization 5
4.1 Robust Generalization with Exact Transformed Low-rank Parameterization 5
4.2 Implicit Bias of Gradient Flow for Adversarial Training of Over-parameterized t-NNs 6
4.3 Robust Generalization with Approximate Transformed Low-rank Parameterization 7
5 Related Works 9
6 Concluding Remarks 10
A Numerical Evaluations of the Theoretical Results 17
A.1 Effects of Exact Transformed Low-rank Weights on the Adversarial Generalization Gap 17
A.2 Implicit Bias of GF-based Adversarial Training to Approximately Transformed Low-rank Weight Tensors 17
A.3 Additional Regularization for a Better Low-rank Parameterized t-NN 19
B Notations and Preliminaries of t-SVD 20
B.1 Notations 20
B.2 Additional Preliminaries of t-SVD 21
C Standarad and Adversarial Generalization Bounds for t-NNs 22
C.1 Standardarad Generalization Bound for t-NNs 22
C.2 Adversarial Generalization Bound for t-NNs 23
C.3 Generalization Bound under Exact Low-tubal-rank Parameterization 25
D Implicit bias towards low-rankness in the transformed domain 27
D.1 Convergence to KKT points of Euclidean norm minimization in direction 29
D.2 Technical Lemmas for Proving Lemma 9 32
D.3 Proof of Theorem 10 34
E Generalization bound of approximately low-tubal-rank t-NNs 36
E.1 Several Useful Results 41
E.2 Adversarial Generalization Gap under Assumption 13 44
F Useful Notions and Lemmas 47
F.1 Tools for Analyzing General DNNs 47
F.2 Some Results for Analyzing t-NNs 49
A Numerical Evaluations of the Theoretical Results
This section presents numerical evaluations for our theoretical results. All training process is conducted on nVidia A100 GPU. For additional information and access to the demo code, please visit the following URL: https://github.com/pingzaiwang/Analysis4TNN/.
A.1 Effects of Exact Transformed Low-rank Weights on the Adversarial Generalization Gap
To validate the adversarial generalization bound in Theorem 6, we have conducted experiments on the MNIST dataset to explore the relationship between adversarial generalization gaps (AGP), weight tensor low-rankness, and training sample size. We consider binary classification of 3 and 7, with FSGM [9] attacks of strength  $20/255$ . The t-NN consists of three t-product layers and one FC layer, with weight tensor dimensions of  $28 \times 28 \times 28$  for  $\underline{\mathbf{W}}^{(1)}, \underline{\mathbf{W}}^{(2)}$ , and  $\underline{\mathbf{W}}^{(3)}$ , and 784 for the FC weight  $\mathbf{w}$ . As an input to the t-NN, each MNIST image of size  $28 \times 28$  is treated as a t-vector of size  $28 \times 1 \times 28$ .
(b)
Figure 1: The adversarial generalization gaps plotted against the training sample size  $(N)$  for t-NNs, both with and without transformed low-rank weight tensors, using the MNIST dataset. In (a), the adversarial generalization gaps are presented against the sample size  $N$ , while in (b), they are plotted against  $1 / \sqrt{N}$ .
Theorem 6 emphasizes: (i) lower weight tensor rank leads to smaller bounds on the adversarial generalization gaps, and (ii) the bound diminishes at a rate of  $O(1 / \sqrt{N})$  as  $N$  increases. We explored this by conducting experiments, controlling the upper bounds of the tubal-rank to 4 and 28 for low and full tubal-rank cases, and systematically increasing the number of training samples.
Fig. 1 presents the results. The curves indicate that t-NNs with lower rank weight tensors have smaller robust generalization errors. Interestingly, the adversarial generalization errors seem to follow a linear relationship with  $1 / \sqrt{N}$ , approximately validating the generalization error bound in Theorem 6 by approximating the scaling behavior of the empirical errors.
A.2 Implicit Bias of GF-based Adversarial Training to Approximately Transformed Low-rank Weight Tensors
We carried out experiments to confirm two theoretical statements related to the analysis of GF-based adversarial training.
Statement A.2.1 Theorem 10 reveals that, under specific conditions, well-trained t-NNs with highly over-parameterized adversarial training using GF show nearly transformed low-rank parameters.
Statement A.2.2 Lemma 22 asserts that the empirical adversarial risk approaches zero, and the F-norm of the weights grows infinitely as  $t$  approaches infinity.
In continuation of the experimental settings in Sec. A.1, we focus on binary classification on MNIST under FGSM attacks. The t-NN is structured with three t-product layers and one FC layer, with weight dimensions set to  $D \times 28 \times 28$  for  $\underline{\mathbf{W}}^{(1)}$ ,  $D \times D \times 28$  for  $\underline{\mathbf{W}}^{(2)}$  and  $\underline{\mathbf{W}}^{(3)}$ , and  $28D$  for the FC weight  $\mathbf{w}$ . Our experiments involve setting values of  $D$  to 128 and 256, respectively, and we track
the effective rank of each weight tensor, the empirical adversarial risk, and the F-norm of the weights as the number of epochs progresses. Since implementing gradient flow with infinitely small step size is impractical in real experiments, we opt for SGD with a constant learning rate and batch-size of 80, following the setting on fully connected layers in Ref. [29].
For Statement A.2.1, we present preliminary results illustrating the progression of the stable ranks of the  $M$ -block-diagonal matrix of tensor weights in Fig. 2 for the settings  $D \in \{128, 256\}$ . Notably, these results show that the effective ranks decrease as more epochs are executed, thereby confirming the influence of implicit bias on transformed low-rankness, as described in Statement A.2.1.
For Statement A.2.2, we present initial numerical findings depicting the progress of the empirical adversarial risk and the F-norm of the weights in Figs. 3 and 4, respectively. These results exhibit a consistent pattern with the theoretical descriptions outlined in Statement A.2.2 and the numerical results reported in Ref. [29] for adversarial training and Ref. [30] for standard training of FNNs. Specifically, we observe a decreasing trend in the empirical risk function and an increasing trend in the weight tensor's F-norm, which align with the expected behavior based on our theoretical framework and corroborate the numerical results presented in Refs. [29, 30].
(b)
Figure 2: Curves of the stable ranks of the  $M$ -block-diagonal matrix derived from the weight tensors, plotted against epoch numbers, using the MNIST dataset. Two t-NN size settings are showcased: (a)  $D = 128$  and (b)  $D = 256$ .
(a)
Figure 3: Curves of the adversarial training loss of t-NNs plotted against epoch numbers, using the MNIST dataset. Two t-NN size settings are showcased: (a)  $D = 128$  and (b)  $D = 256$ .
(b)
Figure 4: Curves of the F-norms of the  $M$ -block-diagonal matrix derived from the weight tensors, plotted against epoch numbers, using the MNIST dataset. Two t-NN size settings are showcased: (a)  $D = 128$  and (b)  $D = 256$ .
A.3 Additional Regularization for a Better Low-rank Parameterized t-NN
It is natural to ask: instead of using adversarial training with  $GF$  in highly over-parameterized settings to train a approximately transformed low-rank t-NN, is it possible to apply some extra regularizations in training to achieve a better low-rank parameterization?
Yes, it is possible to apply additional regularizations during training to achieve a better low-rank representation in t-NNs. Instead of relying solely on adversarial training with gradient flow in highly over-parameterized settings, these extra regularizations can potentially promote and enforce low-rankness in the network.
To validate the concern regarding the addition of an extra regularization term, we performed a preliminary experiment. In this experiment, we incorporated the tubal nuclear norm [48] as an explicit regularizer to induce low-rankness in the transformed domain. Specifically, we add the tubal nuclear norm regularization to the t-NN with three t-product layer  $D = 128$  in Sec. A.2 with a regularization parameter 0.01, and keep the other settings the same as Sec. A.2. We explore how the stable ranks of tensor weights evolve with the epoch number with/without tubal nuclear norm regularization.
The experimental results are depicted in Fig. 5. According to Fig. 5, it becomes evident that the introduction of the explicit low-rank regularizer significantly enforced low-rankness in the transform domain of the weight tensors.
Figure 5: Curves of the stable ranks of the  $M$ -block-diagonal matrix of the weight tensors versus the epoch number in the scenario where  $D = 128$ , both with and without the tubal nuclear norm regularization, using the MNIST dataset.
B Notations and Preliminaries of t-SVD
B.1 Notations
For simplicity, we use  $c$ ,  $c_0$ ,  $C$  etc. to denote constants whose values can vary from line to line. We use We first give the most commonly used notations in Table 1.
Table 1: List of main notations
Notations for t-SVD
x ∈ Rd×1×c	a t-vector	T ∈ Rm×n×c	a t-matrix
M ∈ Rc×c	an orthogonal matrix	M(·)	transform via M in Eq. (1)
*M	t-product	TM	M-block diagonal matrix of T
T::,i	i-th frontal slice of T	rt(·)	tensor tubal rank
||·||sp	tensor spectral norm	||·||F	tensor F-norm
||·||	matrix spectral norm
Notations for data representation
c	number of channels	d	number of features per channel
xi ∈ Rd×1×c	a multi-channel example	yi ∈ {±}	label of multi-channel data xi
S	training sample of size N	δi	scale invariant adv. perturbation
Ra(·)	norm used for attack	x′i	adv. perturbed version of xi
Bx	upper bound on ||x||F	ξ	radius of Ra(·) for adv. attack
Notations for network structure
L	number of t-product layers of a general t-NN
W(l)	weight tensor of l-th t-product layer with dimensionality dl × dl-1 × c
w	weight vector of fully connected layer with dimensionality cdL
f(x; W)	a general t-NN with weights W = (W(1), ..., W(L), w)
BW	bound on product of Euclidean norms of weights of f ∈ Ψ, i.e., BW = BwΠl=1LBl
Notations for model analysis
f(x, y)	adversarial version of f(x) which maps (x, y) to infRα(x′-x)≤ξ yi f(x′)
Bf	bound on the output of f(x, y) given as Bf := (Bx +ξCRa)BW
ℓ(f(x), y)	loss function with range [0, B], and Lipschitz constant Ll (See Assumption 2)
L, L	standard empirical and population risk, respectively
Ladv, Ladv	empirical and population risk, respectively
f, fadv	function class of t-NNs and its adversarial version, respectively
fR, fRadv	function class of low-tubal-rank parameterized t-NNs and adversarial version, resp.
Notations for implicit bias analysis (Sec. 4.2)
q̂i, q̂m, γ	example robust, sample robust, and smoothly normalized robust margin, resp.
f, g, bf, bg, K	auxiliary functions and constants to charaterize ℓ(·, ·) (See Assumption 2)
Notations for the analysis of approximately transformed low-rank parameterized models (Sec. 4.3)
RN, RS, Rt	average, empirical, and localized Rademacher complexity, resp.
fδ, r, fδ, r	function class of nearly low-tubal-rank parameterized t-NNs and adv. version, resp.
||f||S, ||f||L2	empirical L2-norm on sample S and population L2-norm of a function f, resp.
ε = {εi}N	i.i.d. Rademacher variables, i.e., εi equals to 1 or -1 with equal probability
B.2 Additional Preliminaries of t-SVD
We give additional notions and propositions about t-SVD omitted in the main body of the paper.
Definition 6 ([18]). The  $t$ -transpose of  $\mathbf{T} \in \mathbb{R}^{m \times n \times c}$  under the  $M$  transform in Eq. (1), denoted by  $\mathbf{T}^{\top}$ , satisfies
$$
M (\underline {{\mathbf {T}}} ^ {\top}) _ {:,: k} = \left(M (\underline {{\mathbf {T}}}) _ {:,: k}\right) ^ {\top}, k = 1, \dots , \mathsf {c}.
$$
Definition 7 ([18]). The  $t$ -identity tensor  $\mathbf{I} \in \mathbb{R}^{m \times m \times c}$  under  $M$  transform in Eq. (1) is the tensor such that each frontal slice of  $M(\mathbf{I})$  is a  $c \times c$  identity matrix, i.e,
$$
M (\mathbf {I}) _ {:,: k} = \mathbf {I}, k = 1, \dots , \mathsf {c}.
$$
Given the appropriate dimensions, it is trivial to verify that  $\mathbf{T} *_{M} \mathbf{I} = \mathbf{T}$  and  $\mathbf{I} *_{M} \mathbf{T} = \mathbf{T}$ .
Definition 8 ([18]). A tensor  $\underline{\mathbf{Q}}\in \mathbb{R}^{d\times d\times d_3}$  is  $t$ -orthogonal under  $M$  transform in Eq. (1) if it satisfies
$$
\underline {{\mathbf {Q}}} ^ {\top} * _ {M} \underline {{\mathbf {Q}}} = \underline {{\mathbf {Q}}} * _ {M} \underline {{\mathbf {Q}}} ^ {\top} = \underline {{\mathbf {I}}}.
$$
Definition 9 ([19]). A tensor is called  $f$ -diagonal if all its frontal slices are diagonal matrices.
Definition 10 (Tensor t-spectral norm [28]). The tensor  $t$ -spectral norm of any tensor  $\underline{\mathbf{T}}$  under  $M$  transform in Eq. (1) is defined as the matrix spectral norm of its  $M$ -block-diagonal matrix  $\widetilde{\mathbf{T}}_M$ , i.e.,