Lower and Upper Bounds on the Pseudo-Dimension of Tensor Network Models

1 Introduction . Tensor networks ( TNs ) have emerged in the quantum physics community as a mean to compactly represent wave functions of large quantum systems [ 45 , 5 , 52 ] . Their introduction in physics can be traced back to the work of Penrose [ 47 ] and Feynman [ 15 ] . Akin to matrix factorization , TN methods rely on factorizing a high-order tensor into small factors and have recently gained interest from the machine learning community for their ability to efficiently represent and perform operations on very high-dimensional data and high-order tensors . They have been for example successfully used for compressing models [ 43 , 69 , 42 , 29 , 70 ] , developing new insights on the expressiveness of deep neural networks [ 14 , 31 ] and designing novel approaches to supervised [ 56 , 18 ] and unsupervised [ 55 , 25 , 39 ] learning . Most of these approaches leverage the fact that TN can be used to efficiently parameterize high-dimensional linear maps , which is appealing from two perspectives : it makes it possible to learn models in exponentially large feature spaces and it acts as a regularizer , controlling the capacity of the class of hypotheses considered for learning . While the expressive power of TN models has been studied recently [ 17 , 2 ] , the focus has mainly been on the representation capacity of TN models , but not on their ability to generalize in the context of supervised learning tasks . In this work , we study the generalization ability of TN models by deriving lower and upper bounds on the VC-dimension and pseudo-dimension of TN models commonly used for classification , completion and regression , from which bounds on the generalization gap of TN models can be derived . Using the general framework of tensor networks , we derive a general upper bound for models parameterized by arbitrary TN structures , which applies to all commonly used tensor decomposition models [ 20 ] such as CP [ 27 ] , Tucker [ 59 ] and tensor train ( TT ) ) [ 46 ] , as well 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as more sophisticated structures including hierarchical Tucker [ 19 , 23 ] , tensor ring ( TR ) [ 73 ] and projected entangled state pairs ( PEPS ) [ 60 ] . Our analysis proceeds mainly in two steps . First , we formally define the notion of TN learning model by disentangling the underlying graph structure of a TN from its parameters ( the core tensors , or factors , involved in the decomposition ) . This allows us to define , in a conceptually simple way , the hypothesis class HG corresponding to the family of linear models whose weights are represented using an arbitrary TN structure G. We then proceed to deriving upper bounds on the VC/pseudodimension and generalization error of the class HG . These bounds follow from a classical result from Warren [ 66 ] which was previously used to obtain generalization bounds for neural networks [ 3 ] , matrix completion [ 54 ] and tensor completion [ 41 ] . The bounds we derive naturally relate the capacity of HG to the underlying graph structure G through the number of nodes and effective number of parameters of the TN . To assess the tightness of our general upper bound , we derive lower bounds for particular TN structures ( rank-one , CP , Tucker , TT and TR ) . These lower bounds show that , for completion , regression and classification , our general upper bound is tight up to a log factor for rank-one , TT and TR tensors , and is tight up to a constant for matrices . Lastly , as a corollary of our results , we obtain a bound on the VC-dimension of the tensor train classifier introduced in [ 56 ] , which answers one of the open problems listed by Cirac , Garre-Rubio and Pérez-García in [ 13 ] . Related work Machine learning models using low-rank parametrization of the weights have been investigated ( mainly from a practical perspective ) for various decomposition models , including low-rank matrices [ 36 , 49 , 67 ] , CP [ 1 , 37 , 7 ] , Tucker [ 35 , 16 , 26 , 50 ] , tensor train [ 48 , 10 , 44 , 56 , 18 , 53 , 11 , 65 , 68 ] and PEPS [ 12 ] . From a more theoretical perspective , generalization bounds for matrix and tensor completion have been derived in [ 54 , 41 ] ( based on the Tucker format for the tensor case ) . A bound on the VC-dimension of low-rank matrix classifiers was derived in [ 67 ] and a bound on the pseudo-dimension of regression functions whose weights have low Tucker rank was given in [ 50 ] ( for both these cases , we show that our results improve over these previous bounds , see Section 4.2 ) . To the best of our knowledge the VC-dimension of tensor train classifiers has not been studied in the past , but the statistical consistency of the convex relaxation of the tensor completion problem was studied in [ 58 , 57 ] for the Tucker decomposition and in [ 28 ] for the tensor train decomposition . Lastly , in [ 38 ] the authors study the complexity of learning with tree tensor networks using the notion of metric entropy and covering numbers . They provide generalization bounds which are qualitatively similar to ours , but their results only hold for TN structures whose underlying graph is a tree ( thus excluding models such as CP , tensor ring and PEPS ) and they do not provide lower bounds . Summary of contributions We introduce a unifying framework for TN-based learning models , which generalizes a wide range of models based on tensor factorization for completion , classification and regression . This framework allows us to consider the class HG of low-rank TN models for a given arbitrary TN structure G ( Section 3 ) . We provide general upper bounds on the pseudo-dimension and VC-dimension of the hypothesis class HG for arbitrary TN structure G for regression , classification and completion . Our results naturally relate the capacity of HG to the number of parameters of the underlying TN structure G ( Section 4.1 ) . From these results , we derive a generalization bound for TN-based classifiers parameterized by arbitrary TN structures ( Theorem 4 ) . We compare our results to previous bounds for specific decomposition models and show that our general upper bound is always of the same order and sometimes even improves on previous bounds ( Section 4.2 ) . We derive several lower bounds showing that our general upper bound is tight up to a log factor for particular TN structures ( Section 5 ) . A summary of the lower bounds derived in this work , as well as upper bounds implied by our general result for particular TN structures , can be found in Table 1 at the end of the paper . 2 Preliminaries . In this section , we present basic notions of tensor algebra and tensor networks as well as generalization bounds based on combinatorial complexity measures . We start by introducing some notations . For any integer k we use [ k ] to denote the set of integers from 1 to k. We use lower case bold letters for vectors ( e.g . v 2 Rd1 ) , upper case bold letters for matrices ( e.g . M 2 Rd1⇥d2 ) and bold calligraphic letters for higher order tensors ( e.g . T 2 Rd1⇥d2⇥d3 ) . The inner product of two k-th order tensors S , T 2 Rd1⇥···⇥dk is defined by hT , Si = Pd1 i1=1 · · · Pdk ik=1 T i1 ... ikSi1 ... ik . The outer product of two vectors u 2 Rd1 and v 2 Rd2 is denoted by u⌦ v 2 Rd1⇥d2 with elements ( u⌦ v ) i , j = uivj . The outer product generalizes to an arbitrary number of vectors . We use the notation ( Rd ) ⌦p to denote the space of p-th order hypercubic tensors of size d⇥ d⇥ · · ·⇥ d. We denote by YX the space of functions f : X 7 ! Y . sign ( · ) stands for the sign function . Finally , given a graph G = ( V , E ) and a vertex v 2 V , we denote by Ev = { e 2 E | v 2 e } the set of edges incident to the vertex v . 2.1 Tensors and Tensor Networks . Tensor networks A tensor T 2 Rd1⇥···⇥dp can simply be seen as a multidimensional array ( T i1 , ··· , ip : in 2 [ dn ] , n 2 [ p ] ) . Complex operations on tensors can be intuitively represented using the graphical notation of tensor network ( TN ) diagrams [ 5 , 45 ] . In tensor networks , a p-th order tensor is illustrated as a node with p edges ( or legs ) in a graph Td1 d2 dp . An edge between two nodes of a TN represents a contraction over the corresponding modes of the two tensors . Consider the following simple TN with two nodes : A x m n . The first node represents a matrix A 2 Rm⇥n and the second one a vector x 2 Rn . Since this TN has one dangling leg ( i.e . an edge which is not connected to any other node ) , it represents a first order tensor , i.e . a vector . The edge between the second leg of A and the leg of x corresponds to a contraction between the second mode of A and the first mode of x . Hence , the resulting TN represents the classical matrix-product , which can be seen by calculating the i-th component of this TN : A xi = P j Aijxj = ( Ax ) i . Other examples of TN representations of common operations on matrices and vectors can be found in Figure 1 . A special case of TN is the tensor train decomposition [ 46 ] which factorizes a n-th order tensor T in the form G1 d1 r1 G2 d2 r2 Gn-1 rn 2 dn 1 rn 1 Gn dn . This corresponds to T i1 , i2 , ... , in = r1X ↵1=1 · · · rn 1X ↵n 1=1 ( G1 ) i1 , ↵1 ( G2 ) ↵1 , i2 , ↵2 . . . ( Gn 1 ) ↵n 2 , in 1 , ↵n 1 ( Gn ) ↵n 1 , in ( 1 ) where the tuple ( ri ) n 1i=1 associated with the TT representation is called TT-rank . Tensor network structures A tensor network ( TN ) can be fundamentally decomposed in two constituent parts : a tensor network structure , which describes its graphical structure , and a set of core tensors assigned to each node . For example , the tensor in Rd1⇥d2⇥d3⇥d4 represented by the TN Td1 d2 R S d3 d4 is obtained by assigning the core tensors T 2 Rd1⇥d2⇥R and S 2 RR⇥d3⇥d4 to the nodes of the TN structure d1 d2 R d3 d4 . Formally , a tensor network structure is given by a graph G = ( V , E , dim ) where edges are labeled by integers : V is the set of vertices , E ⇢ V [ ( V ⇥ V ) is a set of edges containing both classical edges ( e 2 V ⇥ V ) and singleton edges ( e 2 V ) and dim : E ! N assigns a dimension to each edge in the graph . The set of singleton edges G = E \ V corresponds to the dangling legs of a TN . Given a TN structure G , one obtains a tensor by assigning a core tensor T v 2 N e2Ev Rdim ( e ) to each vertex v in the graph , where Ev = { e 2 E | v 2 e } . The resulting tensor , denoted by TN ( G , { T v } v2V ) , is a tensor of order | G| in the tensor product space N e2 G Rdim ( e ) . Given a tensor structure G = ( V , E , dim ) , the set of all tensors that can be obtained by assigning core tensors to the vertices of G is denoted by T ( G ) ⇢ N e2 G Rdim ( e ) : T ( G ) = { TN ( G , { T v } v2V ) : T v 2 O e2Ev Rdim ( e ) , v 2 V } . ( 2 ) As an illustration , one can check that the set of m ⇥ n matrices of rank at most r is equal to T ( m r n ) . Similarly , the set of 4th order d-dimensional tensors of TT rank at most r is equal to T ( d r d r d r d ) . Finally , for a given graph structure G , the number of parameters of any member of the family T ( G ) in Equation ( 2 ) ( which is the total number of entries of the core tensors { T v } v2V ) is given by NG = X v2V Y e2Ev dim ( e ) ( 3 ) This will be a central quantity in the generalization bounds and bounds on the VC-dimension of TN models we derive in Section 4 . Common tensor network structures In Figure 2 , we show the tensor network structures associated with classical tensor decomposition models such as CP , Tucker [ 59 ] and tensor train ( TT ) [ 46 ] , also known as matrix product state ( MPS ) [ 45 , 52 ] . For the case of the Candecomp/Parafac ( CP ) decomposition [ 27 ] , note that the TN structure is a hyper-graph rather than a graph . We introduced the notion of TN structure focusing on graphs for clarity of exposition in the previous paragraph , but our formalism and results can be straightforwardly extended to hyper-graph TN structures . In addition , we include the tensor ring ( TR ) [ 73 ] ( also known as periodic MPS ) and PEPS decompositions which have initially emerged in quantum physics and recently gained interest in the machine learning community ( see e.g. , [ 12 , 62 , 63 , 71 ] ) . We also show the hierarchical Tucker decomposition initially introduced in [ 19 , 23 ] .

Summary: This work considers the tensor network methods in machine learning. The major contributions are some new lower and upper bounds on the pseudo-dimension for a large class of TN models. The derived results can be applied to machine learning tasks which involves the tensor decomposition models.