Adaptive Learning of Tensor Network Structures

1 INTRODUCTION . Matrix factorization is ubiquitous in machine learning and data science and forms the backbone of many algorithms . Tensor decomposition techniques emerged as a powerful generalization of matrix factorization . They are particularly suited to handle high-dimensional multi-modal data and have been successfully applied in neuroimaging [ 44 ] , signal processing [ 3 , 33 ] , spatio-temporal analysis [ 1 , 31 ] and computer vision [ 20 ] . Common tensor learning tasks include tensor decomposition ( finding a low-rank approximation of a given tensor ) , tensor regression ( which extends linear regression to the multi-linear setting ) , and tensor completion ( inferring a tensor from a subset of observed entries ) . Akin to matrix factorization , tensor methods rely on factorizing a high-order tensor into small factors . However , in contrast with matrices , there are many different ways of decomposing a tensor , each one giving rise to a different notion of rank , including CP , Tucker , Tensor Train ( TT ) and Tensor Ring ( TR ) . For most tensor learning problems , there is no clear way of choosing which decomposition model to use , and the cost of model mis-specification can be high . It may even be the case that none of the commonly used models is suited for the task , and new decomposition models would achieve better tradeoffs between minimizing the number of parameters and minimizing a given loss function . We propose an adaptive tensor learning algorithm which is agnostic to decomposition models . Our approach relies on the tensor network formalism , which has shown great success in the many-body physics community [ 28 , 7 , 6 ] and has recently demonstrated its potential in machine learning for compressing models [ 24 , 39 , 8 , 23 , 14 , 41 ] , developing new insights into the expressiveness of deep neural networks [ 4 , 15 ] , and designing novel approaches to supervised [ 35 , 9 ] and unsupervised [ 34 , 11 , 22 ] learning . Tensor networks offer a unifying view of tensor decomposition models , allowing one to reason about tensor factorization in a general manner , without focusing on a particular model . In this work , we design a greedy algorithm to efficiently search the space of tensor network structures for common tensor problems , including decomposition , completion and model compression . We start by considering the novel tensor optimization problem of minimizing a loss over arbitrary tensor network structures under a constraint on the number of parameters . To the best of our knowledge , this is the first time that this problem is considered . The resulting problem is a bi-level optimization problem where the upper level is a discrete optimization over tensor network structures , and the lower level is a continuous optimization of a given loss function . We propose a greedy approach to optimize the upper-level problem , which is combined with continuous optimization techniques to optimize the lower-level problem . Starting from a rank one initialization , the greedy algorithm successively identifies the most promising edge of a tensor network for a rank increment , making it possible to adaptively identify from data the tensor network structure which is best suited for the task at hand . The greedy algorithm we propose is conceptually simple , and experiments on tensor decomposition , completion and model compression tasks showcase its effectiveness . Our algorithm significantly outperforms a recent evolutionary algorithm [ 19 ] for tensor network decomposition on an image compression task by discovering structures that require less parameters while simultaneously achieving lower recovery errors . The greedy algorithm also outperforms CP , Tucker , TT and TR algorithms on an image completion task and finds more efficient TN structures to compress fully connected layers in neural networks than the TT based method introduced in [ 24 ] . Related work Adaptive tensor learning algorithms have been previously proposed , but they only consider determining the rank ( s ) of a specific decomposition and are often tailored to a specific tensor learning task ( e.g. , decomposition or regression ) . In [ 1 ] , a greedy algorithm is proposed to adaptively find the ranks of a Tucker decomposition for a spatio-temporal forecasting task , and in [ 38 ] an adaptive Tucker based algorithm is proposed for background subtraction . In [ 42 ] , the authors present a Bayesian approach for automatically determining the rank of a CP decomposition . In [ 2 ] an adaptive algorithm for tensor decomposition in the hierarchical Tucker format is proposed . In [ 10 ] a stable rank-adaptive alternating least square algorithm is introduced for completion in the TT format . The problem we consider is considerably more general since we do not assume a fixed tensor network structure ( e.g . Tucker , TT , CP , etc. ) . Exploring other decomposition relying on the tensor network formalism has been sporadically explored . The work which is the most closely related to our contribution is [ 19 ] where evolutionary algorithms are used to approximate the best tensor network structure to exactly decompose a given target tensor . However , the method proposed in [ 19 ] only searches for TN structures with uniform ranks ( with the rank being a hyperparameter ) and is limited to the problem of tensor decomposition . In contrast , our method is the first to jointly explore the space of structures and ( non-uniform ) ranks to minimize an arbitrary loss function over the space of tensor parameters . Lastly , [ 12 ] proposes to explore the space of tensor network structures for compressing neural networks , a rounding algorithm for general tensor networks is proposed in [ 21 ] and the notions of rank induced by arbitrary tensor networks are studied in [ 40 ] . 2 PRELIMINARIES . In this section , we present notions of tensor algebra and tensor networks . We first introduce notations . For any integer k , [ k ] denotes the set of integers from 1 to k. We use lower case bold letters for vectors ( e.g . v 2 Rd1 ) , upper case bold letters for matrices ( e.g . M 2 Rd1⇥d2 ) and bold calligraphic letters for higher order tensors ( e.g . T 2 Rd1⇥d2⇥d3 ) . The ith row ( resp . column ) of a matrix M will be denoted by Mi , : ( resp . M : ,i ) . This notation is extended to slices of a tensor in the obvious way . Tensors and tensor networks We first recall basic definitions of tensor algebra ; more details can be found in [ 17 ] . A tensor T 2 Rd1⇥···⇥dp can simply be seen as a multidimensional array ( T i1 , ··· , ip : in 2 [ dn ] , n 2 [ p ] ) . The inner product of two tensors is defined by hS , T i =P i1 , ··· , ip Si1···ipT i1···ip and the Frobenius norm of a tensor is defined by kT k 2 F = hT , T i . The mode-n matrix product of a tensor T and a matrix X 2 Rm⇥dn is a tensor denoted by T ⇥n X . It is of size d1⇥ · · ·⇥dn 1⇥m⇥dn+1⇥ · · ·⇥dp and is obtained by contracting the nth mode of T with the second mode of X , e.g . for a 3rd order tensor T , we have ( T ⇥2X ) i1i2i3 = P j T i1ji3Xi2j . The nth mode matricization of T is denoted by T ( n ) 2 Rdn⇥ Q i 6=n di . Tensor network diagrams allow one to represent complex operations on tensors in a graphical and intuitive way . A tensor network ( TN ) is simply a graph where nodes represent tensors , and edges represent contractions between tensor modes , i.e . a summation over an index shared by two tensors . In a tensor network , the arity of a vertex ( i.e . the number of legs of a node ) corresponds to the order of the tensor ( see Figure 1 ) . Connecting two legs in a tensor network represents a contraction over the corresponding indices . Consider the following simple tensor network with two nodes : A x m n . The first node represents a matrix A 2 Rm⇥n and the second one a vector x 2 Rn . Since this tensor network has one dangling leg ( i.e . an edge which is not connected to any other node ) , it represents a first order tensor , i.e. , a vector . The edge between the second leg of A and the leg of x corresponds to a contraction between the second mode of A and the first mode of x . Hence , the resulting tensor network represents the classical matrix-vector product , which can be seen by calculating the ith component of this tensor network : A xi = P j Aijxj = ( Ax ) i . Other examples of tensor network representations of common operations on matrices and tensors can be found in Figure 2 . Lastly , it is worth mentioning that disconnected tensor networks correspond to tensor products , e.g. , u v = uv > is the outer product of u and v with components u vi j= uivj . Consequently , an edge of dimension ( or rank ) 1 in a TN is equivalent to having no edge between the two nodes , e.g. , if R = 1 we have A Bi R j = PR r=1 Ai , rBr , j = Ai,1B1 , j = A Bi j. Tensor decomposition and tensor rank We now briefly present the most common tensor decomposition models , omitting the CP decomposition which can not be described using the TN formalism unless hyper-edges are allowed ( which we do not consider in this work ) . For the sake of simplicity we consider a fourth order tensor T 2 Rd1⇥d2⇥d3⇥d4 , each decomposition can be straightforwardly extended to higher-order tensors . A Tucker decomposition [ 36 ] decomposes T as the product of a core tensor G 2 RR1⇥R2⇥R3⇥R4 with four factor matrices Ui 2 Rdi⇥Ri for i = 1 , · · · , 4 : T = G ⇥1 U1 ⇥2 U2 ⇥3 U3 ⇥4 U4 . The Tucker rank , or multilinear rank , of T is the smallest tuple ( R1 , R2 , R3 , R4 ) for which such a decomposition exists . The tensor ring ( TR ) decomposition [ 43 , 25 , 29 ] expresses each component of T as the trace of a product of slices of four core tensors G ( 1 ) 2 RR0⇥d1⇥R1 , G ( 2 ) 2 RR1⇥d2⇥R2 , G ( 3 ) 2 RR2⇥d3⇥R3 and G ( 4 ) 2 RR4⇥d4⇥R0 : T i1 , i2 , i3 , i4 = Tr ( G ( 1 ) : ,i1 , : G ( 2 ) : ,i2 , :G ( 3 ) : ,i3 , : G ( 4 ) : ,i4 , : ) . The tensor train ( TT ) decomposition [ 26 ] ( also known as matrix product states in the physics community ) is a particular case of the tensor ring decomposition where R0 must be equal to 1 ( R0 is thus omitted when referring to the rank of a TT decomposition ) . Similarly to Tucker , the TT and TR decompositions naturally give rise to an associated notion of rank : the TR rank ( resp . TT rank ) is the smallest tuple ( R0 , R1 , R2 , R3 ) ( resp . ( R1 , R2 , R3 ) ) such that a TR ( resp . TT ) decomposition exists . Tensor networks offer a unifying view of tensor decomposition models : Figure 3 shows the TN representation of common models . Each decomposition is naturally associated with the graph topology of the underlying TN . For example , the Tucker decomposition corresponds to star graphs , the TT decomposition corresponds to chain graphs , and the TR decomposition model corresponds to cyclic graphs . The relation between the rank of a decomposition and its number of parameters is different for each model . Letting p be the order of the tensor , d its largest dimension and R the rank of the decomposition ( assuming uniform ranks ) , the number of parameters is in O ( Rp + pdR ) for Tucker , and O pdR2 for TT and TR . One can see that the Tucker decomposition is not well suited for tensors of very high order since the size of the core tensor grows exponentially with p .

Summary: First, the idea of rank incremental method for Tensor network decomposition and determination of TN structures is not novel.  Second, decomposition with weight transfer is obvious and widely used in the rank incremental method.  For generalized tensor networks decomposition, the authors should consider the works.  S.  Handschuh,  “Numerical  Methods  in  Tensor  Networks,” PhD  thesis,Facualty of Mathematics and Informatics, University Leipzig, Germany,Leipzig, Germany, 2015.  Mike Espig, Wolfgang Hackbusch, Stefan Handschuh, and Reinhold Schneider,   Optimization Problems in Contracted Tensor Networks, 2012