Multi-Mode Deep Matrix and Tensor Factorization

1 INTRODUCTION . Low-rank matrices and tensors are pervasive in sciences and engineering . Low-rank matrix completion ( LRMC ) ( Srebro & Shraibman , 2005 ; Candès & Recht , 2009 ; Recht , 2011 ; Hu et al. , 2013 ; Hardt , 2014 ; Chen et al. , 2014 ; Shamir & Shalev-Shwartz , 2014 ; Sun & Luo , 2016 ; Fan & Chow , 2017 ; Fan et al. , 2020 ; Kümmerle & Sigl , 2018 ; Fan et al. , 2019 ) and low-rank tensor completion ( LRTC ) ( Gandy et al. , 2011 ; Acar et al. , 2011 ; Liu et al. , 2012 ; Kressner et al. , 2014 ; Yuan & Zhang , 2016 ; Foster & Risteski , 2019 ) , as shown in Figure 1 , aim to recover the missing entries of a lowrank matrix or tensor . LRMC and LRTC are very usefully in data preprocessing , image and video inpainting , and collaborative filtering ( Fan & Cheng , 2018 ; Liu et al. , 2012 ) . For instance , in collaborative filtering ( e.g . the recommendation problem of Netflix ) , the rating matrix given by users on items is often highly incomplete because each user can only rate a few items and each item is often rated by a few users . If the unknown entries of the rating matrix are predicted , recommendation can then be made , accordingly , for users or items . Since rating matrices often have potentially low-rank structures , we can use LRMC or even LRTC to recover the missing entries for recommendation . Notation Throughout the paper , ‘ x ’ , ‘ x ’ , ‘ X ’ , and ‘ X ’ denote scalar , columns vector , matrix , and tensor respectively . ‘ X ( j ) ’ ( or ‘ Xj ’ ) denotes a matrix with index j . ‖ · ‖F denotes the Frobenius norm of matrix or tensor . ‖ · ‖∗ denotes the nuclear norm of matrix , i.e . sum of the singular values . ‖ · ‖∞ denotes the maximum absolute element in a matrix or tensor . ‘ C ×j A ’ denotes the jmode product of a tensor C with a matrix A. PΩ ( X ) denotes the projection onto the set Ω , i.e. , [ PΩ ( X ) ] ij = [ X ] ij if ( i , j ) ∈ Ω and [ PΩ ( X ) ] ij = 0 otherwise . |Ω| denotes the cardinality of Ω. g ( · ) denotes an activation function and hl denotes the number of units in layer l of a neural network . LRMC In the past decade , a lot of algorithms with theoretical guarantee or/and empirical success have been proposed for LRMC . For example , Candès & Recht ( 2009 ) proved that a low-rank matrix can be recovered exactly with high probability from a few entries sampled uniformly at random , via nuclear norm minimization . Nie et al . ( 2012 ) proposed to minimize the Schatten-p quasi-norm for LRMC . Nuclear norm and Schatten-p quasi-norm minimizations are based on singular value decomposition and hence have high computational cost when the size of the matrix is very large . In addition , they do not incorporate the rank prior that are possibly available in practice , though could be inaccurate . In contrast , low-rank factorization based methods ( Srebro & Shraibman , 2005 ; Srebro & Salakhutdinov , 2010 ; Wen et al. , 2012 ; Hardt , 2014 ; Sun & Luo , 2016 ; Jin et al. , 2016 ; Shang et al. , 2016 ) are scalable to big matrices and can provide high recovery accuracy if the factorization size is properly determined . In fact , regularized matrix factorizations are closely related to nuclear norm and Schatten-p quasi-norm minimizations . For example , it is known ( Srebro et al. , 2005 ; Rennie & Srebro , 2005 ) that ‖X‖∗ = minAB=X ‖A‖F ‖B‖F = minAB=X 12 ( ‖A‖2F + ‖B‖2F ) . For LRMC , we can solve minimize A , B 1 2 ‖PΩ ( Y −AB ) ‖2F + λ 2 ( ‖A‖2F + ‖B‖2F ) , ( 1 ) where A ∈ Rm×d , B ∈ Rd×n , and 0 < d ≤ min { m , n } . Interestingly , Gunasekar et al . ( 2017 ) found and proved that gradient descent for ( 1 ) with λ = 0 and d = m = n converges to the minimum nuclear norm solution , provided that the learning rate is small enough and the initializations are close enough to the origin . Fan et al . ( 2019 ) proposed a class of rank regularizers called factor group sparse regularizer ( FGSR ) as the variational forms for Schatten-p quasi-norms , e.g. , min X=AB 1 q ‖A‖q2 , q + α‖B‖2,1 = ( 1 + 1/q ) αq/ ( q+1 ) ‖X‖ q/ ( q+1 ) Sq/ ( q+1 ) , ( 2 ) where α > 0 and q ∈ { 1 , 12 , 1 4 , . . . } . When q = 1 , one gets Schatten-1/2 quasi-norm . Later , Giampouras et al . ( 2020 ) extended the variational form to arbitrary p ∈ ( 0 , 1 ] . LRTC A few LRMC methods have been directly extended to LRTC ( Acar et al. , 2011 ; Liu et al. , 2012 ; Kressner et al. , 2014 ) . One may categorize LRTC algorithms according to the different tensor factorization models such as CP ( CANDECOMP/PARAFAC ) decomposition ( Kolda & Bader , 2009 ; Jain & Oh , 2014 ) , Tucker decomposition ( Xu et al. , 2013 ; Xie et al. , 2018 ) , tensor singular value decomposition ( Kilmer et al. , 2013 ; Zhang & Aeron , 2016 ) , and tensor ring decomposition ( Zhao et al. , 2016 ; Huang et al. , 2020 ) . Take the CP decomposition based LRTC as an example , one can just learn a CP decomposition from the partial observations ( Acar et al. , 2011 ) . Yuan & Zhang ( 2016 ) provided the sample complexity for the exact recovery of LRTC via nuclear norm minimization , though the optimization is intractable because computing the nuclear norm of a tensor is NP-hard ( Hillar & Lim , 2013 ) . Barak & Moitra ( 2016 ) and Foster & Risteski ( 2019 ) exploit the sum-ofsquares hierarchy for LRTC and provided theoretical guarantees for the recovery . Razin et al . ( 2021 ) studied the implicit regularization in tensor factorization and proved that gradient descent with small learning rate and near-zero initialization gives rise to a bias towards low tensor rank solutions . Liu et al . ( 2019 ) introduced convolutional neural network to tensor completion . The method can model the possibly complex interactions inside tensors while preserving the desired low-rank structure . To complete a third-order ( w.l.g ) tensor based on the Tucker decomposition , we may solve minimize X , A1 , A2 , A3 , C 1 2 ‖X − C ×1 A1 ×2 A2 ×3 A3‖2F , subject to PΩ ( X ) = PΩ ( Y ) , ( 3 ) where we can also consider regularization or constraint on the factors A1 , A2 , A3 , and C. For instance , Xu et al . ( 2013 ) reformulated ( 3 ) to all-mode matricization and solved the problem by alternating minimization . Xie et al . ( 2018 ) proposed to minimize the number of nonzero elements of C and the ranks of the matricizations of X , in addition to ( 3 ) . Deep Matrix Factorization Recently , deep matrix factorization ( DMF ) ( Fan & Cheng , 2018 ; Arora et al. , 2019 ) gains increasing attention in machine learning . There are two types of DMF methods : linear DMF ( Trigeorgis et al. , 2016 ; Zhao et al. , 2017 ; Arora et al. , 2019 ) and nonlinear DMF ( Xue et al. , 2017 ; Wang et al. , 2017 ; Fan & Cheng , 2018 ) . A general formulation is X ≈ g1 ( A1g2 ( A2 · · · gL−1 ( AL−1AL ) · · · ) ) , ( 4 ) where { gi } L−1l=1 are activation functions . When all gi are linear , ( 4 ) is linear DMF . Otherwise , ( 4 ) is nonlinear DMF . Arora et al . ( 2019 ) found that adding depth to a linear matrix factorization can enhance an implicit tendency towards low-rank and can provide higher recovery accuracy than depth-2 factorization , though the implicit regularization in linear DMF may not be captured using simple mathematical norms . In ( Fan & Cheng , 2018 ) , the numerical results showed that nonlinear DMF is able to recover ( with high accuracy ) the missing entries of a full-rank matrix of which the columns are generated by a nonlinear low-dimensional latent variable model . Contributions of this paper This paper first provides theoretical analysis for why and when nonlinear DMF outperforms linear DMF in matrix completion . Second , this paper proposes a new matrix factorization method called two-mode nonlinear DMF . The method can exploit the full nonlinearity of the data while classical nonlinear DMF uses only partial nonlinearity of the data . Third , the paper presents a new tensor decomposition method called multi-mode nonlinear deep tensor factorization , which can be regarded as a high-order generalization of the two-mode nonlinear DMF . Finally , the paper shows that the multi-mode deep matrix and tensor factorization methods have tighter generalization error bounds than conventional factorization methods in matrix and tensor completion . Extensive numerical results verified the effectiveness of the proposed methods . 2 WHY AND WHEN DOES NONLINEAR DMF OUTPERFORM MF ? . Nonlinear DMF methods can outperform linear MF methods empirically in many real tasks such as collaborative filtering ( Xue et al. , 2017 ; Fan & Cheng , 2018 ) , image inpainting ( Fan & Cheng , 2018 ) , and disease modeling ( Wang et al. , 2017 ) . The fundamental reason is that the data matrices in these tasks indeed have some nonlinear structures that can not be exploited by linear MF methods . Nevertheless , in nonlinear DMF , the theoretical guarantee is very limited . In contrast , linear MF or even linear DMF have been well studied and guaranteed with sample complexity ( Hardt , 2014 ; Sun & Luo , 2016 ; Jin et al. , 2016 ) , generalization bound ( Srebro & Shraibman , 2005 ; Fan et al. , 2019 ) , or convergence ( Gunasekar et al. , 2017 ; Arora et al. , 2019 ) . In this section , we will provide some theoretical guarantee for nonlinear DMF in matrix completion . Assumption 1 . Suppose Y = X + E , where Y ∈ Rn1×n2 . The columns of X are generated by x = f ( z ) , where f : Rd → Rn1 denotes a nonlinear mapping , d < min ( n1 , n2 ) , and z ∈ Rd denotes a latent variable . The entries of the noise matrix E are drawn from N ( 0 , σ2 ) . Suppose we observed a few entries of Y randomly ( sampling without replacement ) and want to recover X from the incomplete Y . Consider the following method ( Fan & Cheng , 2018 ) : minimize W1 , ... , WL , Z 1 2 ‖PΩ ( Y −WLg ( WL−1 · · · g ( W1Z ) · · · ) ) ‖2F + λw 2 L∑ l=1 ‖Wl‖2F + λz 2 ‖Z‖2F , ( 5 ) where Z ∈ Rd×n2 , Wl ∈ Rhl×hl−1 , l = 1 , . . . , L , hL = n1 , h0 = d , and h−1 = n2 . λw and λz are regularization parameters . Here we have let all activation functions be the same without loss of generality and omitted the bias terms of the neural network for simplicity . According to the universal approximation theorems ( Cybenko , 1989 ; Pinkus , 1999 ; Lu et al. , 2017 ; Hanin & Sellke , 2017 ) , f can be well approximated by the neural network provided that L is sufficiently large or at least one of h1 , . . . , hL−1 is sufficiently large . The following theorem provides the excess risk bound for ( 5 ) . Theorem 1 . Suppose Z and { Wl } Ll=1 are given by ( 5 ) . Let X̂ = WLg ( WL−1 · · · g ( W1Z ) · · · ) . Suppose ‖Z‖F ∏L l=1 ‖Wl‖F ≤ κ , max ( ‖Y ‖∞ , ‖X̂‖∞ ) ≤ ξ , the Lipschitz constant of g is η . Let N = n1n2 . Then with probability at least 1− 2N−1 , there exists a numerical constant c such that 1√ N ‖X − X̂‖F ≤ 1√ |Ω| ‖PΩ ( Y − X̂ ) ‖F + 1√ N ‖E‖F + cξ ( ∑L l=0 hlhl−1 log ( η L−1ξ−1κ ) |Ω| ) 1/4 . In Theorem 1 , the last term of the RHS of the inequality is equivalent to ψ1 = cξ ( |Ω|−1 ( n2d+ hL−1n1 + L−1∑ l=1 hlhl−1 ) log ( η L−1ξ−1κ ) ) 1/4 . ( 6 ) Most major activation functions ( e.g . ReLU and sigmoid ) are , at worst , 1-Lipschitz with respect to ` 2 norm , which indicates that ηL−1 is not large . When n2 is large enough , ψ1 is linear with the latent dimension d of the columns of X when using ( 5 ) . In practice , it is difficult to know the true d in advance . But the Frobenius norm regularizations on Z and W1 are able to reduce the nuclear norm or even the rank of W1Z theoretically , which means the d in formulation ( 5 ) and Theorem 1 can be larger than the d ( ground truth ) in Assumption 1 . Note that classical MF ( depth-1 ) is a special case of ( 5 ) when L = 1 and g is linear . However , because of the nonlinearity in the data generating model of Assumption 1 , the rank of X can be much higher than the intrinsic dimension of the data , i.e . r : = rank ( X ) d. For example , if f is a q-order polynomial function , r can be as large as ( d+q q ) . For classical MF , the last term of the RHS of the inequality in Theorem 1 reduces to ψ2 = cξ ( |Ω|−1 ( n1r + rn2 ) log ( ξ−1κ′ ) ) 1/4 . ( 7 ) Here κ′ may be less than the κ in ( 6 ) but they are in the log operator . Suppose all the Frobenius norms are the same ( the worst case , may not happen because the norms of the factors in a deep factorization are usually smaller than the norms of the factors in a shallow factorization ) , denoted by β . Then κ = βL+1 and κ′ = β2 , but r d. In the experiments , we observed that κ ≈ ( κ′ ) 1.3 when L = 2 and κ ≈ ( κ′ ) 1.7 when L = 3 , which are much better than the worst cases . Now comparing ( 7 ) with ( 6 ) , we conclude that the nonlinear DMF provides a tighter generalization bound than the classical MF when n2 is much larger than n1 and ∑L−1 l=1 hlhl−1 is not too large . The second condition is realistic because we usually let h1 , . . . , hL−1 < n1 or we can just assume h1 , . . . , hL−1 < r. Consider the case that n2 is not large compared to n1 , e.g . n2 = n1 . In ( 6 ) , suppose d = r/2 , L = 3 , h1 = 2d , h2 = 3d , and η = 1 , we have ψ1 = cξ ( |Ω|−1 ( 2n2r + 2r2 ) log ( κ ) ) 1/4 and ψ2 = cξ ( |Ω|−1 ( 2n2r ) log ( κ′ ) ) 1/4 , where ψ1 > ψ2 . It means the upper bound given by the classical MF is tighter than that given by the nonlinear DMF . In general , the superiority of the nonlinear DMF over the classical MF increases when n2/n1 increases , which is consistent with the numerical results shown in Figure 5 of the appendix .

Summary: The paper provides a multi-mode framework for the deep learning based tensor decomposition, which could be useful for dealing with nonlinear high-dimensional data sets. In particular, it extends the deep matrix factorization (DMF) method and proposes a multi-mode deep matrix factorization method for matrix completion with convergence guarantee. Based on this, it also develops a multi-mode nonlinear deep tensor factorization method with convergence guarantee. The proposed models are solved by various optimization algorithms. Numerical experiments on synthetic and real data sets of the matrix/tensor form have shown that the proposed methods outperform other state of the arts.