Multi-Mode Deep Matrix and Tensor Factorization

1 INTRODUCTION . Low-rank matrices and tensors are pervasive in sciences and engineering . Low-rank matrix completion ( LRMC ) ( Srebro & Shraibman , 2005 ; Candès & Recht , 2009 ; Recht , 2011 ; Hu et al. , 2013 ; Hardt , 2014 ; Chen et al. , 2014 ; Shamir & Shalev-Shwartz , 2014 ; Sun & Luo , 2016 ; Fan & Chow , 2017 ; Fan et al. , 2020 ; Kümmerle & Sigl , 2018 ; Fan et al. , 2019 ) and low-rank tensor completion ( LRTC ) ( Gandy et al. , 2011 ; Acar et al. , 2011 ; Liu et al. , 2012 ; Kressner et al. , 2014 ; Yuan & Zhang , 2016 ; Foster & Risteski , 2019 ) , as shown in Figure 1 , aim to recover the missing entries of a lowrank matrix or tensor . LRMC and LRTC are very usefully in data preprocessing , image and video inpainting , and collaborative filtering ( Fan & Cheng , 2018 ; Liu et al. , 2012 ) . For instance , in collaborative filtering ( e.g . the recommendation problem of Netflix ) , the rating matrix given by users on items is often highly incomplete because each user can only rate a few items and each item is often rated by a few users . If the unknown entries of the rating matrix are predicted , recommendation can then be made , accordingly , for users or items . Since rating matrices often have potentially low-rank structures , we can use LRMC or even LRTC to recover the missing entries for recommendation . Notation Throughout the paper , ‘ x ’ , ‘ x ’ , ‘ X ’ , and ‘ X ’ denote scalar , columns vector , matrix , and tensor respectively . ‘ X ( j ) ’ ( or ‘ Xj ’ ) denotes a matrix with index j . ‖ · ‖F denotes the Frobenius norm of matrix or tensor . ‖ · ‖∗ denotes the nuclear norm of matrix , i.e . sum of the singular values . ‖ · ‖∞ denotes the maximum absolute element in a matrix or tensor . ‘ C ×j A ’ denotes the jmode product of a tensor C with a matrix A. PΩ ( X ) denotes the projection onto the set Ω , i.e. , [ PΩ ( X ) ] ij = [ X ] ij if ( i , j ) ∈ Ω and [ PΩ ( X ) ] ij = 0 otherwise . |Ω| denotes the cardinality of Ω. g ( · ) denotes an activation function and hl denotes the number of units in layer l of a neural network . LRMC In the past decade , a lot of algorithms with theoretical guarantee or/and empirical success have been proposed for LRMC . For example , Candès & Recht ( 2009 ) proved that a low-rank matrix can be recovered exactly with high probability from a few entries sampled uniformly at random , via nuclear norm minimization . Nie et al . ( 2012 ) proposed to minimize the Schatten-p quasi-norm for LRMC . Nuclear norm and Schatten-p quasi-norm minimizations are based on singular value decomposition and hence have high computational cost when the size of the matrix is very large . In addition , they do not incorporate the rank prior that are possibly available in practice , though could be inaccurate . In contrast , low-rank factorization based methods ( Srebro & Shraibman , 2005 ; Srebro & Salakhutdinov , 2010 ; Wen et al. , 2012 ; Hardt , 2014 ; Sun & Luo , 2016 ; Jin et al. , 2016 ; Shang et al. , 2016 ) are scalable to big matrices and can provide high recovery accuracy if the factorization size is properly determined . In fact , regularized matrix factorizations are closely related to nuclear norm and Schatten-p quasi-norm minimizations . For example , it is known ( Srebro et al. , 2005 ; Rennie & Srebro , 2005 ) that ‖X‖∗ = minAB=X ‖A‖F ‖B‖F = minAB=X 12 ( ‖A‖2F + ‖B‖2F ) . For LRMC , we can solve minimize A , B 1 2 ‖PΩ ( Y −AB ) ‖2F + λ 2 ( ‖A‖2F + ‖B‖2F ) , ( 1 ) where A ∈ Rm×d , B ∈ Rd×n , and 0 < d ≤ min { m , n } . Interestingly , Gunasekar et al . ( 2017 ) found and proved that gradient descent for ( 1 ) with λ = 0 and d = m = n converges to the minimum nuclear norm solution , provided that the learning rate is small enough and the initializations are close enough to the origin . Fan et al . ( 2019 ) proposed a class of rank regularizers called factor group sparse regularizer ( FGSR ) as the variational forms for Schatten-p quasi-norms , e.g. , min X=AB 1 q ‖A‖q2 , q + α‖B‖2,1 = ( 1 + 1/q ) αq/ ( q+1 ) ‖X‖ q/ ( q+1 ) Sq/ ( q+1 ) , ( 2 ) where α > 0 and q ∈ { 1 , 12 , 1 4 , . . . } . When q = 1 , one gets Schatten-1/2 quasi-norm . Later , Giampouras et al . ( 2020 ) extended the variational form to arbitrary p ∈ ( 0 , 1 ] . LRTC A few LRMC methods have been directly extended to LRTC ( Acar et al. , 2011 ; Liu et al. , 2012 ; Kressner et al. , 2014 ) . One may categorize LRTC algorithms according to the different tensor factorization models such as CP ( CANDECOMP/PARAFAC ) decomposition ( Kolda & Bader , 2009 ; Jain & Oh , 2014 ) , Tucker decomposition ( Xu et al. , 2013 ; Xie et al. , 2018 ) , tensor singular value decomposition ( Kilmer et al. , 2013 ; Zhang & Aeron , 2016 ) , and tensor ring decomposition ( Zhao et al. , 2016 ; Huang et al. , 2020 ) . Take the CP decomposition based LRTC as an example , one can just learn a CP decomposition from the partial observations ( Acar et al. , 2011 ) . Yuan & Zhang ( 2016 ) provided the sample complexity for the exact recovery of LRTC via nuclear norm minimization , though the optimization is intractable because computing the nuclear norm of a tensor is NP-hard ( Hillar & Lim , 2013 ) . Barak & Moitra ( 2016 ) and Foster & Risteski ( 2019 ) exploit the sum-ofsquares hierarchy for LRTC and provided theoretical guarantees for the recovery . Razin et al . ( 2021 ) studied the implicit regularization in tensor factorization and proved that gradient descent with small learning rate and near-zero initialization gives rise to a bias towards low tensor rank solutions . Liu et al . ( 2019 ) introduced convolutional neural network to tensor completion . The method can model the possibly complex interactions inside tensors while preserving the desired low-rank structure . To complete a third-order ( w.l.g ) tensor based on the Tucker decomposition , we may solve minimize X , A1 , A2 , A3 , C 1 2 ‖X − C ×1 A1 ×2 A2 ×3 A3‖2F , subject to PΩ ( X ) = PΩ ( Y ) , ( 3 ) where we can also consider regularization or constraint on the factors A1 , A2 , A3 , and C. For instance , Xu et al . ( 2013 ) reformulated ( 3 ) to all-mode matricization and solved the problem by alternating minimization . Xie et al . ( 2018 ) proposed to minimize the number of nonzero elements of C and the ranks of the matricizations of X , in addition to ( 3 ) . Deep Matrix Factorization Recently , deep matrix factorization ( DMF ) ( Fan & Cheng , 2018 ; Arora et al. , 2019 ) gains increasing attention in machine learning . There are two types of DMF methods : linear DMF ( Trigeorgis et al. , 2016 ; Zhao et al. , 2017 ; Arora et al. , 2019 ) and nonlinear DMF ( Xue et al. , 2017 ; Wang et al. , 2017 ; Fan & Cheng , 2018 ) . A general formulation is X ≈ g1 ( A1g2 ( A2 · · · gL−1 ( AL−1AL ) · · · ) ) , ( 4 ) where { gi } L−1l=1 are activation functions . When all gi are linear , ( 4 ) is linear DMF . Otherwise , ( 4 ) is nonlinear DMF . Arora et al . ( 2019 ) found that adding depth to a linear matrix factorization can enhance an implicit tendency towards low-rank and can provide higher recovery accuracy than depth-2 factorization , though the implicit regularization in linear DMF may not be captured using simple mathematical norms . In ( Fan & Cheng , 2018 ) , the numerical results showed that nonlinear DMF is able to recover ( with high accuracy ) the missing entries of a full-rank matrix of which the columns are generated by a nonlinear low-dimensional latent variable model . Contributions of this paper This paper first provides theoretical analysis for why and when nonlinear DMF outperforms linear DMF in matrix completion . Second , this paper proposes a new matrix factorization method called two-mode nonlinear DMF . The method can exploit the full nonlinearity of the data while classical nonlinear DMF uses only partial nonlinearity of the data . Third , the paper presents a new tensor decomposition method called multi-mode nonlinear deep tensor factorization , which can be regarded as a high-order generalization of the two-mode nonlinear DMF . Finally , the paper shows that the multi-mode deep matrix and tensor factorization methods have tighter generalization error bounds than conventional factorization methods in matrix and tensor completion . Extensive numerical results verified the effectiveness of the proposed methods . 2 WHY AND WHEN DOES NONLINEAR DMF OUTPERFORM MF ? . Nonlinear DMF methods can outperform linear MF methods empirically in many real tasks such as collaborative filtering ( Xue et al. , 2017 ; Fan & Cheng , 2018 ) , image inpainting ( Fan & Cheng , 2018 ) , and disease modeling ( Wang et al. , 2017 ) . The fundamental reason is that the data matrices in these tasks indeed have some nonlinear structures that can not be exploited by linear MF methods . Nevertheless , in nonlinear DMF , the theoretical guarantee is very limited . In contrast , linear MF or even linear DMF have been well studied and guaranteed with sample complexity ( Hardt , 2014 ; Sun & Luo , 2016 ; Jin et al. , 2016 ) , generalization bound ( Srebro & Shraibman , 2005 ; Fan et al. , 2019 ) , or convergence ( Gunasekar et al. , 2017 ; Arora et al. , 2019 ) . In this section , we will provide some theoretical guarantee for nonlinear DMF in matrix completion . Assumption 1 . Suppose Y = X + E , where Y ∈ Rn1×n2 . The columns of X are generated by x = f ( z ) , where f : Rd → Rn1 denotes a nonlinear mapping , d < min ( n1 , n2 ) , and z ∈ Rd denotes a latent variable . The entries of the noise matrix E are drawn from N ( 0 , σ2 ) . Suppose we observed a few entries of Y randomly ( sampling without replacement ) and want to recover X from the incomplete Y . Consider the following method ( Fan & Cheng , 2018 ) : minimize W1 , ... , WL , Z 1 2 ‖PΩ ( Y −WLg ( WL−1 · · · g ( W1Z ) · · · ) ) ‖2F + λw 2 L∑ l=1 ‖Wl‖2F + λz 2 ‖Z‖2F , ( 5 ) where Z ∈ Rd×n2 , Wl ∈ Rhl×hl−1 , l = 1 , . . . , L , hL = n1 , h0 = d , and h−1 = n2 . λw and λz are regularization parameters . Here we have let all activation functions be the same without loss of generality and omitted the bias terms of the neural network for simplicity . According to the universal approximation theorems ( Cybenko , 1989 ; Pinkus , 1999 ; Lu et al. , 2017 ; Hanin & Sellke , 2017 ) , f can be well approximated by the neural network provided that L is sufficiently large or at least one of h1 , . . . , hL−1 is sufficiently large . The following theorem provides the excess risk bound for ( 5 ) . Theorem 1 . Suppose Z and { Wl } Ll=1 are given by ( 5 ) . Let X̂ = WLg ( WL−1 · · · g ( W1Z ) · · · ) . Suppose ‖Z‖F ∏L l=1 ‖Wl‖F ≤ κ , max ( ‖Y ‖∞ , ‖X̂‖∞ ) ≤ ξ , the Lipschitz constant of g is η . Let N = n1n2 . Then with probability at least 1− 2N−1 , there exists a numerical constant c such that 1√ N ‖X − X̂‖F ≤ 1√ |Ω| ‖PΩ ( Y − X̂ ) ‖F + 1√ N ‖E‖F + cξ ( ∑L l=0 hlhl−1 log ( η L−1ξ−1κ ) |Ω| ) 1/4 . In Theorem 1 , the last term of the RHS of the inequality is equivalent to ψ1 = cξ ( |Ω|−1 ( n2d+ hL−1n1 + L−1∑ l=1 hlhl−1 ) log ( η L−1ξ−1κ ) ) 1/4 . ( 6 ) Most major activation functions ( e.g . ReLU and sigmoid ) are , at worst , 1-Lipschitz with respect to ` 2 norm , which indicates that ηL−1 is not large . When n2 is large enough , ψ1 is linear with the latent dimension d of the columns of X when using ( 5 ) . In practice , it is difficult to know the true d in advance . But the Frobenius norm regularizations on Z and W1 are able to reduce the nuclear norm or even the rank of W1Z theoretically , which means the d in formulation ( 5 ) and Theorem 1 can be larger than the d ( ground truth ) in Assumption 1 . Note that classical MF ( depth-1 ) is a special case of ( 5 ) when L = 1 and g is linear . However , because of the nonlinearity in the data generating model of Assumption 1 , the rank of X can be much higher than the intrinsic dimension of the data , i.e . r : = rank ( X ) d. For example , if f is a q-order polynomial function , r can be as large as ( d+q q ) . For classical MF , the last term of the RHS of the inequality in Theorem 1 reduces to ψ2 = cξ ( |Ω|−1 ( n1r + rn2 ) log ( ξ−1κ′ ) ) 1/4 . ( 7 ) Here κ′ may be less than the κ in ( 6 ) but they are in the log operator . Suppose all the Frobenius norms are the same ( the worst case , may not happen because the norms of the factors in a deep factorization are usually smaller than the norms of the factors in a shallow factorization ) , denoted by β . Then κ = βL+1 and κ′ = β2 , but r d. In the experiments , we observed that κ ≈ ( κ′ ) 1.3 when L = 2 and κ ≈ ( κ′ ) 1.7 when L = 3 , which are much better than the worst cases . Now comparing ( 7 ) with ( 6 ) , we conclude that the nonlinear DMF provides a tighter generalization bound than the classical MF when n2 is much larger than n1 and ∑L−1 l=1 hlhl−1 is not too large . The second condition is realistic because we usually let h1 , . . . , hL−1 < n1 or we can just assume h1 , . . . , hL−1 < r. Consider the case that n2 is not large compared to n1 , e.g . n2 = n1 . In ( 6 ) , suppose d = r/2 , L = 3 , h1 = 2d , h2 = 3d , and η = 1 , we have ψ1 = cξ ( |Ω|−1 ( 2n2r + 2r2 ) log ( κ ) ) 1/4 and ψ2 = cξ ( |Ω|−1 ( 2n2r ) log ( κ′ ) ) 1/4 , where ψ1 > ψ2 . It means the upper bound given by the classical MF is tighter than that given by the nonlinear DMF . In general , the superiority of the nonlinear DMF over the classical MF increases when n2/n1 increases , which is consistent with the numerical results shown in Figure 5 of the appendix .

Summary: This paper studies the nonlinear low-rank completion of matrices and tensors. Specifically, it first presents the theoretical results showing why nonlinear deep matrix factorization is better than the ordinary matrix factorization model. Then, it proposes a model named two-mode nonlinear deep matrix factorization to make full use of the nonlinearity of the nearly square matrices. The authors also extend this method to tensor factorization by further factorizing the factor matrices in the Tucker decomposition using the deep factorization method. Impressive results are obtained using both synthetic and real-world datasets.