Deep geometric matrix completion:  Are we doing it right?

1 INTRODUCTION . Matrix completion deals with the recovery of missing values of a matrix from a subset of its entries , Find X s.t . X S = M S. ( 1 ) HereX stands for the unknown matrix , M ∈ Rm×n for the ground truth matrix , S is a binary mask representing the input support , and denotes the Hadamard product . Since problem ( 1 ) is ill-posed , it is common to assume thatM belongs to some low dimensional subspace . Under this assumption , the matrix completion problem can be cast via the least-squares variant , min X rank ( X ) + µ 2 ‖ ( X −M ) S‖2F . ( 2 ) Relaxing the intractable rank penalty to its convex envelope , namely the nuclear norm , leads to a convex problem whose solution coincides with ( 2 ) under some technical conditions ( Candès & Recht , 2009 ) . Another way to enforce low rank is by explicitly parametrizingX in factorized form , X = Y Z > . The rank is upper-bounded by the inner dimensions of Y , Z > . Recent studies by Arora et al . ( 2019 ) ; Gunasekar et al . ( 2017 ) , suggest that overparametrizing X as a product of several matrices results in a low rank matrix due to an implicit regularization induced by the gradient descent trajectories . According to ( Gunasekar et al. , 2017 ; Arora et al. , 2019 ) , given enough entries and independent of the depth , overparametrized deep matrix factorization ( DMF ) models are equivalent1 to nuclear norm minimization . However , in the data poor regime these methods differ , and an implicit regularization brought forward by the dynamics of the gradient descent algorithm provides a stronger rank regularization . The deeper the network - the stronger the regularization . Optimization approaches as described above can be thought of as instances of self-supervised learning , an informal name given to a machine learning paradigm with the goal of `` predicting any part of the input from any other part '' ( LeCun , 2019 ) . Adopting the machine learning nomenclature , we shall henceforth refer to the given entries ofM as training set or training samples , and denote by test set a subset of the ( unknown ) entries ofM which we shall use for evaluation . The advent of deep learning platforms equipped with efficient automatic differentiation tools allows the exploration of more sophisticated models that incorporate intricate regularizations , both explicit 1Under some technical conditions . See ( Arora et al. , 2019 ) . and implicit . Some contemporary approaches for matrix completion fall under the umbrella of geometric deep learning . These approaches generalize standard ( Euclidean ) deep learning to domains such as general graphs and manifolds . For example , graph convolutional neural networks ( GCNNs ) follow the architecture of standard CNNs , but replace the Euclidean convolution operator with linear filters constructed using the graph Laplacian . We distinguish between graph based approaches which make use of the bi-partite graph structure of the rating matrix ( e.g. , Berg et al . ( 2017 ) ) , and geometric matrix completion techniques which make use of side information in the form of graphs encoding relations between rows/columns ( Kalofolias et al. , 2014 ; Monti et al. , 2017 ) . While these techniques achieve state-of-the-art results , their design is arguably cumbersome and non-intuitive . It has recently been demonstrated that some graph CNN architectures can be greatly simplified , and still perform competitively on several graph analysis tasks ( Wu et al. , 2019 ) . Such simple techniques have the advantage of being easier to analyze and reproduce . Motivated by these results , our intent is to construct a simple architecture for matrix completion that follows from geometric considerations rather than ad-hoc ones . The inspiration for our paper stems from techniques for finding shape correspondence . In particular , the functional maps framework and its variants ( Ovsjanikov et al. , 2012 ; 2016 ) . Most notably the work of ( Litany et al. , 2017 ) who combined functional maps with joint diagonalization to solve partial shape matching problems , and the product manifold filter ( PMF ) ( Vestner et al. , 2017b ; a ) and zoomout ( Melzi et al. , 2019 ) – two greedy algorithms for correspondence refinement by gradual introduction of high frequencies . This last method lent its name to the loss we define in Section 3 . To that end , we propose a simple spectral method for geometric matrix completion that combines the fully linear network structure of DMF with the geometric framework of functional maps . Contribution . Our contributions are as follows : • We propose a geometric interpretation for deep matrix factorization ( Arora et al. , 2019 ) by embedding it as part of the functional maps framework . • We introduce the zoomout loss , a multiresolution spectral geometric loss inspired by recent advances in shape correspondence . • We show that via a simple shallow and fully linear network , it is possible to obtain state-of- the-art results on various recommendation systems datasets . • We demonstrate that in some cases the effect of the geometry is only marginal , and results on par with state-of-the-art can be achieved even without it . 2 BACKGROUND . Spectral graph theory . Let G = ( V , E , Ω ) be a ( weighted ) graph specified by its vertex set V and edge set E , and let Ω be its adjacency matrix . Given a function x ∈ R|V | on the vertices , we define the following quadratic form ( also known as Dirichlet energy ) measuring the variability of the function x on the graph , x > Lx = ∑ ( a , b ) ∈E ωa , b ( x ( a ) − x ( b ) ) 2 . ( 3 ) The matrix L is called the ( combinatorial ) graph Laplacian , and is given by L = D −Ω , where D = diag ( Ω1 ) is the degree matrix . L is symmetric and positive semi-definite and therefore admits a spectral decomposition L = ΦΛΦ > . The graph Laplacian is a discrete generalization of the continuous Laplace-Beltrami operator , and therefore has similar properties . One can think of the eigenpairs ( φi , λi ) as the graph analogues of `` harmonic '' and `` frequency '' . A function x = ∑|V | i=1 αiφi on the vertices of the graph whose coefficients αi are small for large i , demonstrates a `` smooth '' behaviour on the graph in the sense that the function values on nearby nodes will be similar . A standard approach to promoting such smooth functions on graphs is by using the Dirichlet energy ( 3 ) to regularize some loss term . For example , this approach gives rise to the popular bilateral and non-local means filters ( Gadde et al. , 2013 ) . Functional maps . Let G1 = ( V1 , E1 , Ω1 ) , G2 = ( V2 , E2 , Ω2 ) be two graphs , and let Φ , Ψ , be two orthonormal bases for functions defined on the vertices of these graphs . Given two such functions , x = Φα on G1 and y = Ψβ on G2 , one can define a map C between their representations α and β , i.e. , α = Φ > x = CΨ > y = Cβ . The matrix C represents a linear map between the functional spaces on G1 and G2 , known as a functional map . Let us denote the Cartesian product of G1 and G2 by G1 G2 . G1 G2 is a graph with vertex set V1 × V2 , on which two nodes ( u , u′ ) , ( v , v′ ) are adjacent if either u = v and ( u′ , v′ ) ∈ E2 or u′ = v′ and ( u , v ) ∈ E1 . LetX be a function defined on G1 G2 , then the functional map is given by projecting X onto the corresponding bases Φ , Ψ , C = Φ > XΨ . Using the SVD , one can decompose X = UΣV > to interpret C as a mapping between UΣ , a function on one graph , and V , a function on the other . To get X back from the functional map , one can useX = ΦCΨ > . For computational reasons , it is common to use truncated bases Φ , Ψ , in which case the last equality holds only approximately . The structure of the functional map depends on the properties of the chosen bases and the functions it maps . A common choice for a basis is the aforementioned Laplacian eigenbasis , building on the assumption that the signals involved are smooth with respect to the graphs . While this is a useful model , it assumes that the given graphs encode the geometry in an adequate way . In real world problems these graphs are only approximate , constructed from heuristic features associated with the row and the column spaces . Given better graphs , a simpler structure of the functional map emerges . For example , by introducing two orthonormal matrices P andQ , one can make the functional map C = ( ΦP ) > X ( ΨQ ) diagonal . These orthonormal matrices can be thought of as a way of aligning the bases Φ , Ψ , with the principal axes ofX . 3 SPECTRAL GEOMETRIC MATRIX COMPLETION . We assume that we are given a set of samples from the unknown matrix M ∈ Rm×n , encoded by a binary mask S , and two graphs Gr , Gc , encoding relations between the rows and the columns , respectively . Denote the Laplacians of these graphs and their spectral decompositions by Lr = ΦΛrΦ > , Lc = ΨΛcΨ > . Our approach relies on a minimization problem of the form min X Ez ( X ) + µrtrace ( X > LrX ) + µctrace ( XLcX > ) , ( 4 ) with Ez denoting the data term that we discuss in the sequel . The other two terms in ( 4 ) are the Dirichlet energies of X computed on Gr , Gc , as defined in ( 3 ) . These energies measure the variability of X along neighbouring nodes on these graphs . Using X = ΦCΨ > and the spectral decompositions of Lr , Lc , we can equivalently write them in terms of C , trace ( X > LrX ) = trace ( ΨC > Φ > LrΦCΨ > ) = trace ( C > ΛrC ) , trace ( XLcX > ) = trace ( ΦCΨ > LcΨC > Φ > ) = trace ( CΛcC > ) . ( 5 ) As mentioned above , the input graphs are typically constructed from a set of heuristically gathered features which may provide a poor representation of the latent geometry . One way to account for this inaccuracy could be to include the features in our optimization . This will induce , through a complicated nonlinear dependence , a different metric ( i.e. , adjacency matrix ) and a different graph Laplacian . This approach lies at the heart of smoothing models based on conditional random fields ( CRFs ) used in image segmentation ( see for example Krähenbühl & Koltun ( 2011 ) ) . We adopt a different approach by working in the spectral domain . Switching to the spectral domain allows us to modify the metric indirectly by applying orthogonal transformations , P andQ , to the bases Ψ , Φ . The purpose of these transformations is to rotate Ψ , Φ , in a way that will simplify the structure of C. For example , it is possible to diagonalize C by an appropriate choice of P , Q via the SVD decomposition . Since our method relies on the premise that the matrix M is smooth with respect to some graphs , our interest is in modified bases ΦP , ΨQ , which arise from the eigendecomposition of a graph Laplacian . To that end , we shall use Lr , Lc , as proxies for the latent graph Laplacians , and promote bases that approximately diagonalize them by introducing two energy terms , Erdiag ≡ ∥∥off ( P > Φ > LrΦP ) ∥∥2F = ∥∥off ( P > ΛrP ) ∥∥2F , Ecdiag ≡ ∥∥off ( Q > Ψ > LcΨQ ) ∥∥2F = ∥∥off ( Q > ΛcQ ) ∥∥2F , ( 6 ) where off ( · ) denotes the off-diagonal elements . Under these modifications , the Dirichlet energy terms ( 5 ) become , Erdir ≡ trace ( QC > P > ΛrPCQ > ) , Ecdir ≡ trace ( PCQ > ΛcQC > P > ) . ( 7 ) Finally , we introduce the following energy terms to promote the ( approximate ) orthonormality of P , Q : Erorth ( P ) ≡ ∥∥P > P − I∥∥2 F , Ecorth ( Q ) ≡ ∥∥Q > Q− I∥∥2 F . ( 8 ) Zoomout loss . Let us denote the training error achieved by a matrixXp , q ≡ ΦpCp , qΨ > q composed from the first p vectors in Φ and the first q vectors in Ψ as Zp , q ≡ ‖ ( ΦpCp , qΨ > q −M ) S‖2F . We define the zoomout loss as follows : Ez = m , n∑ p=1 , q=1 wp , qZp , q , ( 9 ) with the weights wp , q ≥ 0 . It can be shown that it is enough to use each value of p or q only once and therefore most of the wp , q shall be set to 0 ( see Appendix A ) . The zoomout loss ( 9 ) is inspired by the greedy approaches for shape correspondence proposed by Vestner et al . ( 2017a ) and Melzi et al . ( 2019 ) . These methods approximate the point-to-point map between the shapes ( i.e. , the correspondence ) by a smoothed map obtained through the application of spectral filters , and proceed by projecting the smoothed map onto the set of point-to-point maps . By gradually increasing the filters ’ bandwidth , or resolution , the correspondence becomes more refined in each step . In contrast , we advocate minimizing the training error using all resolutions at once . This simultaneous multi-resolution approach incurs a penalty on the rank of the reconstructed matrix by implicitly giving more weight to the low frequency terms . For example , the sub-matrix C2,2 appears in all the terms Zp , q with p ≥ 2 , q ≥ 2 , emphasizing its importance . This allows us to be sloppy in the estimation of the rank ofM without compromising the results by much . We explore the effect of overparameterization in Section 4 . Following the discussion above , we replace Φ , Ψ with ΦP , ΨQ , obtaining Zp , q ≡ ‖ ( ΦPpCp , qQ > q Ψ > −M ) S‖2F . With this modification , the zoomout loss ( 9 ) should favor bases ΦP , ΨQ , in which most of the energy of C is concentrated in the low frequencies , i.e. , the top-left part . An interesting observation is that by setting p = m , q = n , we get the deep matrix factorization ( DMF ) method from Arora et al . ( 2019 ) ( up to initialization ) . As we show in Section 4 , this model alone , without any additional geometric priors , is sufficient to obtain results on par with the state-of-the-art on some datasets . This puts in question the quality of the geometric information in those cases . The complete minimization objective combines all the described terms weighed with the appropriate weights and minimized with respect toC , P andQ by gradient descent , min C , P , Q Ez + µrE r dir + µcE c dir + ρrE r diag + ρcE c diag + orE r orth + ocE c orth . ( 10 )

Summary: This paper proposes a novel approach for the loss function of matrix completion when geometric information is available. The proposed method consists of two ideas: (1) spectral regularization (i.e., Dirichlet energy) with a re-parameterizing basis and (2) multiresolution of spectral loss (i.e., zoomout loss). In addition, the zoomout loss is motivated by the approach for shape correspondence and can be a generalization of the recent matrix completion method (deep matrix factorization). Empirical results show the best performance compared to other recent methods under small-scale datasets. Moreover, the proposed method outperforms when the geometric model is accurate (verified on the synthetic setting) and this can reflect that the proposed method is a good choice when the graph structures are given.