Deep geometric matrix completion:  Are we doing it right?

1 INTRODUCTION . Matrix completion deals with the recovery of missing values of a matrix from a subset of its entries , Find X s.t . X S = M S. ( 1 ) HereX stands for the unknown matrix , M ∈ Rm×n for the ground truth matrix , S is a binary mask representing the input support , and denotes the Hadamard product . Since problem ( 1 ) is ill-posed , it is common to assume thatM belongs to some low dimensional subspace . Under this assumption , the matrix completion problem can be cast via the least-squares variant , min X rank ( X ) + µ 2 ‖ ( X −M ) S‖2F . ( 2 ) Relaxing the intractable rank penalty to its convex envelope , namely the nuclear norm , leads to a convex problem whose solution coincides with ( 2 ) under some technical conditions ( Candès & Recht , 2009 ) . Another way to enforce low rank is by explicitly parametrizingX in factorized form , X = Y Z > . The rank is upper-bounded by the inner dimensions of Y , Z > . Recent studies by Arora et al . ( 2019 ) ; Gunasekar et al . ( 2017 ) , suggest that overparametrizing X as a product of several matrices results in a low rank matrix due to an implicit regularization induced by the gradient descent trajectories . According to ( Gunasekar et al. , 2017 ; Arora et al. , 2019 ) , given enough entries and independent of the depth , overparametrized deep matrix factorization ( DMF ) models are equivalent1 to nuclear norm minimization . However , in the data poor regime these methods differ , and an implicit regularization brought forward by the dynamics of the gradient descent algorithm provides a stronger rank regularization . The deeper the network - the stronger the regularization . Optimization approaches as described above can be thought of as instances of self-supervised learning , an informal name given to a machine learning paradigm with the goal of `` predicting any part of the input from any other part '' ( LeCun , 2019 ) . Adopting the machine learning nomenclature , we shall henceforth refer to the given entries ofM as training set or training samples , and denote by test set a subset of the ( unknown ) entries ofM which we shall use for evaluation . The advent of deep learning platforms equipped with efficient automatic differentiation tools allows the exploration of more sophisticated models that incorporate intricate regularizations , both explicit 1Under some technical conditions . See ( Arora et al. , 2019 ) . and implicit . Some contemporary approaches for matrix completion fall under the umbrella of geometric deep learning . These approaches generalize standard ( Euclidean ) deep learning to domains such as general graphs and manifolds . For example , graph convolutional neural networks ( GCNNs ) follow the architecture of standard CNNs , but replace the Euclidean convolution operator with linear filters constructed using the graph Laplacian . We distinguish between graph based approaches which make use of the bi-partite graph structure of the rating matrix ( e.g. , Berg et al . ( 2017 ) ) , and geometric matrix completion techniques which make use of side information in the form of graphs encoding relations between rows/columns ( Kalofolias et al. , 2014 ; Monti et al. , 2017 ) . While these techniques achieve state-of-the-art results , their design is arguably cumbersome and non-intuitive . It has recently been demonstrated that some graph CNN architectures can be greatly simplified , and still perform competitively on several graph analysis tasks ( Wu et al. , 2019 ) . Such simple techniques have the advantage of being easier to analyze and reproduce . Motivated by these results , our intent is to construct a simple architecture for matrix completion that follows from geometric considerations rather than ad-hoc ones . The inspiration for our paper stems from techniques for finding shape correspondence . In particular , the functional maps framework and its variants ( Ovsjanikov et al. , 2012 ; 2016 ) . Most notably the work of ( Litany et al. , 2017 ) who combined functional maps with joint diagonalization to solve partial shape matching problems , and the product manifold filter ( PMF ) ( Vestner et al. , 2017b ; a ) and zoomout ( Melzi et al. , 2019 ) – two greedy algorithms for correspondence refinement by gradual introduction of high frequencies . This last method lent its name to the loss we define in Section 3 . To that end , we propose a simple spectral method for geometric matrix completion that combines the fully linear network structure of DMF with the geometric framework of functional maps . Contribution . Our contributions are as follows : • We propose a geometric interpretation for deep matrix factorization ( Arora et al. , 2019 ) by embedding it as part of the functional maps framework . • We introduce the zoomout loss , a multiresolution spectral geometric loss inspired by recent advances in shape correspondence . • We show that via a simple shallow and fully linear network , it is possible to obtain state-of- the-art results on various recommendation systems datasets . • We demonstrate that in some cases the effect of the geometry is only marginal , and results on par with state-of-the-art can be achieved even without it . 2 BACKGROUND . Spectral graph theory . Let G = ( V , E , Ω ) be a ( weighted ) graph specified by its vertex set V and edge set E , and let Ω be its adjacency matrix . Given a function x ∈ R|V | on the vertices , we define the following quadratic form ( also known as Dirichlet energy ) measuring the variability of the function x on the graph , x > Lx = ∑ ( a , b ) ∈E ωa , b ( x ( a ) − x ( b ) ) 2 . ( 3 ) The matrix L is called the ( combinatorial ) graph Laplacian , and is given by L = D −Ω , where D = diag ( Ω1 ) is the degree matrix . L is symmetric and positive semi-definite and therefore admits a spectral decomposition L = ΦΛΦ > . The graph Laplacian is a discrete generalization of the continuous Laplace-Beltrami operator , and therefore has similar properties . One can think of the eigenpairs ( φi , λi ) as the graph analogues of `` harmonic '' and `` frequency '' . A function x = ∑|V | i=1 αiφi on the vertices of the graph whose coefficients αi are small for large i , demonstrates a `` smooth '' behaviour on the graph in the sense that the function values on nearby nodes will be similar . A standard approach to promoting such smooth functions on graphs is by using the Dirichlet energy ( 3 ) to regularize some loss term . For example , this approach gives rise to the popular bilateral and non-local means filters ( Gadde et al. , 2013 ) . Functional maps . Let G1 = ( V1 , E1 , Ω1 ) , G2 = ( V2 , E2 , Ω2 ) be two graphs , and let Φ , Ψ , be two orthonormal bases for functions defined on the vertices of these graphs . Given two such functions , x = Φα on G1 and y = Ψβ on G2 , one can define a map C between their representations α and β , i.e. , α = Φ > x = CΨ > y = Cβ . The matrix C represents a linear map between the functional spaces on G1 and G2 , known as a functional map . Let us denote the Cartesian product of G1 and G2 by G1 G2 . G1 G2 is a graph with vertex set V1 × V2 , on which two nodes ( u , u′ ) , ( v , v′ ) are adjacent if either u = v and ( u′ , v′ ) ∈ E2 or u′ = v′ and ( u , v ) ∈ E1 . LetX be a function defined on G1 G2 , then the functional map is given by projecting X onto the corresponding bases Φ , Ψ , C = Φ > XΨ . Using the SVD , one can decompose X = UΣV > to interpret C as a mapping between UΣ , a function on one graph , and V , a function on the other . To get X back from the functional map , one can useX = ΦCΨ > . For computational reasons , it is common to use truncated bases Φ , Ψ , in which case the last equality holds only approximately . The structure of the functional map depends on the properties of the chosen bases and the functions it maps . A common choice for a basis is the aforementioned Laplacian eigenbasis , building on the assumption that the signals involved are smooth with respect to the graphs . While this is a useful model , it assumes that the given graphs encode the geometry in an adequate way . In real world problems these graphs are only approximate , constructed from heuristic features associated with the row and the column spaces . Given better graphs , a simpler structure of the functional map emerges . For example , by introducing two orthonormal matrices P andQ , one can make the functional map C = ( ΦP ) > X ( ΨQ ) diagonal . These orthonormal matrices can be thought of as a way of aligning the bases Φ , Ψ , with the principal axes ofX . 3 SPECTRAL GEOMETRIC MATRIX COMPLETION . We assume that we are given a set of samples from the unknown matrix M ∈ Rm×n , encoded by a binary mask S , and two graphs Gr , Gc , encoding relations between the rows and the columns , respectively . Denote the Laplacians of these graphs and their spectral decompositions by Lr = ΦΛrΦ > , Lc = ΨΛcΨ > . Our approach relies on a minimization problem of the form min X Ez ( X ) + µrtrace ( X > LrX ) + µctrace ( XLcX > ) , ( 4 ) with Ez denoting the data term that we discuss in the sequel . The other two terms in ( 4 ) are the Dirichlet energies of X computed on Gr , Gc , as defined in ( 3 ) . These energies measure the variability of X along neighbouring nodes on these graphs . Using X = ΦCΨ > and the spectral decompositions of Lr , Lc , we can equivalently write them in terms of C , trace ( X > LrX ) = trace ( ΨC > Φ > LrΦCΨ > ) = trace ( C > ΛrC ) , trace ( XLcX > ) = trace ( ΦCΨ > LcΨC > Φ > ) = trace ( CΛcC > ) . ( 5 ) As mentioned above , the input graphs are typically constructed from a set of heuristically gathered features which may provide a poor representation of the latent geometry . One way to account for this inaccuracy could be to include the features in our optimization . This will induce , through a complicated nonlinear dependence , a different metric ( i.e. , adjacency matrix ) and a different graph Laplacian . This approach lies at the heart of smoothing models based on conditional random fields ( CRFs ) used in image segmentation ( see for example Krähenbühl & Koltun ( 2011 ) ) . We adopt a different approach by working in the spectral domain . Switching to the spectral domain allows us to modify the metric indirectly by applying orthogonal transformations , P andQ , to the bases Ψ , Φ . The purpose of these transformations is to rotate Ψ , Φ , in a way that will simplify the structure of C. For example , it is possible to diagonalize C by an appropriate choice of P , Q via the SVD decomposition . Since our method relies on the premise that the matrix M is smooth with respect to some graphs , our interest is in modified bases ΦP , ΨQ , which arise from the eigendecomposition of a graph Laplacian . To that end , we shall use Lr , Lc , as proxies for the latent graph Laplacians , and promote bases that approximately diagonalize them by introducing two energy terms , Erdiag ≡ ∥∥off ( P > Φ > LrΦP ) ∥∥2F = ∥∥off ( P > ΛrP ) ∥∥2F , Ecdiag ≡ ∥∥off ( Q > Ψ > LcΨQ ) ∥∥2F = ∥∥off ( Q > ΛcQ ) ∥∥2F , ( 6 ) where off ( · ) denotes the off-diagonal elements . Under these modifications , the Dirichlet energy terms ( 5 ) become , Erdir ≡ trace ( QC > P > ΛrPCQ > ) , Ecdir ≡ trace ( PCQ > ΛcQC > P > ) . ( 7 ) Finally , we introduce the following energy terms to promote the ( approximate ) orthonormality of P , Q : Erorth ( P ) ≡ ∥∥P > P − I∥∥2 F , Ecorth ( Q ) ≡ ∥∥Q > Q− I∥∥2 F . ( 8 ) Zoomout loss . Let us denote the training error achieved by a matrixXp , q ≡ ΦpCp , qΨ > q composed from the first p vectors in Φ and the first q vectors in Ψ as Zp , q ≡ ‖ ( ΦpCp , qΨ > q −M ) S‖2F . We define the zoomout loss as follows : Ez = m , n∑ p=1 , q=1 wp , qZp , q , ( 9 ) with the weights wp , q ≥ 0 . It can be shown that it is enough to use each value of p or q only once and therefore most of the wp , q shall be set to 0 ( see Appendix A ) . The zoomout loss ( 9 ) is inspired by the greedy approaches for shape correspondence proposed by Vestner et al . ( 2017a ) and Melzi et al . ( 2019 ) . These methods approximate the point-to-point map between the shapes ( i.e. , the correspondence ) by a smoothed map obtained through the application of spectral filters , and proceed by projecting the smoothed map onto the set of point-to-point maps . By gradually increasing the filters ’ bandwidth , or resolution , the correspondence becomes more refined in each step . In contrast , we advocate minimizing the training error using all resolutions at once . This simultaneous multi-resolution approach incurs a penalty on the rank of the reconstructed matrix by implicitly giving more weight to the low frequency terms . For example , the sub-matrix C2,2 appears in all the terms Zp , q with p ≥ 2 , q ≥ 2 , emphasizing its importance . This allows us to be sloppy in the estimation of the rank ofM without compromising the results by much . We explore the effect of overparameterization in Section 4 . Following the discussion above , we replace Φ , Ψ with ΦP , ΨQ , obtaining Zp , q ≡ ‖ ( ΦPpCp , qQ > q Ψ > −M ) S‖2F . With this modification , the zoomout loss ( 9 ) should favor bases ΦP , ΨQ , in which most of the energy of C is concentrated in the low frequencies , i.e. , the top-left part . An interesting observation is that by setting p = m , q = n , we get the deep matrix factorization ( DMF ) method from Arora et al . ( 2019 ) ( up to initialization ) . As we show in Section 4 , this model alone , without any additional geometric priors , is sufficient to obtain results on par with the state-of-the-art on some datasets . This puts in question the quality of the geometric information in those cases . The complete minimization objective combines all the described terms weighed with the appropriate weights and minimized with respect toC , P andQ by gradient descent , min C , P , Q Ez + µrE r dir + µcE c dir + ρrE r diag + ρcE c diag + orE r orth + ocE c orth . ( 10 )

Summary: This paper aims to solve the matrix completion problem by incorporating geometric information. The proposed approach involves using graphs encoding relations between rows (and columns), applying spectral decomposition to these graphs, and using a multi-resolution spectral geometric loss to reconstruct the functional map which could then be used to directly recover the underlying matrix. The paper evaluates the proposed network on both synthetic and real datasets and shows improvements over the existing geometric methods and convex relaxations.