Matrix Data Deep Decoder - Geometric Learning for Structured Data Completion

1 INTRODUCTION . Matrix completion ( MC ) consists of estimating the missing entries of an n×m matrixX ( usually , of very big dimensions ) given its measurements M on a ( usually , very sparse ) support Ω . An example of such matrices are signals on graphs/manifolds which are Non-Euclidean domains . The classical example of such data are recommender ( recommendation ) systems , where the ratings are signals on ( user item ) couple . The most known Matrix Completion problem is the Netflix problem , where a 1M $ prize was offered for the algorithm that can best predict user ratings in a dataset that contained 480k movies × 18k users ( 8.5B entries ) , with 0.011 % known entries ( Bell et al. , 2009 ) . Many works focused on solutions for the MC problem . In brief , one wishes to obtain the matrix X given matrix M as the specified input on the support Ω . Then , formally the completion task amounts to the minimization problem X̂ = argmin X ‖AΩ ◦ ( X −M ) ‖2F where AΩ is the observation mask matrix ( filled with 1 where data exists in the original problem ) , ◦ is the Hadamard product and ‖.‖2F is the Frobenius norms ( Rennie & Srebro , 2005 ) . Different approaches where presented in order to fill in matrix X . Those approached included imposing different regularization ( priors ) on the matrix and its factors . The most prominent approach consists of imposing a low rank ( Candès & Recht , 2009 ; Recht , 2009 ) on the matrix . Then , priors based on collaborative filtering ( users/items rating patterns ) , content based filtering ( user/items profile ) ( Ghassemi et al. , 2018 ; Jain & Dhillon , 2013 ; Xu et al. , 2013 ; Si et al. , 2016 ) and their combinations . Then Geometric Matrix Completion approaches appeared ( Li & Yeung , 2009 ; Rao et al. , 2015 ; Cai et al. , 2011 ) and proposed describing rows/column graphs which represent similarity , then encoding the structural ( geometric ) information of those graphs via graph Laplacian regularization ( Belkin & Partha , 2002 ; Belkin & Niyogi , 2003 ) and imposing smoothness of the data in those graphs ( Kalofolias et al. , 2014 ; Rao et al. , 2015 ; Ma et al. , 2011 ; Mardani et al. , 2012 ) . Those approaches where generally related to the field of signal processing as entries signals on the rows/columns graphs ( Shuman et al. , 2012 ) . Then Geometric Deep Learning Methods where introduced to learn the domains of geometric data structures ( e.g . single graphs or manifolds ) ( Bronstein et al. , 2016 ; Lefkimmiatis , 2016 ; Defferrard et al. , 2016 ; Niepert et al. , 2016 ; Gilmer et al. , 2017 ; Hamilton et al. , 2017 ; Velickovic et al. , 2017 ; Chen et al. , 2018 ; W. Huang et al. , 2018 ; Klicpera et al. , 2018 ; Abu-El-Haija et al. , 2019 ; Ying et al. , 2018 ; Gao et al. , 2018 ; Hammond et al. , 2011 ) . The current state of the art solution for Matrix completion problem , relies on an extending classical harmonic analysis methods to non-Euclidean domains . When , the geometry of the column/row spaces and their graphs is utilised to provide a Geometric Deep Learning mechanism called the RMGCNN ( Monti et al. , 2017 ) that includes a complex combined CNN and RNN ( Hochreiter & Schmidhuber , 1997 ) networks . In this work we present a simplified method for the MC problem : the Matrix Data Deep Decoder that contains a classical end to end GRAPH convolutional neural network and inspired by the leading methods from the field of image completion - the Deep Image Prior ( Ulyanov et al. , 2020 ) and the Deep Decoder ( Heckel & Hand , 2018 ) . In our method , random noisy input matrix is acted upon by the weights of a neural network ( parametrization ) . By tuning the parameters of the network and minimising the error between its output to the initial corrupted matrix , we find the best candidate for the complete matrix . This method yields state of art results for the MC task . The contributions of our work are : • A novel approach for solving the MC Problem , using deep learning with end-to-end pure convolutional network for graphs . • State-of-the-art performance for the MC problem in both prediction error ( RMSE ) and solution running time1 . Our method significantly outperforms the previous state of art method - the RMGCNN . • We show that a pure graph convolutional neural network is a good prior for the MC problem . This provides a correspondence of convolutionial neural networks methods to MC problems . 2 PRELIMINARIES . 2.1 MATRIX COMPLETION NOTATION . The most prominent prior for the MC problem is assuming the matrix X is of low rank . Low rank is obtained by rank regularization using its nuclear ( trace ) norm ‖X‖∗ – sum of the singular values ofX . The canonical optimization problem with parameter λ∗ , is stated as : X̂ = min X ‖AΩ ◦ ( X −M ) ‖2F + λ∗ ‖X‖∗ 2.1.1 MATRIX FACTORIZATION . To alleviate the computational burden for big datasets , we factorize X = W HT , where W ∈ Rm×k , HT ∈ Rk×n . Here , k m and n is the upper bound on the rank of X . With this factorization , the nuclear norm term can be replaced by the sum of the Frobenius norms leading to the following non-convex ( but still very well-behaved ) problem ( Rao et al. , 2015 ) : X̂ = min W , HT ∥∥AΩ ◦ ( WHT −M ) ∥∥2F + λ∗2 ( ‖W ‖2F + ∥∥HT∥∥2F ) 2.2 GEOMETRIC MATRIX COMPLETION . We introduce the geometric matrix completion framework , using notations as in RMGCNN ( Monti et al. , 2017 ) . 1evaluated on the existing classical benchmark for MC Problems 2.2.1 THE ROW/COLUMN GRAPHS . The matrix X is comprised from signals on non-euclidean domains of rows and columns . We represent those domains by undirected weighted graphs Gr ( e.g . items ) and Gc ( e.g users ) respectively , where : Gr/c = ( V , E , W ) . Gr/c are built either directly from the ratings matrix X , or based on additional data about the rows/columns ( if given ) . Their structure is encoded in Laplacian matrices which are built from the adjacency matrices Wr/c ( definitions are below ) . This procedure is sketched in figure 1 below . 2.2.2 THE ADJACENCY MATRIX : . For a graph G = ( V , E , W ) , the elements of its adjacency matrix ( W ) ij = wij obey : wij = wji , wij = 0 if ( i , j ) /∈ E and wi , j > 0 if ( i , j ) ∈ E. The Adjacency Matrix represents the weights of the proximity between every two vertices and can be built based on the signal patterns or on external features about the rows/columns in methods like euclidean distance of normalized features , Chi square , Gaussian Kernel , K-nn clustering K-means clustering and etc . 2.2.3 THE GRAPH LAPLCIANS . The Laplacian matricesLr andLc are based on the adjacency matricesW and are holding inside the internal Graph Structure . The most common constructions of a Laplacian matrix is an n× n matrix defined as L = D −W . where D is degree matrix , an n× n diagonal matrix ( D ) ii = ∑n j 6=i wij . We adopt the Normalized Graph Laplacian definition as L̃ = D− 1 2LD− 1 2 = I −D− 12WD− 12 . 2.2.4 THE OPTIMIZATION PROBLEM . We use the graph laplacians in the optimization function as an additional prior regularizing the matrix completion problem . We ’ d like more similar items/users get more similar predictions . Mathematically , regarding the columns x1 , . . . , xn for example as a vector-valued function defined on the vertices Vc , the smoothness assumption implies that xj ≈ xj′ if ( j , j′ ) ∈ Ec . Stated differently , we want the following entity ( Trace norm or Dirichlet semi-norm ( Kalofolias et al. , 2014 ) ) : ∑ i , j w c i , j ‖xi − xj‖ 2 2 = tr ( XLcX T ) to be as small as possible , leading to the following optimization problem : X̂ = argmin x∈Rm×n ‖PΩ ◦ ( X −M ) ‖2F + λ∗ ‖X‖∗ + λr tr ( XT LrX ) + λc tr ( XLcX T ) , which , if we will look at the factorized model will be equivalent to , X̂ = argmin W∈Rn×k , HT∈Rk×n ∥∥PΩ ◦ ( WHT −M ) ∥∥2F +λ∗ ∥∥WHT∥∥∗+λr tr ( W TLrW ) +λc tr ( HTLcH ) ) From this perspective , the estimation of the left and the right factors ofX is considered as diffusion of the input fields on the row and column graphs , respectively . This separable form allows no accommodation for the low rank constraint ( which pertains to the product of the graphs ) . 2.3 DEEP NEURAL NETWORKS . In the recent years , deep neural networks and , in particular , convolutional neural networks ( CNNs ) ( Lecun et al. , 1998 ) based methods have been applied with great success to Image completion tasks . Such methods are based on one of the key properties of CNN architectures - the ability to extract the important local stationary patterns of Euclidean data . Image completion with untrained networks ( when only the corrupted image is the input with no other training examples ) can be seen as parallel to the `` Matrix Completion '' task . Two recent works , applying un-trained deep neural networks on corrupted images , showed state-of the art results for this task . We were inspired by those methods and our goal was to generalize them to the Non-Euclidean Domain . 2.3.1 DIP – DEEP IMAGE PRIOR . The method suggests to feed the network with random input Z , forward pass the random input through the network and check how close the output is to the corrupted image , while tuning the network parameters weights . This operation surprisingly reconstructs the clean image ( see Ulyanov et al . ( 2020 ) ) . 2.3.2 DEEP DECODER . The Deep Decoder method showed results even better then the DIP ( see Heckel & Hand ( 2018 ) ) . The method proposed to take a small sample of noise , and pass it through a network , while making some non-linear operations on it and up-sample , then check how far the result is from the corrupted image while fixing the network parameters . This method showed that a deep decoder network is a very concise image prior . The number of parameters it needs to completely specify that image is very small , providing a barrier for over-fitting ( catching only the most important image features ( natural structures ) and ignore noise ) and allowing the network to be amenable to theoretical analysis . 2.4 GEOMETRIC DEEP LEARNING OR DEEP LEARNING ON GRAPHS . In contrast to image matrices , the notion of convolution and pooling for Non-Euclidean matrices needs to be re-defined to give the Non-Euclidean stracture the special meaning that convolutional networks are based on . When those operations are redefined , we can build a `` graph convolutional neural network '' which is parallel to some classical neural network and find the estimate forX .

Summary: This paper aims to tackle the matrix completion problem by drawing connection from prior work in image completion domain. It seems to be a combination of prior work: Multi-graph convolution combined with Dirichlet energy on row and column graph laplacian where the input rating matrix is corrupted with noise. The writing and presentation is significantly below par Iclr acceptance in the current form. Also, considering some of the work mentioned below, SOTA results is an overclaim.