AIR-Net: Adaptive and Implicit Regularization Neural Network for matrix completion

1 INTRODUCTION . The matrix completion ( MC ) problem , which aims to recover a matrix X∗ ∈ Rm×n from its partially observed elements , has arisen in numerous domains , ranging from computer vision ( Wen et al. , 2012 ) , recommender system ( Netflix , 2009 ) , and drug-target interaction ( DTI ) ( Mongia & Majumdar , 2020 ) . This fundamental problem is ill-posed without assumptions on X∗ since we have many completions . So it is essential to impose additional information or priors on the unknown matrix/signal . To describe the prior for natural signal , or restrict the solution in the corresponding space is difficult . Classical methods for MC are mainly based on low-rank , sparsity or piece-wise smoothness assumption ( Rudin et al. , 1992 ; Buades et al. , 2005 ; Romano et al. , 2014 ; Dabov et al. , 2007 ) . These priors describe simple structural signal well , but may lead to a poor approximation of X∗ with complex structures ( Radhakrishnan et al. , 2021 ) especially when the observed entries are not sampled uniformly at random . Recently , deep neural networks ( DNN ) have shown a strong ability in extracting complex structures from large datasets ( Li et al. , 2018 ; Mukherjee et al. , 2021 ) . However , such a large number of data sets can not be obtained in many scenarios . Fortunately , DNN also works in solving some inverse problems without any extra training set ( Ulyanov et al. , 2018 ) . Over-parametric DNN performs well on a single matrix is a mysterious phenomenon . One of the explanations is there exists implicit regularization during training ( Arora et al. , 2019 ; Xu et al. , 2019 ; Rahaman et al. , 2019 ; Chakrabarty & Maji , 2019 ) . Although DNN with implicit regularization outperforms some classical methods , it is insufficient to describe the space of complex X∗ . Extra-explicit regularization can improve its performance in signal recovery ( Metzler et al. , 2018 ; Boyarski et al. , 2019a ; Liu et al. , 2019 ; Li et al. , 2020 ) . However , such explicit priors are often valid only for specific data or sampling patterns . A more flexible regularization is required to meet practical MC problems . We introduce flexibility in this paper by firstly representing the explicit regularization using DNN without any extra training set . The explicit regularization we begin with is Dirichlet Energy ( DE ) , which is formulated as tr ( X > LX ) , with L a Laplacian matrix describing the similarity between columns . Note that L in DE is fixed during iteration . Building an exact L based on incomplete observation is very challenging . Therefore , we parameterize L with DNN , and revise L iteratively during training . Furthermore , We combine the learned DE , which is an adaptive regularizer , with implicit regularization to form a new regularization method for MC named AIR-Net . The interaction between explicit regularization and implicit regularization in solving MC problems is further studied . The results show that combining the two can obtain a new , more flexible regularization model and enhance the low-rank preference of implicit regularization . In many examples , AIR-NET has a more vital feature representation ability and more comprehensive application range and shows state-of-art performance . 2 ADAPTIVE AND IMPLICIT REGULARIZATION NEURAL NETWORK . Our model is proposed as follow : min X , Wi Lall = LY ( A ( X∗ ) , A ( X ) ) + N∑ i=1 λi · RWi ( Ti ( X ) ) ( 1 ) where A ( X ) = X |Ω= { Xi , j , ( i , j ) ∈ Ω 0 , ( i , j ) 6∈ Ω and Ω are the observed coordinates set , and the other entries are missing . Different from other regularization models for MC , here X is represented by a neural network which tends to be low-rank implicitly ( Section 2.1 ) , and RWi is an adaptive regularization with a forward neural network represented Laplacian matrix ( Section 2.2 ) . The detailed notations will be introduced in the corresponding sections . A specific case of Equation 1 for matrix completion is given in Section 2.3 . 2.1 DMF AS AN IMPLICIT REGULARIZATION . In order to make the model suitable for more matrix types , we need a more general data prior . The low-rank is a very general prior in various matrix types . There are two main ways to encode the low-rank prior into model : ( a ) Adding an explicit regularization term such as rank and nuclear norm ( Candès & Recht , 2009 ; Lin et al. , 2010 ) . ( b ) Using a low-dimensional latent variable model to represent X , including matrix factorization ( MF ) and its varieties ( Koren et al. , 2009 ; Fan & Cheng , 2018 ) . The first case suffers from the saturation issue , which is induced by explicit regularization . The second one faces the problem of estimating a proper latent variable dimension . Unlike the existing MF model , which constricts the size of the shared dimension of the factorized matrix , DMF can take a large shared dimension and still preserve the low-rank property without explicit regularization . This is the so-called implicit low-rank regularization of DMF : X ( t ) = W [ L−1 ] ( t ) W [ L−2 ] ( t ) . . .W [ 1 ] ( t ) W [ 0 ] ( t ) ∈ Rm×n , where L is the depth of MF . W [ l ] ( t ) represents the l-th matrix at the step t during training . The results are given under a mild assumption 1 in Section A.2 . This property helps avoiding dimension estimation and saturation issues . As for the details of the implicit low-rank we will discuss in Section A.2 . 2.2 ADAPTIVE REGULARIZER . Apart from the low-rank prior , self-similarity is also a typical prior . The patch in the image and the rating behavior of users are all examples of self-similarity . For example , there is always a certain degree of self-similarity between the blocks in the image . A classical way to encode the similarity prior to X is Dirichlet Energy ( DE ) which is formulated as tr ( X > LX ) . But DE will face two problems in applications : ( a ) L is unknown in MC problem , construct L based on incomplete X may induce worse prior . ( b ) The formulation of DE only encodes the similarity of the columns of X . Other similarities such as block similarity can not be captured . To address both of these issues , we parameterize L with DNN and replace X by a transformed Ti ( X ) to capture the self-similarity flexibly . The adaptive regularization is defined as RWi ( Ti ( X ) ) = tr ( Ti ( X ) > Li ( Wi ) Ti ( X ) ) , i = 1 , 2 , . . . , N where Li ∈ Rmi×mi is parameterized by Wi ∈ Rmi×mi . To keep the Laplacian properties of Li , special design for the parameterized structure is important . We design a forward neural network which encodes the properties of Laplacian matrix in structure . The details are discussed in A.4 . Ti : Rm×n 7→ Rmi×ni transforms X into special domain , which makes the AIR-Net possible to capture various relationships embedded in data . The common choice can be Ti ( X ) = X which captures the relationship between columns . Regularization captures the relationship between rows when Ti ( X ) = X > . Especially if Ti ( X ) = [ vec ( block ( X ) ) 1 , vec ( block ( X ) ) 2 , . . . , vec ( block ( X ) ) ni ] , where vec ( block ( X ) ) j ∈ Rmi is the vectorization of j-th block in X row by row , then the similarity among blocks can be obtained . A natural problem that arises is what the Li looks like during training . Obviously , RWi reaches minimum when Li = 0 , and this is called a trivial solution . The most exciting thing is that when we minimize Equation 1 with the gradient descent algorithm , { Li ( Wi ( t ) ) } converges to a non-trivial solution . Another expected phenomenon is that RWi vanishes at the end and will not cause the so-called saturation issue . The saturation issue is a bias term that dominates the overall estimation error due to explicit regularization . We illustrate these phenomena both by theoretical analysis ( Theorem 2 in Section 3.2 ) and numerical experiments ( Section 4 ) . 2.3 AIR-NET FOR MC . In this subsection , we will focus our model on the MC problem . We select T1 ( X ) = X , T2 ( X ) = X > , N = 2 to capture the relationship both in rows and columns of X . Overall , the theoretical analyses for general inverse problem is based on Equation 2. min X , W r , W c Lall = LY ( A ( X∗ ) , A ( X ) ) + λr · RWr ( X ) + λc · RWc ( X > ) , ( 2 ) where X = W [ L−1 ] W [ L−2 ] . . .W [ 1 ] W [ 0 ] , RWr ( X ) = XLr ( Wr ) X > , RWc ( X > ) = X > Lc ( Wc ) X . Specially , our experiments focus on the MC problem which can reform Equation 2 as follows : min W [ l ] , W r , W c Lall = ∑ ( i , j ) ∈Ω ∣∣Xij −X∗ij∣∣+ λr · RWr ( X ) + λc · RWc ( X > ) , ( 3 ) with l = 0 , 1 , · · · , L − 1 . The parameters in Equation 3 is updated by gradient descent algorithm or its variations . We stop the iteration until ∣∣RWr ( T+1 ) −RWr ( T ) ∣∣ < δ and∣∣RWc ( T+1 ) −RWc ( T ) ∣∣ < δ . The recovered matrix is X̂ ( T ) = W [ L−1 ] ( T ) · · ·W [ 0 ] ( T ) . Some works which combine implicit and explicit regularization also can be regarded as a special case of Equation 1 . Both the Total Variation ( TV ) and DE can be regarded as a fixed L. Therefore , the framework of Equation 1 also contains DMF+TV ( Li et al. , 2020 ) , DMF+DE ( Boyarski et al. , 2019a ) . So far , we can not see any essential difference between Equation 3 and these models . We will illustrate the amazing properties of the model in the next section . 3 THEORETICAL ANALYSIS . In this section , we will analyze the properties based on the dynamics of Equation 3 . Theorem 1 shows that our proposed regularization enhances the implicit low-rank regularization of DMF . Theorem 2 shows that the adaptive regularization will converge to a minimum while capturing the inner structure of data flexibly . Although this paper focus on MC problem , the following theoretical analyzes is satisfied for the general inverse problems . As A and A ( X∗ ) are fixed during optimization , we simplify LY ( A ( X∗ ) , A ( X ) ) as LY ( X ) below . Ui , j is the ( i , j ) th entry of U , U : ,k and Uk , : are the k-th column and the k-th row of U respectively .

Summary: This paper proposes a generalization and extension to deep matrix factorization as presented in the former paper from [Arora et al, NeurIPS 2019]. The extension allows more complex model which include inverse problems. The generalization part is build on a "vanishing" regularization which leads to better dynamics and convergence. These results are given by a theoretical analysis which highlight the effect of the proposed model. Finally the experiments illustrate the advantage of the model compared to state-of-art methods.