AIR-Net: Adaptive and Implicit Regularization Neural Network for matrix completion

1 INTRODUCTION . The matrix completion ( MC ) problem , which aims to recover a matrix X∗ ∈ Rm×n from its partially observed elements , has arisen in numerous domains , ranging from computer vision ( Wen et al. , 2012 ) , recommender system ( Netflix , 2009 ) , and drug-target interaction ( DTI ) ( Mongia & Majumdar , 2020 ) . This fundamental problem is ill-posed without assumptions on X∗ since we have many completions . So it is essential to impose additional information or priors on the unknown matrix/signal . To describe the prior for natural signal , or restrict the solution in the corresponding space is difficult . Classical methods for MC are mainly based on low-rank , sparsity or piece-wise smoothness assumption ( Rudin et al. , 1992 ; Buades et al. , 2005 ; Romano et al. , 2014 ; Dabov et al. , 2007 ) . These priors describe simple structural signal well , but may lead to a poor approximation of X∗ with complex structures ( Radhakrishnan et al. , 2021 ) especially when the observed entries are not sampled uniformly at random . Recently , deep neural networks ( DNN ) have shown a strong ability in extracting complex structures from large datasets ( Li et al. , 2018 ; Mukherjee et al. , 2021 ) . However , such a large number of data sets can not be obtained in many scenarios . Fortunately , DNN also works in solving some inverse problems without any extra training set ( Ulyanov et al. , 2018 ) . Over-parametric DNN performs well on a single matrix is a mysterious phenomenon . One of the explanations is there exists implicit regularization during training ( Arora et al. , 2019 ; Xu et al. , 2019 ; Rahaman et al. , 2019 ; Chakrabarty & Maji , 2019 ) . Although DNN with implicit regularization outperforms some classical methods , it is insufficient to describe the space of complex X∗ . Extra-explicit regularization can improve its performance in signal recovery ( Metzler et al. , 2018 ; Boyarski et al. , 2019a ; Liu et al. , 2019 ; Li et al. , 2020 ) . However , such explicit priors are often valid only for specific data or sampling patterns . A more flexible regularization is required to meet practical MC problems . We introduce flexibility in this paper by firstly representing the explicit regularization using DNN without any extra training set . The explicit regularization we begin with is Dirichlet Energy ( DE ) , which is formulated as tr ( X > LX ) , with L a Laplacian matrix describing the similarity between columns . Note that L in DE is fixed during iteration . Building an exact L based on incomplete observation is very challenging . Therefore , we parameterize L with DNN , and revise L iteratively during training . Furthermore , We combine the learned DE , which is an adaptive regularizer , with implicit regularization to form a new regularization method for MC named AIR-Net . The interaction between explicit regularization and implicit regularization in solving MC problems is further studied . The results show that combining the two can obtain a new , more flexible regularization model and enhance the low-rank preference of implicit regularization . In many examples , AIR-NET has a more vital feature representation ability and more comprehensive application range and shows state-of-art performance . 2 ADAPTIVE AND IMPLICIT REGULARIZATION NEURAL NETWORK . Our model is proposed as follow : min X , Wi Lall = LY ( A ( X∗ ) , A ( X ) ) + N∑ i=1 λi · RWi ( Ti ( X ) ) ( 1 ) where A ( X ) = X |Ω= { Xi , j , ( i , j ) ∈ Ω 0 , ( i , j ) 6∈ Ω and Ω are the observed coordinates set , and the other entries are missing . Different from other regularization models for MC , here X is represented by a neural network which tends to be low-rank implicitly ( Section 2.1 ) , and RWi is an adaptive regularization with a forward neural network represented Laplacian matrix ( Section 2.2 ) . The detailed notations will be introduced in the corresponding sections . A specific case of Equation 1 for matrix completion is given in Section 2.3 . 2.1 DMF AS AN IMPLICIT REGULARIZATION . In order to make the model suitable for more matrix types , we need a more general data prior . The low-rank is a very general prior in various matrix types . There are two main ways to encode the low-rank prior into model : ( a ) Adding an explicit regularization term such as rank and nuclear norm ( Candès & Recht , 2009 ; Lin et al. , 2010 ) . ( b ) Using a low-dimensional latent variable model to represent X , including matrix factorization ( MF ) and its varieties ( Koren et al. , 2009 ; Fan & Cheng , 2018 ) . The first case suffers from the saturation issue , which is induced by explicit regularization . The second one faces the problem of estimating a proper latent variable dimension . Unlike the existing MF model , which constricts the size of the shared dimension of the factorized matrix , DMF can take a large shared dimension and still preserve the low-rank property without explicit regularization . This is the so-called implicit low-rank regularization of DMF : X ( t ) = W [ L−1 ] ( t ) W [ L−2 ] ( t ) . . .W [ 1 ] ( t ) W [ 0 ] ( t ) ∈ Rm×n , where L is the depth of MF . W [ l ] ( t ) represents the l-th matrix at the step t during training . The results are given under a mild assumption 1 in Section A.2 . This property helps avoiding dimension estimation and saturation issues . As for the details of the implicit low-rank we will discuss in Section A.2 . 2.2 ADAPTIVE REGULARIZER . Apart from the low-rank prior , self-similarity is also a typical prior . The patch in the image and the rating behavior of users are all examples of self-similarity . For example , there is always a certain degree of self-similarity between the blocks in the image . A classical way to encode the similarity prior to X is Dirichlet Energy ( DE ) which is formulated as tr ( X > LX ) . But DE will face two problems in applications : ( a ) L is unknown in MC problem , construct L based on incomplete X may induce worse prior . ( b ) The formulation of DE only encodes the similarity of the columns of X . Other similarities such as block similarity can not be captured . To address both of these issues , we parameterize L with DNN and replace X by a transformed Ti ( X ) to capture the self-similarity flexibly . The adaptive regularization is defined as RWi ( Ti ( X ) ) = tr ( Ti ( X ) > Li ( Wi ) Ti ( X ) ) , i = 1 , 2 , . . . , N where Li ∈ Rmi×mi is parameterized by Wi ∈ Rmi×mi . To keep the Laplacian properties of Li , special design for the parameterized structure is important . We design a forward neural network which encodes the properties of Laplacian matrix in structure . The details are discussed in A.4 . Ti : Rm×n 7→ Rmi×ni transforms X into special domain , which makes the AIR-Net possible to capture various relationships embedded in data . The common choice can be Ti ( X ) = X which captures the relationship between columns . Regularization captures the relationship between rows when Ti ( X ) = X > . Especially if Ti ( X ) = [ vec ( block ( X ) ) 1 , vec ( block ( X ) ) 2 , . . . , vec ( block ( X ) ) ni ] , where vec ( block ( X ) ) j ∈ Rmi is the vectorization of j-th block in X row by row , then the similarity among blocks can be obtained . A natural problem that arises is what the Li looks like during training . Obviously , RWi reaches minimum when Li = 0 , and this is called a trivial solution . The most exciting thing is that when we minimize Equation 1 with the gradient descent algorithm , { Li ( Wi ( t ) ) } converges to a non-trivial solution . Another expected phenomenon is that RWi vanishes at the end and will not cause the so-called saturation issue . The saturation issue is a bias term that dominates the overall estimation error due to explicit regularization . We illustrate these phenomena both by theoretical analysis ( Theorem 2 in Section 3.2 ) and numerical experiments ( Section 4 ) . 2.3 AIR-NET FOR MC . In this subsection , we will focus our model on the MC problem . We select T1 ( X ) = X , T2 ( X ) = X > , N = 2 to capture the relationship both in rows and columns of X . Overall , the theoretical analyses for general inverse problem is based on Equation 2. min X , W r , W c Lall = LY ( A ( X∗ ) , A ( X ) ) + λr · RWr ( X ) + λc · RWc ( X > ) , ( 2 ) where X = W [ L−1 ] W [ L−2 ] . . .W [ 1 ] W [ 0 ] , RWr ( X ) = XLr ( Wr ) X > , RWc ( X > ) = X > Lc ( Wc ) X . Specially , our experiments focus on the MC problem which can reform Equation 2 as follows : min W [ l ] , W r , W c Lall = ∑ ( i , j ) ∈Ω ∣∣Xij −X∗ij∣∣+ λr · RWr ( X ) + λc · RWc ( X > ) , ( 3 ) with l = 0 , 1 , · · · , L − 1 . The parameters in Equation 3 is updated by gradient descent algorithm or its variations . We stop the iteration until ∣∣RWr ( T+1 ) −RWr ( T ) ∣∣ < δ and∣∣RWc ( T+1 ) −RWc ( T ) ∣∣ < δ . The recovered matrix is X̂ ( T ) = W [ L−1 ] ( T ) · · ·W [ 0 ] ( T ) . Some works which combine implicit and explicit regularization also can be regarded as a special case of Equation 1 . Both the Total Variation ( TV ) and DE can be regarded as a fixed L. Therefore , the framework of Equation 1 also contains DMF+TV ( Li et al. , 2020 ) , DMF+DE ( Boyarski et al. , 2019a ) . So far , we can not see any essential difference between Equation 3 and these models . We will illustrate the amazing properties of the model in the next section . 3 THEORETICAL ANALYSIS . In this section , we will analyze the properties based on the dynamics of Equation 3 . Theorem 1 shows that our proposed regularization enhances the implicit low-rank regularization of DMF . Theorem 2 shows that the adaptive regularization will converge to a minimum while capturing the inner structure of data flexibly . Although this paper focus on MC problem , the following theoretical analyzes is satisfied for the general inverse problems . As A and A ( X∗ ) are fixed during optimization , we simplify LY ( A ( X∗ ) , A ( X ) ) as LY ( X ) below . Ui , j is the ( i , j ) th entry of U , U : ,k and Uk , : are the k-th column and the k-th row of U respectively .

Summary: In this manuscript, the authors consider matrix completion problems. Leveraging recent advances on implicit regularization in (deep) matrix factorization problems, a new architecture for matrix completion is proposed. Specifically, the authors parameterize the unknown low-rank matrix as a deep linear network, which has been shown to exhibit a low-rank bias when learned via gradient descent. Additional regularization terms based on the Dirichlet energy are added to encourage other structural priors in the recovered solution, such as self-similarity between columns or blocks. The underlying Laplacian matrix is parameterized by learnable weights. The authors analyze the dynamics of gradient flow applied to their optimization problem and show empirically that this approach improves performance in certain settings.