Neuron-Enhanced Autoencoder based Collaborative filtering: Theory and Practice

1 INTRODUCTION . Recommendation system aims to provide personalized recommendation based on various information such as user purchase records , social networks , user features , and item ( or product ) features . With the fast growth of E-commence , social media , and content provider , recommendation systems play more and more important roles in our daily life and have changed our life both explicitly and implicitly . In general , recommendation systems can be organized into three categories ( Adomavicius & Tuzhilin , 2005 ; Zhang et al. , 2019 ) : content based method , collaborative filtering , and hybrid methods . The content based methods recommend similar items to a user or recommend one item to similar users , where the similarity is usually obtained from side information such as genre , occupation , and age . Collaborative filtering ( CF ) assumes that there exist potential correlations within both users and items , which can be implicitly used to predict unknown ratings . Hybrid methods are combinations of content based methods and CF methods ( Adomavicius & Tuzhilin , 2005 ; Zhang et al. , 2019 ; Su & Khoshgoftaar , 2009 ) . CF is at the cores of many recommendation systems . Early CF methods ( Resnick et al. , 1994 ) compute the similarity between users or items directly from the ratings to make prediction . This kind of method is also called memory based CF , which is easy to implement and has high interpretability . One limitation is that the similarity computed from the ratings is not informative owing to the high sparsity of the rating . Another line of CF is model based method ( Ungar & Foster , 1998 ; Shani et al. , 2002 ) that utilizes historical data to train a machine learning model such as Bayesian network ( Breese et al. , 1998 ; Miyahara et al. , 2000 ) for recommendation . Model based methods are more effective than content based methods in learning complex hidden preference and handling the sparsity problem . Note that both content based and model based methods do not work when there are new items or users without ratings , which is known as the cold start problem . A popular strategy for solving the problem is to incorporate side information into CF methods ( Adams et al. , 2010 ; Welling et al. , 2012 ; Zhang et al. , 2017 ) . In the past decades , matrix factorization ( Billsus & Pazzani , 1998 ; Mnih & Salakhutdinov , 2008 ; Koren et al. , 2009 ) and matrix completion ( Candès & Recht , 2009 ; Shamir & Shalev-Shwartz , 2014 ; Sun & Luo , 2015 ; Chen et al. , 2016 ; Fan et al. , 2019 ) have been extensively studied and used in CF . These methods usually exploit the potential low-rank structure of the incomplete rating matrix via embedding items and users into a latent space of reduced dimension , where the observed ratings are approximated by the inner products of the user feature vectors and item feature vectors . The low-rankness is usually obtained by low-rank factorization ( Koren et al. , 2009 ) , nuclear norm minimization ( Candès & Recht , 2009 ) , or Schatten-p quasi norm minimization ( Fan et al. , 2019 ) . Particularly , Lee et al . ( 2016 ) proposed a local low-rank matrix approximation ( LLORMA ) that approximates the rating matrix as a weighted sum of a few low-rank matrices . LLORMA outperformed vanilla low-rank matrix completion methods in collaborative filtering , which indicates that the rating matrices in real applications may have more complicated structures rather than a single low-rank structure . The success of neural networks and deep learning in computer vision and natural language processing inspired researchers to design neural networks for CF ( Salakhutdinov et al. , 2007 ; Dziugaite & Roy , 2015 ; Sedhain et al. , 2015 ; Wu et al. , 2016 ; Zheng et al. , 2016 ; He et al. , 2017 ; van den Berg et al. , 2017 ; Fan & Cheng , 2018 ; Yi et al. , 2020 ) . For instance , Salakhutdinov et al . ( 2007 ) proposed a restricted Boltzmann machines ( Hinton et al. , 2006 ) based CF method called RBM-CF , which showed high performance in the Netflix challenge ( Bennett & Lanning , 2007 ) . Sedhain et al . ( 2015 ) proposed AutoRec , an autoencoder ( Hinton & Salakhutdinov , 2006 ; Bengio et al. , 2007 ) based CF method , which predicts unknown ratings by an encoder-decoder model x̂ = W2σ ( W1x ) , where x denotes the incomplete ratings on one item or of one user and W1 , W2 are weight matrices to optimize . Unlike RBM-CF , which is probabilistic and generative , AutoRec provides a discriminative approach . AutoRec outperformed LLORMA slightly on several benchmark datasets ( Sedhain et al. , 2015 ) . In addition , adding depth is able to improve the performance of AutoRec ( Sedhain et al. , 2015 ) . Inspried by Neural Autoregressive Distribution Estimator ( NADE ) ( Larochelle & Murray , 2011 ) and RBM-CF ( Salakhutdinov et al. , 2007 ) , Zheng et al . ( 2016 ) proposed a method called CF-NADE , in which parameters are shared between different ratings and achieved promising performance in several benchmarks . Muller et al . ( 2018 ) proposed a kernel based reparametrized neural network , in which the weight between two units is set to be a weighted kernel-function of the location vectors . The method works well in data visualization and recommendation systems . Interestingly , Yi et al . ( 2020 ) found that the expected value of the output layer of a neural network depends on the sparsity of the input data . They proposed a simple yet effective method called sparsity normalization to improve the performance of neural networks with sparse input data such as the highly incomplete rating matrices in CF . It is worth mentioning that existing autoencoder based CF methods such as ( Sedhain et al. , 2015 ; Wu et al. , 2016 ; Muller et al. , 2018 ; Yi et al. , 2020 ) use linear activation function in the output of the decoder , i.e. , x̂ = WLhL−1 , where WL denotes the weights of the output layer and hL−1 denotes the features given by the last hidden layer . Thus , these methods are under the assumption that the ratings are linear interactions between user features and item features , though the features can be nonlinear . Such an assumption may not be true or not optimal in real problems , especially when the data are bounded ( e.g . images ) or are collected by sensors ( e.g . medical and chemical sensors ) with nonlinear response functions . We suspect that the rating values given by users on items are from some nonlinear response functions because humans have complex emotion or decision curves ( LeDoux , 2000 ; Baker , 2001 ) . A naive method to incorporate nonlinear interaction is using nonlinear activation functions such as sigmoid function ( with rescaling ) in the output layer of the decoder , which however has much lower performance than using a linear activation function . That ’ s why existing autoencoder based CF methods use only linear activation function . Note that a pre-specified activation function for the output layer of the decoder may work on specific data but may be far away from the possible optimal choice . On the other hand , the theoretical analysis for autoencoder and deep learning based CF is very limited , while there have been many works on the theory of low-rank matrix completion based CF ( Srebro & Shraibman , 2005 ; Candès & Recht , 2009 ; Shamir & Shalev-Shwartz , 2014 ; Fan et al. , 2019 ) . Contribution . In this paper , we present a novel neural network CF method named NE-AECF , an enhanced autoencoder approach for recommendation system . NE-AECF is composed of two different neural networks , one is an autoencoder to reconstruct the incomplete rating matrix , while the other is an element-wise neural network to learn an activation function adaptively for the output layer of the autoencoder . We provide theoretical analysis for NE-AECF , which explains the superiority of our method . Specifically , we prove that the element-wise neural network can reduce the upper bound of the prediction error for the unknown ratings . We also prove that the data sparsity is not problematic but useful and the prediction performance is closely related to the difference between the number of users and the number of items . Further , we demonstrate empirically our NE-AECF on benchmarks : MovieLen-100k and MovieLen-1M , achieving state-of-the-art results . Notation . We use x ( or X ) , x , and X to denote scalar , vector , and matrix respectively . We use ‖x‖ to denote the Euclidean norm of vector x , use ‖X‖F and ‖X‖2 to denote the Frobenius norm and spectral norm of matrix X respectively . The ` 21 norm of matrix is denoted by ‖X‖2,1 : =∑ i ‖xi‖ , where xi denotes the i-th column ofX . The ` ∞ norm of matrix is denoted by ‖X‖∞ : = maxij |Xij | . We use |S| to denote the cardinality of set S. The symbol ′ ′ denotes the Hadamard product between vectors or matrices . The symbol ′◦′ denotes function composition . 2 NEURON-ENHANCED AECF . Suppose we have an incomplete rating matrix X̃ = ( x̃1 , x̃2 , . . . , x̃n ) ∈ Rm×n , where m is the number of users and n is the number of items ( without loss of generality ) . X̃ij ≥ 0 denotes the rating given by user i on item j and X̃ij = 0 indicates an unobserved rating . S denotes the set of observed ratings . We have X̃ij = Xij for all ( i , j ) ∈ S. Our goal is to predict the unobserved ratings Xij , ( i , j ) ∈ [ m ] × [ n ] \S , from X̃ . We want to learn a nonlinear function f : Rm 7→ Rm such that n∑ i=1 ∥∥si ( x̃i − f ( x̃i ) ) ∥∥2 ( 1 ) is as small as possible , where si is a binary vector denoting whether the the corresponding element in xi is zero ( unknown ) or not . The motivation is predicting the missing entries of xi using its observed entries , though the missing entries of xi are filled by zeros before performing f . More formally , we consider the following problem minimize f∈F ∥∥∥S ( X̃ − f ( X̃ ) ) ∥∥∥2 F ( 2 ) where S = ( s1 , s2 , . . . , sn ) , f is performed on each column of X̃ separately , and F denotes a hypothesis set of m to m functions . We have infinite choices for F . For example , F can be a set of functions in the form of neural network with some parameters W ∈ W , whereW denotes a set of matrices under some constraints . In this case , problem ( 2 ) defines a denoising autoencoder or stacked denoising autoencoders ( Vincent et al. , 2010 ) , where the noises are introduced by filling the missing ratings with zeros . Let f be an autoencoder with linear activation function in the output layer . Then ( 2 ) becomes minimize W1 , W2 ∥∥∥S ( X̃ −W2σ ( W1X̃ ) ) ∥∥∥2 F + λ ( ‖W1‖2F + ‖W2‖2F ) , ( 3 ) whereW1 ∈ Rd×m andW2 ∈ Rm×d are weights matrices to learn and λ is a nonnegative constant to control the strength of weight decay . We have omitted the bia terms for simplicity . σ denotes an activation function such as ReLU σ ( x ) = max ( x , 0 ) and Sigmoid σ ( x ) = 1/ ( 1 + exp ( −x ) ) . Note that ( 3 ) is exactly the basic model considered by Sedhain et al . ( 2015 ) , Wu et al . ( 2016 ) , Muller et al . ( 2018 ) , and Yi et al . ( 2020 ) . Once ( 3 ) is used , the following assumption is made implicitly . Assumption 1 . There exist two matricesA ∈ Rm×d andB ∈ Rd×n such that ‖S ( X −AB ) ‖F is small enough . The assumption indicates that if d is much smaller than min ( m , n ) , X can be well approximated by a low-rank matrix , which however may not always hold in real applications . Consider the following data generating model X = h ( A′B′ ) , ( 4 ) where h : R1 7→ R1 is an element-wise nonlinear function and A′ ∈ Rm×d , B′ ∈ Rd×n may be generated by some nonlinear functions . If the nonlinearity of h is high , X can not be well approximated by a rank-d matrix . These analysis indicates that if the element-wise nonlinearity in generatingX is strong , ( 3 ) should use a large d to ensure a small enough training error . The element-wise nonlinearity widely exists in real data . For example , in imaging science , the intensity of pixels are nonlinear responses of photoelectric element to spectrum . In chemical engineering , many sensors have nonlinear responses . In biomedical engineering , the dose-responses are often nonlinear curves . Hence , in collaborative filtering , the ratings may be nonlinear responses to some latent values , according to the studies on response curve in neuroscience and psychology ( LeDoux , 2000 ; Baker , 2001 ) . Therefore , instead of ( 3 ) , one may consider the following problem minimize W1 , W2 ∥∥∥S ( X̃ − h ( W2σ ( W1X̃ ) ) ) ∥∥∥2 F + λ ( ‖W1‖2F + ‖W2‖2F ) , ( 5 ) where h should be determined beforehand . A naive approach to determining h is choosing a bounded or partially bounded nonlinear function according to the range of the data . For example , if the data are image pixels within [ 0 , 1 ] , one may use Sigmoid function . If the data are nonnegative , one may use ReLU . However , such choices only considered the range of the data , which is just a small portion of the nonlinearity . Within the range , the true response functions are not necessarily linear ( ReLU ) or related to exponential ( Sigmoid ) , and can be much more complicated . As it is difficult to choose a suitable nonlinear function h in advance , we propose to learn h from the data adaptively , i.e. , minimize W1 , W2 , h∈H ∥∥∥S ( X̃ − h ( W2σ ( W1X̃ ) ) ) ∥∥∥2 F + λ ( ‖W1‖2F + ‖W2‖2F ) , ( 6 ) where H denotes a hypothesis set of nonlinear functions from R1 to R1 . We have different approaches to learning h. The first approach is combining various activation functions , i.e. , hθ ( z ) = k∑ i θiσi ( z ) , ( 7 ) where σi ( · ) are different activation functions and θ = ( θ1 , . . . , θk ) > are parameters to estimate . However , it is not clear whether ( 7 ) is able to approximate a wide range of nonlinear functions . The second approach is using polynomial functions , i.e. , hθ ( z ) = k∑ i θzk . ( 8 ) It is a k-order polynomial function and can well approximate any smooth functions provided that k is sufficiently large . Another approach is using a neural network , i.e. , hΘ ( z ) = ΘLΘ ( σΘ ( ΘLΘ−1σΘ ( · · ·σΘ ( Θ1z ) · · · ) ) ) , ( 9 ) where Θ1 and ΘLΘ are vectors , Θ2 , . . . , ΘLΘ−1 are matrices , and σΘ is a fixed activation function . According to the universal approximation theorems ( Pinkus , 1999 ; Sonoda & Murata , 2017 ; Lu et al. , 2017 ) , ( 9 ) is able to approximate any continuous functions provided that the network is wide enough or deep enough . Since ( 9 ) is more flexible than ( 7 ) and ( 8 ) in function approximation , we propose to solve the following problem minimize W , Θ ∥∥∥S ( X̃ − hΘ ( gW ( X̃ ) ) ) ∥∥∥2 F + λW LW∑ l=1 ‖Wl‖2F + λΘ LΘ∑ l=1 ‖Θl‖2F , ( 10 ) where W = { W1 , . . . , WLW } , Θ = { Θ1 , . . . , ΘLΘ } , and gW ( X̃ ) = WLW ( σW ( WLW−1σW ( · · ·σW ( W1X̃ ) · · · ) ) ) . ( 11 ) In addition , we assume Wl ∈ Rdl×dl−1 , l ∈ [ LW ] , and Θl ∈ Rpl×pl−1 , l ∈ [ LΘ ] . Note that d0 = dLW = m and p0 = pLΘ = 1 . Comparing ( 10 ) with ( 2 ) , we see that we have replaced f by hW ◦ gΘ with Frobenius-norm constrained weight matrices . Model ( 10 ) is exactly our neuronenhanced autoencoder based collaborative filtering ( NE-AECF ) method . There are two different neural networks . The first one is an autoencoder defined by hΘ◦gW , which is to learn an contraction map from the incomplete rating matrix X̃ to itself or its observed entries more precisely . The second neural network is performed in an element-wise manner to learn an activation function h adaptively for the output layer of the autoencoder or stancked autoencoders . Figure 1 shows an example schematic of NE-AECF , where LW = LΘ = 2 and Z = gW ( X̃ ) .

Summary: This paper considers an autoencoder with one hidden layer. While the output layer uses a linear activation function in several papers in the literature, this paper proposes to use a nonlinear activation function in the output layer. Moreover, instead of using a fixed function (e.g., sigmoid), this non-linear activation function is learned in this paper, using a parametrized form. Generalization bounds are  derived, and experiments are conducted.