Collaborative Filtering with Smooth Reconstruction of the Preference Function

The problem of predicting the rating of a set of users to a set of items in a recommender system based on partial knowledge of the ratings is widely known as collaborative filtering . In this paper , we consider a mapping of the items into a vector space and study the prediction problem by assuming an underlying smooth preference function for each user , the quantization at each given vector yields the associated rating . To estimate the preference functions , we implicitly cluster the users with similar ratings to form dominant types . Next , we associate each dominant type with a smooth preference function ; i.e. , the function values for items with nearby vectors shall be close to each other . The latter is accomplished by a rich representation learning in a so called frequency domain . In this framework , we propose two approaches for learning user and item representations . First , we use an alternating optimization method in the spirit of k-means to cluster users and map items . We further make this approach less prone to overfitting by a boosting technique . Second , we present a feedforward neural network architecture consisting of interpretable layers which implicitely clusters the users . The performance of the method is evaluated on two benchmark datasets ( ML-100k and ML-1M ) . Albeit the method benefits from simplicity , it shows a remarkable performance and opens a venue for future research . All codes are publicly available on the GitLab . 1 INTRODUCTION . Nowadays , recommender systems ( RS ) are among the most effective ways for large companies to attract more customers . A few statistics are sufficient to attract attention towards the importance of RS : 80 percent of watched movies on Netflix and 60 percent of video clicks on Youtube are linked with recommendations ( Gomez-Uribe & Hunt , 2015 ; Davidson et al. , 2010 ) . However , the world of RS is not limited to video industry . In general , recommender systems can be categorized into three groups ( Zhang et al. , 2019 ) : collaborative filtering ( CF ) , content-based RS , and hybrid RS depending on the used data type . In this paper , we focus on CF , which uses historical interactions to make recommendations . There might be some auxiliary information available to the CF algorithm ( like the user personal information ) ; however , a general CF method does not take such side information into account ( Zhang & Chen , 2019 ) . This includes our approach in this paper . Recently , deep learning has found its way to RS and specifically CF methods . Deep networks are able to learn non-linear representations with powerful optimization tools , and their efficient implementations have made then promising CF approaches . However , a quick look at some pervasive deep networks in RS ( e.g. , He et al . ( 2017 ) and Wu et al . ( 2016 ) ) shows that the utilization of deep architectures is limited to shallow networks . Still , it is unclear why networks have not gone deeper in RS in contrast to other fields like computer vision ( Zhang et al. , 2019 ) . We suppose that the fundamental reason that limits the application of a deeper structure is the absence of interpretability ( look at Seo et al . ( 2017 ) , for example ) . Here , interpretability can be defined in two ways ( Zhang et al. , 2019 ) ; first , users be aware of the purpose behind a recommendation , and second , the system operator should know how manipulation of the system will affect the predictions ( Zhang et al. , 2018 ) . This paper addresses both issues by formulating the recommendation as a smooth reconstruction of user preferences . Particularly , our contributions are : • The CF problem is formulated as the reconstruction of user preference functions by minimal assumptions . • An alternating optimization method is proposed that effectively optimizes a non-convex loss function and extracts user and item representations . In this regard , effective clustering methods are proposed and tested . • A feed-forward shallow architecture is introduced , which has interpretable layers and performs well in practice . • Despite the simplicity and interpretability of the methods , their performance on benchmark datasets is remarkable . 1.1 RELATED WORKS . The applied methods in CF are versatile and difficult to name . Below , we explain a number of methods which are well-known and are more related to our work . Multilayer perceptron based models . A natural extension of matrix factorization ( MF ) methods ( Mnih & Salakhutdinov , 2008 ) are Neural Collaborative Filtering ( NCF ) ( He et al. , 2017 ) and Neural Network Matrix Factorization ( NNMF ) ( Dziugaite & Roy , 2015 ) . Both methods extend the idea behind MF and use the outputs of two networks as the user and the item representations . The innerproduct makes the prediction of two representations . Although our work has some similarity to this method , we model users by functions and represent these functions in a so-called frequency domain . Thus , user and item representations are not in the same space . AutoEncoder based models . AutoRec ( Sedhain et al. , 2015 ) and CFN ( Strub et al. , 2016 ) are wellknown autoencoder ( AE ) structures that transform partial observations ( user-based or item-based ) into full row or column data . Our method differs from AE structures as our network use item ( user ) representations and predicts user ( item ) ratings . 2 SMOOTH RECONSTRUCTION FROM NON-UNIFORM SAMPLES . Rating as the output of the preference function . Most of the time , a finite set of features can characterize users and items that constitute the recommendation problem . Although no two users or items are exactly the same , the number of characterization features can be considerably small without losing much information . Assume that item i is characterized by the vector xi ∈ X ⊂ Rd . We further assume that all users observe similar features of an item and user u ’ s ratings are determined by a preference function fu : X → [ cmin , cmax ] . The recovery of a general preference function might need an indefinite number of samples , i.e. , observed ratings . However , we do not expect user attitudes to change too much with small changes in an item ’ s feature . E.g. , if the price is the determinative factor in someone ’ s preference , small changes in the price must not change the preference over this item significantly ( look at figure 1 ) . Reconstruction of bandlimited 1D signals . Let us start with the simplest case . Consider s [ n ] , n = 0 , 1 , . . . , N − 1 , a 1D signal with length N . We call s to have bandwidth M < N2 if there is a representation ŝ [ m ] , m = −M , −M + 1 , . . . , M − 1 , M that can represent s as : s [ n ] = M∑ m=−M ŝ [ m ] ej2π ( mn/N ) ( 1 ) So 2M +1 distinct samples from s would be enough to calculate ŝ . For an analytical approach , it is useful to interpret equation 1 as a discretization of a trigonometric continuous equation : h ( x ) = M∑ ( m=−M ) ame j2π ( mx ) , a ∈ C2M+1 , x ∈ [ 0 , 1 ) ( 2 ) Mirroring . Smoothness usually is used to refer to bandlimited signals around the zero-frequency which can be represented by equation 1 . However , we use the smooth finite-length signal to refer to a real-valued finite-length signal that has intuitively smooth behavior in its non-zero domain . figure 2 shows an example . The trigonometric functions in equation 2 can not approximate such signals well even if we shift and scale the domain to [ 0,1 ] because the original signal is not periodic . One possible solution to make the trigonometric functions still a good representative for the finitelength signal would be mirroring . figure 2 shows the shifted , scaled , and mirrored signals in 1D space . Extension to multi-dimensional real-valued mirrored signals . equation 2 will be simplified for a real-valued s just to include cosine terms . One can obtain the extension of equation 2 for real-valued mirrored signals as : h ( x ) = h ( x1 , x2 , . . . , xd ) = M∑ m1=0 . . . M∑ md=0 Am1 , m2 , . . . , md cos ( π ( m1x1 + ... +mdxd ) ) , ( 3 ) where x ∈ [ 0 , 1 ] d and A is a d-dimensional real tensor . To simplify the notation , we use mTx to refer m1x1 + ... +mdxd and a to refer vectorized A . Starting from m1 , m2 , . . . , md all to be 0 , one can put all the possible values of m as the columns of matrix C = [ c ] ( M+1 ) d×d with the same order as they appear in vectorizing of A . Now we can rewrite equation 3 with matrix operations : h ( x ) = ( M+1 ) d∑ k=1 ak cos ( πCk , :x ) ( 4 ) Vandermonde matrix . Given r non-uniform samples in [ 0 , 1 ] d , the Fourier coefficients , a , are the solution to the linear system of equations h ( xi ) = si , i = 1 , 2 , . . . , r. The Vandermonde matrix for this system is defined as : V = cos ( C [ x1 , x2 , . . . , xr ] ) , ( 5 ) where the cos ( . ) is the element-wise cosine , and [ . . . ] shows the stacking operator of column vectors . So , the linear system of equations can be shortened by : V Ta = s. Here s is the column vector of r observed si put together . In contrast to the 1D case , there is no simple theorem on the conditions to estimate a correctly . Roughly speaking , this needs the rank of V to be larger than the number of unknowns , i.e. , ( M +1 ) d or in other words , the number of samples ( r ) should be enough larger than ( M + 1 ) d. Reconstruction of the preference function from the observed ratings . We can state the problem of rating prediction , as the reconstruction of the preference function of each user ( fu ) given the observed ratings of that user ( Iu ) . If we assign a d-dimensional characterization vector ( xi ) to each item i that is assumed to lie in X = [ 0 , 1 ] d , we can estimate the user u Fourier coefficients as au = ( V T u ) †su . At the starting point we do not know how items are distributed in X which means Vu will be inaccurate . So , optimizing the reconstruction loss gives fair characterstics for the items in X : min xi , i∈I ∑ u∈U ‖V Tu ( V Tu ) †su − su‖2 . ( 6 ) 3 LEARNING REPRESENTATIONS BY MINIMIZING RECONSTRUCTION LOSS . Minimizing equation 6 , aside from the non-convexity of the cost function , implicitly involves solving V Tu au = su , which can in general be an ill-condition system of linear equations , specially when the user u has few recorder ratings . To reliably estimate the Fourier coefficients au ( user representations ) , group similar users into a number of clusters and use a single representative for each cluster ( virtually increasing the number of available ratings ) . In addition , we further consider a Tikhonov ( L2 ) regularizer to improve the condition number . With this approach , we need to solve min { xi , i∈I } , c L =min ∑ u∈U ‖V Tc ( u ) ac ( u ) − su‖ 2 + λ ∑ k∈C ‖ak‖2 , s.t . 0 ≤ xi < 1 , ( 7 ) where c : U → C is the mapping of the users into clusters , C is the set of clusters and Vk is the Vandermonde matrix associated with the cluster k ( considering all the users in a cluster as a superuser ) . Hence , Vk is a function of { xi , i ∈ ∪ { u : c ( u ) =k } Iu } . The penalty parameter λ shall be tuned via cross validation . Moreover , the Fourier coefficients ak for the cluster k are obtained by : ak = ( VkV T k + λI ) −1Vksk , ( 8 ) where sk is the vector of all observed ratings in the cluster k. In the sequel , we propose two approaches for minimizing equation 7 . In the first approach ( Section 3.1 ) , we alternatively find min { xi , i∈I } L and minc L ; as minc L requires a combinatorial search , we introduce an approximate algorithm , named k-representation ( Section 3.1.1 ) inspired by the k-means technique . Each iteration of k-representation consists of assigning each user the cluster with the lowest reconstruction loss , and updating the cluster representatives . In the second approach ( Section 3.2 ) , we train a neural network to jointly characterize the items and cluster the users . For this , the loss function of equation 7 is modified to accommodate for soft clustering .

Summary: This paper proposes an approach based on Fourier transforms to predict ratings in collaborative filtering problems. The paper’s scope (“smooth reconstruction functions”) gets immediately narrowed down to Fourier transforms--it would be nice to provide some motivation for this choice over alternative smooth functions. The paper then clusters the users as a way to reduce the number of parameters in the model, given that the Fourier transform itself does not reduce it. As a further step, the clustering is replaced by a soft-clustering learned by a neural network. In the experiments, the RMSE of the rating prediction problem is worse than some baselines and better than others.