Collaborative Filtering with Smooth Reconstruction of the Preference Function
Authors: Ali Shirali
Reza Kazemi
Arash Amini
Venue:  (2021)
Keywords: collaborative filtering
recommender system
sampling theory

Abstract: The problem of predicting the rating of a set of users to a set of items in a recommender system based on partial knowledge of the ratings is widely known as collaborative filtering. In this paper, we consider a mapping of the items into a vector space and study the prediction problem by assuming an underlying smooth preference function for each user, the quantization at each given vector yields the associated rating. To estimate the preference functions, we implicitly cluster the users with similar ratings to form dominant types. Next, we associate each dominant type with a smooth preference function; i.e., the function values for items with nearby vectors shall be close to each other. 
The latter is accomplished by a rich representation learning in a so-called frequency domain. In this framework, we propose two approaches for learning user and item representations. First, we use an alternating optimization method in the spirit of $k$-means to cluster users and map items. We further make this approach less prone to overfitting by a boosting technique. 
Second, we present a feedforward neural network architecture consisting of interpretable layers which implicitely clusters the users. The performance of the method is evaluated on two benchmark datasets (ML-100k and ML-1M). Albeit the method benefits from simplicity, it shows a remarkable performance and opens a venue for future research. All codes are publicly available on the GitLab.

Under review as a conference paper at ICLR 2021
COLLABORATIVE FILTERING WITH SMOOTH RECON- STRUCTION OF THE PREFERENCE FUNCTION
Anonymous authors Paper under double-blind review
ABSTRACT
The problem of predicting the rating of a set of users to a set of items in a rec- ommender system based partial knowledge of the ratings is widely known collaborative filtering. In this paper, we consider mapping of the items into a vector space and study the prediction problem by assuming an underlying smooth preference function for each user, the quantization at each given vector yields the associated rating. To estimate the preference functions, we implicitly cluster the users with similar ratings to form dominant types. Next, we associate each dom- inant type with smooth preference function; i.e., the function values for items with nearby vectors shall be close to each other. The latter is accomplished by a rich representation learning in a so called frequency domain. In this framework. we propose two approaches for learning user and item representations. First, we use an alternating optimization method in the spirit of k-means to cluster users and map items. We further make this approach less prone to overfitting by boosting technique. Second, we present a feedforward neural network architecture consist- ing of interpretable layers which implicitely clusters the users. The performance the method is evaluated on two benchmark datasets (ML-110K and MI.-IM). Albeit the method benefits from simplicity, it shows a remarkable performance and opens a venue for future research. All codes are publicly available on the GitLab.
INTRODUCTION
Nowadays, recommender systems (RS) are among the most effective ways for large companies to attract more customers. A few statistics are sufficient to attract attention towards the importance of RS: 80 percent of watched movies Netflix and 60 percent of video clicks Youtube are linked with recommendations (Gomez Uribe & Hunt 2013 Davidson et al. 2010). However, the world of RS is not limited to video industry.
In general, recommender systems can be categorized into three groups (Zhang et al. 2019): col- laborative filtering (CF), content-based RS, and hybrid RS depending the used data type. In this paper, we focus on CF, which uses historical interactions to make recommendations. There might be some auxiliary information available to the CF algorithm (like the user personal information); however, a general CF method does not take such side information into account (Zhang & Chen 2019). This includes our approach in this paper.
Recently, deep learning has found its way to RS and specifically CF methods. Deep networks are able to learn non-ling non-linear representations with powerful optimization tools, and their efficient imple- mentations have made then promising CF approaches. However, a quick look at some pervasive deep networks in RS (e.g.. He al. (217)) and Wu et al. (2016) shows that the utilization of deep architectures is limited to shallow networks. Still, it is unclear why networks have not gone deeper in RS contrast to other fields like computer vision (Zhang et al. 2019). We suppose that the fundamental reason that limits the application of deeper structure is the absence of interpretability (look at Seo et al. (2017), for example). Here, interpretability can be defined two ways (Zhang et al. 2019); first, users be aware of the purpose behind recommendationn and second, the sys- tem operator should know how manipulation of the system will affect the predictions (Zhang et al. 2018).

---

Under review as a conference paper at ICLR 2021
This paper addresses both issues by formulating the recommendation smooth reconstruction of user preferences. Particularly, our contributions are:
The CF problem formulated as the reconstruction user preference functions by mini- mal assumptions.
An alternating optimization method is proposed that effectively optimizes a non-convex loss function and extracts and item representations. In this regard, effective clustering methods are proposed and tested.
A feed-forward shallow architecture is introduced, which has interpretable layers and per- forms well in practice.
Despite the simplicity and interpretability of the methods, their performance benchmark datasets is remarkable.
1.1 RELATED WORKS
The applied methods in CF are versatile and difficult to name. Below, we explain a number of methods which are well-known and are more related to our work.
Multilayer perceptron based models. natural extension of matrix factorization (MF) methods (Mnih & Salakhutdinov (2008) are Neural Collaborative Filtering (NCF) (He et al. 2017) and Neural Network Matrix Factorization (NNMF) (Dziugaite & Roy 2015). Both methods extend the idea behind MF and use the outputs of two networks as the and the item representations. The inner- product makes the prediction of two representations. Although our work has some similarity to this method, model users by functions and represent these functions in so-called frequency domain. Thus, user and item representations are not in the same space.
AutoEncoder based models. AutoRec (Sedhain al., 2015) and CFN (Strub al. 2016) well- known autoencoder (AE) structures that transform partial observations (user-based or item-based) into full row column data. Our method differs from AE structures our network item (user) representations and predicts user (item) ratings.
SMOOTH RECONSTRUCTION FROM NON-UNIFORM SAMPLES
Rating as the output of the preference function. Most of the time, finite set of features can charac- terize users and items that constitute the recommendation problem. Although no two users or items are exactly the same, the number of characterization features can be considerably small without los- ing much information.
Assume that item i is characterized by the vector æ; E X' C Rd We further assume that all users observe similar features of an item and user u's ratings are determined by a preference function fa X [Cminn Cmaz)- The recovery of general preference function might need indefinite number of samples, i.e., observed ratings. However, we do not expect user attitudes to change much with small changes in an item's feature. E.g., if the price the determinative factor in someone's 's preference, small changes in the price must not change the preference over this item significantly (look at figure]]
X2
user
liked
X
X
X
X
X
X
X
X
X1
X1
disliked
Figure 1: Preference function is expected have smooth behavior the space of items.
2

---

Under review as a conference paper at ICLR 2021
Reconstruction of bandlimited 1D signals. Let start with the simplest case. Consider s|n|, n 0, N ID signal with length N. We call to have bandwidth M N if there is representation 8[m], m M, + I,...,M 1, M that can represent e as:
M Σ
s(n]
(1)
So 2M + 1 distinct samples from would be enough calculate 3. For analytical approach, it is as a discretization of trigonometric continuous equation:
useful interpret equation |
h(x)
Σ
am e (2m(mz)
a a 22M+1 x E (0,
(m M)
Mirroring. Smoothness usually is used to refer to bandlimited signals around the zero- pro-frequency which can be represented by equation However, we use the smooth finite-length signal to refer to real-valued finite-length signal that has intuitively smooth behavior in its non-zero domain. figure shows an example. The trigonometric functions in equation 2 can not approximate such signals well even if we shift and scale the domain to [0,1] because the original signal is not periodic. One possible solution to make the trigonometric functions still good representative for the finite- length signal would be mirroring. figure 2 shows the shifted, scaled, and mirrored signals in 1D) space.
Figure 2: Mirroring, shifting and scaling.
Extension to multi-dimensional real-valued mirrored signals. equation will be simplified for real-valued just to include cosine terms. One can obtain the extension equation for real-valued mirrored signals as:
M
M Σ =
h(x) 1(11. x.)
Amg.mg.... my cos ( (m111 mata)),
(3)
0
where æ E [0, 1]4 and d-dimensional real tensor. To simplify the notation, we use m mf x to refer mj ... maFa and refer vectorized A. Starting from mi, m2,.... my all to be O, one can put all the possible values of m as the columns of matrix
[c] [c] with the same order rewrite equation 3 with matrix operations:
they appear in vectorizing of A. Now
h(x)= = (M+1)d
ag cos (T))
(4)
Vandermonde matrix. Given non-uniform samples in [0, 1]d, the Fourier coefficients, a, are the solution to the linear system of equations h(x,,) = s;, = 1, ........7 The Vandermonde matrix for this system defined as:
cos (Cl [[ F2,....
(5)
where the cos(.) is the element-wise cosine, and [...] shows the stacking operator of column vectors.
So, the linear system of equations can be shortened by: V y' s. Here is the column vector of r observed 8; put together. contrast to the ID case, there is no simple theorem on the conditions to estimate correctly. Roughly speaking, this needs the rank of V to be larger than the number of unknowns, i.c., (M + 1)d or other words, the number of samples (r) should be enough larger than (M + d)d"
3

---

Under review as a conference paper at ICLR 2021
Reconstruction of the preference function from the observed ratiiggs. We can state the problem of rating prediction, as the reconstruction of the preference function of each user (fu) given the observed ratings of that user (In). If we assign a d-dimensional characterization vector (æ,) to each item i that is assumed to lie in X' = [0, 1] we can estimate the user u Fourier coefficients as au ) Su- At the starting point do not know how items are distributed in X' which means Vu will be inaccurate. So, optimizing the reconstruction loss gives fair characterstics for the items in X:
E
(V.) -sul?
min I.JEZ NEW
LEARNING REPRESENTATIONS BY MINIMIZING RECONSTRUCTION LOSS
Minimizing equation aside from the non-convexity of the cost function, implicitly involves solv- ing Vi a, Su. which can in general be an ill-condition system of linear equations, specially when the user u has few recorder ratings. To reliably estimate the Fourier coefficients au (user representa- tions), group similar users into number of clusters and use single representative for each cluster (virtually increasing the number of available ratings). In addition, we further consider a Tikhonov (I2) regularizer to improve the condition number. With this approach,
need solve
min L = min (
Σ uEW
Vela ae(w) - sall 8a + Σ
las |ax1"
kEC
s.t. 0s. 1
where e -> the mapping of the users into clusters, C the set of clusters and V2 the Vandermonde matrix associated with the cluster l (considering all the users in cluster as super- user). Hence, V& is a function (æ., E Uf [fw) The penalty parameter À shall be tuned via cross validation. Moreover, the Fourier coefficients ag for the cluster k are obtained by:
ag (V.VV' + A
(8)
where 8k is the vector of all observed ratings in the cluster k. In the sequel, we propose two approaches for minimizing equation 7 In the first approach (Section (.1). alternatively find (i77) L and min, L; as min. L requires a combinatorial search, we introduce an approxi- mate algorithm, named k-representation (Section [.I.I] inspired by the k-means technique. Each iteration of k-representation consists of assigning each user the cluster with the lowest reconstruc- tion loss, and updating the cluster representatives. In the second approach (Section 3.2). train neural network to jointly characterize the items and cluster the users. For this, the loss function of equation is modified to accommodate for soft clustering.
3.1 ALTERNATING OPTIMIZATION
3.1.1 A-REPRESENTATION FOR CLUSTERING THE USERS
The total loss in equation can be divided into partial losses of the form
La VI a |x
for each cluster k. We propose the k-representation (Algorithm I to minimize the overall cost it- eratively. We first randomly set ags, k E C; then, each user is assigned a cluster k for which its reconstruction loss (equation 7) is minimized. After dividing the users into clusters, the represen- tative of each cluster updated via equation [8 and we return again to the clustering task. Similar to the k-means, there is no theoretical guarantee that the method converges to the global optimizer; nevertheless, the overall loss is decreased in each iteration. We shall evaluate the performance the method in the Section ?? on both synthetic and real data.
Boosted k-representaation By introducing both the clustering and L2 regularization in equation S we have improved the robustness of the inverse problem. However, increasing the number of clusters is still a potential issue in estimating the cluster representatives. Here, we propose to learn an ensemble of weak binary clusterings instead of learning all clusters together (Algorithm 2 The idea is find the residuals of predicted ratings for each user and fit new clustering to the residuals. Due to the linearity of the prediction, the final representation is the of weak representations for each user.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 k-representation
Input:
item characteristics {z,, E ET available ratings by each user (Tu, E EU) number of clusters [C]
initialization variance of cluster representations o² Le penalty parameter Output:
user clustering c: +
cluster representatives {ak, k C}
procedure k-REPRESENTATION init. Os from (N x²) for all E C do Calculate V& from 2;8
repeat
for all E l do c(u) + argming V 0g - s,
for all k: E C do Update as equation g until convergence
return { [ax) and e end procedure
Algorithm 2 boosted k-representation
Input:
same as Algorithm
Output:
user representatives (av, u E U} procedure BOOSTED &-REPRESENTATION
for all E do au + 0, Su f 8u for ( 1: [log2 [C]] do
{a (() +- k-representation(16 ((8 (s1111))))
for all E u do au t (w) 8u 8
(
(1)
(1-1) Ve yaa
au au + +a
(1)
end for end procedure
3.1.2 OPTIMIZING ITEM CHARACTERISTICS
The second part of the alternating optimization is to minimize equation w.r.t. item characteristies; i.e, {æ;; i E I}, subject to 0 < æ; < 1. To simplify the equations, we rewrite the total loss in equation 7 with small modification
C=
Σ p(lu,))
(10)
(a.i)E0
where lu,i is [Sw,; a.,2 and O is the set of observed ratings. Here, p is saturating function that reduces the effect of outliers. equation 7 is simply the identity function; however, we chose p(y) = (1
1) to better bound the unpredictable errors. We use the Trust Region Reflective (TRF) algorithm of the scipy py python package the optimization algorithm, which selects the updating steps adaptively and without supervision. To facilitate the optimization, we need calculate the gradient of Lu,t w.r.t. {æ;). obvious that Va,lu, is zero for all # i. For j = i and 1,... (dimension of X') we have:
(M+1)²
Lu,i = (Su,i - of a.))
Σ ==1
Tay, (n
(1I)
Orin
Pre-search. Before using the TRF to optimize the loss, we make use of the current function evaluations update item characteristics. Consider are in the thh iteration and the values

---

Under review as a conference paper at ICLR 2021
V-layer Rocsuo - Pd 13*
hem--ong-hot-layee Aucsun pi
User-layer Invanes mn [M] m1
X X-layer hacmon
desp-cat
Figure 3: Neural network inspired from equation 13
(x)(-1) v((-1) aj 111 from the previous iteration available. We define
(t- 2 =x (t-1) argmingez ||a--V (a pp-
(12)
Then, we use { (t- 2) as the input to TRM and run a few iterations to get (x(()), This pre-search makes the optimization process less prone to stagnate local minima.
3.2 RECONST-NET: A FEED-FORWARD SHALLOW NETWORK FOR PREFERENCE
RECONSTRUCTION
Recent advances in deep learning have made the neural networks great tools even for traditional well-known algebraic calculations. Specifically, neural networks can be optimized effectively with various methods and efficient implementations that boost training. Further, some useful techniques like batch normalization, drop-out, etc., are available, which helps avoid overfitting. In this sec- tion, we will reformulate equation 7 for soft clusters and design an architecture that learns items' characteristics and users' representations concurrently. First we modify equation 7 for soft clus- tering. Consider e 4 + ClCl is the assignment function that determines how much each user is belonged each cluster. We intentionally do not constraint norm of the e and let to be scaled appropriately for different user. The total reconstruction loss is:
L -=
1 Vo ax) sull 2 + sull LEC
Σ ||ax|1
(13)
NEWL
REC
figure 3 shows an inspired architecture from 13 consists of six layers (four hiddens) which three of them trainable. The observed ratings will be supplied per item. Each neuron corresponds to an item at the input layer, and the input data should be one-hot vectors. The next layer is X-layer, which is dense layer with units. We interpret the weights from unit the input layer to the X-layer as E; At the V-layer, the item's is representation (æ,) is multiplied by C' (EQ) and forms v, V-layer does have any trainable parameters. Next, there the soft-clustering layer, dense layer with nı neurons. We interpret weights going from the V-layer to unit of the soft-clustering layer as ag, i.e., the representation of the &d soft cluster. Finally, the output layer or equivalently the User-layer is dense layer with MI units. Weights going from the soft-clustering layer to unit determine e(u).
The drop-out layer (not depicted) after V-layer has important role in preventing the network from overfitting. Dropping-ous out makes sense because the observed ratings usually come with a lot of uncertainty in real application, i.e., the same user might rate the same item differently when asked for re-rating, and this is the nature of real data. The drop-out layer prevents overfitting by stopping the network from relying
specific part of the items' characteristics.
One way to increase the capacity of the method and capture non- non-linear interactions is to let the user's representation be nonlinear function of clusters' reconstructed ratings. i.e., to change equation T3

---

Under review as a conference paper at ICLR 2021
Table 1: Datasets summary
DATASET
#USER
#ITEM
#RATINGS
#DENSITY variable
RATING RANGE
Synthetic
50
200
variable
1,
ML-100k
943
1,682
100,000
0.063
1,2,
ML-IM
6,040
3,706
1,000,209
0.045
1, 1..,,..
Table 2: Training settings
METHOD
DATASET
d
A
OTHERS
ML-100K
3
0.1
boosted k-rep. with 4 learners
Alternating optimization 3.1
ML-IM
3
0
boosted k-rep. with 5 learners
RECONST-NET 3.2
1000
3
10
10
f1
10, n2 100, ng
10, drop-out= 0.1
ML-IM
4
10
10
15, n2 100, ng 15, drop-out= 0.1
as:
C= c= I Cu()))((VV) (2,))))))))))) s
| 1|
(14)
DEll
KEC
here 9 can be an arbitrary non-lines non-linear function. In figure have proposed g two additional hidden layers in right panel with tanh activation. Here, we usually choose nı = ny and equals our expectation from the number of soft clusters in the data. Still one interpret the last hidden layer as the soft clustering layer.
3.3 COMBINING MULTI PREDICTORS
Combining user-based and item-hased methods. Till now, have assumed that items mapped to a vector space, and users have preference functions (user-based method), but there is no reason not consider reversely (item-based method). A simple way of combining user-based and item- based methods is to do a linear regression from each method's output to the observed ratings. The validation part of the data will be used for estimating the coefficients of regression. We will see that combining user- and item- flem-based methods significantly improve the prediction of test data. Leveraging ensemble of predictors. A more complicated but effective way of combining is to lever- age an ensemble of combined user-based and item- based methods. Consider we have a predictor (S) at iteration that predicts observed ratings S for training the model. At each iteration, we calculate the residuals of the predicted ratings and pass it to the next predictor: S(++1) = (t()) ((((t) The next predictor, (l++1, uses S((+1) for training its model. The final pre- dictor leveraged from (() 10 1,2,... T. uses the sum of all predictions: S(S) = (S)
EXPERIMENTS
To evaluate the proposed methods, we use three datasets: a synthetic dataset besides the well known ML-100k and ML-IM datasets (Harper & Konstan 2013). The details of each dataset can be found in Table π The synthetic data is created to assess our clustering methods; for this purpose, a num- ber of cluster representatives are randomly chosen in the low-frequncy domain (Guaranteed to be smooth) and then, each user is placed randomly around one of the representatives. The distance of the users to the associated representative (within-cluster variance) varied in different tests. Look at the Appendix for detailed discussion.
The MovieLens dataset F ML-100k and ML-IM are two of the most common benchmarks for col- laborative filtering. For the 1L-100k, we follow the pre-specified train-test split; i.e., 75%, 5%, and 20% of the total available data is used for training, validation, and test, respectively. For ML-IM, these numbers are 90%, 5%, and 5%, respectively.

---

Under review as a conference paper at ICLR 2021
Table RMSE comparison
METHOD
MI-1100k
MI-IM
PMF
0.952
0.883
GMC
[Kalofolias et al.
2014)
0.996
Factorized EAT (Hartford al.
2018
0.920
0,860
IGMC (Zhang & Chen 2019)
0.905
0.857
GC-MC (Berg al. (017)
0.910
0.832
Bayesian tumeSVD (Rendle et al.
2019)
0.886
0.816
SRGCNN (Monti ct al.
2017
0.929
GRALS (Rao al. 2015)
0.945
Alternating optm. (ours)
0,920
0,860
RECONST-NET (ours)
0.909
0.862
0.95
1.4
train batch loss
val. batch
0.90
val. loss
1.2
best loss
ROASE
0.85
user tr.
RMSE
user va.
1.0
item
Rem va.
0.8
va.
0.75
tc.
0.6
20
40
25
50
100
125
150
iter.
epoch
(a) Alternating optimization in ML-1M
(b) RECONST-NET training with ML-100k
Figure 4: Training process for different methods
4.1 PREDICTION EVALUATION
Training process. The training settings of both proposed methods, alternating optimization, and RECONST-NET for each dataset are provided in Table Figure[ssows the RMSE on two different datasets during the training stage. The validation is reserved for parameter tuning, and performance on test data is reported for both methods. Figure clearly reveals the performance gain achieved by combining the user- and item-based techniques. Further, the stair shape decrease of the training loss (and validation loss) confirms the suitability of leveraging the ensemble predictors.
Performance comparison. We further conduct experiments ML- 100k and ML-1M datasets. We have ignored the available side information in both datasets. Therefore. for the performance compar- have included only methods that do not take into account these side information.
ison in Table Although neither of the alternating optimization and RECONST-NET record the best RMSE, they yield very good results despite their simplicity and interpretability.
CONCLUSION
In this article, formulated the rating prediction problem as reconstruction problem with smoothness assumption. The proposed methods are all simple and interpretable but show significant performance comparing state-of-the- -he-the-rrt methods. Specifically, the designed network and evaluated our interpretation in synthetic design. The proposed architecture and rich frequency-domain feature be basis for future research on interpretable recommender systems.
interpreted different layers of
REFERENCES
Rianne van den Berg. Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. arXiv preprint arXiv: 1777.022200 2017.
8

---

Under review as a conference paper at ICLR 2021
James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, et al. The youtube video recommendation system. In Proceedings of the fourth ACM conference Recommender systems, pp. 293-296, 2010.
Gintare Karolina Dziugaite and Daniel M Roy. Neural network matrix factorization. arXiv preprint arXiv:1511.066 2015.
Carlos A Gomez-Uribe and Neil Hunt. The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS), 6(4):
F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (ttis) 5(4):1-19, 2015.
Jason Hartford, Devon R Graham, Kevin Leyton-Brown, and Siamak Ravanbakhsh. Deep models of interactions across sets, arXiv preprint arXiv:/803.0287/ 2018.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural col- laborative filtering. In Proceedings of the 26th international conference on world wide web, pp- 173-182, 2017.
Vassilis Kalofolias, Xavier Bresson, Michael Bronstein, and Pierre Vandergheynst. Matrix comple- tion on graphs. arXiv preprint arXiv: 1717, 2014.
Andriy Mnih and Russ R Salakhutdinov. Probabilistic matrix factorization. In Advances in neural information processing systems, pp. 1257-1264, 2008.
Federico Monti, Michael Bronstein, and Xavier Bresson. Geometric matrix completion with recur- rent mulli-grrph neural networks. In Advances in Neural Information Processing Systems, pp- 3697-3707, 2017.
Nikhil Rao, Hsiang-Fu Yu, Pradeep K Ravikumar, and Inderjit S Dhillon. Collaborative filtering with graph information: Consistency and scalable methods. In Advances in neural information processing systems, pp. 2107-2115, 2015.
Steffen Rendle, Li Zhang, and Yehuda Koren, On the difficulty of evaluating baselines: A study on recommender systems. arXiv preprint arXiv:1905.01333 2019.
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner. and Lexing Xie. Autorec: Autoencoders meet collaborative filtering. In Proceedings of the 24th international conference on World Wide Web, pp. 111-112, 2015.
Sungyong Seo, Jing Huang, Hao Yang, and Yan Liu. Interpretable convolutional neural networks with dual local and global attention for review rating prediction. In Proceedings of the Eleventh ACM Conference on Recommender Systems, pp. 297-305, 2017.
Florian Strub, Romaric Gaudel, and Jérémie Mary. Hybrid recommender system based on autoen- coders. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, pp-
11-16, 2016.
Nguyen Xuan Vinh, Julien Epps, and James Bailey. Information theoretic measures for cluster- ings comparison: Variants, properties, normalization and correction for chance. The Journal of Machine Learning Research, 11:2837-2854, 2010.
Yao Wu, Christopher DuBois, Alice X Zheng, and Martin Ester. Collaborative denoising auto- encoders for top-n recommender systems. In Proceedings of the Ninth ACM International Con- ference Web Search and Data Mining, pp. 153 133-162,70 2016.
Muhan Zhang and Yixin Chen. Inductive matrix completion based on graph neural networks. arXiv preprint arXiv:1904 10.9000.2000 12058, 2019.
Shuai Zhang, Lina Yao, Aixin Sun, Sen Wang, Guodong Long, and Manqing Dong. Neurec: On nonlinear transformation for personalized ranking. arXiv preprint arXiv:1805.03002.22 2018.
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys (CSUR), 52(1):1-38, 2019.

---

Under review as a conference paper at ICLR 2021
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
ARI
0.25
K-Rep Boasted K-Rep
0.25
0.00
RECONST-NET
-0.25
-0.25
-0.25
10a discriminabiiity
10-2
density
#cluster
Figure 5: The performance of the clustering techniques on synthetic data
A APPENDIX
A.1 CLUSTERING EVALUATION
In Section 3.1.1 we proposed the k-representation clustering and its boosted version. Here, we study their performances via experiments synthetic data. We recall that the mentioned methods do not explicitly penalize miss-clustering: instead, they minimize the within-cluster reconstruction loss. As result, we expect these methods perform fairly well when the clusters are distinguishable. To measure the matching between the identified clusters and the original ones, we employ the Adjusted Rank Index (ARI) (look at Vinh al. (2010)). For two clusterings e₁ and C2 with the same domain U. if we form the contingency matrix N with the (i,j) element as Hu, (u) i, 02 (u) = j}l- then, ARI defined based on the elements, column and row sums of N, intuitively shows the rate of agreement between the two clusterings if is randomly chosen from U. It takes the maximum value for identical clusterings and the minimum value 0 when the clusterings are perceived as fully random with respect to each other. As explained, ARI has the advantage of comparing two clusterings even with different number of clusters.
To the ARI metrie, need hard clustering of the users. For this purpose, associate each user in our User-layer (soft clustering layer) in Figure to the neuron (cluster) in the last hidden unit with the largest absolute weight. Although this technique violates the main goal in soft clustering, provides us with a measure of clustering accuracy.
In Figure 5 the performance of &-representation (I). boosted k-representation (2) and modified re- sults from the soft clustering layer (Figure 3) are depicted for three scenarios. As expected, we obtain inferior result from the soft-clustering layer; however, ARI is above zero, the clustering of this layer is not irrelevant. In the left plot in Figure [ the ARI curves in terms of the discrimination within-cluster variances) are presented. We ob- serve that the boosted k-representation works better in cases with higher discrimination index, but loses its performance in case of cluttered clusters.
index of the clusters (the ratio of between-cluser
The center plot in Figure 5 shows ARI changes by varying the density in the rating matrix (ratio of the number observed to non-observed entries). While the k-representatton has the best per- formance low densities, its performance drops when the density exceeds threshold; this might be due to the involved regularization term. Finally, in the right plot of Figure 5) we have changed the number of clusters in all methods; the correct number of clusters is kept fixed at 4. As in these plots, none of the k-representation and its boosted version dominate the other one in all regimes.
10


Reviews:
[{"review_id": "xZR5CamT04h", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is not well-written, so it is hard for me to fully understand the whole paper. It seems to me that the authors tried to model user preferences with smooth functions for collaborative filtering and results on two Movielens datasets showed that the proposed method is comparable to several baseline methods. \n\nPros:\n1.\tThe idea of capturing frequency domain information for collaborative filtering seems interesting to me, although I did not quite understand how this information is obtained in this work.\n2.\tThe authors mentioned that their method can be interpretable, which may be more interesting if they can provide some case studies for explaining recommendations using the proposed method.\n\nCons:\n1.\tThe motivation is not clear to me, i.e., why smooth functions are important in modelling user preferences in collaborative filtering is not well explained. \n2.\tThe authors seem to have a narrow understanding of collaborative filtering. For instance, the authors mentioned that “the utilization of deep architectures is limited to shallow networks” in exiting CF literature. However, there are a lot of recent works building upon state-of-the-art deep learning techniques as far as I know. \n3.\tThe idea of this paper is very similar to a recent work (Harald Steck, Embarrassingly Shallow Autoencoders for Sparse Data, WWW ’19). The main difference may be that this work adopted clustering of users instead of individual user when modelling user interests.\n4.\tThe experimental results are not encouraging. The proposed method is actually much worse than Bayesian TimeSVD++ (Rendle et al., 2019) and IGMC (Zhang & Chen, 2019). So, it is hard to understand why this alternative method is promising in the area.\n5.\tThe presentation could be improved. The authors mentioned that their code has been published but no link was provided in the paper.  \n\nOverall, I think this paper is a preliminary work and is not ready for publication. I think the authors should revise the paper and submit the revised version to another conference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "R7RKCJY35Bk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper has proposed a smooth reconstruction of the preference function in collaborative filtering for recommender system. The evaluations have generally demonstrate the proposed method. However, there are several concerns left.\n\n1, The paper has proposed utilizing interpretable layers to implicitly cluster the users. However, it is not clear how the interpretable layer measure the interpretability using qualitative metric. I don't believe the the weights appeared in the X layers gains the interpretability to tell the model how the users may be clustered. In terms of the contribution of the interpretable layer, there should be ablation study to demonstrate the effectiveness of the interpretable layers(without x-layer).\n\n2, The experiments mostly look good. However, they are not superior compared to the baselines. It is not required to outperform the baseline results that listed in Table 3, but necessary analysis should be conduct why the gap exists. If it is claimed that some baselines use side information, it is vital to eliminate the side informations so that to make a fair experimental comparisons.\n\n3, The experiments could be explored to other datasets to show the generalization capability of the proposed method. Even in Movie-lens dataset, if would be easier to extend to ranking based evaluation(using NDCG, F1 etc), rather than purely rate-based evaluations since both of them are practical metric in recommender system.\n\n4, Some procedure of the proposed methods are not intuitive, e.g.In Alg 2, why take log2|C| as the total size of k-representations?\n\n5. some typos and gammer errors. e.g.  page 4, 'in the section ??'   page 5,  'which selects the updating steps adaptively and without supervision.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6k-u6mGxduV", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach based on Fourier transforms to predict ratings in collaborative filtering problems. The paper’s scope (“smooth reconstruction functions”) gets immediately narrowed down to Fourier transforms--it would be nice to provide some motivation for this choice over alternative smooth functions. The paper then clusters the users as a way to reduce the number of parameters in the model, given that the Fourier transform itself does not reduce it. As a further step, the clustering is replaced by a soft-clustering learned by a neural network. In the experiments, the RMSE of the rating prediction problem is worse than some baselines and better than others. \n\nBesides these technical steps, from a more big-picture perspective, I am not sure if the problem of rating prediction as cast in this paper, misses a key point. The key point I am concerned about is that the observed ratings are missing not at random [a]. For this reason, the collaborative-filtering literature abandoned the minimization of RMSE on the OBSERVED ratings ten years ago. Two different avenues have been pursued since then: most of the papers switched to ranking the entire catalog of items, e.g, see [b] to get started. A few papers continued with rating prediction, but stated the problem correctly by taking into account the fact that the ratings are missing not at random, eg., [c,d].\n\nIn this submission, the problem statement at the top of page 4, and Eq. 6, was not clear to me: while s was defined clearly in the Fourier transform earlier in the paper,  I did not find a definition of s_u in Eq 6 in the context of rating prediction, i.e., is this the vector of ratings of user u? Only the observed ratings? How are the unobserved/missing ratings of user u treated in the proposed approach? \n\nGiven the RMSE-values in the experiments, my best guess is that the model was trained on the observed ratings only, ignoring the key problem that the ratings are missing not at random. \n\nI feel like a rating-prediction paper that ignores the key problem of collaborative filtering, i.e., the fact that ratings are missing not at random cannot be accepted (and should actually be desk-rejected). \n\nI encourage the authors to modify the approach to account for this key problem of collaborative filtering. Alternatively, this approach may be useful for different applications, like compressive sensing problems where the observations are truly random.\n\n[a] Collaborative Prediction and Ranking with Non-Random Missing Data\nby B. Marlin and R. Zemel (RecSys 2009 Best Paper)\n\n[b] Training and testing of recommender systems on data missing not at random\nby H. Steck (KDD 2010)\n\n[c] Probabilistic Matrix Factorization with Non-random Missing Data\nby J.M. Hernández-Lobato, N. Houlsby, and Z. Ghahramani (ICML 2014)\n\n[d] Modeling User Exposure in Recommendation\nby D. Liang et al. (WebConf 2016)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3umt7OwLCy_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary：\n\nIn this paper, the authors regarded the rating prediction as a reconstruction problem with a smoothness assumption. They proposed two approaches for smoothly reconstructing the preference function of users and conducted experiments on both a synthetic dataset and two benchmark datasets (ml-100k and ml-1m). However, there are still some technical issues in this article, such as unconvincing experiments and confusing descriptions. As a result, I suggest the paper should be rejected.\n\nDetailed Comments：\n\nWith respect to the problem of rating predictions in collaborative filtering, this work proposed to estimate the predictions by assuming an underlying smooth preference function for each user, the quantization at each given vector yields the associated rating. Then, they developed two methods including k-representation motivated by k-means and reconst-net, a feed-forward neural network, to do this. The authors also conducted experiments on two benchmark datasets (ml-100k and ml-1m) to test the performance of proposed algorithms.\n\nThe key strengths:\n- The perspective of reconstructing the user preference functions is novel.\n- They proposed two effective methods to estimate the preference functions.\n- An alternating optimization method is proposed that effectively optimizes a non-convex loss function and extracts user and item representations.\n\nReason to Reject: \n\n- The contribution of “the reconstruction of user preference functions” is overclaimed. Much literature in the area of CF regards the rating prediction problem as a reconstruction problem, such as Matrix Factorization (MF) based algorithms [1,2]. The authors should include these methods and elaborate on the difference between them.\n- In the introduction, the authors pointed out that the contemporary deep network in RS is merely limited to shallow networks and they addressed this problem in this work. However, the reconst-net only involves three trainable layers. It is not clear how do you solve the “shallow” problem?\n- The experiments are not convincing. The authors claimed that “their performance on benchmark datasets is remarkable”, but in experiments, we see that the performance of the proposed model is not competitive, e.g., the performance gap on ml-1m is about 5%. In addition, the authors did not dive into these results and analyze them. They should also include some MF-based methods for rating prediction problem, such as [1,2]\n- Many details and explanations are missing and confusing. For example, the intuitions of figure 1 and figure 2 are not clear, and the authors should give more explanations with respect to these figures. In figure 4a, what does the legend represent, such as “user tr.”, “va.” and “item tr.”? \n- some typos:\n  - “in the Section ?? on both synthetic and real data”\n  - “figure 2 shows an example”，” figure” should be “Figure”                                                        \n\n[1] FISM: factored item similarity models for top-N recommender systems. KDD’13\n[2] Factorization meets the neighborhood: a multifaceted collaborative filtering model. KDD’08\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper mostly received negative scores. A few reviewers pointed out that the idea of modeling user preference in the frequency domain seems novel and interesting. However, there are a few concerns around the clarity of the paper, the motivation of the proposed approach, as well as the experimental results being unconvincing (both in terms of execution as well as exploration of the results). The authors did not provide a response. Therefore, I recommend reject. 
Decision: Reject