Neuron-Enhanced Autoencoder based Collaborative filtering: Theory and Practice
Authors: Jicong Fan
Rui Chen
Chris Ding
Venue: ICLR 2022 Submitted (2022)

Abstract: This paper presents a novel recommendation method called neuron-enhanced autoencoder based collaborative filtering (NE-AECF). The method uses an additional neural network to enhance the reconstruction capability of autoencoder. Different from the main neural network implemented in a layer-wise manner, the additional neural network is implemented in an element-wise manner. They are trained simultaneously to construct an enhanced autoencoder of which the activation function in the output layer is learned adaptively to approximate possibly complicated response functions in real data. We provide theoretical analysis for NE-AECF to investigate the generalization ability of autoencoder and deep learning in collaborative filtering. We prove that the element-wise neural network is able to reduce the upper bound of the prediction error for the unknown ratings, the data sparsity is not problematic but useful, and the prediction performance is closely related to the difference between the number of users and the number of items.
Numerical results show that our NE-AECF has promising performance on a few benchmark datasets.

Under review as a conference paper at ICLR 2022
NEURON-ENHANCHT AUTOENCODER BASED COL- LABORATIVE FILTERING: THEORY AND PRACTICE
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents a novel recommendation method called reuron-enhanced au- toencoder based collaborative filtering (NE-AECF). The method uses an addi- tional neural network to enhance the reconstruction capability of autoencoder. Different from the main neural network implemented in a layer-wise manner, the additional neural network is implemented in clement-wise manner. They are trained simultaneously to construct an enhanced autoencoder of which the acti- vation function in the output layer is learned adaptively to approximate possibly complicated response functions in real data. We provide theoretical analysis for NE-AECF investigate the generalization ability of autoencoder and deep learn- ing in collaborative filtering. We prove that the element-wise neural network is able reduce the upper bound of the prediction error for the unknown ratings, the data sparsity is not problematic but useful, and the prediction performance is closely related to the difference between the number of users and the number of items. Numerical results show that our NE-AECF has promising performance on few benchmark datasets.
INTRODUCTION
Recommendation system aims to provide personalized recommendation based various informa- tion such as user purchase records, social networks, user features, and item (or product) features. With the fast growth of E-commence, social media, and content provider, recommendation systems play more and more important roles in our daily life and have changed our life both explicitly and implicitly. general, recommendation systems can be organized into three categories (Adomavi- cius & Tuzhilin, 2005 Zhang et al., (019):: content based method, collaborative filtering, and hybrid methods. The content based methods recommend similar items to a user or recommend one item to similar users, where the similarity is usually obtained from side information such as genre, oc- cupation, and age. Collaborative filtering (CF) assumes that there exist potential correlations within both users and items, which can be implicitly used to predict unknown ratings. Hybrid methods combinations of content based methods and CF methods (Adomavicius & Tuzhilin 2005 Zhang
al. 2019 Su & Khoshgoftaar 2009). CF is at the cores of many recommendation systems.
Early CF methods (Resnick et al., 1994) compute the similarity between users or items directly from the ratings make prediction. This kind of method is also called memory based CF, which is easy to implement and has high interpretability. One limitation is that the similarity computed from the ratings not informative owing the high sparsity of the rating. Another line of CF is model based method (Ungar & Foster 1998 Shani et al.) 2002) that utilizes historical data to train machine learning model such as Bayesian network (Breese al., 1998, Míyahara al. 2000) for recommendation. Model based methods are more effective than content based methods in learning complex hidden preference and handling the sparsity problem. Note that both content based and model based methods do not work when there are items users without ratings, which is known as the cold start problem. A popular strategy for solving the problem is to incorporate side information into CF methods (Adams et al. 2010 Welling et al. 2012 2012|Zhagg et al. 2017).
In the past decades, matrix factorization (Billsus & Pazzani 1998
Mnih & Salakhutdinov 20088 Ko-
ren et al. 2209) and matrix completion (Candès & Recht 2009
Shamir & Shalev-Shwarth [204]]
Sun & Luo] 2015) Chen et al.] (0166] Fan et al. (201)) have been extensively studied and used in CF. These methods usually exploit the potential low-rank structure of the incomplete rating matrix

---

Under review as a conference paper at ICLR 2022
via embedding items and users into a latent space of reduced dimension, where the observed rat- ings are approximated by the inner products of the user feature vectors and item feature vectors. The low-rankness is usually obtained by low-rank factorization (Koren et al. 2009), nuclear norm minimization (Candès & Recht, 2009), or Schatten-p quasi norm minimization (Fan et al. 2019). Particularly, [Le al. (20T6) proposed a local low-rank matrix approximation (LLORMA) that approximates the rating matrix as a weighted sum of a few low-rank matrices. LLORMA outper- formed vanilla low-rank matrix completion methods in collaborative filtering, which indicates that the rating matrices in real applications may have more complicated structures rather than a single low-rank structure.
The success of neural networks and deep learning in computer vision and natural language process- ing inspired researchers to design neural networks for CF (Salakhutdinov et al.) 2007 Dziugaite &
Roy
2015 Sedhain et al. 2015 Wu et al. 2016)] Zheng et al., 2 He et al. 2017 van den Berg
ct al. 2017 Fan & Cheng, 2018 ei al. (020). For instance, (aalkkuutdinov et al. (2007) proposed
restricted Boltzmann machines (Hinton et al. 2006) based CF method called RBM-CF, which
showed high performance in the Netflix challenge (Bennett & Lanning 2007). [Sedhain al. (2015)
proposed AutoRec, an autoencoder (Hinton & Salakhutdinov/ [2006] [Bnnni et al.) (007) based CF method, which predicts unknown ratings by an encoder-decoder model æ = W2o(( where denotes the incomplete ratings on one item or of one user and Wi, W2 are weight matrices to
optimize. Unlike RBM-CF, which is probabilistic and generative, AutoRec provides a discrimina- tive approach. AutoRec outperformed LLORMA slightly on several benchmark datasets (Sedhain iet al. 2015). In addition, adding depth is able to improve the performance of AutoRec (Sedhain al. 2015). Inspried by Neural Autoregressive Distribution Estimator (NADE) (Larochelle &
Murray, (011) and RBM-CF (Salakhutdinov et al.) 2007), Zheng et al. (2016) proposed a method
called CF-NADE, in which parameters are shared between different ratings and achieved promising performance in several benchmarks. Muller et al. (2018) proposed a kernel based reparametrized neural network, in which the weight between two units is set to be a weighted kernel-function of the location vectors. The method works well in data visualization and recommendation systems. Interestingly. Yi et al. (2020) found that the expected value of the output layer of a neural network depends the sparsity of the input data. They proposed simple yet effective method called spar- sity normalization to improve the performance of neural networks with sparse input data such as the highly incomplete rating matrices in CF.
It is worth mentioning that existing autoencoder based CF methods such as (Sedhain et al.) 2015 Wu al. 2016 Muller et al., 2018 ei al., 2020) use linear activation function in the output of the decoder, i.e., #c Wi.h..-1, where Wy. denotes the weights of the output layer and h denotes the features given by the last hidden layer. Thus, these methods are under the assumption that the ratings are linear interactions between user features and item fcatures, though the features can be nonlinear. Such an assumption may not be true or not optimal in real problems, especially when the data are bounded (e.g. images) or are collected by sensors (e.g. medical and chemical sensors) with nonlinear response functions. We suspect that the rating values given by users on items are from some nonlinear response functions because humans have complex emotion or decision curves (LeDoux] 2000; Baker 200T). A naive method to incorporate nonlinear interaction is using nonlin- activation functions such as sigmoid function (with rescaling) in the output layer of the decoder, which however has much lower performance than using a linear activation function. That's why ex- isting autoencoder based CF methods use only linear activation function. Note that a pre-specified activation function for the output layer of the decoder may work on specific data but may be far away from the possible optimal choice. On the other hand, the theoretical analysis for autoencoder and deep learning based CF is very limited, while there have been many works on the theory of
low-rank matrix completion based CF (Srebro & Shraibman 2005 Candès & Recht 2009 Shamir & Shalev-Shwartz 2014: Fan et al. 2019).
Contribution. In this paper, we present novel neural network CF method named NE-AECF, an en- hanced autoencoder approach for recommendation system. NE-AECF is composed of two different neural networks, one is an autoencoder to reconstruct the incomplete rating matrix, while the other is an element-wist neural network to learn an activation function adaptively for the output layer of the autoencoder. We provide theoretical analysis for NE-AECF, which explains the superiority of our method. Specifically, we prove that the element-wise neural network can reduce the upper bound of the prediction error for the unknown ratings. We also prove that the data sparsity is not problematic but useful and the prediction performance is closely related to the difference between the number of
2

---

Under review as a conference paper at ICLR 2022
users and the number of items. Further, demonstrate empirically NE-AECE benchmarks: MovieLen- 100k and MovieLen- IM, achieving state-of-the- art results.
(or X), æ. and X to denote scalar, vector, and matrix respectively. We use ||>|| to denote the Euclidean norm of vector æ, use X || and X| ( to denote the Frobenius norm and spectral norm of matrix X respectively. The f21 norm of matrix is denoted by X [2,11 >; [ where z; denotes the i-th column of X. The &oo norm of matrix is denoted by || loo = max; Xiiyl- We [S| denote the cardinality of set S. The symbol 'o' denotes the Hadamard product between vectors or matrices. The symbol 'o' denotes function composition.
Notation. We
2 NEURON-ENHANCER AECF
Suppose have incomplete rating matrix X -001, ... -..) Rmxn where is the number of users and is the number of items (without loss of generality). Xij 2 denotes the rating given by user i on item j and Xi = indicates an unobserved rating. denotes the set of observed ratings. We have Xij = Xi for all (1,5) E Our goal is to predict the unobserved ratings Xij. (i.j) E [m] x S, from X.
We want learn nonlinear function f R R" such that
Σ
|| E f(
(1)
small as possible, where 8; is binary vector denoting whether the the corresponding element in xy zero (unknown) or not. The motivation predicting the missing entries of æ; using its observed entries, though the missing entries of æ; are filled by zeros before performing f. More formally, we consider the following problem
|s |SO (x-1(x))
minimize JEF
(2)
where S (s1, , ,,)) f hypothesis set of m to m functions. We have infinite choices for F. For example, F can be a set of functions in the form of neural network with some parameters W W, where W denotes a set of matrices under some constraints. In this case, problem 2 defines denoising autoencoder or stacked denoising autoencoders (Vincent et al. 2010), where the noises are introduced by filling the missing ratings with zeros.
performed on each column of X separately, and F denotes
Let be an autoencoder with linear activation function in the output layer. Then 2 becomes
minimize W.
S
X
(I 1
where W1 E dd and W2 mxd are weights matrices to learn and is nonnegative constant control the strength of weight decay. We have omitted the bia terms for simplicity. a denotes an activation function such as
ReLU ((x) max(2x00 and Sigmoid (xx) 1/(1 + cxp(-x)).
Note that (3) is exactly the basic model considered by (Sedhain al. (2015) Wu al. M Muller et al. 2018) and Yi et al. (2020). Once (3) is used, the following assumption is made implicitly.
Assumption 1. There exist two matrices A E Ru and B E gdxd such that S (X AB) | small enough.
The assumption indicates that d much smaller than min(m, n). X can be well approximated by a low-rank matrix, which however may not always hold in real applications. Consider the following data generating model
X h h(A'B')
3

---

Under review as a conference paper at ICLR 2022
where h RR + IR' is element- wise nonlinear function and A' R B' R may be generated by some nonlinear functions. If the nonlinearity of h is high, X cannot be well ap- proximated by rank-d matrix. These analysis indicates that if the clement-wise nonlinearity in generating is strong, 3 should use a large d to ensure a small enough training error.
The element-wiss wise nonlinearity widely exists in real data. For example, in imaging science, the intensity of pixels are nonlinear responses of photoelectric element to spectrum. In chemical en- gineering, many sensors have nonlinear responses. In biomedical engineering, the dose-responses are often nonlinear curves. Hence, in collaborative filtering, the ratings may be nonlinear responses some latent values, according to the studies on response curve in neuroscience and psychology (LeDoux, 2000;
Baker 2001).
Therefore, instead of (3). one may consider the following problem
minimize W:
| の [ ((2)(Woo())))) + (1
(wa]l))(
(5)
where h should be determined beforehand. A naive approach to determining / is choosing a bounded or partially bounded nonlinear function according to the range of the data. For example, the data are image pixels within [0, 1]. one may use Sigmoid function. If the data are nonnegative, one may use ReLU. However, such choices only considered the range of the data, which is just a small portion of the nonlinearity. Within the range, the true response functions are not necessarily linear (ReLU) or related to exponential (Sigmoid), and can much more complicated.
As it is difficult to choose suitable nonlinear function /t in advance, we propose to learn h from the data adaptively, i.c.,
minimize Wi.EH
S X
h( (W || (I (W2))))
where H denotes a hypothesis set of nonlinear functions from RJ R2, We have different ap- proaches to learning h. The first approach is combining various activation functions, i.e.,
ho(z)= (.,,(22)
where o.(+) are different activation functions and = (0).-.., T are parameters to estimate. However, is not clear whether (7) is able to approximate wide range of nonlinear functions. The second approach is using polynomial functions, i.e..
ha(2)
Σ 0z
It is a k-order polynomial function and can well approximate any smooth functions provided that sufficiently large. Another approach is using a neural network, i.e.,
hz(2) 2 (re
de (0
where , and O Ln are vectors, O...... 2 La-1 are matrices, and de a fixed activation function. According to the universal approximation theorems (Pinkus) 1999: Sonoda & Murata 2017] Lu
jet al. 2017 7), p able to approximate any continuous functions provided that the network is wide
enough or deep enough.
Since p more flexible than and X in function approximation, we propose to solve the fol- lowing problem
Lw
Le
minimize W.B
x - he (x)) )1 + Aw S
W
A
Σ
+ ?
(10)
where W (WI.. (W...., WL J. θ (O,...., 1ro). and
gw (X) = Win (ow wow www.WWW. (w..))....)
(II)
4

---

Under review as a conference paper at ICLR 2022
In addition, assume Wi paxd=1, E [Lw] and O do = dLw = m and Po = pLe = 1. Comparing (I with we see that we have replaced f by hw o ge with Frobenius-norm constrained weight matrices. Model (IO) is exactly enhanced autoencoder based collaborative filtering (NE-AECF) method. There are two different neural networks. The first one autoencoder defined by hgo.w. which is to learn contraction map from the incomplete rating matrix X itself its observed entries more precisely. The second neural network is performed in an element-wise manner to learn an activation function adaptively for the output layer of the autoencoder or stancked autoencoders. Figure shows an example schematic of NE-AECF, where Lw Le and Z (w
R=xm--1 [Le]. Note that
neuron-
Inpul
Hidden Layer
Output Layer
Layer
Figure 1: An example schematic of NE-ABCF, The left part demonstrates rating matrix where users and items are represented by each row and column. Every square in the left part corresponds rating. Observed ratings are colored and unobserved rating are left white. Target rating being predicted is marked with question mark.
3 GENERALIZATION ERROR BOUND OF NE-AECF
In this section, we analyze the capability of NE-AECF predicting the unknown ratings of X. Note
that [ST S o (X X) = |ST S@o (X-
X) == Ls. where X he (()W))) We have
1
Ls
Σ ((X,,, X.))
S
(iJ)SS
where ((iij, 81)) = (X., - 2, Note that instead of the square loss, we may use other functions such [Xij X.yl- In the remainder of this paper, {(Xij, Xı²) denotes general loss. Let S A [m] x [n]] S, the generalization error of NE-AECF is quantified by
L
Σ (K))eS=
((xi,, X.y),
Sc
which is measurement of the prediction eπrr of NE-AECF for the unknown ratings in X. We have the following generalization bound.
Theorem 1. Suppose which resulss in incomplete matrix X with unknown ratings replaced by some values such as zero. Suppose nın - |S| > |S] > 50. Let X he (gw((())))))))))))))))))((())( where he is defined by ( and gw is defined by (11). Suppose |
S of ratings of X E RRxx observed uniformaly and randomly,
a₁ Will2.1 d/. [Lw], d == max (di....,dp d.w and 10:l12 Σ by, |@ < b/ 1 E [Le]. Suppose the Lipschitz constants of ow and dg are p and o respectively. Suppose sup,, |f (Yi, X1y) S Tu, is np-liccciitz and | p. Then with
5

---

Under review as a conference paper at ICLR 2022
probability at least 8 over the random sampling S.
Σ
(Xer
Σ
e
Xis)
|
|S|
(s,))s:=
(i,)ES
(12)
Cinpey In |S|
2/2 In
11mme
mm
+ C2ndi4
+ +
In
[S]
[s]
V
|
5²
2/3
3/2
where
U1
plw-l 001-1111101 || [|
(I Lw ai)
Eter
[I be (t),).)) 0/2
=
Ele PiPt-1, U/ = Lapt---1 1opp---loe--lp 1 |
[IE bw ai)
(IL' by
maxq
by bi
and C2 C2 are
some absolute constants.
First, let's show that the bound is non-trivial. Since activation functions often most 1-Lipschitz, we let p 0 == 1. Suppose 01 =.... aLw = and b bL = 1. Since affat I di-1, we
have E (ata, 2/3 3/2
W max; afa;' S 2/222 d. addition, max, 1i/bb S max; vii-
Then the bound in Theorem [bccoes
ne 3/2 d)
X |1F
Ele 1=1 PiPI-1
Le
Cs SLs ò
+Õ
Tep
|S|
S
(13)
11
mn
In
VIS$ISS
|
Note that || S Vmn maxij Xiyl Is > C3 max ( d /mm maxij [Xigh Ete pipi-1)
holds for some constant C3, the bound is non-trivial, Obviously, the condition holds if n is much larger than m and [S] is sufficiently large. A smaller leads to tighter bound.
More specifically, Theorem I provides the following results.
A. The error bound of prediction for the unknown ratings can be reduced via including the additional (element-wise) neural network.
Given fixed autoencoder, denote by 2 the training error without the element-wise neural network of NE-AECE. As discussed in Section [ the additional neural network of NE-AECF aims learn an activation function adaptively for the output layer of the decoder. Hence, the training error C
can be reduced to Ls. Denote C3 Ls = As. In 13 dde-1 H by, and C2ne
22 In are
[S]
introduced by the additional neural network. Denote of by the en without g and I by Then we obtain Cg. < By while
Le b 1) Ciner [S] 1m1
U/ In Us
Ls < Bo -s + (g
+ Canep
S
SS]
Note that -- [ additional network is not too large). Therefore, A can be negative, which means the element- neural network can reduce the upper bound of the prediction error. On the other hand, if we do not the additional neural network but still want reduce B to By As, have to increase the depth or width of the main neural network, which will raise the value of v1 and hence increase the upper bound. These results verified the superiority of our NE-AECF over classical autoencoder based CF methods such as the AutoRec of (Sedhain et al. 2015).
can very close to and v2 much smaller than |S] (provided that the
B. Filling the unknown ratings with zeros reduces the upper bound of prediction error.

---

Under review as a conference paper at ICLR 2022
In Theorem if the unknown ratings replaced by zeros, | become tighter. Specifically, (13) becomes
will decrease. Hence the bound ||
mee ne d mmxxi Xxss
3/2
La Ei
L.s. SLs ò
Ô
pipt-1
VIsI
S
(14)
11gmm V
mn
+ 3Tz
|SS||s|
8
The bound is tight if VSS is much larger than d, the maximum size of the hidden layers.
C. Increasing n reduces the upper bound of prediction error.
Let the sampling rate |S| C and network structures be fixed. Then 14 becomes
3/2
maxij [86s]
Les 1=1 pipi-1
Ls Shs +
+
VCmn/dd
Gmn
(15)
11
mm + 37e
(1 --)mmm In
Therefore, when increases, the bound becomes tighter. In addition. Ls= S Ls when n - co. In real application, if there are more users than items, we need construct a neural network such that the input is vector of each user's rating, where items correspond to features and users correspond samples. In other word, larger difference between the number of users and the number of items leads to tighter upper bound of the prediction error, because we can construct the autoencoder along the smaller size of the rating matrix.
CONNECTION WITH PREVIOUS WORK
The element- weieent-wise neural network of NE-AECF can be regarded activation function adaptively learned from the data. is closely related to the previous work on adaptive activation functions such (Lin et 2013 Agostinelli al. 2014 Hou al., 2017 Goyal et al., 2019). For instance, Lin et al. (0133) proposed to use micro neural networks to improve the convolution operator in convolutional neural networks. Hou al. (2017) showed that applying adaptive activation functions in the regression (second-to-lase layer of a neural network can significantly decrease the bias of the regression. Their adaptive activation function is in the form of piece-wise polynomials. We found that, empirically. in NE-AECF, the improvement given by polynomials (8) and piece-wise polynomials (Hou et al. 2017) are not significant, which may be caused by the unboundedness of polynomials.
As mentioned in Introduction, theoretical study for autoencoder and deep learning based collabo- rative filtering is very limited. In recent years, few researchers have studied the generalization ability or sample complexity of deep neural networks (Bartlett et al. 2017 Neyshabur et al. 2018 Golowich et al. 2018) but their results do not apply to autoencoder based CF and our AE-NECE. Our proof for Theorem I has taken advantage of the result of (Bartlett et al.) 2017). There are two major differences. First, setting is collaborative filtering, in which the traiinn samples trix elements. Hence we utilized the transductive Rademacher complexity proposed by (EI- Yaniv & Pechyony 2009).. Second, computing the complexity of the element-wise neural network required different method rather than that of (Bartlett et al.) 2017).
Shamir & Shalev-Shwartz (2014) provided the following generalization bound for nuclear norm [S]
minimization based CF: Ls- S Ls + Tr ||l.\\/ (m) R. where X denotes the recovered
matrix given by nuclear norm minimization and R stands for the remainder of their result (Theorem
6). Suppose the rank of X is d. Then the term related to the nuclear norm can be as large as
ni V |n(vml v or O 11p 2-m
where we have assumed m < n. According to 15
Crnn
the dominating term in our bound can be Ô (nery neevy
Cd
Note that with the same d, the
7m

---

Under review as a conference paper at ICLR 2022
training of our NE-ABCF is less than the training error of nuclear norm minimization because we are using neural networks. Now we conclude that when n is sufficiently large (compared to Ça), our bound is tighter than that of (Shamir & Shalev-Shwart, 2014
NUMERICAL RESULTS
5.1 EXPERIMENTS ON MOVIELENS DATASETS
In this section, we evaluate the proposed method NE-AECF on two benchmark datasets of collab- orative filtering: Movielens 100K and Movielens 1M (Harper & Konstan 2013). These datasets contains 100 thousand (1 million) real world ratings for 1682 (3900) movies by 943 (6040) users, on a 5-stars scale. We randomly sample 90% of the known ratings as training set, leaving the remaining 10% the test set. Among the training set, 5% are held out for hyperparameter tuning. Following Lee et al. (2016), test items or users without training observations are assigned with default rating of 3. The model performance is evaluated by the root mean squared error defined as
E(id)ess (Xij x,))²
RMSE
(16)
[|S|]
where S denotes the set of test ratings. For all the experiments, we use Adam Kingmaa & Ba 2015) to minimize the objective function of NE-AECF.
We report the mean RMSE based on 50 random splits and compare our NE-AECF with the following
methods: BisaMF (Koren et al., 2009), NNMF (Dziugaite & Roy 2015), AutoRec (Sedhain et al,
2015), CF-NADE (heng et al.
2016). LLOMRA (Lee al. 2016), GC-MC (van den Berg al.
2017 Sparse FC (Muller et al. 2018), DMF+ (Yi al. 2019). and AutoRee W/SN (Yi ct al. 2020
In NE-AECF, the main neural network has one hidden layer. The numbers of hidden units 700 and 2000 for MovieLens- 100k and MovieLens- 1M respectively. For both datasets, the element- wise neural network has one hidden layer, of which the size is 200. The regularization parameters Aw and dy were chosen from (0.01, 1, 10, 50, 100, 200, 5003.
As showr Table methods, which indicates that a more sophisticated model (weighted sum of local low-rank mod- els) fits the data better than a single low-rank model. As expected, our NE-AECF outperformed all baseline methods. Similarly, in Table the RMSE of NE-AECF is less than those of other methods. worth mentioning that we have tried to increase the depth of the main neural network and the element- wise neural network, but the improvements in terms of RMSE are not significant.
Moviel 100k, ILORMA outperformed all low-rank factorization
Table 1: RMSE result of NE-AECF and compared methods on MovieLens-1000 dataset. Model
ML-100k
BiasMF Koren et al.
2009
0.911
GC-MC (van den Berg et al.
2017
0.905
AutoSV D++ Zhang al.
0,904
AutoSVD (Zhagg et al. 2017
0,901
NNMF DDiugatte Roy 2015
0,903
DMF+ Yi et al. 2019)
0,8889
LLORMA (Lee et 2016)
0,8881
AutoRec w/SN (Yi et al.
2020
0.8816 18816
NE-AECF (ours)
0.8791
+ 0.0063
5.2 EXPERIMENTS ON DOUBAN AND FLIXSTER
In this section, we evaluate the proposed NE-AECF on two more datasets Douban and Flixter, which are poplular benchmarks for recommendation systems. For these data, we use the preprocessed sub- sets and splits provided by Monti et al. (011)). These datasets both contain 3000 users and 3000
'The RMSEs of the compared methods shown in all tables are from the original papers, in which the experimental settings the same comparable.
8

---

Under review as a conference paper at ICLR 2022
Table 2: RMSE result of NE-AECF and compared methods Moviel 1M dataset
Model
ML-1M
AutoSVD (Zhang et al. 2017
0,864
AutoSVD+ Zhang al. 2017
0,848
BiasMF Koren et al. 2009
0,845
NNMF (Zzugatte
2015
2015
0,843
LLORMA (Lee et al. 2016)
0,833
DMF+ Yi et al. 2019)
0.8321
GC-MC (aan den Berg et al.
2017
0,832
AutoRec (Sedhain al.
2015
0,831
CF-NADE (Zheng et al.
2016
0.829
AutoRec w/SN (Yi et al.
2020
0,8260
NE-AECF (ours)
0.8252 + 0.0024
items. Douban contains 136,891 ratings with density 0.0152 on rating scale {1, 2, 4, Flixster contains 26, 13 ratings with density 0.0029 on a rating scale (0.5, 1, 1.5, 1.,,....5, 5]. which is a bit dif- ferent from the other three datasets. Among the training samples, 5% used for hyperparameter tuning.
We report the mean RMSE 5 repeated experiments and compare our NE-ABCF with the following methods: PMF (Miih & Salakhutdinoy 22008), GRALS (Rao et al. 2015), sRGCNN (Monti et al. 2017). GC-MC van den Berg et al. 2017). Factorized EAE (Hartford al. 2018 and GRAEM (trrhl et al. 2020). In our NE-AECF, the main neural network has one hidden layer, of which the
size 500, for both datasets. The structure of the element-wise neural network the same as that
used for the MovieLens datasets.
As shown in Table our proposed NE-AECF outperforms other baseline method on both Douban and Flixster. Note that some of the compared methods include extra content like the side information of users and items into the model, while NE-AECT does not require extra content.
Table 3: RMSE result of NE-AECF and compared methods on Douban and Flixster dataset
Model
Douban
Flixster
GRALS (Rao al. 2015)
0.8326
1.245
PMF (Miih & Salakhutding
2008
0.7492
0,9809
sRGCNN (onti al. 2017)
0.801
0.926
Factorized EAE (Hartford et al. 2018)
0.738
0.908
GC-MC (van den Berg al.
2017
0.734
0.917
GRAEM (trahl et al. 2020
0.7497
0,8857
NE-AECF (ours)
0.7286
0.8816
6 CONCLUSION
This paper presented a novel collaborative filtering method called NE-AECE We analyzed the gen- eralization ability for NE-AECF and showed that the element-wise neural network useful re- ducing the upper bound of the prediction error in CF. The theoretical analysis indicates that filling the unknown ratings with zeros can make the error bound tighter. It also provides a guideline to make choice between fem-bbaed autoencoder and bser-based autoencoder. These theoretical find- ings were further validated by the numerical results, in which our NE-AECF has ofate-of-the-an performance in terms of RMSE.
is possible to incorporate side information or graph neural network into our NE-AECF, which can be a future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ryan P. Adams, George E. Dahl, and lain Murray, Incorporating side information into probabilistic matrix factorization using Gaussian processes. In Proceedings of the 26th Conference on Uncer- tainty in Artificial Intelligence, pp. 1-9, 2010.
G. Adomavicius and A. Tuzhilin. Toward the next generation of recommender systems: a survey of the state-of-the-ard and possible extensions. IEEE Transactions Knowledge and Data Engi- neering, 17(6):734-745) 749, 2005.
Forest Agostinelli, Matthew Hoffman, Peter Sadowski, and Pierre Baldi. Learning activation func- tions to improve deep neural networks. arXiv preprint arXiv: (rXiv:1412.6830, 2014.
Frank B Baker. The basies of item response theory. ERIC, 2001.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Opectrally-normaliie margin bounds for neural networks. I. Guyon, U. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pp. 153-160, 2007.
Bennett and S. Lanning. The netflix prize. In Proceedings of the KDD Cup Workshop 2007, pp- 3-6, New York, August 2007. ACM.
Daniel Billsus and Michael Pazzani. Learning collaborative information filters. pp. 46-54, 1998. URL
acm. wttggciiattii on. wwwoonnccfffdffmmm
665311
John Breese, David Heckerman, and Carl Kadie. Empirical analysis of predictive algorithms for collaborative filtering. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelli- gence, pp. 43-52. Morgan Kaufmann, 1998.
Emmanuel J Candès and Benjamin Recht. Exact matrix completion via convex optimization. Foun- dations of computational mathematics, (6):7177777, 2009. ISSN 1615-3375.
Chao Chen, Dongsheng Li, Qin Lv, Junchi Yan, Stephen M. Chu, and Li Shang. Mpma: Mixture probabilistic matrix approximation for collaborative filtering. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI' 16, pp. 1382-1388. AAAI Press, 2016. ISBN 9781577357704.
Gintare Karolina Dziugaite and Daniel M. Roy. Neural network matrix factorization. CoRR, 0ts/1511.06443. 2015.
Ran El- Yaniv and Dmitry Pechyony. Transductive rademacher complexity and its applications. Journal of Artificial Intelligence Research, 35:193-234 2009.
Jicong Fan and Jieyu Cheng. Matrix completion by deep matrix factorization. Neural Networks, 98: 34-11 2018.
Jicong Fan, Lijun Ding, Yudong Chen, and Madeleine Udell. Factor group-sparse regularization for efficient low-rank matrix recovery. In Advances in Neural Information Processing Systems, pp- 5104-5114,, 2019.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pp. 297-299. PMLR, 2018.
Mohit Goyal, Rajan Goyal, Brejesh Lall. Learning activation functions: A new paradigm for understanding neural networks. arXiv preprint a7X1v:190.009529, 2019.
Maxwell Harper and Joseph A Konstan. The MovieLens datasets: History and context. ACM Transactions Interactive Intelligent Systems, 5(4):1-19, December 2015.
10

---

Under review as a conference paper at ICLR 2022
Jason S. Hartford, Devon R. Graham, Kevin SreytonnBrown,, and Siamak Ravanbakhsh. Deep Models of Interactions Across Sets. In Proceedings of the 35th International Conference on Machine Learning. pp. 1914-1923. PMLR, 2018.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural col- laborative filtering. In Proceedings of the 26th international conference world wide web, pp- 173-182, 2017.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-5005 2006.
Geoffrey E Hinton, Simon Osindero, and fee-Whye Teh. A fast learning algorithm for deep belief Neural computation, 18(7):1527-1553 2006.
Le Hou, Dimitris Samaras, Tahsin Kurc, Yi Gao, and Joel Saltz. Convnets with smooth adaptive activation functions for regression. In Artificial Intelligence and Statistics, pp. 430-439. PMLR, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations, 2015.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 3208):30-37, 2009, doi: 10. 20.1009//.20000..6....
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learn- ing Research, pp. 29 37, Fort Lauderdale, FL, USA, 11-13 Apr 2011.PMLR.
Joseph LeDoux. Emotion circuits in the brain. Annual Review of Neuroscience, 23(1):155-184, 2000.
Joonseok Lee, Seungyeon Kim, Guy Lebanon, Yoram Singer, and Samy Bengio. Llorma: Local low-rank matrix approximation. Journal of Machine Learning Research, 17(15):1-24, 2016.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: view from the width. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pp. 6232-6240. 2017.
Koji Miyahara, Michael J. Pazzani, and John Slaney. Collaborative filtering with the simple bayesian classifier. In PRICAI 2000 Topics in Artificial Intelligence, pp. 679-689, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg. ISBN 978-3-540-44533-3
Andriy Mnih and Russ Salakhutdinov. Probabilistic matrix factorization. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20, 2008.
Federico Monti, Michael Bronstein, and Xavier Bresson. Geometric matrix completion with current multi-graph neural networks. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Lorenz Muller, Julien Martel, and Giacomo Indiveri. Kernelized synaptic weight matrices. In Proceedings of the 35th International Conference on Machine Learning, pp. 3654-3663, 2018.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally normalizz margin bounds for neural networks. In International Conference on Leam- ing Representations, 2018.
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143- 195, 1999.
11

---

Under review as a conference paper at ICLR 2022
Nikhil Rao, Hsiang-Fu Yu, Pradeep K Ravikumar, and Inderjit S Dhillon. Collaborative filtering with graph information: Consisteney and scalable methods. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 28. Curran Associates, Inc., 2015.
Paul Resnick, Neophytos lacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. Grouplens: an open architecture for collaborative filtering of netnews. In CSCW '94: Proceedings of the 1994 ACM conference on Computer supported cooperative work, pp. 175-186, New York, NY, USA, 1994. ACM Press.
Ruslan Salakhutdinov. Andriy Mnih, and Geoffrey Hinton. Restricted boltzmann machines for collaborative filtering. In ICML '07: Proceedings ofe 24th international conference on Machine learning, pp. 791-798, 2007.
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. Autorec: Autoencoders meet collaborative filtering. In Proceedings of the 24th International Conference on World Wide Web Companion, pp. 111-112, 2015.
Ohad Shamir and Shai Shalev- Shwartz. Matrix completion with the trace norm: Learning, bounding. and transducing. Journal of Machine Learning Research, 15098)344,,3442,, 2014.
Guy Shani, Ronen Brafman, and David Heckerman. An MDP-based Recommender System. In Proceedings of the Eighteenth Conference Annual Conference on Uncertainty in Artificial Intelli- gence, UAI '02, pp. 453-460, San Francisco, CA. 2002. Morgan Kaufmann.
Sho Sonoda and Noboru Murata. Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis, 43(2):233-263 2017.
Nathan Srebro and Adi Shraibman. Rank, trace-norm and norm. International Conference on Computational Learning Theory, pp. 545-560. Springer, 2005.
Jonathan Strahl, Jaakko Peltonen, Hiroshi Mamitsuka, and Samuel Kaski. Scalable probabilistic matrix factorization with traphbbased priors. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI, pp. 5851-5858. AAAI Press, 2020.
Xiaoyuan Su and Taghi M. Khoshgoftaar. survey of collaborative filtering techniques. Adv. in Artif. Intell., 2009:4:2-4:2, January 2009.
Ruoyu Sun and Zui-Quan Luo. Guaranteed matrix completion via nonconvex factorization. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, pp. 270-289, 2015.
Lyle H. Ungar and Dean P. Foster. Clustering methods for collaborative filtering. In Workshop Recommender Systems at the 15th National Conference on Artificial Intelligence (AAAI'98), pp. 112-125, Madison, Wisconsin, USA, July 1998. AAAI Press.
Rianne van den Berg, Thomas N. Kipf, and Max Welling. Graph convolutional matrix completion. 2017.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research, 11((2), 2010.
Max Welling, Ian Porteous, and Kenichi Kurihara. Exchangeable inconsistent priors for bayesian posterior inference. In 2012 Information Theory and Applications Workshop, ITA 2012, San Diego, CA, USA, February 5-10, 2012, pp. 407-414. IEEE, 2012.
Yao Wu, Christopher DuBois, Alice X. Zheng, and Martin Ester. Collaborative denoising auto- encoders for top-n recommender systems. In Proceedings of the Ninth ACM International Con- ference on Web Search and Data Mining, pp. 153-162, 2016.
Baolin Yi, Xiaoxuan Shen, Hai Liu, Zhaoli Zhang, Wei Zhang, Sannyuya Liu, and Naixue Xiong. Deep matrix factorization with implicit feedback embedding for recommendation system. JEEE Trans. Ind. Informatics, 5(8):4591-460) 2019.
12

---

Under review as a conference paper at ICLR 2022
Joonyoung Yi, Juhyuk Lee, Kwang Joon Kim, Sung Ju Hwang, and Eunho Yang. Why not to use zero imputation? correcting sparsity bias training neural networks. In 8th International Conference on Learning Representations, 2020.
Shuai Zhang, Lina Yao, and Xiwei Xu. Autosvd++: An efficient hybrid collaborative filtering model via contractive auto-encoders. In SIGIR, pp. 957-960. ACM, 2017.
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system: A survey and new perspectives. ACM Comput. Surv., 52(1):1-33)) 2019.
Yin Zheng, Bangsheng Tang, Wenkui Ding, and Hanning Zhou. neural autoregressive approach to collaborative filtering. In Proceedings of the 33nd International Conference on Machine Learn ing, volume 48, pp. 764-773, 2016.
A PROOF FOR THE MAIN THEOREM
First of all, give the following lemmas.
Lemma 1. Let H { E mx his = θ Lo (r. 1.... 21;) .... )). V(i.j) E [m]] x [n]; O Xpp ||1||2 S h, I1ill S bg vI E [Te]; Z E Z, | I 8z). where the Lipschitz constant of is @ Suppose the covering number of 2 with respect to || DF is upper-
2.,)
bounded by Ki and = e (2g I1 b1)
b-)
Then the cover covering number of with respect
|| || is bounded
Le
3 L0 -1 (Le + 1)5x [h pipt-1
NHH (1 S Ki
1=1
Proof. See Section B.1
Lemma D provides upper bound of the covering number of the element- wise neural network. The following lemma shows an upper bound of the covering number of the main neural network.
Lemma 2 (Theorem 3.3 of Bartlett
(017)). reformulated). Let Z E [[w]. and max(m, di...., dLw) ) < D. Denote the Lipschitz constant of by Suppose the reference matrices (M1,... ..., MLW) are given. Define
al.
WIW (iw-r (... (...o (W,
where W.
E Rdd-dd,
C= (Fw((()) W- (W1.... WLw), I at || Millo S d/. [Lw].. The for any e > 0,
IC, (1+) S
X 2
((L
I of
Lw
Σ
Lw
a
2/3
3
22
Now we can get an upper bound for the covering number of the entire neural network in NE-AECF.
Lemma 3. The covering number Huve (He (Fw()))) with respect to
|| satisfies
2he 1) 2Lo
X
2D
Law II
La
2/3
(H, <
II
67
2
a1
bi maxi
6Leghe-10 -1pw p-w--11 || | (II
Lw at)
(II by
La
bt
PiP-1
ln
Proof. See Section B.2
13

---

Under review as a conference paper at ICLR 2022
Now calculate the upper bound of the Rademacher complexity of Hw,e via using Lemma and Dudley entropy integral bound.
2/3
Lemma 4. Let = 4p2:tu-111@2(0
2 12 1 In 2n
(M Lw
at)
Lw -=
(I
Le
bi
a1
02 Ei= PIPI-1, and 03 GLep 1ioopt--1gte-1- X F (Ili
Lw ai
Le by)
maxi b1bT Sup-
pose [gw (X)) S p. The Rademacher complexity of Hw.e bounded as
4p + 12 U1 + 12 In
12 02 In μ uu--us
Rs(Hw.) &
+
(17)
s
5
VS
Proof. See Section
B.3
The following lemma provides a sample complexity bound for transductive learning, which is con- sistent with the objective funetion and evaluation metric (RMSE) widely used in collaborative filter- ing.
Lemma (Corollary 1 of (El- Yaniv Pechyony 2009), reformulated), Let H fixed hypothesis set and suppose supi.j X [e (Ytj, S Tt. Suppose a fixed set S of distinct indices is uniformly and randomly split to two subsets Suain and Swes. where") Swal
|Suail
50. Then with probability
at least 5 over the random split, we have
Σ
e (Vij, Xzy) 5 Stain
Σ
eYyj, Xú²) + 4Rs(Eo 7
SSeet
(ee
(K
(18)
11 (|Strain)
+ Steal)
(|Stainl + Steal 1)
+ 3Te/
ln
v SSaiin
Swstl
Strain Steal
8
Then T'eeorm 1 can be proved as follows.
Proof. Accoding the Rademacher contraction property, we have Rs(L H) S RsH).. where denotes the lipschitz constant of f. Using Lemma 5] with a slightly different notation and Lemma where 1L002 01 provided that the element-wise neural network small enough, we have
e (Nux X1s)
Σ
(Xes, X13)
S
(i.))s
(J)ES
16p
48 / U1 + u In
48p U2 In / 10
1/e
[S|
[
VIS1
11 (1S)
(S
(IS) + 15a))
1
+ 3Te
VISTSS
S S4
8
(19)
Σ
Xiis X.3)
S
(g)ES
Ciner
|n|
U/2 In vg
+ C2nel1
|S|
|S]
117ymn V|S||Sa
mn
+ 3TT
S
8'
where C1 and C2 are some fixed constants, and
Lw II i=1
Lw
2/3
3/2
Le
of = (bw 1 X
aj
Σ
0
bi
a1
1=1
We these assumptions simplify the theorem.
14

---

Under review as a conference paper at ICLR 2022
and J/s = Leju 1oppw-lloo-10 respectively, finish the proof.
(I a) (II/2 b1). Rename of and o'g as 01 and D3
B PROOF FOR LEMMAS
B. PROOF FOR LEMMA
Proof. Let Se, == { E RP44XpL |0i||2 S b, |Orl| S kj. E [Lo].
It is known that there
exists an net Sg, obeying
36/
WSSe. || (1, S
such that |00 - @illr I 44 We have
hi
hig h:j hiille
= 11010 0 Le 1 (---001 ... O
Le (1
(1 ...)) OL (0 (0)
O (0).-1(-.. (4)) O ( (
La (000-11-000000
θ (0.o-1(... (14)) ...))
...)))))))))
< O (1 (1))))
θ Le (0
1 (0 11 (0 + Le 10 (()))))))))
L ((..-1 (0 ... σL
+ o 10
( 4nj) ....)) 1 o (1)
Le-1 irall
|
Le
+ 6
/e |
+ e
|
O
O
0 11122 +..
- --1 |aizl
||0] 6
2
Le
+ 0 Le
||æ,| - Ziyllr l1
Le
Le
< -1
S2y CLo
II
by + S219 CLo-
b
S2g
b +
+ 1 - E.g
by
ILLe
L
1
(20)
15

---

Under review as a conference paper at ICLR 2022
In (a), used the facts |XY|] follows that
$
|X||a|| and (oo(X) s xelx-Y Y|| recursively.
|
H|FF
his
2
(a)
Le
Le
S 0
Σ 2
LLe II
bi + CLa-1
II 1hhe-1
by +
61
by
+
||
- F
II
V
ij
1
Lig
Li
2
=V2e 2--1
ELe II bi + ELe-1
II 1#b8-1
bi +
bt
1 7 +
II by
1 21?
IfL0
1=
Le
Le
2
SV 28
ELe
II LFL0
bi + ELe-1
II 1#
bi +
bi
82 +
II
bt
1+1
(21)
c/ c((vLe)
In (a), we used the fact (x + 8)² & 2(x² + 1²). Let cl =
1I E [Le]. Let
-2gho-11 1 II dk
c/v2
We arrive at
V2gLe-1
1=1 by
La
-H-H S.
(22)
It means that H is an c-cover of H. Then the covering number of H. is bounded as
NHH, |1 ()),))
<N(2,1 (11,,)))
N (S ll ((r,,))
Le
60-0-1 L IIaz bb:) PiP-1
<e
II
(23)
1=1
Le
60-0-1 L
11ke he by
PIPI-1
Kig
l=l
ILe
Ce
PIPI-1
Kig
where Ce = -60 Los
Ihe by and max b/b,1 This finished the proof.
B.2 PROOF FOR LEMMA
Proof. is easy show that
82 1Z
WEw (ow ow (W, (W) (W.))..)))
(24)
p-w X
|
(25)
[=1
16

---

Under review as a conference paper at ICLR 2022
Combining Lemma
1 and Lemma
we have
In. |l (F,,))
Le
Ce
<Inkg +
Σ
PIPL-1
In
X1 In 2D
Lw
Lw
2/3
Lg
II af
Σ
a
II
b
(26)
2²
at
6Lag pLww-- ||
(IL ai)
(II by
Le
maxi
by
PiPI-1
In
B.3 PROOF FOR LEMMA
Before proof, we give the following lemma, which is a variant of the Dudley entropy integral bound on Rademacher complexity.
Lemma 6 (Lemma A.5 of Bartlett et al. 2017)). Let F be a real-valued function class taking values [0, 1] and assume that F. Then
12 VS
4a
R(Fis) < inf
V
(ns((Fis (le.,)de)
(27)
a.0
VS
S Sa
Proof. For convenience, let
Lw
Lw
2/3
Le
21 (p2(Lw-1) 2(L6 || 7 In 2D
II at
Σ
0
II b7
ai
82 Eha PIPI-1, and 83 6Lop-u-1 bo
Te
|
Fww -
ai)
(II 1. bi) max b1b, Then
In, || || マ U2 In
13
from (66). Let p maxHeh
H |ox and H (H. pll
H3. follows that (H, ||
|| e) <
01
U2 In
't3
e
According to Lemma
we have
4a
12
V
" U1
4 l 03
(P) S inf
+ U/2n
In
de
a>0
VS
S
a
2
c
4a
12
U1
+ U2
inf
+ 02 In / -u de
020
V5
S
2
(8)
4a
12
|
20 +
U2
S inf
+ v 02 In 213
de
020
VS
5
(28)
4a 12
14 2 U2 In VS
(V5 -a)) )
inf
+ v U2 1n
a>0
VS
S
(0) 4
12 / 201 U2
12(S - 1) U2 In 1 23
In
5
SVS
4 +
12 1 201 + U/ In
12 Un In p '23
5
S
VS
17

---

Under review as a conference paper at ICLR 2022
In (a), we used the fact In S J. The inequality (b) holds according to vz + vz + v8. In
(c), we have let a Vs. which though may not be the best choice. We arrive
4p + 12 + u2 In + 12 U2 In 1 U/ Rs(Hw. - g
5
VS
18


Reviews:
[{"review_id": "uH3qzkmICG", "reviewer": "Reviewer_rgcR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The proposed method is easy to implement.\n2. A detailed theoretical analysis is provided to prove the effectiveness of the element-wise neural network.\n3. This paper draws some interesting conclusions like, the data sparsity is useful.\n\nWeaknesses\n1. The assumptions in theorem 1 may be rough and untenable. For example, the authors suppose that the ratings are observed uniformly and randomly. However, the missing ratings in recommender systems are usually not missing at random. Multiple types of biases have impact on the observable ratings. I am not sure if the conclusions of this paper hold when the ratings are not uniformly observed.\n2. The authors investigate the rating prediction problem in recommender systems. However, most real-world recommender systems pay more attention to the item recommendation problem concerning the ranks of items and using different losses such as cross-entropy. \n3. The experiments are insufficient.  Although they test the proposed method on four datasets, they only conduct experiments for performance comparison. In two subsections, they evaluate the method on movielens-10k (1M), and Douban and Flixster, respectively. I think there is no need to separate the performance comparison into two subsections while providing nothing new. The additional two datasets are also small, and the experimental results are therefore less convincing. Since the proposed method has a very simple structure, it is meaningful to investigate if the proposed method shows the same advantages when using different nonlinear activation functions, neuron sizes and layers. Besides, the experimental results are not significant. \n4. The writing is not concise. I think there is no need to provide too much basic knowledge in section 2. Instead, more experiments are needed.\n", "summary_of_the_paper": "This paper proposes to use an additional element-wise neural network to enhance the reconstruction ability of the autoencoder for recommendation. A detailed theoretical analysis is provided that proves the element-wise neural network is able to reduce the upper bound of the prediction error for the unknown ratings.", "summary_of_the_review": "This paper proposes to use an additional element-wise neural network to enhance the autoencoder for recommendation, and provides detailed theoretical analysis. However, the authors do not comprehensively evaluate their model and only very few experiments are conducted. The fundamental assumption of the proposed method is problematic and the experimental results only show slight improvements compared with existing method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZP-FNFEhPWv", "reviewer": "Reviewer_WENm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1 The idea of introducing an elementwise function (approximated by a neural network) is new.\n\n2 The results show that the newly introduced neural network is helpful in improving the prediction accuracy.\n\nWeaknesses:\n\n1 Collaborative rating prediction is a very well-studied problem, for which there are lots of existing works. Moreover, in most real recommender systems, item ranking is more consistent with a real setting.\n\n2 The time complexity seems rather high. First, the authors use an item-oriented autoencoder, in which there may be lots of users associated with a typical item. Second, the elementwise function is expensive. Third, the number of hidden units is much larger than a typical matrix factorization-based method.\n\n3 The authors do not provide sufficient details or justification on using a large number of hidden units and an additional elementwise function. Moreover, treating unobserved ratings as zeros may introduce bias, which is also not justified.\n", "summary_of_the_paper": "In this paper, the authors study a traditional collaborative filtering problem with users' ratings, where the goal is to predict ratings that are unobserved. Specifically, the authors propose an enhanced autoencoder-based method called NE-AECF. The main idea of NE-AECF as shown in Eq(10) and Figure 1 is that it contains an additional module for each predicted rating by an autoencoder, i.e., h_\\theta(g_w) in Eq(10) and the most right part near the output layer in Figure 1.", "summary_of_the_review": "In this paper, the authors study a traditional collaborative filtering problem with users' ratings, where the goal is to predict ratings that are unobserved. Specifically, the authors propose an enhanced autoencoder-based method called NE-AECF. The main idea of NE-AECF as shown in Eq(10) and Figure 1 is that it contains an additional module for each predicted rating by an autoencoder, i.e., h_\\theta(g_w) in Eq(10) and the most right part near the output layer in Figure 1.\n\nThe authors conduct experiments on some public datasets and show that the newly introduced neural network is helpful in improving the prediction accuracy.\n\nSome comments:\n\n1 Collaborative rating prediction is a very well-studied problem, for which there are lots of existing works. Moreover, in real recommender systems, item ranking is more consistent with a real setting.\n\n2 The time complexity seems rather high. First, the authors use an item-oriented autoencoder, in which there may be lots of users associated with a typical item. Second, the elementwise function is expensive. Third, the number of hidden units is much larger than a typical matrix factorization-based method.\n\n3 The authors do not provide sufficient details or justification on using a large number of hidden units and an additional elementwise function. Moreover, treating unobserved ratings as zeros may introduce bias, which is also not justified.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K-1pdNUDppV", "reviewer": "Reviewer_7kpq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major concerns: \n\n1. The theoretical contribution is not easy and intuitive to follow. Moreover, I am not sure how significant or generalizable the theoretical contribution is. \n\n2. The empirical results are not solid enough. For example, in Table 1, the proposed method seems to have achieved the best performance, while CF-UIcA w/ SN yielded lower RMSE according to the SN paper. Also, the experiment of ML 10M is missing. \n\n3. SN is simple and boosts most algorithms; it will be nice to present the proposed idea similarly, i.e., boosting most existing algorithms, such as CF-UIcA. There is no obvious difficulty in not doing that. Please correct me if I am wrong. \n\n4. Could we add one more non-linear layer between hidden and output? \n\nI will increase the score if there are more promising empirical results. \n", "summary_of_the_paper": "This paper proposes an imputation method that consists of two networks. One network is AE, and one network is element-wise non-linear transformation. The authors proved the proposed method has a better generalization bound for unknown ratings. Furthermore, the authors demonstrated the proposed method achieved promising results in a few datasets. \n", "summary_of_the_review": "This work proposes imputing the missingness through two NNs, addressing the non-linear transformation between hidden and output. The authors provided theoretical justification for why the proposed method is better, but the empirical experiments can be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F3lPlTIMHri", "reviewer": "Reviewer_83pQ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Evaluation based on RMSE on the OBSERVED ratings in experiments on recommender systems triggers an immediate reject of the paper in my view. \n\nThe reason is that this implicitly assumes that the data are missing at random, while in fact they are missing not at random (MNAR), which is one of the core properties of the recommendation problem (and  which makes it different from matrix completion and compressive sensing). This fact was realized in the literature a decade ago (Merlin et al, 2009, RecSys conference best paper), which has led the community away from using RMSE on the OBSERVED ratings, and towards using metrics that take into account ALL user-item-interactions, whether observed or not. Taking into account ALL user-item interactions can most easily be accomplished with ranking mertics, which have since been used in the literature. But it can also be done in the context of rating prediction, which however becomes a much more complicated problem in this case. Some references on the data being MNAR and ranking metrics:\n- B. Marlin and R. Zemel: Collaborative Prediction and Ranking with Non-Random Missing Data, Recsys 2009\n- H. Steck: Training and testing of recommender systems on data missing not at random, KDD 2010\n\nAnd some references on rating prediction done right, i.e., accounting for the data being MNAR:\n- J.M. Hernández-Lobato, N. Houlsby, and Z. Ghahramani. Probabilistic matrix factorization with non-random missing data. ICML 2014.\n- D. Liang, L. Charlin, J. McInerney, D. M. Blei. Modeling User Exposure in Recommendation. WWW 2016.\n\nApart from that, two  rather small data sets are used in the experiments (Movielens 100k and 1 million), which raises the question of scalability of the proposed approach.\n", "summary_of_the_paper": "This paper considers an autoencoder with one hidden layer. While the output layer uses a linear activation function in several papers in the literature, this paper proposes to use a nonlinear activation function in the output layer. Moreover, instead of using a fixed function (e.g., sigmoid), this non-linear activation function is learned in this paper, using a parametrized form. Generalization bounds are  derived, and experiments are conducted.\n", "summary_of_the_review": "Evaluation based on RMSE on the OBSERVED ratings in experiments on recommender systems triggers an immediate reject of the paper in my view. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed an enhanced autoencoder for collaborative filtering by adding another element-wise neural network for rating predictions. Overall the scores are negative, where reviewers pointed out concerns around the motivation, time complexities, and most importantly, using rating prediction as the evaluation setting which ignores the missing-not-at-random nature of the recommender systems. The authors didn't provide any response. Therefore, I vote for rejection.
Decision: Reject