Title: Collaborative Deep Embedding via Dual Networks
Authors: Yilei Xiong
Dahua Lin
Haoying Niu
JIefeng Cheng
Zhenguo Li
Venue: ICLR 2017

Abstract: Despite the long history of research on recommender systems, current approaches still face a number of challenges in practice, e.g. the difficulties in handling new items, the high diversity of user interests, and the noisiness and sparsity of observations. Many of such difficulties stem from the lack of expressive power to capture the complex relations between items and users. This paper presents a new method to tackle this problem, called Collaborative Deep Embedding. In this method, a pair of dual networks, one for encoding items and the other for users, are jointly trained in a collaborative fashion. 
Particularly, both networks produce embeddings at multiple aligned levels, which, when combined together, can accurately predict the matching between items and users. Compared to existing methods, the proposed one not only provides greater expressive power to capture complex matching relations, but also generalizes better to unseen items or users. On multiple real-world datasets, this method outperforms the state of the art.


Review discussion:
Since recall@N is used as metric. It makes sense to do a negative sampling based approach baseline, as opposed to rate prediction models.
E.g. Bayesian personalized recommendation
Thanks for the suggestion. The use of rate prediction is a common practice in existing methods, like WMF and CDL. We did perform empirical comparison of different loss functions in this work, e.g. L2 and logistic. We found that weighted L2 loss consistently results in the best performance among all choices. This is mentioned in the paper.
Following the suggestion in this comment, we conducted an experiment on Bayesian Personalized Recommendation (BPR) on CiteULike, with parameters tuned to obtain the optimal performance. This is a pure collaborative filtering method from [1]. Our experimental results showed that its performance is similar to WMF. Particularly, the Recall@50, 100, 200 for BPR are respectively 39.11%, 49.16%, 59.96%, while those for WMF are  40.45%, 50.25%, 59.95%. We just added these results to the paper.
[1]Rendle, Steffen, et al. "BPR: Bayesian personalized ranking from implicit feedback." Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence. AUAI Press, 2009.
I worry that this method is very dependent on the density of the dataset. Netflix is very dense, and Ciao is sub-sampled to be dense (I didn't spot the total number of interactions in citeUlike, and therefore its density, can you add this?).
Can you comment on how density affects the results or better yet how the Ciao results would look without subsampling?
The densities for CiteULike, MovieLens and Ciao are respectively 0.22%, 0.24%, 0.25%. The densities are at a comparable level. This level of densities are common in datasets for recommendation. Compared to classical methods like collaborative filtering, the proposed method takes into account not only the user history but also the item features, and thus is generally less sensitive to densities. From the experimental results, we found that the recall metrics vary a lot across the three datasets (which have comparable densities). Here, the relevance of the features has more significant impact than the densities.
The sub-sampling step is a common practice in this area.
For example, this practice is also adopted in Wang and Blei 2011 and Wang et al 2015. Both papers are cited. We found empirically that precluding entries that are extremely sparse may reduce the computational cost, but it does not lead to notable impact on the performance metrics.
The responses to the pre-review questions are not strong; especially w.r.t. the question about dataset density and why the dataset had to be subsampled, the authors responded that subsampling is common in recommender systems work, including the papers cited. This isn't a particularly strong justification of why subsampling is a good idea, and in particular doesn't answer the question of "how would the results look without subsampling," which I think is a question that could easily have been answered directly.
Especially given that the goal of dealing with the cold-start issue is so heavily emphasized in the paper, in seems odd to sample the data to reduce sparsity.
Other than that, the pre-review questions seem to have been answered satisfactorily.
The contribution of the paper is to propose user and item embedding methods, as a means of learning complex non-linear interactions between users and items. This is fairly similar to recent work on deep RS, though the network formulation has some differences.
Overall this is an reasonably put together paper that makes a contribution in an important area, though there are still some shortcomings that should be addressed, namely:
1) The evaluation is unusual. Recall@M is the only result reported, though this is not usually an evaluation seen in recommender systems research. At the very least other performance measures (rmse or AUC) should be reported for completeness, even if the results are not strong
2) Given that the contribution is fairly simple (i.e., the "standard" recommender systems task, but with a new model) it's a shame that unusual data samples have to be taken. This should be a case where it's possible to report results against competing methods using *exactly* the same data they used, and exactly the same error measure, for the fairest comparison possible.
Without the above it's hard to tell how much the performance improvements are really due to the method being better, versus the choice of datasets and the choice of loss functions.
Thanks for the comments.
Following the suggestion of the reviewer, we further compared our method with others on MovieLens, on both settings, namely with and without subsampling. Note that without subsampling, the number of users increases from 4,663 to 131,821, while the density decreases from 0.25% to 0.15%. The results are listed below:
=== with subsampling
Recall@50  Recall@100 Recall@200
WMF     37.14%      48.81%     60.25%
CDL     38.11%      49.73%     61.00%
Ours    44.95%      59.15%     72.56%
=== without subsampling
Recall@50  Recall@100 Recall@200
WMF    35.95%      46.48%     56.69%
CDL    37.01%      47.41%     57.34%
Ours   44.35%      57.30%     69.84%
We can see that without subsampling, the recall values decrease moderately for all methods.
However, the general trend remains similar -- our proposed method still outperforms others by a remarkable margin.
As to the evaluation metric, there have been different choices in literatures. Recall@M is a common choice adopted by our baselines [1, 2], and is closely related to how a recommendation system is assessed in practice. That being said, we also did additional tests to compute other metrics (including AUC and RMSE), as listed below:
AUC   CiteULike   MovieLens       Ciao
WMF      91.01%      89.27%     79.82%
CDL      96.36%      89.30%     83.74%
Ours     93.70%      96.74%     81.80%
RMSE    CiteULike   MovieLens   Ciao
WMF         0.070       0.070   0.129
CDL         0.106       0.071   0.088
Ours        0.120       0.128   0.126
We can see that different metrics are not quite correlated. Sometimes, they even exhibit opposite trends.
Previous work [3] has performed systematic study of this issue, and shown that "Common methodologies based on error metrics (such as RMSE) are not a natural fit for evaluating the top-N recommendation task.
Rather, top-N performance can be directly measured by alternative methodologies based on accuracy metrics (such as precision/recall)."
[1] Wang C, Blei D M. Collaborative topic modeling for recommending scientific articles[C]//Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2011: 448-456.
[2] Wang H, Wang N, Yeung D Y. Collaborative deep learning for recommender systems[C]//Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2015: 1235-1244.
[3] P. Cremonesi, Y. Koren, R. Turrin: Performance of recommender algorithms on top-n recommendation tasks. RecSys 2010.
This paper provides a minor improvement paper of DeepRS. The major improvement comes from the coupling of user-item factors in prediction.
While the motivation is clear, the improvement of the model architecture is minor.
I think the author should improve the paper to discuss more on the impact of introduction of coupling, which might make this paper stronger.
Specifically, conduct isolate experiment to change loss, architecture gradually, from a non-coupled network to a final proposed coupled network to demonstrate the importance of coupling.
Another important missing part of the paper seems to be time complexity. Since coupled net would be much more costly to generate recommendations, a discussion on how it would impact real world usages should be added.
Overall, I think this is a paper that should be improved before accepted.
Thanks for the comments.
First, as to the statement "This paper provides a minor improvement paper of DeepRS, which is by the same author", we are not able to identify which paper it refers to. Actually, none of the previous publications of the listed authors were about Deep RS.
The key idea of this work is to couple both the user network and the item network to make predictions. The idea of coupled learning instead of the model architecture is our major contribution. The CDL method, which we compare with as a baseline, can actually be considered as a "less coupled" version, where the training of the item encoder and the rating matrix factorization are done alternatively. Our experimental results have shown that the proposed method yields better results on multiple datasets -- this may be considered as an important evidence that shows the significance of coupling. Intuitively, coupled learning tailors the representations towards accurate recommendation, and thus it can often lead to better recommendation results.
It is true that the dual networks take a bit longer to train.
However, the time complexity remains in an acceptable level (about 10 hours to train a model for MovieLens, our largest dataset). Note that once the networks are trained, they work very efficiently when deployed for service. For existing users or items, encoding features can be calculated in advance and then memorized -- we are as fast as traditional method like WMF under this scenario. The encoding of new users or new items only needs to go through several layers of feed-forward computation (on average, 0.4 milliseconds to encode a user/item). We will add these details to the paper.
revised the comment wrt to the author
The authors proposed to learn embeddings of users and items by using deep neural network for a recommendation task. The resulting method has only minor differences from the previous CDL, in which neural networks were also used for recommendation tasks. In the experiments, since the proposed method, DualNets have use more item features than WMF and CDL, the comparisons are unfair.
Thanks for the comments.
Our model differs from CDL in several key aspects.
(1) Our model encodes both users and items using deep neural networks. The learning objective encourages cooperation of both networks such that the coupling of the resultant embeddings on both sides result in higher prediction accuracy.
(2) The learning task is to estimate the parameters of the encoding networks, without pursuing the optimal embeddings of items or users directly.
As a result, new users and new items are no longer second-class citizens -- they are encoded in exactly the same way as those in the training set.
(1) CDL only encodes items using deep neural network, while the representations of the users are still obtained by matrix factorization. The learning process, driven by the combined objective, devoted a considerable amount of efforts to minimize the reconstruction error of the auto-encoder, which does not necessarily lead to improved recommendation.
(2) Whereas it partly addresses the cold-start problem. There remains inconsistencies between known items and new ones -- the embedding of known items is resulted from a tradeoff between the matrix factorization accuracy and the closeness to the SDAE feature, while the embedding of new items are purely based on the encoding of inherent features.
It is true that WMF, a representative method for pure collaborative filtering, only utilizes rating matrix -- the comparison of WMF and other methods shows the gain of incorporating item contents. Our method, DualNets and CDL are hybrid methods, which incorporates both rating matrix and item contents. DualNets uses the same item features as CDL does.
Please see the review

Meta-review:
role: PC
data: comment: The paper presents a collaborative filtering method, using dual deep nets for users and items. The nets can take advantage of content in addition to ratings. This contribution is just below the bar, in that its novelty relative to existing methods is limited and the results are good but not sufficiently impressive, especially since they focus exclusively on Recall@N. In the response, the authors do present results on other metrics but the results are mixed.

Decision: Reject