WORKSHOP REPORT

Report on RecSys 2016 Workshop on New Trends inContent-Based Recommender Systems
Toine Bogers
Marijn Koolen
Aalborg University Copenhagen
Huygens ING
Copenhagen, Denmark
Amsterdam, the Netherlands
toine@hum.aau.dk
marijn.koolen@uva.nl
Cataldo Musto Pasquale Lops Giovanni Semeraro
University of Bari “Aldo Moro”, Bari, Italy{cataldo.musto | pasquale.lops | giovanni.semeraro} @uniba.it

Abstract
This article reports on the CBRecSys 2016 workshop, the third edition of the workshop on
New Trends in Content-based Recommender Systems, co-located with RecSys 2016 in Boston, MA.
Content-based recommendation has been applied successfully in many different domains, but it
has not seen the same level of attention as collaborative ﬁltering techniques have. Nevertheless,there are many recommendation domains and applications where content and metadata playa key role, either in addition to or instead of ratings and implicit usage data. The CBRecSys
workshop series provides a dedicated venue for work dedicated to all aspects of content-based
recommender systems.

1

Motivation and Goals

Content-based recommendation has been applied successfully in a variety of domains [8], yet it has
not seen the same level of attention as collaborative ﬁltering techniques have. Competitions likethe Netﬂix Prize1 , CAMRA2 , and the Yahoo! Music KDD Cup 2011 [10] spurred on advancesin collaborative ﬁltering and how to utilize ratings and usage data from a community of users.
However, there exist many recommendation domains and applications where metadata and item
content play a key role, either in addition to or instead of ratings and implicit usage data. For somedomains the relationship between content and usage data has seen thorough investigation already(e.g. movies [15]). For many other domains, however, such as books, news, scientiﬁc articles, andWeb pages we still do not know if and how these data sources should be combined to provide the1 http://www.netflixprize.com/
2 http://www.dai-labor.de/camra2010/

ACM SIGIR Forum45

Vol. 51 No. 1 June 2017

best recommendation performance. On top of that, in certain contexts of search, content-based
recommendation plays a role as well. Personalised IR is strongly related to recommendation, but
recent research in book search also identiﬁed search scenarios that combine aspects of retrieval andrecommendation [11].
The CBRecSys workshop series aims to address this by providing a venue for papers dedicatedto all aspects of and new trends in content-based recommender systems. This includes both recommendation in domains where textual content is abundant (e.g., books, news, scientiﬁc articles,
jobs, educational resources, and Web pages) as well as dedicated comparisons and combinations ofcontent-based techniques with collaborative ﬁltering approaches. After two successful editions organized at RecSys 2014 in Silicon Valley [6, 7] and RecSys 2015 in Vienna [4, 5], the third editionof the workshop, CBRecSys 2016, was co-located with RecSys 2016 in Boston, MA.

2

Workshop Focus & Format

Authors were encouraged to submit papers on a variety of topics related to content-based recommendation, including:• Opinion mining for text/book recommendation• Semantic recommendation• Content-based recommendation to alleviate cold-start problems• Deep learning for content representation• Serendipity and diversity• Cross-domain recommendation
In particular, papers submitted to the the workshop focused on the following topics. As in previous years, we received several submissions that examined how semantic information—usually inthe form of Linked Open Data resources—can be integrated in recommendation algorithms. Another popular approach in 2016, related to semantic recommendation, was the use of text mining
techniques on textual resources, such as reviews, to improve the quality of the recommendations.
Deep learning, currently one of the most popular machine learning techniques, also made its way
to CBRecSys. Finally, one of the accepted submissions examined the problem of content mining for
multimedia content. See Section 2.2 for more details about the accepted submissions.
A total of 14 papers were submitted, of which 9 were accepted for an oral presentation (foran acceptance rate of 64%). All papers were reviewed by at least 3 reviewers from the programcommittee, consisting of international experts in the ﬁeld. In terms of attendance, CBRecSys 2016
proved to be a popular workshop ﬁlling up the maximum room capacity of 30 with around another
30 attendees on the waiting list.

ACM SIGIR Forum

46

Vol. 51 No. 1 June 2017

2.1

Keynotes

The third edition of CBRecSys featured two excellent keynote presentations. In the opening keynote,
Barry Smyth (University College Dublin) discussed the role that user-generated reviews could playin a new generation of recommender systems. Barry presented an overview of the work he and his
group have done on leveraging opinion mining techniques to extract features and sentiment from
unstructured review text, and subsequently how to incorporate this information in recommendation
ranking and explanation.
The closing keynote was given by Bamshad Mobasher (DePaul University), who contrasted
content and context by focusing on the role of semantic and social knowledge in context-aware
recommendation. He presented three different approaches to this: (1) an approach where user proﬁles are represented as mixtures of the latent topics, allowing for a uniﬁed model of users, items, andthe metadata associated with contexts; (2) an approach where contextual information in obtained bymining social annotations or other textual features associated with a user’s current situation and usedin combination with user preference histories to compute a utility function over the set of items; and(3) an approach that emphasizes the role of a domain ontology in the form of a concept hierarchy as
an integral part of a user’s evolving contextual proﬁle.

2.2

Accepted Papers

This section brieﬂy summarizes the eight accepted papers presented at the workshop. In their work,
Lenhart and Herzog [14] addressed the problem of recommending sports news, where users havestrong emotional attachments to particular sports, teams and players. They propose a weighted hybrid content-based and collaborative ﬁltering system, with user proﬁles based on reading behaviour
as implicit feedback. The content-based model only takes recent articles into account, to avoid stale
news. A collaborative ﬁltering component is added that which allows recommending trending news
and older articles that have recently been read. They implemented a recommender dashboard in alarge news site and used online evaluation on 1% of users. The content-based approach leads tohigh accuracy but low diversity. The addition of collaborative ﬁltering increases diversity throughrecommending trending news and older (yet still relevant) articles.
Lak et al. [13] investigated different text-based content analysis techniques for ranking news.
One year of historical data is used to extract text features and combine them with author reputation,
article freshness and genre. Both SVM and Random Forests algorithms are used to train classiﬁers
on three types of text features: keyword popularity, TF-IDF and word2vec. Performance drops as
older articles are included for all three types of features, but least for word2vec. Overall TF-IDF
gives the best performance.
Deldjoo et al. [9] proposed to enrich content-based movie recommendation through automatic
extraction of low-level visual features from shots and key frames. The MovieLens dataset is extended
with movie trailers from YouTube. The extracted features include dynamic features such as object
and camera motion and static features such as color variance and lighting key. The use of visual features improves recommendation performance with respect to baselines that use genre and popularity
information.
Bauman et al. [3] presented a new method for recommending not only items but also speciﬁc
aspects of items, in the domains of restaurant, hotel beauty/spa recommendations. The method
extracts aspect-based sentiments from analysis in user reviews and predicts aspect-based sentiment

ACM SIGIR Forum

47

Vol. 51 No. 1 June 2017

of users. The system is trained and tested on Yelp data, with evaluation showing improved rating
prediction when the system can predict the sentiment of aspects that users mention in their reviews.
Rosati et al. [16] used RDF graph embeddings (rdf2vec) based on short random walks on local
sub-graphs in DBpedia and Wikidata to obtain entity sequences. They experiment with different
link directions (incoming, outgoing) and types (e.g. RDF type, subject) and test their methods on
ratings data for movies and books. Evaluation shows that for the dense movie rating data set, collaborative ﬁltering outperforms content-based approaches. But for book rating data from LibraryThing,which is more sparse, their content-based approach is better. This suggests recommender systems
can effectively exploit content-based strategies for domains and user populations where overlapping
ratings are rare.
Vagliano et al. [20] proposed a dynamic hybrid algorithm that traverses resource and category
links in DBpedia to recommend movies. They conducted a user study using a mobile applicationand compared the proposed approach against other Linked-Data recommender systems, and foundthat their approach hits a sweet spot of good accuracy and good novelty, where the recommended
list of movies contains a good mix of known (but not watched) and unknown movies.
Ahn et al. [1] proposed a rank aggregation model for quote recommendation exploiting complementary strengths of different algorithms for different aspects: Random Forests for word discrimination, Convolutional Neural Networks for local semantic features and Recurrent Neural Networks
to exploit word and sentence ordering. Recommended quotes are chosen from three quote data
sets containing half a million quotes, with mentions and contextual data from Twitter, blogs andGutenberg books. Evaluation shows the three algorithms are highly complementary and together
signiﬁcantly outperform state-of-the-art baselines.
Alzoghbi et al. [2] used learning-to-rank over preference pairs for research paper recommendation. Their approach uses only publicly available metadata and is trained on preference pairs, withpapers marked as interesting taken as positive examples and other papers from the same conference
as negative examples.
Korpusik et al. [12] used recurrent neural networks to predict which Twitter users are potential
buyers of a certain product, to personalise product recommendation. The authors identiﬁed regularexpression patterns associated with buying or desiring a product and trained recurrent neural networks on tweets both with and without sequence information, with the best performance achieved
with a feed-forward network exploiting sequential information.

2.3

Discussion Session about Current and Future Challenges

The workshop was concluded by a discussion session on current trends and future challenges incontent-based recommender systems. One of the most important challenges that was brought up
by several of the participants was the difficulty of comparing the results of different recommendation
algorithms across data sets as reported in research. Currently, it is difficult to know whether newlyproposed algorithms actually outperform other state-of-the-art approaches, because of differencesin the data sets used, ﬁltering performed on these data sets, use of evaluation metrics, and otherelements of the experimental setup. An additional problem is that details about the experimental
setup or evaluation regimen are often missing or lacking in detail, which makes it even harder tocompare a new algorithm to other state-of-the-art approaches.
Naturally, this issue is important not just for content-based recommendation, but for recom-

ACM SIGIR Forum48

Vol. 51 No. 1 June 2017

mender systems research in general, and was signaled already in 2014 by Said & Bellogín [17].
They also proposed RIVAL, a toolkit for transparent and objective benchmarking of recommendersystems software. This would enable making robust and comparable assessments of their recommendation quality [18]. Another possible (and partial) solution to this problem could be to set up
an recommender systems evaluation infrastructure that stores the results of recommendation runs,
as well as the data sets and details about the experimental setup(s). Similar to the PROMISE evaluation infrastructure3 for information retrieval research, this would greatly streamline the process of
directly comparing new algorithms to older state-of-the-art algorithms.
Another challenge brought up was the need for data sets that enable the study of feature preferences by users: which aspects, properties, or features of a product do they prefer as opposed to only
studying item-level preferences. More research is also needed on how to elicit and collect such feature
preference information. A better understanding of feature preferences could lead to higher-quality
recommendations. It would also enable the development of approaches to multi-criteria feature selection that take other dimensions besides accuracy into account, such as the potential ‘explainability’of recommendation.
Finally, the topic of explaining content-based recommendation evoked additional discussion,
especially in light of the EU’s General Data Protection Regulation (GDPR) law4 which will comeinto effect in 2018. Speciﬁc provisions of the law may be interpreted in such as way that EU citizens
might soon have a way to demand explanations of the decisions that algorithms make about them.
In addition to a possible legal requirement, generating explanations for recommendations can also
improve the transparency, scrutability, trust, effectivenes, persuasiveness, efficiency, and satisfactionof a recommender system [19]. Participants quickly agreed that the relevant importance of thisdepends on the application domain, which was followed by a lively discussion about the relative
importance of explanations for the different domains.

3

Future Workshop

The continued popularity of the CBRecSys workshops in terms of both submissions and participation
clearly shows a consistent interest in research on content-based recommendation. We are therefore
planning a fourth and ﬁnal edition of the CBRecSys workshop at RecSys 2017 in Como, Italy.

4

Acknowledgments

We would like to thank ACM and RecSys for hosting this workshop, the RecSys workshop chairs
Elizabeth Daly and Dietmar Jannach. We would also like to thank the program committee: Jon Atle
Gulla, Shlomo Berkovsky, Ludovico Boratto, Robin Burke, Iván Cantador, Federica Cena, Paolo
Cremonesi, Marco de Gemmis, Ernesto William De Luca, Tommaso Di Noia, Peter Dolog, Fabio
Gasparetti, Christina Gena, Frank Hopfgartner, Juan F. Huete, Jaap Kamps, Silvia Likavec, BabakLoni, Fedelucio Narducci, Casper Petersen, Shaghayegh Sahebi, Alan Said, Marko Tkalĉiĉ, and3 See http://promise-noe.eu for more information.

4 Available at http://ec.europa.eu/justice/data-protection/.

ACM SIGIR Forum

49

Vol. 51 No. 1 June 2017

Bei Yu. Final thanks are due to the paper authors, our invited speakers Barry Smyth and Bamshad
Mobasher, and the participants for an interesting and lively workshop.
The workshop material (list of accepted papers, invited talk, and the workshop schedule) can befound on the CBRecSys 2016 workshop website at http://cbrecsys2016.aau.dk/. The proceedings were published as a CEUR Workshop Proceedings volume, available at http://ceur-ws.org/
Vol-1673/.

References[1] Y. Ahn, H. Lee, H. Jeon, S. Ha, and S.-G. Lee. Quote Recommendation for Dialogs andWritings. In CBRecSys ’16: Proceedings of the 3rd Workshop on New Trends on Content-Based
Recommender Systems, pages 39–42, 2016.
[2] A. Alzoghbi, V. A. A. Ayala, P. M. Fischer, and G. Lausen. Learning-to-Rank in Research
Paper CBF recommendation: Leveraging Irrelevant Papers. In CBRecSys ’16: Proceedings of the
3rd Workshop on New Trends on Content-Based Recommender Systems, pages 43–46, 2016.
[3] K. Bauman, B. Liu, and A. Tuzhilin. Recommending Items with Conditions Enhancing User
Experiences Based on Sentiment Analysis of Reviews. In CBRecSys ’16: Proceedings of the 3rdWorkshop on New Trends on Content-Based Recommender Systems, pages 19–22, 2016.
[4] T. Bogers and M. Koolen. Report on RecSys 2015: Workshop on New Trends in ContentBased Recommender Systems. SIGIR Forum, 49(2):141–146, 2015.
[5] T. Bogers and M. Koolen. Second Workshop on New Trends in Content-based Recommender Systems (CBRecSys 2015). In RecSys ’15: Proceedings of the Ninth ACM Conference on
Recommender Systems, pages 339–340, 2015.
[6] T. Bogers, M. Koolen, and I. Cantador. Workshop on New Trends in Content-based Recommender Systems (CBRecSys 2014). In RecSys ’14: Proceedings of the Eighth ACM Conference
on Recommender Systems, pages 379–380, 2014.
[7] T. Bogers, M. Koolen, and I. Cantador. Report on RecSys 2014: Workshop on New Trends
in Content-Based Recommender Systems. SIGIR Forum, 49(1):20–26, 2015.
[8] M. de Gemmis, P. Lops, C. Musto, F. Narducci, and G. Semeraro. Semantics-Aware ContentBased Recommender Systems. In F. Ricci, L. Rokach, and B. Shapira, editors, Recommender
Systems Handbook, pages 119–159. Springer, 2015.
[9] Y. Deldjoo, M. Elahi, and P. Cremonesi. Using Visual Features and Latent Factors for Movie
Recommendation. In CBRecSys ’16: Proceedings of the 3rd Workshop on New Trends on ContentBased Recommender Systems, pages 15–18, 2016.
[10] G. Dror, N. Koenigstein, Y. Koren, and M. Weimer. The Yahoo! Music Dataset and KDDCup ’11. In JMLR Workshop and Conference Proceedings, volume 18 of Proceedings of KDD Cup
2011, pages 3–18, 2012.

ACM SIGIR Forum

50

Vol. 51 No. 1 June 2017

[11] M. Koolen, T. Bogers, A. van den Bosch, and J. Kamps. Looking for books in social media:
An analysis of complex search requests. In Advances in Information Retrieval: 37th European
Conference on IR Research (ECIR 2015), LNCS. Springer, 2015.
[12] M. Korpusik, S. Sakaki, F. Chen, and Y.-Y. Chen. Recurrent Neural Networks for Customer
Purchase Prediction on Twitter. In CBRecSys ’16: Proceedings of the 3rd Workshop on New Trends
on Content-Based Recommender Systems, pages 47–50, 2016.
[13] P. Lak, C. Babaoglu, A. B. Bener, and P. Pralat. News Article Position Recommendation Basedon the Analysis of Article’s Content – Time Matters. In CBRecSys ’16: Proceedings of the 3rdWorkshop on New Trends on Content-Based Recommender Systems, pages 11–14, 2016.
[14] P. Lenhart and D. Herzog. Combining Content-based and Collaborative Filtering for Personalized Sports News Recommendations. In CBRecSys ’16: Proceedings of the 3rd Workshop on
New Trends on Content-Based Recommender Systems, pages 3–10, 2016.
[15] I. Pilászy and D. Tikk. Recommending New Movies: Even a Few Ratings Are More ValuableThan Metadata. In RecSys ’09: Proceedings of the Third ACM Conference on Recommender Systems,
pages 93–100. ACM, 2009.
[16] J. Rosati, P. Ristoski, T. D. Noia, R. D. Leone, and H. Paulheim. RDF Graph Embeddings
for Content-based Recommender Systems. In CBRecSys ’16: Proceedings of the 3rd Workshop on
New Trends on Content-Based Recommender Systems, pages 23–30, 2016.
[17] A. Said and A. Bellogín. Comparative Recommender System Evaluation: Benchmarking Recommendation Frameworks. In RecSys ’14: Proceedings of the 8th ACM Conference on Recommender Systems, pages 129–136, 2014.
[18] A. Said and A. Bellogín. RIVAL: A Toolkit to Foster Reproducibility in Recommender System
Evaluation. In RecSys ’14: Proceedings of the 8th ACM Conference on Recommender Systems, pages
371–372, 2014.
[19] N. Tintarev and J. Masthoff. Designing and Evaluating Explanations for Recommender Systems. In F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor, editors, Recommender Systems
Handbook, pages 479–510. Springer, 2011.
[20] I. Vagliano, C. Figueroa, O. R. Rocha, M. Torchiano, C. Faron-Zucker, and M. Morisio.
ReDyAl: A Dynamic Recommendation Algorithm based on Linked Data. In CBRecSys ’16:
Proceedings of the 3rd Workshop on New Trends on Content-Based Recommender Systems, pages
31–38, 2016.

ACM SIGIR Forum

51

Vol. 51 No. 1 June 2017