EVENT REPORT

Report on the Fourth Workshop on
Recommendation in Complex Environments(ComplexRec 2020)
Toine Bogers
Aalborg University Copenhagentoine@hum.aau.dk
Marijn Koolen
Royal Netherlands Academy of Arts & Sciences
marijn.koolen@di.huc.knaw.nl
Casper Petersen
SamPension
casper.ptrsn@gmail.com

Bamshad Mobasher
DePaul University
mobasher@cs.depaul.edu

Alexander Tuzhilin
New York University
atuzhili@stern.nyu.edu

Abstract
During the past decade, recommender systems have rapidly become an indispensable elementof websites, apps, and other platforms that are looking to provide personalized interaction
to their users. As recommendation technologies are applied to an ever-growing array of nonstandard problems and scenarios, researchers and practitioners are also increasingly facedwith challenges of dealing with greater variety and complexity in the inputs to those recommender systems. For example, there has been more reliance on fine-grained user signals
as inputs rather than simple ratings or likes. Many applications also require more complex
domain-specific constraints on inputs to the recommender systems. The outputs of recommender systems are also moving towards more complex composite items, such as package orsequence recommendations. This increasing complexity requires smarter recommender algorithms that can deal with this diversity in inputs and outputs. The ComplexRec workshop
series offers an interactive venue for discussing approaches to recommendation in complex
scenarios that have no simple one-size-fits-all solution.

1

Introduction

During the past decade, recommender systems have rapidly become an indispensable element of
websites, apps, and other platforms that are looking to provide personalized interaction to theirusers. As recommendation technologies are applied to an ever-growing array of non-standard
problems and scenarios, researchers and practitioners are also increasingly faced with challenges
ACM SIGIR Forum

1

Vol. 54 No. 2 December 2020

of dealing with greater variety and complexity in the inputs to those recommender systems. For
example, there has been more reliance on fine-grained user signals as inputs rather than simple
ratings or likes. Many applications also require more complex domain-specific constraints on
inputs to the recommender systems.
The outputs of recommender systems are also moving towards more complex composite items,
such as package or sequence recommendations. This increasing complexity requires smarter recommender algorithms that can deal with this diversity in inputs and outputs. For the past three
years (Bogers et al. [2017, 2018]; Koolen et al. [2019]), the ComplexRec workshop series has offered
an interactive venue for discussing approaches to recommendation in complex scenarios that haveno simple one-size-fits-all solution.

2

Workshop focus

For the fourth edition of ComplexRec, we narrowed the focus of the workshop and contributions to
the workshop about topics related to one of these two main themes on complex recommendation:complex inputs and complex outputs.

2.1

Complex inputs

An important source of complexity comes from the various types of inputs to the system beyond
users and items, such as features, queries and constraints. There are active user inputs (interaction), implicit user inputs (task, context, preferences), item inputs (features or attributes) anddomain inputs (eligibility, availability). In group-based recommendation, the user input can be a
combination of inputs for multiple individual users as well as group aspects such as the composition of the group and how well they know each other. In the context of education for instance,
inputs range from the knowledge and skill level of the learner, their preferences for modes oflearning, the amount of time available for studying and rehearsing, and the constraints of thelocation, resources and physical space. These constraints may even conflict with each other orwith preferences of different stakeholders.
An additional challenge is providing users with ways to have control over the inputs. For
instance by selecting and weighting or ranking user and item features, or providing interactive
queries to steer the recommendation. In recommender systems designed to assist customers andtroubleshoot problems, or in systems that help office workers with writing reports, one of the inputs
is the problem statement, which can be provided in the form of a few keywords, natural language
questions, or longer narrative statements. These require increasing levels of natural language
understanding to transform into relevant features and constraints that have to be combined withmany other types of inputs, including the knowledge and skill level of the person for whom theadvice or report is intended.
There are many examples of other domains where recommender systems have to deal with acomplex set of inputs, such as:• Food, where there may be dietary requirements, constraints on the availability of ingredients,
cooking utilities, time to cook and budget, paired accompaniments such as wine, as well as
taste preferences and requirements for cuisine and number of courses;
ACM SIGIR Forum2

Vol. 54 No. 2 December 2020

• Tourism, with similar constraints on time and budget, distance, preferences for types oflocations, accommodation and mode of transport, as well as inputs related to the travelgroup;• Transportation, which has to take into account different numbers and types of packages,
available vehicles and drivers, as well as constraints from customers being available to receivea package, as well as different priority levels.

2.2

Complex outputs

Another type of complexity that we focused on in ComplexRec 2020 is the complexity of theoutputs of a recommender system to move away from a straightforward ranked list of items as
output. An example of such complex output is package recommendation: suggesting a set or
combination of items that go well together and are complementary on dimensions that matterto the user. In many domains the sequence in which items are recommended is also important.
Moreover, different users may want different information about items, so the output complexity
goes beyond ranking and also manifests itself in how the interface should allow the user to view thetype of information that is most relevant to them. Another example of complexity in recommendersystems output are environments where the system’s goal is to create new, composite items thatmust satisfy certain constraints (such as menu recommendation, or recommendations for productdesigns).
There are many examples of domains where the outputs of the recommender system are complex in one or more of the above-mentioned ways, such as:• Food, where a recommender system could help the users by suggesting a balanced diet byrecommending a set of dishes over a longer period, or in composing entirely new dishes basedon available ingredients;• Tourism, where travel packages should consist of an optimized, complementary set of locations, accommodations, and activities;• Fashion, where complete matching outfits could be recommended to the user.

3

Format & Topics

ComplexRec 2020 was organized as an interactive, fully on-line, half-day workshop. We encouraged
authors to submit short papers and position papers of 4–8 pages in length dedicated to any
aspect of recommendation in complex environments. We invited contributions that addressed thechallenges associated with constructing recommender systems that must handle complex inputs
and/or outputs. These topics of interest for the workshop included, but were not limited to:• Recommenders with novel complex inputs;• Recommenders with interesting combinations of inputs;• Constraint-driven recommender systems;
ACM SIGIR Forum

3

Vol. 54 No. 2 December 2020

• Novel knowledge-based recommender systems;• Novel modes of user interactions with complex inputs;• Query-driven and interactive recommender systems;• Algorithms and models that effectively integrate complex inputs;• Recommending complex items, such as packages and sequences;• Recommending composite items with complex feature interactions;• Algorithms for generating personalized items based on feature preferences;• Novel NLP approaches for dealing with complex inputs and outputs.
Accepted submissions were then invited for short presentations. Evaluation criteria for acceptance included novelty, diversity, significance for theory/practice, quality of presentation, and thepotential for sparking interesting discussion at the workshop. All submitted papers were reviewed
by the Program Committee.
A total of 6 papers were submitted, of which 5 were accepted for an oral presentation. All
papers were reviewed by at least 3 reviewers from the program committee, consisting of international experts in the field. Likely in part due to its online nature, ComplexRec 2020 had a record
number of attendees with around 70 participants at the workshop at any given time.

4

Workshop program

4.1

KeynoteThe workshop featured a keynote presentation by Christine Bauer, assistant professor at the
Human-Centred Computing group at the Department of Information and Computing Sciences at
Utrecht University. In her talk “Ratings in, rankings out. Keep it simple, they said. But weneed more than that”, she reflected on the complexity of recommender systems by reaching out
to related fields such as context-aware computing and pervasive advertising for inspiration.
Focusing on contextual aspects as complex input, she demonstrated that there are a huge number of relevant signals to go beyond most of our current “context-aware recommender systems”.
She proposed to have a view to related fields that deal with context as deeply complex input. In
context-aware computing, six years of research provided 3,741 distinct contextual elements for 36
different contextual models [Bauer and Novotny, 2017]. For instance for music recommendation,some of the many relevant contextual inputs are personal preferences, mood, cultural backgroundand how it relates to genres, temporal aspects like time of year (e.g. Christmas or summer) andtime of day, purpose (to help relax, motivate to exercise or as background while focusing on work),
location (in the car, in Paris), music setup (e.g. type of speakers) and whether you are listening
on your own or with others, such as your children, partner, friends, colleagues or students.
On the output side, complexity arises in considering what we want to present, how, for who,
and why. A ranked list as output may seem like an appropriate one-size-fits-all solution, but there
ACM SIGIR Forum

4

Vol. 54 No. 2 December 2020

are many factors that play a role. The appropriate rank cut-off is determined not only on theitems in the ranking, but also contextual factors like the device, user preferences and task and how
the ranking is presented. Apart from presenting items in a particular order, some items can behighlighted, for instance through distinct colors, icons or moving images, to draw the attention ofusers. Beyond individual items there are different considerations for item bundles, complementary
goods, sequences, repeated recommendations, etc. Completeness is important in recommending
items that are part of a whole, like chapters in a book. For some sequences of items, the order
is not important (e.g. television series where each program stands on its own), but others it is
crucial.
Finally, she drew our attention to what she calls situationalization as a dimension next topersonalization. Whether recommendations need to be addapted to the situtation can be combinedwith whether recommendations should be personalized, either to a single user or a group of users[Lasinger and Bauer, 2013; Bauer and Lasinger, 2014].

4.2

Accepted papers

In total, five papers were accepted for presentation and they covered a broad set of complex recommendation scenarios. Moskalenko et al. [2020] presented WikiRecNet, a system for providing
personalized recommendations of Wikipedia articles to editors by exploiting the text content andlink structure of the articles and built on top of Graph Convolutional Networks and Doc2Vec.
Their approach was shown to outperform BM25, CB and kNN baselines. Parra et al. [2020] proposed a transfer-learning model, CuratorNet, based on CNN (ResNet) and trained using BPR
for personalized ranking of items from an art store. Their evaluation showed that their modeltends to perform better than two baselines. Mavridis et al. [2020] described various challenges
and viable solutions for some of the ML-powered ranking algorithms powering Booking.com, withfocus on modeling, experimentation and serving. They also show the increase in business value
as a result of these considerations. Wadhwa et al. [2020] attempted to predict a user’s inclination
towards specific price bands using historical user-item, and to use these predictions for creating
recommendations to user through re-ranking. Their approach showed improvements in off-line
evaluation metrics. Ahlers [2020] discusssed the implications of “smart-city” infrastructure for future developments of recommender systems, such as offering inhabitants to adapt their behaviour,for example in the choice of mobility with personalised options. However, he emphasizes that sucha complex recommendation scenario is an as-yet underspecified problem.

4.3

Closing Discussion

The workshop finished with a short plenary discussion that was seeded with the following questions
by the organisers:
1. What is the current state-of-the-art in ComplexRec? What has worked well? What is
missing and where are the gaps?
2. What are the current key challenges in recommender systems research and practice when
dealing with complex scenarios (and more specifically with complex and non-standard inputs
or outputs)?
ACM SIGIR Forum

5

Vol. 54 No. 2 December 2020

3. What are some emerging domains where complexity in the types of inputs or outputs may
require new recommender systems algorithms, designs, or architectures?
4. What are the future potential research directions and challenges in ComplexRec and whatshould be the focus of future workshops in this area?
Several participants pointed to the need for reducing biases inherent in real-world data as
one of the key challenges for not only complex recommendation, but all recommender systems
research. Another interesting discussion was related to how complexity is related to the cost ofinteracting with recommendation: in the case of one-off, expensive purchases, dealing with the
input and output complexity becomes more important than in the case of, for instance, selecting
which song to play next on a music streaming service. Workshop participants agreed that a better
understanding of the relation between cost and complexity would be a valuable direction for futurework.
In response to question 3, many domains with complex inputs or outputs were identified that
could require new algorithms, designs or architectures. Some of the example domains brought
up by the workshop participants included domains with multiple stakeholders, such as educationand smart cities; the health/medical domain, with many different personalized restraints, suchas privacy, dietary, health, and legal); smart homes, where more and more systems need to be
integrated and collaborate with each other; and domains where packages of related items (fashion,cooking) and/or sequences of items (music, travel, entertainment) should be recommended.
One exciting new research direction is recommending new items based on existing features
or partials. For instance, recommending new recipes by extrapolating from changes in what istrending, or by taking into account different cultural or dietary requirements.

5

Acknowledgments

We would like to thank ACM and RecSys for hosting this workshop, the RecSys workshop chairs
Elizabeth Daly and Dietmar Jannach. We would also like to thank the program committee:
Panagiotis Adamopoulos, Robin Burke, Iván Cantador, Pablo Castells, Paolo Cremonesi, Peter
Dolog, Fabio Gasparetti, Marco de Gemmis, Cristina Gena, Dietmar Jannach, Ernesto William
De Luca, Cataldo Musto, Fedelucio Narducci, Tommaso Di Noia, Shaghayegh Sahebi, and Nafiseh
Shabib. Final thanks are due to the paper authors, our invited speaker Christine Bauer, and theparticipants for an interesting and lively workshop.
The workshop material (list of accepted papers, keynote, and the workshop schedule) can
be found on the workshop website at https://complexrec2020.aau.dk/. The proceedings
were published as a CEUR Workshop Proceedings volume, available at http://ceur-ws.org/
Vol-2697/. The recording of the entire workshop are available on the RecSys YouTube channel
at https://youtu.be/nAevW75MzQA.

ACM SIGIR Forum

6

Vol. 54 No. 2 December 2020

References
Dirk Ahlers. Making sense of the urban future: Recommendation systems in smart cities. In
Proceedings of the Fourth Workshop on Recommendation in Complex Scenarios. CEUR-WS,
2020.
Christine Bauer and Peter Lasinger. Adaptation strategies to increase advertisement effectivenessin digital media. Management Review Quarterly, 64(2):101–124, 2014.
Christine Bauer and Alexander Novotny. A consolidated view of context for intelligent systems.
Journal of Ambient Intelligence and Smart Environments, 9(4):377–393, 2017.
Toine Bogers, Marijn Koolen Bamshad Mobasher, Alan Said, and Alexander Tuzhilin. Complexrec
2017. In Proceedings of the First Workshop on Recommendation in Complex Scenarios, volume
1892, pages 1–28. CEUR-WS, 2017.
Toine Bogers, Marijn Koolen, Bamshad Mobasher, Casper Petersen, and Alan Said. Complexrec
2018. In Proceedings of the Second Workshop on Recommendation in Complex Scenarios,
pages 1–37, 2018. URL http://toinebogers.com/workshops/complexrec2018/resources/
proceedings.pdf.
Marijn Koolen, Toine Bogers, Bamshad Mobasher, and Alexander Tuzhilin. Complexrec 2019. In
Proceedings of the Third Workshop on Recommendation in Complex Scenarios, volume 2449,
pages 1–39. CEUR-WS, 2019.
Peter Lasinger and Christine Bauer. Situationalization, the new road to adaptive digital-out-ofhome advertising. In Proceedings of IADIS International Conference e-Society, pages 162–169,
2013.
Themis Mavridis, Soraya Hausl, Andrew Mende, and Roberto Pagano. Beyond algorithms: Ranking at scale at booking.com. In Proceedings of the Fourth Workshop on Recommendation inComplex Scenarios. CEUR-WS, 2020.
Oleksii Moskalenko, Diego Saez-Trumper, and Denis Parra. Scalable recommendation of wikipedia
articles to editors using representation learning. In Proceedings of the Fourth Workshop on
Recommendation in Complex Scenarios. CEUR-WS, 2020.
Denis Parra, Pablo Messina, Manuel Cartagena, Felipe del Rio, and Patricio Cerda-Mardini. Curatornet: Visually-aware recommendation of art images. In Proceedings of the Fourth Workshop
on Recommendation in Complex Scenarios. CEUR-WS, 2020.
Soumya Wadhwa, Ashish Ranjan, Selene Xu, Jason H.D. Cho, Sushant Kumar, and Kannan
Achan. Personalizing item recommendation via price understanding. In Proceedings of the
Fourth Workshop on Recommendation in Complex Scenarios. CEUR-WS, 2020.

ACM SIGIR Forum

7

Vol. 54 No. 2 December 2020