EVENT REPORT

Report on the 10th Joint Workshop on
Interfaces and Human Decision Making for
Recommender Systems (IntRS 2023) at ACM
RecSys 2023
Peter Brusilovsky

Marco de Gemmis

Alexander Felfernig

University of Pittsburgh
USA

University of Bari Aldo Moro
Italy

Graz University of Technology
Austria

peterb@pitt.edu

marco.degemmis@uniba.it

alexander.felfernig@ist.tugraz.at

Pasquale Lops

Marco Polignano

Giovanni Semeraro

University of Bari Aldo Moro
Italy

University of Bari Aldo Moro
Italy

University of Bari Aldo MoroItalypasquale.lops@uniba.it

marco.polignano@uniba.it

giovanni.semeraro@uniba.it

Martijn C. Willemsen
Eindhoven University of Technology & JADSThe Netherlands
M.C.Willemsen@tue.nlAbstract
The 10th edition of the Joint Workshop on Interfaces and Human Decision Making for Recommender Systems was held as part of the 17th ACM Conference on Recommender Systems(RecSys), the premier international forum for the presentation of new research results, systems and techniques in the broad field of recommender systems. The workshop was organized
as a hybrid event: the physical session took place on September 18th at the venue of themain conference, Singapore, with the possibility for authors to present remotely. The IntRS
workshop brings together an interdisciplinary community of researchers and practitioners
who share research on new recommender systems (informed by psychology), including new
design technologies and evaluation methodologies, and aim to identify critical challenges andemerging topics in the field. This year we focused particularly on topics related to HumanCentered AI, Explainability of decision-making models, User-adaptive XAI systems, which
are becoming more and more popular in the last years, especially in domains where recommended options might have ethical and legal impacts on users. The integration of XAI
with recommender systems is crucial for enhancing their transparency, interpretability, andaccountability. This topic attracted a lot of interest from the community, as demonstrated by
the fact that several workshop papers proposed methods for recommendation explanations.
Date: 18 September 2023.
Website: https://intrs2023.wordpress.com.

ACM SIGIR Forum

1Vol. 57 No. 2 – December 2023

1Introduction

Recommender systems were originally developed as interactive intelligent systems that can proactively guide users to items that match their preferences. Despite its origin on the crossroads of HCIand AI, the majority of research on recommender systems gradually focused on objective accuracy
and ranking criteria paying less and less attention to how users interact with the system as well as
the efficacy of interface designs from users’ perspectives. The current studies are moving beyond
the isolated algorithm or graphical user interface to evaluate the user experience comprehensively,
while attempting to enhance both the effectiveness and efficiency of the recommender systems.
The series of workshops on Interfaces and Human Decision Making for Recommender Systems
focuses on the “human side” of recommender systems. The goal of the research stream featured
at the workshop is to improve users’ overall experience with recommender systems by integrating
different theories of human decision making into the construction of recommender systems andexploring better interfaces for recommender systems. The integration of human aspects into therecommendation process is a prerequisite for making such systems effective and efficient in theirinteraction with the end user.

2Workshop Theme and Topics

The 10th Joint Workshop on Interfaces and Human Decision Making for Recommender Systems(IntRS’23) follows successful workshops on the same topic organized at RecSys conferences in 2014- 2022. The workshop series was created by merging two original RecSys workshops series: HumanDecision Making and Recommender Systems (Decisions@RecSys – 2010–2013) and Interfaces for
Recommender Systems (InterfaceRS’12). The idea of merging the two workshops was motivatedby the strong inter-relationship between the user interface and human decision making topics.
The combination of these two aspects seems to be highly attractive. Earlier workshops, such asthe IntRS’15 workshop in Vienna, the IntRS’16 in Boston, the IntRS’17 in Como, the IntRS’18 inVancouver, the IntRS’19 in Copenhagen were highly appreciated and widely attended. The virtualedition of IntRS’20 and hybrid sessions at IntRS’21 and IntRS’22 opened workshop participation
to a broader audience and further increase the number of attendees.
The main research strands covered by the workshop are:• User interfaces for recommender systems: visual interfaces, explanation interfaces, conversational recommender systems, mechanisms to incorporate User Experience for embedded
customization in the user interface;• Interaction, user modeling and decision making: cognitive, affective, and personality-baseduser models for recommender systems, decision biases, cognitive biases, persuasive recommendation and argumentation, explainable recommendation models;• Evaluation: user-centric evaluation, beyond-accuracy objectives and metrics (novelty, diversity, serendipity, trustability, fairness, and utility), case studies, benchmarking platforms,
empirical studies of new interfaces and interaction designs, evaluations in real-world contexts;• Influence of recommender systems on user’s behavior: An interesting research direction thathas recently received renewed interest is to investigate how users interact with recommenders

ACM SIGIR Forum

2Vol. 57 No. 2 – December 2023

based upon their cognitive model of the system. We believe that the paradigm that describes
the relationship between humans and recommender systems is changing and evolving toward“symbiotic recommender systems”, in which both parties learn by observing each other.

3

Workshop Contributions

3.1

KeynoteThe workshop had Marko Tkalčič as invited keynote speaker, who shared his perspectives on
how cognitive models of personality and emotions can improve recommender systems. He is associate professor at the Faculty of Mathematics, Natural Sciences and Information Technologies(FAMNIT) at the University of Primorska in Koper, Slovenia. He aims at improving personalizedservices (e.g. recommender systems) through the usage of psychological models in personalization algorithms. To achieve this, he uses diverse research methodologies, including data mining,machine learning, and user studies. He presented his talk on ”From Amateur Musicianship to
Computational Predictors of Media Consumption Experiences” to highlight the importance ofcognitive and affective models when acquiring knowledge about users in recommender systems.
Behaviour implicit data is only part of the user modeling process. To overcome this limitation,
in his talk Professor Tkalčič showed how the usage of psychology-inspired models can lead to abetter acquisition of user preferences.

3.2

Paper Sessions

All paper submissions were reviewed by at least three members of the program committee. In
the reviewing process, we ensured that authors received appropriate and constructive feedback.
The reviews were examined by the workshop chairs, who accepted three contributions as regularpapers, and three contributions as short papers. Authors presented and discussed their work at
the workshop. We provide a brief description for each contribution below.
Regular Papers
In [Majjodi et al., 2023], the authors presented a study that investigated how the user’s level ofdomain knowledge may affect user evaluation of recommendations, as well as the effectiveness ofinterfaces and interaction methods. The study was performed in the context of food recommendation and showed that a knowledge-based preference elicitation method led to healthier recipechoices. Another contribution of this paper concerned the interplay between the user’s nutritional
knowledge and the preference elicitation method. The sudy revealed that users with higher levelsof food knowledge experience additional benefits when using a knowledge-based recommender.
The work [Manzoor et al., 2023] investigated the impact of three types of factors (user-related,system-related, and context-related) on users’ perceived meaningfulness of a conversational recommender system performance. The main lesson learnt from this work is that it is vital to consider
the target user base and its familiarity with technology and provide more advanced features likeaccess to item metadata, reviews proliferation, or explanations for item recommendations, tomeet their expectations. Other interesting findings are that different age groups prefer varying
ACM SIGIR Forum

3Vol. 57 No. 2 – December 2023

lengths of dialogs, and that female users tend to have a higher need for informative responses. In[Szymanski et al., 2023], the authors investigated the problem of designing explanations for nonexpert users in critical contexts such as health. The work presented the results of a user study
performed through a mobile app that provides knowledge-based recommendations to users experiencing chronic musculoskeletal pain, regarding how to effectively manage pain flare-ups. The
study showed that hybrid explanations were largely preferred compared to unimodal explanations(textual and visual) because they were perceived as more useful.
Short Papers
The first short paper [Starke et al., 2023] studied the problem of how to design communication
strategies that encourage consumers to engage in sustainable behavior. The authors explored theimpact of linguistic (i.e., message) and visual framing of product information on sustainable fashion
decision-making. Linguistic framing can take multiple forms: the proposed approach investigate
the effectiveness of abstract versus concrete messages, e.g. ”Eco-friendly T-shirt” versus ”T-shirt
with recycled cotton”. The visual strategy used eco-labels, graphic representations intended todesignate items that have a favorable environmental and social impact. The study highlighted
the importance of using visuals to make easier cognitive processing. The only effect found forthe linguistic framing was that a concrete description are preferred among sustainable options.
The other two short papers focused on recommendation explanations. In [Lubos et al., 2023],
an online user study was performed to assess the effectiveness of consequence-based explanations,a novel type of explanation that the authors introduced to emphasize the individual impact ofconsuming a recommended item on the user, which makes the effect of following recommendations
clearer. The outcomes of the study highlight the importance of this kind of explanations, whichwere well-received by users and effectively improved user satisfaction in recommender systems. The
method proposed in [Silva et al., 2023] exploited Large Language Models (LLMs) to generate movie
recommendation explanations. The authors presented an interesting demo at the workshop, which
encouraged participants to experiment with a web app to evaluate the effectiveness of explanations
customized according to user-defined criteria, such as novelty or popularity.

4Conclusion and Future Directions

During the ten years of IntRS workshops, a range of topics related to interfaces and decisionmaking in recommender systems were presented and discussed. The popularity of topics changed
year to year following the development of the field [Brusilovsky et al., 2020]. In particular, the papers at 2023 workshop reflected the increasing interest to explainability, which was re-invigoratedby LLMs. However, year by year, the workshop kept true to its focus on the “human side” ofrecommender systems, providing a platform for a community of researchers interested in humancentered recommender systems to meet and discuss their work. We hope to continue this trendin 2024. As 10 years is an important milestone, we also plan to organize a special issue onhuman-centered recommender systems in one of the major journals focusing on personalization
and recommender systems. If this field is of interest to you, we hope to see you among the authorsof the next workshop or the special issue!

ACM SIGIR Forum

4Vol. 57 No. 2 – December 2023

Acknowledgments
The IntRS chairs would like to thank the RecSys 2023 workshop chairs, Ludovico Boratto, MiZhang, and Victor Sheng, for their guidance during the workshop organization. We also wish tothank all authors and all presenters, and the members of the program committee. All of themsecured the usual workshop’s high-quality standards.

Program Committee• Ludovico Boratto, University of Cagliari, Italy• Li Chen, Hong Kong Baptist University, China• José A. Galindo, University of Seville, Spain• Sergiu Gordea, AIT Austrian Institute of Technology GmbH, Austria• Denis Helic, Graz University of Technology, Austria• Andreas Holzinger, Human-Centered AI Lab, Austria• Anthony Jameson, Chusable AG, Gamprin, Liechtenstein• Dietmar Jannach, University of Klagenfurt, Austria• Bart Knijnenburg, Clemson University, United States• Joseph Konstan, University of Minnesota, United States• Dominik Kowald, Know-Center, Graz, Austria• Viet-Man Le, Graz University of Technology, Austria• Julia Neidhardt, TU Wien, Austria• Behnam Rahdari, University of Pittsburgh, United States• Marko Tkalčič, University of Primorska, Slovenia• Chun-Hua Tsai, University of Nebraska at Omaha, United States• Eduardo Veas, Graz University of Technology, Austria• Wolfgang Wörndl, Technical University of Munich, Germany
We also express out gratitude to ACM RecSys 2023 organizers for hosting the workshops andproviding meeting facilities.

References
Peter Brusilovsky, Marco de Gemmis, Alexander Felfernig, Pasquale Lops, John O’Donovan, Giovanni Semeraro, and Martijn C. Willemsen. Interfaces and human decision making for recommender systems. In Fourteenth ACM Conference on Recommender Systems, pages 613–618.
ACM, 2020. doi: 10.1145/3383313.3411539. URL https://doi.org/10.1145%2F3383313.34
11539.
Sebastian Lubos, Thi Ngoc Trang Tran, Seda Polat Erdeniz, Merfat El Mansi, Alexander Felfernig,Manfred Wundara, and Gerhard Leitner. Concentrating on the Impact: Consequence-based
Explanations in Recommender Systems. In Proceedings of the 10th Joint Workshop on Interfacesand Human Decision Making for Recommender Systems (IntRS 2023), pages 63–73. CEURWS.org, 2023.
ACM SIGIR Forum

5Vol. 57 No. 2 – December 2023

Ayoub El Majjodi, Alain D. Starke, Mehdi Elahi, and Christoph Trattner. The Interplay between
Food Knowledge, Nudges, and Preference Elicitation Methods Determines the Evaluation ofa Recipe Recommender System. In Proceedings of the 10th Joint Workshop on Interfaces andHuman Decision Making for Recommender Systems (IntRS 2023), pages 1–18. CEUR-WS.org,2023.
Ahtsham Manzoor, Wanling Cai, and Dietmar Jannach. Factors Influencing the Perceived Meaningfulness of System Responses in Conversational Recommendation. In Proceedings of the 10th
Joint Workshop on Interfaces and Human Decision Making for Recommender Systems (IntRS2023), pages 19–34. CEUR-WS.org, 2023.
Itallo Silva, Alan Said, Leandro Balby Marinho, and Martijn Willemsen. Leveraging Large Language Models for Recommendation and Explanation. In Proceedings of the 10th Joint Workshop
on Interfaces and Human Decision Making for Recommender Systems (IntRS 2023), pages 74–81. CEUR-WS.org, 2023.
Alain D. Starke, Kimia Emami, Andrea Makarová, and Bruce Ferwerda. Using Visual and Linguistic Framing to Support Sustainable Decisions in an Online Store. In Proceedings of the 10th
Joint Workshop on Interfaces and Human Decision Making for Recommender Systems (IntRS2023), pages 53–62. CEUR-WS.org, 2023.
Maxwell Szymanski, Cristina Conati, Vero Vanden Abeele, and Katrien Verbert. Designing andPersonalising Hybrid Multi-Modal Health Explanations for Lay Users. In Proceedings of the 10th
Joint Workshop on Interfaces and Human Decision Making for Recommender Systems (IntRS2023), pages 35–52. CEUR-WS.org, 2023.

ACM SIGIR Forum6Vol. 57 No. 2 – December 2023