# A Comprehensive Review of User Interaction for Recommendation Systems

Carla Alexandra Martins, Carina Friedrich Dorneles, Marco Antônio Winckler

► To cite this version:

Carla Alexandra Martins, Carina Friedrich Dorneles, Marco Antônio Winckler. A Comprehensive Review of User Interaction for Recommendation Systems. iSys - Brazilian Journal of Information Systems, 2024, 10.5753/isys.2024.4064. hal-04877488

HAL Id: hal-04877488

https://inria.hal.science/hal-04877488v1

Submitted on 9 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Comprehensive Review of User Interaction for Recommendation Systems
## Uma Revisão Abrangente da Interação do Usuário em Sistemas de Recomendação

Carla Alexandra Martins¹ Ⓒ, Carina Friedrich Dorneles¹ Ⓒ, Marco Antônio Winckler² Ⓒ

¹Statistics and Informatics Department - Federal University of Santa Catarina (UFSC)
Florianópolis, Santa Catarina -Brazil

²Department of Informatics at Polytech Nice, Université Côte d'Azur (UCA)
Sophia Antipolis, Nice - France

acarlam@gmail.com, carina.dorneles@ufsc.br,
marco.winckler@univ-cotedazur.fr

**Abstract.** In recent years, Recommendation Systems have become integral to the online experiences of consumers, particularly those that effectively integrate user interactions into their algorithms, enhancing both efficiency and adaptability. This article presents a comprehensive systematic review of the literature addressing classical problems in recommendation systems, specifically focusing on the consideration of user interaction. We employed a rigorous systematic literature review methodology, critically analyzing various proposals to identify their limitations, characteristics, and potential avenues for further research. Our investigation involved mapping relevant studies that examine how user interaction with recommendation systems is addressed and determining the extent to which this aspect has been explored. We established strict inclusion and exclusion criteria to select academic publications, resulting in a curated set of 29 scientific papers. The findings offer a snapshot of the primary characteristics of the identified works, revealing significant gaps that can inform future research directions. Our analysis indicates that most studies addressing user interaction emphasize preference elicitation and feedback mechanisms, predominantly focusing on improving the accuracy of recommendation rankings, with a notable concentration on the e-commerce domain.

**Keywords.** Recommendation systems; User behavior; User interaction, User feedback.

## 1. Introduction

Recommendation Systems (RSs) have become vital tools in assisting users with their decision-making processes, whether selecting products to purchase, choosing music
to listen to, or identifying relevant news articles. Nonetheless, as pointed by many authors, knowing the users and their preferences is essential for tailoring effective recommendations [Adomavicius and Tuzhilin 2005, Christensen and Schiaffino 2011, Jannach et al. 2011, Hebrado et al. 2011, Carrer-Neto et al. 2012, Bobadilla et al. 2013, Aggarwal and Aggarwal 2016, Hwangbo et al. 2018]. For that reason, RSs aim to capture users' preferences and past interactions with items (such as purchases and likes), which might ultimately requires aggregating user preferences across a wide array of domains, including products, music, movies, books, jobs, videos, and services.

User preferences might be gathered explicitly by asking users (ex. to provide rating to item), or implicitly by tracking user activities (such as website visits, viewed items, etc.) [Liu et al. 2010]. A significant amount of information about users can be collected in a non-intrusive manner, simply by observing the users' interactions. While there are many ways of interacting with the system (e.g. selecting items, spending time reading, scrolling, rating items, confirming a purchase...), not all user interactions are directly accountable for a user preference or interest. Thus, one of the tricky questions when designing a RS is to determine which user interactions are relevant for supporting a recommendation.

User interactions are often motivated by specific tasks with defined goals [Parush 2015, Diaper and Stanton 2003]. The user interaction within RSs can be divided into two parts [Jugovac and Jannach 2017a]: (i) preference elicitation, where the system personalizes recommendations and prompts users to express their preferences, and (ii) interaction during result presentation, or feedback, when users engage with and evaluate the provided recommendations. When users receive tailored suggestions that align with their interests, their engagement increases, fostering loyalty. Interesting enough, the dynamic nature of user interactions allows systems to adapt to evolving preferences, ensuring that recommendations remain relevant over time. This adaptability enhances the effectiveness of recommendation systems, ultimately driving better outcomes for users and providers.

While the fundamental interaction process is similar across many information systems, RSs stand out due to their tailored outputs, which depend on a variety of shared parameters with the user, including: the nature and extent of user input, the familiarity of the recommendations, the transparency of the system's logic, and the number of recommendations [Swearingen and Sinha 2002]. To improve the accuracy of recommendations, it is essential to integrate additional information alongside user ratings [Patro et al. 2020]. An RS can be conceptualized as comprising a user model, a community, an item model, a recommendation algorithm, and an interaction style [Jawaheer et al. 2014].

The importance of user interaction lies in its influence on the perceived relevance of the information delivered by the system. Each interaction consists of user actions, allowing the system to capture user behavior over time to enhance future recommendation predictions [Rocha Silva et al. 2020a, Mo et al. 2014, Guo and Li 2020a, Wang et al. 2016, Gan and Xiao 2019, Nguyen et al. 2020, Albanese et al. 2004, Sinha and Dhanalakshmi 2019, Baeza-Yates et al. 2005]. RSs can employ various functionalities to promote user interaction, such as strategically positioning items to boost the likelihood of user selection [Xu et al. 2020]. Proposals to create persona prototypes
based on user behavior metrics also leverage interactions to deepen the understanding of recommendation complexities [Misztal-Radecka and Indurkhya 2020]. The overarching goal is to capture user behavior dynamically, crafting a framework that tailors recommendations to specific contexts [Dao et al. 2012, Adomavicius and Tuzhilin 2010, Qian Gao 2021, Su et al. 2021, Setiowati et al. 2018, Tarus et al. 2018, Wu et al. 2016].

In order to create intelligent RSs, it is essential to capture user interactions and apply this input to their algorithms, thus enhancing both flexibility and user value [Pu et al. 2011]. However, successful implementation begins with a thorough understanding of how to accurately capture user interactions, including fluctuations in user preferences [Rashid et al. 2008]. It is critical to adopt processes that accommodate these dynamic user characteristics. Research shows that incorporating temporal features into datasets can enhance rating predictions and ranking accuracy. Consequently, exploring the contextual aspects of user feedback has the potential to improve overall performance [Jawaheer et al. 2014, Adomavicius and Tuzhilin 2015]. Such adaptability enables RSs to respond more effectively to users' changing behaviors, influenced by various external factors or time periods [Wang et al. 2009]. By considering these aspects, recommendation systems can offer a more user-friendly interaction experience.

This study aims at investigating the relationship between classical problems in recommendation systems (ex. improving predictions and ranking accuracy) and how these studies address user interaction and changes in user preferences. We examined research proposing solutions for user behavior prediction, rating prediction, user modeling, item modeling, and precision improvement. The outcomes of these analyses provide a comprehensive overview of the merits and limitations of various proposals concerning user interactions. This broad examination can serve as a valuable resource for future research aimed at enhancing user-system interactions, including identifying dynamic user behaviors and other influential factors [Oard and Kim 2001, Silva and Winckler 2022, Xu et al. 2019].

Our review adheres to the established guidelines [Kitchenham and Charters 2007] to systematically searching for, identifying, examining, categorizing, and discussing the current state of recommendation challenges, methodologies, and insights related to user interactions in the scientific literature. This article focuses on the following contributions:

* systematically gathering and examining research proposals related to how existing works addressing classical problems in recommendation systems consider user interaction in their solutions, including changes in user preferences;

* identifying and describing the actions that users might perform while interacting with the recommendation systems;

* analyzing the main features of recommendation systems and how they are related to the user interaction with recommendation systems;

* providing a snapshot of the primary characteristics of the identified works;

* highlighting gaps found in the literature that can guide future research.

The structure of the article is as follows: section 2 briefly introduces the fundamental concepts that motivated the research on user interaction with RSs; section 3 details the methodology employed for obtaining the studies used to extract responses to our research questions, offering an overview of the process and describing the study extraction
and selection steps; the response stratification is elaborated in section 4, along with in-
sights to guide future research, which are briefly discussed at the end of this section;
lately, the section 6 presents the conclusions drawn from our findings.

## 2. Background

In this section, we describe the background for our study, showing first the theories from the field of Information Systems that could be applied in the context of user interaction and recommendation systems. After, we describe the fundamental concepts that are important to understand the rest of the article.

### 2.1. Theoretical Background

In the context of user interaction and recommendation systems, some theoretical frame-
works from the field of Information Systems can provide a robust foundation for under-
standing user behavior and system design. Among these, Cognitive Load Theory (CLT)
stands out as particularly relevant [Reese et al. 2016]. CLT focuses on how individuals
process information and the impact of cognitive load on their ability to comprehend and
interact with system outputs. By applying this theory, researchers can investigate how
the design of recommendation systems and information retrieval interfaces influences
user cognitive load, ultimately affecting their engagement and decision-making processes.
This approach is essential in environments where users must navigate complex informa-
tion landscapes, as it highlights the importance of optimizing system design to enhance
user experience and effectiveness.

The ISO Standard 9241-210:2019 [ISO 2019] emphasizes the need of designing
systems that prioritize user needs and behaviors. UCD advocates for understanding users'
preferences, motivations, and interaction patterns, which can significantly inform the de-
velopment of more intuitive and effective retrieval systems. By incorporating UCD princi-
ples, researchers can analyze how user interactions with information systems are shaped
by interface design and functionality, leading to improvements in user satisfaction and
system performance.

Additionally, the Information Systems Success Model (ISSM) [Delone and McLean 1992] provides a comprehensive perspective by examining the relationships between system quality, information quality, user satisfaction, and net benefits. This model allows for the exploration of how the quality of retrieved information and system attributes influences user interaction. By integrating ISSM into the analysis, researchers can better understand the drivers of successful user experiences in information retrieval systems and identify key areas for improvement.

In summary, leveraging these theoretical frameworks, Cognitive Load Theory, User-Centered Design, and the Information Systems Success Model, can significantly enhance the theoretical underpinnings of research in user interaction and retrieval systems. Each theory offers unique insights that can inform the design, evaluation, and refinement of systems to better meet user needs and improve interaction outcomes.
## 2.2. Fundamental Concepts

In this article, we are considering only RSs based on filtering methods. In this context, RSs are divided into three main approaches based on filtering methods [Zheng et al. 2009]: (i) content-based (CB), whose recommendations are created based on attributes of items and users. These attributes can include personal details like age, gender, occupation, and other individual information; the item attributes refers to descriptive information that sets one item apart from another, which helps the system distinguish between items; (ii) collaborative filtering (CF), whose method looks at the past interactions of users with various items to make recommendations. These interactions can be anything from buying a product, listening to a song, giving a product a rating, watching a movie, clicking on a news article, and so on; and hybrid (Hybrid), whose idea is to blend the content-based and collaborative filtering methods. This approach aims to provide recommendations with fewer shortcomings than using either method alone, and it's widely adopted by major platforms [Ricci et al. 2022].

A user can interact with an RS in two ways: implicitly and explicitly [Parra and Amatriain 2011, Jawaheer et al. 2010, Moling et al. 2012]. The action is explicit when the user selects or rates an item on the interface [Jawaheer et al. 2014]. On the other hand, the interaction is implicit when the relevance of an item is based on the user history [Nichols 1998, Oard and Kim 1998, Parra et al. 2011, Choi et al. 2012, Go et al. 2010, Schoinas and Tjortjis 2019, Hu et al. 2008, Baltrunas and Amatriain 2009, Kim et al. 2000], such as the number of clicks, number of page visits, the number of times some user played a song, and so on, i.e., the number and time in which the implicit interaction occurs may represent the interest of users [Cao et al. 2019, Yi et al. 2015]. Compared to the explicit interaction, the implicit interaction is much more expressive as hardly users provide explicit feedback [Rocha Silva et al. 2020b, Oh et al. 2008], only 15% of users according to [Yi et al. 2015].

Some works consider time as a crucial factor in defining user preferences [Ginty and Smyth 2002, Kang et al. 2020, Ortiz Viso 2020, Gan and Xiao 2019, Feng et al. 2019, Neelima and Rodda 2016], as it serves as a basis for calculating values of variables used as measures to define preferences, such as time of permanence in certain items, navigation time and the number of actions repetitions over time [Kang et al. 2020]. As time passes, the sequence in which items are clicked can also reflect the course of evolving interests for a specific user [Gan and Xiao 2019]. An SR also needs to consider the evolution of the user based on their needs, not just their personal preferences that are variable over time [Ortiz Viso 2020, Alonso et al. 2009, Chao et al. 2005]. An RS can work on generating logs that mark the frequency and duration of occurrence of user interactions [Feng et al. 2019, Feng and Wei-wei 2018]. The logs are available on the server, browser, or proxy, and scores are assigned, which can be used to calculate the subsequent recommendations [Neelima and Rodda 2016, Dumais et al. 2014].

Other works in the literature propose metrics to measure user interaction with the interface in several dimensions [Feng et al. 2019]. The metrics assess user interaction, and the values measured serve as a parameter to filter and eliminate unnecessary
sary information and recommend what is important [Jianjun 2020, Walek 2017]. The metrics defined are simple, such as permanence time, number of visits, purchases made, the favorite topic, favorite content, and session duration [Crespo et al. 2011]. [Misztal-Radecka and Indurkhya 2020] proposes another set of metrics, and the idea is to build archetypes that serve as a structure in predicting new interactions, influencing the SR [Ma et al. 2020]. In [Crespo et al. 2011], the authors define the following items for this measurement: (a) total interaction time (in minutes); (b) favorite topic (ordered list of topics); (c) favorite content (ordered list of contents); (d) session duration (ordered list of content/duration pairs); (e) comments (number); (f) direct feedback (number of “starts”); (g) repetitive readings (number of iterations); (h) session length/content length.

In addition to the time variable, user interaction actions are influenced by the context in which the user is inserted. The context can constrain selection actions, guiding the user towards more specific items [Mo et al. 2014]. For example, in [Qian Gao 2021], the authors mention that depending on the environment in which they are, the user selects a style of music, or in [Xu et al. 2020], the authors show that depending on the context, the explanatory text of a document recommendation can be better detailed. In the analyzed studies, few considered the user's context during interaction.

It is possible to identify the user interaction that contributes to the enhancement of the interface for using the recommendations, investing in the placement of the items to be chosen, and explaining the recommended items [Xu et al. 2020]. For this, a structure can be defined in the interface/document body, and attributes can be used to understand user interaction by reviewing the text or selecting the item. The feedback of such information begins to influence future review propositions [Yang and Zhang 2020].

User interaction may also influence RSs through the sequential interaction of the user in the selection (or not) of items, with the entire interaction being stored over time. After the interaction is captured, it is modeled with complex relationships for understanding implicit data recommendation [Nguyen et al. 2020]. In this case, the idea is to use the dynamic elements of interaction and time that contribute to the possibility of variation in the result of the recommendations.

When a structure is defined with attributes to measure user interaction in different contexts, and the contexts influence item selection, there is a personalization of the recommendations for which such interactions are mapped [Qian Gao 2021]. In this case, the dynamic elements are context and interaction and can influence the diversity of results in recommendation. Some studies [Meshram and Kaza 2021, Zheng et al. 2017] deepen the specialization of user interaction learning to feedback to the RS and, thus, contribute to the precision of future item propositions.

[Nguyen et al. 2020] presented the Sequential Implicit To Explicit (SITE), which directly models both types, item sequence, and sequential user interactions with an item, to capture user interaction. The SITE method models several types of interactions as action sequences, making possible to represent complex relationships among the user interactions with an item. It is also possible to monitor how user preferences change over time by examining the sequence of items they have interacted with in the past.
Observing how users behave, is useful for making improvements in the system that may contribute to the recommendations' precision through interaction. However, it is necessary to understand that user interaction also suffers various influences of variables internal (desires and needs) and external to the user. In general, user interaction is directly connected to the variations in the decision to choose the recommended item, such as time, context, and the change of interests that impact interaction.

**3. Methodology**

To ensure the relevance of our study, we initially investigated the existing literature to determine if similar reviews were available, finding none. Our review adheres to the systematic methodology described by Kitchenham [Kitchenham and Charters 2007]. We started by formulating a detailed review protocol, defining research questions, methods for addressing them, search strategies, and inclusion/exclusion criteria. Then we run the study through article selection, quality assessment, and analysis of results. The last step, concerns the presentation of the findings. As suggested by [Kitchenham and Charters 2007, Kitchenham 2012], we used the tool StArt<fnref n="1" /> tool [Fabbri et al. 2016a] to conduct the systematic review procedures in a structured way.

**3.1. Research questions**

Three main research questions (RQ) guided our investigation:

**RQ1:** Which are the main features of RSs considered in the studies? This question aims to analyze the features covered by the recommendation algorithm itself. More specifically, we want to know which filtering methods are used to produce recommendations, which mathematical/machine learning models are implemented by the algorithms, and what data is used in the experiments to test the features.

**RQ2:** What kind of action does the user perform when interacting with the system? For that, we start by analyzing which actions allowing users to elicit preference and to provide feedback to the system were available. Then, we also investigated how the user behavior might influence the RSs (for example to improve precision) and how such as user behaviour is specified by the RSs.

**RQ3:** What are the study's current research gaps for future investigation? It aims to point out open questions that can be explored in future research.

**3.2. Search string and research sources**

The review of the literature focused on publications of indexed scientific articles. For crafting the search query to find such as articles, we examined the research questions, research objectives, and relevant prior research. The search string is as follows: ((``RSs'' OR ``recommender systems''') AND (``user interaction'')). We conducted our search across four major computer science databases, ACM, IEEE, Scopus, and Springer, selected due to their comprehensive coverage, relevance, and high impact in the field of computer science research [Cavacini 2015]. The included publications span from 2009 to 2023. The automated searches covered all

<footnote n="1">¹http://lapes.dc.ufscar.br/tools/start_tool/</footnote>
fields available in the research sources since we searched using options like "anywhere" in the fields equivalent to "search within".

### 3.3. Inclusion and exclusion criteria

After performing the search process, we conducted a first selection of articles by reading their titles, abstracts, and keywords, selecting them according to the inclusion/exclusion criteria presented respectively in Tables 1 and 2. Every article was assessed based on its pertinence to our research questions.

<table><caption>Table 1. Inclusion Criteria</caption><thead><tr><td>Criterion</td><td>Description</td></tr></thead><tbody><tr><td>IC1</td><td>Studies that show actions of users interaction with RSs</td></tr><tr><td>IC2</td><td>Studies that treat user interactions with RSs</td></tr><tr><td>IC3</td><td>Studies that consider dynamism in RSs and how this reflects user interaction</td></tr><tr><td>IC4</td><td>Studies that correct RS problems through user interaction</td></tr><tr><td>IC5</td><td>Studies that define RS metrics to assess user interaction</td></tr></tbody></table>

The criteria for inclusion and exclusion demonstrated effectiveness in limiting the studies not aligned with our research goals. We excluded works related to tutorials, posters, and technical reports as they are not systematically peer-reviewed. We also exclude Ph.D. thesis assuming the authors might have published the relevant parts of thesis in the form of scientific articles already available in the literature. We also excluded works not published in scientific journals or conferences. The exclusion criterion EC5, concerning studies with protected or paid access, not available through the Capes agreement.

<table><caption>Table 2. Exclusion Criteria</caption><thead><tr><td>Criterion</td><td>Description</td></tr></thead><tbody><tr><td>EC1</td><td>Title, abstract, or keywords without the search terms</td></tr><tr><td>EC2</td><td>Studies not published in scientific journals or conferences as research papers</td></tr><tr><td>EC3</td><td>Duplicated studies</td></tr><tr><td>EC4</td><td>Studies not in English</td></tr><tr><td>EC5</td><td>Studies with protected or paid access</td></tr><tr><td>EC6</td><td>Studies that do not address user interaction towards RSs</td></tr></tbody></table>

### 3.4. Search and selection procedure

We start by inserting all items of the research protocol in the tool StArt, including the following fields: objective, research questions, search string, keywords, inclusion/exclusion criteria, and the databases. Figure 1 shows the search and selection procedure. Then we run the search query (as defined in Subsection 3.2) in all data sources. For each data source, we have created a ".bib" file containing the list of corresponding studies found. Later, we imported these files into the Start tool. This procedure generated 1050 studies. Then, we applied the remaining exclusion and inclusion criteria, resulting in 24 works. A snowballing process [Wohlin 2014] (consisting of extracting studies related to a start set of papers, using a backward/forward strategy, taking the reference list to identify new
<figure><img src="image_2.png" /><figcaption>Figure 1. Search and selection procedure</figcaption></figure>

works to include, and looking forward to identifying new works based on those papers
citing the paper being examined) was also applied. This process resulted in 208 further
studies, from which 47 duplicated ones were excluded, with 161 remaining. After apply-
ing all exclusion criteria (after Snowballing process), we got two more works, resulting
in a final total of 29 studies that have been considered for deep analysis. From these 29
studies, we got 20 papers published in conferences and 9 articles published in scientific
journals. We used a single backward and forward iteration. No further iterations were
needed as the resulting articles began to repeat.

**3.5. Threats to the validity**

We have taken some actions to eliminate threats to the validity of the presented search and selection process. These actions aim to minimize the barriers that could compromise the conduction of the research and reduce the quality and reliability of the presented results. First, the review followed the protocol defined in [Kitchenham and Charters 2007] and was revised by specialists, guaranteeing that we carried out all steps defined for execut- ing the research. Second, the selected data sources correspond to well-known research platforms containing curated and hosted full-text publications from select publishers. In support of this search, we adopted the Snowballing strategy to reduce any threat of rel- evant studies that have been ignored. Finally, during the entire process of searching and selecting studies, we count on the support of the StArt tool [Fabbri et al. 2016b], allowing the generation of documentation of the whole procedure and enabling the review at any time.

**4. Results and discussions**

Table 3 reports the 29 studies selected for deep analysis. Hereafter, we start by providing
some overall statistics about the studies, then we present a series of graphics and tables
<table><caption>Table 3. List of studies selected for deep analysis of interaction aspects of Recommendation Systems approaches</caption><thead><tr><th>Paper/year</th><th>Filtering</th><th>Machine Learning Models (MLM)</th><th>Domain</th></tr></thead><tbody><tr><td>[Meshram and Kaza 2021]</td><td>CBF</td><td>Partially observable Markov decision process (POMDP)</td><td>E-commerce</td></tr><tr><td>[Nguyen and Cho 2020]</td><td>Hybrid</td><td>Hybrid generative model based on LDA</td><td>Generic</td></tr><tr><td>[Chong and Abeliuk 2019]/2019</td><td>CF</td><td>Pearson correlation and matrix factorization (MF)</td><td>Joke</td></tr><tr><td>[Liu et al. 2020]</td><td>CF</td><td>Latent factor model (LFM)</td><td>Movie</td></tr><tr><td>[Zhou et al. 2021]</td><td>CF</td><td>Filtering matrix with gradient descent; Deep learning (DL) technology;</td><td>Movie</td></tr><tr><td>[Yang and Zhang 2020]</td><td>Hybrid</td><td>ON-LSTM together</td><td>E-commerce</td></tr><tr><td>[Saranya et al. 2020]</td><td>CBF</td><td>Heterogeneous user-item (HUI) network and cosine similarity</td><td>Music, Movie</td></tr><tr><td>[Shibamoto et al. 2019]</td><td>CF</td><td>Not identified</td><td>Photography</td></tr><tr><td>[Xu et al. 2020]</td><td>Hybrid</td><td>Use of a posthoc heuristic to identify the explanations</td><td>Documents</td></tr><tr><td>[Misztal-Radecka and Indurkhya 2020]</td><td>CF</td><td>Density-based spatial hierarchical algorithm, Clustering of Applications with Noise (HD BSCAN).</td><td>Movies and Docs</td></tr><tr><td>[Chang et al. 2021]</td><td>CBF</td><td>Not identified</td><td>E-commerce</td></tr><tr><td>[Jianjun 2020]</td><td>CF</td><td>Collaborative filtering algorithm and Pearson's correlation;</td><td>E-commerce</td></tr><tr><td>[Guo and Li 2020b]</td><td>Hybrid</td><td>K-means and Funk-SVD algorithms and hybrid recommendation algorithm KMFSCF;</td><td>E-commerce</td></tr><tr><td>[Qian Gao 2021]</td><td>CBF</td><td>Use of CA-GNN</td><td>E-commerce</td></tr><tr><td>[Nguyen et al. 2020]</td><td>CBF</td><td>Not identified</td><td>E-commerce</td></tr><tr><td>[Walek 2017]</td><td>CBF</td><td>Not identified</td><td>E-commerce</td></tr><tr><td>[Widiyaningtyas et al. 2021]</td><td>Hybrid</td><td>UPCSim algorithm</td><td>Movie</td></tr><tr><td>[Koren et al. 2009]</td><td>Hybrid</td><td>Matrix to control the time change</td><td>Movie</td></tr><tr><td>[Gao et al. 2023]</td><td>Hybrid</td><td>Static graph neural networks</td><td>E-commerce</td></tr><tr><td>[Zheng et al. 2017]</td><td>Hybrid</td><td>DeepCoNN</td><td>E-commerce</td></tr><tr><td>[Zhuo et al. 2022]</td><td>CF</td><td>Pointwise and the pairwise approaches</td><td>Music, Movie</td></tr><tr><td>[Zhu et al. 2022]</td><td>CF</td><td>Graph neural network (GNN) and a Mixture-of-expert (MoE)</td><td>Movie, Image</td></tr><tr><td>[Xie et al. 2022]</td><td>Hybrid</td><td>Gain-tuning dynamic negative sampling model (GDNS)</td><td>Movie, E-commerce</td></tr><tr><td>[Wang et al. 2022]</td><td>Hybrid</td><td>Heter. Infor. Network (HIN) augmented target policy model</td><td>Movie, E-commerce</td></tr><tr><td>[Jin et al. 2022]</td><td>CF</td><td>TWo-side Interactive NetworkS (TWINS) model</td><td>E-commerce</td></tr><tr><td>[Deepak and Santhanavijayan 2022]</td><td>Hybrid</td><td>Graph-based semantic strategy for query expansion and recommendation;</td><td>Generic</td></tr><tr><td>[Sabitha et al. 2022]</td><td>CBF</td><td>Modified RBFNN</td><td>E-commerce</td></tr><tr><td>[Peng et al. 2019]</td><td>CF</td><td>Deep neural network</td><td>Movie, Book</td></tr><tr><td>[Zheng et al. 2023]</td><td>Hybrid</td><td>AutoML for deep RS</td><td>E-commerce, social media</td></tr></tbody></table>

that illustrate our findings with respect to the research questions presented at section 3.1.

## 4.1. Overview of the selected studies

Figure 2 presents the distribution of studies from 2009 to 2023. As we shall see, the first study mentioning user interaction in RSs was a article published in a IEEE journal ([Koren et al. 2009] dating to 2009. We could not find any other publication on the topic until 2017. However, from 2017 onward, there is an increasing number of publications in the area. IEEE appears as the main data source of studies followed by ACM, but fewer studies can still be found at Springer and Scopus.

## 4.2. How findings answer research questions

Hereafter we present our findings with respect to our research questions.

### 4.2.1. Analyzing RQ1 - the main features of RSs

Table 3 presents a general overview of the studies considering the research question **RQ1**
- Which are the main features of RSs considered in the studies?

The first feature analyzed was the application domain, which have been classified as *generic* when studies were using more than two different domain datasets to perform their experiments. The most commonly application domain was *e-commerce*, followed by *movies*, *music*, *jokes*, *photography*, and *documents*. As shown in column *domain* of
<figure><img src="image_3.png" /><figcaption>Figure 2. Distribution of studies from 2009 to 2023</figcaption></figure>

the Table 3, the proposals tend to be specific to an application domain. Table 4 provides the number of studies per application domain along the years.

The second feature analyzed was the possible influences of filtering methods in the precise construction of RSs or even in the intervention in specifying the system requirements. The column *Filtering* in Table 3 presents the filtering method in each study. Figure 3 shows the distribution among the three methods: *CF* (i.e. *collaborative filtering*), *CBF* (i.e. *content-based filtering*), and *Hybrid* (when both collaborative and content-based filtering are available). Interestingly, the filtering methods are well distributed among the selected works, with a slight increase in the hybrid approach. This homogeneous distribution shows no specific filtering approach addressing user interaction.

Another feature investigated concerns what are the improvements proposed by RSs (e.g. *behaviour prediction*, *rating prediction*, *precision improvement...*). As shown at Figure 4, the majority of studies focus on the enhancement of recommendation algorithms to improve the *precision of the items recommended* by the system to its users. Most of the studies propose a solution that tries (in some way) to indicate in a more precise or personalized way the items for users [Chong and Abeliuk 2019, Liu et al. 2020, Saranya et al. 2020, Shibamoto et al. 2019, Misztal-Radecka and Indurkhya 2020, Jianjun 2020, Guo and Li 2020b, Walek 2017, Widiyaningtyas et al. 2021, Koren et al. 2009]. These works use some mathematical or machine learning models as the core of their recommended system proposal.

Table 3 presents the mathematical/machine learning models (MLM) implemented by RSs. A mathematical model uses high-level information, such as attributes, to perform calculations and tasks, and these attributes extracted from user interaction actions might also influence the construction of such MLMs. A mathematical model may represent all business domains [Diskin 2000]. Thus, as RSs increasingly use MLMs, it is essential to have an alignment between user interaction and the model in these systems, which reflects
<figure><img src="image_4.png" /><figcaption>Figure 3. Filtering methods</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4. Improvements proposed by RSs</figcaption></figure>
<table><caption>Table 4. Classification of studies by application domain</caption><thead><tr><th>Year</th><th>documents</th><th>e-Commerce</th><th>generic</th><th>jokes</th><th>movies</th><th>music</th><th>photography</th><th>book</th></tr></thead><tbody><tr><td>2009</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>-</td><td>-</td><td>-</td></tr><tr><td>2017</td><td>-</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>2019</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>-</td><td>1</td><td>1</td></tr><tr><td>2020</td><td>2</td><td>5</td><td>-</td><td>-</td><td>2</td><td>1</td><td>-</td><td>1</td></tr><tr><td>2021</td><td>-</td><td>2</td><td>-</td><td>-</td><td>2</td><td>-</td><td>-</td><td>-</td></tr><tr><td>2022</td><td>-</td><td>4</td><td>1</td><td>-</td><td>4</td><td>1</td><td>1</td><td>-</td></tr><tr><td>2023</td><td>-</td><td>2</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>-</td></tr></tbody></table>

the business vision and the models that will use such methods. Hereafter we present the main findings concerning the feature MLMs:

* The proposal described in [Meshram and Kaza 2021] shows a Markov model as an approximation of the dynamic interaction of users showing their interests. In the Markov model, a state describes the intensity level of preferences, whereas a higher state means a higher level of interest for an item. The user interaction for an item is determined by the transition dynamics for that item. In [Nguyen and Cho 2020], the proposal is a hybrid generative model that can predict users by analyzing repetitive interaction and latent group preference components to discover new interactions or exogenous effects. The work described in [Chong and Abeliuk 2019] analyzes the impact of RSs and quantifies the inequalities resulting from them to visualize and compare different biases. The intuition behind this is to illustrate that frequently recommended popular items create a bias that impacts and modifies user preferences and interactions. In [Liu et al. 2020, Zheng et al. 2023], an innovative enhanced LFM-based framework is introduced, using historical user interactions, check-in data, and user social connections to enhance recommendation accuracy. In [Gao et al. 2023], the proposed is based on neural network and graphs for implicit interaction learning.

* In [Chang et al. 2021], the authors introduce a graph neural network model named SURGE (SeqUential Recommendation with Graph neural nEworks). The main idea addresses two issues: (i) the user interactions in their historical sequences, which often can not sufficiently reflect their actual preferences; and (ii) the changing in users' dynamic preferences, which is challenging to capture in their historical sequences. In [Chang et al. 2021], the authors introduce a graph neural network model named SURGE (SeqUential Recommendation with Graph neural nEworks). The main idea addresses two issues: (i) the user interactions in their historical sequences, which often can not sufficiently reflect their actual preferences; and (ii) the changing in users' dynamic preferences, which is challenging to capture in their historical sequences. The work discusses how important it is to consider the dynamics of user interaction in RSs (published in 2021, it shows the current relevance of the topic). Another work that presented studies on sequential user interaction was described in [Nguyen et al. 2020], where the authors present a hybrid generative model capable of predicting user interactions by considering multiple factors.
<figure><img src="image_6.png" /><figcaption>Figure 5. Datasets used in the experiments</figcaption></figure>

* The user interaction is also considered in works that focus on rating prediction [Zhou et al. 2021, Yang and Zhang 2020]. In [Zhou et al. 2021], both variance and the Pearson correlation coefficient (PCC) are employed to assess the user's focus on the feature dimensions and to derive the weight vector for interactive features. In [Yang and Zhang 2020], the authors proposed to use two models, one for users and another for projects. User reviews and item reviews are provided as inputs to the models. The output of the two models is used in the graph convolutional neural network layer for heterogeneous information. The corresponding rating is generated as an output through a shared layer.

The feature *dataset used to carry out the experiments* (shown at Figure 5) allowed us to identify what data the users provide as input to the system. As we shall see at Figure 6, a deeper analysis of those dataset shows that most of works (20%) use the *ratings provided by users* (such as ratings of products, movies, restaurants, food, and so on) [Chong and Abeliuk 2019, Liu et al. 2020, Zhou et al. 2021, Misztal-Radecka and Indurkhya 2020, Qian Gao 2021], as well as *user-supplied reviews* [Koren et al. 2009, Zheng et al. 2017, Yang and Zhang 2020, Saranya et al. 2020]. *User data* (such as user id, date and time of login, genre, age, and others) are considered in some works [Widiyaningtyas et al. 2021, Koren et al. 2009, Zheng et al. 2017], as well as any *user interaction* with the RSs [Nguyen et al. 2020, Jianjun 2020, Chang et al. 2021] (such as clicks, likes, login, scroll up and down). More often, there are no detailed descriptions about which attributes have been considered in the works, so that some of attributes were inferred by considering the nature of the datasets employed in the studies.

The last feature considered in the study was the overall output
<figure><img src="image_7.png" /><figcaption>Figure 6. Input data</figcaption></figure>

generated by the algorithms. This point is not trivial to identify because most of the work focuses on developing solutions that aim to improve the accuracy of RSs, such as discussed in [Meshram and Kaza 2021, Chong and Abeliuk 2019, Liu et al. 2020, Saranya et al. 2020, Shibamoto et al. 2019, Misztal-Radecka and Indurkhya 2020, Chang et al. 2021, Jianjun 2020, Guo and Li 2020b, Walek 2017, Widiyaningtyas et al. 2021, Koren et al. 2009, Zheng et al. 2017]. For these specific cases, we have found that most of the output can be classified as a "Ranking" of recommended products. The selected works also focus on user interaction prediction [Nguyen and Cho 2020, Qian Gao 2021] and rating prediction [Zhou et al. 2021, Yang and Zhang 2020]. In all cases, the output involves a similarity or prediction score value. Other works focus on interaction modeling [Nguyen et al. 2020] and create user history based on log data of the system [Xu et al. 2020].

### 4.2.2. Analyzing RQ2 - User actions

Table 5 shows the features that have been analyzed to answer the research question **RQ2** - *What actions do users take during their interactions when interacting with the system?* For that, we tried to identify what kind of user interactions are taken into account by RSs.

According to [Jugovac and Jannach 2017b], there are two phases in which a user interacts with the RS: (i) elicitation of preferences when the system does not know the user's preferences and acts to achieve them; and (ii) presentation of the results, or *feed-back*, when the RS makes the result available, and the user needs to make an evaluation.
<figure><img src="image_8.png" /><figcaption>Figure 7. User Actions</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 8. Actions to elicit preferences</figcaption></figure>

Actions related to eliciting preferences can be: select item, *check-in*, etc. *Feedback* actions can be: rate item, like, comment, etc. To answer the research question RQ2, we analyzed the dataset experiments to identify the attributes used in each experimental dataset. The results in terms of how each work considers the user actions and elicit preferences are presented in the column *User Actions* in Table 5 and their distribution is illustrated at Figure 7. As we shall see that most work, 45%, consider *preference elicitation (PE)*, while 30% consider the *user feedback* while interacting with the resulting recommendation, and 25% consider *both* types of action.

Figure 8 illustrates common actions used as elicitation user preferences, which can be *implicitly* or *explicitly*. The user actions enabling to elicit preferences can vary greatly, but very often lie in one of the following categories: *item select*, *shopping* (which can be further classified as *register*, *visualization*, *check-in*), *geolocation* or *play item*. The third column in Table 5 presents the user interactions explicitly mentioned in the selected works. Identifying this type of interaction is not trivial since the works generally refer to “user interaction” given some examples, such as clicking a button or link, playing music/video, or adding an item to a shopping cart.
<figure><img src="image_10.png" /><figcaption>Figure 9. Types of User Feedback</figcaption></figure>

<table><caption>Table 5. User Actions in a RS</caption><thead><tr><th>Work/Year</th><th>User actions</th><th>Preference elicitation</th><th>User feedback</th><th>Specification</th></tr></thead><tbody><tr><td>[Meshram and Kaza 2021]</td><td>PE</td><td>Items selection</td><td>NA</td><td>Ad-hoc description</td></tr><tr><td>[Nguyen and Cho 2020]</td><td>PE</td><td>Play music/videos, login in Communities</td><td>NA</td><td>NA</td></tr><tr><td>[Chong and Abeliuk 2019]</td><td>Feedback</td><td>NA</td><td>Items evaluations</td><td>NA</td></tr><tr><td>[Liu et al. 2020]</td><td>Feedback</td><td>NA</td><td>Item evaluation and classification</td><td>NA</td></tr><tr><td>[Zhou et al. 2021]</td><td>Feedback</td><td>NA</td><td>Items classification</td><td>NA</td></tr><tr><td>[Yang and Zhang 2020]</td><td>Feedback</td><td>NA</td><td>User comments</td><td>NA</td></tr><tr><td>[Saranya et al. 2020]</td><td>Feedback</td><td>NA</td><td>Items classification</td><td>Ad-hoc description e ad-hoc design</td></tr><tr><td>[Shibamoto et al. 2019]</td><td>PE</td><td>Geolocation</td><td>NA</td><td>NA</td></tr><tr><td>[Xu et al. 2020]</td><td>PE</td><td>Access and read item</td><td>NA</td><td>NA</td></tr><tr><td>[Misztal-Radecka and Indurkhya 2020]</td><td>PE</td><td>Select item category</td><td>NA</td><td>NA</td></tr><tr><td>[Chang et al. 2021]</td><td>Both</td><td>Item selection and following</td><td>Like</td><td>NA</td></tr><tr><td>[Jianjun 2020]</td><td>PE</td><td>Visit page</td><td>NA</td><td>Ad-hoc design</td></tr><tr><td>[Guo and Li 2020b]</td><td>PE</td><td>Item selection and visualization</td><td>NA</td><td>NA</td></tr><tr><td>[Qian Gao 2021]</td><td>PE</td><td>Item selection</td><td>NA</td><td>NA</td></tr><tr><td>[Nguyen et al. 2020]</td><td>Both</td><td>Item click and visualization</td><td>Item evaluation</td><td>NA</td></tr><tr><td>[Walek 2017]</td><td>PE</td><td>Item selection, visualization and purchase</td><td>NA</td><td>NA</td></tr><tr><td>[Widianingtyas et al. 2021]</td><td>Both</td><td>Item selection</td><td>Item classification</td><td>NA</td></tr><tr><td>[Koren et al. 2009]/2009</td><td>Both</td><td>Item selection</td><td>Comments</td><td>NA</td></tr><tr><td>[Gao et al. 2023]</td><td>Both</td><td>Adding-to-cart or purchasing</td><td>Item classification</td><td>NA</td></tr><tr><td>[Zheng et al. 2017]</td><td>Both</td><td>Items selection</td><td>Item evaluation</td><td>NA</td></tr><tr><td>[Zhuo et al. 2022]</td><td>PE</td><td>Users Likes</td><td>NA</td><td>NA</td></tr><tr><td>[Zhu et al. 2022]</td><td>Feedback</td><td>NA</td><td>Observed Interaction</td><td>NA</td></tr><tr><td>[Xie et al. 2022]</td><td>PE</td><td>NA</td><td>NA</td><td>NA</td></tr><tr><td>[Wang et al. 2022]</td><td>Feedback</td><td>NA</td><td>Items evaluation</td><td>NA</td></tr><tr><td>[Jin et al. 2022]</td><td>Both</td><td>Item selection</td><td>Item evaluation</td><td>NA</td></tr><tr><td>[Deepak and Santhanavijayan 2022]/2022</td><td>PE</td><td>NA</td><td>NA</td><td>NA</td></tr><tr><td>[Sabitha et al. 2022]</td><td>Both</td><td>Item selection</td><td>Item evaluation, reviews</td><td>NA</td></tr><tr><td>[Peng et al. 2019]</td><td>Feedback</td><td>NA</td><td>Item evaluation</td><td>NA</td></tr><tr><td>[Zheng et al. 2023]</td><td>Both</td><td>Click</td><td>Item classification</td><td>NA</td></tr></tbody></table>

We also analyzed how the user behaves concerning recommended items. We sought to identify the actions indicated in the works that, as comprehensively as possible, consider user interactions on the recommended items. In this case, the most performed interactions are item classification and evaluation and, in some cases, the possibility to comment. The fourth column in Table 5 and Figure 9 present the user interactions mentioned in the selected works. As in the previous question, identifying this type of interaction was also not trivial, since the works generally refer to “user interaction” given some examples, such as ratings, item evaluation, and user comments.

We also tried to identify how user interaction may influence the RS to improve the precision in recommending. We aim to determine how users' actions can increase the accuracy of recommended items. In this sense, we only analyzed the works whose final result focused on results ranking, which represented 55% of the studied works. The
other works were not analyzed since they do not focus on improving precision but on user modeling, prediction of ratings or user interaction. In general, interactions with users always impact improving recommendation accuracy. Any action the user takes, listed in the second column of Table 5, can positively enhance accuracy. For example, the user's action in selecting an item through a click indicates interest in that item, which can be stored as an indication of preference. In this way, any user action is positive for accuracy improvement, including actions demonstrating their lack of interest.

### 4.2.3. Analyzing RQ3 - Research gaps

The identification of open questions in our study was based on a critical analysis of the literature reviewed. We did not use a specific method like thematic analysis; rather, we examined the limitations and future research directions mentioned across the studies in our corpus. This approach allowed us to identify recurring gaps and under-explored areas relevant to user interaction in recommendation systems. We suggest that this method provides a valid perspective, focusing on gaps highlighted by leading studies in the field. Our analysis identifies three open questions that can be studied more deeply when considering user interaction in recommendation systems. In this final section, we summarize these issues while making recommendations for future research.

**Conceptual modeling of user interaction in recommendation systems.** A conceptual model helps us understand the more abstract intent of an application. The better the conceptual model relates to users' mental conceptual models, the easier it is to use it to explain what they intend to do with the application. Therefore, constructing a conceptual model that reflects all possible user interactions, whether related to the elicitation of preferences or feedback on results, would be very interesting to help develop recommendation systems that better understand user needs. In [Jawaheer et al. 2014], the authors present as future directions the understanding of the reasons users rate items is essential in improving user feedback. A conceptual modeling design goes further and models all kinds of interactions.

**Specification of user requirements for interaction in recommendation systems.** A specification is a document used in software engineering that specifies what the user expects the software to be able to do. As recommendation systems have similar standards worldwide, it would be interesting to formalize the users' needs to define a standard for specifying the users' needs [Kumar et al. 2010].

**Specification of dynamic user behavior patterns.** In our perception, it is clear that there are dynamic behavior patterns in users who use a recommendation system. However, in our investigation, we have yet to identify anything that can specify these behavior patterns in the literature that contributes to pre-defining this for the system before its implementation. Not considering such patterns makes it impossible to reproduce this behavior in systems, not even to plan how they can work [Chen et al. 2020].

**Mapping and specification of user dynamic behavior.** As mentioned earlier, the examination of the contextual aspects of user feedback holds potential for enhancing performance. [Jawaheer et al. 2014]. In this scenario, the system could, for example,
adapt more effectively to the dynamic behavior of the user, which can be identified in some situations that consider the user behavior, the external context, or even periods of time. It could be interesting for recommendation systems to consider such variables so that the user may have a friendlier experience interacting with the system.

# 5. Discussion

In this section, we discuss the implications of the synthesized findings for the first two research questions (RQ1 and RQ2).

## 5.1. Key Features of Recommendation Systems (RQ1)

The studies showcase a rich diversity of features across domains, filtering methods, and recommendation models, each designed to enhance recommendation accuracy and user interaction. A primary characteristic observed is the domain specificity of the datasets used. Many proposals adopt domain-specific features to achieve more accurate recommendations, with e-commerce, movies, music, and other entertainment sectors most frequently represented. This trend reflects the popularity of these domains and the availability of large, rich datasets. However, our findings indicate that studies employing generic datasets across multiple domains may offer more broadly applicable insights, allowing for generalization and adaptability across RS applications.

Filtering methods (collaborative, content-based, and hybrid) are fairly evenly represented, with a notable increase in hybrid approaches. The hybrid approach combines the strengths of both collaborative and content-based filtering, making it especially valuable for applications requiring diverse data sources to capture user preferences. This balance suggests a need for future work to assess how these filtering methods could be tailored to optimize user interaction by adapting dynamically to user inputs.

Recent advances in recommendation models include the application of Markov models, neural networks, and hybrid generative models. Many studies emphasize the use of sequential and contextual data to improve recommendation accuracy. These models account for users' dynamic preferences by leveraging patterns in historical interactions, thereby personalizing recommendations based on implicit cues such as clicks, time spent, and scrolling. Such approaches underscore the importance of capturing evolving user preferences and suggest the potential for enhancing personalization through real-time data analysis and feedback loops.

Finally, the user input and output data in these studies reveal that, while explicit feedback like ratings and reviews is widely used, there is an increasing reliance on implicit behavioral data to enrich interaction models. The integration of implicit signals allows RSs to capture subtler aspects of user behavior, ultimately facilitating more accurate and responsive systems. This focus on user-centric features implies that future research should consider more granular data collection methods and sophisticated processing techniques, particularly in contexts beyond popular domains, to explore how RS features might impact user interaction quality across a broader spectrum of applications.
## 5.2. User Interaction Actions (RQ2)

The studies reviewed provide insights into two primary phases of user interaction in recommendation systems (RSs): preference elicitation and result presentation, as described by [Jugovac and Jannach 2017b]. Preference elicitation involves actions users take to express initial preferences, while result presentation encompasses user feedback on the recommended items. Our findings indicate that user interactions across these phases are diverse but can broadly be classified into explicit actions (e.g., selecting items, providing ratings) and implicit cues (e.g., click behaviors, time spent on a page), which are captured in Table 6.

Preference elicitation actions, such as item selection or check-ins, are foundational for systems to establish initial user profiles. Studies show that 45% of the analyzed works focus on these actions, underscoring their significance in generating accurate recommendations. For example, each item selected, checked in, or even viewed by a user contributes to a nuanced preference profile, allowing RSs to make more informed recommendations.

In the result presentation phase, which 30% of the reviewed studies focus on, actions such as rating, liking, and commenting on items enable systems to refine recommendations based on explicit user feedback. These interactions not only help validate recommendations but also allow RSs to continuously adjust to evolving user preferences. Table 6 illustrates the common feedback actions, highlighting how these systems leverage user evaluations to prioritize recommended items and improve ranking accuracy.

User interactions also impact the RS's precision, particularly in ranking results. Studies focused on accuracy improvement, representing 55% of the reviewed works, emphasize that any user action, whether explicit or implicit, serves as valuable feedback for enhancing recommendation accuracy. Actions like item clicks or shopping cart additions signal user interest, which can significantly refine the recommendation algorithm's predictions by reinforcing items that align with the user's demonstrated preferences. Furthermore, even interactions indicating disinterest (e.g., ignoring or skipping recommendations) can help RSs reduce the level of priority of certain items, ensuring future recommendations better match user intent.

In summary, user interactions, through both preference elicitation and feedback, play a crucial role in shaping and refining RS functionality. Understanding these interactions as a continuous feedback loop enables RSs to better align with user expectations, providing more precise and personalized recommendations. This analysis points to the value of future research on optimizing interaction-based feedback mechanisms, potentially enhancing RS adaptability across diverse user behaviors and domains.

## 6. Conclusion

Analyzing user behavior on recommended items is essential for developing more accurate and personalized recommendation systems. It is a critical issue to design more interactive recommendations [Jugovac and Jannach 2017a] and must be deeply investigated. The objective of this article was to present how works that propose solutions to traditional problems in recommendation systems consider user interaction. We developed a qualitative research study using a literature systematic review, analyzing journal articles,
professional publications, and conference proceedings papers. The initial search process, after the snowballing, resulted in 208 studies, from which 47 duplicated ones were excluded, with 161 remaining. After applying all exclusion criteria, we got 29 studies that have been considered for deep analysis. From these 29 studies, 20 papers were published in conferences, and 9 articles were published in scientific journals.

This systematic review provides a comprehensive analysis of user interaction in recommendation systems, highlighting several key takeaways. First, it emphasizes the importance of understanding user preferences and actions during interactions, which significantly influence the effectiveness and personalization of recommendation systems. By categorizing user interactions into preference elicitation and feedback, the study underscores the dual phases of user engagement that shape recommendation outcomes. Moreover, it identifies prevalent domains and filtering methods used in current research, revealing a trend towards domain-specific approaches that enhance recommendation accuracy. The findings also point to critical gaps in the literature, particularly in the integration of user dynamics into recommendation algorithms, suggesting that future research should focus on developing more adaptive systems that respond to evolving user preferences. Ultimately, this review not only synthesizes existing knowledge but also lays the groundwork for future investigations into improving user interaction within recommendation systems, fostering advancements that can lead to more personalized and effective user experiences.

## References

[Adomavicius and Tuzhilin 2005] Adomavicius, G. and Tuzhilin, A. (2005). Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. *IEEE transactions on knowledge and data engineering*, 17(6):734–749.

[Adomavicius and Tuzhilin 2010] Adomavicius, G. and Tuzhilin, A. (2010). Context-aware recommender systems. In *Recommender Syst. handbook*, pages 217–253. Springer.

[Adomavicius and Tuzhilin 2015] Adomavicius, G. and Tuzhilin, A. (2015). *Context-Aware Recommender Systems*, pages 191–226. Springer.

[Aggarwal and Aggarwal 2016] Aggarwal, C. C. and Aggarwal, C. C. (2016). *Recommender systems: The textbook*. Springer.

[Albanese et al. 2004] Albanese, M., Picariello, A., Sansone, C., and Sansone, L. (2004). Web personalization based on static information and dynamic user behavior. In *Proc. of the 6th Annual ACM International Workshop on Web Information and Data Management*, page 80–87. ACM.

[Alonso et al. 2009] Alonso, S., Cabrerizo, F. J., Chiclana, F., Herrera, F., and Herrera-Viedma, E. (2009). Group decision making with incomplete fuzzy linguistic preference relations. *International Journal of Intelligent Systems*, 24(2):201–222.

[Baeza-Yates et al. 2005] Baeza-Yates, R., Hurtado, C., Mendoza, M., and Dupret, G. (2005). Modeling user search behavior. In *Third Latin American Web Congress (LA-WEB'2005)*, pages 10–pp. IEEE.
[Baltrunas and Amatriain 2009] Baltrunas, L. and Amatriain, X. (2009). Towards time-dependant recommendation based on implicit feedback. In *Workshop on context-aware recommender systems (CARS’09)*, pages 25–30. ACM.

[Bobadilla et al. 2013] Bobadilla, J., Ortega, F., Hernando, A., and Gutiérrez, A. (2013). Recommender systems survey. *Knowledge-Based Systems*, 46:109–132.

[Cao et al. 2019] Cao, Z., Qiao, X., Jiang, S., and Zhang, X. (2019). An efficient knowledge-graph-based web service recommendation algorithm. *Symmetry*, 11(3).

[Carrer-Neto et al. 2012] Carrer-Neto, W., Hernández-Alcaraz, M. L., Valencia-García, R., and García-Sánchez, F. (2012). Social knowledge-based recommender system. application to the movies domain. *Expert Systems with applications*, 39(12):10990–11000.

[Cavacini 2015] Cavacini, A. (2015). What is the best database for computer science journal articles? *Scientometrics*, 102(3):2059–2071.

[Chang et al. 2021] Chang, J., Gao, C., Zheng, Y., Hui, Y., Niu, Y., Song, Y., Jin, D., and Li, Y. (2021). Sequential recommendation with graph neural networks. In *Proc. of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval*, page 378–387. ACM.

[Chao et al. 2005] Chao, D. L., Balthrop, J., and Forrest, S. (2005). Adaptive radio: achieving consensus using negative preferences. In *Proc. of the 2005 ACM International Conference on Supporting Group Work*, pages 120–123. ACM.

[Chen et al. 2020] Chen, S., Andrienko, N., Andrienko, G., Adilova, L., Barlet, J., Kindermann, J., Nguyen, P., Thonnard, O., and Turkay, C. (2020). LDA ensembles for interactive exploration and categorization of behaviors. *IEEE Transac. on Visualization and Comput. Graphics*, 26(9):2775–2792.

[Choi et al. 2012] Choi, K., Yoo, D., Kim, G., and Suh, Y. (2012). A hybrid online-product recommendation system: Combining implicit rating-based collaborative filtering and sequential pattern analysis. *electronic commerce research and applications*, 11(4):309–317.

[Chong and Abeliuk 2019] Chong, S. and Abeliuk, A. (2019). Quantifying the effects of recommendation systems. In *2019 IEEE International Conference on Big Data (Big Data)*, pages 3008–3015.

[Christensen and Schiaffino 2011] Christensen, I. A. and Schiaffino, S. (2011). Entertainment recommender systems for group of users. *Expert systems with applications*, 38(11):14127–14135.

[Crespo et al. 2011] Crespo, R. G., Martínez, O. S., Lovelle, J. M. C., García-Bustelo, B. C. P., Gayo, J. E. L., and de Pablos, P. O. (2011). Recommendation system based on user interaction data applied to intelligent electronic books. *Computers in Human Behavior*, 27(4):1445–1449.

[Dao et al. 2012] Dao, T. H., Jeong, S. R., and Ahn, H. (2012). A novel recommendation model of location-based advertising: Context-aware collaborative filtering using ga approach. *Expert Systems with Applications*, 39(3):3731–3739.
[Deepak and Santhanavijayan 2022] Deepak, G. and Santhanavijayan, A. (2022). Uqscm-rfd: A query-knowledge interfacing approach for diversified query recommendation in semantic search based on river flow dynamics and dynamic user interaction. *Neural Comput. Appl.*, 34(1):651-675.

[Delone and McLean 1992] Delone, W. and McLean, E. (1992). Information systems success: The quest for the dependent variable. *Information Systems Research*, 3:60-95.

[Diaper and Stanton 2003] Diaper, D. and Stanton, N. (2003). *The Handbook of Task Analysis for Human-Computer Interaction*, page 568. Taylor Francis Group.

[Diskin 2000] Diskin, Z. (2000). On mathematical foundations for business modeling. In *Proc. 37th International Conference on Technology of Object-Oriented Languages and Systems. TOOLS-Pacific 2000*, pages 182-187.

[Dumais et al. 2014] Dumais, S., Jeffries, R., Russell, D. M., Tang, D., and Teevan, J. (2014). Understanding user behavior through log data and analysis. *Ways of Knowing in HCI*, pages 349-372.

[Fabbri et al. 2016a] Fabbri, S., Silva, C., Hernandes, E., Octaviano, F., Di Thommazo, A., and Belgamo, A. (2016a). Improvements in the start tool to better support the systematic review process. EASE '16, New York, NY, USA. Association for Computing Machinery.

[Fabbri et al. 2016b] Fabbri, S., Silva, C., Hernandes, E., Octaviano, F., Di Thommazo, A., and Belgamo, A. (2016b). Improvements in the start tool to better support the systematic review process. In *Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering. ACM*.

[Feng and Wei-wei 2018] Feng, L. and Wei-wei, G. (2018). Retracted: Recommendation algorithm based on tag time weighting. In *2018 3rd International Conference on Smart City and Systems Engineering (ICSCSE)*, pages 755-758.

[Feng et al. 2019] Feng, M., Peck, E., and Harrison, L. (2019). Patterns and pace: Quantifying diverse exploration behavior with visualizations on the web. *IEEE Transactions on Visualization and Computer Graphics*, 25(1):501-511.

[Gan and Xiao 2019] Gan, M. and Xiao, K. (2019). R-rnn: Extracting user recent behavior sequence for click-through rate prediction. *IEEE Access*, 7:111767-111777.

[Gao et al. 2023] Gao, C., Zheng, Y., Li, N., Li, Y., Qin, Y., Piao, J., Quan, Y., Chang, J., Jin, D., He, X., and Li, Y. (2023). A survey of graph neural networks for recommender systems: Challenges, methods, and directions. *ACM Trans. Recomm. Syst.*, 1(1).

[Ginty and Smyth 2002] Ginty, L. M. and Smyth, B. (2002). Evaluating preference-based feedback in recommender systems. In *Irish Conference on Artificial Intelligence and Cognitive Science*, pages 209-214. Springer.

[Go et al. 2010] Go, G., Yang, J., Park, H., and Han, S. (2010). Using online media sharing behavior as implicit feedback for collaborative filtering. In *2010 IEEE Second International Conference on Social Computing*, pages 439-445. IEEE.
[Guo and Li 2020a] Guo, S. and Li, C. (2020a). Hybrid recommendation algorithm based on user behavior. In *2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC)*, volume 9, pages 2242–2246. IEEE.

[Guo and Li 2020b] Guo, S. and Li, C. (2020b). Hybrid recommendation algorithm based on user behavior. In *IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC)*, volume 9, pages 2242–2246.

[Hebrado et al. 2011] Hebrado, J. L., Lee, H. J., and Choi, J. (2011). The role of transparency and feedback on the behavioral intention to reuse a recommender system. In *CONFIRM 2011 PROCEEDINGS*. AIS eLibrary.

[Hu et al. 2008] Hu, Y., Koren, Y., and Volinsky, C. (2008). Collaborative filtering for implicit feedback datasets. In *2008 Eighth IEEE international conference on data mining*, pages 263–272.

[Hwangbo et al. 2018] Hwangbo, H., Kim, Y. S., and Cha, K. J. (2018). Recommendation system development for fashion retail e-commerce. *Electronic Commerce Research and Applications*, 28:94–101.

[ISO 2019] ISO (2019). ISO 9241-210:2019, ergonomics of human-system interaction part 210: Human-centred design for interactive systems.

[Jannach et al. 2011] Jannach, D., Zanker, M., Felfernig, A., and Friedrich, G. (2011). An introduction to recommender systems. *New York: Cambridge*, 10:1941904.

[Jawaheer et al. 2010] Jawaheer, G., Szomszor, M., and Kostkova, P. (2010). Comparison of implicit and explicit feedback from an online music recommendation service. In *Proc. of the 1st international workshop on information heterogeneity and fusion in recommender systems*, pages 47–51. ACM.

[Jawaheer et al. 2014] Jawaheer, G., Weller, P., and Kostkova, P. (2014). Modeling user preferences in recommender systems: A classification framework for explicit and implicit user feedback. *ACM Trans. Interact. Intell. Syst.*, 4(2):47–51.

[Jianjun 2020] Jianjun, M. (2020). Research on collaborative filtering recommendation algorithm based on user behavior characteristics. In *2020 International Conference on Big Data Artificial Intelligence Software Engineering (ICBASE)*, pages 425–428.

[Jin et al. 2022] Jin, J., Chen, X., Chen, Y., Zhang, W., Rui, R., Jiang, Z., Su, Z., and Yu, Y. (2022). Who to watch next: Two-side interactive networks for live broadcast recommendation. In *Proc. of the ACM Web Conference 2022*, page 3206–3216. ACM.

[Jugovac and Jannach 2017a] Jugovac, M. and Jannach, D. (2017a). Interacting with recommenders—overview and research directions. *ACM Trans. Interact. Intell. Syst.*, 7(3).

[Jugovac and Jannach 2017b] Jugovac, M. and Jannach, D. (2017b). Interacting with recommenders—overview and research directions. *ACM Trans. Interact. Intell. Syst.*, 7(3).

[Kang et al. 2020] Kang, S., Jeong, C., and Chung, K. (2020). Tree-based real-time advertisement recommendation system in online broadcasting. *IEEE Access*, 8:192693–192702.
[Kim et al. 2000] Kim, J., Oard, D. W., and Romanik, K. (2000). User modeling for information access based on implicit feedback. *University Of Maryland*.

[Kitchenham 2012] Kitchenham, B. A. (2012). Systematic review in software engineering: Where we are and where we should be going. In *Proceedings of the 2nd international workshop on Evidential assessment of..., 2012*, page 1–2. ACM.

[Kitchenham and Charters 2007] Kitchenham, B. A. and Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report.

[Koren et al. 2009] Koren, Y., Bell, R., and Volinsky, C. (2009). Matrix factorization techniques for recommender systems. *Computer*, 42(8):30–37.

[Kumar et al. 2010] Kumar, M., Ajmeri, N., and Ghaisas, S. (2010). Towards knowledge assisted agile requirements evolution. In *Proceedings of the 2nd International Workshop on Recommendation Systems for Software Engineering*, page 16–20. ACM.

[Liu et al. 2010] Liu, N. N., Xiang, E. W., Zhao, M., and Yang, Q. (2010). Unifying explicit and implicit feedback for collaborative filtering. In *Proc. of the 19th ACM international conference on Information and knowledge management*, pages 1445–1448. ACM.

[Liu et al. 2020] Liu, T., He, Z., and Wang, P. (2020). Sorrrs: Social recommendation incorporating rating similarity and user relationships analysis. In *2020 7th International Conference on Information, Cybernetics, and Computational Social Systems (ICCSS)*, pages 118–123.

[Ma et al. 2020] Ma, P., Gao, Q., et al. (2020). Eeg signal and feature interaction modeling-based eye behavior prediction research. *Computational and Mathematical Methods in Medicine*, 2020.

[Meshram and Kaza 2021] Meshram, R. and Kaza, K. (2021). Monte carlo rollout policy for recommendation systems with dynamic user behavior. In *2021 International Conference on Communication Systems NETworkS (COMSNETS)*, pages 86–89.

[Misztal-Radecka and Indurkhya 2020] Misztal-Radecka, J. and Indurkhya, B. (2020). Persona prototypes for improving the qualitative evaluation of recommendation systems. In *Adjunct Publication of the 28th ACM Conf. on User Modeling, Adaptation and Personalization*, page 206–212. ACM.

[Mo et al. 2014] Mo, Y., Chen, J., Xie, X., Luo, C., and Yang, L. T. (2014). Cloud-based mobile multimedia recommendation system with user behavior information. *IEEE Systems Journal*, 8(1):184–193.

[Moling et al. 2012] Moling, O., Baltrunas, L., and Ricci, F. (2012). Optimal radio channel recommendations with explicit and implicit feedback. In *Proc. of the sixth ACM conference on Recommender systems*, pages 75–82. ACM.

[Neelima and Rodda 2016] Neelima, G. and Rodda, S. (2016). Predicting user behavior through sessions using the web log mining. In *2016 International Conference on Advances in Human Machine Interaction (HMI)*, pages 1–5.
[Nguyen and Cho 2020] Nguyen, M.-D. and Cho, Y.-S. (2020). A hybrid generative model for online user behavior prediction. *IEEE Access*, 8:3761–3771.

[Nguyen et al. 2020] Nguyen, T., Ngo Van, L., and Than, K. (2020). Modeling the sequential behaviors of online users in recommender systems. In *Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications II*, volume 11413, pages 570–579. SPIE.

[Nichols 1998] Nichols, D. (1998). Implicit rating and filtering. In *Proceedings of the 5th DELOS Workshop on Filtering and Collaborative Filtering*. ERCIM.

[Oard and Kim 1998] Oard, D. W. and Kim, J. (1998). Implicit feedback for recommender systems. *University Libraries Of South Carolina*.

[Oard and Kim 2001] Oard, D. W. and Kim, J. (2001). Modeling information content using observable behavior. *Journal of the Association for Information Science and Technology*.

[Oh et al. 2008] Oh, J., Lee, S., and Lee, E. (2008). A user modeling using implicit feedback for effective recommender system. In *2008 International Conference on Convergence and Hybrid Information Technology*, pages 155–158.

[Ortiz Viso 2020] Ortiz Viso, B. (2020). Evolutionary approach in recommendation systems for complex structured objects. In *Fourteenth ACM Conference on Recommender Systems*, page 776–781. ACM.

[Parra and Amatriain 2011] Parra, D. and Amatriain, X. (2011). Walk the talk: Analyzing the relation between implicit and explicit feedback for preference elicitation. In *Proc. of User Modeling, Adaption and Personalization: 19th International Conference, UMAP 2011, Girona, Spain.*, pages 255–268. Springer.

[Parra et al. 2011] Parra, D., Karatzoglou, A., Amatriain, X., Yavuz, I., et al. (2011). Implicit feedback recommendation via implicit-to-explicit ordinal logistic regression mapping. *Proc. of the CARS-2011*, 5.

[Parush 2015] Parush, A. (2015). Chapter 13 - first, user research. just do it. In *Conceptual Design for Interactive Systems*, pages 79–88. Morgan Kaufmann.

[Patro et al. 2020] Patro, S. G. K., Mishra, B. K., Panda, S. K., Kumar, R., Long, H. V., Taniar, D., and Priyadarshini, I. (2020). A hybrid action-related k-nearest neighbour (har-knn) approach for recommendation systems. *IEEE Access*, 8(1):90978–90991.

[Peng et al. 2019] Peng, D., Yuan, W., and Liu, C. (2019). Harsam: A hybrid model for recommendation supported by self-attention mechanism. *IEEE Access*, 7:12620–12629.

[Pu et al. 2011] Pu, P., Chen, L., and Hu, R. (2011). A user-centric evaluation framework for recommender systems. In *Proceedings of the Fifth ACM Conference on Recommender Systems*, RecSys ’11, page 157–164, New York, NY, USA. Association for Computing Machinery.

[Qian Gao 2021] Qian Gao, P. M. (2021). Graph neural network e context-aware user behavior prediction and recommendation system research. In *Computational Intelligence and Neuroscience*, volume vol. 2020, page 14. Hindawi.
[Rashid et al. 2008] Rashid, A. M., Karypis, G., and Riedl, J. (2008). Learning preferences of new users in recommender systems: an information theoretic approach. *Acm Sigkdd Explorations Newsletter*, 10(2):90–100.

[Reese et al. 2016] Reese, D. D., Pawluk, D. T., and Taylor, C. R. (2016). Chapter 6 - engaging learners through rational design of multisensory effects. In Tettegah, S. Y. and Noble, S. U., editors, *Emotions, Technology, and Design*, Emotions and Technology, pages 103–127. Academic Press, San Diego.

[Ricci et al. 2022] Ricci, F., Rokach, L., and Shapira, B. (2022). *Recommender Systems: Techniques, Applications, and Challenges*, pages 1–35. Springer.

[Rocha Silva et al. 2020a] Rocha Silva, T., Winckler, M., and Bach, C. (2020a). Evaluating the usage of predefined interactive behaviors for writing user stories: an empirical study with potential product owners. *Cognition, Technology Work, 22*, 437–457.

[Rocha Silva et al. 2020b] Rocha Silva, T., Winckler, M., and Trætteberg, H. (2020b). Ensuring the consistency between user requirements and task models: A behavior-based automated approach. *Proc. ACM Hum.Comput. Interact.*, 4.

[Sabitha et al. 2022] Sabitha, R., Vaishnavi, S., Karthik, S., and Bhavadharini, R. (2022). User interaction based recommender system using machine learning. *Intelligent Automation Soft Comput.*, 31(2):1037–1049.

[Saranya et al. 2020] Saranya, Sowmya, A. S., Mohammed Shebin, K. K., and Mohan, A. (2020). Social recommendation system using network embedding and temporal information. In *2020 5th International Conference on Computing, Communication and Security (ICCCS)*, pages 1–7.

[Schoinas and Tjortjis 2019] Schoinas, I. and Tjortjis, C. (2019). Musif: a product recommendation system based on multi-source implicit feedback. In *Proc. of Artificial Intelligence Applications and Innovations: 15th IFIP WG 12.5 International Conference, AIAI 2019, Greece, 2019*, pages 660–672. Springer.

[Setiowati et al. 2018] Setiowati, S., Adji, T. B., and Ardiyanto, I. (2018). Context-based awareness in location recommendation system to enhance recommendation quality: A review. In *2018 International Conference on Information and Communications Technology (ICOIACT)*, pages 90–95. IEEE.

[Shibamoto et al. 2019] Shibamoto, E., Kittirojrattana, C., Koopipat, C., Hansuebsai, A., and Takano, K. (2019). A recommendation system of sightseeing places based on user's behavior of taking and editing photos. In *IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM)*, pages 1–6.

[Silva and Winckler 2022] Silva, T. R. and Winckler, M. (2022). Assessing user interface design artifacts: A tool-supported behavior-based approach.

[Sinha and Dhanalakshmi 2019] Sinha, B. B. and Dhanalakshmi, R. (2019). Evolution of recommender system over the time. *Soft Computing*, 23(23):12169–12188.

[Su et al. 2021] Su, Z., Lin, Z., Ai, J., and Li, H. (2021). Rating prediction in recommender systems based on user behavior probability and complex network modeling. *IEEE Access*, 9:30739–30749.
[Swearingen and Sinha 2002] Swearingen, K. and Sinha, R. (2002). Interaction design for recommender systems. In *Designing Interactive Systems*, pages 312–334.

[Tarus et al. 2018] Tarus, J. K., Niu, Z., and Kalui, D. (2018). A hybrid recommender system for e-learning based on context awareness and sequential pattern mining. *Soft Computing*, 22:2449–2461.

[Walek 2017] Walek, B. (2017). Creating adaptive web recommendation system based on user behavior. *Journal of Physics: Conference Series*, 933:012014.

[Wang et al. 2022] Wang, X., Li, Q., Yu, D., and Xu, G. (2022). Off-policy learning over heterogeneous information for recommendation. In *Proc. of the ACM Web Conference 2022*, page 2348–2359. ACM.

[Wang et al. 2016] Wang, Y., Shang, W., and Li, Z. (2016). The application of factorization machines in user behavior prediction. In *2016 IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS)*, pages 1–4.

[Wang et al. 2009] Wang, Y., Stash, N., Aroyo, L., Hollink, L., and Schreiber, G. (2009). Semantic relations for content-based recommendations. In *Proc. of the Fifth International Conference on Knowledge Capture*, page 209–210. ACM.

[Widiyaningtyas et al. 2021] Widiyaningtyas, T., Hidayah, I., and Adji, T. B. (2021). User profile correlation-based similarity (UPCSim) algorithm in movie recommendation system. *Journal of Big Data*, 8(1):52.

[Wohlin 2014] Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. *Proc. of the 18th International Conference on Evaluation and Assessment in Software Engineering*.

[Wu et al. 2016] Wu, S., Liu, Q., Wang, L., and Tan, T. (2016). Contextual operation for recommender systems. *IEEE Transactions on Knowledge and Data Engineering*, 28(8):2000–2012.

[Xie et al. 2022] Xie, Y., Wang, Z., Yang, C., Li, Y., Ding, B., Deng, H., and Han, J. (2022). Komen: Domain knowledge guided interaction recommendation for emerging scenarios. In *Proc. of the ACM Web Conference 2022*, page 1301–1310. ACM.

[Xu et al. 2019] Xu, X., Chikersal, P., Doryab, A., Villalba, D. K., Dutcher, J. M., Tumminia, M. J., Althoff, T., Cohen, S., Creswell, K. G., Creswell, J. D., et al. (2019). Leveraging routine behavior and contextually-filtered features for depression detection among college students. *Proc. of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies*, 3(3):1–33.

[Xu et al. 2020] Xu, X., Hassan Awadallah, A., T. Dumais, S., Omar, F., Popp, B., Rounthwaite, R., and Jahanbakhsh, F. (2020). Understanding user behavior for document recommendation. In *Proc. of The Web Conference 2020*, page 3012–3018. ACM.

[Yang and Zhang 2020] Yang, Z. and Zhang, M. (2020). Textog: A recommendation model for rating prediction based on heterogeneous fusion of review data. *IEEE Access*, 8:159566–159573.
[Yi et al. 2015] Yi, P., Li, C., Yang, C., and Chen, M. (2015). An optimization method for recommendation system based on user implicit behavior. In *2015 Fifth International Conference on Instrumentation and Measurement, Computer, Communication and Control (IMCCC)*, pages 1537–1540.

[Zheng et al. 2017] Zheng, L., Noroozi, V., and Yu, P. S. (2017). Joint deep modeling of users and items using reviews for recommendation. In *Proc. of the Tenth ACM International Conference on Web Search and Data Mining*, page 425–434. ACM.

[Zheng et al. 2023] Zheng, R., Qu, L., Cui, B., Shi, Y., and Yin, H. (2023). AutoML for deep recommender systems: A survey. *ACM Trans. Inf. Syst.*, 41(4).

[Zheng et al. 2009] Zheng, Z., Ma, H., Lyu, M., and King, I. (2009). Wsrec: A collaborative filtering based web service recommender system. In *2009 IEEE international conference on web services*, pages 437–444. IEEE.

[Zhou et al. 2021] Zhou, D., Hao, S., Zhang, H., Dai, C., An, Y., Ji, Z., and Ganchev, I. (2021). Novel sddm rating prediction models for recommendation systems. *IEEE Access*, 9:101197–101206.

[Zhu et al. 2022] Zhu, Q., Zhang, H., He, Q., and Dou, Z. (2022). A gain-tuning dynamic negative sampler for recommendation. In *Proc. of the ACM Web Conference 2022*, page 277–285. ACM. https://doi.org/10.1145/3485447.3511956.

[Zhuo et al. 2022] Zhuo, J., Zhu, Q., Yue, Y., and Zhao, Y. (2022). Learning explicit user interest boundary for recommendation. In *Proc. of the ACM Web Conference 2022*, page 193–202. ACM.