# Service-Aware Interactive Presentation of Items for Decision-Making
## Materials and Methods
### Review Analysis
#### 3.2.3 Binding Review Sentences to Evaluation Dimensions
Therefore, we formed the result list \(\mathcal{L}\) by selecting four homes that were comparable from the viewpoint of a search query, and which also had a similar amount of consumer feedback in terms of lines of text. We investigated User Experience (UX) with the applications in two learning tasks, each one using a different apartment:

* _Task1_: question answering using the functions provided by INTEREST, i.e., interactive charts, temporal and dimension-dependent review selection, and word highlighting. * _Task2_: question answering using the basic list of reviews (Baseline) with temporal filter. The study was a within-subjects design one. We considered each treatment condition as an independent variable and every participant received the two treatments. We counterbalanced the order of tasks to minimize the impact of result biases and the effects of practice and fatigue. People joined the user study on a voluntary basis, without any compensation, and they gave their informed consent to participate in the study. The consent form described the research purposes, procedure, duration, confidentiality of data, sharing of results, right to refuse or withdraw, and contact persons. The user study took place live, in video calls with shared screen due to COVID-19 pandemic, i.e., we did not use any platform such as Amazon Mechanical Turk or similar to recruit participants or to manage the tasks of the study. ### The Experiment

One person at a time participated in the study, which lasted approximately 30 min. We did not impose any time restrictions in solving the tasks and we allowed her/him to take as much time as (s)heneeded in order to comply with diverse backgrounds and levels of confidence with technology. Then, we asked her/him to answer a pre-test questionnaire designed to assess demographic information, cultural background, and familiarity with booking and e-commerce platforms. For each task of the study the participant used either the Apartment Monitoring implementation of INTEREST, or the Baseline application, in the context of the organization of a holiday in Florence: within the task, we asked her/him to look at the proposed homes and to answer two questions that required either scoring (thumb up or down) an evaluation dimension of an apartment, or identifying the content corresponding to the dimension within the available reviews. For each home, all participants answered the same questions, which were based on the following templates. * Give a thumb up/thumb down evaluation of _dimension_ of _home_ provided by _host_ in _time frame_. For instance, \"Give a thumb up/thumb down evaluation of the surroundings of Toscanella apartment provided by host [PERSON] during the last year\". * List the characteristics of _dimension_ of _home_ provided by _host_ in _time frame_. For example, \"List the characteristics of the host of Il Podesta apartment provided by host [PERSON] during the last six months.\"

While the participant carried out a task, the experimenter took notes about her/his voice comments using the think aloud protocol [94]; we did not put any time restrictions on question answering. For the study we focused on User Experience (UX). 1. After the completion of each task, the participant filled in a post-task questionnaire to evaluate the model (s)he had used. We selected the Italian version of the UEQ questionnaire [95] that supports a quick assessment of a comprehensive impression of user experience covering perceived ergonomic quality, perceived hedonic quality, and perceived attractiveness of a software product. However, as UEQ does not cover user awareness and control, we extended it with three items aimed at investigating these aspects. For this purpose we took inspiration from the ResQue questionnaire for recommender systems [96]. Participants answered each item of our questionnaire by selecting a rating in a 7-point Likert scale. In UEQ, questions are proposed as bipolar items, e.g., [annoying 1 2 3 4 5 6 7 enjoyable].