REASONER: An Explainable Recommendation Dataset with Comprehensive Labeling Ground Truths
Authors: Xu Chen
Jingsen Zhang
Lei Wang
Quanyu Dai
Zhenhua Dong
Ruiming Tang
Rui Zhang
Li Chen
Xin Zhao
Ji-Rong Wen
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Explainable Recommendation
Recommendation Dataset
Labeling Ground Truths

Abstract: Explainable recommendation has attracted much attention from the industry and academic communities. It has shown great potential to improve the recommendation persuasiveness, informativeness and user satisfaction. In the past few years, while a lot of promising explainable recommender models have been proposed, the datasets used to evaluate them still suffer from several limitations, for example, the explanation ground truths are not labeled by the real users, the explanations are mostly single-modal and around only one aspect. To bridge these gaps, in this paper, we build a new explainable recommendation dataset, which, to our knowledge, is the first contribution that provides a large amount of real user labeled multi-modal and multi-aspect explaination ground truths. In specific, we firstly develop a video recommendation platform, where a series of questions around the recommendation explainability are carefully designed. Then, we recruit about 3000 high-quality labelers with different backgrounds to use the system, and collect their behaviors and feedback to our questions. In this paper, we detail the construction process of our dataset and also provide extensive analysis on its characteristics. In addition, we develop a library, where ten well-known explainable recommender models are implemented in a unified framework. Based on this library, we build several benchmarks for different explainable recommendation tasks. At last, we present many new opportunities brought by our dataset, which are expected to promote the field of explainable recommendation. Our dataset, library and the related documents have been released at https://reasoner2023.github.io/.

REASONER: An Explainable Recommendation Dataset with Comprehensive Labeling Ground Truths
Xu Chen 1.2 Jingsen Zhang 1.2, Lei Wang Quanyu Dai Zhenhua Dong', Ruiming Tang'. Rui Zhang Li Chen Wayne Xin Zhao Ji-Rong Wen
Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China 2 Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China DDpartmentt of Computer Science, Hong Kong Baptist University, Hong Kong HHuawei Noah's s Ark Lab, China 5 www..uizhang.inf
Abstract
Explainable recommendation has attracted much attention from the industry and academic communities. has shown great potential to improve the recommenda- tion persuasiveness, informativeness and user satisfaction. In the past few years, while lot promising explainable recommender models have been proposed, the datasets used evaluate them still suffer from several limitations, for example, the explanation ground truths are not labeled by the real users, the explanations are mostly single-modal and around only one aspect. To bridge these gaps, in this paper, we build new explainable recommendation dataset, which, to our knowledge, the first contribution that provides large amount of real user labeled multi-modal and multi-aspect explanation ground truths. In specific, we firstly develop video recommendation platform, where series of questions around the recommendation explainability are carefully designed. Then, we recruit about 3000 high-quality labelers with different backgrounds to use the system, and collect their behaviors and feedback to our questions. In this paper, we detail the construction process of our dataset and also provide extensive analysis on its characteristics. addition, we develop a library, where many well-known explainable recommender models are implemented a unified framework. Based on this library, we build several benchmarks for different explainable recommendation tasks. At last, we present many new opportunities brought by our dataset, which are expected to promote the field of explainable recommendation. Our dataset, library and the related documents have been released at https//reasooer22233,
htt.:aao.onne2222..iu.......o github.io/.
Introduction
Recommender system has become an indispensable component in a large amount of real-world applications, ranging from e-commence [32] and entertainment (46) to education (30] and health- caring 37). Basically, recommender system bridges the content providers with the users. For the content providers, better recommendations can effectively expose the items which are more likely to be clicked or purchased by the users, and thus, achieve better business targets. For the users, an effective recommender model can help to quickly discover the items of interest, and thus, save the user time and alleviate the problem of information overloading, Recently, explainable recommendation has been recognized as an important research direction [36] 2). Essentially, providing explanations for the recommended items can further enhance the utilities of the content providers and users. For
Eqqul contribution. Corresponding author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table 1: Comparison between different explainable recommendation datasets. "Real-U" means that whether the ground truths are labeled by the real users. "Multi-E" indicates that whether the dataset has multi-aspect explanations. "App-S" represents that whether the dataset can be generally used in different studies. "Multi-M" shows that whether the dataset contains multi-modal explanations. In addition, we also present the number of labelers in each dataset.
Reference
Real-U
Multi-E
App-S
Multi-M
# ILahelers
Extra 20
/
x
X
<100
VECF
x
<100
AFRec
24
x
x
< <0000
Fx3 [4
x
x
X
X
< 100
MTER 38
x
/
x
<100
JRASA 288
/
100-300
UseFCE 29
/
/
X
<100
ArgEx [15
V
X
100-300
REASONER
Rs 3000
the content providers, appropriate explanations can better persuade the users to interact with the recommended items [II.. For the users, reasonable explanations reveal more informative item features to facilitate user decisions [18].
In the past few years, while a lot of promising explainable recommender models have been proposed, the datasets used to evaluate them still suffer from several limitations: to begin with, many work evaluate the models by recruiting annotators to label the ground truths on existing public datasets (27). However, the annotators are not the real users in the datasets. Thus, the labeled results may deviate from the real user behavior reasons. Then, previous work mostly evaluate the explanations from only one aspect, for example, "whether the explanation can better persuade the users" [33 or "whether the explanation can reveal more informative item features"' 6 41. However, the recommendation explanations may serve for different purposes (1 thus, these datasets may fail to comprehensively evaluate the model explainability. At last, while recent years have witnessed few pioneer studies on evaluating the explanations from multiple aspects [39 Ii 3], the labeled datasets are not large enough, which may fail to evaluate the models in a statistically sufficient manner.
To alleviate the above problems, in this paper, we propose to build a large real user labeled dataset, which can evaluate the recommendation explanations from multiple aspects. We call our dataset as REASONER, where RE indicates that the ground truths are labeled by the real users, ASON represents that our dataset can evaluate multi-aspect explanations, ER is short for explainable recommendation. While this dataset is important for the explainable recommendation domain, is not easy to build it. To begin with, if we build our dataset by extending an existing one, then it is hard to find the real users in the original dataset to label the explanation ground truths. If we do not rely on an existing dataset, then how to obtain the user-item interactions can be challenge. In addition, different from the other domains, where the labeling tasks are usually objective (e.g., identifying the contents of image), we have to collect user subjective data, which is more sensitive to the designs of the labeling tasks, for example, "how to ask each question" and "what are the orders of the questions". Without proper designs, the annotators may misunderstand the questions or can be interfered by the external factors, which may lower the labeling accuracy. Last but not least, how to control the dataset quality is also challenge. Our dataset aims collect user subjective preferences, it can be difficult to purify it by directly judging whether the labeled results are right or wrong, since different users may have various opinions. We need find effective rules to filter the results according to their reasonableness.
To overcome the above challenges, we propose to build our dataset in a two-step manner. To begin with, we develop a video recommendation platform, and carefully design a series of questions around the recommendation explainability. For example, "which features are the reasons that you would like to watch this video?" and "which features are most informative for this video?". Then, we recruit about 3000 labelers to our platform and answer the above questions. This makes that the labelers of the explanation ground truth are exactly the real users that produce the user-item interaction To control the quality of our dataset, set many rules check and filter the labeled results, For example, "whether the labeling time is too short?" and "whether the answers of the same
In the following, do not discriminate the users and labelers.

---

Tom
Chris
Chris
Tom
User
Be
Intoraction
Canegory
Pauture
Type
Tom Cruise
Chris Pine
Enhancing Persuasiveness
Enhancing Informativeness
Addreso
Anrwer
Vr Set
user may click the video
The user know more about the video
Rigllen
Assowrr 10 00
Int Set
(a)
(b)
Figure 1: (a) All the information collected in dataset.
(b) Different ground truths for the
recommendation explanations with different purposes.
user conflict with each other?". Beyond the above dataset, we also develop library. which contains many well-known explainable recommender models. With this library, we build several benchmarks based on our dataset for different explainable recommendation tasks. Due to the space limitation, we present the details on the library and benchmarks in the appendix. last, we point out several new opportunities brought by our dataset. We have released the above-mentionel dataset, library and documents at https///ressoer22233. 2atp:soonr2223..tguuuuuii io/ under CC BY-NC 4.0 license.
Related Datasets
The datasets for evaluating explainable recommender models can be divided into two classes. In the first class, the datasets are usually generated by recruiting labelers to answer the questions around the model explainability. For example, asks the labelers imagine themselves as the real users and label the explanation ground truth. 24 requires the labelers to compare the explanations generated from the designed models and baselines. In [0 22), the labelers have to evaluate the helpfulness of the explanations in terms of facilitating better user decisions.
[35] 38 let the labelers to express their satisfactions and rate the informativeness of the explanations. (20 regards the user reviews as the
explanation ground truths, and projects similar review sentences into the same cluster to facilitate model optimization. In this type of datasets, the labelers are usually not the real users, thus the labeled results may not accurately reflect the real user preference. In the second class, the explanation dataset completely constructed by the labelers. For example, the authors in [28] recruit 286 labelers to evaluate the transparency and effectiveness of some given explainable recommender models. In 29) the authors employ 20 annotators to study the effectiveness of the feature-bate -ased explanations. 152 annotators recruited, and the target is to study the
15
perception on the explanation types. While the results in these datasets can well represent the real user preference, the labeling questions highly model-specific, which be hardly used in the other studies. Comparing with the above datasets, REASONER has many significant advantages. For example, it is labeled by the real users and can be used evaluate different models. In addition, has multi-aspect explanation ground truths with different modalities. For clearly understanding the advantages of REASONER, we compare it with the previous explainable recommendation datasets Table
Dataset Construction
To align the labelers with the real users, our dataset is constructed based on two steps. The first step is building a recommendation platform, and designing many questions about the recommendation explainability. The second step is recruiting labelers use the above platform and collecting their behaviors and answers to the above questions. In the following, we detail these two steps.
3.1 Building the Recommendation Platform
Comparing with traditional recommender systems, our platform is specially designed for collecting the user explainable behaviors. In this section, firstly introduce the items on the platform and the

---

explanation candidates that the labelers can select from. Then, we describe the questions designed for collecting the explanation ground truths. Finally, we present the complete labeling process for each labeler based on our platform.
3.1.1 Items
We select videos as the recommendation items, since they are content intensive for providing sufficient explanation candidates. Considering that too long labeling time may distract the user attention, the length of the videos confined to be shorter than three minutes. The selected videos can cover different categories, such as technology and music, which are expected to comprchensively represent the user preferences. All the videos are from an industrial short-video website.
3.1.2 Explanation candidates
We collect the explanation ground truths by asking the labelers to select from many video features, egg. "which features are the reasons that you would like to watch this video?". The most important features are:
Tags. These are textual features, which are collected from three sources: (1) the original tags (2) The overall comments, which posted by the viewers after watching the video. (3) The emee-sycchonnieed comments, which are provided by the viewers as they watching the video in a real-time manner [25). The above information is from the same website as that of the videos. Since the overall and time-synchronized comments are originally raw texts, we process them by neural language processing tools 43 to extract the candidate tags. We combine all the tags from different sources, and manually remove the tags which are duplicated, meaningless or containing swear words.
provided by the video authors.
These are visual features, which are extracted from the videos. To make the previews more representative, use standard key frame extraction too for preview generation. We manually check the similarities between different previews, and remove the duplicated ones. For each video, finally generate five previews for the labelers.
Previews.
In addition to the tags and previews, we also present the video titles and brief introductions to the labelers. Besides, the labelers can also write the answers in a reserved text box, if they find that all the above features cannot represent their ideas.
3.1.3 Labeling questions
As mentioned before, the recommendation explanations may serve for different purposes. For the content provides, the explanations be used to enhance the recommendation persuasiveness. For the users, the explanations can be leveraged to provide more informative item features for faster user decisions. As result, there in Figure [b), the user is a fan of Tom Cruise. If we explain the video by "TTm Cruise", then the user may click the video, and the purpose of the content provided is achieved. From this perspective, "Tom Cruise" is the ground truth for improving the recommendation persuasiveness. However, while Tom Cruise is in the video, he only appears for very short time, and the real leading actor Chris Pine. If we explain the video by "Chris Pine", then the user can make more accurate and informed decisions. From this perspective, "Chris Pine" is the ground truth for enhancing the recommendation informativeness. To obtain the explanation ground truths from multiple aspects, we design series of questions for the labelers to answer, and we introduce the most important ones follows:
be multiple ground truths for the explanations. For example,
Q1. Which features are the reasons that you would like to watch this video? This question is asked before the user watches the video. The selected features the user expectations, which reveal the reasons that motivate the users to watch the videos. This question aims to obtain the explanation ground truth in terms of enhancing the recommendation persuasiveness.
Q2. Which features are most informative for this video? This question is asked after the user has watched the video. At this time, the user has fully understood the video, thus the selected features can objectively represent the video. By this question, we would like to obtain the explanation ground truth in terms of enhancing the recommendation informativeness.

---

Repeat for 6 times
Step
Registration
Email, Name, Age.
Video
Before watching
Watching: requined to completee
watching The arswers
Education, Ocoupation, Reskdont, Habit
Brtef Introduction, Tags, Proviows
she like this video" explain why (Q1)
questions from
Figure 2: The complete labeling processing for each user to collect our dataset.
Q3. Which features are you most satisfied with? This question is also asked after the user has watched the video. It aims to obtain the explanation ground truth in terms of improving the user satisfaction. Comparing with Q1, this ground truth is obtained after the user has fully understood the video contents. Comparing with Q2, this ground truth involves the user subjective factors.
Q4. Please give a rating (from to S) to this video according to your preference. This question is designed to collect the user overall preference on the video.
Q5. How do you comment this video? This question aims to collect more detailed user preference on the video, and the user express her opinions in text box.
By the above questions, we collect the explanation ground truths from three aspects, that is, per- suasiveness (Q1), informativeness (Q2) and user satisfactions (Q3). In summary, we present all the information collected in our dataset in Figure a).
3.1.4 Workflow of the labeling task
Based on the above questions, in this section, introduce the complete labeling process for each user (see Figure
In specific, there are five important steps:
Step 1: User sign up. To begin with, the users have to sign up platform by providing their basic personal information. And then, we also require the users to complete a test on big five personality. This test includes 15 questions, which are standard and widely used the world Based on the answers to these questions, we can understand the personality a user from five dimensions, that is, openness, conscientiousnesss, extraversion, agreeableness and neuroticism. We refer the readers to [22 [3) for more detailed introduction on the personality test. The big five test may provide us with the opportunities study the influence of different user personalities on the selection tendency of the recommendation explanations. For the above personal information, we have obtained the user permissions use it for research purposes. To protect the user privacy, our dataset has been desensitized based on IDs.
Step 2: Platform recommendation. After the user has logged into the platform, the system will recommend her with three videos. For each video, the features introduced in Section s.tiionn.1.. 3.1.2 are also presented the user. The videos are recommended in completely random manner, that is, different videos are exposed to the user with the same probability. This setting aims to collect more diverse user preferences without the information cocoons effect. addition, our dataset may also contribute to the field of debiased recommendation 77, where collecting uniform item exposure traffic can be expensive in real-world scenarios 45
Step 3: User selection and before-watching question answering. Once the user has received the recommendations, for each video, she has carefully check its features, and decides whether she would like to watch this video. the user decide to watch the video, then we require her to answer Q1 in Section 3.1.3 that is, "which features the Otherwise, the user has to select the features which make her not want to watch the video.
that you would like watch this video?".
Step 4: Video watching. In this step, the user has to watch the video. Since the video only lasts for a short time (<3min), the user can quickly understand it with little effort. For the videos that the user does not want to watch in the previous step, we also require her to watch them and answer the
hhtps:/wwww.wyyoooliiirrssayyyy httm
5

---

questions in the following steps, which may help to study whether the videos of interests can be missed due to the improper explanations.
Step 5: After-watching question answering. After the video has been watched, we require the user answer series of questions. To begin with, the user has to provide her rating and comments the video (corresponding to Q4 and Q5). Then, we require the user to select the features which are most informative for the videos (corresponding Q2). At last, the user needs to select the features that they are most satisfied with (corresponding to Q3).
For each user, step 1 is only done for once, step 2 to step 5 are repeated for least six times. Since the system recommends three videos every time, each user receives more than 3 x (=18) recommendations in the complete labeling process.
3.2 Collecting the Dataset
Based on the above platform, we then recruit labelers to use it and complete the above labeling process. To control dataset quality, we design many rules to purify the labeled results.
3.2.1 Labelers
To guarantce the dataset quality, we cooperate with a professional company to recruit the labelers. For each labeler, we require that she should have been employed by the company before and have better working records. In addition, we need that the background of the labelers should be diverse enough. In total, we recruit about 3000 labelers, and we pay each labeler $8 USD for completing the labeling task. The overall cost of our dataset is about $28, 000 USD.
3.2.2 Dataset quality control
In traditional labeling tasks, such as the image recognition or entity tagging, the ground truths are objective, and it is easy to judge whether the labeled results are right or wrong, which makes hard to control the dataset quality. However, our dataset aims to collect the user subjective and personalized preferences. There is no strict right or wrong, which brings difficulties to control the dataset quality. To solve this problem, we firstly set series of rules judge whether the labeled results are reasonable, and then remove the unreasonable ones. In specific, the rules are mainly designed around the following aspects:
A1. Labeling time. On our platform, the time cost of the user for answering each question has been recorded in our database. For each question, we gather all the time costs of the users, and check whether there are outliers. For all the questions, if the same user is identified as an outlier for more than five times, then the user and all her labeled results are removed from the dataset.
A2. Repeated answers. For this aspect, we check whether the user answers to different questions are highly similar. For example, some labelers may always select the first three tags for QI across all the videos. Some labelers may use the same comment for different videos. In general, if a user has more than 1/3 similar answers, then remove this user and her labeled results.
A3. Conflict answers to the same question. To check whether the labelers have carefully done our task, we intentionally insert many questions which have been answered by the labelers before. For the same labeler, we examine whether the answers to the same question are consistent. If user has more than 1/3 inconsistent answers, then the user is removed.
A4. Conflict answers across different questions. Intuitively, if the labelers have carefully completed our task, then there should be some reasonable relations between different answers. For example, if the user give a very low rating, then the comments should be mostly negative. At first, we planned to summarize all the unreasonable relations and develop an automatic tool to identify them. However, we find that the relation patterns between different answers are too complex, and building reliable enough tool may cost too much time. To solve these problems, we recruit five students in our lab to manually check each record the dataset. To begin with, we require each student to independently check the dataset, and find out the unreasonable relations. Then, we merge all the unreasonable results, and the students to vote for each one to determine whether it reasonable or not. For the unrcasonable results, we remove the corresponding records in the final datasets.

---

15%
12%
22%
Sk /month
1%
Santing
14%
5- 10k /mooth
48%
52%
10-15k /mntth
1%
Youn Avmoloon
16 20k imonth
7%
over 20k mmntth
69%
15%
14%
15%
(a) User Gender
(c) User Income
(e) Video Category
11%
5% 3%
8%
9%
26%
22%
Serior High
60-90s
Brsheorrs Degroe Marte's Degree
40%
41
125-30
500-85
20-25
over 35
90-1200
150-180s
120-150g
25%
27%
(b) User Education
(d) User Age
Video Duration
Figure 3: Statistics of the users and videos.
2000
2477
Interaation Number
Uber Numbbog
2000
2000
Video
1145
1000
500
2704
2072
125
(0.20)
(50,40)
(0.18) (10.15) (15,20] 220
19.0
(10,15)
[15,70]
(5.50)
230
Video
User Numbar
Review Length
Figure 4: (a) The distribution of the number of users who watched videos in different ranges. (b) The distribution of the number of videos watched by users in different ranges. (c) The distribution of the user review length in words.
Statistical analysis
In this section, introduce the basic statistics of dataset, where focus the following aspects: (1) the basic statistics of the users and videos. (2) The statistics on the user-video, user-tag and video-tag interactions.
(3) The similarities between the ground truths for different explanation aspects. (4) The statistics on the users who tend to select single- or multi-modality explanations. (5) reviews.
The statistics the length of the
our dataset, there are total 2997 users, 4672 videos and 6115 tags. In Figure Ja-d), we present the statistics of the users based their genders, incomes, educations and ages. We users can cover different populations, which makes our dataset more general and representative. In Figure e-),), present the statistics of the vidcos based their categories and lengths. We can see, the categories of the videos are diverse, and for different categories, there are almost the same number of videos, which help to explore more comprehensive and unbiased user preferences.
the
In Figure aa) and bb), we present the distribution of the vier-video interactions. Each point (x. represents that there are users (or videos) who interact with videos (or users). For clear presentation, we merge the number videos (or users) into four bins. Figure 5 we show distribution of the user-tag interactions. Since there are three types of tags for Q1, Q2 and Q3 as mentioned in Section 3.1.3 we present them separately. Each point (z, y) these figures represents that there are y users who totally select tags across different videos, where the x-axis is also clustered into four bins. Fi we show the distribution of the video-tag interactions. Similar to Figure 5 present the results for Q1, Q2 and Q3 separately. Each point (z, represents that there are y videos whose tags have been totally selected for & times by different users.
In our dataset, we label the ground truths for multi- explanations. An interesting problem is that how different are the tags selected by user for same video on different explanation aspects. To answer this question, we measure the tag similarities based on the Jaccard distance. For every two explanation aspects, we firstly compute the tag similarity for each iser-item pair, and then report the result by averaging all the vidro pairs in Table 2 We see, for different explanation purposes,

---

1032
954
859
1200
1082
User Number
1000
600
591
600
470
800
488
400
200
[0,80] (50,90) 120, a20
(0,60) [80,90] (80,120),1200
[0,50] (60,90) (90, 120 120
Tag Number
Tag Number
Tag Number
Figure Distribution of the user-tag interactions. These three graphs respectively display the distribution of the number of users who label different numbers of Q1, Q2, and Q3 tags.
Video Number
2000
2000
1894
2054
2000
1500
1998
1500
1445
1500
1000
796
1000
734
1000
500
188
500
425
[0,00] (80, (00,00) 190,
(0.90) (60, 90) (90, 120
(0.60) (60,90) [90,125) x20)
Tag Number
Tag Number
Tag
Figure 6: Distribution of the video-tag interactions. These three graphs respectively display the distribution the number of videos labeled with different numbers of Q1, Q2, and Q3 tags.
the overlaps between the selected tags are usually not large (about 0.35 to 0.45). This demonstrates that one may need to use different tags to achieve different explanation purposes.
In our dataset, the users can select multi-mode explanations. Here, we compute the ratio between the numbers of users who select multi-modal explanations and all the users. We present the results for QI, Q2 and Q3 separately in Table 3, where we can see about 38% users tend to select both textual and visual explanations, while the other users would like select single-modal explanations. At last, we present the distribution of the user review lengths in Figure (c). We can see most of the reviews contain more than words, which can comprehensively reveal the user preferences.
Table The average Jaccard similarity for different explanation aspects.
Explanation aspects
Q1-Q2
Q1-Q3
Q2-Q3
Jaccard similarity
0.423
0.364
0.410
Table 3: Ratios between the number users who select multi-modal explanations and the total number of users.
Explanation aspect
Q1 0.389
Q2
Q3
Ratio
0.390
0.359
Library
Besides the above dataset, also develop library, where many well-known explainable rec- ommender models are implemented in a unified framework. With this library, we build several benchmarks based dataset for different explainable recommendation tasks. Due the space limitation, more details on the library and benchmarks are presented in the appendices.
5.1 The Structure of the Library
We show the structure of our library in Figure The configuration module is the base part of the library and responsible for initializing all the parameters. We support three methods to specify the parameters, that is, the command line, parameter dictionary and configuration file. Based on the configuration module, there are four higher layer modules, that is,
Data module: this module converts the raw data into the model inputs. There are two components: the first one is responsible for loading the data and building vocabularies for the user reviews. The second part aims to process the data into the formats required by the model inputs, and generate the sample batches for model optimization.

---

Execution
Rating Prediction
Top-k Recommendation
Review Generation
Data
Model
Trainer
Evaluation
Batch Processing
Natural Language-based
Negative
Value Metries
Sampling
Ranking Metries
Training
Dataloader
Feature-based
Loop
Text Metrics
Configuration
Command Line
Parameter Dicts
Config Files
Figure 7: The structure of our library. There are six parts including the configuration, data, model, trainer, evaluation and execution modules.
Model module: this module implements the explainable recommender models. There are two types of methods in our library. The first one includes the caature-based explainable recommender models, and the second one contains the models with natural language explanations. We delay the detailed introduction of these models in the next section.
Trainer module: this module is leveraged implement the training losses, such as the Bayesian Personalized Ranking (BPR) and Binary Cross Entropy (BCE). In addition, this module can also record the complete model training process.
Evaluation module: this module is designed to evaluate different models, and there are three types of evaluation tasks, that is, rating prediction, top-k recommendation and review generation.
Upon the above four modules, there is an execution module on the upper-most layer. is responsible for optimizing models for different tasks, such as rating prediction and review generation.
5.2 Examples for Using Our Library
In this section, we introduce how to use our library. We present simple example in Figure B(a), where one can directly execute lag ses__reddictonppy review-based model, respectively. each of these commands, one needs to specify three parameters to indicate the names of the model, dataset and configuration file, respectively.
rrview_generate.p to run feature-based or
Take tag_ predictiion for example, it sequentially executes the following steps: (1) Configuration. In this step, the parameters related to the model architecture and optimization process from different sources (commend line, configuration dictionary and files) are integrated into dictionary. (2) Data loading. In this step, the dataloader is selected according to model type. For the review-aware models, this step reads all the records and build the vocabulary. (3) Data Formatting. The training, validation and test sets are processed into the formats required by the model input in sample batch manner. (4) Initialization. The corresponding model class will be defined and initialized according to parameter values in configuration. (5)-(6) Training. Selecting the optimization approach to train the model. (7) Evaluation. Mcasuring the model performance on different tasks.
Our library is highly extensible, and there are three steps to realize a new model: (1) implementing the basic functions of the model, including the model architecture, preference score prediction etc. (2) Customizing the training approaches in train.py. (3) Indicating the parameters in the config file.
New Opportunities
Based on the information collected in our dataset (see Figure T a)), we believe that our dataset can bring the following new opportunities for the domain of explamable recommendation:
Explainable recommendation with extensive personal information. In most of the previous studies, the datasets used for training explainable recommender models do not contain sufficient personal information. By our dataset, people can access the desensitized user profiles, which may facilitate lot research directions. For example, one can study the explanation fairness, where the sensitive variables can the age, education, income and so on. addition, people can also use the extensive personal information to enhance the predictive accuracy of the explanations.

---

Usage Example:
Tag-aware models: python tag_ -oo. -dataastt -on.:a----
Review-aware models: python www.://wweerrtioooyyy-odddddddde - -config=NRTyamll
core
[a]
The Running flow of :
(I) Configuration
(3) config Config[config sia__iistt.iinl_commeee dict
(2) data -ettg
(2) Data Loading
(3) train_data gett_bacciffl/mddll Lype, train_ siin__yyeejdaaa.rriisses config)
valid data get__bachiff(mmddl_nooo train wilittppeeddattawlddsstt, config)
(3) Data Formatting
test_ = get pett_batcchfl/mmdde_wye trrin__tyeeeddaaee saai__yveeddaaa..eessst,, config)
(4) Initialization
(4) model = get wwl_mmeellmddee_mmmeelooooo
(5) trainer get_ Srriiee((model_tyeel(cooom model, train_ data, valid data)
($) - (6) Training
(6) best Bast__mdel_ppee path, best_ traneecccttannneerraannnnoooo loop()
(7) Evaluation
(7) results trainerrewaluattebesttttt sacpresst__dde____t___ttte test_data)
(b)
Figure 8: (a) Examples of the feature- and review-hased models. (b) The running flow of our library.
Multi-aspect explainable recommendation. For almost all the previous explainable recommender models, there is only one aspect ground truths for the explanations in the dataset. Thus, the learned models may not perform well on the other aspects. For example, if the model only trained to improve the recommendation persuasiveness, then the generated explanations may not help to enhance (or even hurt) the recommendation informativeness. By our dataset, people can simultaneously consider different explanation aspects, and learn more comprehensive explainable model to serve the on- on-line users. We argue that there can much room in this direction, example, if different explanation purposes have contradictions, how to design models to better overcome such contradictions.
Multi-modal explainable recommendation. To our knowledge. most previous explainable recommender models only consider singlc-modal explanations. However, in the real-world scenarios, the users always need to perceive multi-modal information. By our dataset, people can conduct lot of studies around the multi-modal explanations, for example: (1) which parts of users would like to select textual or visual features? (2) How to build a unified model to generate both textual and visual explanations? (3) What is the relation between the user personalities and the explanation modalities? And how to build model to predict the explanation modality given a user personal information?
Beyond the above research opportunities on explainable recommendation, our dataset can also contribute the other research domains. For example, since dataset is collected with uniform item exposure probabilities, people can use our dataset to study debiased recommendation. In our dataset, we have collected user big five personality information, thus, one can use it investigate esychology-informe recommendation.
Conclusion and Future Work
In this paper, introduced new explainable recommendation dataset, where we detailed its construction process and extensively analyzed its characters. Our dataset can support multi-modal explanations, and the ground truths annotators exactly the real users who produce the interactions. In addition, our dataset support multi-aspect explanations, and we have about 3000 users in the dataset. We developed an explainable recommendation toolkit, and built several benchmarks based on the above dataset for different explainable recommendation tasks. In addition, we also analyze the new opportunities brought by our dataset. Actually, our dataset only a small step towards more measurable explainable recommendation, and there is still long way to go. In the future, we plan to extend our dataset to the other domains, such as news, music and so on. In addition, we would also like employ more labelers to enlarge our dataset.
10

---

Acknowledgments and Disclosure of Funding
This work is supported part by National Key R&D Program of China (20222D0120103)) National
Natural Science Foundation of China (No. 62102420). Beijing Outstanding Young Scientist Program Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China, Public Computing Cloud, Renmin University of China, fund for building world-class universities (disciplines) of Renmin University of China, Intelligent Social Governance Platform. The work is sponsored by Huawei Innovation Research Programs. We gratefully acknowledge the support from Mindsport CANN (Compute Architecture for Neural Networks) and Ascend Ai Processor used for this research. We sincerely thank the contributions Zhenlei Wang, Rui Zhou, Kun Lin, Zeyu Zhang, Jiakai Tang and Hao Yang from Renmin University of China for checking the unreasonable results in the dataset construction process.
NO. BIJWZYJHO1 21100000000100000000000
References
[1] Krisztian Balog and Filip Radlinski. Measuring recommendation explanation quality: The conflicting goals of explanations. In SIGIR, 2020.
[2] Krisztian Balog, Filip Radlinski, and Shushan Arakelyan. Transparent, scrutable and explainable user models for personalized recommendation. In SIGIR, pages 265-274, 2019.
[3] Krisztian Balog, Filip Radlinski, and Alexandros Karatzoglou. On interpretation and measure- ment of soft attributes for recommendation. In SIGIR, pages 890-899, 2021.
[4] Konstantin Bauman, Bing Liu, and Alexander Tuzhilin. Aspect based recommendations: Recommending items with the most valuable aspects based on user reviews. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pages 717-725, 2017.
[5] Bo Chen, Yue Ding, Xin Xin, Yunzhe Li, Yule Wang, and Dong Wang. Airec: Attentive intersection model for tag-aware recommendation. Neurocomputing, 421:105-114, 2021.
[6] Chong Chen, Min Zhang, Yiqun Liu, and Shaoping Ma. Neural attentional rating regression with reveew-level explanations. In Proceedings of the 2018 world wide web conference, pages 1582-1592, 2018
[7] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions Information Systems, 2020.
[8] Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, and Hongyuan Zha. Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation. In SIGIR, pages 765-774, 2019.
[9] Xu Chen, Zheng Qin, Yongfeng Zhang, and Tao Xu. Learning to rank features for recommenda- tion over multiple categories. In SIGIR, pages 305-314 2016.
[10] Li Dong, Shaohan Huang, Furu Wei, Mirella Lapata, Ming Zhou, and Ke Xu. Learning to generate product reviews from attributes. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Volume Long Papers, pages
2017.
[11] Fatih Gedikli, Dietmar Jannach, and Mouzhi Ge. How should i explain? comparison of different explanation types for recommender systems. International Journal of Human- Computer Studies, 3272)2367-382, 2014.
[12] Lewis R Goldberg. The development of markers for the big-five factor structure. Psychological assessment, 4(1):26, 1992.
"aaps//wwww.inddspor..cn
11

---

[13] Samuel D Gosling, Peter J Rentfrow, and William B Swann Jr. A very brief measure of the big-five personality domains. Journal of Research in personality. 37(6):504 -284, 2003.
[14] Xiangnan He, Tao Chen, Min- Yen Kan, and Xiao Chen. Trirank: Review- aware explainable recommendation by modeling aspects. In Proceedings of the 24th ACM international on conference on information and knowledge management, pages 1661-1670, 2015.
[15] Diana et al. Hrrmandez-Bocanegee perception of review-based recommendations. In UMAP, 2020.
Effects of argumentative explanation types on the
[16] Yunfeng Hou, Ning Yang, Yi Wu, and Philip S Yu. Explainable recommendation with fusion of aspect information. World Wide Web, 22:221-240, 2019.
[17] Ruoran Huang, Nian Wang, Chuanqi Han, Fang Yu, and Li Cui. Tnam: tag awae neural attention model for top-n recommendation. Neurocomputing, 385:1-12, 2020.
[18] Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin- Mihhai Barbu, and Jürgen Ziegler. Let me explain: Impact of personal and impersonal explanations on trust in recommender systems, In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1-12, 2019.
[19] Hongmei Li, Xingchun Diao, Jianjun Cao, Lei Zhang, and Qin Feng. Tag-aware recommen- dation based on bayesian personalized ranking and feature mapping. Intelligent data analysis, (3(3):641-653 2019.
[20] Lei Li, Yongfeng Zhang, and Chen. Extra: Explanation ranking datasets for explainable recommendation. In SIGIR, pages 2463-2469, 2021.
[21] Lei Li, Yongfeng Zhang, and Li Chen. Personalized transformer for explainable recommenda- tion. arXiv preprint arXiv: 105. 11601, 2021.
[22] Lei al. Li. Towards controllable explanation generation for recommender systems via neural template. In WebCoff 2020.
[23] Piji Li, Zihao Wang, Zhaochun Ren, Lidong Bing, and Wai Lam. Neural rating regression with abstractive tips generation for recommendation. In SIGIR, pages 345 2017.
[24] Yang et Li. Attribute-ave aware explainable complementary clothing recommendation. World Wide Web, 2021.
[25] Zhenyu Liao, Yikun Xian, Xiao Yang, Qinpei Zhao, Chenxi Zhang, and Jiangfeng Li. Tscset: A crowdsourced time-syne comment dataset for exploration of user experience improvement. In 23rd International Conference on Intelligent User Interfaces, pages 641-652, 2018
[26] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[27] Christopher Lin, Ece Kamar, and Eric Horvitz. Signals in the silence: Models of implicit feed- back in recommendation system for crowdsourcing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014.
[28] Cataldo Musto, Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. Justifying recom- mendations through aspect-based sentiment analysis of users reviews. In Proceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization, pages 4-12, 2019.
[29] Sidra al. Naveed. On the use of feature-based collaborative explanations: An empirical comparison of explanation styles. In UMAP, 2020.
[30] Charbel Obeid, Inaya Lahoud, Hicham El Khoury, and Pierre-Antoine Champin. Ontology- based recommender system in higher education. In Companion proceedings of the the web conference 2018, pages 1031-1034 2018
[31] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
12

---

[32] J Ben Schafer, Joseph Konstan, and John Riedl. Recommender systems in e-commerce. In Proceedings of the 1st ACM conference on Electronic commerce, pages 158-166, 1999.
[33] Amit Sharma and Dan Cosley. Do social explanations work? studying and modeling the effects of social explanations in recommender systems. In Proceedings of the 22nd international conference on World Wide Web, pages 113--1144, 2013.
[34] Jianshan Sun, Mingyue Zhu, Yuanchun Jiang, Yezheng Liu, and Le Wu, Hierarchical attention model for personalized tag recommendation. Journal of the Association for Information Science and Technology, 72(2):173 (22)):73-189. 189, 2021.
[35] Yiyi Tao, Yiling Jia, Nan Wang, and Hongning Wang. The fact: Taming latent factor models for explainability with factorization trees. In SIGIR, pages 295-304, 2019.
[36] Nava Tintarev and Judith Masthoff. A survey of explanations in recommender systems. In 2007 IEEE 23rd international conference on data engineering workshop, pages 801-810. IEEE, 2007.
[37] Thi Ngoc Trang Tran, Alexander Felfernig, Christoph Trattner, and Andreas Holzinger. Rec- ommender systems in the healthcare domain: oftte-of-the-art and research issues. Journal of Intelligent Information Systems, 57:171-200 2021.
[38] Nan Wang, Hongning Wang, Yiling Jia, and Yue Yin. Explainable recommendation via multi- task learning in opinionated text data. In SIGIR, pages 165-174, 2018
[39] Bingbing Wen, Yunhe Feng. Yongfeng Zhang, and Chirag Shah. Expscore: Learning metrics for recommendation explanation. In Proceedings of the ACM Web Conference 2022, pages 3740-3744 2022.
[40] Yikun al. Xian. Ex3: Explainable attribute-aware item-set recommendations, In Recsys, 2021.
[41] Jingsen Zhang, Xu Chen, Jiakai Tang, Weiqi Shao, Quanyu Dai, Zhenhua Dong, and Rui Zhang. Recommendation with causality enhanced natural language explanations. In Proceedings of the ACM Web Conference 2023, pages 876-886, 2023.
[42] Yongfeng Zhang, Guokun Lai, Min Zhang, Yi Zhang, Yiqun Liu, and Shaoping Ma. Explicit factor models for explainable recommendation based on phrase-level sentiment analysis. In SIGIR, pages 83-92, 2014.
[43] Yongfeng Zhang, Haochen Zhang, Min Zhang, Yiqun Liu, and Shaoping Ma. Do users rate or review? boost phrase-level sentiment labeling with review-level sentiment classification. In SIGIR, pages 1027-1030, 2014.
[44] Weibin Zhao, Aoran Zhang, Lin Shang, Yonghong Yu, Li Zhang, Can Wang, Jiajun Chen, and Hongzhi Yin. Hyperbolic personalized tag recommendation. In International Conference on Database Systems for Advanced Applications, pages 216-231. Springer, 2022.
[45] Chang Zhou, Jianxin Ma, Jianwei Zhang. Jingren Zhou, and Hongxia Yang. Contrastive learning for debiased candidate generation in large-scale recommender systems. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 3985-3995 2021.
[46] Renjie Zhou, Samamon Khemmarat, and Lixin Gao. The impact of youtube recommendation system on video views. In Proceedings of the 10th ACM SIGCOMM conference on Internet measurement, pages 404-410, 2010.
[47] Yi Zuo, Shengzong Liu, Yun Zhou, and Huanhua Liu. Tral: A tag-aware recommendation algorithm based on attention learning. Applied Sciences, 13(2):814, 2023.
[48] Yi Zuo, Yun Zhou, Shengzong Liu, and Yupeng Liu. A tag-aware recommendation algorithm based on deep learning and multi-objective optimization. In 2023 International Conference on Pattern Recognition, Machine Vision and Intelligent Algorithms (PRMVIA), pages 42-46. IEFF, 2023.
13

---

A The Implemented Models in library
In our library, we implement two types of explainable recommender models, which are widely studied in the research community. The first one are feature-bast bassee explainable recommender models, where the features can be the tags, item aspects and on. The second one are the models with natural language explanations.
More specifically, we implement the following representative feature- based explainable recommender models:
EFM 42 predicts the user preferences and generates explainable recommendations based on explicit product features and user opinions from the review information.
TriRank [4] models the user-item-asped ternary relation as a heterogencous tripartite graph based on user ratings and reviews, and it devises a vertex ranking algorithm for recommendation.
LRPPM
y is tensor-matrix factorization algorithm which captures the user preferences using ranking- aaniing-assed optimization objective over various item aspects.
SULM 4 enhances recommendations by recommending not only item but also the specific aspects by using aspect-level sentiment analysis.
MTER 38 is tensor factorization method which models the task of item recommendation using three-way tensor over the users, items and features. We omit the modeling of the opinions in the original implementation for adapting our data.
AMF [6 improves the recommendation accuracy by using the auxiliary information extracted from the user review aspects.
TRDM 48 introduces a two-stage approach to generate accurate item recommendations and effective bas-based potential features simultaneously for enhancing recommendation accuracy and diversity.
TRAL 47
proposes attention-ban baaed learning to capture diverse tag-based features, and compress these features with an attention pooling layer to enhance recommendation accuracy.
HPTR employs hyperbolic distance to measure semantic relevance between entities, which better captures hierarchical structures presented in tag information.
AlRec [5] enhances tag-aware recommender system by employing a hierarchical attention network to capture multi-aspect preferences and leveraging tag intersection to improve conjunct feature learning.
captures distinct user preferences and informative elements by employing separate attention networks for element-level influence and information-level attentiveness.
HAN-TR 34
TNAM 17 addresses the issues of tag weight assignment in recommender systems by introducing tag-based neural attention network that captures users' specific tag attention.
BPR-T [19 addresses high dimension and sparsity issues of tagging information by integrating tag mapping into a Bayesian personalized ranking collaborative filtering model.
In addition to the above shallow models based on matrix factorization, we also implement the following deep feature- based explainable recommender models (called DERM for short):
DERM-MLP is deep recommender model for jointly predicting the ratings and tags. The two tasks share the set of user/item/tag embeddings. The hidden states well as the tag embeddings put into different layers corresponding to the different tasks.
DERM-MF firstly obtains hidden state based the user/item embeddings using matrix factoriza- tion, and then the outputs are computed by a neural network.
DERM-C combines matrix factorization and Multi-Layer Perceptron (MLP) to derive the hidden states, and the outputs are merged in a concatenated manner.
DERM-H leverages the tags to profile the users and items, and then use the same architecture as DERM-MLP P for predicting the ratings and tags.
For the models with natural language explanations, we implement the following representative methods:
14

---

Table 4: Statistics of the datasets.
Dataset
REASONER
Users
2,997
Items
4,672
Tags
6,115
Interactions
58,497
Avg. # words / review
17
Att2Seq [0] is review generation model which uses LSTM as the decoder, and output the texts directly based on the user/item IDs and rating information.
NRT 23 simultancously predicts the reviews and ratings based on the input user-item pair, where the two tasks share the same embedding and hidden layers.
PETER 21 leverages Transformer to generate the user reviews, which is state-of-the-an review generation model.
B Benchmark
B.1 Experiment Setup
Considering that we have three types of ground truths for the explanations, we evaluate the model performance by predicting the tags for QI, Q2, Q3, Q1+Q2, Q2+Q3, Q1+Q3 and Q1+Q2+Q3, respectively. When we have to predict multiple types of ground truths, we extend the original models to their multi-task versions by sharing the embedding parameters. We randomly split the dataset into the training, validation and testing sets according to the ratio of 8:1:1. For the review generation task, use the most 20,000 frequently mentioned words to construct the vocabulary, and the maximum length of the generated sentences is set to 17, which is equal to the average length of reviews in dataset. The dataset statistics are presented in Table For all the models, the batch size is set as 256. We tune the other key hyper-parameters by grid search. In specific, we tune the learning rate and the weight of L2 regularization in the range of [0.1. 0.01, 0.001, 0.0001] and [0.001, 0.0001, 0] respectively. For the deep models, we tune the hidden size and the layer number in the range of [32, 64, 128. 256] and [1, 2, 3, 4] respectively. More details of the experiment setting are shown in our project, which has been released at https:/reasoner22222 2ttrssone2222..iuubbio///// We use RMSE and MAE as the metrics to evaluate the performance of the rating prediction task. For the task of tag prediction, F1 and NDCG are selected to evaluate the model performance. To evaluate the quality of the generated reviews, we leverage the metrics including BLEU 31 and ROUGE for model comparison.
B.2 Experiment Results
The comparison results of the feature-based explainable recommender models on the tasks of tag and rating predictions are presented in Table IT The comparison results of the models with natural language explanations on the task of review generation and rating prediction are presented in Table 12 We use the tags with top 10 prediction scores to calculate Fl and NDCG, and the results are percentage values with "% omitted. For RMSE and MAE, lower value indicates better performance. For each evaluation metric, we use bold fonts to label the best performance. Since the TriRank we implemented does not support to predict multiple types of tags simultaneously, we omit in the corresponding tables.
15

---

Table 5: The benchmarking results of the feature-basse based explainable recommender models on predict- ing the tags for persuasiveness and ratings.
Persuasiveness
Rating Prediction
Metrics
F1
NDCG
RMSE
MAE
EFM
26.90
18.89
1.68
1.24
TriRank
18.36
13.98
2.90
2.58
IRPPM
37.31 37.31 10.23
23.25
1.22
0.96 10.00
SULM
41.68
25.77
1.65
1.30
MTER
5.66
2.65
2.27
1.96
AMF
27 27.93
17.6240.17
2.28
1.86
DERM-MLP
37.74
23.45.0002
1.30
1.06
DERM-MF
36.57
21.4940.17
1.32-0.00
1. 1.14
DERM-C
37.17
23.21
1.30
1.07
DERM-H
35.58=0.07
22.02+0.18
1.27
1.04+0.01
TRDM
30.24
13.44
1.19=0.00
0.95 10.01
TRAL
5.93+
2.61
1.29
1.08
HPTR
38.61 10.20
23.05
2.16
1.88 10.68
AIRec
38.05
23.07
1.30
1.08
HAN-TR
34.96-0.60
18.58
2.10
10.75
TNAM
5.97 10.01
2.60
1.37
1.17
BPR-T
32.70.0.62
18.32
1.23
0.98
Table 6: The benchmarking results of the Teature-based explainable recommender models on predict- ing the tags for informativeness and ratings.
Informativeness
Rating Prediction
Metrics
F1
NDCG
RMSE
MAE
EFM
3.38
3.91
1.68
1.24
TriRank
18.78
14.50.0.09
2.90 2..0
2.58 10.00
LRPPM
37.85
38.35
1.22
0.96
SULM
43.25 10.59
42,97 10.40
1.65
1.30
MTER
8.40.10.86
6.13
2.04
1..6
AMF
28.63 28.63...24
28.95
2.28
1.86
DERM-MLP
38.60 38.60.0.06
38.99
1.30
1.06 10.610.01
DERM-MF
37. 37.1020.09
35.37
1.32
1.1440.00
DERM-C
37. 37.96-0.110
38.43
1.30
1.07
DERM-H
36.3 36.36_0.56
35.8540.41
1.29
1.06+0.01
TRDM
31.49 =2.84
24.09+2.50
1.19
0.95
TRAL
6.04
4.59+0.01
1.28
1.08+0.00
HPTR
39.55 39.55=a.00
37.73+1.88
1.27
1.05
AIRec
38. 38.90
39.33
1.31=0.00
1.08
HAN-TR
1..0
30.69 15.89
2.10
1.83 10.75
TNAM
37.77=0.35
37.97 20.20
1.36
1.16
BPR-T
33.83 20.43
31.15
1.23
0.98 10.01
16

---

Table 7: The benchmarking results of the feature-based explainable recommender models on predict- ing the tags for satisfaction and ratings.
Satisfaction
Rating Prediction
Metrics
F1
NDCG
RMSE
MAE
FFM
4.57
1.79
1.68.0.00
1.24 10.01
TriRank
16.82
13. 13.1610.02
2.00
2.58 10.00
LRPPM
36.04_0.:16
22.35
1.22-0000
0.96
SULM
40.46 1.062
24.80 10.19
1.64
1.29 10.07
MTER
11.92
2.85+1.07
2.26
1.96
AMF
27. 27.16
17.05
2.28
1.86
DERM-MLP
36.76
22.53+0.12
1.30
1.06+0.01
DERM-MF
35.40.0.23
20.59
1.32=0.00
1.14
DERM-C
36 36.20=0.28 22.28
10.0
1.07
DERM-H
34.65=00.43
21.33
1.28
1.05
TRDM
31.29
14.7440.69
1.19
0.95
TRAL
5.89 1.005
2.52
20.00
1.08 10.00
HPTR
35.35=2.90
17.78+3.92
1.81
1.56
AIRec
37.10 10009
22.86
10.01
1.08 10.01
HAN-TR
33.95=1.80
18.06
2.10
1.83
TNAM
5.89 10.05
2.52
1.37 10.00
1.17
BPR-T
33.82.0.29
19.52
1.23
0.98 10.00
Table 8: The benchmarking results of the feature- based explainable recommender models on jointly predicting the tags for persuasiveness, informativeness and ratings.
Persuasiveness
Informativeness
Rating
Metrics
F1
NDCG
F1
NDCG
RMSE
MAE
EFM
15.6940.03
12.74
5.38
3.94
1.66
1. 1.23
IRPPM
37.32 37.22
23.26.0.00
37.89
38.37
1.22
0.96
SULM
41.34 10.53
25.68 _0.20
42.70 10.50
42.82
1.67 10.06
1.31 1.30
MTER
35.53
21.88
36. 00.30
36.09
1.36
1.09
AMF
27.67
17.45
28. 23 10.33
28.57 10.34
2.28 10.00
1.86
DERM-MLP
38.4940.15
23.80
39. 14
39.38
1.30
1.0640.01
DERM-MF
36.8440.04
22.55
37.58 37.58.0.11
37.44 37.44_0.08
1.32
1. 1.15.0000
DERM-C
37. 37.85+0.28
23.4340.21
38.82+0.06
39.1040.09
1.30
1.08
DERM-H
37. 37.47:10.6
23.23.10.22
38. 38.17:40.2
38.32
1. 1.28
1.0440.01
TRDM
32.91+1.06
15.51
33.01+0.50
25.02+0.37
1.19+0.00
0.94
TRAL
5.93
2. 2.602
6.06
4.60+0.02
1.29+0.0
1.08++.000
HPTR
38. 38.68+4.22
23.11+1.20
36.20+4.79
31.40.19.27
2. 2.1640.71
1.88+0.68
AIRec
38.73
23.97
39.43
39.66.10.09
1.31
1.08
HAN-TR
33.32-t0.75
16.83
33.31
27.72+0.46
1.29+0.02
1.0640.03
TNAM
5.91 10.05
2.63.10.02
37.47 10.46
37.35
1.36
1.17
BPR-T
33.15
18.9040.44
34.05.1.20
31.51
1.24
0.99
17

---

Table 9: The benchmarking results of the feature-based explainable recommender models on jointly predicting the tags for persuasiveness, satisfaction and ratings.
Pensuasiveness
Satisfaction
Rating
Metrics
F1
NDCG
F1
NDCG
RMSE
MAE
EFM
15.58+0.03
12.84
4.58
1.77
1.66
1.23++.00
LRPPM
37.32
23.26
36.06
22.37 10.07
1.20
0.9610.00
SULM
40.36
35.71 0.222
40.15
24.70
1..760006
1.31
MTER
5.83
2.56
5.36
2.2340.12
2.04
1.74
AMF
27.71+0.21
17.52=0.21
26.90+0.18
16.9440.16
2.28+0.00
1.86+0.00
DERM-MLP
38.37 +0.09
23.71
37.3240.02
22.87 40.03
1.30
1.0640.01
DERM-MF
36.90+0.12
22.63+0.12
35.78+0.10
21.74=0.12
1.32+0.00
1.15#0.00
DERM-C
38.03+0.11
23.6040.06
36.9540.10
22.72+0.06
1.31
1.08++0.00
DERM-H
37. 37.49+0.24
23.32+0.18
36. 36.11+0.29
22.34+0.17
1.29+0.01
1.04+0.01
TRDM
32.57+1.92
15.11
30.91+1.77
13.86+1.40
1.19
0.94=0.000
TRAL
5.91
2.63+0.02
5.89+0.05
2.52+0.01
1.29+0.00
1.08+0.00
HPTR
38. 08.64
22.93+1146
32.96+1.03
14. 14.78+4100
2.16+0.71
1,88+0.68
AIRec
38. 38.6940.06
23.9440.05
37.63
23.08
1.30
1.07
HAN-TR
35.95+2.15
19. 19.75+3.93
34.9542.02
19.13:3.74
2.09+0.80
1,83+0.75
TNAM
5.91
2.63
1.89
2.52
1.39
1.17
BPR-T
30.07
18.93
33. 30.75
19.57
1.23
0.99
Table 10: The benchmarking results of the feature-based explainable recommender models on jointly predicting the tags for informativeness, satisfaction and ratings.
In Informaatvennss
Satisfaction
Rating
Metries
NDCG
NDCG
RMSE
MAE
EFM
5.15
3.75
4.57+0.54
1.7540.21
1.66+0.00
1. 1.2300.00
LRPPM
37.8910.19
38.37
36.0610.14
20.07
1.22
0.96
SULM
42.70.0.40
42.84
40.12
24.70
1.67
1.31
MTER
6.100
4.56.0.18
5.64
2.37
2.04
1.74.0.66
AMF
28.1740.28
28.5344032
26. 26.79+0.06
16.8940.09
2.28+
1.8640.00
DERM-MLP
39.23
39.45
37.4040.07
22.93
1.30
1.064
DERM-MF
37. 37.60+0.13
37 37.49+0.17
35. 35.77++.16
21.76+0.16
1.32+0.00
1. 1.154:0.00
DERM-C
38.7740.13
39.08
36.84+0.23
22.5940.16
1.30
1.07
DERM-H
38.13+0.50
38.45+0.47
36. 36.44+0.28
22.55+0.21
1.27+0.01
1. 1.04+0.01
TRDM
33. 33.15
25.24+1.92
30.42+1.50
13.83
1.19
0.94
TRAL
6.04+
4.56+0.022
5.84+0.01
2.43+
1.20 1.29+0.00
1.08+0.00
HPTR
37 37.36+3.15
32.22+7.35
35.35+2.98
17. 17.79+3.92
1.8 10.77
1.56++.72
AIRec
39. 39.4640.11
39.72+0.07
37. 37.65+0.11
23.13 23.13:40.07
1.30+
1.08
HAN-TR
35.7943.09
11..8 16.85
34.0442.94
18.48
1.31 1
1.10
TNAM
37.01
37.01
5.89
2.52
1.36
1.15
BPR-T
34. 34.11 40.32
31.57
33.94
19.75
1.23
0.98
18

---

Table 11: The benchmarking results of the caature-bbsed explainable recommender models on jointly predicting the tags for persuasiveness, informativeness and satisfaction and ratings.
Persuasiveness
Informativeness
Satisfaction
Rating Prediction
Metrics
F1
NDCG
F1
NDCG
F1
NDCG
RMSE
MAE
EFM
11.66=0.15 8.52
4.97
3.70
1.33
2.24
1.664a.01
1.23
LRPPM
37.32=0.24
23.26
37.91
38.46 18.46
36.08
22.35
a.20
0.90
SULM
41.12:20.00
25.35
42.35
42.66.0.34
40.00.0.49
24.66
1.69
1.33
MTER
36. 36.16=0.06 22.38
36.75.0.09
36.94_0.24
34.84-20.08
21.52220.03
1.34
1.08
AMF
20.83
19.19
28.35
20.64
27.09
17.03
2.28
1.80
DHRM-MLP
23.81 38.00
10.07
39.3330.09
19.07
37.52.0.08
22.97.10.0
10.02
1.07
DERM-MF
37 37.42
23. 16..06
38.26
38.46 28.46
36.60=1.01
19
1.33
1.16
DERM-C
38. 20.05
23.53
39.03
39.20
20.1
37.19
22.79
1.30
1.08
DERM-H
37 20.64
23.36
38.5220.44
18.83
36.70.2.40
22.60
1.28
1.05
TRDM
2..50 15.6441.83
33.94
26.47 1:.89
31.79a1.17
14.77
1.19 10.00
0.94
TRAL
1..11 2.501
5.91
4.48
35.42:10.2
21.09
10.00
1.07
HPTR
38.82 12.14
39. 39.31
38. 32.01
37. 37.64:1026
21.46
1.77
1.50
AlRec
38.89 23.98
39.61
39.82
37 37.85
23.1640.08
1.30
1.07 1.07
HAN-TR
37. 37.95=0.63
38. 38.65 23.46.0.02
39. 39.01
37.07=0.82
22. 22.7440.04
1.80
1.57
TNAM
37 37.96_0.13
23.51 +0.02
38.74
38.81
5.89
2.44
1..0
1.17
BPR-T
33.41 20.44
19.39
34.37
32.15
34.29
20.11
1.24
1.00
Table 12: The benchmarking results of the models with natural language explanations in our library. For BLEU and ROUGE, the results are percentage values with "%" omitted. "_" means the evaluation metric is not available for the model.
BLEU (%)
ROUGE- (%)
ROUGE-2 (%)
Metrics
B-1
B-4
R
R
Att2Seq
19.96
3.25_
22.13
19.73
26.40.0.78
5.56
5.19
6.26
NRT
17.6741.10
2.92
20.60
16.04
30.02
5.23
4.20
7.33 40.42
PETER
17.6541.18 2.35
20.00+1.07
15.68 15.68
28,5941.44
4.99
3.95
7.00
19


Reviews:
[{"review_id": "f3iX8YyUE7", "reviewer": "Reviewer_VntB", "summary": "", "questions": "", "limitations": "- As this paper already mentioned in their conclusion section, though the explanation information is comprehensive, however, the data scale is limited. This dataset only has 2,997 users, 4,672 items, and 6,115 tags. If considering the user-item interaction, it only has 58,497, which is much fewer than some existing small recommendation dataset benchmark, such as  MovieLens 100K. Thus, it may have limited contribution. It would \n\n- This datasets may not be aligned with real-world recommender systems. Most of the short video recommendation factors are not considered in the explanation such as the trending topics, the fashion styles, the dwelling times, the session behavior, the time periods, and etc. It would be better to collect the dataset from real user usage scenario rather than ask labeler to watch the video and collect the feedback. Some complex context information may also be important for explaining the short video watching behaviors of users. \n\n- No baselines are compared for this benchmark. It would be better to see how existing SOTA RecSys methods perform on this new benchmark. \n\n\n", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The strengths are as follows:\n- This dataset contains explanations for short videos from multiple aspects and comprehensive\n- The persona information is available for  recommendation\n- The data collection process is in details and clear\n- This will motivate the current explainable recommendation research", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "T9g1L5zwFo", "reviewer": "Reviewer_zpBA", "summary": "", "questions": "", "limitations": "Yes.  ", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This dataset is constructed by about 3000 real users with different backgrounds, which contains\n(i) Comprehensive user information: this dataset has collected user profiles like age, gender, income, hobby and physiology information, i.e. personalities.\n(ii) Multi-aspect Explanation Ground Truth: this dataset provides the explanation ground truths from the perspectives of persuasiveness, informativeness and satisfaction.\n(iii) Multi-modal Explanations:  in the dataset, users can select both textual and visual explanations according to their preferences for each video.\nThis dataset will make significant contribution to recommendation system research community, including explainable recommendation, sentiment analysis and sequential recommendation.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "dpe9fyR84U", "reviewer": "Reviewer_1MRi", "summary": "", "questions": "", "limitations": "1) The authors should explicitly discuss the potential limitations of the newly constructed dataset, including possible biases in the data collection process or inherent limitations due to the specific domain (i.e., video recommendation).\n2) Consideration should also be given to addressing any potential negative societal impacts of this work. The authors may want to discuss how the dataset can be used responsibly to ensure fairness and privacy.\n3) The authors should consider discussing how the subjectivity of explanation satisfaction could potentially influence the model's evaluation.\n4) The dataset depends on subjective labeler feedback, which may introduce inherent biases and inconsistencies. Although the authors have taken measures to ensure data quality, the subjectivity remains a limitation.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1) The work offers a significant improvement over current datasets used for evaluating explainable recommendation models, addressing limitations such as absence of real-user labelling and limited aspects of explanation.\n2) The authors' efforts to ensure data quality are noteworthy. Rules to check and filter labeled results have been set, considering factors like labeling time and the consistency of a user's responses.\n3) The authors have made an extensive analysis of their dataset, including an in-depth presentation of its construction process. This transparency enhances the reproducibility of their work.\n4) The work opens new opportunities for future studies in the field of explainable recommendation.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "2LmviseUz5", "reviewer": "Reviewer_pC4V", "summary": "", "questions": "", "limitations": "I think there is no obvious limitation and potential negative societal impact of this work.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.The paper provides datasets labeled by real users with multi-aspect, multi-modal explanations, in favor of explainable recommendations, and recommendation fairness.\n\n2.The construction of the dataset is clearly described and well-designed, and detailed documentation is provided.\n\n3.Some interesting statistical analyses are provided and the dataset deserves to be further analyzed and explored.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "9Kz8H9BkLQ", "reviewer": "Reviewer_ZxYM", "summary": "", "questions": "", "limitations": "1. It is recommended that the authors compare the distribution gap between the 3,000 labelers and the real users to ensure the representativeness of the labeling process. By conducting such a comparison, the authors can assess how closely the labelers' behavior aligns with that of actual users, providing valuable insights into the dataset's reliability and validity.\n2. Providing analysis and justification for the manual rules implemented in the dataset construction process would enhance the transparency and understanding of the dataset's quality. By presenting the rationale and empirical evidence supporting these rules, the authors can strengthen the confidence in the dataset and address any concerns regarding the rationality of the rules.\n3. Conducting case studies to showcase the evaluation of multi-aspect explanations would be beneficial in demonstrating the dataset's capability. By selecting representative examples and thoroughly evaluating the explanations across multiple aspects, the authors can provide concrete evidence of the dataset's ability to handle and evaluate complex recommendation scenarios.\n", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The motivation behind this work is compelling as it addresses several limitations present in previous studies. These limitations include discrepancies between labeled results and actual user behavior, evaluations focused on a single aspect, and inadequate sizes of labeled datasets.\n2. The contributions are substantial. This library is an invaluable asset for researchers and practitioners in the field, equipping them with a comprehensive toolkit to construct and assess explainable recommendation systems effectively.\n3. The authors have conducted an extensive array of analyses, providing a thorough evaluation of the proposed dataset. These analyses showcase the dataset's performance comprehensively, offering insights into its strengths and capabilities.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors present a new dataset for explainable recommendations, based on multi-modal short video recommendations, and they give extensive analysis of the dataset.

The reviewers generally agreed that this paper gives a substantial contribution, especially because it overcomes limitations of existing datasets. Furthermore, the reviewers appreciated the strong methodology and execution of the paper.

The reviewers noted weaknesses such as the limited data scale, potentially limited alignment with real-world recommender systems, and data diversity. The authors put in significant effort during the discussion period to improve the paper, and in the end all reviewers were positive about the paper.

Overall, the paper is a strong contribution, and I recommend acceptance.
Decision: Accept (Poster)