SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xu Chen $^{1,2}$ , Jingsen Zhang $^{1,2,*}$ , Lei Wang $^{1,2,*}$ , Quanyu Dai $^{4,\dagger}$ , Zhenhua Dong $^{4}$ , Ruiming Tang $^{4}$ , Rui Zhang $^{5}$ , Li Chen $^{3}$ , Wayne Xin Zhao $^{1,2}$ , Ji-Rong Wen $^{1,2}$
<sup>1</sup>Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China
$^{2}$ Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
<sup>3</sup>Department of Computer Science, Hong Kong Baptist University, Hong Kong
$^{4}$ Huawei Noah's Ark Lab, China  $^{5}$ www.ruizhang.info
Explainable recommendation has attracted much attention from the industry and academic communities. It has shown great potential to improve the recommendation persuasiveness, informativeness and user satisfaction. In the past few years, while a lot of promising explainable recommender models have been proposed, the datasets used to evaluate them still suffer from several limitations, for example, the explanation ground truths are not labeled by the real users, the explanations are mostly single-modal and around only one aspect. To bridge these gaps, in this paper, we build a new explainable recommendation dataset, which, to our knowledge, is the first contribution that provides a large amount of real user labeled multi-modal and multi-aspect explanation ground truths. In specific, we firstly develop a video recommendation platform, where a series of questions around the recommendation explainability are carefully designed. Then, we recruit about 3000 high-quality labelers with different backgrounds to use the system, and collect their behaviors and feedback to our questions. In this paper, we detail the construction process of our dataset and also provide extensive analysis on its characteristics. In addition, we develop a library, where many well-known explainable recommender models are implemented in a unified framework. Based on this library, we build several benchmarks for different explainable recommendation tasks. At last, we present many new opportunities brought by our dataset, which are expected to promote the field of explainable recommendation. Our dataset, library and the related documents have been released at https://reasoner2023.github.io/.
Recommender system has become an indispensable component in a large amount of real-world applications, ranging from e-commence [32] and entertainment [46] to education [30] and health-caring [37]. Basically, recommender system bridges the content providers with the users. For the content providers, better recommendations can effectively expose the items which are more likely to be clicked or purchased by the users, and thus, achieve better business targets. For the users, an effective recommender model can help to quickly discover the items of interest, and thus, save the user time and alleviate the problem of information overloading. Recently, explainable recommendation has been recognized as an important research direction [36, 2]. Essentially, providing explanations for the recommended items can further enhance the utilities of the content providers and users. For
Table 1: Comparison between different explainable recommendation datasets. "Real-U" means that whether the ground truths are labeled by the real users. "Multi-E" indicates that whether the dataset has multi-aspect explanations. "App-S" represents that whether the dataset can be generally used in different studies. "Multi-M" shows that whether the dataset contains multi-modal explanations. In addition, we also present the number of labelers in each dataset.
Reference	Real-U	Multi-E	App-S	Multi-M	# Labelers
Extra [20]	✓	X	✓	X	<100
VECF [8]	X	X	✓	X	<100
AFRec [24]	X	X	X	X	<100
Ex3 [40]	X	X	X	X	<100
MTER [38]	X	✓	X	X	<100
JRASA [28]	✓	✓	X	X	100-300
UseFCE [29]	✓	✓	X	X	<100
ArgEx [15]	✓	✓	X	✓	100-300
REASONER	✓	✓	✓	✓	≈ 3000
the content providers, appropriate explanations can better persuade the users to interact with the recommended items [11]. For the users, reasonable explanations can reveal more informative item features to facilitate user decisions [18].
In the past few years, while a lot of promising explainable recommender models have been proposed, the datasets used to evaluate them still suffer from several limitations: to begin with, many work evaluate the models by recruiting annotators to label the ground truths on existing public datasets [27]. However, the annotators are not the real users in the datasets. Thus, the labeled results may deviate from the real user behavior reasons. Then, previous work mostly evaluate the explanations from only one aspect, for example, "whether the explanation can better persuade the users" [33] or "whether the explanation can reveal more informative item features" [6, 41]. However, the recommendation explanations may serve for different purposes [1], thus, these datasets may fail to comprehensively evaluate the model explainability. At last, while recent years have witnessed a few pioneer studies on evaluating the explanations from multiple aspects [39, 1, 3], the labeled datasets are not large enough, which may fail to evaluate the models in a statistically sufficient manner.
To alleviate the above problems, in this paper, we propose to build a large real user labeled dataset, which can evaluate the recommendation explanations from multiple aspects. We call our dataset as REASONER, where RE indicates that the ground truths are labeled by the real users, ASON represents that our dataset can evaluate multi-aspect explanations, ER is short for explainable recommendation. While this dataset is important for the explainable recommendation domain, it is not easy to build it. To begin with, if we build our dataset by extending an existing one, then it is hard to find the real users in the original dataset to label the explanation ground truths. If we do not rely on an existing dataset, then how to obtain the user-item interactions can be a challenge. In addition, different from the other domains, where the labeling tasks are usually objective (e.g., identifying the contents of an image), we have to collect user subjective data, which is more sensitive to the designs of the labeling tasks, for example, "how to ask each question" and "what are the orders of the questions". Without proper designs, the annotators may misunderstand the questions or can be interfered by the external factors, which may lower the labeling accuracy. Last but not least, how to control the dataset quality is also a challenge. Our dataset aims to collect user subjective preferences, it can be difficult to purify it by directly judging whether the labeled results are right or wrong, since different users may have various opinions. We need to find effective rules to filter the results according to their reasonableness.
To overcome the above challenges, we propose to build our dataset in a two-step manner. To begin with, we develop a video recommendation platform, and carefully design a series of questions around the recommendation explainability. For example, "which features are the reasons that you would like to watch this video?" and "which features are most informative for this video?". Then, we recruit about 3000 labelers to use our platform and answer the above questions. This makes that the labelers of the explanation ground truth are exactly the real users that produce the user-item interactions<sup>3</sup>. To control the quality of our dataset, we set many rules to check and filter the labeled results. For example, "whether the labeling time is too short?" and "whether the answers of the same
3 In the following, we do not discriminate the users and labelers.
Figure 1: (a) All the information collected in our dataset. (b) Different ground truths for the recommendation explanations with different purposes.
user conflict with each other?". Beyond the above dataset, we also develop a library, which contains many well-known explainable recommender models. With this library, we build several benchmarks based on our dataset for different explainable recommendation tasks. Due to the space limitation, we present the details on the library and benchmarks in the appendix. At last, we point out several new opportunities brought by our dataset. We have released the above-mentioned dataset, library and documents at https://reasoner2023.github.io/ under a CC BY-NC 4.0 license.
The datasets for evaluating explainable recommender models can be divided into two classes. In the first class, the datasets are usually generated by recruiting labelers to answer the questions around the model explainability. For example, [8] asks the labelers to imagine themselves as the real users and label the explanation ground truth. [24] requires the labelers to compare the explanations generated from the designed models and baselines. In [40, 22], the labelers have to evaluate the helpfulness of the explanations in terms of facilitating better user decisions. [35, 38] let the labelers to express their satisfactions and rate the informativeness of the explanations. [20] regards the user reviews as the explanation ground truths, and projects similar review sentences into the same cluster to facilitate model optimization. In this type of datasets, the labelers are usually not the real users, thus the labeled results may not accurately reflect the real user preference. In the second class, the explanation dataset is completely constructed by the labelers. For example, the authors in [28] recruit 286 labelers to evaluate the transparency and effectiveness of some given explainable recommender models. In [29], the authors employ 20 annotators to study the effectiveness of the feature-based explanations. In [15], 152 annotators are recruited, and the target is to study the user perception on the explanation types. While the results in these datasets can well represent the real user preference, the labeling questions are highly model-specific, which can be hardly used in the other studies. Comparing with the above datasets, REASONER has many significant advantages. For example, it is labeled by the real users and can be used to evaluate different models. In addition, it has multi-aspect explanation ground truths with different modalities. For clearly understanding the advantages of REASONER, we compare it with the previous explainable recommendation datasets in Table 1.
To align the labelers with the real users, our dataset is constructed based on two steps. The first step is building a recommendation platform, and designing many questions about the recommendation explainability. The second step is recruiting labelers to use the above platform and collecting their behaviors and answers to the above questions. In the following, we detail these two steps.
3.1 Building the Recommendation Platform
Comparing with traditional recommender systems, our platform is specially designed for collecting the user explainable behaviors. In this section, we firstly introduce the items on the platform and the
explanation candidates that the labelers can select from. Then, we describe the questions designed for collecting the explanation ground truths. Finally, we present the complete labeling process for each labeler based on our platform.
3.1.1 Items
We select videos as the recommendation items, since they are content intensive for providing sufficient explanation candidates. Considering that too long labeling time may distract the user attention, the length of the videos are confined to be shorter than three minutes. The selected videos can cover different categories, such as technology and music, which are expected to comprehensively represent the user preferences. All the videos are from an industrial short-video website.
3.1.2 Explanation candidates
We collect the explanation ground truths by asking the labelers to select from many video features, e.g., "which features are the reasons that you would like to watch this video?" The most important features are:
- Tags. These are textual features, which are collected from three sources: (1) the original tags provided by the video authors. (2) The overall comments, which are posted by the viewers after watching the video. (3) The time-synchronized comments, which are provided by the viewers as they watching the video in a real-time manner [25]. The above information is from the same website as that of the videos. Since the overall and time-synchronized comments are originally raw texts, we process them by neural language processing tools [43] to extract the candidate tags. We combine all the tags from different sources, and manually remove the tags which are duplicated, meaningless or containing swear words.
- Previews. These are visual features, which are extracted from the videos. To make the previews more representative, we use a standard key frame extraction tool for preview generation. We manually check the similarities between different previews, and remove the duplicated ones. For each video, we finally generate five previews for the labelers.
In addition to the tags and previews, we also present the video titles and brief introductions to the labelers. Besides, the labelers can also write the answers in a reserved text box, if they find that all the above features cannot represent their ideas.
3.1.3 Labeling questions
As mentioned before, the recommendation explanations may serve for different purposes. For the content provides, the explanations can be used to enhance the recommendation persuasiveness. For the users, the explanations can be leveraged to provide more informative item features for faster user decisions. As a result, there can be multiple ground truths for the explanations. For example, in Figure 1(b), the user is a fan of Tom Cruise. If we explain the video by "Tom Cruise", then the user may click the video, and the purpose of the content provided is achieved. From this perspective, "Tom Cruise" is the ground truth for improving the recommendation persuasiveness. However, while Tom Cruise is in the video, he only appears for a very short time, and the real leading actor is Chris Pine. If we explain the video by "Chris Pine", then the user can make more accurate and informed decisions. From this perspective, "Chris Pine" is the ground truth for enhancing the recommendation informativeness. To obtain the explanation ground truths from multiple aspects, we design a series of questions for the labelers to answer, and we introduce the most important ones as follows:
Q1. Which features are the reasons that you would like to watch this video? This question is asked before the user watches the video. The selected features are the user expectations, which reveal the reasons that motivate the users to watch the videos. This question aims to obtain the explanation ground truth in terms of enhancing the recommendation persuasiveness.
Q2. Which features are most informative for this video? This question is asked after the user has watched the video. At this time, the user has fully understood the video, thus the selected features can objectively represent the video. By this question, we would like to obtain the explanation ground truth in terms of enhancing the recommendation informativeness.
4https://github.com/FFmpeg/FFmpeg
Figure 2: The complete labeling processing for each user to collect our dataset.
Q3. Which features are you most satisfied with? This question is also asked after the user has watched the video. It aims to obtain the explanation ground truth in terms of improving the user satisfaction. Comparing with Q1, this ground truth is obtained after the user has fully understood the video contents. Comparing with Q2, this ground truth involves the user subjective factors.
Q4. Please give a rating (from 1 to 5) to this video according to your preference. This question is designed to collect the user overall preference on the video.
Q5. How do you comment this video? This question aims to collect more detailed user preference on the video, and the user can express her opinions in a text box.
By the above questions, we collect the explanation ground truths from three aspects, that is, persuasiveness (Q1), informativeness (Q2) and user satisfactions (Q3). In summary, we present all the information collected in our dataset in Figure 1(a).
3.1.4 Workflow of the labeling task
Based on the above questions, in this section, we introduce the complete labeling process for each user (see Figure 2). In specific, there are five important steps:
Step 1: User sign up. To begin with, the users have to sign up our platform by providing their basic personal information. And then, we also require the users to complete a test on big five personality. This test includes 15 questions, which are standard and widely used over the world<sup>5</sup>. Based on the answers to these questions, we can understand the personality of a user from five dimensions, that is, openness, conscientiousness, extraversion, agreeableness and neuroticism. We refer the readers to [12, 13] for more detailed introduction on the personality test. The big five test may provide us with the opportunities to study the influence of different user personalities on the selection tendency of the recommendation explanations. For the above personal information, we have obtained the user permissions to use it for research purposes. To protect the user privacy, our dataset has been desensitized based on IDs.
Step 2: Platform recommendation. After the user has logged into the platform, the system will recommend her with three videos. For each video, the features introduced in Section 3.1.2 are also presented to the user. The videos are recommended in a completely random manner, that is, different videos are exposed to the user with the same probability. This setting aims to collect more diverse user preferences without the information cocoons effect. In addition, our dataset may also contribute to the field of debiased recommendation [7], where collecting uniform item exposure traffic can be too expensive in real-world scenarios [45].
Step 3: User selection and before-watching question answering. Once the user has received the recommendations, for each video, she has to carefully check its features, and decides whether she would like to watch this video. If the user decide to watch the video, then we require her to answer Q1 in Section 3.1.3, that is, "which features are the reasons that you would like to watch this video?" Otherwise, the user has to select the features which make her not want to watch the video.
Step 4: Video watching. In this step, the user has to watch the video. Since the video only lasts for a short time (<3min), the user can quickly understand it with little effort. For the videos that the user does not want to watch in the previous step, we also require her to watch them and answer the
<sup>5</sup>https://www.psyto toolkit.org/survey-library/big5-bfi-s.html
questions in the following steps, which may help to study whether the videos of interests can be missed due to the improper explanations.
Step 5: After-watching question answering. After the video has been watched, we require the user to answer a series of questions. To begin with, the user has to provide her rating and comments on the video (corresponding to Q4 and Q5). Then, we require the user to select the features which are most informative for the videos (corresponding to Q2). At last, the user needs to select the features that they are most satisfied with (corresponding to Q3).
For each user, step 1 is only done for once, step 2 to step 5 are repeated for at least six times. Since the system recommends three videos every time, each user receives more than  $3 \times 6$  (=18) recommendations in the complete labeling process.
3.2 Collecting the Dataset
Based on the above platform, we then recruit labelers to use it and complete the above labeling process. To control dataset quality, we design many rules to purify the labeled results.
3.2.1 Labelers
To guarantee the dataset quality, we cooperate with a professional company to recruit the labelers. For each labeler, we require that she should have been employed by the company before and have better working records. In addition, we need that the background of the labelers should be diverse enough. In total, we recruit about 3000 labelers, and we pay each labeler  $8 USD for completing the labeling task. The overall cost of our dataset is about$ 28,000 USD.
3.2.2 Dataset quality control
In traditional labeling tasks, such as the image recognition or entity tagging, the ground truths are objective, and it is easy to judge whether the labeled results are right or wrong, which makes it not hard to control the dataset quality. However, our dataset aims to collect the user subjective and personalized preferences. There is no strict right or wrong, which brings difficulties to control the dataset quality. To solve this problem, we firstly set a series of rules to judge whether the labeled results are reasonable, and then remove the unreasonable ones. In specific, the rules are mainly designed around the following aspects:
A1. Labeling time. On our platform, the time cost of the user for answering each question has been recorded in our database. For each question, we gather all the time costs of the users, and check whether there are outliers. For all the questions, if the same user is identified as an outlier for more than five times, then the user and all her labeled results are removed from the dataset.
A2. Repeated answers. For this aspect, we check whether the user answers to different questions are highly similar. For example, some labelers may always select the first three tags for Q1 across all the videos. Some labelers may use the same comment for different videos. In general, if a user has more than 1/3 similar answers, then we remove this user and her labeled results.
A3. Conflict answers to the same question. To check whether the labelers have carefully done our task, we intentionally insert many questions which have been answered by the labelers before. For the same labeler, we examine whether the answers to the same question are consistent. If a user has more than 1/3 inconsistent answers, then the user is removed.
A4. Conflict answers across different questions. Intuitively, if the labelers have carefully completed our task, then there should be some reasonable relations between different answers. For example, if the user give a very low rating, then the comments should be mostly negative. At first, we planned to summarize all the unreasonable relations and develop an automatic tool to identify them. However, we find that the relation patterns between different answers are too complex, and building a reliable enough tool may cost too much time. To solve these problems, we recruit five students in our lab to manually check each record in the dataset. To begin with, we require each student to independently check the dataset, and find out the unreasonable relations. Then, we merge all the unreasonable results, and let the students to vote for each one to determine whether it is reasonable or not. For the unreasonable results, we remove the corresponding records in the final datasets.
Figure 3: Statistics of the users and videos.
Figure 4: (a) The distribution of the number of users who watched videos in different ranges. (b) The distribution of the number of videos watched by users in different ranges. (c) The distribution of the user review length in words.
4 Statistical analysis
In this section, we introduce the basic statistics of our dataset, where we focus on the following aspects: (1) the basic statistics of the users and videos. (2) The statistics on the user-video, user-tag and video-tag interactions. (3) The similarities between the ground truths for different explanation aspects. (4) The statistics on the users who tend to select single- or multi-modality explanations. (5) The statistics on the length of the user reviews.