SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In our dataset, there are in total 2997 users, 4672 videos and 6115 tags. In Figure 3(a-d), we present the statistics of the users based on their genders, incomes, educations and ages. We can see the users can cover different populations, which makes our dataset more general and representative. In Figure 3(e-f), we present the statistics of the videos based on their categories and lengths. We can see, the categories of the videos are diverse, and for different categories, there are almost the same number of videos, which can help to explore more comprehensive and unbiased user preferences.
In Figure 4(a) and 4(b), we present the distribution of the user-video interactions. Each point  $(x, y)$  represents that there are  $y$  users (or videos) who interact with  $x$  videos (or users). For clear presentation, we merge the number of videos (or users) into four bins. In Figure 5, we show the distribution of the user-tag interactions. Since there are three types of tags for Q1, Q2 and Q3 as mentioned in Section 3.1.3, we present them separately. Each point  $(x, y)$  in these figures represents that there are  $y$  users who totally select  $x$  tags across different videos, where the x-axis is also clustered into four bins. In Figure 6, we show the distribution of the video-tag interactions. Similar to Figure 5, we present the results for Q1, Q2 and Q3 separately. Each point  $(x, y)$  represents that there are  $y$  videos whose tags have been totally selected for  $x$  times by different users.
In our dataset, we label the ground truths for multi-aspect explanations. An interesting problem is that how different are the tags selected by a user for the same video on different explanation aspects. To answer this question, we measure the tag similarities based on the Jaccard distance. For every two explanation aspects, we firstly compute the tag similarity for each user-item pair, and then report the result by averaging all the user-video pairs in Table 2. We can see, for different explanation purposes,
Figure 5: Distribution of the user-tag interactions. These three graphs respectively display the distribution of the number of users who label different numbers of Q1, Q2, and Q3 tags.
Figure 6: Distribution of the video-tag interactions. These three graphs respectively display the distribution of the number of videos labeled with different numbers of Q1, Q2, and Q3 tags.
the overlaps between the selected tags are usually not large (about 0.35 to 0.45). This demonstrates that one may need to use different tags to achieve different explanation purposes.
In our dataset, the users can select multi-modal explanations. Here, we compute the ratio between the numbers of users who select multi-modal explanations and all the users. We present the results for Q1, Q2 and Q3 separately in Table 3, where we can see about  $38\%$  users tend to select both textual and visual explanations, while the other users would like to select single-modal explanations. At last, we present the distribution of the user review lengths in Figure 4(c). We can see most of the reviews contain more than ten words, which can comprehensively reveal the user preferences.
Table 2: The average Jaccard similarity for different explanation aspects.
Explanation aspects	Q1-Q2	Q1-Q3	Q2-Q3
Jaccard similarity	0.423	0.364	0.410
Table 3: Ratios between the number of users who select multi-modal explanations and the total number of users.
Explanation aspect	Q1	Q2	Q3
Ratio	0.389	0.390	0.359
5 Library
Besides the above dataset, we also develop a library, where many well-known explainable recommender models are implemented in a unified framework. With this library, we build several benchmarks based on our dataset for different explainable recommendation tasks. Due to the space limitation, more details on the library and benchmarks are presented in the appendices.
5.1 The Structure of the Library
We show the structure of our library in Figure 7. The configuration module is the base part of the library and responsible for initializing all the parameters. We support three methods to specify the parameters, that is, the command line, parameter dictionary and configuration file. Based on the configuration module, there are four higher layer modules, that is,
Data module: this module converts the raw data into the model inputs. There are two components: the first one is responsible for loading the data and building vocabularies for the user reviews. The second part aims to process the data into the formats required by the model inputs, and generate the sample batches for model optimization.
Figure 7: The structure of our library. There are six parts including the configuration, data, model, trainer, evaluation and execution modules.
Model module: this module implements the explainable recommender models. There are two types of methods in our library. The first one includes the feature-based explainable recommender models, and the second one contains the models with natural language explanations. We delay the detailed introduction of these models in the next section.
Trainer module: this module is leveraged to implement the training losses, such as the Bayesian Personalized Ranking (BPR) and Binary Cross Entropy (BCE). In addition, this module can also record the complete model training process.
Evaluation module: this module is designed to evaluate different models, and there are three types of evaluation tasks, that is, rating prediction, top-k recommendation and review generation.
Upon the above four modules, there is an execution module on the upper-most layer. It is responsible for optimizing models for different tasks, such as rating prediction and review generation.
5.2 Examples for Using Our Library
In this section, we introduce how to use our library. We present a simple example in Figure 8(a), where one can directly execute tag_prediction.py or review_generate.py to run a feature-based or review-based model, respectively. In each of these commands, one needs to specify three parameters to indicate the names of the model, dataset and configuration file, respectively.
Take tag_prediction.py for example, it sequentially executes the following steps: (1) Configuration. In this step, the parameters related to the model architecture and optimization process from different sources (commend line, configuration dictionary and files) are integrated into a dictionary. (2) Data loading. In this step, the dataloader is selected according to model type. For the review-aware models, this step reads all the records and build the vocabulary. (3) Data Formatting. The training, validation and test sets are processed into the formats required by the model input in a sample batch manner. (4) Initialization. The corresponding model class will be defined and initialized according to the parameter values in configuration. (5)-(6) Training. Selecting the optimization approach to train the model. (7) Evaluation. Measuring the model performance on different tasks.
Our library is highly extensible, and there are three steps to realize a new model: (1) implementing the basic functions of the model, including the model architecture, preference score prediction etc. (2) Customizing the training approaches in train.py. (3) Indicating the parameters in the config file.
6 New Opportunities
Based on the information collected in our dataset (see Figure 1(a)), we believe that our dataset can bring the following new opportunities for the domain of explainable recommendation:
- Explainable recommendation with extensive personal information. In most of the previous studies, the datasets used for training explainable recommender models do not contain sufficient personal information. By our dataset, people can access the desensitized user profiles, which may facilitate a lot research directions. For example, one can study the explanation fairness, where the sensitive variables can be the age, education, income and so on. In addition, people can also use the extensive personal information to enhance the predictive accuracy of the explanations.
(b)
Figure 8: (a) Examples of the feature- and review-based models. (b) The running flow of our library.
- Multi-aspect explainable recommendation. For almost all the previous explainable recommender models, there is only one aspect ground truths for the explanations in the dataset. Thus, the learned models may not perform well on the other aspects. For example, if the model is only trained to improve the recommendation persuasiveness, then the generated explanations may not help to enhance (or even hurt) the recommendation informativeness. By our dataset, people can simultaneously consider different explanation aspects, and learn a more comprehensive explainable model to serve the on-line users. We argue that there can be much room in this direction, for example, if different explanation purposes have contradictions, how to design models to better overcome such contradictions.
- Multi-modal explainable recommendation. To our knowledge, most previous explainable recommender models only consider single-modal explanations. However, in the real-world scenarios, the users always need to perceive multi-modal information. By our dataset, people can conduct a lot of studies around the multi-modal explanations, for example: (1) which parts of users would like to select textual or visual features? (2) How to build a unified model to generate both textual and visual explanations? (3) What is the relation between the user personalities and the explanation modalities? And how to build a model to predict the explanation modality given a user personal information?
Beyond the above research opportunities on explainable recommendation, our dataset can also contribute to other research domains. For example, since our dataset is collected with uniform item exposure probabilities, people can use our dataset to study debiased recommendation. In our dataset, we have collected user big five personality information, thus, one can use it to investigate psychology-informed recommendation.
In this paper, we introduced a new explainable recommendation dataset, where we detailed its construction process and extensively analyzed its characters. Our dataset can support multi-modal explanations, and the ground truths annotators are exactly the real users who produce the interactions. In addition, our dataset supports multi-aspect explanations, and we have about 3000 users in the dataset. We developed an explainable recommendation toolkit, and built several benchmarks based on the above dataset for different explainable recommendation tasks. In addition, we also analyze the new opportunities brought by our dataset. Actually, our dataset is only a small step towards more measurable explainable recommendation, and there is still a long way to go. In the future, we plan to extend our dataset to the other domains, such as news, music and so on. In addition, we would also like employ more labelers to enlarge our dataset.
This work is supported in part by National Key R&D Program of China (2022ZD0120103), National Natural Science Foundation of China (No. 62102420), Beijing Outstanding Young Scientist Program NO. BJWZYJH012019100020098, Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China, Public Computing Cloud, Renmin University of China, fund for building world-class universities (disciplines) of Renmin University of China, Intelligent Social Governance Platform. The work is sponsored by Huawei Innovation Research Programs. We gratefully acknowledge the support from Mindspore<sup>6</sup>, CANN (Compute Architecture for Neural Networks) and Ascend Ai Processor used for this research. We sincerely thank the contributions of Zhenlei Wang, Rui Zhou, Kun Lin, Zeyu Zhang, Jiakai Tang and Hao Yang from Renmin University of China for checking the unreasonable results in the dataset construction process.
[1] Krisztian Balog and Filip Radlinski. Measuring recommendation explanation quality: The conflicting goals of explanations. In SIGIR, 2020.
[2] Krisztian Balog, Filip Radlinski, and Shushan Arakelyan. Transparent, scrutable and explainable user models for personalized recommendation. In SIGIR, pages 265-274, 2019.
[3] Krisztian Balog, Filip Radlinski, and Alexandros Karatzoglou. On interpretation and measurement of soft attributes for recommendation. In SIGIR, pages 890-899, 2021.
[4] Konstantin Bauman, Bing Liu, and Alexander Tuzhilin. Aspect based recommendations: Recommending items with the most valuable aspects based on user reviews. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 717-725, 2017.
[5] Bo Chen, Yue Ding, Xin Xin, Yunzhe Li, Yule Wang, and Dong Wang. Airec: Attentive intersection model for tag-aware recommendation. Neurocomputing, 421:105-114, 2021.
[6] Chong Chen, Min Zhang, Yiqun Liu, and Shaoping Ma. Neural attentional rating regression with review-level explanations. In Proceedings of the 2018 world wide web conference, pages 1583-1592, 2018.
[7] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 2020.
[8] Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, and Hongyuan Zha. Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation. In SIGIR, pages 765-774, 2019.
[9] Xu Chen, Zheng Qin, Yongfeng Zhang, and Tao Xu. Learning to rank features for recommendation over multiple categories. In SIGIR, pages 305-314, 2016.
[10] Li Dong, Shaohan Huang, Furu Wei, Mirella Lapata, Ming Zhou, and Ke Xu. Learning to generate product reviews from attributes. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 623-632, 2017.
[11] Fatih Gedikli, Dietmar Jannach, and Mouzhi Ge. How should i explain? a comparison of different explanation types for recommender systems. International Journal of Human-Computer Studies, 72(4):367–382, 2014.
[12] Lewis R Goldberg. The development of markers for the big-five factor structure. Psychological assessment, 4(1):26, 1992.
<sup>6</sup>https://www.mindspore.cn
A The Implemented Models in library
In our library, we implement two types of explainable recommender models, which are widely studied in the research community. The first one are feature-based explainable recommender models, where the features can be the tags, item aspects and so on. The second one are the models with natural language explanations.
More specifically, we implement the following representative feature-based explainable recommender models:
EFM [42] predicts the user preferences and generates explainable recommendations based on explicit product features and user opinions from the review information.
TriRank [14] models the user-item-aspect ternary relation as a heterogeneous tripartite graph based on user ratings and reviews, and it devises a vertex ranking algorithm for recommendation.
LRPPM [9] is a tensor-matrix factorization algorithm which captures the user preferences using ranking-based optimization objective over various item aspects.
SULM [4] enhances recommendations by recommending not only item but also the specific aspects by using aspect-level sentiment analysis.
MTER [38] is a tensor factorization method which models the task of item recommendation using a three-way tensor over the users, items and features. We omit the modeling of the opinions in the original implementation for adapting our data.
AMF [16] improves the recommendation accuracy by using the auxiliary information extracted from the user review aspects.
TRDM [48] introduces a two-stage approach to generate accurate item recommendations and effective tag-based potential features simultaneously for enhancing recommendation accuracy and diversity.
TRAL [47] proposes attention-based learning to capture diverse tag-based features, and compress these features with an attention pooling layer to enhance recommendation accuracy.
HPTR [44] employs hyperbolic distance to measure semantic relevance between entities, which better captures hierarchical structures presented in tag information.
AIRec [5] enhances tag-aware recommender system by employing a hierarchical attention network to capture multi-aspect preferences and leveraging tag intersection to improve conjunct feature learning.
HAN-TR [34] captures distinct user preferences and informative elements by employing separate attention networks for element-level influence and information-level attentiveness.
TNAM [17] addresses the issues of tag weight assignment in recommender systems by introducing a tag-based neural attention network that captures users' specific tag attention.
BPR-T [19] addresses high dimension and sparsity issues of tagging information by integrating tag mapping into a Bayesian personalized ranking collaborative filtering model.
In addition to the above shallow models based on matrix factorization, we also implement the following deep feature-based recommender models (called DERM for short):
DERM-MLP is a deep recommender model for jointly predicting the ratings and tags. The two tasks share the set of user/item/tag embeddings. The hidden states as well as the tag embeddings are put into different layers corresponding to the different tasks.
DERM-MF firstly obtains a hidden state based on the user/item embeddings using matrix factorization, and then the outputs are computed by a neural network.
DERM-C combines matrix factorization and Multi-Layer Perceptron (MLP) to derive the hidden states, and the outputs are merged in a concatenated manner.
DERM-H leverages the tags to profile the users and items, and then use the same architecture as DERM-MLP for predicting the ratings and tags.
For the models with natural language explanations, we implement the following representative methods:
Table 4: Statistics of the datasets.
Dataset	REASONER
# Users	2,997
# Items	4,672
# Tags	6,115
# Interactions	58,497
Avg. # words / review	17
Att2Seq [10] is a review generation model which uses LSTM as the decoder, and output the texts directly based on the user/item IDs and rating information.
NRT [23] simultaneously predicts the reviews and ratings based on the input user-item pair, where the two tasks share the same embedding and hidden layers.
PETER [21] leverages Transformer to generate the user reviews, which is a state-of-the-art review generation model.
B Benchmark
B.1 Experiment Setup
Considering that we have three types of ground truths for the explanations, we evaluate the model performance by predicting the tags for Q1, Q2, Q3, Q1+Q2, Q2+Q3, Q1+Q3 and Q1+Q2+Q3, respectively. When we have to predict multiple types of ground truths, we extend the original models to their multi-task versions by sharing the embedding parameters. We randomly split the dataset into the training, validation and testing sets according to the ratio of 8:1:1. For the review generation task, we use the most 20,000 frequently mentioned words to construct the vocabulary, and the maximum length of the generated sentences is set to 17, which is equal to the average length of reviews in dataset. The dataset statistics are presented in Table 4. For all the models, the batch size is set as 256. We tune the other key hyper-parameters by grid search. In specific, we tune the learning rate and the weight of L2 regularization in the range of [0.1, 0.01, 0.001, 0.0001] and [0.001, 0.0001, 0] respectively. For the deep models, we tune the hidden size and the layer number in the range of [32, 64, 128, 256] and [1, 2, 3, 4] respectively. More details of the experiment setting are shown in our project, which has been released at https://reasoner2023.github.io/. We use RMSE and MAE as the metrics to evaluate the performance of the rating prediction task. For the task of tag prediction, F1 and NDCG are selected to evaluate the model performance. To evaluate the quality of the generated reviews, we leverage the metrics including BLEU [31] and ROUGE [26] for model comparison.
B.2 Experiment Results
The comparison results of the feature-based explainable recommender models on the tasks of tag and rating predictions are presented in Table 5-11. The comparison results of the models with natural language explanations on the task of review generation and rating prediction are presented in Table 12. We use the tags with top 10 prediction scores to calculate F1 and NDCG, and the results are percentage values with "%" omitted. For RMSE and MAE, a lower value indicates better performance. For each evaluation metric, we use bold fonts to label the best performance. Since the TriRank we implemented does not support to predict multiple types of tags simultaneously, we omit it in the corresponding tables.
Table 5: The benchmarking results of the feature-based explainable recommender models on predicting the tags for persuasiveness and ratings.
Metrics	Persuasiveness	Rating Prediction
F1	NDCG	RMSE	MAE
EFM	26.99±0.35	18.89±0.23	1.68±0.00	1.24±0.01
TriRank	18.36±0.07	13.98±0.06	2.90±0.00	2.58±0.00
LRPPM	37.31±0.23	23.25±0.10	1.22±0.00	0.96±0.00
SULM	41.68±0.63	25.77±0.22	1.65±0.08	1.30±0.06
MTER	5.66±1.74	2.65±1.13	2.27±0.64	1.96±0.62
AMF	27.93±0.08	17.62±0.17	2.28±0.00	1.86±0.00
DERM-MLP	37.74±0.12	23.45±0.02	1.30±0.01	1.06±0.01
DERM-MF	36.57±0.14	21.49±0.17	1.32±0.00	1.14±0.00
DERM-C	37.17±0.19	23.21±0.01	1.30±0.00	1.07±0.01
DERM-H	35.58±0.07	22.02±0.18	1.27±0.01	1.04±0.01
TRDM	30.24±1.57	13.44±1.11	1.19±0.00	0.95±0.01
TRAL	5.93±0.05	2.61±0.02	1.29±0.00	1.08±0.00
HPTR	38.61±0.20	23.05±1.28	2.16±0.71	1.88±0.68
AIRec	38.05±0.07	23.07±0.03	1.31±0.00	1.08±0.01
HAN-TR	34.96±0.60	18.58±3.37	2.10±0.80	1.83±0.75
TNAM	5.97±0.01	2.60±0.01	1.37±0.01	1.17±0.01
BPR-T	32.70±0.62	18.32±0.82	1.23±0.00	0.98±0.01
Table 6: The benchmarking results of the feature-based explainable recommender models on predicting the tags for informativeness and ratings.
Metrics	Informativeness	Rating Prediction
F1	NDCG	RMSE	MAE
EFM	5.38±0.28	3.97±0.19	1.68±0.00	1.24±0.01
TriRank	18.78±0.10	14.50±0.09	2.90±0.00	2.58±0.00
LRPPM	37.85±0.22	38.35±0.15	1.22±0.00	0.96±0.00
SULM	43.25±0.59	42.97±0.40	1.65±0.08	1.30±0.06
MTER	8.40±0.86	6.13±0.68	2.04±0.68	1.74±0.66
AMF	28.63±0.24	28.95±0.29	2.28±0.00	1.86±0.00
DERM-MLP	38.60±0.06	38.99±0.03	1.30±0.01	1.06±0.01
DERM-MF	37.10±0.09	35.37±0.14	1.32±0.00	1.14±0.00
DERM-C	37.96±0.10	38.43±0.05	1.30±0.00	1.07±0.01
DERM-H	36.36±0.55	35.85±0.41	1.29±0.01	1.06±0.01
TRDM	31.49±2.84	24.09±2.80	1.19±0.00	0.95±0.01
TRAL	6.04±0.06	4.59±0.01	1.28±0.00	1.08±0.00
HPTR	39.55±0.09	37.73±1.88	1.27±0.07	1.05±0.09
AIRec	38.90±0.06	39.33±0.06	1.31±0.00	1.08±0.01
HAN-TR	34.95±1.70	30.69±5.89	2.10±0.79	1.83±0.75
TNAM	37.77±0.35	37.97±0.20	1.36±0.00	1.16±0.00
BPR-T	33.83±0.43	31.15±0.33	1.23±0.01	0.98±0.01
Table 7: The benchmarking results of the feature-based explainable recommender models on predicting the tags for satisfaction and ratings.
Metrics	Satisfaction	Rating Prediction
F1	NDCG	RMSE	MAE
EFM	4.57±0.46	1.79±0.19	1.68±0.00	1.24±0.01
TriRank	16.82±0.03	13.16±0.02	2.90±0.00	2.58±0.00
LRPPM	36.04±0.16	22.35±0.08	1.22±0.00	0.96±0.00
SULM	40.46±0.62	24.80±0.19	1.64±0.09	1.29±0.07
MTER	5.97±1.92	2.85±1.07	2.26±0.65	1.96±0.62
AMF	27.16±0.19	17.05±0.21	2.28±0.00	1.86±0.00
DERM-MLP	36.76±0.07	22.53±0.12	1.30±0.01	1.06±0.01
DERM-MF	35.40±0.23	20.59±0.35	1.32±0.00	1.14±0.00
DERM-C	36.20±0.28	22.28±0.22	1.29±0.01	1.07±0.01
DERM-H	34.65±0.43	21.33±0.47	1.28±0.01	1.05±0.02
TRDM	31.29±0.63	14.74±0.69	1.19±0.00	0.95±0.01
TRAL	5.89±0.05	2.52±0.01	1.29±0.00	1.08±0.00
HPTR	35.35±2.90	17.78±3.92	1.81±0.77	1.56±0.72
AIRec	37.10±0.09	22.86±0.08	1.30±0.01	1.08±0.01
HAN-TR	33.95±1.80	18.06±4.15	2.10±0.80	1.83±0.75
TNAM	5.89±0.05	2.52±0.01	1.37±0.00	1.17±0.00
BPR-T	33.82±0.29	19.52±0.25	1.23±0.00	0.98±0.00