Venue: NAACL 2025

# ALERT: An LLM-powered Benchmark for Automatic Evaluation of

Recommendation Explanations

Yichuan Li\({}^{1*}\), Xinyang Zhang\({}^{2}\), Chenwei Zhang\({}^{2}\), Mao Li\({}^{2}\), Tianyi Liu\({}^{2}\), Pei Chen\({}^{2}\),

**Vifan Gao\({}^{2}\), Kyumin Lee\({}^{1}\), Kaize Ding\({}^{3}\), Zhengyang Wang\({}^{2}\), Zhihan Zhang\({}^{4}\), Jingbo Shang\({}^{5}\), Xian Li\({}^{2}\), Trishul Chilimbi\({}^{2}\) +\({}^{1}\)Worcester Polytechnic Institute, \({}^{2}\)Amazon.com, \({}^{3}\)Northwestern University, ++\({}^{4}\)University of Notre Dame, \({}^{5}\)University of California, San Diego**

Footnote †: This work was mainly done during Yichuan’s internship at Amazon.

Correspondence to: yli29@wpi.edu

###### Abstract

Recommendation explanation systems have become increasingly vital with the widespread adoption of recommender systems. However, existing recommendation explanation evaluation benchmarks suffer from limited item diversity, impractical user profiling requirements, and unreliable and unscalable evaluation protocols. We present ALERT, a model-agnostic recommendation explanation evaluation benchmark. The benchmark comprises three main contributions: _1)_ a diverse dataset encompassing 15 Amazon e-commerce categories with 2,761 user-item interactions, incorporating implicit preferences through purchase histories; _2)_ two novel LLM-powered automatic evaluators that enable scalable and human-preference aligned evaluation of explanations; and _3)_ a robust divide-and-aggregate approach that synthesizes multiple LLM judgments, achieving 70% concordance with expert human evaluation and substantially outperforming existing methods. ALERT facilitates comprehensive evaluation of recommendation explanations across diverse domains, advancing the development of more effective explanation systems. The implementation is available at [https://github.com/bigheiniu/ALERT-LLMRecomBenchmark](https://github.com/bigheiniu/ALERT-LLMRecomBenchmark).

## 1 Introduction

Recommender systems have become essential tools for navigating user preferences and mitigating information overload in the current era of information explosion Isinkaye et al. (2015); Zhao et al. (2021); Resnick and Varian (1997). As these systems evolve, there is a growing demand for convincing and useful explanations that provide insights into the recommendation process, fostering user trust and enhancing overall satisfaction Zhang et al. (2020); Balog and Radlinski (2020); Chen et al. (2022).

Despite the critical role of explanations, their effective evaluation remains a significant challenge, particularly in representing real-world scenarios and developing automated evaluation methods that accurately reflect user preferences. Current benchmark datasets Chen et al. (2023); Lu et al. (2023); Li et al. (2023); Balog and Radlinski (2020) suffer from two major limitations:

Lack of Diverse Categories and User Preference DataExisting benchmarks Lu et al. (2023); Li et al. (2023) mainly focus on a single category, such as movies or books, failing to capture the complexity and variety of real-world recommendation scenarios. The explanatory features vary significantly across product domains; for instance, movie recommendations typically emphasize artistic elements like genres and actors, whereas beauty product recommendations focus on functional attributes such as moisturizing or exfoliating properties. This domain-specific variation presents a significant challenge in evaluating the cross-domain generalization capabilities of explanation models. Furthermore, while user preferences are fundamental to modeling and evaluating recommendation explanations, as they inform the underlying rationale for user-item interactions, current user profile collection methodologies are insufficient. Several benchmarks Lu et al. (2023); Li et al. (2023) rely on questionnaire-based approaches to collect user profiles and demographic attributes. However, this methodology presents inherent limitations in real-world applications, often yielding incomplete preference profiles and failing to capture the dynamic nature of user preferences.

Absence of Scalable and Human-Preference Aligned Evaluation Protocols.Existing evaluation protocols can be categorized into two main types: _human evaluation_ and _automated evaluation_. While _human evaluation_ Hernandez-Bocanegra et al. (2020); Musto et al. (2019) are valuable, they are

often time-consuming, expensive, and challenging to scale, hindering rapid iteration and improvement of explanation models. _Automatic evaluation methods_Li et al. (2017); Li et al. (2023, 2023); Geng et al. (2022); Xie et al. (2022) often require references to compare the generated results. This approach poses challenges for evaluating large language model-based recommendation explanation methods Lei et al. (2024) that can generate free-form explanations without following a specific structure. Moreover, these automatic evaluators often misuse user reviews as ground truth for explanations, ignoring differences between pre-purchase explanations and post-purchase reviews. This approach fails to capture the persuasive nature of explanations and includes irrelevant factors like delivery experiences, compromising the evaluation's validity.

To address these challenges, we propose **A**utomatic **LLM**Evaluation of **R**ecommendation explana**T**ions (ALERT), a novel recommendation model-agnostic explanation benchmark. Our approach offers several key innovations addressing the shortcomings of existing recommendation explanation benchmarks and evaluation methodologies:

* **ALERT Dataset**: We introduce a comprehensive benchmark dataset spanning 15 Amazon product categories, to balance the diversity while also balancing the cost, we including 2761 user-item interactions, leveraging user purchase behaviors from Amazon Hou et al. (2024). This diverse dataset surpasses existing benchmarks Chen et al. (2023); Lu et al. (2023), providing a more realistic and challenging evaluation environment. By incorporating users' purchase history, we enable recommendation explanation models to learn implicit preferences from actual user behavior.
* **LLM-powered Automatic Evaluation Protocols**: We introduce two novel evaluation methods using Large Language Models as judges (LLM-as-a-Judge), specifically designed for evaluateing recommendation explanation quality without the reference requirement. Unlike existing LLM-as-a-Judge approaches Zheng et al. (2024); Chiang et al. (2024) used for general instruction-following ability, we develop criteria tailored to recommendation explanations. We also propose an divide and aggregation approach to combine multiple LLM judgments under each criterion, achieving high human agreement.
* **Validated Reliability**: To ensure the reliability of our evaluation framework, we create a meta-evaluation dataset with intentionally created negative examples and expert-level human annotations. Our comprehensive meta-evaluation demonstrates a strong correlation with expert human judgments, achieving 70% human agreement and outperforming existing automated and crowd-sourced approaches in evaluating recommendation explanations.

## 2 Task Description of Recommendation Explanations

The primary goal of this benchmark is to inspire the development of new methods for generating post-hoc, free-style recommendation explanations that motivate users to purchase recommended items. Following prior literature Zhang et al. (2020); Chen et al. (2022), we define the task as: _"generating a recommendation explanation by considering the user's profile and black-box recommended items, aimed at increasing user satisfaction and purchase inclination"_ as shown in Fig. 1 (A). Formally, for a user \(u\), let \(i\) denote the recommended item, with \(X_{i}\) representing the item's context (e.g., title, price); let \(\mathcal{X}_{u}\) represent the user's purchase history which is \(\mathcal{X}_{u}=\{X_{j}\}^{|\mathcal{X}_{u}|}_{j=0}\). It should be noticed that \(X\). is a text sequence. The task is to generate an explanation \(\mathbf{E}_{ui}\) that encourages user \(u\) to purchase item \(i\), described as \(\mathbf{E}_{ui}=G(\mathcal{X}_{u},X_{i})\), where \(G(\cdot)\) is the explanation generation model.

## 3 Benchmark Description

Our benchmark, ALERT, consists of two main components: curated benchmark datasets and LLM-powered automatic evaluators, as illustrated in Fig. 1 (B) that work together to provide a comprehensive evaluation framework for recommendation explanation models. These components are designed to provide evaluation environment that simulate authentic recommendation scenarios while facilitating scalable evaluation that better capture human preferences and decision-making patterns.

### Curated Benchmark Dataset Components

ALERT enables reference-free evaluation through a comprehensive dataset that captures user-item interactions and their contextual information. The dataset curation balances representativeness with quality control.

Recommended Items.We treat the user's most recent purchase as the item requiring recommendation explanation. We leverage the rich context 

available in Amazon product listings, incorporating the item's title (which typically encapsulates key features), price point, and merchant information. This comprehensive item representation enables explanation models to generate more contextually relevant explanations that highlight specific product aspects aligned with user preferences.

User History.We construct user profiles based on purchase histories. This approach offers several advantages over traditional demographic data collection Chen et al. (2023); Lu et al. (2023): _1)_ Enhanced privacy protection through the exclusion of sensitive demographic information; _2)_ Concrete evidence of user preferences through actual purchase decisions; _3)_ Simplified data collection through existing transaction records. Each user profile consists multiple dimensions, including previously purchased item categories, purchase frequency patterns, and typical price ranges, enabling explanation models to generate personalized recommendations based on demonstrated user behavior.

Dataset Quality Control.To ensure the quality and diversity of purchase behaviors, we retained items with titles containing at least six alphabetic characters and selected users with 5-20 purchase transactions. This balance provides sufficient behavioral data while maintaining dataset manageability. Furthermore, we exclusively considered 5-star rated purchases and randomly sampled a maximum of 200 purchase behaviors per category to prevent category bias while maintaining dataset diversity. After filtering, the statistical distribution of ALERT is presented in Fig. 2.

### LLM-powered Automatic Evaluators

Evaluating the quality of recommendation explanations is a complex task that requires a well-defined set of criteria and a robust evaluation process. In this section, we introduce an automatic evaluation framework powered by Large Language Models (LLMs) that addresses these challenges. We begin by presenting a set of carefully designed evaluation criteria that capture the key aspects of high-quality recommendation explanations, enhancing interpretability and reducing ambiguity in the evaluation process. Next, we describe two types of LLM-based evaluators, ALERT-Gen and ALERT-Disc, which leverage these criteria to assess explanation quality. To effectively handle the complexity of the evaluation task, we propose a divide and aggregate approach that combines the judgments of multiple LLM evaluators. Finally, we discuss the evaluation metric used in ALERT to quantify the performance of recommendation explanations.

#### 3.2.1 Evaluation Criteria Development

The evaluation criteria were developed collaboratively with an e-commerce editorial team and refined through user studies. These criteria balance technical system requirements with user needs in

Figure 1: Evaluation framework for ALERT. (A) is the task description of recommendation explanation. (B) are two LLM-powered automatic evaluators used in this benchmark. ALERT-Gen is the generative recommendation explanation evaluation and ALERT-Disc is the discriminate recommendation explanation evaluation.

Figure 2: Distribution of data categories and statistics in ALERT.

 

e-commerce settings, providing a framework to assess recommendation explanation effectiveness.

Footnote 1: [https://github.com/en/hugging-learning/](https://github.com/en/hugging-learning/)

* **Reasoning**: Provide reasons based on user preferences, item attributes, or past behaviors, tailored to each user's specific context. Consistency with user history and item details is essential.
* **Clear and Concise Language**: Use simple, conversational language that avoids jargon and is easily understandable by the average customer.
* **Engaging Narrative**: Allow customers to envision using and enjoying the product by providing an engaging narrative.
* **Neutral Tone**: Maintain a neutral, informative tone, avoiding overtly sales-oriented language while offering clear, objective reasons for recommendations.

#### 3.2.2 LLM-as-a-Judge Architectures

With the evaluation criteria in place, we now introduce our LLM-based evaluation framework that leverages these criteria to assess the quality of recommendation explanations. Our framework employs two types of LLM evaluators, ALERT-Gen and ALERT-Disc, each offering unique advantages in capturing different aspects of explanation quality. To effectively handle the complexity of the evaluation task and ensure a comprehensive assessment, we propose a divide and aggregate approach that combines the judgments of these evaluators.

ALERT-Gen.The generative evaluator, powered by Claude-3-Sonnet, performs comparative analysis of explanation pairs using Chain-of-Thought Wei et al. (2022) reasoning. For each comparison, ALERT-Gen generates: _1)_ explicit reasoning steps showing how the comparison was made by following the evaluation criterion, and _2)_ a final preference statement in the format "[[A/B/C]] Model A/B is better than Model B/A or Tied." To mitigate position bias, we systematically swap the position of explanations being compared.

ALERT-Disc.Although generative evaluation provides both comparisons and rationale of the judgment, it is typically more computationally expensive and may not optimally reflect human preferences Lambert et al. (2024). The discriminative evaluator assigns a score to each model's response based on predefined evaluation criteria. The scores are then compared, and the response receiving the higher score is judged as superior. The scores are then compared, and the response receiving the higher score is judged as superior. We implement ALERT-Disc using ArmoRM Wang et al. (2024), a), which has achieved the state-of-the-art performance in RewardBench Lambert et al. (2024) during June 2024.

Divide and Aggregate Evaluation Process.As illustrated in Fig. 3, the divide and aggregate process begins by breaking down the evaluation criteria into individual components. Each criterion is used to prompt an LLM evaluator to compare two explanations and determine the preferred one. The individual preferences are then aggregated through majority voting to arrive at a final judgment. By decomposing the evaluation task into smaller, more focused assessments and combining the results, we ensure a comprehensive and robust evaluation of recommendation explanations.

#### 3.2.3 Evaluation Metrics

To quantify the performance of recommendation explanations, we employ the widely adopted _win rate_ metric, which has been used in previous studies on evaluating the quality of generated text Zheng et al. (2024); Lin et al. (2024). The win rate calculates the percentage of user-item interactions for which the evaluated explanation is preferred over a baseline explanation. In our experiments, we use the PreferenceLogic as the baseline, which generates explanations by connecting user preferences and item attributes through prompting an LLM.

## 4 Meta-Evaluation of ALERT-Gen and ALERT-Disc

To rigorously assess the efficacy of our proposed automatic evaluation methods, ALERT-Gen and ALERT-Disc, we conduct a comprehensive meta-evaluation framework that examines three critical dimensions: their reliability in detecting suboptimal explanations, their correlation with expert human judgments, and their computational efficiency in terms of time and cost savings. This systematic evaluation approach enables us to vali

Figure 3: Diagram of the divide and aggregation evaluation process for ALERT-Gen and ALERT-Disc.

 

date our methods through both controlled experimental conditions and real-world human preference data, while demonstrating the practical advantages of automated evaluation over traditional manual evaluation procedures.

### Baselines of Evaluation Methods

We compare our proposed evaluation methods against several established baseline approaches: _1)_ MTurk: Crowdsourced annotation from Amazon Mechanical Turk, using majority voting over 9 annotators per annotation task. _2)_ MT-Bench [35]: an LLM-as-a-Judge approach for evaluating LLM instruction following ability; and _3)_ LLM-hasEvaluator [34]: a Likert-style LLM-as-a-Judge evaluator that evaluates recommendation explanations based on persuasiveness, transparency, accuracy, and satisfaction. Additionally, we develop variants of our evaluation methods that process all judge criteria in a single call, rather than dividing the evaluation criteria and aggregating preferences through multiple calls. These variants are denoted with the suffix "_w./o_ DA." to both ALERT-Gen and ALERT-Disc.

### Inferior Explanations Detection Test

To examine the evaluation methods' ability to identify explanation quality differences, we construct a set of controlled test cases by introducing systematic interruption to explanations generated by PreferenceLogic (discussed in SS 5.1). This semi-synthetic evaluation methodology enables validation without the need for additional human annotations [36]. The variations are categorized into two types: _1) Data Interruption_ which includes item replacement, user replacement, and user profile empty. _2) Prompt interruption_1 intentionally disrupts the explanation prompt to generate negative examples that are not easily identifiable.

Footnote 1: The prompt for generating interrupted explanation can be found on Appendix A

Detection Results.The results presented in Tab. 1 lead to several observations. First, the User-Replace and Empty-User conditions pose the greatest challenges for the evaluators. Under the User-Replace condition, the proposed ALERT-Gen and ALERT-Disc evaluators significantly outperform other evaluators, achieving accuracies of 0.627 and 0.564, respectively. Moreover, the MTurk accuracy of 0.073 suggests the presence of systematic biases in the annotations, where workers relied on superficial features rather than conducting a meaningful evaluation of user-explanation alignment. Inverting these predictions would yield higher accuracies (e.g., 0.927 instead of 0.073), further highlighting the biases. These results demonstrate the superior ability of the proposed evaluators in detecting inconsistencies between user profiles and explanations. The Empty-User condition emerges as the most challenging, with all evaluators struggling to achieve high accuracy. This difficulty likely arises from the evaluators' tendency to infer user preferences from detailed item descriptions, even in the absence of user information. Finally, the results consistently validate the effectiveness of the ALERT-Gen and ALERT-Disc evaluators across diverse test conditions, underscoring the versatility and reliability of the proposed approach in detecting many potential issues in recommendation explanations.

### Correlation with Human Judgment

This meta-evaluation ensures that our automated evaluation closely approximates expert human evaluation, potentially reducing the need for resource-intensive manual evaluations in future research. To collect reliable human judgments, three professional data annotators with extensive experience in natural language processing and e-commerce. These annotators were not involved in the paper's authorship or research, ensuring an unbiased evaluation. Their expertise surpasses that of average crowd workers, guaranteeing high-quality annotat

\begin{table}
\begin{tabular}{c c c c|c c c c c} \hline \hline \multirow{2}{*}{Evaluators} & \multicolumn{3}{c}{Data Interruption} & \multicolumn{3}{c}{Prompt Interruption} \\  & Item-Replace & User-Replace & Empty-User & Minor-Benefits & Misinfo. & Jargon & Bad-Experience & AVG. \\ \hline MTurk* & 0.922 & 0.073 & 0.196 & 0.091 & 0.345 & 0.900 & 0.556 & 0.440 \\ \hline MT-Bench & 0.354 & 0.333 & 0.292 & 0.333 & 0.250 & 0.350 & 0.333 & 0.321 \\ LLMAsEvaluator & 0.627 & 0.254 & 0.274 & 0.182 & 0.259 & 0.700 & 0.555 & 0.408 \\ \hline ALERT-Gen w./o D.A. & **0.980** & 0.529 & 0.345 & 0.454 & 0.273 & 0.800 & 0.556 & 0.563 \\ ALERT-Disc w./o D.A. & **0.980** & 0.455 & **0.471** & 0.636 & 0.709 & 0.950 & **0.889** & 0.727 \\ \hline ALERT-Gen & **0.980** & **0.627** & 0.418 & 0.545 & 0.309 & 0.900 & 0.667 & 0.635 \\ ALERT-Disc & **0.980** & 0.564 & 0.431 & **0.727** & **0.870** & **1.000** & 0.778 & **0.764** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Inferior explanations detection result. The highest accuracy is marked by bold. Random guess would achieve an accuracy 0.5.

 

tions. A comprehensive Standard Operating Procedure (SOP) was developed to break down the final preference decision into a sequence of sub-questions. This approach enables a more nuanced evaluation and includes early stopping rules to optimize annotation efficiency. For each subquestion, carefully selected examples were provided to illustrate preferred options and their rationales, further enhancing annotation quality and consistency. The inter-annotator agreement (IAA) of 0.24 on Krippendorff's alpha indicates moderate agreement, which is reasonable given the complexity and subjectivity involved in evaluating recommendation explanations. To ensure a comprehensive evaluation set, 88 pairs of recommendation explanations were randomly selected from the 5 recommendation explanation models mentioned in SS 5.1, namely user-based, item-based, review extraction, review summarization, and attr2seq. This sampling method guarantees representation across different quality levels and model types, providing a diverse and representative dataset for evaluation.

Footnote 3: [https://www.cai.com/en/en/en/en/en/en_en/en](https://www.cai.com/en/en/en/en/en/en_en/en) 

resenting time and cost reductions of over 99.7% and 99.96% compared to MTurk. This makes our methods highly scalable for large-scale evaluations that would be infeasible with human annotators. _2) Comparison with MT-Bench:_ Although ALERT-Gen requires more API calls due to its divided and aggregate nature, it provides better human alignment with limited time. Furthermore, our ALERT-Disc method outperforms MT-Bench in both time (10s vs. 25s) and cost (\(0.09vs.1.02\)), demonstrating its superiority among automatic evaluation methods.

_3) Flexibility:_ while slightly slower and more expensive than ALERT-Disc, still offers significant improvements over human evaluation, providing researchers with options based on their speed, cost, or evaluation approach requirements.

## 5 Leaderboard Results

This section presents the benchmark performance of diverse recommendation explanation models on the full ALERT dataset.

### Recommendation Explanation Methods

We evaluate several popular recommendation explanation models:

* **User-based[48, 49]**: Generales explanations based on user similarities, typically stating, Users with similar tastes to yours like this item. They have also purchased these items."
* **Item-based[48, 49]**: Produces explanations based on similarities between recommended items and the user's purchase history, often using the statement, This item is similar to one you've already bought."
* **Review Extraction**: Directly utilizes review content, which can be informative but potentially noisy and not always directly explanatory [46, 48].
* **Review Summarization**: Synthesizes multiple reviews to create a comprehensive overview of user opinions and item features, offering a generalized rather than personalized explanation.
* **Attr2Seq[104]**: Generates reviews based on hidden attribute information learned from past user-item interaction reviews.
* **RecExplainer-B[49]**: Employs multi-task learning to align LLM-based recommendation explanation models with recommendation systems. The training incorporates tasks such as attribute prediction, user history prediction,

Figure 4: Case study of generative judge reasoning. Orange content is comments for Model A’s output while blue content is comments for Model B’s output.

\begin{table}
\begin{tabular}{c|l l} \hline \hline Evaluator & Time (est.) & Cost (est.) \\ \hline MTurk & 363min & $285.12 \\ MT-Bench & 25s & $1.02 \\ ALERT-Gen & Imin & $4.02 \\ ALERT-Disc & 10s & $0.09 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Evaluation efficiency analysis comparing time and monetary costs.

 

and explanation ability knowledge distillation from proprietary LLMs, enhancing the model's capacity to generate relevant and personalized explanations.

Additionally, we propose **PreferenceLogic3**, which leverages LLMs to generate explanations by connecting user preferences with item attributes. This serves as the baseline for our ALERT-Gen and ALERT-Dis evaluators in calculating the _Win Rate_.

Footnote 3: Prompting details are provided in Appendix A

For Item-based and User-based methods, we implement BiasedMF Koren et al. (2009) following Lu et al. (2023) to identify similar items and users, then employ LLMs to generate coherent explanations. Models requiring LLM training (Attr2Seq and RecExplainer-B) utilize Mistral-7B-Instruct Jiang et al. (2024), while non-training methods employ Claude-3-Sonnet. Training data comprises historical interactions of users and items present in the evaluation dataset, excluding their mutual interactions.

### Results and Analysis

Fig. 5 presents the _win rates_ calculated by ALERT-Gen and ALERT-Disc for various recommendation explanation models. Our analysis reveals several key findings: Fig. 5 presents the _win rates_ calculated by ALERT-Gen and ALERT-Disc for various recommendation explanation models. Our analysis reveals several key findings. _1)_. Both user-based and item-based recommendation explanations show the lowest _win rates_, approaching 0%, suggesting that these traditional methods may be insufficient for generating high-quality, personalized explanations in modern recommendation systems. _2)_.The performance of review-based explanation methods is significantly lower than that of our prompting-based approaches, indicating that directly leveraging review content may not capture the nuanced preferences and context required for effective explanations. Notably, _3)_.PreferenceLogic achieves commendable performance in both types of automatic evaluation, even surpassing the training-based Attr2Seq approach. This highlights the potential of LLMs for generating high-quality recommendation explanations without extensive domain-specific training. _4)_.Furthermore, RecExplainer-B outperforms Attr2Seq, demonstrating the importance of multi-task learning in recommendation explanation models. This approach enables the model to better understand specific data types and learn explanation generation capabilities from other LLMs, underscoring the benefits of in-domain data training for recommendation explanation tasks. These findings suggest that while traditional methods may be insufficient for modern recommendation explanation tasks, LLM-based approaches - particularly those leveraging multi-task learning - show significant promise in generating relevant, personalized, and high-quality explanations.

### Evaluation Criteria Decomposition

Analysis of the granular performance metrics from ALERT-Gen and ALERT-Disc across evaluation criteria (see Tab. 4) reveals distinct patterns. RecExplainer demonstrates superior overall performance, achieving notable success in reasoning and narrative engagement (71.4% win rate for both metrics). While Attr2Seq exhibits balanced performance across metrics, it particularly excels in communication clarity (49.0%). ReviewSumm and ReviewExtract show competence in reasoning but demonstrate deficiencies in maintaining neutral tone. Conversely, User-Based and Item-Based approaches maintain strong tone neutrality (>57%) but underperform in reasoning and engagement.

## 6 Related Work

Benchmark for Recommendation Explanation Generation.The field of recommendation expla

Figure 5: Automatic benchmarking results. The recommendation explanation are compared against prompting Claude3 Sonnet’ output.

 

nation evaluation has seen significant developments in recent years, with several benchmarks contributing to our understanding of explanation quality and effectiveness. Datasets such as REASONER Chen et al. (2023) and Sel-Explain Lu et al. (2023) have provided valuable insights into specific domains like movies or books. These benchmarks have employed various approaches to capture user preferences, ranging from questionnaires to the analysis of user reviews. Evaluation protocols in the field span from human evaluations Hernandez-Bocanegra et al. (2020); Musto et al. (2019) to automated methods Li et al. (2017, 2023); Geng et al. (2022); Xie et al. (2022) designed to assess explanation quality efficiently. As the field progresses, there is growing interest in developing benchmarks that can address a wider range of product categories, capture more nuanced user preferences, and accommodate diverse explanation types, including the free-form explanations generated by large language models.

Recommendation Explanation GenerationRecommendation explanation generation has evolved significantly, moving from matrix factorization methods that align user-item interactions with explicit features Zhang et al. (2014); Chen et al. (2016) to more sophisticated natural language generation approaches. The advent of Large Language Models (LLMs) has revolutionized this field, enabling the generation of fluid, human-like explanations Chu et al. (2024); Geng et al. (2022); Lei et al. (2024). LLM-based models leverage vast textual data and general world knowledge to extract nuanced user preferences and item attributes, facilitating more contextually rich explanations. These models often employ multi-task learning and prompting techniques to generate free-form explanations directly. However, they face challenges such as content hallucination, which can lead to inaccurate explanations. To address this, recent research has explored retriever-augmented generation (RAG) techniques Yang et al. (2024); Li et al. (2023), which anchor the generation process to key aspects retrieved from reviews, thereby improving factual accuracy while maintaining the fluidity of free-form explanations. This approach represents a promising direction in balancing the creativity of LLM-generated explanations with the need for accuracy and relevance in recommendation systems.

LLM-as-A-JudgeThe emergence of Large Language Models (LLMs) as evaluators, known as LLM-as-A-Judge, represents a significant advancement in automated assessment techniques. This approach aims to bridge the gap between human evaluation, which remains the gold standard but is often prohibitively expensive and time-consuming, and traditional automated metrics. Several benchmarks have successfully employed this method, including MT-Bench Zheng et al. (2024), AlpacaEval Li et al. (2023), ArenaHard Chiang et al. (2024), and WildBench Lin et al. (2024). In the context of recommendation explanations, LLM-as-A-Judge offers the potential to assess explanation quality across multiple dimensions, such as relevance, persuasiveness, and coherence, without the need for reference explanations. This flexibility is particularly valuable for evaluating free-form explanations generated by advanced models. Additionally, some researchers have explored discriminative judge models Wang et al. (2024) that predict scores for generated explanations, further enhancing the scalability and consistency of automated evaluation methods. As the field progresses, LLM-as-A-Judge approaches are likely to play an increasingly important role in developing and refining recommendation explanation systems.

## 7 Conclusion

This paper introduces ALERT, a novel recommendation model-agnostic explanation benchmark that addresses key limitations in existing evaluation methods. ALERT contains a diverse dataset from 15 Amazon e-commerce categories and introduces two innovative LLM-powered automatic evaluation methods for flexible evaluation of free-form explanations without ground-truth references. Our meta-evaluation experiments demonstrate that ALERT's automated evaluation approaches: ALERT-Gen and ALERT-Disc achieves high correlation with human judgments, outperforming existing methods. By providing an efficient, scalable, and reliable approach, ALERT aims to accelerate the development of more transparent and user-friendly recommendation explanation models.

\begin{table}
\begin{tabular}{c|c c c c} \hline \hline Method & Reason. & Clear. & Engag. & Nertral \\ \hline User-Based & 0.166 & 0.166 & 0.181 & 0.583 \\ Item-Based & 0.100 & 0.200 & 0.200 & 0.577 \\ ReviewSumm & 0.485 & 0.363 & 0.161 & 0.171 \\ ReviewExtract & 0.315 & 0.271 & 0.184 & 0.166 \\ Attr2Seq & 0.297 & 0.489 & 0.363 & 0.500 \\ RecExplainer-B & 0.714 & 0.500 & 0.714 & 0.625 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Granular performance comparison of recommendation explanation methods.

 



## 8 Limitations

While our study introduces a novel benchmark ALERT for recommendation explanation, we acknowledge several limitations in our current approach. Firstly, we did not conduct a thorough analysis of potential data leakage during LLM pretraining. Given that our benchmark data were crawled from websites, there is a possibility that some of this information may have been included in the training data of certain LLMs. This potential overlap could inadvertently favor specific types of LLMs or prompting methods that can access these evaluation data, potentially skewing our results. Additionally, our evaluation framework lacks an online testing component to further validate the effectiveness of the LLM-as-evaluator approach. While our offline evaluations provide valuable insights, we recognize the importance of real-world testing to fully assess the practical applicability and robustness of our proposed methods. Implementing online tests would offer a more comprehensive understanding of how our benchmark and evaluation techniques perform in dynamic, real-time scenarios. We acknowledge the significance of addressing these limitations to enhance the reliability and generalizability of our findings. Future work should prioritize investigating the extent of data leakage and its impact on recommendation explanation model performance, as well as conducting online tests to validate the LLM-as-a-Judge approach in practical settings. These efforts will contribute to a more robust and comprehensive evaluation framework for recommendation explanation systems.

## 9 Ethical Considerations

We have taken the following steps to address ethical concerns related to the data collection and evaluation experiments in this work:

* [leftmargin=0pt]
* **Fair Compensation for Annotations**. In our meta-evaluation process, we collected human annotations through Amazon Mechanical Turk (AMTurk). Workers were compensated at a rate of $0.36 per task, which is significantly higher than the platform average. This ensures fair payment for their time and effort. Additionally, we engaged expert human editors, both full-time and part-time, for data annotation. Regular meetings were held with these editors from the outset of the annotation process to maintain consistency and quality.
* **User Privacy Protection**. The dataset used in this study was sourced from the Amazon Reviews 2023 dataset ([https://amazon-reviews-2023.github.io/](https://amazon-reviews-2023.github.io/)), which does not contain sensitive user information. To further protect user privacy, we will not release any user IDs associated with the user-interaction data in our benchmark.
* **Data Sensitivity**. Our benchmark focuses on recommendation explanations and does not include any tasks or data related to sensitive topics or real-world security vulnerabilities. The scenarios used in our benchmark are based on product recommendations and reviews, which minimizes the risk of exposing or exploiting sensitive information.

In conclusion, based on these precautions, we believe that the risks associated with the data collection and usage of this benchmark for evaluating recommendation explanation systems are minimal. We remain committed to addressing any unforeseen ethical concerns that may arise and to continuously improving our practices to ensure responsible research in this field.

## References

* Balog and Radlinski (2020) Krisztian Balog and Filip Radlinski. 2020. Measuring recommendation explanation quality: The conflicting goals of explanations. In _Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval_, pages 329-338. Cited by: SS1.
* Chen et al. (2016) Learning to rank features for recommendation over multiple categories. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_, pages 305-314. Cited by: SS1.
* Chen et al. (2016) Xu Chen, Zheng Qin, Yongfeng Zhang, and Tao Xu. 2016. Learning to rank features for recommendation over multiple categories. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_, pages 305-314. Cited by: SS1.
* Chen et al. (2023) Xu Chen, Yongfeng Zhang, and Ji-Rong Wen. 2022. Measuring why" in recommender systems: A comprehensive survey on the evaluation of explainable recommendation. arXiv preprint arXiv:2202.06466. Cited by: SS1.
* Li et al. (2020)

Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating l1ms by human preference. _Preprint_, arXiv:2403.04132.
* Chu et al. (2024) Zhixuan Chu, Yan Wang, Qing Cui, Longfei Li, Wenqing Chen, Sheng Li, Zhan Qin, and Kui Ren. 2024. Llm-guided multi-view hypergraph learning for human-centric explainable recommendation. _arXiv preprint arXiv:2401.08217_.
* Dong et al. (2017) Li Dong, Shaohan Huang, Furu Wei, Mirella Lapata, Ming Zhou, and Ke Xu. 2017. Learning to generate product reviews from attributes. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 623-632.
* Geng et al. (2022) Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In _Proceedings of the 16th ACM Conference on Recommender Systems_, pages 299-315.
* Hernandez-Bocanegra et al. (2020) Diana C Hernandez-Bocanegra, Tim Donkers, and Jurgen Ziegler. 2020. Effects of argumentative explanation types on the perception of review-based recommendations. In _Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization_, pages 219-225.
* Hou et al. (2024) Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. 2024. Bridging language and items for retrieval and recommendation. _arXiv preprint arXiv:2403.03952_.
* Isinkaye et al. (2015) Folasade Olubusola Ishiskaye, Yetunde O Folajimi, and Bolande Adefowoke Ojokoh. 2015. Recommendation systems: Principles, methods and evaluation. _Egyptian informatics journal_, 16(3):261-273.
* Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lensepel, Guillaume Bour, Guillaume Lample, Lello Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Theophile Gervet, Thibaut Lavril, Thomas Wang, Timothete Lacroix, and William El Sayed. 2024. Mixtral of experts. _Preprint_, arXiv:2401.04088.
* Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. _Computer_, 42(8):30-37.
* Kouki et al. (2019) Pigi Kouki, James Schaffer, Jay Pujara, John O'Donovan, and Lise Getoor. 2019. Personalized explanations for hybrid recommender systems. In _Proceedings of the 24th International Conference on Intelligent User Interfaces_, IUI '19, page 379-390, New York, NY, USA. Association for Computing Machinery.
* Lambert et al. (2024) Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024a. Reward-bench: Evaluating reward models for language modeling. _Preprint_, arXiv:2403.13787.
* Lambert et al. (2024b) Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024b. Reward-bench: Evaluating reward models for language modeling. _Preprint_, arXiv:2403.13787.
* Lei et al. (2024c) Yuxuan Lei, Jianxun Lian, Jing Yao, Xu Huang, Defu Lian, and Xing Xie. 2024c. Recexplainer: Aligning large language models for explaining recommendation models. In _Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_, volume 1341 of _KDD '24_, page 1530-1541. ACM.
* Li et al. (2023a) Jiacheng Li, Zhankui He, Jingo Shang, and Julian McAuley. 2023a. Ucepic: Unifying aspect planning and lexical constraints for generating explanations in recommendation. _Preprint_, arXiv:2209.13885.
* Li et al. (2017) Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017. Neural attentive session-based recommendation. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_, pages 1419-1428.
* Li et al. (2023b) Lei Li, Yongfeng Zhang, and Li Chen. 2023b. Personalized prompt learning for explainable recommendation. _ACM Transactions on Information Systems_, 41(4):1-26.
* Li et al. (2023c) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023c. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval).
* Lin et al. (2024) Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeez Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. _Preprint_, arXiv:2406.04770.
* Lu et al. (2023) Hongyu Lu, Weizhi Ma, Yifan Wang, Min Zhang, Xiang Wang, Yiquin Liu, Tat-Seng Chua, and Shaooping Ma. 2023. User perception of recommendation explanation: Are your explanations what users need? _ACM Trans. Inf. Syst._, 41(2).
* Musto et al. (2019) Cataldo Musto, Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. 2019. Justifying recommendations through aspect-based sentiment analysis of users reviews. In _Proceedings of the 27th ACM conference on user modeling, adaptation and personalization_, pages 4-12.
* Resnick and Varian (1997) Paul Resnick and Hal R Varian. 1997. Recommender systems. _Communications of the ACM_, 40(3):56-58.

 

Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. 2024a. Arithmetic control of l1ms for diverse user preferences: Directional preference alignment with multi-objective rewards. In _ACL_.
* Wang et al. (2024b) Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024b. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In _EMNLP_.
* Wang et al. (2022c) Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024c. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. _Preprint_, arXiv:2406.12845.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Xie et al. (2022) Zhouhang Xie, Sameer Singh, Julian McAuley, and Bodhisattwa Prasad Majumder. 2022. Factual and informative review generation for explainable recommendation. _Preprint_, arXiv:2209.12613.
* Yang et al. (2024) Mengyuan Yang, Mengying Zhu, Yan Wang, Linkun Chen, Yilei Zhao, Xiuyuan Wang, Bing Han, Xiaolin Zheng, and Jianwei Yin. 2024. Fine-tuning large language model based explainable recommendation with explainable quality reward. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 9250-9259.
* Zeng et al. (2024) Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In _The Twelfth International Conference on Learning Representations_.
* Zhang et al. (2024) Xiaoyu Zhang, Yishan Li, Jiavin Wang, Bowen Sun, Weizhi Ma, Peijie Sun, and Min Zhang. 2024. Large language models as evaluators for recommendation explanations. _Preprint_, arXiv:2406.03248.
* Zhang et al. (2020) Yongfeng Zhang, Xu Chen, et al. 2020. Explainable recommendation: A survey and new perspectives. _Foundations and Trends& in Information Retrieval_, 14(1):1-101.
* Zhang et al. (2014) Yongfeng Zhang, Guokun Lai, Min Zhang, Yi Zhang, Yiqun Liu, and Shaoping Ma. 2014. Explicit factor models for explainable recommendation based on phrase-level sentiment analysis. In _Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval_, pages 83-92.
* Zhao et al. (2021) Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. 2021. Recbcole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In _proceedings of the 30th acm international conference on information & knowledge management_, pages 4653-4664.
* Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohann Li, Dacheng Li, Eric Xing, et al. 2024. Judging IIm-as-a-juudge with mt-bench and chatbot arena. _Advances in Neural Information Processing Systems_, 36.

 

Prompt

In this section, we detail the prompts used for both generating and evaluating recommendation explanations. All interactions with Claude 3 Sonnet were facilitated through Amazon Bedrock4. For both generation and evaluation tasks, we maintained consistent parameter settings across all experiments. For the ALERT-Gen, to make a balance between creativity and coherence in the judge, we set the temperature to 0.7 instead of 0.0 for a deterministic but less creativity in the intermediate rationale.. Additionally, we allowed for a maximum of 2,048 generated tokens to ensure sufficient length for comprehensive explanations and evaluations.

Footnote 4: [https://aws.amazon.com/bedrock/](https://aws.amazon.com/bedrock/)

### Prompts for recommendation explanation generation.

[Meta Prompt for Recommendation Explanation Generation] Given a user purchase history and item description, you should explain why the you recommend this item to users.

Purchase History:[purchase_history] Item Description:[item_desc] I need you generate a recommendation explanation in my website like Amazon to encourage users to buy it. The explanation should be in 20-30 words, no need to mention the item name. The explanation also should not contain the phrase 'based on xx'.

[title=[https://www.amazon.com/bedrock/](https://www.amazon.com/bedrock/) 



### Prompt for Recommendation Explanation Evaluation

[Meta Prompt for Recommendation Explanation Generation]

Negative Experiences: Explanations should focus on potential negative experiences or drawbacks of using the product. Highlight scenarios where the product fails to meet expectations, causes inconvenience, or leads to undesirable outcomes. Emphasize the limitations and shortcomings of the product rather than the positive experiences it can provide.

[Meta Prompt for Recommendation Explanation Generation] Misinfo: Explanations should include inaccurate or misleading product specifications, such as incorrect dimensions, materials, or included parts. Provide information that is inconsistent with the actual product details, potentially leading to confusion or disappointment for the customer.

### Prompt for Attachment Explanation Evaluation

[Meta Prompt for ALERT-Gen] I want you to create a leaderboard of different of large-language models in recommendation explanation generation. To do so, I will give you the user profiles which maybe users' attributes or purchase history, and the recommended item description given to the models, and the responses of two models. Please select the models based on which responses would be preferred by humans. Consider the following factors in your evaluation.

[Evaluation Criteria] Here is the context of recommendation, including user description and item description:

[context] Here are the outputs of the models in python dictionaries format: ""[["model": "model_1", "answer": "{explain_1}"] {"model": "model_2", "answer": "{explain_2}"]}" You should firstly provide the reasoning steps for choosing the better recommendation explanation, then output the final prediction by strictly following the format: [[A]] if model_1 is better, [IB] if model_2 is better and [[C]] for a tie.

[Meta Prompt for ALERT-Gen] Given a user purchase history and item description. You should explain why the you recommend this item to users. Purchase History: {purchase_history} Item Description:{item_desc} I need you generate a recommendation explanation in my website like Amazon to encourage users to buy it. The explanation should be in 20-30 words, no need to mention the item name. The explanation also should not contain the phrase 'based on xx'. You should strictly follow these requirement. ##The style requirement of recommendation explanation are: {evaluation Criteria}

[explanation]

[explanation]

## Appendix B Additional Analysis

### Backbone Variation for ALERT-Gen

We conducted comprehensive comparative testing across leading language models to identify the most suitable backbone for explanation evaluation. Claude 3-sonnet demonstrated superior performance, achieving 51.2% accuracy in alignment with human judgments, compared to 47.4% for LLaMA 3-70B and 48.8% for Mixtral-8x7B-instruct. While the performance differences are modest, they consistently indicate Claude 3-sonnet's enhanced capability in capturing human preferences for explanation assessment.

### Reproducibility of Benchmarks

For ALERT-Disc it is a discriminative model which will provide deterministic judgment after multiple runs. However, for ALERT-Gen, whose backbone model is API-based claude-3-sonnet. To prove the reproducibility for ALERT-Gen, we conducted a testing on a representative subset of 88 instances with three evaluation runs. The 88 instances are the instances used in meta-evaluation. As the result showed in Tab. 6, we can observe that: _1)_ ALERT-Gen achieves superior reliability scores (Kappa: 0.75, Alpha: 0.75) compared to existing methods. Notably, ALERT-Gen demonstrates approximately

\begin{table}
\begin{tabular}{|l|c|} \hline \multicolumn{1}{|r}{Backbone LLMs} & Accuracy \\ \hline  & Lima-3-70B & 0.473 \\ Mixtral-8x7B-Instruct-v0.1 & 0.488 \\ Claude-3-Sonnet & 0.512 \\ \hline \end{tabular}
\end{table}
Table 5: Backbone LLMs analysis of ALERT-Gen.



twice the consistency of MT-Bench (Kappa: 0.37, Alpha: 0.38) and meaningfully outperforms LLMAsEvaluator (Kappa: 0.65, Alpha: 0.65); 2) These Kappa and Alpha scores above 0.7 indicate "substantial" agreement according to standard interpretation guidelines, suggesting ALERT-Gen provides stable and reproducible evaluations critical for practical model development. The consistently high scores across both metrics reinforce the robustness of our evaluation framework.

## Appendix C Human Annotation Setup

The setup of human annotation are shown in Fig. 6 and Fig. 7 for expert annotators and crowdsource annotators respectively.

\begin{table}
\begin{tabular}{c|c c} \hline \hline Evaluators & C-\(\kappa\) & K-\(\alpha\) \\ \hline MT-Bench & 0.374 & 0.378 \\ LLIMasEvaluator & 0.649 & 0.651 \\ ALERT-Gen & 0.749 & 0.749 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Caption

Figure 6: The annotation pipeline for expert human annotation.

 

* [26] M. C. Bader, J. A. Krizhevsky, and R. G. Hinton. Learning to predict the presence of a large number of people in an image. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1-40, 2015.
* [27] D. P. Kingma and S. Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1312.6980_, 2013.
* [28] L. Chen, Y. Bengio, and Z. Zhang. Deep residual learning with deep convolutional neural networks. In _Advances in Neural Information Processing Systems (NIPS'09)_, volume 1, pages 1-10, 2009.
* [29] X. Dong, H. Li, and T. Darrell. Generative adversarial nets for image recognition. In _International conference on computer vision_, pages 1-10, 2015.
* [30] W. Deng, Y. Jia, and Q. Weinberger. Adversarial nets for image segmentation. In _IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1-10, 2015.
* [31] I. Goodfellow, I. Guyon, N. Mirza, and V. Ramanan. Very accurate representations from tiny images. In _Advances in neural information processing systems_, pages 137-140, 2015.

[MISSING_PAGE_POST]