Title: Max Explainability Score with Confidence Interval (MES-CI): A Quantitative Metric for Interpretability in Knowledge Graph-Based Recommender System
Venue: NeurIPS 2025
Accepted: no
Keywords: - Explainable AI

Recommender Systems

Knowledge Graphs

Confidence Interval

Explainability Evaluation Metrics

Reinforcement Learning
Abstract: Knowledge graph-based recommender systems (KGRS) utilize structured semantic relationships to generate personalized and interpretable recommendations, leveraging the inherent connectivity within knowledge graphs to enhance transparency. While KGRS offer significant advantages in explainability, quantifying the reliability and impact of these explanations remains challenging due to the complexity of underlying models and the multiple pathways that influence recommendation outcomes. This paper critically analyzes existing evaluation metrics for explainability in KGRS, identifying their limitations and advocating for a balanced framework that integrates interpretability with predictive accuracy. This research builds upon the existing Max Explainability Score (MES) by introducing an enhanced scoring mechanism, the Max Explainability Score with Confidence Interval (MES-CI). MES-CI overcomes the limitations of evaluating the explainability of generated recommendations using a single-point score by providing a more comprehensive and balanced assessment. It incorporates confidence intervals alongside confidence score percentages, offering a clearer representation of explainability reliability. Furthermore, the applicability of this refined metric is examined across multiple datasets, with case studies demonstrating its effectiveness in improving transparency and user trust in AI-driven recommendation systems.

Abstract

Knowledge graph-based recommender systems (KGRS) utilize structured semantic relationships to generate personalized and interpretable recommendations, leveraging the inherent connectivity within knowledge graphs to enhance transparency. While KGRS offer significant advantages in explainability, quantifying the reliability and impact of these explanations remains challenging due to the complexity of underlying models and the multiple pathways that influence recommendation outcomes. This paper critically analyzes existing evaluation metrics for explainability in KGRS, identifying their limitations and advocating for a balanced framework that integrates interpretability with predictive accuracy. This research builds upon the existing Max Explainability Score (MES) by introducing an enhanced scoring mechanism, the Max Explainability Score with Confidence Interval (MES-CI). MES-CI overcomes the limitations of evaluating the explainability of generated recommendations using a single-point score by providing a more comprehensive and balanced assessment. It incorporates confidence intervals alongside confidence score percentages, offering a clearer representation of explainability reliability. Furthermore, the applicability of this refined metric is examined across multiple datasets, with case studies demonstrating its effectiveness in improving transparency and user trust in AI-driven recommendation systems.

1 Introduction

Recommender systems (RSs) are designed to suggest relevant items to users based on their prefer20 ences and can be developed using traditional approaches such as content-based filtering, collaborative 21 filtering, and hybrid methods [1]. While these systems achieve high accuracy, they often lack trans22 parency and fail to provide clear explanations for their recommendations. To address this limitation, 23 explainable recommendation systems (XRS) have emerged, offering personalised recommendation 24 algorithms specifically designed to clarify the reasoning behind suggested items. By incorporating 25 detailed explanations, XRS enhances user trust and understanding, ensuring that recommendations 26 are not only accurate but also interpretable [2]. 27

XRS are predominantly built using knowledge graphs (KG), which are structured as sub28

ject-predicate-object (SPO) triples within a heterogeneous graph. The KGs have gained significant 29

traction for their ability to harness structured semantic relationships, enabling more precise and 30

personal recommendations. By enhancing contextual understanding, KGs address limitations present 31

in traditional recommender models, offering deeper insights into user preferences [3]. However, 32

while KGs contribute to explainability, quantifying or systematically evaluating this interpretability 33

remains a challenge [4]. 34

Researchers typically assess the effectiveness of XRS explainability through user studies, online 35 experiments, and offline evaluations based on historical datasets. For instance, Cao et al. [5] assessed 36 the effectiveness of their PR4SR model by conducting a study with 50 participants. Each participant 37 was assigned 30 randomly selected cases from the Beauty, Cellphones, and Baby datasets. The study 38 included a detailed questionnaire that provided information such as the user's session history, the 39 starting node for path reasoning, and an explanation of the generated path. Participants were asked to 40 evaluate the selection of the starting node and the explainable paths based on their comprehension of 41 the session history. Similarly, Liu et. al. [6] conducted a crowdsourced evaluation, enlisting the 100 42 most active users to assess the explanations provided by their Ante-RNN model. However, explain43 ability evaluation metrics derived from qualitative user studies often remain subjective and prone to 44 confirmation biases [7], posing challenges in obtaining objective assessments of interpretability. 45

46

47

48

49

50

51

52

53

54

55

While Rosenfeld [8] introduced a quantitative explainability metric, it still requires human interaction to evaluate discrepancies between the agent model and its corresponding logical explanation. Tiwary et al. [9] proposed the Max Explainability Score (MES) as a quantitative metric for assessing recommendation explainability within KG-based frameworks. MES quantifies explainability by capturing surprising information, which is influenced by factors such as attributable features, their quality (i.e., the probability of occurrence in the recommendation generation path), information value (i.e., entropy derived from traversal paths), the relevance of the recommended item, and the rewards associated with traversal paths. However, MES relies on a single-point evaluation. A more comprehensive approach would incorporate confidence interval limits alongside the evaluation score, such as a 95% confidence level, providing a more detailed and reliable assessment of explainability.

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

This paper seeks to address the gap in explainability evaluation for knowledge graph-based recommender systems (KGRS) by reviewing existing metrics and enhancing the MES. Specifically, we extend MES beyond a single-point score by incorporating a confidence interval, offering a more reliable assessment of explainability. Through case studies, we validate the proposed explainability metric's applicability across diverse domains, setting a foundation for future research in XRS.

The primary contributions of this study encompass the following key aspects:

A comprehensive review of current quantitative explainability metrics for KGRS.

An extension of MES to include confidence interval-based scoring, ensuring a more nuanced evaluation of explainability.

This paper is structured as follows: Section 2 reviews background and related work on XRS explainability. Section 3 outlines the proposed explainability measurement algorithm, while Section 4 details the experimental framework. Section 5 provides an in-depth analysis, including a case study on evaluation mechanisms. Finally, Section 6 presents conclusions and future research directions.

2 Background and related work

This section outlines the foundational concepts and examines relevant research.

2.1 Why Explainability?

Explainability in AI refers to the ability to clearly convey the reasoning behind a model's output. It addresses critical questions such as: How does the system operate? What factors influence specific predictions? Why were particular recommendations made for the user? Which explanation best supports a given prediction or recommendation? By providing transparency into data processing mechanisms, explainability not only enhances user comprehension but also helps uncover potential biases and limitations within the system. A well-explained AI model fosters trust and confidence, making it more reliable for both users and decision-makers. Since many AI models function as black boxes, it is crucial to ensure users have clear insights into how predictions or recommendations are generated [10]. According to GDPR and other relevant regulations, users have the fundamental right to understand the rationale behind AI-generated decisions. Transparency in AI fosters greater trust and engagement, reinforcing its credibility and encouraging wider adoption in real-world applications.

Recent advancements in KGs, reinforcement learning (RL), and large language models (LLMs) are 83 playing a significant role in shaping explainable AI (XAI), enabling AI systems to provide clearer, 84 more interpretable insights [11]. As RSs become increasingly sophisticated, ensuring the validity and 85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

reliability of their generated explanations is crucial. Without proper validation, explanations may lack consistency or fail to provide meaningful insights into how recommendations are derived. Therefore, there is a growing need for robust evaluation frameworks that assess the quality, relevance, and impact of explanations, ultimately improving transparency and accountability in AI-driven decision-making.

2.2 Explainability implementation in XRS

Explainability in AI is achieved through two main approaches: embedded methodologies and post-hoc techniques. Embedded methods integrate explainability directly within the AI model, ensuring that the reasoning behind predictions is generated internally before producing an outcome. In contrast, post-hoc techniques analyze decisions after the output has been generated, using methods such as feature importance analysis, surrogate models, and counterfactual reasoning to provide insights into the model's decision-making process [3].

To enhance XRS, researchers incorporate KGs into RSs as auxiliary information, improving both performance and interpretability. This integration leverages structured relationships between entities, making recommendations more intuitive and explainable. Several KG-based RS methodologies have been explored, including post-hoc approaches, KGE methods, path-based embedding strategies, and unified frameworks, each contributing to improved explainability and user trust in AI-generated recommendations [2].

Post-hoc explainability methods generate explanations after model development by applying soft matching algorithms to enhance recommendation transparency. A widely recognized post-hoc approach for assessing model outcomes is determining the significance of various features in influencing predictions. Feature importance can be quantified using post-hoc methods, such as Shapley values, which are derived from game theory principles [12]. This method enables recommendations to align more closely with user preferences while ensuring interpretability.

KGE-based approaches integrate explainability directly into the recommendation generation process by embedding structured semantic relationships. These methods determine entity similarity by evaluating the distance between their embeddings, often integrating item-side attributes [13] or user preferences [14] within user-item KGs. By embedding entities effectively, these methods enhance both prediction accuracy and explanation consistency.

Path-based recommendation techniques leverage the KG structure and its meta-paths to infer meaningful connections between users and items. Li et al. [15] proposed the complex-to-concise (C2C) meta-multigraph, designed to facilitate message propagation from complex structures to more concise representations as it traverses the depth of the meta-multigraph. Xian et al. [16] developed a policy-guided path reasoning algorithm that applies reinforcement learning (RL) to identify the most relevant paths between user-item pairs, optimizing recommendation reliability while maintaining explainability.

Unified approaches combine embedding-based and path-based techniques, benefiting from the strengths of both semantic embeddings and structured KG traversal patterns. These hybrid methods aim to maximize interpretability while retaining high predictive accuracy, ensuring that recommendations are both transparent and relevant. By integrating entity representations with structured reasoning frameworks, unified methods improve user trust and facilitate the deployment of AI-driven personalized RSs [3].

Overall, leveraging KGs in RSs enhances explainability by providing structured, interpretable relationships between entities, allowing AI systems to generate recommendations that align with user expectations while maintaining transparency. As research progresses, developing more robust frameworks that balance interpretability, and predictive performance will remain crucial for advancing explainable AI-driven recommendation technologies.

2.3 Explainability evaluation in XRS

XRS algorithms are traditionally evaluated using standard metrics such as root mean square error

133 (RMSE) and mean absolute error (MAE) for rating predictions, as well as accuracy , recall , Hit Rate , 134 F-measure , and normalized discounted cumulative gain (NDCG) for top-n recommendations. While 135 these metrics provide insight into the predictive performance of RSs, assessing explainability requires 136

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

2.3.1 Explainability - qualitative evaluation

Qualitative evaluation of explainability in RS increasingly incorporates user-centered methods such as user studies, offline simulations, and online A/B testing to assess user satisfaction, trust, and perceived transparency. User studies involve direct interaction with participants through interviews, surveys, or usability tasks to gather subjective feedback on the quality and helpfulness of explanations. Offline assessments use historical data to simulate user responses and analyze interpretability or transparency metrics. Meanwhile, online assessments track real-time user behavior, such as dwell time, conversion rates, and interaction patterns, to measure the actual impact of explanations on user engagement and decision-making [18]. These approaches, although effective, are constrained by practical limitations such as sample size, scalability, and the subjective nature of human feedback.

2.3.2 Explainability - quantitative evaluation

Quantitative evaluation of explainability in AI systems is demonstrated in prior work [8], where explainability measures are derived from performance variations across models of different fidelity. These measures include the number of rules in the generated explanations, the quantity of features utilized for explanation generation, and the stability of the system's explanatory framework. Despite its contributions, this approach does not explicitly address explainability evaluation in RSs, particularly those leveraging KGs and related technologies. Additionally, it requires human interaction to compare the system's model outputs with the logical reasoning presented in its explanations, adding another layer of complexity.

Recently, Tiwary et al. [9], proposed MES, a quantitative metric designed to evaluate the explainability of AI-driven RSs, particularly those leveraging KG. Unlike traditional explainability measures that rely on subjective user studies or post-hoc evaluations, MES provides a structured approach to assessing how well a RS conveys transparent and interpretable insights. MES quantifies explainability by capturing surprising information, which is influenced by several key factors:

Attributable Features : The characteristics contributing to a recommendation, ensuring that users can trace the reasoning behind suggested items.

Feature Quality : The probability of occurrence of these features within the recommendation generation path, reflecting their reliability.

Information Value : The entropy derived from traversal paths in the KG, indicating how much new or unexpected information is introduced.

Rewards Assigned to Traversal Paths : The significance of different paths taken within the KG to arrive at a recommendation, reinforcing the credibility of the explanation.

MES is particularly valuable in KGRS, where structured semantic relationships can enhance transparency. By advancing beyond conventional explainability scores, MES contributes to the development of fair, user-centric, and accountable AI-driven RSs, reinforcing trust and usability in AI applications.

2.4 Analysis of MES

The MES formula is structured around two key components of explainability: Reward gain and Information gain. The central question is whether these parameters are meaningful within the context of XAI and KGRS? AI models generate predictions and recommendations by analyzing structured relationships within data, ensuring that outputs align with historical user behaviors, actions, and future needs. This interconnected approach allows AI systems to uncover patterns in user interactions, providing insights into preferences and decision-making. [19]. KGs play a crucial role in enhancing explainability by systematically capturing and representing entity relationships. They enable AI models to traverse structured pathways, offering a clear rationale behind recommendations. This graph-based traversal serves two essential purposes: selecting relevant items for users and justifying those selections with interpretable reasoning. Unlike opaque black-box models, KG-based AI systems provide transparency, fostering trust and confidence [20]. However, a key challenge lies in determining the most reliable explanatory pathway among multiple connections between users and recommended items. Not all connections hold equal importance, and identifying the most interpretable and relevant explanation requires well-defined methodologies.

This is where the MES framework introduces reward gain and entropy gain to quantify explainability. 189

190

191

192

193

Reward gain : Reflects the value or significance of a traversal path in the KG. The model assigns rewards to specific paths based on their relevance, credibility, and alignment with user preferences. The higher the reward for a path, the more logically grounded and interpretable the explanation.

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

Information gain : Captures the information richness and surprise factor of a given path. A recommendation is more explainable if the traversal path provides meaningful new information rather than redundant or overly predictable connections.

The explainability of a recommendation is influenced by the reward assigned to a path within the KG and the quality of information it provides. A well-designed explainability framework must strike a balance between semantic relevance (ensuring recommendations are logically grounded) and novel insight (offering valuable, non-trivial information). This equilibrium fosters a deeper understanding, trust, and engagement among users interacting with AI-generated recommendations, ultimately driving broader adoption and improved usability of interpretable AI systems.

A key limitation of MES in its original form is its reliance on single-point evaluation, which may not fully capture the variability and confidence in explainability assessments. To address this, we introduce confidence interval-based scoring, allowing for a more nuanced and equitable evaluation. By incorporating confidence scores-such as a 95% confidence level-the proposed metric offers a more robust measure of explainability, ensuring that recommendations are both interpretable and statistically sound. In the following section, we introduce our proposed approach, which quantitatively assesses the explainability performance of KGRS by integrating structured methodologies that balance interpretability and predictive accuracy.

3 Overviews of the proposed approach

The proposed approach aims to enhance the explainability evaluation of KGRS by introducing a refined framework built upon the MES. While traditional MES operates on single-point evaluation, the enhanced framework integrates confidence interval-based scoring, ensuring a more nuanced and statistically robust assessment of explainability. By establishing confidence scores, such as a 95% confidence level, we improve reliability and precision in measuring the transparency of recommendations.

The proposed approach, as shown in Algorithm 1 , systematically assesses the explainability score of each path that connects a recommended product to a user, ensuring transparency and interpretability in AI-driven recommendations. The process begins by computing the standard deviation of all explainability scores associated with the product's various pathways. This step quantifies the variability in explainability across different paths, providing a measure of consistency and deviation within the recommendation framework.

Once the standard deviation is established, the approach proceeds to evaluate the confidence interval , specifically determining the 95% lower and upper bounds of the explainability score. This interval estimation enhances the robustness of the explainability assessment, offering a statistical range that accounts for potential fluctuations in how recommendations are justified.

Following this, the system identifies the path with the highest explainability score, selecting the most interpretable and relevant route that best aligns with user understanding and trust. The final output presents the maximum explainability score (MES) alongside its associated 95% confidence interval bounds, ensuring that the explanation is not only optimal but also statistically validated. By incorporating confidence intervals, this refined methodology strengthens the credibility of explainable recommendations, fostering greater user engagement and trust in AI-driven decision-making processes.

235

236

237

238

239

240

241

242

243

244

245

4 Experimentations

4.1 Dataset

The experiment utilizes domain-specific datasets from Amazon, focusing on two e-commerce categories: clothing and beauty, which were previously employed in prior research [9]. Each dataset comprises six key entities-user, product, product's feature words, related products, brand, and category-interconnected through eight distinct relationship types, as depicted in Figure 1. Users can purchase multiple products, each associated with specific categories and brands, while also exhibiting additional behavioral connections such as joint purchases, product views, and relationships with related products. Product descriptions incorporate feature words mentioned by users, further enriching the dataset. Together, these elements form user-centric KGs with clearly identifiable entities and structured relationships, facilitating more interpretable and personalized recommendations.

Produced_By

Brand

Also_Bought

Related

User

Purchase→

Product

Also_Viewed

Product

Described_As

Mention

Word

Belongs_To→ Category

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

4.2 Experimental methodology

Building on the previously established MES explainability framework, we have enhanced its capabilities while maintaining a similar experimental setup. The experiment involves developing an XRS model using RL based on users' historical purchases, with separate training (70%) and test (30%) datasets derived from transaction records. Initially, relevant entities-including product relationships and user reviews-are extracted from an application-specific e-commerce dataset. A label property graph (LPG) is employed for KG construction, structuring entities and relationships. The experiment then progresses to KGE generation using the TransE algorithm to create vector representations. The RL framework integrates KGE within an Actor-Critic model, where path traversal actions that align with predefined patterns are rewarded, while deviations incur penalties. Model effectiveness is evaluated using the test dataset, applying metrics such as NDCG, recall, hit rate (HR), and precision to assess recommendation quality. Ultimately, the explainability framework generates a quantitative explainability score with confidence interval, offering valuable insights into recommendation transparency and reinforcing interpretability in AI-driven decision-making.

5 Results and Analysis

This section presents an analysis of the experiments conducted with the proposed algorithm. Additionally, it explores the mechanisms used for explainability evaluation, demonstrated through a case study. Lastly, it examines the limitations inherent in the algorithm.

5.1 Validation sets and benchmarks

Having expanded the capabilities of the previously published explainability evaluation mechanism, MES, we adopted a similar evaluation framework as described in that paper. Both e-commerce datasets include corresponding test datasets, accounting for approximately 30% of the total transactions. To validate the model's performance, we utilized metrics such as NDCG, precision, HR, and recall.

The central focus of this paper is to evaluate the explainability of recommended items. Despite an extensive review of existing literature, we found MES to be the only reliable explainability evaluation mechanism, as it measures explainability by considering both information gain and reward gain. Building upon this foundation, we extend the metric by incorporating confidence intervals for the defined explainability measures, enhancing robustness and reliability. Consequently, we deliberately excluded discussions on model efficacy, limiting the scope strictly to the explainability of recommendations.

5.2 Explainability Evaluation - A Case Study

To assess explainability, consider an example from the Beauty dataset. User 21001 had previously purchased products [11808, 7381, 9141, 10747], as shown in Table 1. Based on these past purchases, the RS identifies path patterns to suggest new products. In this case, our KG and RL-based XRS generated a top-10 recommendation list: [11772, 8471, 9576, 5351, 1690, 5603, 8015, 1465, 10872, 5934]. The test dataset contains the actual products the user is expected to purchase in the future, which in this example is [11772]. This setup enables an evaluation of the system's predictive accuracy while offering insights into the transparency and interpretability of its recommendations.

Providing users with explanations for the recommended top-10 products would enhance their understanding of the model's decision-making process. Users may wonder how the system determines these recommendations, how it recognizes their preferences, and how it interprets their needs. Clear and well-structured explainability fosters trust in the RS, ensuring users feel confident in its relevance and utility.

This paper leverages the existing MES framework to enhance the explainability of the recommended top-10 products. As shown in Table 2, six possible explanations exist for recommending product 11772 to user 21001. However, determining which of these explanations is most meaningful and aligns with user preferences is essential. A key consideration is whether the user would find any of these explanations satisfactory or if they have specific criteria for evaluating them.

According to the MES framework, users seek to maximize both the rewards gained from the provided explanation and the informational value it conveys. In this scenario, among the six possible explana-

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

tions, the statement "User 21001 purchased product 10747, which was produced by brand 201, who also produced product 11772" achieved the highest explainability score. Consequently, the model selects the explanation with the highest score to ensure optimal transparency and user trust in the recommendations.

As previously mentioned, the objective of this paper is to introduce confidence interval-based scoring for MES, which currently operates as a single-point value. Table 3 presents the explainability for all the top-10 recommended products, including their MES values and the corresponding confidence intervals. As outlined in Table 2, the process of selecting the most meaningful explanation for a recommended product involves identifying the highest-scoring explainability. Table 3 extends this approach by displaying the explainability scores of all the top-10 recommended products alongside their MES values and their respective confidence intervals, offering a more comprehensive assessment of recommendation transparency.

The evaluation of confidence intervals follows Algorithm 1, where the model first calculates the standard deviation of all possible explainability scores. Based on the desired confidence level, it then determines the corresponding confidence interval. Table 3 presents the 95% confidence interval for the respective MES values. In some instances, the MES-CI field is blank, as there was only one single possible explanation for the recommended product in those cases. The inclusion of confidence interval values strengthens user trust by providing a clearer assessment of MES and the corresponding explainability of recommendations.

5.3 Limitation

While the MES-CI framework enhances explainability assessment by incorporating confidence 316 intervals, it has certain limitations. It relies on the statistical distribution of explainability scores, 317 assuming a well-behaved spread, which may not accurately reflect true reliability in cases of high 318 variance or skewness. Additionally, for recommended products with only one possible explanation, 319 MES-CI cannot provide meaningful confidence bounds, limiting interpretability in those instances. 320 The framework is also sensitive to sample size, as a smaller number of explanations can result in 321 wider or unstable confidence intervals, affecting robustness. Addressing these limitations could 322 further improve the reliability and practical usability of MES-CI in XAI systems. 323

324

325

326

327

328

329

330

331

332

6 Conclusion and future works

This paper builds upon the existing MES framework by introducing a confidence interval-based scoring mechanism to improve the explainability of recommendations. By incorporating confidence intervals, the proposed approach enhances user trust and provides a more nuanced evaluation of explainability beyond a single-point metric. The results demonstrate that this refined methodology offers a deeper understanding of recommendation transparency, allowing users to assess both the reliability and informational value of explanations. The approach ensures that the most meaningful explanations that maximize both rewards and information gain-are selected, thereby improving the overall effectiveness of XRS.

Future research will explore additional enhancements to the MES-CI framework to address its limita333 tions, such as refining confidence interval calculations for sparse datasets and ensuring adaptability in 334 dynamic user contexts. Investigating alternative statistical models for explainability quantification 335 may further improve the robustness of confidence intervals. Additionally, extending MES-CI to 336 multi-modal RSs and real-time personalization scenarios will help validate its applicability across 337 diverse AI-driven environments. These efforts will contribute to the continuous advancement of 338 transparent, interpretable, and user-centric AI systems. 339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

References

S. Al-Ghuribi, S. Mohd Noah, T. Mohammed, N. Tiwary, and N. Saat. A comparative study of sentimentaware collaborative filtering algorithms for arabic recommendation systems. IEEE Access , 12:174441174454, 2024. doi:10.1109/ACCESS.2024.3489658.

N. Tiwary, S. Mohd Noah, F. Fauzi, and T. Yee. A review of explainable recommender systems utilising knowledge graphs and reinforcement learning. IEEE Access , 12:91999-92019, 2024. doi:10.1109/ACCESS.2024.3422416.

Q. Guo, F. Zhuang, C. Qin, H. Zhu, X. Xie, H. Xiong, and Q. He. A survey on knowledge graph-based recommender systems. IEEE Transactions on Knowledge and Data Engineering , 34(8):3549-3568, 2020. doi:10.1109/TKDE.2020.3028705.

Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Computing Surveys , 55(13s), 2023. 10.1145/3583558.

Yang Cao, Shuo Shang, Jun Wang, and Wei Zhang. Explainable session-based recommendation via path reasoning. IEEE Transactions on Knowledge and Data Engineering , 37(1):278-290, 2025. 10.1109/TKDE.2024.3486326.

Peng Liu, Lemei Zhang, and Jon Atle Gulla. Dynamic attention-based explainable recommendation with textual and visual fusion. Information Processing & Management , 57(6):102099, 2020. https://doi.org/10.1016/j.ipm.2019.102099.

Yi-Shan Lin, Wen-Chuan Lee, and Z. Berkay Celik. What do you see? evaluation of explainable artificial intelligence (xai) interpretability through neural backdoors. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , KDD '21, page 1027-1035, New York, NY, USA, 2021. Association for Computing Machinery. 10.1145/3447548.3467213.

Avi Rosenfeld. Better metrics for evaluating explainable artificial intelligence. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems , AAMAS '21, page 45-50, Richland, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems. https://dl.acm.org/doi/10.5555/3463952.3463962.

Neeraj Tiwary, Shahrul Azman Mohd Noah, Fariza Fauzi, and Tan Siok Yee. Max explainability score-a quantitative metric for explainability evaluation in knowledge graph-based recommendations. Computers and Electrical Engineering , 116:109190, 2024. https://doi.org/10.1016/j.compeleceng.2024.109190.

Enayat Rajabi and Kobra Etminani. Knowledge-graph-based explainable ai: A systematic review. Journal of information science , 50(4):1019-1029, 2024. https://doi.org/10.1177/01655515221112844.

Nourhan Ibrahim, Samar Aboulela, Ahmed Ibrahim, and Rasha Kashef. A survey on augmenting knowledge graphs (kgs) with large language models (llms): models, evaluation metrics, benchmarks, and challenges. Discover Artificial Intelligence , 4(1):76, 2024. https://doi.org/10.1007/s44163-024-00175-8.

Yong-Geon Lee, Jae-Young Oh, Dongsung Kim, and Gibak Kim. Shap value-based feature importance analysis for short-term load forecasting. Journal of Electrical Engineering & Technology , 18(1):579-588, 2025. 10.1007/s42835-022-01161-9.

Y. Zhang, Y. Shi, D. Yang, and X. Gu. Exploiting explicit item-item correlations from knowledge graphs for enhanced sequential recommendation. Information Systems , 128:102470, 2025. doi:10.1016/j.is.2024.102470.

S. Geng, Z. Fu, J. Tan, Y. Ge, G. De Melo, and Y. Zhang. Path language modeling over knowledge graphsfor explainable recommendation. In Proceedings of the ACM Web Conference 2022 , pages 946-955, New York, NY, USA, 2022. Association for Computing Machinery. doi:10.1145/3485447.3511937.

C. Li, H. Xu, and K. He. Meta-multigraph search: Rethinking meta-structure on heterogeneous information networks. Knowledge-Based Systems , 289:111524, 2024. doi:10.1016/j.knosys.2024.111524.

Y. Xian, Z. Fu, S. Muthukrishnan, G. de Melo, and Y. Zhang. Reinforcement knowledge graph reasoning for explainable recommendation. In SIGIR'19 , pages 285-294, New York, NY, USA, 2019. Association for Computing Machinery. doi:10.1145/3331184.3331203.

Marek Pawlicki, Aleksandra Pawlicka, Federica Uccello, Sebastian Szelest, Salvatore D'Antonio, Rafał 390 Kozik, and Michał Chora´ s. Evaluating the necessity of the multiple metrics for assessing explainable ai: A 391 critical examination. Neurocomputing , 602:128282, 2024. https://doi.org/10.1016/j.neucom.2024.128282. 392

Chun-Hua Tsai and Peter Brusilovsky. The effects of controllability and explainability in a so393 cial recommender system. User Modeling and User-Adapted Interaction , 31(3):128282, 2021. 394 https://doi.org/10.1007/s11257-020-09281-5. 395

Yongfeng Zhang and Xu Chen. Explainable recommendation: A survey and new perspectives. Foundations 396 and Trends® in Information Retrieval , 14(1):1-101, 2020. 10.1561/1500000066. 397

S. Ji, S. Pan, E. Cambria, P. Marttinen, and . P. Yu. A survey on knowledge graphs: Representation, 398 acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems , 33(2):494399 514, 2022. doi:10.1109/TNNLS.2021.3070843. 400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: [TODO] The abstract and introduction clearly outline the paper's claims, detailing its contributions while highlighting key assumptions and limitations.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: [TODO] A dedicated Section 5.3 has been introduced to outline the limitations of the proposed approach.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [TODO] We have not used any theoretical assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: [TODO] We have already provided the algorithm used in this paper. We will also share the additional supplement material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: [TODO] We will make it available for open access through our GitHub repository and will also provide our code base as supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿✴✴♥✐♣/a115✳❝❝✴ ♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿ ✴✴♥✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: [TODO] Adedicated section has been included for experiments, with additional information provided in the appendix and supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: [TODO] The 95% confidence interval has been employed as a statistical measure to evaluate the reliability and consistency of explainability in the generated recommendations. By constructing this interval, we can quantify the uncertainty surrounding the explainability scores and determine the range within which the true quality of explainability is likely to fall with a high degree of confidence.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: [TODO] The details has been provided in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ❤/a116/a116♣/a115✿✴✴♥❡✉/a114✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❊/a116❤✐❝/a115●✉✐❞❡❧✐♥❡/a115 ?

Answer: [Yes]

Justification: [TODO] We have adhered to ethical guidelines throughout this work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: [TODO] This paper focuses on explainable AI, a crucial aspect in ensuring transparency and accountability in AI systems. Explainability is particularly significant in compliance with regulations such as the General Data Protection Regulation (GDPR) and European Union (EU) laws governing AI. These regulations emphasize the need for interpretable decision-making processes, allowing users to understand and challenge automated recommendations. By advancing explainability metrics, such as MES-CI, this work

contributes to the broader efforts of aligning AI-driven recommendations with legal and ethical standards.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [TODO] The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: [TODO] This paper appropriately cites all referenced works.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ♣❛♣❡/a114/a115✇✐/a116❤❝♦❞❡✳❝♦♠✴❞❛/a116❛/a115❡/a116/a115 has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [TODO] The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [TODO] No crowdsourcing or research involving human subjects has been conducted in this study.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [TODO] No crowdsourcing or research involving human subjects has been

713

conducted in this study.

Guidelines: 714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: [TODO] The LLM is used for editing and formatting purpose only.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( ❤/a116/a116♣/a115✿✴✴♥❡✉/a114✐♣/a115✳❝❝✴❈♦♥❢❡/a114❡♥❝❡/a115✴✷✵✷✺✴▲▲▼ ) for what should or should not be described.

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

A Technical Appendices and Supplementary Material

This section includes technical appendices featuring supplementary results, figures, graphs, and mathematical proofs.

A.1 Data

A.1.1 Data Description

The experimental data was sourced from previous research [16] and is available at the provided link. The experiment employed two domain-specific datasets from the Amazon e-commerce platform, specifically the Clothing and Beauty domains. These datasets were selected due to their richness in user-product interactions and the availability of diverse product metadata, making them ideal for evaluating knowledge-aware and XRS. As illustrated in Figure 1, each dataset was modeled using a structured schema incorporating six key entity types: User, Product, Product's Feature Words, Related Products, Brand, and Category.

The relationships among these entities were both varied and semantically meaningful, encompassing eight distinct relationship types that reflected different aspects of e-commerce interactions. For instance, users interacted with products through purchases, which were further contextualized by product categories (e.g., shirts, makeup kits) and brands (e.g., Nike, L'Oréal). Additionally, each product was associated with a set of related products through behavioral patterns captured in the dataset-such as 'also viewed', 'also bought', or 'bought together' relationships. These links provided valuable insights into co-purchasing behavior and product affinity, which are essential for collaborative filtering and path-based reasoning.

Moreover, the datasets included feature words that were extracted from user reviews and product descriptions. These words served as textual signals reflecting product characteristics and user sentiments. The connections between products and these feature words enabled the model to incorporate semantic information into the recommendation process, enhancing interpretability and enabling more personalized results.

To effectively organize and represent this complex, multi-relational data, a user-centric KG was formed. This KG helped identify and formalize the various entities and their relationships in the form of directed triples ( h, r, t ) where h represents the head entity, r the relationship, and t the tail entity. For example, a triple like (User A, purchased, Product B) or (Product B, produced_by, Brand C) encodes a specific piece of structured knowledge that the model can leverage. This structured representation allowed the recommender system to perform path-based reasoning, uncover latent connections, and generate recommendations that are both contextually relevant and explainable.

Overall, this phase established a rich, semantically grounded foundation that supported subsequent stages in the pipeline, including knowledge graph embedding, reinforcement learning, and the generation of transparent, personalized recommendations..

A.1.2 Data Summary

Table 4 provides a statistical summary that offers valuable insights into the entities, relationships, and characteristics of the datasets used in the experiment, enabling a deeper understanding of the data structure. The top section of the table outlines the number of entities present within each dataset, while the bottom section focuses on the relationships between head and tail entities, presenting the mean and standard deviation values. Among the various relationships, 'mention' and 'described as' are consistently prevalent across all datasets, underscoring their significance in capturing product features. However, these relationships may contain redundant words, which can be refined using the Term Frequency - Inverse Document Frequency (TF-IDF) technique to filter out less relevant feature words. Additionally, within the Product and Related Products datasets, the 'also bought' relationship stands out as the most dominant, highlighting its strong influence in modeling user purchasing behavior.

A.1.3 Data Snippet

Table 5 provides a detailed overview of the Amazon 'clothing' e-commerce dataset used in the experiment. The upper section summarizes key entities within the dataset, with anonymized details for users, products, and related product entities. The lower section

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

highlights the three most significant relationships in the dataset, helping to contextualize interactions between entities. For instance, User #431, represented in the user entity, purchases Product #8374 from the product entity, forming the relationship 'user purchasing product.' Additionally, User #431 references specific feature words such as #8028 and #12395, which are recorded in the word entity while purchasing Product #8374. Similarly, Product #8374 is linked to multiple feature words, including #8028, #12395, and others, establishing the 'product described by words' relationship. The subsequent columns illustrate associations between products and various entities, including brand, category, and related products. Each row corresponds to a sequential product identifier. For example, Product #1 in the product entity is manufactured by Brand #8 in the brand entity and classified under Categories #0, #25, #423, #711, and #39 in the category entity.

Additionally, #Product_1 is frequently purchased under the 'Also Bought' relationship alongside related products such as #157044, #40113, and many others. Furthermore, #Product_1 is often viewed under the 'Also Viewed' category alongside related products like #7644, #7647, #8868, and others. Lastly, #Product_1 is commonly purchased together with related products such as #40113 and #148881.

This structured representation highlights the intricate network of connections within the Amazon 'clothing' e-commerce dataset, illustrating the dynamic interplay between entities

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

and their relationships. Essentially, entities function as dimensions, while relationships serve as factual links that connect these dimensions.

A.2 Experiment

A.2.1 Experimental Environment

For a comprehensive overview of the experiment's specifics, detailed information can be found in Table 6, where the experimental settings are documented.

A.2.2 Experimental Setup

The research utilized a comprehensive transaction dataset that was systematically divided into training and test subsets. Each subset encompassed critical information such as users' product purchase histories, the occurrence of feature-related word mentions, and detailed product descriptions. This structured data enabled the development of a semantically enriched RS. Table 7 presents a summary of statistical information, including the mean and standard deviation for each subset across two distinct domains: Clothing, and Beauty. These statistics offer insight into user behavior and interaction patterns, highlighting variations in the number of products purchased and the frequency of feature word mentions.

For example, in the "Clothing" domain, users purchased an average of 7.08 products with a standard deviation of 3.59, reflecting a moderate variation in user activity. When broken down by dataset split, users in the training set purchased an average of 5.45 products (SD = 2.49), whereas those in the test set purchased 1.62 products on average (SD = 1.13). This disparity underscores the segmentation strategy, which simulates real-world scenarios where models are trained on historical data and evaluated on unseen user behavior. Similar trends were observed across the other domains, providing a diverse and robust experimental foundation.

The experimental framework was implemented using Python as the primary programming language, with PyTorch employed as the deep learning library due to its flexibility and efficiency in building complex neural models. The proposed RS integrated KG and RL technologies to generate semantically informed and context-aware recommendations. The KG served to represent structured domain knowledge, capturing intricate relationships between users, products, and attributes. RL was used to dynamically model user preferences over time, enabling the system to learn optimal recommendation strategies through interaction and feedback.

This hybrid approach aligns with the broader objectives of XAI, as it facilitates transparency in recommendation reasoning by leveraging semantic relationships and adaptive decisionmaking. To evaluate the effectiveness of the RS, standard top-N recommendation metrics were employed, including NDCG, Recall, HR, and Precision. These metrics provide a comprehensive assessment of the system's ability to rank relevant items accurately, capture user interests, and deliver consistent performance across varied domains.

A.3 Experiment Methodology

A.3.1 Data Preprocessing

The data preprocessing phase plays a critical role in preparing the dataset for semantic recommendation. This stage involves systematically extracting key components such as

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

relevant entities (e.g., products, attributes), relationships (e.g., user-product interactions, product-feature associations), and detailed user behavior data. To ensure that the input data is both meaningful and manageable, a robust filtering mechanism is employed.

Specifically, the filtering criterion focuses on retaining only those feature words that contribute valuable semantic information. Feature words that appear more than 5,000 times are excluded, as they are likely to be overly common or generic (e.g., 'good,' 'product,' 'buy') and thus contribute little to the specificity required for effective recommendations. Additionally, to further refine the dataset, only words with a Term Frequency-Inverse Document Frequency (TF-IDF) score greater than 0.2 are retained. This threshold ensures that the selected words are not only infrequent but also contextually important across the document corpus, highlighting their relevance to specific products or user preferences.

Through this process, the final dataset is enriched with significant and pertinent feature words, which enhances the quality of the KG and ultimately improves the performance and explainability of the RS.

A.3.2 Knowledge Graph Generation

In this phase, a critical step involves constructing a Label Property Graph (LPG) from the training dataset. The LPG serves as a foundational data structure that models the semantic and relational aspects of the domain. It is built by extracting and organizing key entities-such as users, products, and feature words-along with their bi-directional relationships and interactions. These relationships include user-product interactions (e.g., purchases or ratings), product-feature associations, and semantic links between products and descriptive terms. By incorporating directionality, the graph captures not only the existence of relationships but also the flow of influence or interaction between entities, enhancing the expressiveness of the model.

Each node in the LPG is assigned a label (e.g., "User", "Product", "Feature") and associated properties (such as user ID, product category, TF-IDF score, or purchase frequency), enabling a structured and attribute-rich representation of the dataset. This labeling schema helps in differentiating node types during the learning process and allows the model to leverage domain semantics effectively. The construction of the LPG thus provides a comprehensive framework for encoding and navigating the underlying knowledge within the data.

Although the LPG is primarily generated from the training dataset, both the training and test datasets contribute to the definition of labels and properties, ensuring consistency across the entire evaluation pipeline. This alignment is essential, as it enables the recommender system to generalize learned representations from the training phase and apply them effectively to unseen data during testing. Furthermore, the LPG plays a vital role in supporting explainability by maintaining interpretable relationships and features that can be traced back to individual recommendations. As a result, the LPG not only enhances model performance but also facilitates transparency and trust in the recommendation process, aligning with the goals of semantic modeling and XAI.

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

A.3.3 Knowledge Graph Embedding

The third phase of the experiment focuses on learning low-dimensional vector representations of entities and relationships within the KG using the TransE (Translating Embeddings) algorithm. TransE is a widely used KGE technique that models relationships by interpreting them as translations in the vector space. In this approach, for a given triplet ( h, r, t ) -where h is the head entity, r is the relation, and t is the tail entity-TransE seeks to learn embeddings such that the vector representation of the head entity plus the relation vector approximates the tail entity vector, i.e., h + r ≈ t .

To manage computational efficiency and scalability, embedding vectors are generated in mini-batches. For the Beauty, and Cell Phones domains, a batch size of 32 is selected to balance memory consumption and convergence speed. This batching strategy ensures that the learning process remains efficient across datasets of varying scales.

The embedding process is carried out over 30 epochs, during which the model iteratively updates the representations of entities and relationships based on observed interactions and structural dependencies in the graph. During each epoch, the model adjusts its internal parameters by minimizing a smooth margin-based ranking loss function, which encourages correct triplets to have lower distances between embeddings than negative triplets. This loss function plays a key role in shaping meaningful geometric relationships in the embedding space, capturing semantic similarities and structural patterns.

By the end of this phase, the embedding model yields a set of well-trained vectors that encode the semantic proximity and relational dynamics among entities in the KG. These learned embeddings are essential for downstream tasks, such as making recommendations and generating explanations, as they provide a compact, information-rich representation of the complex interconnections within the dataset. This phase lays the groundwork for integrating semantic understanding into the recommender system and supports the broader goal of enhancing explainability in AI-driven recommendations.

A.3.4 Reinforcement Learning Model

In this phase, the focus shifts to analyzing users' historical purchase behaviors and leveraging this information to train a RL agent capable of making personalized product recommendations. The model is designed to explore the KG by simulating a sequence of decisions that reflect a user's potential navigation paths through the graph. Each path represents a trajectory from a user node to a product node, capturing the semantic associations and behavioral patterns that inform recommendation outcomes.

To maintain computational tractability and ensure semantic relevance, the exploration of the KG is constrained to a maximum of three hops. Each hop allows the agent to choose among several candidate actions-such as transitioning to a related product, feature, or category node-based on the current node and its neighboring relationships. These actions determine the agent's trajectory and influence the subsequent choices available in future hops. Importantly, even if the agent ends its traversal at a product node, there is no guarantee that the node corresponds to a product previously seen in the training or test set for that user, introducing a realistic challenge of exploring novel but relevant items.

The RL agent operates in a step-wise learning framework, where it continuously updates its policy and parameter weights at each decision point rather than deferring learning until the end of the episode. This temporal difference learning approach allows the agent to react dynamically to immediate feedback and refine its decision-making strategy over time. The agent receives positive rewards for successfully navigating paths that lead to target-labeled products-i.e., products aligned with the user's known interests or preferences. These rewards signal successful behavior, guiding the agent to prioritize similar paths in the future.

The RL algorithm is carefully designed to incentivize the discovery and exploitation of user behavior patterns. By observing and internalizing how users interact with different product attributes and categories, the agent builds a behavioral model that encapsulates user preferences. This model becomes highly valuable during the recommendation phase, as it enables the agent to proactively identify products that are not only semantically related to the user's historical interests but are also behaviorally consistent with their past choices. Over time, the RL agent learns to balance exploration (discovering new products) and exploitation (recommending known preferences), ultimately enhancing the accuracy, relevance, and explainability of the RS.

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

A.3.5 Experimental Parameter Settings and Evaluation Strategy

The experimental setup adopted a well-defined set of default parameter settings to establish a consistent and reproducible baseline for evaluating the model's performance across multiple datasets. These parameters were carefully selected based on empirical observations and prior research to balance computational efficiency with model accuracy and explainability. To generate latent representations , the models employed knowledge graph embeddings trained using a 1-hop scoring function , enabling the system to capture meaningful local semantics between entities and relationships. Each embedding vector had a dimensionality of 100, providing sufficient capacity to represent nuanced interactions within the knowledge graph. The embedding training process used a learning rate of 0.5 and was capped at a maximum of 30 epochs to prevent overfitting while ensuring convergence.

The Actor-Critic reinforcement learning (RL) model was optimized for interpretability and path efficiency, particularly by constraining the maximum path length to three hops . This design choice aimed to generate concise and intuitive explanation paths that end users can easily comprehend. To manage computational complexity and maintain recommendation quality, the action space was pruned to a maximum of 250 actions per state. An action dropout rate of 0.5 was applied during training to encourage the agent to explore diverse paths, thereby enhancing both learning generalization and the diversity of explanations.

Key RL parameters included a discount factor γ = 0 . 99 , reflecting a high degree of importance placed on future rewards, which is crucial for long-term planning in sequential decision-making tasks. The policy and value networks were defined with the following weight matrix configurations:

W 1 ∈ R 400 × 512 : Input to first hidden layer

W 2 ∈ R 512 × 256 : First hidden to second hidden layer

W P ∈ R 256 × 250 : Policy head (action probability)

W V ∈ R 256 × 1 : Value head (expected return)

Training procedures were dataset-specific. For the CDs & Vinyl dataset, the model was trained for 5 epochs using the Adam optimizer with a learning rate of 0.001 and a batch size of 64. For other datasets (e.g., Clothing, Beauty), the model was trained for 100 epochs with a lower learning rate of 0.0001 and a batch size of 32 to accommodate differences in dataset scale and convergence behavior. The entropy loss weight was set to 0.001, promoting exploration by penalizing overly confident predictions.

To ensure continuity and reproducibility , both the embedding and RL models were configured to resume training from the most recent checkpoint, enabling seamless recovery and consistent results.

During the evaluation and product recommendation phase , the Actor-Critic model was evaluated across all users in each dataset using default parameter settings. For three-hop sampling, the default parameters were:

where K i denotes the number of candidate actions at hop i .

The evaluation phase also incorporated the Maximum Explainability Score with Confidence Interval (MES-CI) module to further refine the quality of recommendations. MES-CI facilitated the selection of semantic paths that optimized not only the relevance of recommendations but also their interpretability and user trust, thereby aligning the system with the broader goals of explainable and user-centric AI.

A.4 Recommendation

The final phase of the experiment is dedicated to evaluating the explainability of the proposed RS. This phase is critical as it not only measures the accuracy and effectiveness of the model's predictions but also ensures that the recommendations are interpretable and aligned with the principles of XAI. The evaluation is conducted in two main steps: the Prediction Step and the Recommendation Step.

Prediction step : In this step, the model estimates a set of key parameters for each candidate product that could potentially be recommended to a user. These parameters are derived from the semantic paths explored in the KG and provide insights into both the system's reasoning and the user's behavior. The four predicted parameters are:

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

-Likelihood of Traversal (P) : This refers to the probability that a user will traverse a specific semantic path in the KG. It reflects the system's confidence that the user would follow a given sequence of relationships and entities based on historical behavior.

-Product Affinity Score (S) : This score represents the predicted degree of interest or preference a user has for a particular product. It is derived from the user's previous interactions, preferences for features, and relationships encoded in the graph.

Cumulative Reward (Rw) : This metric captures the total reward the RL agent expects to gain by following a specific path that ends in a candidate product. It quantifies how well a given path aligns with the user's known preferences, as learned during training.

-Path Entropy (H) : Entropy reflects the level of uncertainty associated with a given path. A high entropy value suggests a lack of clarity or confidence in the path's outcome, while a low entropy indicates a more deterministic and explainable path. This helps in filtering out ambiguous or less reliable recommendation paths.

Recommendation step : Once the prediction parameters are obtained, the model proceeds to the Recommendation Step, where it utilizes these parameters to determine the optimal explainability path for each user-product pair. This is accomplished using the Maximum Explainability Score with Confidence Interval (MES-CI) algorithm. The MES-CI algorithm balances three competing objectives-accuracy, confidence, and informativeness-to select paths that are not only effective in predicting user preferences but also transparent and easy to interpret.

After identifying the optimal semantic path, the system recommends the associated product(s) to the user. Importantly, the selected path also serves as the explanation for why the product is being recommended. For instance, the system might explain that a product is recommended because it shares features with items the user previously purchased, comes from a favored brand, or is frequently bought together with past purchases.

By incorporating explainability into both the prediction and recommendation processes, the RS ensures that its outputs are not only personalized and accurate, but also transparent and trustworthy. This dual emphasis enhances user satisfaction and trust, and aligns the system with broader XAI goals, promoting adoption in real-world applications where understanding the rationale behind AI decisions is critical.

A.5 Execution Steps:

Max Explainability Score with Confidence Interval (MES-CI): A Quantitative Metric for Interpretability in Knowledge Graph-Based Recommender Systems

Datasets Datasets used in this paper is available at the provided link

The execution time is listed in the Table 8.

Requirements

Python >= 3.12

PyTorch> = 2.5

How to run the code

Proprocess the data first:

''bash python preprocess.py -dataset <dataset_name>

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

''

"<dataset_name>" should be one of "beauty", "cloth" (refer to utils.py).

2. Train knowledge graph embeddings (TransE in this case):

''bash python train_transe_model.py -dataset <dataset_name>

''

3. Train RL agent:

''bash python train_RL_agent.py -dataset <dataset_name>

''

4a. Evaluation - MESCI - Example call ''bash python test_RL_agent.py -dataset beauty -run_path True -run_eval True -users 21001 -debug True

''

If "run_path" is True, the program will generate paths for recommendation according to the trained policy.

If "run_eval" is True, the program will evaluate the recommendation performance based on the resulting paths.

''bash python test_RL_agent.py -dataset <dataset_name> -run_path True -run_eval True -users <user_id> -debug True

''

If "run_path" is True, the program will generate paths for recommendation according to the trained policy.

If "run_eval" is True, the program will evaluate the recommendation performance based on the resulting paths.

4c. Performance Evaluation ''bash python test_RL_agent.py -dataset <dataset_name> -run_path True -run_eval True ''

If "run_path" is True, the program will generate paths for recommendation according to the trained policy.

If "run_eval" is True, the program will evaluate the recommendation performance based on the resulting paths.

A.6 Logs-Amazon Beauty

A.6.1 Training - Embedding Model

[INFO] Namespace(dataset='beauty', name='train_transe_model', seed=123, gpu='1', epochs=30, batch_size=16, lr=0.5, weight_decay=0, l2_lambda=0, max_grad_norm=5.0, em-bed_size=100, num_neg_samples=5, steps_per_checkpoint=100000, checkpoint_folder='checkpoint', log_folder='log', log_file_name='train_log.txt', is_resume_from_checkpoint=0, logging_mode='a', device=device(type='cuda', index=0), dir='./tmp/Amazon_Beauty/train_transe_model', checkpoint_dir='./tmp/Amazon_Beauty/train_transe_model/checkpoint', log_dir='./tmp/Amazon_Beauty/train_transe_model/log')

[INFO] Parameters:['purchase', 'mentions', 'describe_as', 'produced_by', 'belongs_to', 'also_bought', 'also_viewed', 'bought_together', 'user.weight', 'product.weight', 'word.weight', 'related_product.weight', 'brand.weight', 'category.weight', 'purchase_bias.weight', 'men-tions_bias.weight', 'describe_as_bias.weight', 'produced_by_bias.weight', 'belongs_to_bias.weight', 'also_bought_bias.weight', 'also_viewed_bias.weight', 'bought_together_bias.weight']

[INFO] Epoch: 01 | Words: 4128430/405897991 | Lr: 0.49491 | Smooth loss: 14.59327

[INFO] Epoch: 01 | Words: 8258715/405897991 | Lr: 0.48983 | Smooth loss: 12.55730

[INFO] Epoch: 01 | Words: 12396929/405897991 | Lr: 0.48473 | Smooth loss: 12.04881

[INFO] Epoch: 02 | Words: 16532862/405897991 | Lr: 0.47963 | Smooth loss: 11.54240 1097 [INFO] Epoch: 02 | Words: 20672014/405897991 | Lr: 0.47454 | Smooth loss: 11.39307 1098 [INFO] Epoch: 02 | Words: 24796721/405897991 | Lr: 0.46945 | Smooth loss: 11.35870 1099 [INFO] Epoch: 03 | Words: 28926876/405897991 | Lr: 0.46437 | Smooth loss: 11.08418 1100 [INFO] Epoch: 03 | Words: 33060941/405897991 | Lr: 0.45927 | Smooth loss: 10.84952 1101 [INFO] Epoch: 03 | Words: 37189105/405897991 | Lr: 0.45419 | Smooth loss: 10.87816 1102 [INFO] Epoch: 04 | Words: 41320252/405897991 | Lr: 0.44910 | Smooth loss: 10.80380 1103 [INFO] Epoch: 04 | Words: 45447898/405897991 | Lr: 0.44402 | Smooth loss: 10.48704 1104 [INFO] Epoch: 04 | Words: 49586669/405897991 | Lr: 0.43892 | Smooth loss: 10.52798 1105 [INFO] Epoch: 04 | Words: 53717858/405897991 | Lr: 0.43383 | Smooth loss: 10.51342 1106 [INFO] Epoch: 05 | Words: 57854228/405897991 | Lr: 0.42873 | Smooth loss: 10.20127 1107 [INFO] Epoch: 05 | Words: 61980330/405897991 | Lr: 0.42365 | Smooth loss: 10.18586 1108 [INFO] Epoch: 05 | Words: 66114677/405897991 | Lr: 0.41856 | Smooth loss: 10.21038 1109 [INFO] Epoch: 06 | Words: 70248616/405897991 | Lr: 0.41347 | Smooth loss: 9.99730 1110 [INFO] Epoch: 06 | Words: 74383499/405897991 | Lr: 0.40837 | Smooth loss: 9.92230 1111 [INFO] Epoch: 06 | Words: 78516640/405897991 | Lr: 0.40328 | Smooth loss: 9.92832 1112 [INFO] Epoch: 07 | Words: 82652426/405897991 | Lr: 0.39819 | Smooth loss: 9.79496 1113 [INFO] Epoch: 07 | Words: 86786576/405897991 | Lr: 0.39309 | Smooth loss: 9.63500 1114 [INFO] Epoch: 07 | Words: 90915476/405897991 | Lr: 0.38801 | Smooth loss: 9.66572 1115 [INFO] Epoch: 08 | Words: 95050419/405897991 | Lr: 0.38291 | Smooth loss: 9.63382 1116 [INFO] Epoch: 08 | Words: 99180036/405897991 | Lr: 0.37783 | Smooth loss: 9.37495 1117 [INFO] Epoch: 08 | Words: 103312880/405897991 | Lr: 0.37274 | Smooth loss: 9.38763 1118 [INFO] Epoch: 08 | Words: 107450397/405897991 | Lr: 0.36764 | Smooth loss: 9.39443 1119 [INFO] Epoch: 09 | Words: 111579451/405897991 | Lr: 0.36255 | Smooth loss: 9.17835 1120 [INFO] Epoch: 09 | Words: 115711107/405897991 | Lr: 0.35746 | Smooth loss: 9.18696 1121 [INFO] Epoch: 09 | Words: 119848199/405897991 | Lr: 0.35237 | Smooth loss: 9.17020 1122 [INFO] Epoch: 10 | Words: 123980140/405897991 | Lr: 0.34728 | Smooth loss: 9.02826 1123 [INFO] Epoch: 10 | Words: 128115148/405897991 | Lr: 0.34218 | Smooth loss: 8.93950 1124 [INFO] Epoch: 10 | Words: 132244427/405897991 | Lr: 0.33710 | Smooth loss: 8.91189 1125 [INFO] Epoch: 11 | Words: 136375962/405897991 | Lr: 0.33201 | Smooth loss: 8.86128 1126 [INFO] Epoch: 11 | Words: 140511594/405897991 | Lr: 0.32691 | Smooth loss: 8.70399 1127 [INFO] Epoch: 11 | Words: 144644269/405897991 | Lr: 0.32182 | Smooth loss: 8.71882 1128 [INFO] Epoch: 11 | Words: 148774311/405897991 | Lr: 0.31673 | Smooth loss: 8.70265 1129 [INFO] Epoch: 12 | Words: 152907720/405897991 | Lr: 0.31164 | Smooth loss: 8.47386 1130 [INFO] Epoch: 12 | Words: 157039204/405897991 | Lr: 0.30655 | Smooth loss: 8.52233 1131 [INFO] Epoch: 12 | Words: 161168616/405897991 | Lr: 0.30147 | Smooth loss: 8.50717 1132 [INFO] Epoch: 13 | Words: 165296435/405897991 | Lr: 0.29638 | Smooth loss: 8.34437 1133 [INFO] Epoch: 13 | Words: 169429053/405897991 | Lr: 0.29129 | Smooth loss: 8.30853 1134 [INFO] Epoch: 13 | Words: 173561225/405897991 | Lr: 0.28620 | Smooth loss: 8.29629 1135 [INFO] Epoch: 14 | Words: 177694465/405897991 | Lr: 0.28111 | Smooth loss: 8.19751 1136 [INFO] Epoch: 14 | Words: 181820934/405897991 | Lr: 0.27603 | Smooth loss: 8.10413 1137 [INFO] Epoch: 14 | Words: 185954101/405897991 | Lr: 0.27093 | Smooth loss: 8.11419 1138 [INFO] Epoch: 15 | Words: 190086525/405897991 | Lr: 0.26584 | Smooth loss: 8.05563 1139 [INFO] Epoch: 15 | Words: 194219304/405897991 | Lr: 0.26075 | Smooth loss: 7.90554 1140 [INFO] Epoch: 15 | Words: 198349234/405897991 | Lr: 0.25567 | Smooth loss: 7.91409 1141 [INFO] Epoch: 15 | Words: 202485461/405897991 | Lr: 0.25057 | Smooth loss: 7.92219 1142 [INFO] Epoch: 16 | Words: 206619102/405897991 | Lr: 0.24548 | Smooth loss: 7.73590 1143 [INFO] Epoch: 16 | Words: 210751866/405897991 | Lr: 0.24039 | Smooth loss: 7.74630 1144 [INFO] Epoch: 16 | Words: 214884114/405897991 | Lr: 0.23530 | Smooth loss: 7.74051 1145 [INFO] Epoch: 17 | Words: 219023607/405897991 | Lr: 0.23020 | Smooth loss: 7.62290 1146 [INFO] Epoch: 17 | Words: 223159476/405897991 | Lr: 0.22510 | Smooth loss: 7.56469 1147 [INFO] Epoch: 17 | Words: 227291204/405897991 | Lr: 0.22001 | Smooth loss: 7.56370 1148 [INFO] Epoch: 18 | Words: 231420203/405897991 | Lr: 0.21493 | Smooth loss: 7.48413 1149 [INFO] Epoch: 18 | Words: 235552208/405897991 | Lr: 0.20984 | Smooth loss: 7.40282 1150 [INFO] Epoch: 18 | Words: 239684246/405897991 | Lr: 0.20475 | Smooth loss: 7.39227 1151 [INFO] Epoch: 19 | Words: 243814072/405897991 | Lr: 0.19966 | Smooth loss: 7.36665 1152 [INFO] Epoch: 19 | Words: 247936836/405897991 | Lr: 0.19458 | Smooth loss: 7.23048 1153 [INFO] Epoch: 19 | Words: 252069613/405897991 | Lr: 0.18949 | Smooth loss: 7.23733 1154 [INFO] Epoch: 19 | Words: 256208565/405897991 | Lr: 0.18439 | Smooth loss: 7.22914 1155

1192

1193

1194

1195

1196

1197

1198

1199

1200

1201

1202

1203

1204

1205

1206

1207

1208

1209

1210

1211

1212

1213

A.6.2 Training - RL Agentl

[INFO] Namespace(dataset='beauty', name='train_RL_agent', seed=123, gpu='1', epochs=100, batch_size=32, lr=0.0001, max_acts=250, max_path_len=3, gamma=0.99, ent_weight=0.001, act_dropout=0, state_history=1, hidden=[512, 256], debug=0, steps_per_checkpoint=50000, checkpoint_folder='checkpoint', log_folder='log', log_file_name='train_log.txt', is_resume_from_checkpoint=1, logging_mode='a', device=device(type='cuda', index=0), dir='./tmp/Amazon_Beauty/train_RL_agent', checkpoint_dir='./tmp/Amazon_Beauty/train_RL_agent/checkpoint', log_dir='./tmp/Amazon_Beauty/train_RL_agent/log')

[INFO] Parameters:['l1.weight', 'l1.bias', 'l2.weight', 'l2.bias', 'actor.weight', 'actor.bias', 'critic.weight', 'critic.bias']

[INFO] Save model to ./tmp/Amazon_Beauty/train_RL_agent/checkpoint/policy_model_epoch_1.ckpt [INFO] epoch/step=2/50000 | loss=14.78206 | ploss=14.78685 | vloss=9267978.91805 | entropy=4.79005 | reward=99.75579

[INFO] epoch/step=2/100000 | loss=-4.97828 | ploss=-4.97337 | vloss=6457813.60289 | entropy=4.91355 | reward=70.13095

[INFO] Save model to ./tmp/Amazon_Beauty/train_RL_agent/checkpoint/policy_model_epoch_2.ckpt [INFO] epoch/step=3/150000 | loss=-5.12558 | ploss=-5.12053 | vloss=5468618.27898 | entropy=5.05146 | reward=59.12274

[INFO] epoch/step=3/200000 | loss=-5.14505 | ploss=-5.13992 | vloss=4560310.82483 | entropy=5.12293 | reward=49.15593

[INFO] Save model to ./tmp/Amazon_Beauty/train_RL_agent/checkpoint/policy_model_epoch_3.ckpt

1787

1788

A.6.3 Test - RL Agent (MES-CI) - for a user

[INFO] model epoch=100 | count (users)=1 | ndcg=100.00000 | recall=100.00000 | hit_ratio=100.00000 1796

1797

| precision=10.00000 | invalid_users=0.00000 | execution_timestamp=2025-05-15 09:33:00.890819

train_labels[uid] : 21001 [11808, 7381, 9141, 10747] 1798

1799

1800

1801

test_labels[uid] : 21001 [11772]

pred_labels[uid] : 21001 [11772, 8471, 9576, 5351, 1690, 5603, 8015, 1465, 10872, 5934]

i: j : 21001 0

pred_labels[uid] : 21001 11772 1802

1803

1804

pred_labels_path[uid] : 21001 (11772, 'user 21001 has purchase product 10747 which was produced_by by brand 201 who produced_by product 11772')

1805

1806

1807

1808

1809

1810

1811

1812

1813

1814

1815

1816

1817

1818

1819

1820

1821

1822

1823

1824

1825

1826

1827

1828

1829

1830

1831

1832

1833

1834

1835

1836

1837

1838

1839

1840

1841

pred_labels_details[uid] : 21001 (np.float32(3.6954694), np.float32(1.0), np.float32(1.0986123), np.float32(5693.659), [('self_loop', 'user', 21001), ('purchase', 'product', 10747), ('pro-duced_by', 'brand', 201), ('produced_by', 'product', 11772)], np.float32(8.231993), np.float32(-0.4814049), np.float32(0.020631433), np.float32(4231.7305), np.float32(1.8997045), np.float64(1.6209097319169405), np.float64(2.178499180741751), 6)

pred_probs=1.0 | pred_entropy=1.0986123085021973 | pred_reward=5693.6591796875 | | pred_path=[('self_loop', 'user', 21001), ('purchase', 'product', 10747), ('pro-duced_by', 'brand', 201), ('produced_by', 'product', 11772)] | path_prob_diff_user_mean=-0.4814049005508423 | path_entropy_diff_user_mean=0.02063143253326416 | path_rewards_diff_user_mean=4231.73046875

explainability_score: 1.8997045

explainability_score: CI - Lower 1.6209097319169405

explainability_score: CI - Upper 2.178499180741751 Count: 6

[ (11772, 'user 21001 has purchase product 10747 which was purchase by user 9748 who purchase product 11772', np.float32(0.8712647), np.float32(15.429942), (np.float32(3.6954694), np.float32(2.0), np.float32(1.0397208), np.float32(3274.7021),

[('self_loop', 'user', 21001), ('purchase', 'product', 10747), ('purchase', 'user', 9748), ('purchase', 'product', 11772)], np.float32(8.231993), np.float32(0.5185951), np.float32(-0.038260102), np.float32(1812.7737), np.float32(0.8712647), np.float64(0.5924699717088107), np.float64(1.150059420533621), 6)),

(11772, 'user 21001 has purchase product 10747 which was also_viewed by related_product 23132 who also_viewed product 11772', np.float32(1.4115256), np.float32(18.995838), (np.float32(3.6954694), np.float32(1.0), np.float32(1.0986123), np.float32(4418.364),

[('self_loop', 'user', 21001), ('purchase', 'product', 10747), ('also_viewed', 'related_product', 23132), ('also_viewed', 'product', 11772)], np.float32(8.231993), np.float32(-0.4814049), np.float32(0.020631433), np.float32(2956.4353), np.float32(1.4115256), np.float64(1.1327308826966647), np.float64(1.690320331521475), 6)),

(11772, 'user 21001 has purchase product 10747 which was produced_by by brand 201 who produced_by product 11772' , np.float32(1.8997045), np.float32(20.948555), (np.float32(3.6954694), np.float32(1.0), np.float32(1.0986123), np.float32(5693.659),

[('self_loop', 'user', 21001), ('purchase', 'product', 10747), ('produced_by', 'brand', 201), ('pro-duced_by', 'product', 11772)], np.float32(8.231993), np.float32(-0.4814049), np.float32(0.020631433), np.float32(4231.7305),

np.float32(1.8997045), np.float64(1.6209097319169405), np.float64(2.178499180741751) , 6)),

(11772, 'user 21001 has purchase product 11808 which was also_viewed by related_product 23062 1842 who bought_together product 11772', 1843

np.float32(1.720593), np.float32(20.232107), (np.float32(3.6954694), np.float32(1.0), 1844 np.float32(1.0986123), np.float32(5225.7563), 1845

1846

1847

[('self_loop', 'user', 21001), ('purchase', 'product', 11808), ('also_viewed', 'related_product', 23062), ('bought_together', 'product', 11772)], np.float32(8.231993), np.float32(-0.4814049), np.float32(0.020631433), np.float32(3763.828), 1848 np.float32(1.720593), np.float64(1.44179825120405), np.float64(1.9993877000288602), 6)), 1849

(11772, 'user 21001 has purchase product 11808 which was also_viewed by related_product 23062 1850 who also_bought product 11772', 1851

np.float32(1.720593), np.float32(20.232107), (np.float32(3.6954694), np.float32(1.0), 1852 np.float32(1.0986123), np.float32(5225.7563), 1853

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new metric—Max Explainability Score with Confidence Interval (MES-CI)—to quantitatively evaluate explainability in knowledge graph-based recommender systems (KGRS). Building on the existing Max Explainability Score (MES), MES-CI introduces a statistical confidence interval around the score, addressing the limitation of single-point evaluations. The new formulation considers both reward gain (from path traversal in the knowledge graph) and information gain (entropy), and complements this with interval estimates to better quantify reliability in explanations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper meaningfully extends the MES metric by integrating confidence intervals, which is a significant step toward statistically sound explainability. This addition transforms explainability from a static, deterministic score into a distribution-aware assessment. The proposed algorithm (Algorithm 1) formalizes the procedure clearly, including computation of standard deviation, confidence bounds, and path ranking.\n2. The algorithm is fully specified, and the experimental setup—including dataset construction, RL modeling using Actor-Critic with TransE embeddings, and case study walkthrough—is described in enough detail to support reproducibility.\n\nWeaknesses\n1. While the metric enhancement is meaningful, the experiments are largely adaptations of prior MES-based setups. The work does not evaluate how MES-CI changes recommendation behavior or how users might interact differently with CI-enhanced explanations.\n2. The proposed MES-CI is intended to generalize across RS models, but it is only tested in a very specific RL-based KGRS pipeline. A more persuasive argument would include evaluations across multiple XRS architectures (e.g., path-based, attention-based, and post-hoc explainers) to show general utility.\n3. The paper promotes the benefit of “user trust” via MES-CI, yet no empirical user feedback or proxy user interaction metric (e.g., dwell time, click-through) is presented. The link between CI-enhanced explanations and actual human trust is assumed, not demonstrated.\n\nQUESTIONS:\n1. How sensitive is MES-CI to the number of available paths per recommendation? Some products may have only one or two valid explanation paths—can MES-CI be reliably computed in such cases?\n\n2. Have you validated the normality assumption underlying the confidence interval computation? Are path-level explainability scores approximately normal? If not, would non-parametric CI estimation be more appropriate?\n\nLIMITATIONS:\nThe authors have explicitly addressed the primary limitations of MES-CI, including: 1) Reliance on statistical assumptions (e.g., normality, sufficient sample size), 2) Inability to generate CI when only a single path is available, 3) Sensitivity to variance and skewness in score distributions.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research introduced MES-CI, a metric for evaluation on explanations on recommender systems. MES-CI is an extension of the existing MES metric, and it innovates by adding confidence intervals to improve the reliability of the values provided for the explanations generated by a Knowledge Graph-Based Recommender System.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The problem of providing explanations to personalized recommendations is well within the scope of the conference topics.\n2. The paper is somehow clear, with a structure easy to follow. The appendices help deepen the understanding of the implementation and allow the experiments to be replicated. For readers who are not very specialized in explainable recommendation systems, it provides a good introduction.\n\n\n**Weaknesses**\n1. The paper contains some redundancy, particularly in the repeated discussion of standard recommendation metrics and the use of confidence intervals for MES. A more concise presentation would improve clarity.\n\n2. It is not clear how much MES-CI contributes newer concepts compared to the original MES metric. After a quick review of MES, MES-CI seems incremental and not very innovative compared to MES-CI. Furthermore, the metric’s effectiveness may be sensitive to the number of explanations generated, potentially limiting its applicability or significance in certain scenarios\n\n3. The research misses comparisons with two important baselines:  PRINCE (Ghazimatin et al., 2020) and CredPaths (Yang et al., 2018)\n* Ghazimatin, A., Balalau, O., Saha Roy, R., & Weikum, G. (2020). Prince: Provider-side interpretability with counterfactual explanations in recommender systems. In Proceedings of the 13th International Conference on Web Search and Data Mining (pp. 196-204).\n* Yang, F., Liu, N., Wang, S., & Hu, X. (2018). Towards interpretation of recommender systems with sorted explanation paths. In 2018 IEEE International Conference on Data Mining (ICDM) (pp. 667-676). IEEE.\n\n4. The evaluation of the proposed metric could be strengthened; it only compares the values of the generated intervals (MES-CI) with MES values. In the evaluation, the generated explanations are left in numerical form, which prevents a real comparison of how good the explanations are relative to each other. Additionally, the authors claim that the confidence intervals make recommendations more trustworthy for users, but this is not tested with real users. Notice that PRINCE (Ghazimatin et al., 2020) performed both a qualitative evaluation but also a study on Amazon Mechanical Turk (AMT) to gain insights directly from users' perceptions of the explanation. I would recommend the authors of MES-CI to conduct a similar evaluation to provide stronger evidence over the claims on MES-CI.\n\n5. In the theoretical framework for quantitative evaluation in explainability, only one previous work (from 2021) is mentioned in addition to MES, which suggests that more research on these methodologies might be lacking. Again, the authors miss a fair comparison with two well known methods for explaining recommendations based on knowledge graphs: PRINCE and CredPaths.\n\nQUESTIONS:\n1)\tIn your case study using the Beauty dataset for user 21001, the explanations are presented using only product and brand IDs, without mapping them to actual product names or interpretable attributes. At the same time, you claim that \"The inclusion of confidence interval values strengthens user trust by providing a clearer assessment of MES and the corresponding explainability of recommendations.\" Given that the explanations rely on numerical IDs, which are not directly meaningful to end users, could you elaborate on how these would concretely contribute to enhancing user trust in a real-world scenario? Have you considered user studies, human evaluations, or example renderings with product names or categories to assess how users would perceive and interpret these explanations?\n2)\tYou explicitly state that the focus of the paper is not on recommendation efficacy but solely on the explainability of recommended items. However, Table 3 presents a case study where the recommended items vary in their MES scores—even including items with lower MES than others not recommended. Given that explainability is inherently tied to the recommendation itself (i.e., users care about explanations for why a certain item was recommended over others), could you clarify how your method balances recommendation quality with explainability? Did you explore whether lower-MES items might be less useful to users despite being part of the top-N recommended list?\n3) Could you provide a clear and thurough account of similarities and differences with the methods PRINCE and CredPaths?\nGhazimatin, A., Balalau, O., Saha Roy, R., & Weikum, G. (2020). Prince: Provider-side interpretability with counterfactual explanations in recommender systems. In Proceedings of the 13th International Conference on Web Search and Data Mining (pp. 196-204).\nYang, F., Liu, N., Wang, S., & Hu, X. (2018). Towards interpretation of recommender systems with sorted explanation paths. In 2018 IEEE International Conference on Data Mining (ICDM) (pp. 667-676). IEEE.\n4) Can you run an evaluation comparing with both PRINCE and CredPaths?\n\nLIMITATIONS:\nSomehow, they describe the technical limitations of the approach introduced (MES-CI), but the authors do not explain potential societal impacts. For instance, they do not explain that providing biased explanations of recommendations could enhance the filter bubble or echo chamber effects of recommender systems in the news domain, which can have a strong societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new metric, the Max Explainability Score with Confidence Interval (MES-CI), for evaluating the interpretability of recommendations from Knowledge Graph-based Recommender Systems. The work is an extension of a previously proposed metric, the Max Explainability Score (MES), which provides a single-point score for an explanation. The authors argue that a single score is insufficient to capture the reliability of an explanation. The core contribution is the incorporation of a confidence interval around the MES score, which is calculated based on the standard deviation of explainability scores from all possible explanation paths for a given recommendation. The paper provides an algorithm for MES-CI and demonstrates its use through a case study on an e-commerce dataset, arguing that MES-CI offers a more robust and nuanced assessment of explainability, thereby enhancing user trust.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper tackles a significant problem in explainable AI. As recommender systems become more complex, developing robust, quantitative metrics to evaluate their explainability is crucial for transparency, accountability, and user trust. \n2. The paper is well-structured and easy to follow. The inclusion of a detailed case study helps to operationalize the metric and makes the concept of multiple explanation paths tangible for the reader. \n3.  The idea of assessing the reliability of an explanation score is a good one. A single-point metric can indeed be misleading, and the authors' proposal to add a measure of statistical confidence is an intuitive step towards a more comprehensive evaluation framework. \n\nWeaknesses\n1. The primary weakness is the incremental nature of the contribution. The core proposal is to apply a standard statistical tool—the confidence interval—to an existing, recently published metric (MES). The paper does not propose a new way to measure explainability itself but rather a method to analyze the output of an existing metric. \n2. The empirical evidence supporting the utility of MES-CI is thin. The evaluation consists of a single case study for one user from the Beauty dataset. While illustrative, this is insufficient to demonstrate the general applicability or superiority of the metric. There are no large-scale experiments, comparisons with other explainability evaluation methods (even qualitative ones), or user studies to show that MES-CI aligns with human judgments of explanation quality or reliability. The paper explicitly limits its scope to the explainability metric itself, not the recommendation model's performance, but the evaluation of the metric itself is not extensive.\n\nQUESTIONS:\n1. Could you provide a more rigorous justification for using the variance of scores from all possible paths to define the confidence in the single maximum-scoring path? An alternative view is that the properties of the best path (e.g., its score relative to the next-best path) might be more indicative of confidence than the overall variance. How does your proposed CI distinguish between a scenario with one great explanation and many poor ones, versus a scenario with several good, similarly-scored explanations?\n2.  In the case study (Table 3), several recommendations have no MES-CI value because only one explanation path was found. How frequently does this occur in your experiments across the full datasets?\n\nLIMITATIONS:\nYes, the authors have included a dedicated \"Limitation\" section (5.3). They correctly identify that the metric relies on the statistical distribution of scores, which may not always be appropriate, and that it is not applicable when only a single explanation path exists. They also note its sensitivity to the number of explanations (sample size).  This is a fair and transparent assessment of the immediate limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}