SlowGuess/NAACL_2021_OCR huggingface dataset:
Yaxin Zhu; Yikun Xian; Zuohui Fu; Gerard de Melo, Yongfeng Zhang  
Rutgers University, New Brunswick
{yaxin.a.zhu,yikun.xian,zuohui.fu}@rutgers.edu {gerard.demelo,yongfeng.zhang}@rutgers.edu
Knowledge graphs (KG) have become increasingly important to endow modern recommender systems with the ability to generate traceable reasoning paths to explain the recommendation process. However, prior research rarely considers the faithfulness of the derived explanations to justify the decision-making process. To the best of our knowledge, this is the first work that models and evaluates faithfully explainable recommendation under the framework of KG reasoning. Specifically, we propose neural logic reasoning for explainable recommendation (LOGER) by drawing on interpretable logical rules to guide the path-reasoning process for explanation generation. We experiment on three large-scale datasets in the e-commerce domain, demonstrating the effectiveness of our method in delivering high-quality recommendations as well as ascertaining the faithfulness of the derived explanation.
Compared with traditional recommender systems (RS), explainable recommendation is not only capable of providing high-quality recommendation results but also offers personalized and intuitive explanations (Zhang and Chen, 2020). Incorporating a knowledge graph (KG) into recommender systems has become increasingly popular, since KG reasoning is able to generate explainable paths connecting users to relevant target item entities. At the same time, there is increasing demand for systems to ascertain the faithfulness of the generated explanation, i.e., assess whether it faithfully reflects the reasoning process of the model and is consistent with the historic user behavior.
However, previous work has largely neglected faithfulness in KG-enhanced explainable recommendation (Xian et al., 2020a; Fu et al., 2020a). A number of studies (Lakkaraju et al., 2019; ter Hoove et al., 2018; Wu and Mooney, 2018) argue that
faithful explanations should also be personalized and gain the capability to reflect the personalized user historic behavior. However, to the best of our knowledge, none of the existing explainable recommendation models based on KGs have considered faithfulness in the explainable reasoning process and its evaluation on the generated explainable paths. For instance, PGPR (Xian et al., 2019; Zhao et al., 2020) infers explainable paths over the KG without considering personalized user behavior, and its prediction on next potential entities is merely based on the overall knowledge-driven rewards. CAFE (Xian et al., 2020b) builds user module profiles to guide the path inference procedure. However, as illustrated in Subramanian et al. (2020), such neural module networks only implicitly abstract the reasoning process and lack of considering the faithfulness of explanations.
In this paper, we propose a new KG-enhanced recommendation model called LOGER to produce faithfully explainable recommendation via neural logic reasoning. To fully account for heterogeneous information and rules about users and items from the KG, we leverage an interpretable neural logic model for logical reasoning, enhanced by a general graph encoder that learns KG representations to capture semantic aspects of entities and relations. These two components are iteratively trained via the EM algorithm by marrying the merits of interpretability of logical rules and the expressiveness of KG embeddings. Subsequently, the learned rule weights are leveraged to guide the path reasoning to generate faithful explanations. The derived logical rules are expected to be consistent with historic user behavior and the resulting paths genuinely reflect the decision making process in KG reasoning. We experiment on three large-scale datasets for e-commerce recommendation that cover rich user behavior patterns. The results demonstrate the superior recommendation performance achieved by our model compared to the state-of-the-art base
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3083-3090 June 6-11, 2021. ©2021 Association for Computational Linguistics
lines, with the guarantee of the faithfulness on the generated path-based explanations. The contributions of this paper are threefold.
- We highlight the significance of considering faithfulness in explainable recommendation.
- We propose a novel approach that incorporates interpretable logical rules into KG path reasoning for recommendation and explanation generation.
- We experiment on three large-scale datasets showing promising recommendation performance as well as faithful path-based explanation.
A knowledge graph (KG) for recommendation is defined as  $\mathcal{G} = \{(e_h,r,e_t)\mid e_h,e_t\in \mathcal{E},r\in \mathcal{R}\}$ , where  $\mathcal{E}$  denotes the entity set consisting of sets of users  $\mathcal{U}$ , items  $\mathcal{I}$ , and other entities, while  $\mathcal{R}$  denotes the relation set. Each triplet  $(e_h,r,e_t)$  represents a fact indicating head entity  $e_h$  interacts with tail entity  $e_t$  via relation  $r$ . In recommendation tasks, we are particularly interested in user-item interactions  $\{(u,r_{ui},v)\mid u\in \mathcal{U},r_{ui}\in \mathcal{R},v\in \mathcal{I}\}$  with the special relation  $r_{ui}$  meaning purchase in e-commerce or like in movie recommendation.
The problem of KG reasoning for explainable recommendation is formulated as follows. Given an incomplete KG  $\mathcal{G}$  with missing user-item interactions, for every user  $u\in \mathcal{U}$ , the goal is to select a set of items as recommendations  $\{v|(u,r_{ui},v)\notin \mathcal{G},v\in \mathcal{I}\}$  along with a set of paths as explanations connecting each pair of the user and a predicted item. The key challenge is to not only guarantee the recommendation quality with the rich information in KG, but also generate faithful explanations that reflect the actual decision-making process of the recommendation model and are consistent with historic user behavior.
We introduce the novel neural LOGic Explainable Recommender (LOGER) for producing faithfully explainable recommendations with a KG. As illustrated in Fig. 1, it consists of three components: (i) a KG encoder for learning embeddings of KG entities and relations to capture their semantics, (ii) a neural logic model for conducting interpretable logical reasoning to make recommendations, and (iii) a rule-guided path reasoner for generating faithfully explainable paths. Both KG encoder and neural logic model are trained iteratively via the EM algorithm (Neal and Hinton, 1998) so that they mutually
benefit to make recommendations via logical reasoning. Additionally, personalized rule importance scores are derived for every user and leveraged to guide the path reasoning for faithful explanation generation.
3.1 KG Encoder
Let  $X_{hrt}$  be a binary random variable indicating whether a triplet  $(e_h,r,e_t)$  is true or not,  $X_{\mathcal{G}} = \{X_{hrt} \mid (e_h,r,e_t) \in \mathcal{G}\}$  be a random variable regarding all observed triplets in the KG  $\mathcal{G}$ , and  $X_H = \{X_{hrt} \mid (e_h,r,e_t) \in H\}$  be a random variable of hidden user-item interactions in  $H = \{(u,r_{ui},v) \mid u \in \mathcal{U}, v \in \mathcal{I}, (u,r_{ui},v) \notin \mathcal{G}\}$ . The KG encoder is generally defined as a triplet-wise function  $f_\theta : \mathcal{E} \times \mathcal{R} \times \mathcal{E} \mapsto [0,1]$  parametrized by  $\theta$  that maps each triplet to a real-valued score. For any triplet  $(e_h,r,e_t) \in \mathcal{G} \cup H$ , we can interpret its truth probabilistically via the KG encoder  $f_\theta$  as  $q(X_{hrt}|\theta) = \text{Bernoulli}(X_{hrt}|f_\theta(e_h,r,e_t))$ . The KG encoder  $f_\theta$  can be instantiated with any existing KG embedding (Ji et al., 2020) or graph neural network (Wu et al., 2020) model.
3.2 Neural Logic Model
We focus on composition rules for user-item interactions, i.e.,  $r_{ui}$  is a composition of relations  $r_1, \ldots, r_j$  if  $(u, r_1, e_1) \wedge \dots \wedge (e_{j-1}, r_j, v) \Rightarrow (u, r_{ui}, v)$ ,  $\forall u \in \mathcal{U}, v \in \mathcal{V}, e_1, \ldots, e_{j-1} \in \mathcal{E}$ . Given a set of logical rules  $L$  mined from the KG, the goal of this component is, for every user  $u \in \mathcal{U}$ , to emit a set of personalized rule importance scores  $y_u = \{y_{u,l}\}_{l \in L}$  to capture the historic user behavior. To achieve this, we build upon Markov Logic Networks (Qu and Tang, 2019), an interpretable probabilistic logic reasoning method that models the joint distribution of all triplets via a set of logical rules  $L$ , i.e.,  $p(X_{\mathcal{G}}, X_H | w) = \frac{1}{Z} \exp \left( \sum_{l \in L} w_l n_l \right)$ , where  $w = \{w_l\}_{l \in L}$  with  $w_l$  being the global weight of rule  $l \in L$ , and  $n_l$  denotes the number of true groundings of rule  $l$  over observed and hidden triplets. Accordingly, we define the personalized rule importance score to be  $y_{u,l} = \frac{w_l n_l(u)}{\sum_{l' \in L} n_{l'}(u)}$ , where  $n_l(u)$  is the number of groundings of rule  $l$  over the observed triplets in  $\{(u, r_{ui}, v) \in \mathcal{G}\}$ . However, it is intractable to directly maximize the log likelihood of observed triplets to learn the global weights  $w$ , i.e.,  $\max_w \log p(X_{\mathcal{G}} | w)$ . Instead, we employ the EM algorithm to iteratively optimize the objective to acquire optimal global weights.
Figure 1: Illustration of the proposed method for explainable recommendation including (i) a KG encoder, (ii) a neural logic model, and (iii) a rule-guided path reasoner.
E-Step We introduce a mean-field variational distribution  $q(X_H|\theta)\approx \prod_{(e_h,r,e_t)\in H}q(X_{hrt}|\theta)$  over hidden user-item interactions in  $H$ . The goal of the E-step is to estimate  $q(X_H|\theta)$  by minimizing the KL divergence between  $q(X_H|\theta)$  and the posterior distribution  $p(X_H|X_{\mathcal{G}},w)$  with fixed  $w$ . For each triplet  $(e_h,r,e_t)\in H$ , we denote by  $L_{hrt}$  the set of rules associated with the triplet and by  $G_{hrt}$  the corresponding groundings of all logical rules in  $L_{hrt}$ . Following Qu and Tang (2019), the optimal  $q(X_H|\theta)$  can be achieved under the fixed-point condition, i.e.,  $q(X_{hrt}|\theta)\approx p(X_{hrt}|X_{G_{hrt}},w)$ , for all  $(e_h,r,e_t)\in H$ . Here,  $q(X_{hrt}|\theta)$  is approximated by the KG encoder  $f_{\theta}$ , and  $p(X_{hrt}|X_{G_{hrt}},w)$  can be estimated with the global weights  $w$  of the rules in  $L_{hrt}$  from the last iteration:
$$
p \left(X _ {h r t} = 1 \mid X _ {G _ {h r t}}, w\right) = \sigma \left(\frac {\sum_ {l \in L _ {h r t}} w _ {l}}{\left| L _ {h r t} \right|}\right), \tag {1}
$$
where  $\sigma (\cdot)$  is the sigmoid function. In other words, if a hidden triplet  $(e_h,r,e_t)$  is asserted to be true by the rules (e.g.,  $p(X_{hrt} = 1\mid X_{G_{hrt}},w) > 0.5)$ , the probability  $q(X_{hrt} = 1\mid \theta)$  given by the KG encoder is also expected to be high. Therefore, to learn the parameter  $\theta$ , we aim to maximize the log-likelihood function over all observed triplets in  $\mathcal{G}$  and the plausibly true hidden triplets in  $H^{+} = \{(e_{h},r,e_{t})\mid p(X_{hrt} = 1|X_{G_{hrt}},w)\geq \tau \}$ , which leads to the objective
$$
\ell (\theta) = \sum_ {\left(e _ {h}, r, e _ {t}\right) \in \mathcal {G} \cup H ^ {+}} \log q \left(X _ {h r t} = 1 \mid \theta\right), \tag {2}
$$
where  $\tau$  is a hyperparameter.
M-Step The goal of the M-step is to learn the global rule weights  $w$  by maximizing the log-likelihood function  $E_{q(X_H)}[\log p(X_G,X_H;w)]$  given a fixed  $\theta$  from the E-step. Since the log-likelihood term models the joint distribution over all triplets, which is hard to compute for a large KG, we approximate it with the pseudolikelihood (Besag, 1975):  $\ell_{PL}(w) = \sum_{(e_h,r,\epsilon_t)\in \mathcal{G}\cup H}E_q(X_H|\theta)[\log p(X_{hrt}|X_{G_{hrt}},w)]$ . Then, we can invoke gradient ascent to acquire the optimal  $w$ , with the gradient defined as:
$$
\begin{array}{l} \nabla_ {w _ {l}} \ell_ {P L} (w _ {l}) = \sum_ {\left(e _ {h}, r, e _ {t}\right) \in \mathcal {G}} \frac {1 - p _ {h r t}}{\left| L _ {h r t} \right|} + \tag {3} \\ \sum_ {(e _ {h}, r, e _ {t}) \in H} \frac {q (X _ {h r t} = 1 | \theta) - p _ {h r t}}{| L _ {h r t} |}, \\ \end{array}
$$
where  $p_{hrt} = p(X_{hrt} = 1|X_{G_{hrt}},w)$ . Once the optimal global weights are acquired, we can make a recommendation by calculating the ranking score of a user  $u\in \mathcal{U}$  and an item  $v\in \mathcal{I}$  as  $q(X_{urv}|\theta) + \alpha p(X_{urv} = 1|X_{G_{urv}},w)$ , where  $r = r_{ui}$  and  $\alpha \in \mathbb{R}$  is a hyperparameter.
3.3 Rule-Guided Path Reasoner
We draw on the KG encoder  $f_{\theta}$  and the personalized rule importance scores  $y_{u}$  from the last two steps to generate explainable paths for every user  $u$ . Specifically, we train an LSTM-based path reasoning network  $\phi$  that takes the start user embedding as input and predicts a sequence of entities and relations to form a path. For every user  $u$ , we restrict the reasoner to generate the paths that follow the rules with the largest scores in  $y_{u}$ . The details of  $\phi$
and path reasoning are described in the Appendix.
Dataset We experiment on three domain-specific e-commerce datasets from Amazon, namely Cellphones, Grocery, and Automotive. There are two requirements that lead to the selection of these categories in our experiments. First, the constructed KG should contain rich user behavior patterns, e.g., user mentioned features or preferred styles, etc. This is the major difference from most of the existing work (Zhao et al., 2019), which only extends knowledge on the item side. Second, the KGs are assumed to be large-scale. We select several large subsets from Fu et al. (2020b), where the constructed KG can be regarded as an updated version of those of Ai et al. (2019) based on the Amazon review dataset (Ni et al., 2019). The remaining three datasets are the ones that satisfy both of the aforementioned requirements. Statistical details of datasets are provided in the Appendix.
Baselines & Metrics We consider several state-of-art baselines in the following experiments. CKE (Zhang et al., 2016) uses semantic representations derived from TransR (Lin et al., 2015) to enhance the matrix factorization process. RipplesNet (Wang et al., 2018) is a hybrid method combining regularization and path formats, and augmenting user representations with a memory-network-like approach. PGPR (Xian et al., 2019) designed a policy-guided graph search algorithm for recommendation over KGs. HeteroEmbed (Ai et al., 2018) aims to learn the embeddings of a heterogeneous graph including users, items, and relations for recommendation. KGAT (Wang et al., 2019) explicitly models higher-order KG connectivity and learns node representations by propagating the embedding of neighbors with corresponding importance discriminated by an attention mechanism. We adopted the same metrics as Ai et al. (2018) to evaluate the recommendation performance of all models: Precision, Recall, Normalized Discounted Cumulative Gain (NDCG), and Hit Rate (HR).
4.1 Recommendation Results
We first evaluate the recommendation quality of our model. The results of all methods across all three datasets are reported in Table 1. In general, our method significantly outperforms all state-of-the-art baselines on all metrics. Taking Cellphones
as an example, our method achieves an improvement of  $6.01\%$  in NDCG against the best baseline (underlined), and an improvement of  $5.82\%$  in Hits@10. Similar trends can be observed on other benchmarks as well. Note that both our model and HeteroEmbed adopt TransE for KG representation learning, yet our model achieves better performance, mainly attributed to the iterative learning of graph encoder and neural logic model.
4.2 Faithfulness of Explanation
We aim to measure whether the generated explainable paths are consistent with the historic user behavior via a faithfulness metric and a user study.
Measuring Faithfulness Inspired by previous work (Maaten and Hinton, 2008; Serrano and Smith, 2019; Subramanian et al., 2020), we define the faithfulness to be the Jensen-Shannon (JS) divergence of rule-related distributions from training and test sets. Specifically, we randomly sample 50 users from the training set. For each user  $u$ , we further sample around 1,000 paths between the user and the connected item nodes, and calculate the rule distribution over these paths, denoted by  $F(u)$ . We compare the proposed LOGER with two baselines, PGPR, and KGAT, each of which is used to generate 20 explainable paths for every selected user in the test phase. Similarly, we can calculate the rule distribution over these 20 paths, denoted by  $Q_f(u)$ . The JS scores are defined as follows.
$$
\mathrm {J S} _ {f} = \mathbb {E} _ {u \sim \mathcal {U}} [ D _ {\mathrm {J S}} (Q _ {f} (u) \| F (u)) ]
$$
$$
\mathrm {J S} _ {w} = \mathbb {E} _ {u \sim \mathcal {U}} [ D _ {\mathrm {J S}} (Q _ {w} (u) \| F (u)) ]
$$
Here,  $Q_w(u)$  is the rule weight distribution derived from the personalized rule importance scores of our method or the path weights of baselines. Smaller values of two JS scores correspond to better faithfulness of the explainable paths. This faithfulness evaluation is motivated in terms of the consistency of the explainable paths with respect to the user historic behavior.
User Study Additionally, we conduct a user study to evaluate the faithfulness of the explainable paths. We display 50 sampled KG paths starting from one user towards purchased items in the training set to represent examples of user historical behaviors. For comparison, we also present 10 explainable paths generated by three methods for the same user in the test dataset. We ask 20 human subjects to rank these methods based on whether
Cellphones	Grocery	Automotive
Precision	Recall	NDCG	HR	Precision	Recall	NDCG	HR	Precision	Recall	NDCG	HR
CKE	0.0360	0.1760	0.1847	0.3067	0.0612	0.2528	0.3070	0.4511	0.0458	0.1871	0.2257	0.3621
RippleNet	0.0419	0.2141	0.2177	0.3715	0.0591	0.2682	0.2858	0.4800	-	-	-	-
PGPR	0.0462	0.2148	0.2366	0.3801	0.0649	0.2710	0.3174	0.4926	0.0589	0.2315	0.2804	0.4409
KGAT	0.0476	0.2274	0.2365	0.3835	0.0702	0.2916	0.3381	0.5020	0.0601	0.2500	0.2859	0.4514
HeteroEmbed	0.0527	0.2543	0.2626	0.4226	0.0785	0.3316	0.3701	0.5572	0.0695	0.2923	0.3314	0.5082
LOGER	0.0622	0.2977	0.3227	0.4808	0.0906	0.3754	0.4370	0.6121	0.0743	0.3091	0.3653	0.5346
Table 1: Recommendation quality of all methods on three datasets. The results are computed based on the top-10 recommendation on the test set. The best results are highlighted in bold and the second best results are underlined.
Cellphones	Grocery
JSf	JSw	Avg. Rank	JSf	JSw	Avg. Rank
PGPR	0.56	0.49	2.52	0.42	0.38	2.27
KGAT	0.53	0.45	2.14	0.39	0.41	2.08
LOGER	0.47	0.32	1.52	0.34	0.28	1.75
Table 2: Results of measuring the faithfulness of the generated paths obtained by three methods. Bold numbers indicate the best results.
(a) Precision
(c) NDCG
Figure 2: Recommendation quality under varying sizes of estimated hidden triples.
(b) Recall
(d) Hit Rate
the generated paths are consistent with those from the training set. Then, we calculate the average ranking scores (Avg. Rank) by averaging the rank given by each human tester on each method.
Results The results on the Cellphones and Grocery datasets are reported in Table 2. We observe that our method LOGER achieves the lowest JS scores and average ranking score, which reveal the effectiveness of our model in producing more faithful explanations in both quantitative measurements and in the user study.
We further study how hidden triplets used in training KG encoder (Eq. 2) influence the recommendation performance. We experiment on
the Cellphones data under different sizes of hidden triplet sets  $H^{+}$ . We choose the sizes of  $\{10, 20, 30, 40, 50\}$  and keep all other settings unchanged. The results are plotted in Fig. 2, including our model (red circles) and the best baseline HeteroEmbed (blue crosses). We find that our model consistently outperforms the baseline in all the metrics under different numbers of hidden triplets. Better recommendation performance can be achieved with more hidden triplets included in training the KG encoder, because more candidate items will enhance the capability of our model to discern the logical rules of good quality and hence benefit the recommendation prediction.
In this paper, we propose LOGER for faithfully explainable recommendation, which generates explainable paths based on personalized rule importance scores via neural logic reasoning that adequately captures historic user behavior. We experiment on three large-scale datasets for e-commerce recommendation showing superior recommendation quality of LOGER as well as the faithfulness of the generated explanations both quantitatively and qualitatively. We hope to encourage future work that values explainability and in particular the faithfulness of explanations. Our code is available at https://github.com/orcax/LOGER.
We thank the reviewers for the valuable feedback and suggestions. This work was supported in part by NSF IIS-1910154 and IIS-2007907. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.
Qingyao Ai, Vahid Azizi, Xu Chen, and Yongfeng Zhang. 2018. Learning heterogeneous knowledge base embeddings for explainable recommendation. Algorithms.
Qingyao Ai, Yongfeng Zhang, Keping Bi, and W Bruce Croft. 2019. Explainable product search with a dynamic relation embedding model. TOIS.
Julian Besag. 1975. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society: Series D (The Statistician), 24(3):179-195.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795.
Zuohui Fu, Yikun Xian, Ruoyuan Gao, Jieyu Zhao, Qiaoying Huang, Yingqiang Ge, Shuyuan Xu, Shi-jie Geng, Chirag Shah, Yongfeng Zhang, et al. 2020a. Fairness-aware explainable recommendation over knowledge graphs. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 69-78.
Zuohui Fu, Yikun Xian, Yaxin Zhu, Yongfeng Zhang, and Gerard de Melo. 2020b. Cookie: A dataset for conversational recommendation over knowledge graphs in e-commerce. arXiv preprint arXiv:2008.09237.
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S Yu. 2020. A survey on knowledge graphs: Representation, acquisition and applications. arXiv preprint arXiv:2002.00388.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. 2019. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 131-138.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, page 2181-2187. AAAI Press.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. Opendialkg: Explanable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 845-854.