SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To perform InF, we leverage domain knowledge and expert-defined thresholding to determine which concepts in  $C_i$  for  $(s_i, a_i)$  critically contribute towards the agent's ability to reach goal  $G$ . Thresholds are determined via qualitative analysis of an RL agent's policy. In continuous domains, these threshold values are extracted by finding the values in the agent's state space that determine turning points in the agent's ability to reach goal  $G$ . For example, qualitative analysis may produce that an agent's  $pos_x > 0.15$  is a critical turning point for the agent's ability to land close to the center, and if  $pos_x > 0.15$ , the agent's action is crucially linked with decreasing its  $pos_x$ . Thus,  $C_{iw/InF}$  represents the concept set after thresholding is applied, in which concepts  $c \in C_i$  that do not meet the determined thresholds are filtered out. As a result, an abstracted explanation, using InF, templates  $C_{iw/InF}$  into a natural language explanation  $\mathcal{E}_{C_{iw/InF}}^i$ . In Appendix A, we provide more details on how concepts are filtered via qualitative analysis in InF and include an ablation study analyzing the sensitivity of chosen thresholds. Note, while we qualitatively determine thresholds, the InF method can be automated in future work.
Temporal Grouping (TeG) TeG automatically groups explanations over a sequence of time that represents a consecutive pattern. For example, if in Lunar Lander,  $\mathcal{E}_{C_i}^i$  and  $\mathcal{E}_{C_i}^{i + 2}$  is "Fire left engine because it decreases tilt" and  $\mathcal{E}_{C_i}^{i + 1}$  and  $\mathcal{E}_{C_i}^{i + 3}$  is "Fire right engine because it decreases tilt", applying TeG produces "For the next 4 steps, alternate firing left and right engine to decrease tilt".
To perform TeG, we analyze an agent's rollout at deployment to extract series of  $(s_{[i,i + n]},a_{[i,i + n]})$  with a repeated pattern of concept sets  $C_{[i,i + n]}$ . Therefore, a grouped explanation provides a single  $\mathcal{E}_{C_{iw / TeG}}^i$ , that templates the repeated pattern within  $C_{[i,i + n]}$  across the  $n$  state-action pairs. We posit that TeG is likely to be important in domains where actions are sampled at high frequency (e.g. Lunar Lander or Robotics), requiring an abstraction over actions to provide meaningful explanations for consecutively alike state-action pairs.
Overall, the "Explanation Abstraction" component in S2E (see Fig. 1c) determines whether InF, TeG or both should be applied to  $\mathcal{E}_{C_i}^i$  for providing an abstracted, concept-based explanation. In Section 6 we demonstrate the effectiveness of InF and TeG in high frequency domains, such as Lunar Lander, for improved end user understanding.
5 Model Evaluations
In this section, we perform quantitative evaluations to validate the joint embedding model  $M$  in S2E by evaluating  $M$ 's explanation retrieval accuracy (Sec. 5.2), and  $M$ 's ability to inform reward shaping during agent training (Sec. 5.3).
5.1 Evaluation Domains
Connect 4 is an adversarial board game in which the objective is to achieve a four-in-a-row, in any direction. The state space is a discrete, 2D array representation of token positions. Also, the action
space is discrete and each player selects the column to place a token in. Lunar Lander is a trajectory optimization problem in which the lander must land on a landing pad. We utilize LunarLander-v2 from OpenAI Gym [4]. The state space is a continuous 1D array including the lander's linear position and velocity, angular velocity, angle, and leg contact. The game has four discrete actions: fire left engine, fire right engine, do nothing, and fire main engine.
Concept Collection Concepts for a domain may be extracted from expert players or commentators. However for Lunar Lander and Connect 4, there are no existing datasets for mining concepts. Thus, we leverage expert domain knowledge to attribute concepts to state-action pairs. Figure 2 lists the set of concepts we consider when attributing concepts to a  $(s_i, a_i)$  in Connect 4 and Lunar Lander, as well as provides example  $\mathcal{E}_{C_i}^i$ . Concepts for Connect 4 are derived from the strategies outlined in [3]. Note, "NULL" describes all other  $(s_i, a_i)$  that are not attributed to a concept. Alternatively, concepts for Lunar Lander are derived from the domain's existing reward function [4]. In this domain, the existing dense reward function includes the primary concepts important towards successful task completion. In Appendices B.1 and B.2, we provide an expanded list of example  $\mathcal{E}_{C_i}^i$ , spanning all concepts derived for Connect 4 and Lunar Lander.
Note, for both domains, the concepts are defined via mathematical or logical representations of the state. For example, in Connect 4, the concept of "BW", blocking an opponent win, can explicitly be encoded by board representations where any three-in-a-row pattern is blocked with a token from the opponent player. Similarly, in Lunar Lander, the "POS", position concept, is modelled by mathematical representation of the agent's position over time. Deriving concepts via mathematical or logical representations allow us to automatically collect concepts from states, as well as use such mathematical or logical rules to evaluate that concepts and states are accurately paired. In many applications, it may be infeasible to derive mathematical or logical rules from a state representation. In these scenarios, concepts can be collected via crowd sourcing, [20], or obtained via "think-aloud" procedures [14].
5.2 Evaluation of Joint Embedding Models
Below we demonstrate high recall rates of the joint embedding models in S2E in both the Connect 4 and Lunar Lander domain.
Datasets Recall from Section 4.1 that to train  $M$  we utilize a dataset  $D = \{D^{a}, D^{m}\}$ . For Connect 4,  $D_{C4}$  includes approximately 3 million samples, and for Lunar Lander,  $D_{LL}$  includes approximately 2 million samples. We randomly sample  $D_{C4}^{a}$  and  $D_{LL}^{a}$  from an expert RL policy. We obtain  $D_{C4}^{m}$  and  $D_{LL}^{m}$ , by selecting  $z$  incorrect concepts from  $C_{C4}$  and  $C_{LL}$ , as replacement for each sample in  $D_{C4}^{a}$  and  $D_{LL}^{a}$  (more detailed numbers in Appendix C).
Model Architectures & Training  $M_{C4}$  and  $M_{LL}$  denote the joint embedding models for Connect 4 and Lunar Lander, respectively. The model architectures for  $M_{C4}$  and  $M_{LL}$  leverage LSTMs to learn explanation embedding  $exp_{\text{embed}}^i$  for a given  $\mathcal{E}_{C_i}^i$ . For Connect 4, to learn  $s_{\text{embed}}^i$  for  $(s_i, a_i)$ , we leverage Convolutional Neural Networks to learn local and spatial relationships between tokens on a 2D game board. For Lunar Lander, we leverage Fully Connected Networks to learn  $s_{\text{embed}}^i$ . More details (split, learning rate, etc) are in Appendix C.
Results Similar to image-to-text retrieval [31, 6], we evaluate  $M_{C4}$  and  $M_{LL}$  via recall rate, at  $k = \{1,2,3\}$ , which evaluates whether the retrieved  $\mathcal{E}_{C_i}^i$  is ranked in the top  $k$ . Figure 3a provides the average recall rates of  $M_{C4}$  and  $M_{LL}$ , across 5 random seeds. We observe that  $M_{LL}$  performs with near  $100\%$  accuracy, whereas  $M_{C4}$  has an average recall rate of  $88\%$ . Given the lower recall rates by  $M_{C4}$ , in Figure 3b we examine the false positive and false negative explanation retrievals. We see that  $M_{C4}$  has greatest challenge in correctly retrieving  $\mathcal{E}_{C_i}^i$  with "BW" and "3IR_BL". We posit that "BW" and "3IR_BL" may occur variation in comparison to other concepts, leading to higher incorrect rate of  $M_{C4}$  and  $M_{LL}$  are in Appendix C.4.
Note, incorrect retrievals of  $\mathcal{E}_{C_i}^i$  for a given  $(s_i,a_i)$  can have a negative downstream impact within S2E. Specifically, when S2E is leveraged during the RL agent's training, incorrect retrievals of
(a)
(b)
Figure 3: (a) recall@k of  $M_{C4}$  and  $M_{LL}$ , and (b) confusion matrix for  $M_{C4}$ 's  $\mathcal{E}_{C_i}$  retrievals.
Figure 4:  $M_{C4}$  improves agent learning rate by  $\sim 200$  training steps compared to SoTA agent without reward shaping (a), while  $M_{LL}$  maintains similar agent learning rate to SoTA agent with expert-defined reward shaping (b and c).
$\mathcal{E}_{C_i}^i$  can incorrectly providing shaping rewards to the agent and in return impacting learned agent policy. Similarly, when S2E is leveraged at deployment to provide end-user understanding, incorrect retrievals of  $\mathcal{E}_{C_i}^i$  can confuse end-users and hinder their understanding of the agent's behavior. However, in Section 5.3 our results demonstrate that the percentage of incorrect retrievals from the joint embedding models do not significantly impact the RL agent's learned policy. Similarly, in Section 6, we demonstrate that the percentage of incorrect explanation retrievals are not significantly detrimental to user performance. Nevertheless, it is important that the joint embedding models within S2E have high recall rate, especially when applied in high-stakes or mission critical scenarios.
5.3 Evaluation of  $M$  for Reward Shaping in Agent Training
We validate that both joint embedding models in S2E,  $M_{C4}$  and  $M_{LL}$ , inform reward shaping comparable to expert-defined dense, reward functions.
RL Agent & Shaping Rewards Our S2E framework is model agnostic and  $M$  does not make any assumptions on the type of RL model utilized. Given our domains are complex games, we leverage a state-of-the-art (SoTA) RL algorithm MuZero [47] to evaluate  $M$ 's ability to inform reward shaping. We use the open-source version of MuZero available in [52] (details in Appendix D.1).
Recall from Section 4 that  $M_{C4}$  and  $M_{LL}$  inform when to provide shaping rewards, but the shaping values are expert determined. For Lunar Lander, there exists a SoTA dense, reward function [4]. In Connect 4, to our knowledge, there is no SoTA dense, reward function. Thus, we perform a hyperparameter sweep to assign shaping values for each concept (values provided in Appendix D.2).
Results To evaluate the efficacy of  $M_{C4}$  and  $M_{LL}$  in informing reward shaping, we measure the agents' Reward and Win%. In Lunar Lander, we evaluate  $M_{LL}$ 's ability to inform reward shaping (MuZero+S2E-RS) in comparison to a baseline MuZero agent that is informed by an existing expert-defined reward shaping (MuZero+E-RS). Figure 4b and Figure 4c demonstrate that both the MuZero+S2E-RS and MuZero+E-RS agents achieve comparable Reward and Win%. While performance is comparable between both agents, in MuZero+S2E-RS,  $M_{LL}$  provides an automated method to determine when a reward value should be presented, which otherwise has to be manually encoded, such as in MuZero+E-RS via an expert-defined function.
In Connect 4, since there does not exist a SoTA reward shaping function, we compare  $M_{C4}$ 's ability to inform reward shaping (MuZero+S2E-RS) against a baseline MuZero agent with sparse rewards (MuZero+No-RS). Figure 4a shows that MuZero+S2E-RS has improved learning rate and requires  $\sim 200\mathrm{k}$  less training samples to achieve similar Win% in comparison to MuZero+No-RS. We also evaluate MuZero+S2E-RS against an upper bound in which we provide expert-defined reward shaping (MuZero+E-RS) to analyze how incorrect retrievals from  $M_{C4}$  may affect the agent's learning rate. Although  $M_{C4}$  retrieves incorrect explanations  $13.9\%$  of times (see Appendix C.4), we observe that MuZero+S2E-RS's learning rate is not significantly affected (Fig. 4a).
Overall, these results demonstrate the joint embedding models' ability to effectively inform reward shaping. In Connect 4, we demonstrate that, even with an imperfect joint embedding model, S2E can inform reward shaping and improve the agent's learning rate compared to the SoTA agent trained on sparse rewards. Similarly, in Lunar Lander, we demonstrate that our S2E informs reward shaping comparably to the existing SoTA agent that is trained with an expert-defined dense, reward function.
6 User Evaluation
In this section, we validate S2E with end-users, demonstrating that our retrieved  $\mathcal{E}_{C_i}$  significantly improve user task performance in both Connect 4 and Lunar Lander.
Study Procedure Participants performed an online study in which they played several games from the domains in four stages. Specifically, 1) Practice included playing 2 practice games, 2) Pre-Test included playing 3 scored games, 3) Explanation included interacting with an expert player (well-trained RL agent) with exposure to explanations from an assigned study condition, if applicable and 4) Post-Test included playing 3 more scored games. Details about the study procedures are in Appendix E.1.
Metrics We measure the difference between participant Pre-Test and Post-Test Adjusted Task Score (ATS) to analyze any task improvement with exposure to the study conditions. Participant ATS is defined by the expert-defined reward functions utilized by the RL agents during training (evaluated in Sec. 5.3, detailed in Appendix D.2). If  $R(s,a)$  denotes the reward associated with a given  $(s,a)$  and  $N$  defines the total actions taken by the user in a game, then ATS is defined as a normalized aggregation of user rewards received in a game:  $ATS = \sum_{n}^{N}R(s_{n},a_{n}) / N$ .
6.1 Connect 4 Study Specifics
Study Conditions We conducted a five-way between-subjects study with the following conditions. Participants could receive: 1) None (Baseline): no information about the agent's action to be played, 2)  $\mathcal{E}_A$  (Baseline): action-based explanation that contains the action to be played, 3)  $\mathcal{E}_V$  (Baseline): value-based explanation stating the action to be played along with the action's value in comparison to the values of all other actions for a given state, 4) GT  $\mathcal{E}_{C_i}$ : concept-based explanation using expert-defined, ground-truth concepts for a state-action pair, and 5) S2E  $\mathcal{E}_{C_i}$ : a concept-based explanation from S2E for a state-action pair. Note, given that Connect 4 has a discrete state space, and actions are sampled at low frequency, the domain does not need further abstracted  $\mathcal{E}_{C_i}$  via InF and TeG. Additionally,  $\mathcal{E}_V$  is similar to the action-cost condition in [48], as well as the Q-values presented per action in [27]. Note, precondition-based explanations in [48]
Figure 5: User adjusted task scores (ATS) in Connect 4. Statistical significance:  $^* p <   0.05$  \*\*  $\mathfrak{p} <   0.01$  , and \*\*\*  $\mathfrak{p} <   0.001$  Details in Appendix E.3.
require hierarchical domains, and causal-based explanations in [36] require a learned structured causal model which prevent direct comparison with our framework and domains.
Results The following results are from an IRB-approved study with participants recruited via Amazon Mechanical Turk (n=75, details in Appendix E.2). Our data is analyzed with a one-way ANOVA and a Tukey HSD post-hoc test, given that the assumptions of homoscedasticity (Levene's Test, p>0.3), and normality (Shapiro-Wilke, p>0.1) are met. Figure 5 shows a significant effect of explanation type on ATS (F(4, 70)=8.56, p<0.001). Specifically, we observe improvement in participant's ATS with exposure to our S2E  $\mathcal{E}_{C_i}$ , in comparison to None (t(70) = 3.08, p<0.05),  $\mathcal{E}_A$  (t(70)=-3.25, p<0.05) and  $\mathcal{E}_V$  (t(70)=4.03, p<0.01). Additionally, we observe similar ATS improvement when exposed to S2E  $\mathcal{E}_{C_i}$  in comparison to GT  $\mathcal{E}_{C_i}$ . These results indicate the benefit of our S2E produced  $\mathcal{E}_{C_i}$  in helping participants better understand Connect 4 and improve their ATS.
6.2 Lunar Lander Study Specifics
The underlying physics needed for Lunar Lander results in agent  $(s_i, a_i)$  being sampled at high frequency and multiple, repetitive concepts being attributed to  $(s_i, a_i)$  pairs. For example, "decrease velocity" is a valid concept for every  $(s_i, a_i)$  until the end of a game. Recall from Section 4.3 that  $\mathcal{E}_{C_i}$  may need to be further abstracted in these scenarios to provide meaningful explanations to end-users. Therefore, to evaluate the utility of abstracted concept-based explanations, we introduce 3 more study conditions that utilize our InF and TeG methods outlined in Section 4.3.
Study Conditions We perform a seven-way between-subjects user study. Similar to Connect 4, we include None,  $\varepsilon_{A}$ ,  $\varepsilon_{V}$ , and S2E  $\varepsilon_{C_i}$ . Unique to our Lunar Lander study participants
Figure 6: User ATS in Lunar Lander;  $*\mathrm{p} <   0.05$ $^{**}\mathrm{p} <   0.01$
could receive 1) S2E  $\mathcal{E}_{C_i\mathrm{w / TeG}}$ : a concept-based explanation from S2E with additional abstractions performed using the TeG method, 2) S2E  $\mathcal{E}_{C_i\mathrm{w / InF}}$ : a concept-based explanation from S2E with additional abstractions performed using the InF method, and 3) S2E  $\mathcal{E}_{C_i\mathrm{w / InF},\mathrm{TeG}}$ : a concept-based explanation from S2E with additional abstractions performed using both TeG and InF.
Results The following results are from an IRB-approved study with participants recruited via Amazon Mechanical Turk (n=105, details in Appendix E.2). Our data is analyzed with a one-way ANOVA and a Tukey HSD post-hoc test, given that the assumptions of homoscedasticity (Levene's Test,  $p > 0.7$ ), and normality (Shapiro-Wilke,  $p > 0.1$ ) are met. Figure 6 shows a significant effect of explanation type on participant ATS ( $F(6,98) = 3.67$ ;  $p < 0.01$ ). Specifically, we see significant improvement in participant's ATS with exposure to S2E  $\mathcal{E}_{C_i\mathrm{w / InF},\mathrm{TeG}}$ , in comparison to None ( $t(98) = 3.15$ ,  $p < 0.05$ ),  $\mathcal{E}_A(t(98) = -3.35$ ,  $p < 0.05$ ),  $\mathcal{E}_V(t(98) = 3.59$ ,  $p < 0.01$ ) and  $\mathcal{E}_{C_i}$  ( $t(98) = -3.19$ ,  $p < 0.05$ ). These results demonstrate the need of our S2E abstraction methods when providing concept-based explanations to end-users, and that usage of both InF and TeG in high-frequency RL domains are crucial for ATS improvement.
7 Discussion and Conclusions
Our work introduces a unified framework, S2E, that involves learning a joint embedding model between agent state-action pairs and concept-based explanations to provide a dual benefit to the agent and end-user. We additionally outline a desiderata for what may constitute as a "concept" in sequential decision making problems, beyond the scope of prior concept-based explanations for sequential decision making. Our model evaluations demonstrate that the joint embedding model in S2E provides an automatic method for determining when reward shaping should be provided to improve agent learning rates. Our user evaluations demonstrate that concept-based explanations can significantly improve user task performance (Connect 4), but when considering high-frequency RL domains (Lunar Lander), the additional abstraction methods from S2E are important for producing abstracted concept-based explanations that significantly improve user task performance.
Limitations: We present several areas of future work that aims to improve the generalizability of S2E. For instance, the concept-based explanations in our work are derived from mathematical representations and expert knowledge in each domain (see Sec. 5.1). To expand the generalizability of the S2E method to other complex domains, such as Robotics or open-world games, future work should explore how to collect and extract concepts in scenarios where mathematical representations of concepts may be infeasible. A future direction may include collecting expert commentary [20] or think-aloud [14] sessions from which concepts are automatically derived. Additionally, in many real-world applications, having access to large amounts of data for a domain may be infeasible. Thus, future work should also explore how to adapt the joint embedding model within S2E for few shot learning. Additionally, for application to high-stakes domain, future work should explore how to remediate incorrect explanation retrievals from the joint embedding model within S2E. While the small percentage of incorrect explanation retrievals do not significantly impact user task performance and agent learning in our tested domains, inaccurate explanation retrievals may have significant effects in mission-critical tasks. It is also important to highlight that our current concept-based explanations follow a fixed template for each unique concept set (see Appendices B.1, B.2). However, there are multiple ways of explaining the same concept, and future work includes learning a joint embedding model with greater language variety in the explanations combined with automatically generating such templates using language models. Furthermore, the InF method within S2E utilizes manually defined thresholds to produce abstracted concept-based explanations, and future work entails developing an automated InF method.
Broader Impacts: S2E shows promise in applying the Protégé Effect to Human-AI Interaction, and that explanations of agent behavior are beneficial to both agents and users. These insights may promote AI agent architects to consider the utility of self-explaining agents in accelerating learning as well as providing transparency. Additionally, our work shows promise in using concept-based explanations of well-trained AI agents as a teaching tool, helping users improve their task performance. A potential risk of S2E is purposely training a joint embedding model to associate misleading concept-based explanations with state-action pairs. This could lead to a badly performing agent, and the agent providing deceitful explanations of its actions. Future work in explaining black-box AI systems should explore how to detect and prevent deceptive explanations.
[55] Luowei Zhou, Chenliang Xu, Parker Koch, and Jason J Corso. Watch what you just said: Image captioning with text-conditional attention. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pages 305–313, 2017.
[56] Kai Zhu and Tao Zhang. Deep reinforcement learning based mobile robot navigation: A review. Tsinghua Science and Technology, 26(5):674-691, 2021.
[57] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaii, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
A Information Filtering (InF) Details
Recall from Section 4.3 that we utilize expert-defined thresholding and domain knowledge to determine which concepts within  $C_i$  for a  $(s_i, a_i)$  critically contribute towards the agent's ability to reach goal  $G$ . Specifically, these thresholds are expert-defined upper and lower bounds on the agent's state values that denote the agent's ability to succeed or fail in its goal. In our InF method, these thresholds are not mathematically derived, but are derived from RL-expert analysis. For a given domain, an RL expert visualizes multiple policy rollouts and analyze the different state values over time to manually determine the upper and lower bounds (turning points) that influence the agent's ability to reach G. Below we demonstrate how thresholds in InF for one of our evaluation domains, Lunar Lander.
The concepts relevant to Lunar Lander include, position, velocity, tilt, side and main fuel, right and left leg contacts, and landing, and these concepts are derived from the existing, expert-defined, dense reward function [4] (see Section 5.1). Within the dense, reward function, binary concept components are rewarded via constant shaping values, while the continuous concepts are rewarded through a continuous shaping function (see Appendix D.2). In our work, we consider applying thresholding to concepts defined via a continuous function since these scenarios typically result in multiple concepts being rewarded over multiple time steps, and are more likely to warrant information filtering.