SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the context of Lunar Lander, Figure 7 and Figure 8 show the two concepts on which thresholding is applied when examining a well trained agent's policy, specifically the lander's x-position and tilt. Recall that thresholds for InF can be determined by examining an agent's state values to identify turning points in the agent's ability to reach  $G$ . In Figure 7, we extract that a suitable threshold for the x-position concept is 0.15. We identify that such x-position value corresponds to the time step at which the agent starts firing its left engine to critically correct its increasing position value. Similarly, in Figure 8, we extract that suitable thresholds for the tilt concept are 0.01 and -0.05. We find that such tilt values correspond with frequently alternating firing of left and right engine to critically correct an increasing tilt value. We do not perform a thresholding analysis on velocity, as with domain analysis we observe that decreasing the agent's velocity is directly associated with firing the main engine and is performed to prevent crashing. Additionally, we do not consider the agent's y-position for thresholding as the agent follows a steady, downward linear trajectory, and visible turning points cannot be extracted by qualitative analysis. While we qualitatively determine thresholds, the InF method can be automated in future work.
Overall, in our InF method, if  $C_i$  denotes the original concept set for a given  $(s_i, a_i)$ , then  $C_{i\mathrm{w / InF}}$  represents the concept set after thresholding is applied, in which concepts  $c \in C_i$  that do not meet the threshold are filtered out. As a result, an abstracted explanation, using InF, templates  $Ct_i$  into a natural language explanation  $\mathcal{E}_{C_{iw / InF}}^i$ .
A.1 Thresholding Sensitivity Analysis
As stated above, the thresholds are manually derived from RL-expert analysis. Specifically, an RL expert visualizes multiple policy rollouts and analyzes the different state values over time to determine approximate upper and lower bounds (turning points) that influence the agent's ability to reach  $G$ . Given the thresholding approach in InF, different threshold values can result in filtering of different concepts and therefore produce different abstracted, concept-based explanations. To study the sensitivity of our chosen thresholds in Lunar Lander, we analyze how different threshold values for each concept affect the number of concepts filtered in a policy rollout.
Figure 9 demonstrate what fraction of concepts are filtered (y-axis) as the threshold values change (x-axis). When looking at Figure 9a, analyzing threshold values for the x-position concept, we see our chosen value is within the elbow of the curve, denoting that the rate of filtration of the concept slows down after 0.15. Similarly, when analyzing the threshold values for the tilt concept in Figure 9b and Figure 9c, we see the lower and upper bound of the tilt thresholds are also within the "elbow" of each curve. Note, when analyzing the upper bound threshold for the tilt concept (Figure 9b), the lower bound tilt threshold is fixed. Similarly, when analyzing the lower bound threshold for the tilt concept (Figure 9c), the upper bound tilt threshold is fixed. Overall, our threshold values being within the elbow of each curve provide some validation on their soundness. However, to study the complete impact of varying thresholding values for each concept, additional user studies with abstracted concept-based explanations produced from varying threshold values are required. We
Figure 7: The position (top) and corresponding actions (bottom) of a trajectory sampled from a well-trained Lunar Lander agent policy.
Figure 8: The tilt angle (top) and corresponding actions (bottom) of a trajectory sampled from a well-trained Lunar Lander agent policy.
(c)
Figure 9: Sensitivity analysis on how the rate of concept filtration changes across various threshold values for Lunar Lander concepts of x-position (a) and tilt (b and c).
consider such analyses beyond the scope of our work, and one to further explore when considering future methods on automating the information filtering submodule within our S2E framework.
B Concept-Based Explanations for Experimental Domains
Below we present an exhaustive list of the different  $\mathcal{E}_{C_i}^i$ , utilized for each domain, paired with an instance of a  $(s_i,a_i)$  pair for context. Note, the rows denote the possible concept sets  $C_i$ , and their corresponding fixed, templated explanations starting with "because". The action  $a_i$  is always appended to the beginning of the explanation.
B.1 Connect 4
\( (s_i,a_i) \)	\( C_i \)	Corresponding \( \mathcal{E}_{C_i}^i \)
\(\{3\text{IR}\}\)	"Play column 4 because it creates a three-in-a-row."
\(\{3\text{IR\_BL}\}\)	"Play column 7 as a neutral move, it creates a three-in-a-row that is blocked by the opponent from a win."
\(\{3\text{IR},\text{CD}\}\)	"Play column 4 because it provides center dominance and creates a three-in-a-row."
\(\{BW\}\)	"Play column 6 because it blocks the opponent from a win"
\(\{BW,3\text{IR}\}\)	"Play column 5 because it creates a three-in-a-row and blocks the opponent from a win."
\(\{BW,CD\}\)	"Play column 4 because it provides center dominance and blocks the opponent from a win."
\(\{CD\}\)	"Play column 4 because it provides center dominance"
\(\{W\}\)	"Play column 7 because it leads to a four-in-a-row win"
\(\{NULL\}\)	"Play column 6 as a generic move not tied to a particular strategy"
Table 1: Example state action pairs with associated concept lists and concept-based explanations for the Connect 4 domain.
B.2 Lunar Lander
\( (s_i,a_i) \)	\( C_i \)	Corresponding \( \mathcal{E}_{C_i}^i \)
{POS, VEL,TILT, SF}	"Fire main engine because it moves lander closer to the center, decreases lander speed to avoid crashing, decreases tilt of lander, and conserves side fuel usage."
{POS, VEL,TILT, MF}	"Fire side engine because it moves lander closer to the center, decreases lander speed to avoid crashing, decreases tilt of lander, and conserves main fuel usage."
{POS, VEL,TILT}	"Do nothing because it moves lander closer to the center, decreases lander speed to avoid crashing, and decreases tilt of lander."
{POS, VEL,TILT, LLEG, SF}	"Fire main engine because it moves lander closer to the center, decreases lander speed to avoid crashing, decreases tilt of lander, encourages left leg contact, and conserves side fuel usage."
{POS, VEL,TILT, LLEG, MF}	"Fire side engine because it moves lander closer to the center, decreases lander speed to avoid crashing, decreases tilt of lander, encourages left leg contact, and conserves main fuel usage."
{POS, VEL,TILT, LLEG}	"Do nothing because it moves lander closer to the center, decreases lander speed to avoid crashing, decreases tilt of lander and encourages left leg contact."
{POS, VEL,TILT, RLEG, SF}	"Fire main engine because it moves lander closer to the center, decreases lander speed to avoid crashing, decreases tilt of lander and encourages right leg contact, and conserves side fuel."
{POS, VEL,TILT, RLEG, MF}	"Fire side engine because it moves lander closer to the center, decreases lander speed to avoid crashing, decreases tilt of lander, encourages right leg contact, and conserves main fuel."
{POS, VEL,TILT, RLEG}	"Do nothing because it moves lander closer to the center, decreases lander speed to avoid crashing, decreases tilt of lander, and encourages right leg contact."
{POS, VEL,TILT, LLEG,RLEG, SF}	"Fire main engine because it moves lander closer to the center, decreases lander speed to avoid crashing, decreases the tilt of the lander, encourages left and right leg contact and conserves side fuel."
{POS, VEL,TILT, LLEG,RLEG, MF}	"Fire side engine because it moves lander closer to the center, decreases the lander speed to avoid crashing, decreases the tilt of the lander, encourages left and right leg contact, and conserves main fuel."
{POS, VEL, TILT, LLEG, RLEG}	"Do nothing because it moves lander closer to the center, decreases the lander speed to avoid crashing, decreases the tilt of the lander, encourages left and right leg contact."
{L}	"Do nothing because it results in a land."
C Joint Embedding Model Details
C.1 Dataset Details
MisAligned and Aligned Data Breakdown Recall from Section 5 that  $D_{C4}$  represents the dataset used for training and evaluating  $M$  in Connect 4. Specifically,  $D_{C4}$  includes 2,948,742 total samples, in which  $D_{C4}^{a} = 327$ , 638 aligned samples and  $D_{C4}^{m} = 2$ , 621, 104 misaligned samples. Similarly,  $D_{LL}$  represents the dataset used for training and evaluating  $M$  in Lunar Lander. Specifically,  $D_{LL}$  includes 1,941,042 total samples, in which  $D_{LL}^{a} = 323$ , 507 aligned samples and  $D_{LL}^{m} = 1$ , 617, 535 misaligned samples. Recall that to create the misaligned dataset,  $D^{m}$ , we utilize  $z$  incorrect concept sets available in each domain as replacement for the existing correct concepts utilized in  $\mathcal{E}_{C_i}^i$  for each  $d_{i}\in D^{a}$ . Specifically, for Connect 4,  $z = 8$  and we leverage all possible non-aligned concept sets for perturbation. In Lunar Lander,  $z = 5$  and we randomly choose five non-aligned concept sets for perturbation. We randomly choose 5 of 13 possible concept sets to generate misaligned sample pairs per state to avoid a larger than necessary dataset. As a refresher, the list of possible concept sets for each domain are presented in Figure 2.
Concept Breakdown in Aligned Data Table 3 and Table 4 provide a breakdown of the occurrences of each concept set within the training data for the joint embedding models,  $M_{C4}$  (Connect 4) and  $M_{LL}$  (Lunar Lander). We specifically show the breakdown in the aligned datasets,  $D_{C4}^{a}$  and  $D_{LL}^{a}$ , given that such data is the raw data collected via rollout simulations in each domain. Recall that the additional misaligned data in the training set,  $D_{C4}^{m}$  and  $D_{LL}^{m}$ , are synthetically generated via perturbations, and used for the contrastive learning. When the misaligned data is included in the training set, given the perturbations, the number of occurrences of each concept (aligned or misaligned) become equal.
Table 2: Example state action pairs with associated concept lists and concept-based explanations for the Connect 4 domain.
Concept Set	# of Samples in \(D_{C4}^{aligned}\)
{3IR}	43,896
{3IR_BL}	21,911
{3IR, CD}	6,901
{BW}	25,352
{BW, 3IR}	10,668
{BW, CD}	4,940
{CD}	30,591
{W}	23,137
{NULL}	160,242
Table 3: Concept set occurrence analysis of the aligned data within the training set used to train  ${M}_{C4}$  .
C.2 Model Architectures
Figure 10 shows the model architecture for  $M_{C4}$ . Recall that the inputs to  $M_{C4}$  include  $\langle [s_i, s_{i-1}], g_i, \mathcal{E}_{C_i}^i, y_i \rangle$ . In the diagram, we don't explicitly show  $y_i$  but it is used to denote whether
Concept Set	# of Samples in \(D_{LL}^{aligned}\)
{POS, VEL, TILT, MF}	136,917
{POS, VEL, TILT, SF}	62,140
{POS, VEL, TILT}	53,944
{POS, VEL, TILT, LLEG, MF}	1,610
{POS, VEL, TILT, LLEG, SF}	2,981
{POS, VEL, TILT, LLEG}	4,219
{POS, VEL, TILT, RLEG, MF}	1,523
{POS, VEL, TILT, RLEG, SF}	2,399
{POS, VEL, TILT, RLEG}	3,228
{POS, VEL, TILT, LLEG, RLEG, MF}	1,730
{POS, VEL, TILT, LLEG, RLEG, SF}	12,026
{POS, VEL, TILT, LLEG, RLEG}	39,981
{L}	809
Table 4: Concept set occurrence analysis of the aligned data within the training set used to train  ${M}_{LL}$  .
$s_i$  and  $\mathcal{E}_{C_i}^i$  are aligned or misaligned (see Section 4). The game boards ( $s_i$  and  $s_{i-1}$ ) in Connect 4 are represented as 6x7 2D array which are passed into two consecutive CNNs after padding to 7x7 2D arrays. The first set of CNNs (CNN1 and CNN3) for  $s_i$  and  $s_{i-1}$  have the following parameters: input_channels=1, output_channels=4, kernel_size=(3,3), stride=1. The second set of CNNs (CNN2 and CNN4) have the following parameters: input_channels=4, output_channels=6, kernel_size=(3,3), stride=1. The embedding outputs of CNN2 and CNN4 are then concatenated with the game information  $g_i$ , which includes boolean representations of the player/opponent as well as whether the game is over. The concatenated embedding is then passed through three FCNs. Specifically, FCN1's output dimension is 64. FCN2's output dimension is 32. FCN3's output dimension is 16. The output of FCN3 represents our final state embedding,  $s_{\text{embed}}^i$ . The concept-based explanation  $\mathcal{E}_{C_i}^i$  is first passed through an embedding layer and then passed to a LSTM network to produce our  $exp_{\text{embed}}^i$ . The input size to the LSTM is of dimension 32, and output dimension is 16.
Figure 11 shows the model architecture for  $M_{LL}$ . The inputs to  $M_{LL}$  include  $\langle [s_i, s_{i-1}], g_i, \mathcal{E}_{C_i}^i, y_i \rangle$ . In the diagram, we don't explicitly show  $y_i$ , but it is used to denote whether  $s_i$  and  $\mathcal{E}_{C_i}^i$  are aligned or misaligned (see Section 4). The game boards  $(s_i$  and  $s_{i-1})$  in Lunar Lander are represented as 1x10 arrays which are passed into two consecutive FCNs. Specifically, the game board representation includes the 1x8 game state representation from OpenAI Gym [4], along with an inclusion of whether the side fuel or main fuel is in use. The first set of FCNs (FCN1 and FCN3) for  $s_i$  and  $s_{i-1}$  have an output dimension of 64. The second set of FCNs (FCN2 and FCN4) have an output dimension of 32. The embedding outputs of FCN2 and FCN4 are then concatenated with the game information  $g_i$ , which includes a boolean representation of whether the game is over. The concatenated embedding is then passed through three more FCNs Specifically, FCN5 has an output dimension of 16. FCN6 has an output dimension of 8. Similar to the architecture for  $M_{C4}$ , the concept-based explanation  $\mathcal{E}_{C_i}^i$  is first passed through an embedding layer and then passed to a LSTM network to produce our  $exp_{\text{embed}}^i$ . The input size to the LSTM is of dimension 32, and output dimension is 8.
To train and evaluate  $M_{C4}$  and  $M_{LL}$ , we utilize a 60%-20%-20% train-valid-test split on  $D_{C4}$  and  $D_{LL}$  (see total dataset size in Appendix C.1). The models are trained with learning rate of 0.001, batch size of 128, Adam Optimizer, and trained with 10 epochs. To train  $M_{C4}$  and  $M_{LL}$ , we utilize a desktop computer with a NVIDIA GTX 1060 6GB GPU and an Intel i7 processor. The runtime for training and testing  $M_{C4}$  and  $M_{LL}$  takes approximately 30 minutes. See link to code in Appendix F.
C.4 Additional Evaluations
Learned Embedding Spaces of  $M_{C4}$  and  $M_{LL}$  In Figure 12a and Figure 12b, we provide a visualization of our learned joint embedding models,  $M_{C4}$  and  $M_{LL}$ . The TSN-E visualizations is created using the models' training dataset. We do not draw any conclusions via these visualizations, but present them as an intuitive way to interpret our high dimensional, learned embedding spaces.
Figure 10: Joint Embedding Model  $M$  Architecture for Connect 4.
Figure 11: Joint Embedding Model  $M$  Architecture for Lunar Lander.
(b)
Figure 12: (a) and (b) demonstrate visualizations of the learned embedding spaces of the joint embedding models learned for Connect  $4, M_{C4}$ , and Lunar Lander,  $M_{LL}$ .
In the "Wild" Evaluation of  $M_{C4}$  and  $M_{LL}$  We performed an evaluation of how well  $M_{C4}$  and  $M_{LL}$  continue to perform when utilized during an RL agent's training to inform reward shaping. Specifically, we analyzed the model's Recall@1, examining how often  $\mathcal{E}_{C_i}^i$  was incorrectly retrieved for a given  $(s_i, a_i)$ . For Lunar Lander,  $M_{LL}$ 's average Recall@1, across the 5 seeds, during the
Figure 13: Evaluation of  $M_{C4}$ 's ability to retrieve correct  $\mathcal{E}_{C_i}^i$  during the RL agent's training.
agent's training continued to be  $100\%$ . Figure 13 provides an analysis of the Connect 4 joint embedding model's,  $M_{C4}$ , average Recall@1. We see on average,  $M_{C4}$  has an average Recall@1 of  $86.1\%$ , with  $13.9\%$  incorrect retrievals. As mentioned in Section 5.3 results, these incorrect retrievals do not significantly affect  $M_{C4}$ 's ability to inform reward shaping and the agent's learning rate.
D RL Agent Details
D.1 RL Model Training
We utilized the open-source version of MuZero from [52], and utilize the provided hyperparameters. Please see Muzero's Github Repo from [52]  $^{1}$ , as well as our code linked in Appendix F, for more details. To train and validate our MuZero agents we leverage an NVIDIA GTX 1060 6GB GPU and an Intel i7 processor. The runtime for training a Connect 4 MuZero agent to around 400k training steps takes approximately 30 hours. The runtime for training a Lunar Lander MuZero agent to around 400k training steps takes approximately 9 hours.
D.2 Shaping Rewards Per Concepts
For Lunar Lander, we utilize the existing dense reward function to determine the relevant concepts as well as their shaping values. Therefore each  $c \in C_{LL}$  has an expert-defined shaping value. The existing shaping values can be found in the LunarLander.py provided by OpenAI Gym [4]. Table 5 summarizes the continuous functions outlined in [4] that we use to reward each concept  $c \in C_{LL}$ . Alternatively, there are no existing SoTA reward shaping values for the game of Connect 4. Therefore we perform a hyperparameter sweep to determine an optimal set of shaping values, and Table 6 summarizes our utilized shaping values for each concept  $c \in C_{C4}$ .
Lunar Lander Concept	Shaping Values
POS (position)	-100(√(posx)2+(posy)2)
VEL (velocity)	-100(√(velx)2+(vely)2)
TILT (tilt)	-100(|anglelander|)
RLEG (right leg contact)	10(boolRLEG)
LLEG (left leg contact)	10(boolLLEG)
MF (main fuel)	-0.3(boolmain)
SF (side fuel)	-0.03(boolside)
L (land)	100
Table 5: Corresponding shaping values from [4] for each Lunar Lander concept.
<sup>1</sup>https://github.com/werner-duvaud/muzero-general
Connect 4 Concept	Shaping Values
3IR (Three-in-Row)	1
3IR_BL (Three-in-Row Blocked from Win)	0
CD (Center Dominance)	1
BW (Block Win)	5
W (Win)	10
NULL (Neutral State)	0
Table 6: Corresponding shaping values determined from hyperparameter sweep for each Connect 4 concept.
E User Study Details
E.1 Procedure Details
Participants performed an online study in which they were asked to play several games from the domain. The user studies for each domain were broken into the following four stages: Practice, Pre-Test, Explanation, Post-Test. First, prior to the Practice stage, participants were introduced to the game including background information, game rules, and their task. Then in the Practice, participants played 2 practice games to get further acquainted with the specified domain. In Pre-Test, participants played 3 more games, this time being aware that their games were being scored. In Explanation, participants interacted with an expert player (well-trained RL agent) and were exposed to explanations about the player's actions via their assigned study condition. Note, participants in the "None" study condition did not receive any accompanying explanations during the Explanation stage. Specific to Connect 4, this stage consisted of observing the expert player (RL agent) play 3 games. In Lunar Lander, the Explanation stage consisted of observing the expert player (RL agent) play 1 game. In the Post-Test, participants played 3 more scored games, similar to the Pre-Test. After the Post-Test stage, participants completed a post-questionnaire survey that collected demographics data as well as their perceived task performance and their perceived utility of their explanation condition.
Some specifics unique to each domain include that for Connect 4, participants always played first, and their opponent player was a well-trained MuZero agent. Additionally, for Lunar Lander, to reduce the impact of learning effects, the starting position and initial force applied to the lander was randomized across all 8 games; however, the random combinations remained fixed across participants. Also, participants played the version of Lunar Lander implemented by OpenAI Gym [4].
E.2 Participant Information
Connect 4 We recruited 84 participants from Amazon Mechanical Turk, for an IRB-approved study and participants were required to be novice players of Connect 4. Of the 84 participants, 9 were filtered out for not finishing the study. The remaining 75 participants, 15 per study condition, included 40 males and 35 females, all over the age of 18 (M=31.57, SD=7.12). The study took on average 20 minutes, and participants were compensated $5.00.
Lunar Lander We recruited 118 participants from Amazon Mechanical Turk, for an IRB-approved study. Participants were required to be novice players of Lunar Lander. Of the 118 participants, 13 were filtered out for not finishing the study or demonstrating no effort. The remaining 105 participants, 15 per study condition, included 50 males and 55 females, all over the age of 18 (M=30.57, SD=6.75). The study took on average 20 minutes, and participants were compensated $5.00.
E.3 Additional Evaluation Details
Participant ATS Note, the adjusted task score (ATS) adjusts participant's Post-Test average task performance by their Pre-Test average task performance. Note, this adjustment does not change any of the results or statistical analyses and is purely for visualization purposes. Specifically, we perform the adjustment to visualize the relative changes among participant ATS using a common starting point.
Figure 14: Analysis of perceived performance Perf shows participants across all conditions believed their performances improved after exposure to the expert player and assigned explanations.
Figure 15: Analysis of perceived explanation utility  $EU$  shows participants across all explanation conditions believed their assigned study condition improved task understanding.
Additionally, to keep the ATS graphs legible, we present the mean and standard deviations for each study condition here, as opposed to including error bar lines on the graphs. The following are the mean and standard deviations in the Connect 4 user study conditions. Specifically, (1) None (Baseline):  $\mathbf{M} = -0.039$ ,  $\mathrm{SD} = 0.222$ , (2)  $\mathcal{E}_A$  (Baseline):  $\mathbf{M} = -0.041$ ,  $\mathrm{SD} = 0.114$  (3)  $\mathcal{E}_V$  (Baseline):  $\mathbf{M} = -0.104$ ,  $\mathrm{SD} = 0.346$  (4)  $\mathbf{GT}\ \mathcal{E}_{C_i}$ :  $\mathbf{M} = 0.256$ ,  $\mathrm{SD} = 0.318$  (5)  $\mathbf{S2E}\ \mathcal{E}_{C_i}$ :  $\mathbf{M} = 0.215$ ,  $\mathrm{SD} = 0.194$ . The following are the mean and standard deviations in the Lunar Lander user study conditions. Specifically (1) None (Baseline):  $\mathbf{M} = -0.051$ ,  $\mathrm{SD} = 0.191$ , (2)  $\mathcal{E}_A$  (Baseline):  $\mathbf{M} = -0.049$ ,  $\mathrm{SD} = 0.136$  (3)  $\mathcal{E}_V$  (Baseline):  $\mathbf{M} = -0.067$ ,  $\mathrm{SD} = 0.211$  (4)  $\mathbf{S2E}\ \mathcal{E}_{C_i}$ :  $\mathbf{M} = -0.043$ ,  $\mathrm{SD} = 0.192$ , (5)  $\mathbf{S2E}\ \mathcal{E}_{C_iw/TeG}$ :  $\mathbf{M} = 0.100$ ,  $\mathrm{SD} = 0.154$  (6)  $\mathbf{S2E}\ \mathcal{E}_{C_iw/InF}$ :  $\mathbf{M} = 0.034$ ,  $\mathrm{SD} = 0.190$ , (7)  $\mathbf{S2E}\ \mathcal{E}_{C_iw/InF,TeG}$ :  $\mathbf{M} = 0.156$ ,  $\mathrm{SD} = 0.182$ .