## Appendix D BEERL on Other Environments
### Cartpole from OpenAI Gym
We can see that pos_y input feature contributes negatively to several reward components and at the RL level it significantly contributes to the position and velocity reward component. This explanation makes sense because the lander is still far from the moon's surface. Furthermore, we can also see that the angle (\(\theta\)) input feature contributes negatively to the angle (\(\theta\)) reward component because the lander is rotated. The explanation of when the lander has landed on the moon (Figure A6 position C) is shown in Figure A9b. At the Q-level explanation, we can see that the contributions of the leg contact input features that contribute to the respective leg reward components are significant compared to others. However, at the RL-level, the contribution of pos_y input feature to the position reward component is the highest due to reward prioritization, while the contributions of the respective legs elements become the second and third. Figure A9: The local explanations of the Lunarlander of three different states as shown in Figure A6. (**a**) The local explanations of the Lunarlander when the agent chooses fire_right action on position A in Figure A6. (**b**) The local explanations of the Lunarlander when the agent chooses fire_left action on position C in Figure A6. When comparing two actions, the contrastive explanation shows the benefit of choosing an action over the other, as shown in Figure A10. By applying the MSX concept, the contributions of _\"pos_x[I] to pos[R]\", \"pos_x[I] to vel[R]\", \"vel_y[I] to vel[R]\"_, and _\"c_leg1[I] to vel[R]\"_ are enough to choose fire_right action and outweigh the disadvantage of not choosing no_fire action. Furthermore, our automatic MSX compression (Algorithm 1) reduces the length of the explanation to only present the contributions of position and velocity reward components to be presented to the user. The base value of each pair of action and reward components is shown in Tables A6 and A7, where the former is calculated at the Q-level while the latter is at the RL-level. We can see that the range at the RL-level is significantly larger than the Q-level because of the reward prioritization. The base value of the position reward component has the highest value, which is relevant to this environment where the main goal is to position the lander to the origin. All above explanations are detailed and the focus value can be used to summarize whether the explanations fulfill the desired properties. For this environment, the relevance table is set as shown in Table A8 which generates the focus value shown in Table A9. The position component (\(R_{pos}\)) is highly correlated with the position input features (\(pos\_x,pos\_y\)). Similarly, other pairs are also highly correlated where we set the relevance value as 1 such as: velocity component (\(R_{vel}\)) with speed (\(speed_{h},speed_{v},speed_{a}\)); angle component (\(R_{angle}\)) with angular speed (\(speed_{a}\)) and angle input feature; and leg components (\(R_{leg1}\) and \(R_{leg2}\)) with leg input feature (\(leg_{1}\) and \(leg_{2}\), respectively), etc. On the other hand, the main engine (\(R_{main-eng}\)) should not consider the angular speed (\(speed_{a}\)) because changes in the angular speed cannot be corrected by the action of the main engine; therefore, it has zero relevance value. We can see that the position reward component has the highest focus value, and \(pos\_y\) contributes significantly higher than the others. The velocity (\(R_{vel}\)) reward component has a low value because \(pos\_y\) has a higher contribution than \(vel\_x\), \(vel\_y\) or the angular velocity (\(\omega\)), which ideally is the focus of this component. Similarly, the angle reward component has a low value because \(vel\_x\) and \(pos\_y\) are the second and third most contributing feature instead of the angular velocity (\(\omega\)). Overall, the weighted mean has a lower value than the unweighted mean focus value, which means that high priority components (i.e., velocity and angle) do not focus on the desired input features.