## Appendix D BEERL on Other Environments
### Cartpole from OpenAI Gym  
In this environment, a simple control problem is simulated where a pole on a cart needs to be balanced so it can stand vertically by adjusting the cart in a horizontal direction. The state of this environment consists of the following: cart position(_pos_c_), cart velocity(_vel_c_), pole angle(_w_p_), and pole angular velocity(_\(\theta\_p\)_). Since the cart is moving horizontally, there are two possible actions that push the cart to the left or right. The original reward was calculated by giving the \(+1\) reward in each time step when the pole is upright. Otherwise, zero rewards will be given if the pole falls. However, to accommodate the decomposition of the reward, we modify the reward calculation as follows:  
* Cart position reward: \(R_{c\_pos}(s)=1.0-|\frac{pos\_c}{approx\_c}|\);
* Pole angle reward: \(R_{p\_\theta}(s)=1.0-|\frac{\theta\_p}{approx\_c}|\);
* Total reward: \(R_{total}(s)=k_{c\_pos}\times R_{c\_pos}(s)+k_{p\_\theta}\times R_{p\_\theta}(s)\). We set the reward weight as 0.2 and 0.8 for the cart position reward (\(k_{c\_pos}\)) and pole angle reward (\(k_{p\_\theta}\)), respectively. The higher weight for the pole angle reward is given because it is the main task of this environment. In this manner, the agent is trained not only to keep the pole upright but also to position the cart at the center of the work area. The global explanations of the agent in this environment are shown in Figure 11. At the Q-function level, all input features other than the position of the cart (_pos_c_) have higher contributions to the position of the cart (_c_pos_) than the pole angle's (_p_\(\theta\)_) reward component. However, after applying the reward prioritization, the pole angle reward component dominates the contribution at the RL level. We can also see that the cart and pole angular velocity's (_vel_c_ and _\(\omega\_p\)_) input features are the two most important features. Both have significant contributions to both the cart position (_c_pos_) and pole angle (_p_\(\theta\)_) reward components. When we compare the contribution of all input features and the reward components, as shown in Figure 12, both the velocity input features (_vel_c_ and _\(\omega\_p\)_) and the _p_\(\theta\)_ reward component are the most contributing elements in the trained agent. For local explanations, we can see from Figure A4 that, in that particular situation, opposing contributions are made by the cart position (_pos_c_) and velocity (_vel_c_) input features to both reward components. This information may not be obtained if we use the existing method (feature importance and reward decomposition) exclusively. Figure A5 presents the contrastive explanation in which the agent chooses the push-right action instead of push-left. The sum contribution of '_\(\omega\_p[I]\) to c_pos[R]_' and '_\(\omega\_p[I]\) to p_\(\beta\)' is enough to outweigh the disadvantage of not choosing the push-left action. This explanation is further compressed by Algorithm 1 to present that only the contribution of \(\omega\_p\) input feature is enough to outweigh the disadvantage of the same situation. Figure 17 presents the base values of each action and reward component for both in normalized and weighted values. In contrast to the antenna tilt use case, we want this agent to prioritize the angle of the pole more than the position of the cart. Thus, after applying the reward weight, we can see that the pole angle has higher base values than the cart position reward component. We want the cart position and velocity input feature to contribute more to the cart position reward component. Additionally, all input features other than cart position should have a significant impact on the pole angle reward component.