# Analysis of Explainable Goal-Driven Reinforcement Learning in a Continuous Simulated Environment
## 3 Methods and Proposed Architecture
### Introspection-Based Method  
[PERSON] et al. [26] proposed Equation (3) to compute the probability of success according to the introspection-based method. This solution consists of estimating the probability using the Q-value through a mathematical transformation, so no additional memory is needed. In the equation, \(\sigma\) is a parameter for stochastic transitions (in this case 0). \(Q^{*}(s,a)\) is the estimated Q-value. \(R^{T}\) is the total reward obtained at the terminal state. \([...]^{P_{S}\leq 1}_{P_{S}\geq 0}\) is the rectification used to restrict the value between \([0,1]\). For continuous environments, a normalization of the results was proposed instead of limiting it with the equation. The change is illustrated in Equation (4) where \(\vec{p}_{S}\) relates to the normalization of a probability and \(\hat{p}\) computes the probability as shows in Equation (3) without the rectification. Algorithm 2 shows an adaptation of the introspection-based method in addition to the DQN algorithm [29]. In the algorithm, the notation used is as in Algorithm 1. The calculation for the probability of this method was carried out in line 17. \[\hat{P_{S}}\approx\left[(1-\sigma)\cdot(\frac{1}{2}\cdot\log\frac{Q^{*}(s,a)}{ R^{T}}+1)\right]^{\hat{P_{S}}\leq 1}_{\hat{P_{S}}\geq 0} \tag{3}\]  
\[\hat{P_{S}}\approx\hat{p_{S}}\in\hat{p} \tag{4}\] \[\hat{p_{S}}=\frac{\hat{p}-\min(\hat{p})}{\max(\hat{P})-\min(\hat{ P})}\] \[\hat{p}=(1-\sigma)\cdot(\frac{1}{2}\cdot\log\frac{Q^{*}(s,a)}{R^ {T}}+1)\]  
Figure 1: Diagram of the neural network used to calculate the Q-values using the DQN algorithm and the probability of success using the learning-based method. When computing the probability of success, the neural architecture used a sigmoid activation function in the last dense layer in order to restrict the output values between 0 and 1. Three consecutive \(96\times 96\) gray images of the car racing game are used as inputs. ```
1:Initialize \(Q\) and \(\mathbb{P}\) functions
2:Initialize \(\hat{Q}\) and \(\hat{\mathbb{P}}\) function objectives
3:Initialize memory \(D\) with long \(N\)
4:for episode 1 to M do
5: Initialize state queue \(S\) with initial state
6:repeat
7:\(\epsilon\leftarrow\) update \(\epsilon\) with \(\epsilon\)-decay
8:\(s_{t}\leftarrow\) dequeue current state from \(S\)
9: Select an action \(a_{t}\) according to \(s_{t}\) using policy \(\epsilon\)-greedy
10: Take action \(a_{t}\), observe reward \(r_{t}\), and next state \(s_{t+1}\)
11: Enqueue next state \(s_{t+1}\) into \(S\)
12: Store transition \((s_{t},a_{t},r_{t},s_{t+1},done_{t})\) in \(D\)
13: Take a random sample of transitions \((s_{j},a_{j},r_{j},s_{j+1},done_{j})\) from \(D\)
14:\(y_{j}=\left\{ \begin{array}{ll}r_{j}&\text{if }s_{t+1}\text{ is final}\\ r_{j}+\gamma\max_{a^{\prime}}\hat{Q}(s_{j+1},a^{\prime})&\text{otherwise} \end{array} \right.\)
15: Update \(Q\) with respect to \(y_{j}\)
16:\(y_{pj}=\left\{ \begin{array}{ll}r_{j}&\text{if }s_{j+1}\text{ is final}\\ r_{j}+\gamma\max_{a^{\prime}}\hat{\mathbb{P}}(s_{t+1},a^{\prime})&\text{otherwise} \end{array} \right.\)
17: Update \(\mathbb{P}\) related to \(y_{pj}\)
18: Each C steps, \(\theta_{\hat{Q}}\leftarrow\theta_{\hat{Q}}\) y \(\theta_{\mathbb{P}}\leftarrow\theta_{\mathbb{P}}\)
19:until\(s_{t}\) is terminal or \(r_{t}\) had been negative 25 times
20:endfor
21:Return \(Q\), \(\mathbb{P}\)
```  
**Algorithm 1** Explainable goal-driven learning approach to calculate the probability of success using the learning-based method.