# Decision-Making in Fallback Scenarios for Autonomous Vehicles: Deep Reinforcement Learning Approach
## Appendix C  
In this section, the initial iteration of the interaction is calculated with Equations (1)-(9) and the equations in the background section. The initial information provided by the AVs before the calculations is presented in Table  
The initial actions of the ego vehicle were randomly selected. The \(a_{E}\) is \(a_{7}\) which turns right at a slow speed. According to Action \(a_{7}\), the ego vehicle moves toward the right driving lane \(-0.15\) m at a velocity of \(0.15\) m/s. Vehicles A and B exhibit constant actions \(a_{4}\) and \(a_{6}\), respectively. At all times, Vehicle A drove in the left driving lane \(0.15\) m at a velocity of \(0.10\) m/s, and Vehicle B drove in the right driving lane \(-0.15\) m at a velocity of \(0.20\) m/s.  
\[a_{E}=a_{7},a_{A}=a_{4},a_{B}=a_{6}.\]  
For the \(r_{1}\) reward function, \(r_{1}=0\) because the ego vehicle has not yet reached its goal point.  
\[r_{1}=\begin{cases}+100,&\text{if }x_{OE}\geq x_{OG}\\ 0,&\text{otherwise}\\ &=0.\end{cases}\]  
For the \(r_{2}\) reward function, \(r_{2}\) is \(0.70\) as per the following calculation.  
\[\begin{array}{l}r_{2}=100\times(x_{OE}-x_{OE,i})\\ =100\times(x_{EG}+x_{OG}-x_{OE,i})\\ =100\times(-3.9930+5.0000-1.0000)\\ =100\times 0.0070\\ =0.70,\end{array}\]  
where the current position of the ego vehicle \(x_{OE}\) is \(x_{EG}+x_{OG}\) because \(x_{EG}\) is \(x_{OE}-x_{OG}\), the \(x_{EG}\) is \(-3.9930\); the initial position of the ego vehicle \(x_{OE,i}\) is \(1.0000\) m; and the \(x_{OG}\) is \(5.0000\) m.  
For the \(r_{3}\) reward function, \(r_{3}\) is \(-1\) because only one step progressed, that \(n_{step}\) is \(1\).  
\[\begin{array}{l}r_{3}=-1\times n_{step}\\ =-1\times 1\\ =-1\end{array}\]  
For the integration of the total reward function, \(r\) is \(-0.30\) as per the following calculation.  
\[\begin{array}{l}r=\sum\limits_{j=1}^{3}r_{j}\\ =r_{1}+r_{2}+r_{3}\\ =0+0.70-1\\ =-0.30.\end{array}\]  
Depending on the progress made thus far, move on to the next state \(s^{\prime}\).  
\[s^{\prime}=(-3.9621,0.1500,-0.0076,-0.9706,0.0000,-0.0076,1.0222,0.3000,-0.0 077).\]  
The one-step interaction data \((s,a,r,s^{\prime})\) are recorded in the reply buffer. The current state \(s\) is updated with \(s^{\prime}\).  
\[s=s^{\prime}.\]The next action was selected utilizing the following equations. The value of the Q function was calculated and updated as follows:  
\[\begin{array}{c}h_{1}=f_{h_{1}}\big{(}W_{h_{1}}\cdot s+b_{h_{1}}\big{)}\\ h_{2}=f_{h_{2}}\big{(}W_{h_{2}}\cdot h_{1}+b_{h_{2}}\big{)}\\ Q(s,a)=f_{\text{Q}}\big{(}W_{\text{Q}}\cdot h_{2}+b_{\text{Q}}\big{)}.\end{array}\]  
We then apply epsilon greed, where exploration and exploitation are determined using the epsilon greed rate \(\varepsilon\). The initial exploration and exploitation ratio is 0.99. This implies a 99% exploration probability and a 1% exploitation probability. The exploration and exploitation rates decrease as the stages progress. During the exploration, an action was selected randomly. If exploitation is selected, policy \(\pi\) outputs an action \(a\) for the maximum value of Q.  
\[\pi(s)=\underset{a}{\text{argmax}}Q(s,a).\]  
When as many interaction data as the number of batch sizes are collected in the replay buffer, the Q function is later learned by updating the weight bias according to the following formula:  
\[\begin{array}{c}L(\theta)=\frac{1}{2}\big{[}\{r+\gamma\text{max}Q(s^{\prime },a^{\prime};\mathbf{\omega})\}-Q(s,a;\theta)\big{]}^{2}\\ \
abla_{\theta}L(\theta)=\big{[}\{r+\gamma\text{max}Q(s^{\prime},a^{\prime}; \theta)\}-Q(s,a;\theta)\big{]}\
abla_{\theta}Q(s,a;\theta)\\ \theta^{\prime}=\theta+\alpha\
abla_{\theta}L(\theta)\end{array}\]  
The input and output data obtained by repeating Steps 1-10 are summarized in Table 11. This interaction is repeated until the end of an episode.