# Decision-Making in Fallback Scenarios for Autonomous Vehicles: Deep Reinforcement Learning Approach
## 6 Training and Evaluation
### Metric and Training Results  
The metrics considered for the fallback capacity were the three optimal heuristic maneuvers derived in Section 3.2: slow following, lane change, and lane change after yielding. We evaluated the decision-making algorithm by comparing the training results from heuristic maneuvers. If the training results converge to a particular heuristic maneuver, the ego vehicle successfully learns the optimal policy to respond appropriately to the corresponding fallback scenario. The proposed decision-making algorithm was trained 100 times with the same hyper-parameters. The training was performed in a single synchronous environment, and the time required to complete the training was measured and presented. This was conducted to generalize the training results based on the statistical evaluation. The 100 results were converged into five groups: (1) lane changes, (2) slow following, (3) rear-end collisions, (4) front-end collisions, and (5) side collisions. Therefore, lane changes and slow following were successful results of the training fallback technique, and the three collisions were considered failures. Figure 5 presents the quantitative training results for each episode's total reward and maximum Q-value. The Q-value is related to the loss function and is presented as an indicator for evaluating the learning process and policy results. 1. _Lane change_ involved the ego vehicle moving from the current lane to the right lane, which occurred 38 of 100 times (38%). In Figure 5a,b, the average of 38 training results and each training result are depicted in red and shaded red, respectively. The total reward converged within the range of 470-480 after 200 episodes. The maximum Q-value increased monotonically. Moreover, each training time required an average of 2.5 h for 500 episodes. Figure 4: Learning environment of fallback scenario. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Parameter** & **Ego Vehicle** & **Vehicle A** & **Vehicle B** \\ \hline \(x\) (m) & 1.00 & 2.00 & 0.00 \\ \(y\) (m) & 0.15 & 0.15 & \(-\)0.15 \\ \(\phi\) (rad) & 0.00 & 0.00 & 0.00 \\ \(\dot{x}\) (m/s) & - & 0.05 & 0.15 \\ \(\dot{\phi}\) (rad/s) & - & 0.00 & 0.00 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Initial conditions of ego vehicle, Vehicles A and B. 2. _Slow following_ involved the ego vehicle following the front slow vehicle, which occurred 9 out of 100 times (9%). In Figure 5a,b, the averages of the nine training results and each training result are depicted in green and shaded green, respectively. The total reward converges to 370-380 after 200 episodes. The maximum Q-value increases slightly. Moreover, each training time required an average of 8.5 h for 500 episodes. 3. _Rear-end collision_ involved the front of Vehicle B colliding with the rear of the ego vehicle, which occurred 25 of 100 times (25%). In Figure 5a,b, the average of the 25 training results and each training result are depicted in blue and shaded blue, respectively. The total reward converged in the range of 120-130 after 100 episodes. The maximum Q-value increased slightly. 4. _Front-end collision_ involved the front of the ego vehicle colliding with the rear of Vehicle A, which occurred 17 of 100 times (17%). In Figure 5a,b, the average of the 17 training results and each training result are depicted in cyan and shaded cyan, respectively. The total reward converged in the range of 100-110 after 100 episodes. The maximum Q-value increased slightly and was indistinguishable from the rear-end collision results. 5. _Side collision_ involved the side of the ego vehicle colliding with the front of Vehicle B, which occurred 11 of 100 times (11%). In Figure 5a,b, the average of the 11 training results and each training result are depicted in magenta and shaded magenta, respectively. The total reward converged within the range of 30-40 after 200 episodes. The maximum Q-value decreased slightly. Thus, the _lane change_ maneuver was the optimal fallback maneuver that achieved its purpose and provided the most efficient (speed) results.