# Decision-Making in Fallback Scenarios for Autonomous Vehicles: Deep Reinforcement Learning Approach
## 4 Fallback Scenario Formulation
### Reward for Achieving the Goal  
An ego vehicle must be designed to reach the goal line without collision. A reward function was defined to solve the fallback scenario. The reward function adequately provided success, failure, and transfer rewards to drive safety. The reward for success was reaching the destination without colliding with nearby vehicles. A reward is a key component that affects learning; a bad reward design results in a bad policy or even divergence of learning. Rewards can be classified as episodic, transitional, or conditional. An episodic reward was provided when learning episodes were complete. When an episode was completed while achieving the goal, the agent was awarded a positive reward of +100; otherwise, the agent did not receive that reward. A transitional reward was awarded at each conditional time step during an episode. This depends on the design and can be constant or variable. Furthermore, both rewards can be awarded simultaneously. _Reward_\(r\) was formulated to encapsulate a tripartite sub-reward (\(r_{1},r_{2},r_{3}\)), as expressed in Equation (3). \[r=\sum_{j=1}^{3}r_{j}, \tag{3}\]  
The first sub-reward \(r_{1}\) is a sparse reward awarded only when a goal is achieved, as expressed in Equation (4). This reward is a strong motivation for the ego vehicle to achieve a goal because it provides many rewards simultaneously compared to subsequent rewards. Specifically, when the ego vehicle successfully navigated to the target location without colliding, a large reward of +100 was awarded. Conversely, nothing (zero reward) was awarded if the ego vehicle failed and the current episode ended. \[r_{1}= \begin{cases}+100,&\text{if }x_{OE}\geq x_{OG}\\ 0,&\text{otherwise}\end{cases} . \tag{4}\]  
Here, \(x_{OE}\) and \(x_{OG}\) denote the current and target locations of the ego vehicle, respectively. The second sub-reward \(r_{2}\) is a positive and dense reward awarded at every time step after the interaction, as described in Equation (5). This sub-reward serves as a directional indicator, revealing whether the current behavior positively or negatively influences goal attainment. It offers vital navigational insights by progressively rewarding the agents as they approach their desired outcomes. For example, if the ego vehicle starts at 1.0 m and the position of \(n_{k}\)-step is 3.5 m, the reward is 100 \(\times\) (3.5 \(-\) 1.0) = 250. In the next step \(n_{k+1}\), if the position of the ego vehicle is 2.5 m, the reward is 100 \(\times\) (3.5 \(-\) 1.0) = 150. When comparing the rewards of steps \(n_{k}\) and \(n_{k+1}\), it can be concluded that this is a bad case because the position of Step \(n_{k+1}\) is lower than that of Step \(n_{k}\). \[r_{2}=100\times(x_{OE}-x_{OE,i}), \tag{5}\]  
where \(x_{OE,i}\) is an initial position of the ego vehicle and is a constant value. The third sub-reward, \(r_{3}\) was a negative, dense reward awarded at every time step after the interaction, as described in Equation (6). This sub-reward imposes a time penalty to increase the efficiency of the learning process, leading to faster optimal learning. An ego vehicle can reach a target point at low speed, as a practical example. However, this policy was inefficient in terms of time consumption. Therefore, the ego vehicle must achieve its goal at the highest speed to reduce the penalty and increase the total reward. \[r_{3}=-1\times n_{step}, \tag{6}\]  
where \(n_{step}\) is the number of steps in an episode. The expected maximum total reward, according to Equations (3)-(6), was computed as 480. When the ego vehicle reaches the goal point, the maximum value of the first sub-reward is +100. The maximum value of the second sub-reward was 400, where the driving distance \(x_{EC}\) was 400 from the initial position of ego vehicle \(x_{OEj}\), and 1.00 m to the goal position of \(x_{OG}\) 5.00 m. To obtain the maximum total reward, the third sub-reward must be minimized. Then, the minimum value of the third sub-reward was \(-20\) when the ego vehicle drove a distance of 4.00 m with a maximum velocity of 0.2 m/s.