# Decision-Making in Fallback Scenarios for Autonomous Vehicles: Deep Reinforcement Learning Approach
## 2 Background  
The proposed decision-making algorithm is primarily based on reinforcement learning. Recently, naive reinforcement learning has utilized artificial neural networks of supervised learning as a plugin to overcome the typical drawbacks (discrete) and achieve synergy through collaboration. Various advanced deep reinforcement learning algorithms have been developed, for example, deep deterministic policy gradient (DDPG), trust region policy optimization (TRPO), proximal policy optimization (PPO), etc. These advanced algorithms demonstrated outstanding results in various applications. However, we implemented the proposed algorithm as a deep Q-network (DQN), which is outdated but considered adequate for our research for the following reasons. The DQN is a neural network that renders the state space continuous and the action space discrete. Furthermore, it has a lower computational strength than the above algorithms.  
Reinforcement learning is a machine-learning algorithm that learns optimal decision-making patterns based on sequential interaction data between agents and the environment. The aim is to determine the optimal policy \(\pi^{*}\). In the current state \(s\) with the optimal policy, the agent selects the optimal action \(a\) that is expected to yield the largest total reward \(Q(s,a)\) in the future.  
\[\pi^{*}(s)=\underset{a}{\text{argmax}}Q(s,a),\]  
where \(Q(s,a)\) denotes the state-action value function. A state-action value function that adopts an artificial neural network is called deep reinforcement learning. Because the artificial neural network contains the weights, we denote it again as \(Q(s,a;\theta)\) and refer to it as the Q-network. As the interaction continued, more accurate and diverse driving data containing collision and/or collision-free data were utilized to train the Q-network. Therefore, the Q-network network exhibits higher values.  
Sequential interaction data for learning the Q-network were collected through the interaction between the agent and the environment. The agent receives State \(s\) from the environment and outputs action \(a\). The environment receives the behavior of the agent and outputs the next state \(s^{\prime}\) and reward \(r\). Through one interaction, data \(d\) are collected as \((s,a,s^{\prime},r)\).  
The training data for the Q-network were such that the input data were the state \(s\), and the output data were the temporal difference (TD) target \(r+\gamma\text{max}Q(s^{\prime},a^{\prime};\Theta)\), calculated from the interaction data where \(a^{\prime}\) is an arbitrary next action.  
The Q-network is an approximation function that minimizes the optimization problem by estimating training-output data. The loss function of the Q-network is expressed as  
\[L(\theta)=\frac{1}{2}\big{[}\big{\{}r+\gamma\text{max}Q\big{(}s^{\prime},a^{ \prime};\mathbf{\omega}\big{)}\big{\}}-Q(s,a;\Theta)\big{]}^{2}.\]  
The gradient is as follows:  
\[\
abla_{\theta}L(\theta)=\big{[}\big{\{}r+\gamma\text{max}Q\big{(}s^{\prime}, a^{\prime};\Theta\big{)}\big{\}}-Q(s,a;\theta)\big{]}\
abla_{\theta}Q(s,a; \Theta).\]  
It is updated with a batch and determines the minimum such that stochastic gradient descent is utilized to minimize the objective function and gradient value.  
\[\Theta^{\prime}=\Theta+\alpha\
abla_{\Theta}L(\theta),\]  
Reinforcement learning algorithms repeat learning until an optimal policy is found.  
The remainder of the detailed implementation-associated background, such as hyperparameters, is explained in Section 5.