# A Reliability Quantification Method for Deep Reinforcement Learning-Based Control
## 4 Reliability Quantification  
### Applicability of Random Network Distillation  
In RND, an intrinsic reward is provided when an agent experiences an unknown state, and exploration of the unknown state is encouraged. Thus, the value of the intrinsic reward was used as the value of inadequacy to the state, and RND was used for reliability evaluation. A neural network consists of multiple layers, and features are extracted from each layer. Because the output of each layer may be changed significantly by smaller changes in the input data it cannot be possible to extract similar features from similar states. Thus, an uncertainty evaluation for extracted features is more appropriate than one for the input data. Therefore, the uncertainty of the features extracted before the last hidden layer was evaluated. The structure of the neural network is defined according to RND, as shown in Figure 2. The neural network comprises two parts: a DQN network and an RND network that evaluates the uncertainty of the extracted features. The DQN network consists of five fully connected layers (FC). The numbers of nodes in these layers are 256, 256, 128, 128, and 9, respectively. The activation functions in the DQN are the ReLU function in the hidden layers and the linear function in the output layer. The target network and predictor network consist of five FCs, with their numbers of nodes being 512, 512, 256, 128, and 15, respectively. The activation functions in the hidden layers of the RND networks and output layers are a softsign function and linear function. Any DRL method can be applied by setting up a neural network instead of a DQN. During training, the parameters of the neural networks that evaluate uncertainty are updated to minimize the output values. The parameters of the neural network predicting the Q values were updated to minimize the loss function of the DQN. To ensure that reliability quantification does not affect the DRLs, these updated processes are independent. The training hyperparameters are listed in Table 1.  
After training until the loss values converged, the trained model was evaluated in the simulation by varying the initial position within the ranges shown in Equation (16). To evaluate the action of the agent and the uncertainty of the learning environment, the ranges were wider than those in the learning environment, and the episode did not end if the agent exited the learning environment. \[\left(\mathbf{x_{evalu}},\mathbf{y_{evalu}}\right)\in\left\{\left(\mathbf{x},\mathbf{y} \right)\middle| \begin{pmatrix}\mathbf{x}\in\left\{-\mathbf{700},-\mathbf{630},\ldots,\mathbf{63 0},\mathbf{700}\right\}\\ \mathbf{y}\in\left\{-\mathbf{700},-\mathbf{630},\ldots,\mathbf{630},\mathbf{700}\right\}\end{pmatrix}  \right\}\right\} \tag{16}\]The evaluation trajectories are shown in Figure 3. The unfilled and filled red markers indicate the agent's initial and last positions, respectively, and the shade of color indicates the time history. As shown in Figure 3, the agent can reach the goal regardless of where the initial position is set within the learning environment. If the initial position is set outside the environment the agent cannot reach the goal. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Learning steps & 20,000,000 [steps] \\ Batch size & 1024 \\ Learning rate & 0.001 \\ Target model update & 0.01 \\ Optimizer & Adam \\ L1 regularization & 0.0001 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Hyperparameters for training. Figure 3: Trajectories of the trained model (RND). Figure 2: Structure of the neural networks considering RND. Figure 4 shows the distribution of the uncertainty. The figure shows the average uncertainty values when the agent passed through the points. During the training process, the agent acted randomly in the early steps and experienced a wide range of states in the environment. At the end stage of training, the agent moves to the given goal from its initial point on the shortest course.