# A Reliability Quantification Method for Deep Reinforcement Learning-Based Control
## 6 Switching of Trained Model  
It was demonstrated that the proposed method could evaluate the reliability of the trained model described in the previous section. The reliability value becomes 0 in the untrained state and 1 in the well-trained state. In RND, because the range of uncertainty depends on the model, a comparison of the uncertainty between models is not possible. The proposed method ensured that the reliability ranges were identical, enabling the reliability  
Figure 16: Initial position where the agent achieves the goal.  
Figure 15: Comparison of distribution between uncertainty by RND and reliability by proposed model (untrained).  
of the models to be compared. Therefore, the effectiveness of switching trained models was demonstrated as an application method. The quality of the sampled training data is a major issue in DRL. However, it is difficult to solve this problem completely. One solution is to switch between the trained models. Using several trained models, the untrained states of each model can cover each other. However, it is difficult to set criteria for switching models. As a solution, the reliability value can be used as a criterion for the switching model. Therefore, the effectiveness of switching models is discussed in this section.  
Before demonstrating the switching models, four models with biased experiences were developed. The ranges of the initial positions were different for these models. The initial positions of the \(i\)-th model were determined randomly within the range expressed in Equation (23). The \(i\)-th model trained the states of the \(i\)-th quadrant. The reward and observations were the same as those described in the previous sections.  
\[\left(x_{init}^{i},y_{init}^{i}\right)\in\left\{(\text{d-cos}(\theta),d\text {-sin}(\theta))\right|\begin{pmatrix}200\leq d\leq 300\\ (\frac{7}{2})(i-1)\leq\theta<(\frac{7}{2})i\end{pmatrix}\right\}\,(i=1,\;2,\; 3,\;4) \tag{23}\]  
After training, all models were evaluated. The trajectories and distribution of the reliability are shown in Figures 17 and 18, respectively. The gray area in Figure 17 indicates the range of the initial states. According to the results, the trained models achieved their goals and exhibited high reliability within their trained quadrants.  
Figure 17: Trajectories of DQN models trained using biased experiences.  
Figure 18: Reliability distributions of DQN models trained using biased experiences.  
The trained models were then switched. In the simulation, the optimal actions and reliability values of all models were calculated at every time step. Then, the action of the agent with the maximum reliability was used as an action in the simulation. The results of the trajectories and reliability are shown in Figure 19. According to the evaluation results, the agent achieved the goal from all initial positions and exhibited high reliability over a wider area in the learning environment.  
In addition, the effectiveness of the model switching was validated by applying it to a DDPG-based control. The trajectories and reliability distributions of the DDPG-trained model using biased experience are shown in Figures 20 and 21. Then, the trajectories and reliability distribution of the switching models are shown in Figure 22. According to the evaluation results, the agent reached the goal from all initial positions and exhibited high reliability over a wider area.  
Through the validation of trained model switching applied to DQN-based and DDPG-based controls, it can be concluded that the performance of DRL-based controls is improved by adopting the switching model technique according to the proposed reliability quantification method.