# A Reliability Quantification Method for Deep Reinforcement Learning-Based Control
## 4 Reliability Quantification
### Reliability Quantification Method  
In the previous section, issues related to the application of RND to uncertainty evaluation were addressed. Thus, an improved method suitable for evaluating reliability was proposed. To evaluate reliability, two networks, a reference network, and an evaluator network, were used. Reliability was obtained by calculating the absolute error between the reference and evaluator networks, and a softsign function was applied to the absolute error to scale a range from 0 to 1. The reliability value was calculated using Equation (17), and the calculation flow of the reliability is shown in Figure 6.  
The scheme is similar to that of RND, but there are some differences. In RND, the target and predictor networks can have different structures, and their initial parameters are determined randomly; however, the reference and evaluator networks should have the same structure and initial parameters. Because the reference and evaluator networks are the same neural networks before training, their outputs are the same if the same input date is provided. This ensured that the reliability value reached zero before training. During the training process, the parameters of the reference network were fixed, and those of the evaluator network were updated to maximize the reliability value. Thus, the loss function can be described using Equation (18), where \(Ref(s|\theta_{r})\), \(Evalu(s|\theta_{e})\), \(\theta_{r}\), and \(\theta_{e}\) are the reference network, the evaluator network, the parameters of the reference network, and the parameters of the evaluator network, respectively. The shape of the loss function is shown in Figure 7.  
\[Loss_{improve}(\theta_{e})=\mathbb{E}[-\text{softsign}(|Ref(s|\theta_{r})-Evalu (s|\theta_{e})|)] \tag{18}\]  
The gradient of the loss function decreased as reliability increased. Therefore, the larger the reliability value, the more difficult it is to change. This means that sufficiently trained experience fades slowly and is similar to human memory.  
Furthermore, because the trained model is largely affected by recent training data, the reliability of past trained situations should decrease step by step. In this regard, the parameters of the evaluation network were updated to minimize the reliability of irrelevant  
Figure 6: Reliability calculation flow.  
training data, \(S_{irr}\), and irrelevant data were generated randomly. The loss function that considers the forgetting experience is described by Equation (19), where \(s_{irr}\) is a randomly generated irrelevant state.  
\[Loss_{forget}(\theta_{e})=\mathbb{E}[\text{softsign}(|Ref(s_{irr}|\theta_{r})- Evalu(s_{irr}|\theta_{e})|)] \tag{19}\]  
Finally, the loss function for reliability quantification is described as Equation (20).  
\[Loss_{reliability}(\theta_{e})=Loss_{improve}(\theta_{e})+Loss_{forget}(\theta_{e}) \tag{20}\]  
The regularization term used to prevent overtraining is defined in Equation (21), where \(p\) and \(\lambda\) are a dimension and a power of regularization, respectively. This restricted the distance of the parameters between the evaluation and reference networks.  
\[\lambda\frac{1}{p}|\theta_{e}-\theta_{r}|^{p} \tag{21}\]