## 4 Design of the DRFTM
### Trust Update Based on Deep Reinforcement Learning
#### 4.3.3 Process of Deep Reinforcement Learning  
In this paper, the sliding time window is used to store the trust characteristics of sensor nodes, and the trust state of nodes is determined by random forest. Additionally, the deep reinforcement learning model can record the update strategy of previous trust updates, determine the best trust update strategy according to the update strategy of previous trust updates, the trust evidence score of sensor nodes, and the trust state of nodes, and choose action \(A_{t}\) to maximize the reward \(R_{t}\) according to the optimal strategy. The following details the deep reinforcement learning parameters used in the DRFTM:  
* State: the deep reinforcement learning model predicts the state of the environment at the time point based on the random forest algorithm. States are defined as \[S_{t}=\left\{\phi_{t-1};T^{c},T^{d},T^{cn},T^{cn},T^{rec};C_{P}\right\},\] (22) where \(\phi_{t-1}\) is the smoothing factor of the previous trust update strategy, \(\left\{T^{c},T^{d},\right.\)\(\left.T^{en},T^{enc},T^{rec}\right\}\) is the trust evidence score of the sensor node, and \(C_{P}\) is the node trust state determined by the random forest according to the result of the last round of trust update combined with the current trust evidence.
* Action: the action of deep reinforcement learning is to select the smoothing factor \(\phi\), that is, the trust update strategy, and it is expressed as \[A_{t}=(\phi_{1},\phi_{2},\ldots,\phi_{m}),\] (23) where \((\phi_{1},\phi_{2},\ldots,\phi_{m})\) is the \(m\) smoothing factor of trust renewal strategy.
* Reward: reward is defined as the detection accuracy of the trust state of the node determined by the trust model at a given time, which is defined as \[R_{t}=\frac{m_{t}}{M_{t}},\] (24) where \(m_{t}\) is the number of malicious nodes detected by the trust model and \(M_{t}\) is the actual total number of malicious nodes.  
When the Sink node collects a new round of trust evidence, that is, a new state \(S_{t}\) appears in the network, a trust update strategy, represented as an action \(A_{t}\), is chosen based on the new trust evidence and the trust state of the node determined by the trust evidence. Given the state, the action \(A_{t}\) is determined by the optimal trust update policy \(\pi\). This action \(A_{t}\) is then executed, resulting in a reward \(R_{t}\) obtained from the environment. The environment operates according to a Markov decision process (MDP) and transitions to a new state \(S_{t+1}\), in accordance with the MDP rules. The goal of reinforcement learning is to maximize the expected long-term discounted rewards by identifying the best policy. Q-values can be utilized to find the optimal policy and can be updated using the following formula:  
\[Q_{t+1}(s_{t},a_{t})=(1-\alpha)Q_{t}(s_{t},a_{t})+\alpha[R+\gamma\text{max}Q_{ t}(s_{t+1},a_{t+1})], \tag{25}\]  
where \(\alpha\) is the learning rate. The best strategy can be expressed as  
\[\pi^{*}=\underset{a\in A}{\text{argmax}}Q^{*}(s,a). \tag{26}\]  
DQN uses neural network \(Q(s,a;w)\) to approximate \(Q^{*}(s,a)\), takes state as input, and outputs Q value estimation for each action. Relating inputs are approximated to relating outputs using weight or coefficients. The objective is to minimize the error by adjusting the network's weights through gradient descent. First, the weights of the deep Q network coefficients are randomly initialized. Over time, the deep Q-network updates its weights based on the difference between the expected reward and the true reward. In each iteration of the deep Q-network, the objective is to minimize the loss function as follows:  
\[L(w)=\sum_{(s_{t},a_{t})}(y_{t}-Q(s_{t},\ a_{t};\ w))^{2}, \tag{27}\]  
where \(Q(s_{t},\ a_{t};\ w)\) is the predicted value and \(y_{t}\) is the estimated value of the actual value, which can be expressed as  
\[y_{t}=R_{t}+\gamma\underset{a_{t}\in A}{\text{max}}Q(s_{t},\ a_{t};\ w), \tag{28}\]  
where \(R_{t}\) is the corresponding reward and \(\gamma\) is the discount factor.