# A Trusted Routing Scheme Using Blockchain and Reinforcement Learning for Wireless Sensor Networks
## 3 Approach
### RLBC Routing Algorithm  
In this part, we elaborate the reinforcement learning-based routing scheduling scheme by using the global, dynamic, and trusted routing information provided by the proposed blockchain platform. The information obtained by the learning model from the blockchain network includes the timestamp \(t_{i}\) of the token transaction (all transactions represented by array \(T_{t}\)), the transfer amount \(n_{i}\) of tokens for each hop routing node (all amounts represented by array \(N_{t}\)), the amount of remaining tokens, and the address array \(A_{t}\) passed by. The specific reinforcement learning and blockchain based (RLBC) algorithm is summarized in Algorithm 2. Environment \(E\) represents a real routing environment, where the routing nodes update their states and make decisions about actions. State \(x\) represents the current position of the packets that need to be transferred, i.e., the packets locate currently at the \(x\)-th routing node. It means that the learning model has \(n\) states if there are \(n\) routing nodes. Policy \(\pi(x)\) is the action \(a\) to be taken from the current state \(x\) and the action \(a\) represents forwarding the packets to the next-hop routing node \(R_{a}\). It means that when a state the routing node has \(m\) next hop nodes to choose from, the current state has \(m\) actions. Therefore, the action space \(\mathcal{A}(x)\) represents the collection of all actions at routing node \(i\). For example, \(R_{1}\) has two and \(R_{2}\) has three actions to choose from: \(\mathcal{A}(1)=\{a_{1},a_{2}\}\), \(\mathcal{A}(2)=\{a_{1},a_{2},a_{3}\}\) (according to the actual situation, the \(a_{1}\), \(a_{2}\) in the two arrays are not necessarily the same). Note that \(q_{t}(x,a_{k})\) is the differential queue length between \(R_{x}\) and \(R_{a_{k}}\) that \(q_{t}(x,a_{k})=B_{R_{x}}-B_{R_{a_{k}}}\). The size of this metric affects the probability that \(R_{x}\) transmits the packets to \(R_{a_{k}}\). The larger this metric is, the greater the probability will be. Therefore, if a routing node releases a lower queue length, it can increase the probability that other routing nodes transmit packets to it. ```
1:Environment \(E\); Action Space \(\mathcal{A}\) ; Initial State \(x_{0}\); Reward Discount \(\gamma\); Learning Rate \(\alpha\);
2:Policy \(\pi\);
3:\(Q_{t}(x,a)=0\), \(P(x,a)=\frac{1}{|\mathcal{A}(x)|}\);
4:\(x=x_{0}\);
5:for\(T=1,2,...\)do
6:\(a=\pi^{p}(x)\);
7:\(r=\) reward by routing action \(a\);
8:\(x^{\prime}=\) next state by routing action \(a\);
9:\(a^{\prime}=\pi(x^{\prime})\);
10:\(Q_{t}(x,a)=Q_{t}(x,a)+\alpha(r+\gamma Q_{t}(x^{\prime},a^{\prime})-Q_{t}(x,a ))\);
11:\(\pi(x)=\arg\max_{a_{k}}q_{t}(x,a_{k})\cdot Q_{t}(x,a_{k})\);
12:\(x=x^{\prime}\);
13:endfor
```  
**Algorithm 2** Reinforcement learning and blockchain based (RLBC) routing algorithm. In the core Equation (1) at line 8 of Algorithm 2, \(Q_{t}\) represents the Q-table of the packets sent to the routing node \(R_{t}\). When the packets reach the next routing node \(R_{\pi}\) from current routing node \(R_{i}\), it enters the next state. The action is moving the packets to the next-hop routing node \(R_{\pi}\). \[Q_{t}(x,a)=Q_{t}(x,a)+\alpha(r+\gamma Q_{t}(x^{\prime},a^{\prime})-Q_{t}(x,a )); \tag{1}\]  
Parameter \(0\leq\alpha\leq 1\) is the learning rate of the Q-learning. Where \(\alpha\) equals to one, means that the learning algorithm ignores the initial \(Q_{t}\). The reward discount \(\gamma\) represents the specific gravity of the next state. The larger its value is, the greater the influence of the next state on the current \(Q_{t}\) will be. The reward value \(r\) is an important parameter to motivate the routing nodes to make better routing decisions. Its value is determined by the routing information obtained by the learning model from the blockchain. The information includes the timestamp \(t_{i}\), the amount \(n_{i}\) of transferred tokens, and the address array \(A_{t}\).