# Device-to-Device (D2D) Multi-Criteria Learning Algorithm Using Secured Sensors
## 3 Proposed Multi-Criteria Learning Algorithm Using Secured Sensors
### D2D Authentication with Multi-Criteria Reinforcement Learning
To compute this, the source node distributes \(n\) number of probes' packets in a fixed time interval, \(t\), and as a result, the neighboring node \(j\) determines the value of \(lcost_{i,j}\) for node \(i\), as defined in Equations (4) and (5):  
\[lcost_{i}=\left(PRR_{(i,j)}\ +\frac{1}{ave_{dtime}}\right)+1/d_{err} \tag{4}\]  
\[ave_{dtime}=\frac{\left(p_{n}\ -p_{i}\ \right)}{\text{t}} \tag{5}\]  
where \(p_{n}\) and \(p_{i}\) denote the reception time for the first and last probe packets, \(t\) is the given time interval, and \(d_{err}\) is the data error, used to measure the number of retransmissions. The proposed algorithm utilizes reinforcement learning [33] for computing and selecting the routing states using network conditions and experiences. The reinforcement algorithm is comprised of agents, states, S, and a set of actions, A, per state. Using reinforcement learning, node \(i\) exploits the \(R(i)\) values and selects the next hop using energy, speed, radio coverage, and link cost metrics. On receiving the data, the next-hop performs the re-computation of the \(R(i)\) value and forwards the data through its selected routing states. This process is continued for each neighbor selection until network data are received at the sink node. Additionally, when device \(i\) needs to route the data at the time \(t_{0}\), it performs a set of actions and selects the neighbor node based on the computed route rank. The value of route rank is dynamically changed by evaluating the network and nodes' statistics. Later, the device \(i\) gains a reward, \(Rwd\), and enters the next state, i.e., \((S,\ a,\ Rwd)\). A node has only a single reward value at any time. If any node has no reward value at any moment, then it will not be allowed to participate in the routing phase. On entering into the next state, the device \(i\) updates its forwarding table by adding the value of the reward. Moreover, the preceding device retrieves the updated information of device \(i\). This practice of reinforcement learning is exploited by the proposed algorithm for finding the most optimal routes for forwarding the IoT data towards the sink node. At the end of the learning period, the entries of forwarding tables are converged to a numeric value that indicates the optimal route from the source device to the sink node. Converged forwarding tables with computation of route rank not only decreases the unnecessary data diverting but also increases the packets reception ratio over the communication channels in the existence of malicious nodes. Figure 2 illustrates the flow of reinforcement learning by exploiting the computed route rank. It uses the multi-criteria of the nodes to determine its rank value and accordingly assign the reward. Based on the updated forwarding tables and reward values the proposed algorithm offers convergence results and increases the route lifetime in terms of energy, speed, and link cost. The convergence levels depend on the number of iterations until end-to-end routes are established with the efficient distribution of constraint resources. Figure 3 shows the message flow for the selection of the next-hop between the source node and its neighbors. The source node floods the route request packet in its radio coverage and identifies the nearest neighbors. In a case when no reply has been received, then it resends the request packet. Once it has found the list of neighbors, then the process of data discovery is initiated, utilizing the node-level table to fetch the statistics. Based on the fetched data, the proposed algorithm computes the route rank using a multi-criteria process and the assigned reward value by exploiting reinforcement learning. Thus, selected nodes advertise their status for the connection in the routing phase, and sensors' data is forwarded to the sink node. Figure 3: Next-hop selection procedure. Figure 2: Route rank using reinforcement learning.