## 3 Design of SROA
### SROA Protocol Description
Before data forwarding, the sender node will wrap the collection of candidate forwarders and the calculated suppressing time into the header. ```
1.Procedure Clustering(all nodes)
2.Get all nodes \(N=\{n_{1},\,\dots,\,n_{m}\}\) where \(m\) is the number of nodes;
3.Get all the locations of \(N\);
4.//Calculate the optimal \(k\)
5.Calculate the silhouette coefficient for \(N\);
6.\(k\) is the highest silhouette score;
7.Select centroid \(c_{1}\) randomly where \(c_{1}\in N\);
8.//cluster the network, apply \(k\)-means++
9.For\(j=1\); \(j<=k\); \(j+\)+
10.For\(i=1\); \(i<=m\); \(i++\)
11.Calculate distance between \(n_{i}\) and previously \(c_{i}\)
12.New centroid \(c_{i+1}\in N\) is selected with longer distance;
13.Endfor
14.Endfor
15.Assign \(n_{i}\in N\) to the nearest \(c_{j}\in C\) by \(k\)-means++;
16.//select CH and PCH for each cluster
17.For\(j=1\); \(j<k\); \(j++\)
18.Calculate the average energy \(Ea\) for \(Cl_{j}\);
19.For\(n_{i}\in Cl_{j}\) and \(Ei>E_{a}\)
20.Select CH and PCH by distance and residual energy;
21.Update the cluster status;
22.Endfor
23.End Procedure
```  
**Algorithm 1**The procedure of Clustering. **(4) Packet transmission**: When a node receives the packet, it will first check the package header. If the candidate set contains itself, it starts the waiting timer according to the fields in packet header. Then, the data forwarding repeats the above steps until the data packet is received by the sink node. Therefore, a complete routing path has been built. Then, the successive packets from the same source node are sent directly along the calculated routing path. When a transmission failure occurs, Q-Learning will run again and converge to alternative routes. The routing procedure of SROA protocol is described in Algorithm 2. ```
1.Procedure routing(node \(n_{i}\))
2.Initialize \(V(s)\);
3.Get \(E_{res}^{n_{i}}\), \(P_{b}^{n_{i}}\), and locations of Neighbor\({}_{i}\);
4.If (\(n_{i}\) 1 = sink node)
5.If (\(n_{i}\) is not a CH or PCH)
6.find the CH(head node) of \(n_{i}\);
7.transmit data packet to CH;
8.Return;

### Design of Reward Function

The reward function is a critical part of Q-Learning, so we go over the reward function in depth. The SROA adopts three performance indicators in the reward function to assess the action interacting with the environment, including remaining energy, network latency, and link quality, to make the protocol more energy-efficient and reliable. The Q-value calculated represents the quality of routing decisions. When taking action \(a_{j}\) successfully in the transmission, the reward is denoted by \(R_{s;s_{j}}^{a_{j}}\), which is defined as follows:

\[ \begin{array}{l}R_{s;s_{j}}^{a_{j}}=-R_{0}-\alpha_{1}*[c(en)+\alpha_{2}*(c( delay)-c(pdr))]\\ where\ \alpha_{1},\alpha_{2}\in(0,1)\end{array}  \tag{15}\]

The reward function takes constant cost, energy cost, delay cost, and link quality cost into account. Due to the occupation of the channel bandwidth during communication, \(R_{0}\) represents the constant cost. Hence, the constant cost increase with the number of relay hops. If the reward function only contains the constant cost, it will lead to selecting just the shortest path. Nevertheless, the shortest path is not always the best path owing to the imbalanced energy use and transmission reliability. As a result, additional factors, such as remaining energy, network latency, and package delivery, must be addressed. In addition, network latency and package delivery ratio are the indicators of transmission, so a link sensitivity factor denoting \(\alpha_{2}\) is introduced for balancing energy and to link the quality of the path. When the link sensitivity factor is set to zero, the selected path takes only the energy into account. As a result, the sensitivity factor in the formula is the weight assigned to the link cost. _c(en)_ denotes the energy-related cost.