## 3 Methodology
### A Centralized Routing Protocol for Lifetime and Energy Optimization Using GA and LSPI
The Q-values of the state-action pair are approximated using Equation (23)  
\[\hat{Q}(s,a)=w_{1}EC_{max}(s,a)+w_{2}EC_{sum}(s,a)=w\varphi(s,a)^{T} \tag{23}\]  
where \(w_{1}\) and \(w_{2}\) are the weights associated to the basic functions \(EC_{max}(s,a)\) and \(EC_{sum}(s,a)\), respectively, \(w\) is the weight matrix of size \(2\times 1\) and \(\varphi(s,a)^{T}\) is the transpose of the basic function matrix of size \(1\times 2\). The weight, \(w\) in the Equation (23) is approximated using a set of samples \(\mathcal{D}=\left\{(s_{i},a_{i},r_{i},\delta_{i})|i=1,2\ldots,M\right\}\) as given in Equation (24). \[\hat{w}=\hat{X}^{-1}\hat{y} \tag{24}\]  
The set of samples \(\mathcal{D}\) is obtained by taking a random action in a given state. The algorithm used to generate the random samples by the sink is given in Algorithm 3. ```
0: List of RTs, Maximum number of samples, \(|\mathcal{D}|_{max}\), Number of iterations, \(N\)
0: Set of samples, \(\mathcal{D}\)
1:\(\mathcal{D}=\{\}\)
2: Initialize a random RT as \(s_{0}\)
3:for\(i=1\) to \(N\)do
4:\(s_{i}=s_{o}\)
5: The agent Choose a random RT, \(a_{i}\)
6: The agent receives a reward, \(r_{i}\) using Equation (27)
7:\(s_{i}\gets s_{o}\)
8:\(\delta_{i}=s_{o}\)
9:if\((s_{i},a_{i},\delta_{i},r_{i})\
otin\mathcal{D}\)then
10:\(\mathcal{D}\leftarrow(s_{i},a_{i},\delta_{i},r_{i})\)
11:endif
12:if\(|\mathcal{D}|=|\mathcal{D}|_{max}\)then
13:break
14:endif
15:endfor
16:Return \(\mathcal{D}\)
```  
**Algorithm 3** Samples Generation Algorithm. Assuming \(\hat{X}^{0}=0\) and \(\hat{y}^{0}=0\) initially, Equations (25) and (26) will give the approximated values of \(\hat{X}^{i+1}\) and \(\hat{y}^{i+1}\), respectively, of a new sample \((s_{i},a_{i},r_{i},\delta_{i})\)  
\[\hat{X}^{i+1}\leftarrow\hat{X}^{i}+\varphi(s_{i},a_{i})\left[\varphi(s_{i},a_{i })-\gamma\varphi(\delta_{i},\underset{a\in A}{\text{argmin}}(\delta_{i},a)) \right]^{T} \tag{25}\]  
where \(\gamma\) is the discount factor. Figure 2:0  
The reward obtained is modeled as in Equation (27)  
Figure 2:1  
where \(EC_{v}\) is the energy consumption by the \(v\)th sensor node. The sink evaluates the \(EC_{v}\) after each learning round using the difference between the formerly estimated sensor residual energy \(ESRE_{v}^{Previous}\) and the later estimated sensor residual energy, \(ESRE_{v}^{Current}\). Therefore \(EC_{v}\) is as given in Equation (28). Figure 2:2  
The network lifetime is maximized while minimizing the energy consumption of the WSN by learning the RT that minimizes the Q-value using the epsilon-greedy technique. Given a number, \(z\in(0,1)\) randomly generated in each learning round and a probability epsilon value, \(\epsilon\in[0,1]\), the agent selects its action in each learning round with the policy given in Equation (29). \[a_{t}= \begin{cases}\underset{a}{\text{argmin}}(\varphi(s,a)^{T}w),\ if\ z<1-\epsilon\\ \text{Random action, otherwise}.\end{cases}  \tag{29}\]  
The designed CRPLEOGALSPI for learning the optimal RT to maximize network lifetime and minimize the network energy consumption is given in Algorithm 4. The novelty of the proposed protocol is in the use of the LSPI, which tends to converge faster to the best RT in solving the objective function subject to constraints when compared to Q-learning.