## 3 The Multi-Agent Framework
### Framework Design
#### 3.4.2 Selecting a Cooperative Neighbour for Relaying  
When a sensor, say \(v_{i}\), has a packet to transfer, it needs to decide which cooperative neighbour is the best choice for relaying the packet. The selection of a cooperative neighbour is based on each cooperative neighbour's properties, which are embodied in the previous request message (As time progresses, the information embodied in the request message may be outdated. Thus, if a cooperative neighbour's status significantly changes, it will send a new message to \(v_{i}\) regarding its current status.). These properties include the remaining energy of \(v_{j}\), the normalised effective sensing area of \(v_{j}\), the distance between \(v_{j}\) to the sink node and the average number of hops from \(v_{i}\) to \(v_{j}\). Generally, sensor \(v_{i}\) prefers to forward packets to a cooperative neighbour, which has more remaining energy, because such a cooperative neighbour is not likely to run out of energy. \(v_{i}\) also prefers a small normalised effective sensing area, because even if such a cooperative neighbour is depleted, the overall coverage area will not be greatly affected, since the sensing area of that cooperative neighbour is overlapped by multiple sensors. A short distance from \(v_{j}\) to the sink and a small number of hops from \(v_{i}\) to \(v_{j}\) are also preferred, as a short distance and a small number of hops usually imply little transmission delay. Therefore, for \(v_{i}\), there are different benefits for choosing different cooperative neighbours to relay a packet. Such a benefit, for \(v_{i}\) choosing \(v_{j}\), can be calculated using Equation (8). \[\mathcal{B}(v_{j})=\gamma_{1}\cdot Rem_{E}ng(v_{j})-\gamma_{2}\cdot\eta(v_{j}) -\gamma_{3}\cdot d_{v_{j}\to s}-\gamma_{4}\cdot hop_{i\to j}, \tag{8}\]  
where \(Rem_{E}ng(v_{j})\) means the remaining energy of \(v_{j}\), \(\eta(v_{j})\) indicates the normalised effective sensing area of \(v_{j}\), \(d_{v_{j}\to s}\) is the distance from \(v_{j}\) to the sink node, \(hop_{i\to j}\) denotes the number of hops from \(v_{i}\) to \(v_{j}\) and \(\gamma_{1}\), \(\gamma_{2}\), \(\gamma_{3}\), \(\gamma_{4}\) are coefficients that are used to normalise the four values in the same magnitude. After calculation of the benefit for selecting each cooperative neighbour, sensor \(v_{i}\) employs another Q-learning algorithm to select a cooperative neighbour to forward the packet (Algorithm 2). In Line 1, \(v_{i}\) initialises the Q-value of each cooperative neighbour as zero. \(\mathbf{\pi}\) is a vector, which demonstrates the probability with regard to selecting each cooperative neighbour to transfer the packet, _i.e._, \(\mathbf{\pi}\)\(=\)\(\langle\pi(v_{1}),...,\pi(v_{|N^{\prime}_{1}|})\rangle\). \(v_{i}\) initially sets each element in \(\mathbf{\pi}\) to \(\frac{1}{|N^{\prime}_{i}|}\) for simplicity. Then, in Lines 2-4, \(v_{i}\) updates the Q-value of each cooperative neighbour based on the benefit of selecting each cooperative neighbour, \(\mathcal{B}(v_{j})\), where \(\alpha_{2}\) is the Q-learning rate. \(v_{i}\) averages the benefits of selecting its cooperative neighbours and uses this average value and the newly created Q-values to update the probability vector \(\mathbf{\pi}\), where \(\tau\) is the policy learning rate (Lines 5-8). The probability vector update method was developed by [PERSON] _et al._[38]. In Line 9, the function \(Normalise()\) is used to constrain \(\mathbf{\pi}\) to a legal probability distribution, _i.e._, \(\sum_{j=1}^{|N^{\prime}_{1}|}\pi(v_{j})=1\), and \(Normalise(\mathbf{\pi})\) has the minimum Euclidean distance to \(\mathbf{\pi}\). Finally, in Line 10, \(v_{i}\) selects a cooperative neighbour to transfer the packet based on the probability vector \(\mathbf{\pi}\). This selection is akin to the roulette, namely that cooperative neighbour \(v_{j}\) with a large value of \(\pi(v_{j})\) is more likely to be selected. Algorithm 2 is a direct policy search algorithm that learns stochastic policies.