## 3 The Multi-Agent Framework
### Framework Design
#### 3.4.1 Cooperative Neighbour Set Creation  
The creation of a cooperative neighbour set is based on the sensors' historical information. We use an example to describe the creation procedure. A sensor, \(v_{j}\), often relays packets for \(v_{i}\). Then, \(v_{j}\) may want itself to be \(v_{i}\)'s cooperative neighbour. \(v_{j}\) then sends a request message to \(v_{i}\) to suggest \(v_{i}\) add \(v_{j}\) as one of \(v_{i}\)'s cooperative neighbours. The request message includes the information regarding \(v_{j}\) itself, such as the remaining energy of \(v_{j}\), the normalised effective sensing area of \(v_{j}\), the distance between \(v_{j}\) to the sink node and the average number of hops from \(v_{i}\) to \(v_{j}\) (that is derived from the previously relayed packets, each of which contains a hop-count field). Once \(v_{i}\) receives this request packet, \(v_{i}\) will evaluate the reward of adding \(v_{j}\) as one of \(v_{i}\)'s cooperative neighbours. \(v_{i}\) estimates that if it adds \(v_{j}\) as a cooperative neighbour, \(v_{i}\)'s storage consumption may increase and \(v_{j}\)'s energy consumption may rise (because in the future, \(v_{i}\) will send more packets to \(v_{j}\) than before). Such a consumption is a negative reward. On the other hand, however, in the future, the packet routing delay could be reduced, because the routing procedure becomes more targeted. Furthermore, the energy consumption of other sensors could be reduced, since packets would be relayed by fewer sensors. The storage and energy consumption can be estimated using Equations (1), (2) and (4). The estimation of packet routing delay is based on two factors: the communication radius of a sensor, \(r\), and the distance between sensor node \(v_{i}\) and sink node \(s\), which is recorded as \(d_{v_{i}\to s}\). If there is more than one sink in the sensor network, such a distance demonstrates the shortest distance. In this study, it is assumed that each sensor has the same communication radius, \(r\). The distance between sensor nodes to the sink node can be obtained during the initialisation stage of a sensor network. Once the two factors, \(r\) and \(d_{v_{i}\to s}\), are known, the estimation of packet routing delay can be obtained using Equation (6), where \(\delta\) is a constant coefficient, which is greater than one. \[del=\delta\cdot\frac{d_{v_{i}\to s}}{r} \tag{6}\]  
The reason for using \(r\) and \(d_{v_{i}\to s}\) to calculate the estimation of the packet routing delay is that the result of using \(d_{v_{i}\to s}\) to divide \(r\) can approximately indicate the number of hops from sensor \(v_{i}\) to sink \(s\). Since a packet transmitted between two neighbouring sensors needs one time slot, the number of hops from sensor \(v_{i}\) to sink \(s\) can approximately reflect the number of time slots needed to transmit a packet from \(v_{i}\) to \(s\), namely the packet delay from \(v_{i}\) to \(s\). Hence, if \(v_{i}\) wants to add \(v_{j}\) as one of its cooperative neighbours, the estimated increase of storage consumption is \(S^{(2)}\cdot\hat{t}_{ij}^{(3)}\), where \(\hat{t}_{ij}^{(3)}\) indicates the intended time slots during which \(v_{i}\) keeps \(v_{j}\) as one of its cooperative neighbours; the estimated increase of energy consumption on \(v_{j}\) is \(\zeta\cdot Ene_{R_{j}}(k)\), where \(k\) denotes the total size of messages (measured in bits) passed from \(v_{i}\) to \(v_{j}\) and \(\zeta\) is a constant coefficient, which is greater than one; the estimated routing delay reduction is \(\zeta\cdot(del_{i}-del_{j})\); the estimated reduction of energy consumption on other sensor nodes is \(\zeta\cdot hop_{i\to j}\cdot(Ene_{T}(k,\frac{r}{\delta})+Ene_{R}(k))\). After the estimation, the reward of \(v_{i}\), namely \(\mathcal{R}_{i}\), is derived by summing up the four estimated values. Here, the four estimated values can be assigned different weights in different situations to demonstrate their different importance levels.