## 3 The Multi-Agent Framework
### Framework Design
#### 3.4.1 Cooperative Neighbour Set Creation
For example, if in some situations, energy consumption is the most important, the energy consumption will be assigned the highest weight among these estimated values. In this paper, we do not consider special use cases. Thus, the four estimated values are assigned the same weight, _i.e._, one. Once the reward value, \(\mathcal{R}_{i}\), is obtained, sensor \(v_{i}\) has to make a decision regarding whether to add \(v_{j}\) as one of its cooperative neighbours. A Q-learning algorithm is developed for this decision making process (see Algorithm 1). Q-learning is one of the most-studied reinforcement learning (RL) algorithms and has been applied with success in several domains, from relatively simple toy problems, such as cliff-walking [30], to more complex ones, such as web-based education [31] and face recognition [32]. One of the attractive features of Q-learning is that it assumes no knowledge about the environment (such as state transition functions or reward functions). ```
1 Initialise \(Q\) value for each available action arbitrarily;
2 for\(k=0\) to a predefined integer do;
3 calculate \(\pi\);
4 for each available action \(a\in A_{i}\)do;
5\(Q_{k+1}(a)=Q_{k}(a)+\pi(a)\alpha_{1}(\sum_{a}\mathcal{R}(a)\pi(a)-Q_{k}(a))\);
6 endfor
7 endfor
8\(a_{opt}\gets argMax(Q)\);
9\(v_{i}\) takes the action \(a_{opt}\);
```  
**Algorithm 1**Learning progress of sensor \(v_{i}\)  
Sensor \(v_{i}\) first arbitrarily initialises the Q-value of each available action (Line 1). There are two available actions for sensor \(v_{i}\): whether or not to add \(v_{j}\) as one of its cooperative neighbours. Next, \(v_{i}\) launches the learning process for the Q-value of each available action (Lines 2-7). In Line 2, the number of iterations is set to \(30\), which can yield adequate results. In Line 3, \(\pi\) is calculated using Equation (7) for each available action, \(a\), which is the \(\epsilon\)-greedy exploration method [33], where \(0<\epsilon<1\) is a small positive number and \(n\) is the number of available actions of a sensor. The \(\epsilon\)-greedy exploration method defines a semi-uniform probability distribution, in which the current best action is selected with probability \(1-\epsilon\) and a random action is chosen with probability \(\epsilon\). \[\pi(a)= \begin{cases}(1-\epsilon)+(\,\epsilon/n\,),\text{if $Q(a)$ is the highest value}\\ \epsilon/n\,\,,\text{otherwise}\end{cases}  \tag{7}\]  
Line 5 demonstrates the Q-value update equation, where \(0<\alpha_{1}<1\) is the learning rate and \(\mathcal{R}(a)\) is the estimated reward received by \(v_{i}\) after taking action \(a\). After finishing learning, \(v_{i}\) executes the action that could maximise the Q-value (Lines 8-9). Finally, if \(v_{i}\) decides to add \(v_{j}\) as one of its cooperative neighbours, \(v_{i}\) will send an acknowledgement message back to \(v_{j}\) to let \(v_{j}\) know that \(v_{j}\) has been added to \(v_{i}\)'s cooperative neighbour set. Conversely, if \(v_{i}\) does not decide to add \(v_{j}\) as its cooperative neighbour, \(v_{i}\) just ignores \(v_{j}\)'s request message and does not send any message back in order to save energy consumption. Having this algorithm, the expected behaviour of sensors can be derived by tracking the actions with the highest Q-value over the learning process of each sensor. The importance of obtaining a Q-learning algorithm with \(\epsilon\)-greedy exploration is also justified through a large number of applications. For example, [PERSON] _et al._[34] applied a Q-learning algorithm with \(\epsilon\)-greedy exploration to develop a decentralised resource allocation mechanism; [PERSON] and [PERSON] [35] studied the problem of learning demand functions; and [PERSON] _et al._[36] investigated the development of bidding strategies.