# An On-Demand Charging for Connected Target Coverage in WSRNs Using Fuzzy Logic and Q-Learning +
## 4 Fuzzy Q-Charging Algorithm
### State Space, Action Space and \(Q\) Table  
In our Q-learning-based model, the network is considered the environment while the MC is the agent. A state is defined by the current charging location of the MC, and an action is a move to the next charging location. Each MC maintains its own Q-table, which is a two-dimensional array. Each row represents a state, and each column represents an action. An item \(Q(D_{j},D_{i})\) in the \(j\)-th row and \(i\)-th column represents the Q-value corresponding to the action when the MC moves from the current charging square \(D_{j}\) to the next charging location \(D_{i}\). Figure 4 shows an illustration of our Q-table. In the figure, the gray row represents the Q-values concerning all possible actions when the MC stays at the charging location \(D_{c}\). The green cell depicts the maximum Q-value regarding the next charging location.  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Notation** & **Definition** \\ \hline \(D_{i}\) & The \(i\)-th charging location \\ \hline \(Q(D_{i},D_{j})\) & Action value of the action moving from \(D_{i}\) to \(D_{j}\) \\ \hline \(r(D_{i})\) & Reward obtained after the MC moves to \(D_{i}\) \\ \hline \(\alpha\) & The learning rate of the Q-learning algorithm \\ \hline \(\gamma\) & The discount factor of the Q-learning algorithm \\ \hline \(E_{th}\) & The threshold for sending a charging request \\ \hline \(E_{sf}\) & The safe charging level \\ \hline \(\theta\) & The safe energy factor \\ \hline \(E_{max}\) & The maximum energy capacity of the sensors \\ \hline \(p_{j}^{i}\) & The per second energy that a sensor \(S_{j}\) is charged when the MC stays at \(D_{i}\) \\ \hline \(e_{j}\) & Energy consumption rate of sensor \(S_{j}\) \\ \hline \(E_{j}\) & Remaining energy of sensor \(S_{j}\) \\ \hline \(\mathcal{T}_{i}\) & The optimal charging time at \(D_{i}\) \\ \hline \(L_{\tau},E_{min}\) & Fuzzy input variables \\ \hline \(w_{j}\) & The priority index of \(S_{j}\) \\ \hline \(\xi_{j}\) & The energy severity index of \(S_{j}\), \(\xi_{j}=\frac{e_{j}}{E_{j}}\). \\ \hline \hline \end{tabular}
\end{table}
Table 1: List of notations.  
Figure 3: The flow of Fuzzy Q-learning-based charging algorithm.  
Let \(D_{c}\) be the current charging location and \(D_{i}\) be an arbitrary charging location, then the Q-value of action moving from \(D_{c}\) to \(D_{i}\) is iteratively updated by using the Bellman equation as follows:  
\[Q(D_{c},D_{i})\gets Q(D_{c},D_{i})+\alpha(r(D_{i})+\gamma\max_{1\leq j\leq l }Q(D_{i},D_{j})-Q(D_{c},D_{i})). \tag{4}\]  
The equation's right side consists of two elements, including the current Q-value and the temporal difference. The temporal difference measures the gap between the estimated target, i.e., \(r(D_{i})+\gamma\max_{1\leq j\leq l}Q(D_{i},D_{j})\), and the old Q-value, i.e., \(Q(D_{c},D_{i})\). \(\alpha\) and \(\gamma\) are two hyper-parameters whose names are learning rate and discount factor, respectively. \(r(D_{i})\) is our proposed reward function, which will be detailed in Section 4.5.  
In the following, we first describe our algorithms to determine the optimal charging time and the safety energy level in Sections 4.3 and 4.4. Then, we present the details of the reward function and the mechanism for updating the Q-table in Sections 4.5 and 4.6.