# Mobile Charging Scheduling Approach for Wireless
## 3 System Model and Problem Formulation
### Problem Formulation
Then, the JSSTC problem is formulated as  
\[\begin{array}{ll}max&T_{life}(\Pi)\\ &C1:\quad\quad\theta_{k}=\{0,1,\ldots,n\},k=\{1,\ldots,K\},n,K\in\mathbf{N}^{* }\\ &C2:\quad\quad\varsigma_{k}=\{0.1,0.2,\ldots,1\},k=\{1,\ldots,K\},K\in\mathbf{N} ^{*}\\ s.t.&C3:\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad  
the action to be performed in the next time step. Then, MC interacts with the environment based on the action predicted by deep learning. At any time step, the environment gives feedback based on the actions performed by the MC, which is defined as a reward. Figure 2 shows the flow chart of making charging decisions based on the DRL. The exploration process can be described by the tuple \(\{S,\mathcal{A},R,S^{\prime}\}\), where \(S\) is the state space of the system, \(\mathcal{A}\) is the action space, \(R\) is the reward feedbacked by the environment, and \(S^{\prime}\) is the new state space after the MC performing the action. The details of the state information, action, policy, and reward are described below.  
(1) State.  
The state information of this paper mainly consists of three parts, including the state of the sensors, MC, and the charging service station, described with \(s(k)=(s_{sn}(k),s_{MC}(k),s_{d}(k))\), which can help the MC determine the charging destination. The state information of each sensor is described with a tuple \(s_{sn_{i}}(k)=\left(x_{sn_{i}},y_{sn_{i}},e_{sn_{i}}^{c},E_{sn_{i}}^{r},E_{sn _{i}}^{MAX}\right)\); the state information of MC is described with \(s_{MC}(k)=\left(x_{MC},y_{MC},e_{MC}^{m},p_{MC}^{c},E_{MC}^{max},E_{MC}^{min},E _{MC}^{r},E_{MC}^{r}\right)\); and the state information of charging service station is described with \(s_{d}(k)=(x_{d},y_{d})\). The elements in the state information can be divided into static elements and dynamic elements. Therefore, in the above state information, \(e_{sn_{i}}^{c},E_{sn_{i}}^{r}\), \(x_{MC},y_{MC}\), and \(E_{MC}^{r}\) are dynamic elements; the others are static.  
(2) Action.  
We define that at any time step, the action \(a(k)\) consists of discrete action pairs, which are the charging destination at next time step \(a_{\theta}(k)\) and the correspond ding upper threshold \(a_{\zeta}(k)\), described with \(a(k)=(a_{\theta}(k),a_{\zeta}(k))\). For the convenience of sorting, we assume that the index of the charging service station is 0. The MC needs to start from the charging service station and return to the charging service station to replace the battery. Therefore, the action \(a_{\theta}(k)\) determined at time step \(k\) is an integer number \(a_{\theta}(k)\in\mathcal{A}_{\theta}=\{0,1,\ldots,n\}\), and the action \(a_{\zeta}(k)\) determined at time step \(k\) is a float number \(a_{\zeta}(k)\in\mathcal{A}_{\zeta}=\{0.1,0.2,\ldots,1\}\), described in Figure 3.  
(3) Transition.  
After the MC performs the charging action, the state of the network will change from \(s(k)\) to \(s(k+1)\). The energy consumption of the MC and the sensors can be obtained by Equations (4) and (5). It should be noted that MC can only observe the updated state information, and the underlying process is simulated in the base station.  
Figure 2: The flowchart of making charging decisions based on the DRL.  
(4) Reward.  
We utilize \(r(s(k),a(k))\) to describe the reward the agent gets after performing action \(a(k)\) at state \(s(k)\). Since the main purpose of this paper is to maximize the survival time of the network before the termination condition is met, this is equivalent to maximizing the duration of MC performing charging tasks. The charging task performed by MC includes charging and moving. Therefore, the sum of the moving time and the charging time corresponding to each time step is regarded as the reward of MC, which can be obtained by (1) and (3). In addition, the increased number of failed sensors due to charging decision will worsen the charging performance. Thus, we regard the number of new failed sensors as a penalty. Therefore, the reward \(r(s(k),a(k))\) is obtained by  
\[r(s(k),a(k))=t_{\text{MC}}^{m}(k)+t_{\text{MC}}^{c}(k)-\phi\cdot(n_{d}(k)-n_{d}( k-1)), \tag{7}\]  
where \(\phi\) is the penalty coefficient.