# Mobile Charging Scheduling Approach for Wireless
## 3 System Model and Problem Formulation
### The Details of MDDRL-JSSTC  
The training process of MDDRL-JSSTC is shown in Algorithm 1, and its structure is shown in Figure 5. The core framework of MDDRL-JSSTC is the deep Q-network (DQN); therefore, the parameterized Q-network and parameterized target Q-network can be defined as \(Q(s(k),a(k);\theta)\) and \(Q(s(k+1),a(k+1);\theta)\). The convolutional neural network (CNN) is widely utilized as Q-network in DQN because of its excellent nonlinear mapping ability. In this paper, the charging sequence and charging upper threshold included in the charging decision have complex time characteristics. CNN lacks the ability to store time-series information and cannot predict future decisions based on historical information. The recurrent neural network (RNN) has good information memory ability, and the input of its hidden layer is not only related to the output of the previous layer but also related to the output of this layer in the previous layer of time step. Therefore, RNN accurately represents the complex temporal correlation between the current input sequence and the past information. However, traditional RNNs have a drawback that they can only store short-term temporal memories. When too long a temporal series is an input in RNN, its gradient will explode or disappear, making the neural network invalid. The gated recurrent unit (GRU) is a variant of RNN which can better capture the dependencies of large intervals in temporal data, thus effectively alleviating the problems of gradient explosion and gradient disappearance. The input and output structure of GRU is the same as that of traditional  
Figure 4: A scheduling example of JSSTC. RNN, mainly composed of four parts, the input at the \(k\)-th time step \(x(k)\), the hidden layer state at \(k-1\) time step \(h(k-1)\), the output at the \(k\)-th time step \(y(k)\), and the hidden layer state transferred to the next time step \(h(k)\). The structure of the GRU is shown at the top of Figure 6. It mainly includes two gates: the reset gate and the update gate. The reset gate determines how much memory is saved and resets, and the update gate determines how much memory is retained to be updated to the current gate unit. At any time step, the GRU is updated based on the following:  
\[u(k)=\sigma(w_{u}\cdot[h(k-1),x(k)]), \tag{9}\]  
\[r(k)=\sigma(w_{r}\cdot[h(k-1),x(k)]), \tag{10}\]  
\[\overset{\sim}{h}(k)=\tanh(w_{h}\cdot[r(k)*h(k-1),x(k)]), \tag{11}\]  
\[h(k)=(1-u(k))*h(k-1)+u(k)*\overset{\sim}{h}(k), \tag{12}\]  
where \(\sigma\) is the sigmoid function, described as \(\sigma=1/(1+e^{-x})\). \(\sigma\) can convert the data into a value from 0 to 1 and then act as a gate signal. tanh is the activation function, expressed as \(\tanh=(e^{x}-e^{-x})/\left((e^{x}+e^{-x})\right.\). tanh can scale the data to the range \(-1\) to 1. Furthermore, \(w_{u}\), \(w_{r}\), and \(w_{h}\) are the weights that need to be updated, respectively. Since the charging decision at any time step in this paper includes the charging destination and the corresponding charging upper threshold, it can be analyzed that both the past and the future charging decisions impact the current charging decision. To find a charging policy that maximizes the network lifetime, we introduce the bidirectional gated recurrent unit (Bi-GRU) in Q-network, which can predict the charging decision of the current time step based on historical and future information. Bi-GRU consists of a forward and backward GRU, as described in Figure 6. At any time step, the outputs of forward GRU and backward GRU are \(h_{f}(k)\) and \(h_{b}(k)\), respectively. Figure 5: The structure of MMDRL-JSSTC. The process of obtaining the Q value in MDDRL-JSSTC is briefly described as follows:  
1. The state information of any time step \(s(k)\) is embedded to obtain a high-dimensional vector, and this vector is utilized as the input of Bi-GRU, denoted as \(x(k)\);
2.