# Mobile Charging Scheduling Approach for Wireless
## 3 System Model and Problem Formulation
### The Details of MDDRL-JSSTC
After inputting the input information to Bi-GRU, the hidden state information \(h(k)\) and output information \(y(k)\) of any time step will be obtained, where the hidden state information is spliced by forward hidden state information and backward hidden state information, expressed as \(h(k)=h_{f}(k)\oplus h_{b}(k)\);
3. Two fully connected layers are connected behind the Bi-GRU layer and then obtain the Q value. It should be noted that because the actions performed by MC at any time step are two-dimensional, the Q value here is a two-dimensional list. Furthermore, the weights mentioned above are included in the \(\theta\) and \(\theta\prime\), and the specific training process will be briefly described as follows. In MMDRL-JSSTC, the action value function is defined as \(Q^{\Pi}(s,a)\). The action value at \(k\)-th time step is obtained by  
\[Q^{\Pi}(s(k),a(k))=\mathbb{E}_{\Pi}\Big{[}\sum_{k=0}^{K}\gamma^{k}r(k)\Big{|} s(k),a(k)=a_{\theta}(k),a_{\varsigma}(k)\Big{]}, \tag{13}\]  
The action value function is updated according to the Bellman optimal equation, which can be expressed as  
Figure 4:0 Figure 4:1  
Q-network is updated by minimizing the loss function \(L_{i}(\theta_{i})\)  
Figure 4:2  
where \(a(k)\sim\rho(\cdot)\) represents the probability distribution of the action, and \(y_{j}\) is  
Figure 4:3  
Figure 4:4 The structure of the Q-network in MMDRL-JSSTC. The gradient of the loss function is defined as  
\[ \begin{split}\
abla_{\theta_{i}}L_{i}(\theta_{i})=\mathbb{E}_{s(k),a( k)\sim\rho(\cdot);s(k+1)}\big{[}(r+\gamma\cdot max_{x^{\prime}}Q(s(k+1),a(k+1); \theta_{i-1})-\\ Q(s(k),a(k);\theta_{i})\cdot\
abla_{\theta_{i}}Q(s(k),a(k);\theta_{i}) )\big{]}\end{split}  \tag{17}\]  
```
Input: Input the state information \(s\), the maximum number of episode is \(N_{c}\), maximum training step in one episode is \(K\), the target network parameter update interval is \(N_{u}\)-step, discount factor \(\gamma\) Output: \(\theta\) and \(\theta^{\prime}\)
1. Initialization: Initialize the capacity of experience replay buffer \(\mathbb{B}\) to \(\Xi\), initialize the weight of Q-network \(\theta\) randomly, \(\theta^{\prime}\longleftarrow\theta\)
2. for episode = 1 to \(N_{u}\)do
3. Initialize the network environment
4. for\(k\) = 1 to \(K\)do
5. Select an action \(a_{\theta}(k)\) from \(\mathcal{A}_{\theta}\), based on \(\varepsilon\)-greedy at \(k\)-th time step
6. Select an action \(a_{\varsigma}(k)\) from \(\mathcal{A}_{\varsigma}\), based on \(\varepsilon\)-greedy at \(k\)-th time step
7. Combine \(a_{\theta}(k)\) and \(a_{\varsigma}(k)\) to an action pair \(a(k)=(a_{\theta}(k),a_{\varsigma}(k))\)
8. \(\varepsilon=\varepsilon-\Delta\varepsilon\)
9. Perform the action \(a(k)\) on the environment, the environment feedback a reward \(r(k)\), and a new state \(s(k+1)\). 10. Save the interaction process as tuple \(\{s(k),a(k),r(k),s(k+1)\}\), and store it in \(\mathbb{B}\)
11. Repeat performing 5-10
12. Sample \(\{s(j),a(j),r(j),s(j+1)\}\) randomly from minibatch \(\Gamma\)
13. if any conditions in (6) is not satisfied at / + 1-th time step \(y(j)=r(j)\)
14. break. 15. else Obtain the two-dimensional action value \(y_{\theta}(j)\) and \(y_{\varsigma}(j)\) with (14) Update the parameters via (18)
16. end if
17. Update the parameter of target network with \(\theta^{\prime}\longleftarrow\theta\) every \(N_{u}\) steps
18. end for
19. end for
```  
**Algorithm 1****MDDRL-JSSTC**