## 3 Materials and Methods
### DRL-Based Dynamic Collaborative Resource Scheduling Algorithm  
Although the MILP model can provide an accurate solution in small networks, it is time-consuming and cannot be worked out in large networks. Also, it cannot adapt to diversified types of service requests and dynamic network environments. To this end, a DRL-based AI service-oriented dynamic collaborative resource scheduling algorithm based on distributed data parallelism is proposed. Notably, our proposed DQN model is trained under sets of AI services during the time period \(t\). As the time period \(t\) approaches infinity, it can be regarded as a dynamic planning problem. That is to say, the dynamic computing resource scheduling problem. Under diversified types of AI service requests and changing workloads on edge data centers, the proposed DRL-based algorithm is good at providing an elastic collaborative scheduling strategy of edge computational resources based on distributed data parallelism. We first formulate the dynamic, collaborative resource scheduling problem of edge computational resources as a Markov decision process. It is represented by a quadruple of \(<\)state, action, reward, Q value\(>\). For the state, it contains the remaining computing resource capacity \(C_{i,j}^{t}\) of edge data centers and the cost of \(N_{i,j}\) for processing one computational request task on each edge data center. For the action, the computational request tasks \(A_{i,j}^{t,k,f}\) allocated on each edge data center of different suppliers in each time frame. The Q value represents the state-action function obtained through neural network training. In other words, it is the maximum reward expectation obtained by selecting different actions for different states, and the reward is formulated in Equation (7). \[Reward=\left\{ \begin{aligned} \alpha\cdot\left(\sum\limits_{t,i,j,k,f}M^{k,f} \cdot A_{i,j}^{t,k,f}-\sum\
olimits_{t,i,j}N_{i,j}\cdot C_{i,j}\cdot X_{i,j}^ {t}\right)-\beta\cdot\sum\
olimits_{t,i,j}\sigma_{i,t^{\prime}}^{t}\left( \begin{array}{c}if\ successfully\ assigned\\ if\ not\ successfully\ assigned\end{array}\right)\end{aligned} \right. \tag{7}\] \[-R,\]  
where \(\alpha\) and \(\beta\) are the respective weight coefficients for a profit of computational service and backlog of computational request tasks on the edge data center in the ECFN of the smart grid. The process steps of the DRL-based dynamic, collaborative resource scheduling algorithm are shown as follows. 1. Initialize the parameters on edge data centers of different edge computational resource suppliers. 2. Input the request \(B^{k,f}\) and the current state \(s_{t}\) of edge data center in the \(t\)-th time frame into DNN, which consists of an M-layer convolutional neural network and a N-layer fully connected network. 3. Based on \(B^{k,f}\) and \(s_{t}\), the corresponding Q value is calculated for each action, and then DNN outputs the set of Q values for all possible actions \(Q^{\pi_{t}}(s_{t},a_{t})\) under the current strategy \(\pi_{t}\), where \(a_{t}\) indicates the action in \(t\)-th time frame. 4. The action with the largest value is selected from the set of Q values to interact with the environment. The action can also be selected utilizing the \(\varepsilon\)-Greedy strategy. That is to say that the action is selected with the \(\varepsilon\) probability with the highest Q value. The action is randomly selected to fully explore the action space with (1-\(\varepsilon\)) probability. The value \(\varepsilon\) will gradually increase after each action selection. 5. After executing the action \(a_{t}^{*}=max_{a_{t}}Q^{\pi_{t}}(s_{t},a_{t})\), the status \(s_{t}\) of remaining computational resource capacity on the edge data center is updated to the next state \(s_{t+1}\) and return the reward value \(r_{t}\). 6. The quadruple entry \(<\)\(s_{t}\),\(s_{t+1}\),\(a_{t}\),\(r_{t}\)\(>\) is saved in the memory. 7. \(L\) sets of data entries from the memory are quantitatively and randomly extracted at regular intervals.