## 3 System Design
### System Framework of DGRL  
The system block diagram of the controller is shown in Figure 2. The _Environment_ represents the network to which the controller is connected. The controller captures the current network status from the environment and uses the _Online Policy_ in the Actor neural network to make policy. The previous state, action made by the controller, reward value of the environment feedback, and the latest observed SDN state form a four-tuple \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Reference** & **Metrics** & \begin{tabular}{c} **Experimental** \\ **Platform** \\ \end{tabular} & **Drawbacks** \\ \hline [PERSON] et al. [31] & Delay, throughput & Simulations:OMNet++ & Limited scenarios \\ \hline [PERSON] et al. [32] & Throughput & Real-world scenarios & \begin{tabular}{c} Limited metrics, \\ heavyweight to \\ deploy \\ \end{tabular} \\ \hline [PERSON] et al. [33] & Criticality of path & \
\begin{tabular}{c} Own testbed \\ Simulations \\ \end{tabular} &
\begin{tabular}{c} Lack comparisons \\ and metrics \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 1: _Cont._ \begin{table}
\begin{tabular}{c c} \hline \hline
**Symbol** & **Description** \\ \hline \(N\) & The number of nodes in the network \\ \(A\) & The adjacency matrix of the network \\ \(X\) & The feature matrix of all nodes \\ \(l\) & One-hot coding of current node \\ \(d\) & One-hot coding of target node \\ \(F\) & The number of features of nodes \\ \(\mu\) & The output action decision vector of the Actor model \\ \(f\) & The number of packets that received by its destination node \\ \(T_{S_{j}}\) & The time when packet \(i\) sent by its source node \\ \(T_{Tj}\) & The time when packet \(j\) received by its destination node \\ \(Pr_{n_{i}}\) & The number of packets received at node \(n_{i}\) as a destination \\ \(Pd_{n_{i}}\) & The number of packets dropped at node \(n_{i}\) \\ \(\beta_{i}\) & The ratio of the number of packets forwarded by node \(n_{i}\) to \\  & the total number of packets forwarded in the network \\ \(\overline{\beta}\) & The average of \(\beta_{i}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Terms and notations used in DGRL. record (\(LastState\), \(Action\), \(Reward\), \(NewState\)). The four-tuple is stored in the experience pool for training multiple neural network models. One is the \(Online\)\(Q\) in Critic NN, and the other is the decision network \(Online\)\(Policy\) in Actor NN. The \(Online\)\(Policy\) is used to make decisions \(\mu\) based on the state of the current environment: state \(\rightarrow\) action, while the \(Online\)\(Q\) is used to fit the reward of the environment for the controller's decision: (state, action) \(\rightarrow\) reward. Both the \(Online\)\(Policy\) and the \(Online\)\(Q\) have exactly the same target network as their structure, named \(Target\)\(Policy\) and \(Target\)\(Q\), respectively. The purpose is to use soft updates to assist the training process to achieve convergence and avoid large gradient fluctuations during training. In general, the Actor NN and Critic NN both contain two parts, namely online network and target network. \(Online\)\(Policy\) outputs real-time actions for the Actor NN to use in real time. The \(Target\)\(Policy\) is used to update the Critic NN. The output of \(Online\)\(Q\) and \(Target\)\(Q\) are both the value of one state, while the input is different. \(Online\)\(Q\) takes the actual actions taken by the Actor NN and the current state as input. \(Target\)\(Q\) uses the output of \(Target\)\(policy\). Details about the feedforward processes for the Actor NN and the Critic NN can be found in Section 3.3. The structure diagram of the Actor neural network and Critic neural network is shown in Figure 3. Actor neural network is at the top of the figure, and Critic neural network is at the bottom of the figure. The blue rectangles in Figure 3 represent tensor transformation operations, the golden circles represent neurons, and the gray circles represent tensors.