# A3C-R: A QoS-Oriented Energy-Saving Routing Algorithm for Software-Defined Networks
## 4 QoS and Energy-Saving Routing Based on A3C
### Design of A3C-R Routing and Environment Interaction  
We realize the interaction process between A3C-R routing and the environment in the SDN environment. The SDN mainly includes the data plane, the control plane, and the intelligent plane, as well as the southbound interface and the northbound interface. The training process of the A3C algorithm is mainly located in the intelligent plane. First, the data plane collects network status information and then transmits it to the control plane through the southbound interface to extract valid status information. Finally, the effective state information is input to the A3C-R routing algorithm of the intelligent plane through the northbound interface for asynchronous training. After the A3C-R routing algorithm training converges, it can output the link weights of the network topology, and then the control plane generates a flow table after receiving the link weight information and sends it to the data plane. In this way, the interaction between A3C-R routing and the environment is completed. The interaction process between A3C-R routing and the environment is shown in Figure 3. The design of the specific state, action, and reward during the interaction between A3C routing and the environment is as follows:  
1. State  
The state design in this paper mainly includes QoS information such as network delay, bandwidth, and packet loss rate, as well as network link energy consumption. We collected various network performance indicators according to the network information collection module, and then calculated the normalized network link energy consumption, delay, bandwidth, and packet loss rate through \(\phi_{f}\), \(Delay(P)\), \(D(P)\), and \(Loss(P)\prime\) respectively. Finally, the above state information is recorded as a state set \(s=\left\{\phi_{f}\prime,Delay(P)\prime,D(P)\prime,Loss(P)\prime\right\}\).
2. Action  
The design of the action is the link weights of the network topology. Through the continuous interaction between the A3C-R routing algorithm and the environment, we can gradually train the optimal link weights suitable for the current network environment. We can record the action output by the A3C-R routing algorithm as \(a=\left\{w_{1}\text{,}w_{2},\cdots,w_{n}\right\}\), where \(w\) is the weight value of each link in the network topology, and \(n\) is the total number of links in the network topology. The A3C-R algorithm, through the interaction with the network environment and continuous learning, generates link weights that comprehensively consider QoS and energy-saving requirements. To calculate the optimal routing path, the A3C-R algorithm utilizes the Dijkstra algorithm, which takes the optimized link weights generated by A3C-R as input and computes the optimal path from the source node to the destination node in the network topology. This path is not only the shortest in terms of link weights but also ensures that it meets the QoS and energy-saving requirements, as determined by the A3C-R algorithm. Finally, based on the link weights optimized by the A3C-R algorithm and the path calculated by the Dijkstra algorithm, the optimal routing strategy is implemented in the network.  
Figure 3: The interaction process between A3C-R routing and the environment.  
Reward  
The design of the reward mainly takes into account the energy consumption of the current network link, as well as network performance indicators such as network delay, bandwidth, and packet loss rate. The design of the reward is shown in Formula (21).  
\[r=\delta\phi_{f}t+\beta Delay(P)t+\lambda\frac{1}{D(P)t}+\gamma Loss(P)t,(\delta,\beta,\lambda,\gamma)\in[0,1] \tag{21}\]  
Among them, \(\delta\), \(\beta\), \(\lambda\), and \(\gamma\) are the weight parameters of link energy consumption, delay, bandwidth, and packet loss rate, respectively. In order to ensure the optimal balance of various performance indicators, we set the above parameters to 0.25, and their sum is 1.