# A3C-R: A QoS-Oriented Energy-Saving Routing Algorithm for Software-Defined Networks
## 4 QoS and Energy-Saving Routing Based on A3C
### A3C-R Routing Algorithm Process  
During the training process of the QoS-oriented energy-saving routing algorithm A3C-R, the network status, including network energy consumption, and QoS indicators such as delay, bandwidth, and packet loss rate, are the main inputs. Through the asynchronous training mechanism of the A3C algorithm, multiple agents are executed to complete the independent interaction with the environment, and the experience value obtained during the interaction process is sampled to complete the update of the entire network. Finally, the weights of various network links that meet the requirements of QoS and network energy consumption are output, so as to calculate the optimal routing strategy. The process of A3C-R routing algorithm oriented to QoS and energy saving is shown in Algorithm 1. ```
Input: The set of network states \(\left\{\phi_{f}t,Delay(P)t,D(P)t,Loss(P)t\right\}\) Output: The set of network link weights \(\left\{w_{1}w_{2},\cdots,w_{n}\right\}\) Initialize: Critic parameter \(\phi\) and Actor parameter \(\theta\) of the global network. Critic parameter \(\phi\prime\) and Actor parameter \(\phi\prime\) of Worker thread network. Initialize: Global shared counter \(T\), global maximum shared count \(T_{max}\), thread step size \(t=1\). (1) Repeat: (2) Reset \(\phi=\phi\prime\), \(\theta=\theta\prime\) (3) \(t\gets t+1\), \(s_{t}=\left\{\phi_{f}t,Delay(P)t,D(P)t,Loss(P)t\right\}\) (4) Repeat (5) The Worker thread executes \(a_{t}\leftarrow\pi(a_{t}|s_{t};\theta\prime)\) (6) Obtain \(r_{t},s_{t+1}\gets a_{t}\) (7) \(t\gets t+1\), \(T\gets T+1\) (8) Until \(s_{t}\) is the terminal state, or \(t-t_{start}=t_{\max}\) (9) \(R=\left\{ \begin{array}{ll}0,&s_{terminal}\\ V(s_{t};\phi\prime),&others\end{array} \right.\) (10) For \(i\in\left\{t-1,t-2,\cdots,t_{start}\right\}\)do (11) Calculate \(R\gets r_{i}+\gamma R\) at each moment (12) \(d\theta\gets d\theta+\
abla_{\theta\prime}\log\pi(a_{t}|s_{i};\theta \prime)(R-V(s_{i};\phi\prime))+c\
abla_{\theta\prime}H(\pi(s_{i},\theta\prime))\) (13) \(d\phi\gets d\phi+\partial\left(R-V(s_{i};\phi\prime)^{2}\right)/\partial\phi\prime\) (14) End For (15) Global network asynchronous update \(\theta\gets d\theta\), \(\phi\leftarrow\phi\prime\) (16) Until \(T>T_{max}\)
```  
**Algorithm 1** A3C-R Routing Algorithm Process for QoS and Energy Saving  
The A3C-R algorithm, designed for QoS-oriented and energy-efficient intelligent routing, begins with a series of rigorous initialization steps. First, the Critic and Actorparameters of the global network are initialized, where the Critic evaluates state values, and the Actor generates decision actions. Simultaneously, the Critic and Actor parameters for each Worker thread are also initialized, establishing a solid foundation for multi-agent asynchronous training. In addition to parameter initialization, key global variables such as the global shared counter, maximum shared update steps, and thread step size are configured. These settings play a crucial role in defining the iteration process and termination conditions, ultimately influencing the algorithm's convergence efficiency. The core training process is detailed in Algorithm 1, lines 1-15, which describes the parameter update mechanism during training. Specifically, in lines 2-3, the agent retrieves the initial network state, including key indicators such as energy consumption, latency, bandwidth, and packet loss rate, which serve as the basis for decision making. In lines 4-9, the Worker thread executes actions based on the current state, then obtains the corresponding reward value to assess the effectiveness of the action, along with the next state for further evaluation. Finally, in lines 10-15, the agent updates global network parameters based on reward values, state information, and policy gradients, continuously refining the routing strategy. This parameter update mechanism enables A3C-R to dynamically adjust routing decisions, balancing QoS requirements and energy efficiency.