# A3C-R: A QoS-Oriented Energy-Saving Routing Algorithm for Software-Defined Networks
## 4 QoS and Energy-Saving Routing Based on A3C  
### A3C-R Routing Framework  
A3C-R uses the asynchronous multi-thread training mechanism of the A3C algorithm to execute multiple agents, including a Global network and multiple Worker threads, and their network structures are exactly the same. Each Worker thread includes the Actor network and the Critic network to interact independently with the environment. The Actor network can realize the mapping from state to action, and the Critic network is to evaluate whether the action is good or bad. During the independent exploration process of multiple Workers and the environment of the A3C algorithm, it not only breaks the correlation between training data but also saves storage space, reduces computing power requirements, and shows obvious advantages in training speed and convergence. The A3C algorithm asynchronous update framework is shown in Figure 2.  
In the traditional Actor network and the Critic network, the Actor network refers to the policy function \(\pi_{\theta}(a|s)\), and the Critic network refers to the value function \(V_{\phi}(s)\). The policy function \(\pi_{\theta}(a|s)\) defines the probability distribution of selecting action a given the current state \(s\). It is used to guide the agent's decision-making process. The value function, \(V_{\phi}(s)\), represents the expected cumulative reward that the agent can obtain starting from state, \(s\). It evaluates the quality of the current state and helps the agent optimize its policy. The strategy function and the value function are continuously learned during the training process, and the reward starting at time, t, can be recorded as \(R(\tau_{t:T})\), and the calculation process of the approximate value, \(\hat{R}(\tau_{t:T})\), is shown in Formula (14). \[\hat{R}(\tau_{t:T})=r_{t+1}+\gamma V_{\phi}(s_{t+1}) \tag{14}\]  
Among them, \(s_{t+1}\) and \(r_{t+1}\) are the state and reward at time \(t+1\), respectively, and the strategy function \(\pi_{\theta}(a|s)\) and function \(V_{\phi}(s)\) are learned during the update process. On the  
Figure 2: Asynchronous update framework of A3C algorithm. one hand, the parameter \(\phi\) can be updated to make the value function, \(V_{\phi}(s_{t})\), close to the real return value, \(\hat{R}(\tau_{t:T})\). The calculation process Formula (15) is shown. \[\min_{\phi}\bigl{(}\hat{R}(\tau_{t:T})-V_{\phi}(s_{t})\bigr{)}^{2} \tag{15}\]  
On the other hand, the value function is used as a baseline update to reduce the variance of the policy gradient, and the update process is shown in Formula (16). \[\theta\leftarrow\theta+\alpha\gamma^{t}\bigl{(}\hat{R}(\tau_{t:T})-V_{\phi}( s_{t})\bigr{)}\
abla_{\theta}\log\pi_{\theta}(a_{t}|s_{t}) \tag{16}\]  
The traditional Actor network and Critic network have the phenomenon that no matter which action is used in a certain state, they can obtain better action value, ignoring the differences between actions, making the training process unstable. Therefore, the A3C algorithm introduces an advantage function. The process is shown in Formula (17). \[A(s,a)=Q(s,a)-V(s,a) \tag{17}\]  
Among them, \(Q(s,a)\) is the action value function, \(V(s)\) is the estimated cumulative state value function in the states, and the training sequence \((s_{t},a_{t},r_{t},s_{t+1})\) is given. The calculation process of \(Q(s,a)\) is shown in Formula (18). \[Q(s,a)=r_{t}+\gamma V(s_{t+1}) \tag{18}\]  
The policy update of the Actor network will adopt the policy gradient training method. The calculation process is shown in Formula (19). \[\theta\leftarrow\theta+\alpha\underset{t}{\sum}\
abla_{\theta}\log\pi_{\theta }(s_{t}|a_{t})+\beta\
abla_{\theta}H(\pi_{\theta}(\cdot|s_{t})) \tag{19}\]  
Among them, \(\alpha\) is the learning rate, and the entropy value of strategy \(\pi\) is introduced to optimize the exploration process by organizing premature convergence to the suboptimal determination strategy. \(H\) is the entropy of the strategy. The entropy function, \(H(\pi_{\theta}(s_{i}))\), measures the randomness of the policy, \(\pi_{\theta}\).