# Faulty Links' Fast Recovery Method Based on Deep Reinforcement Learning
## 3 Faulty-Path Recovery Scheme Based on DDPG-LBBP
### DDPG-LBBP Agent Interacts with the Environment  
The GRU neural network is used in the DDPG-LBBP algorithm framework for the neural network training process of the DDPG algorithm. The GRU neural network, as an efficient recurrent neural network architecture, primarily comprises two gating mechanisms: a reset gate and an update gate. These gates enable the GRU to efficiently handle and extract relevant information from the input multi-dimensional network topology data, resulting in a superior training effect. The DDPG-LBBP algorithm leverages the sophisticated gating mechanisms of the GRU neural network, specifically the update gate and the reset gate. The update gate assesses the level of impact that historical state information has on the current state, while the reset gate modulates the degree of interaction between the current input and the historical state information. This approach streamlines the algorithm's architecture, enhancing training efficiency while maintaining model accuracy. The interaction process between the DDPG-LBBP agent and the environment is shown in Figure 3.  
The DDPG-LBBP agent interacts repeatedly with the network environment, adjusting the action output according to the dynamic changes in data transmission delay, link occupancy rate, packet loss rate and reward value in the network topology, and dynamically adjusting the link weight in the network topology. During the interaction between the agent and the environment, the following procedures are integral to the mapping of states, actions, and reward values:  
1. State mappingThe state serves as a representation of the real-time conditions within the network environment where the agent operates. The DDPG-LBBP algorithm incorporates the data transmission delay, link occupancy, and packet loss rate within the network topology as input state information. The SDN controller gathers and processes this state information, subsequently transmitting the actual network status to the agent. Specifically, the state of the nodes in the network topology at time \(t\) is denoted as \(s_{t}=\left[delay_{t}\big{(}e_{ij}\big{)},bandwidth_{e_{ij}}(t),Lose_{e_{ij}} (t)\right]\), where \(delay_{t}\big{(}e_{ij}\big{)}\) represents the data transmission delay, \(bandwidth_{e_{ij}}(t)\) signifies the link utilization, and \(Lose_{e_{ij}}(t)\) indicates the packet loss rate. (2) Action mapping  
The action represents a strategy formulated by the agent based on the evolving state and reward values. The DDPG-LBBP algorithm undergoes training by taking in the state \(s_{t}\) and reward value \(r_{t}\) at time \(t\) as inputs and subsequently outputs the optimized \(\theta^{Q}\) action value after the iterative training converges. The action set outputted by the DDPG-LBBP agent is denoted as \(action=\left[a_{w_{1,2}},a_{w_{1,3}},\cdots,a_{w_{ij}}\right]\), wherein each action value \(a_{w_{ij}}\) within the set corresponds to the link weight \(weight_{e_{ij}}\) assigned to a specific link \(e_{ij}\). (3) Reward value mapping  
The reward value is the immediate benefit of the action \(a_{t}\) taken by the feedback agent in the current state \(s_{t}\). The main optimization goals of the DDPG-LBBP algorithm are low  
Figure 3: The interaction process between the DDPG-LBBP agent and the environment. link utilization, low transmission delay, and low packet loss rate. The link utilization, transmission delay, and packet loss rate are standardized as the basis for calculating the reward value. The specific calculation process of the reward value is shown in Equation (8):  
\[reward=\frac{\eta}{Delay^{\prime}{}_{t}(p_{s,A})}+\frac{\tau}{Loss_{p_{s,A}}(t)} +\frac{\xi}{Bandwidth_{p_{s,d}}(t)} \tag{8}\]  
In Equation (8), the reward value factor parameter \(\eta,\tau,\xi\in(0,1),\eta+\tau+\xi=1\) is adjusted based on the relative importance of transmission delay, link utilization, and packet loss rate within the specific network topology to set the parameters \(\eta,\tau,\xi\).