## 3 System Scheme and Implementation
### CTN-MADDPG Intelligent Network Model  
This paper constructs a hybrid deep reinforcement learning model based on the 1D-CNN and LSTM concatenated for each agent. The Actor-Critic network model structure consists of three network modules: the one-dimensional convolution module, LSTM module, and fully connected layer module. CTN-MADDPG utilizes the one-dimensional convolutional neural network layer to learn the features of the network state information and extracts the temporal information of the learned features through the LSTM layer, which can effectively improve the generalization ability and traffic prediction capability of the model and achieve optimal routing and forwarding. The structure of the CTN-MADDPG intelligent network model is shown in Figure 2.  
In this algorithm, the one-dimensional convolutional network module is added to the Actor-Critic architecture, which consists of a one-dimensional convolutional layer (Conv1d), max-pooling layer (MaxPool1d), and ReLU activation function layer.  
Furthermore, the second network module in the Actor-Critic architecture of this paper is the LSTM module. The LSTM module is used to learn the feature vectors processed by the one-dimensional convolutional layer, exploring their internal rules and obtaining their hidden layer states. By combining LSTM, the effective temporal information from action values and observed states can be introduced and processed. Based on the action information from the previous time step, the actions to be taken at the current time step can be effectively predicted and properly adjusted so as to accelerate the training efficiency of the network and achieve a better control effect.  
In this paper, LSTM is applied to the traffic scheduling task of the data center, which makes LSTM learn how to predict the state and reward. Through this approach, it infers that the network states obtained by each agent interact with the network topology environment. At the point, the agent wants to obtain the action- and state-value functions of the model output when forwarding data packets, which depends not only on the current state \(s\) but also on the additional auxiliary signal \(h_{t-1}\) from the self-propagation of the LSTM memory unit, which contains the key features extracted from the network topology in the previous states. The forward propagation of all states relies on the same LSTM, which runs through the entire episode. The cell state and hidden state transmitted inside the LSTM serve as the long-term memory and short-term memory of the agent, respectively, helping the agent  
Figure 2: The structure of CTN-MADDPG intelligent network model.  
to better grasp the global information and cope with the problem of hidden states in the current state in real situations.  
In this paper, the third network module of the Actor-Critic architecture is the fully connected layer module. The last layer of the Actor network outputs the routing actions. The last layer of the Critic network changes the output dimension to 1, which serves as feedback from the Critic network to the Actor network, guiding the Actor network to update and select the optimal action strategy.