## 2 System Architecture  
The algorithm is implemented based on the three-layer architecture of SDN. The three-layer architecture comprises the multi-agent layer, the controller layer, and the data layer from top to bottom. The multi-agent layer enables programmatic control of the network, including traffic monitoring, load balancing, and security policy implementation. Other functions are achieved through multi-agent control. Multi-agents accelerate learning by sharing knowledge and experience. The controller layer is the core of SDN architecture, which is responsible for centralized management of data layer network topology, interacts with the agent through the northbound interface to deliver the flow table, and interacts with the data layer through the southbound interface to collect data layer network topology information. The data layer includes network devices such as switches and routers, which are responsible for the actual data packet forwarding and processing. The processing flow of the architecture includes data layer information collection, status input, policy generation, policy delivery, and flow table delivery. The architecture is shown in Figure 1.  
Figure 1: Three-layer architecture of SDN. The interaction time interval between a single agent and the environment is optimized and iterated based on feedback from reward information. The reinforcement learning process is modeled as a Markov Decision Process (MDP) and follows the Markov decision process. When there are multiple agents in the architecture interacting with the environment, the entire system becomes a multi-agent system. Agent-environment interaction follows the MDP framework: at each time step \(t\), the agent observes the state \(s_{t}\), selects an action \(a_{t}\) via policy \(\pi_{t}\), receives a reward \(r_{t}\), and transitions to state \(s_{t+1}\) according to \(P(s_{t+1}|s_{t},a_{t})\). This iterative process aims to maximize the expected cumulative reward. Individual agents still follow the MDP process, while multi-agent systems treat other agents as influencing factors in the environment. In MDP, actions are selected based on known probabilities, and then the reward value is obtained in the next state. MDP includes the five elements of Equation (1). \[\langle S,A,P(s_{t+1}|s_{t},a_{t}),\gamma,R\rangle \tag{1}\]  
where \(S\) is the state space (set of states), with \(|S|\) denoting its cardinality, \(A\) represents a set of actions, \(t\) denotes the time step index, and \(P(s_{t+1}|s_{t},a_{t})\) represents the transition probability at time \(t\), mapping the state-action pair at time \(t\) to the state distribution at time \(t+1\). Immediate reward function \(r(s_{t},a_{t},s_{t+1})\) measures the short-term feedback for taking action \(a_{t}\) in a state \(s_{t}\) to reach \(s_{t+1}\). The discount factor \(\gamma\in[0,1]\) prioritizes immediate rewards over future rewards, balancing short-term and long-term objectives in the reinforcement learning process. The schematic diagram of the interaction between multiple agents and the environment is shown in Figure 2. Figure 2 illustrates the dynamic interaction between \(n\) agents and the network environment in a multi-agent MDP. Each agent \(i\) observes a local state \(s_{i}^{t}\) at time \(t\), selects an action \(a_{i}^{t}\) based on its policy \(\pi_{i}\), and collectively influences the environment's transition to the next state \(s_{t+1}\). The shared reward \(r_{t+1}\) reflects the global impact of joint actions (e.g., reduced end-to-end delay). This diagram highlights that agents treat each other's actions as part of the environment's dynamics, necessitating collaborative learning to maximize cumulative rewards. In multi-agent reinforcement learning, each agent operates in a shared environment, where they simultaneously select actions \(a_{i}\in A_{i}\) based on the current global state \(s\).