# Multi-Agent Reinforcement Learning Framework in SDN-IoT for Transient Load Detection and Prevention
## 5 Proposed MADDPG Framework  
In this section, we proposed our novel MARL SDN-IoT framework with the objective to optimize the network through multipath routing while blocking DDoS traffic. The proposed architecture is the first of its kind in an SDN that utilizes two agents in a cooperative environment integrated with the MADDPG algorithm. The MADDPG integrated MARL architecture, as shown in Figure 7, has an RL set up consisting of two main parts: two intelligent agents in the controller and an observed environment comprising data forwarding devices and IoT devices. The design is modelled as an MDP with a state space, an action space, and a reward function. In Algorithm 1, state S in MDP is defined as essential features in the agent's environment that are of interest. The state consists of a finite set \(\left\{\mathsf{s}^{1},\ldots,\mathsf{s}^{\mathsf{N}}\right\}\) where \(\left|\mathsf{S}\right|=\) N. N is the size of the state space. The action, A defined as a finite set \(\left\{\mathsf{a}^{1},\ldots,\mathsf{a}^{\mathsf{K}}\right\}\) where \(\left|\mathsf{A}\right|=\) K is applied in the environment  
Figure 5: Proposed MARL MADDPG concept. Figure 6: Proposed concept—OpenFlow switches. to control the system's state to a new state. \(K\) represents the size of the action space [43]. With possible transitions, an action \(\text{a}\in\text{A}\) in a state \(\text{s}\in\text{S}\) changes the state from s to a new state \(\text{s}\prime\in\text{S}\). This results in a transition state of \(\text{P}(\text{s},\text{a},\text{s}\prime)\). The reward function is based on the action of the agent in a state. \(\text{R}\left(\text{s},\text{ a},\text{s}\prime\right)\) represents the reward after transition from s to a new state s\(\prime\). A discount factor \(\gamma\) determines the agent's action that leads to distant future rewards relative to immediate rewards. A discount factor should range between \(0\leq\gamma<1\). An MDP is a 5-tuple \(\left(S,\text{ }A,\text{ }P,\text{ }R,\text{ }\gamma\right)\), where;  
S is a set of states  
A is a set of actions  
P \(\left(\text{s},\text{ a},\text{ s}\prime\right)\) is the probability that action a in state s at time t  
will lead to state \(\text{s}\prime\) at time t + 1  
R \(\left(\text{s},\text{ a},\text{ s}\prime\right)\) is the immediate reward received after a transition  
from state s to s\(\prime\), due to action a  
\(\gamma\) is the discounted factor, which is used to generate a discounted reward  
Figure 7: Proposed MARL MADDPG framework. The environment, as shown in Figure 7, has an SDN-enabled network consisting of OpenFlow switches that relate statistics and forward data flows of the network under the instructions of the SDN controller. The IoT devices, as part of the environment, generate data packets and elephant flows to destination nodes via the SDN-IoT gateway and forwarding devices. The environment also consists of zombie or compromised IoT devices that generate malicious packets to random destinations in the network. The two agents interact with the environment using actions that alter the state of the environment and receive rewards for their respective actions. The MARL agents execute the actions via the SDN controller, which has the responsibility to make decisions, collect network statistics, and enforce agent's actions by updating the flow table of the OpenFlow switches. Agent 1 is responsible for multipath routing of genuine transient packets and elephant flows in the network, while agent 2 is responsible for identifying and blocking DDoS flows. As depicted in Figure 8, the MARL MADDPG flow diagram illustrates the monitoring of network metrics and the action enforcement in the network through the RYU controller, an open source SDN controller [45]. The state space defines the environment and refers to the state of flows in the network. The state space s(t) is defined as a traffic matrix at time step t \(TM_{t}\).