# Multi-Agent Reinforcement Learning Framework in SDN-IoT for Transient Load Detection and Prevention
## 5 Proposed MADDPG Framework
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{Scenario 1} & No. of devices & 20 \\  & No. of controller & 1 \\  & No. of switches & 5 \\ \hline \multirow{3}{*}{Scenario 2} & No. of devices & 60 \\  & No. of controller & 1 \\  & No. of switches & 8 \\ \hline Bandwidth & \multicolumn{2}{c}{50–100 Mbps} \\ Traffic Type & TCP & \\ Queue Type & Tail Drop & \\ Transmission Rate & 2 Mbps & \\ SDN Controller & Ryu & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Network parameters. #### 7.2.2 Jitter

Jitter in SDN represents the small intermittent delay in data transmission. It is caused by a number of factors including network congestion, interference, and collisions. Jitter represents the variation in the delay of packets transmitted from the source to destination nodes. Figure 9: MADDPG set up in Mininet. #### 7.2.3 End to End Delay

Transmission delay refers to the time taken for data packets to move from source to destination nodes in the SDN network. #### 7.2.4 Packet Loss

Packet loss ratio is calculated as the difference between the number of packets received over the number of packets sent in the network. If the destination does not receive the packets, then the packet loss ratio is high; otherwise it is reduced. #### 7.2.5 Bandwidth Usage

It is estimated in bits per second, and a higher bandwidth helps more data to pass through it. This metric also evaluates how the network allocates the bandwidth for data transmission. When the optimum bandwidth is allocated, then the data transmission is efficient and reliable. To provide scalable and reliable services for current network traffic beyond improving network bandwidth, the system must provide a flexible mode for traffic management. #### 7.2.6 DDoS Detection Rate

The detection rate of DDoS detection is defined as the number of packets successfully predicted as intruder packets. #### 7.2.7 Global Reward

As shown in Figure 10, after 3000 episodes, MADDPG gained a 25% cumulative global average reward over DDPG for the 20-nodes topology. The least reward gap for MADDPG was obtained from episode 1 to 1500 at 9%, while the later episodes gave a 16% reward. Due to exploration, the reward gap decreased along the episodes. For MADDPG, the average reward gap decreased from episode 1000 to 1500, while DDPG had a minimum decrease from episode 2000 to 2500. For the 60-nodes topology, the average global reward of MADDPG increased to 33% for the 3000-episode state-action transitions. MADDPG obtained the least average reward of 8% from episodes 1 to 1500, with a 24% reward for the later episodes. DDPG obtained an average least incremental reward gap from episodes 1500 to 2000, while MADDPG's least incremental gap was obtained from episode 1000 to 1500 and episodes 1500 to 2000. Figure 10: Reward vs. no. of episodes for SA-DDPG and MA-DDPG. #### 7.2.8. Jitter

As shown in Figure 11, as the number of nodes involved in packet transmittal increases, the average jitter increases due to link utilization and congestion. MADDPG for 20-nodes topology decreased jitter by 33% to 2000 ms compared with DDPG of 4000 ms. For the 60-nodes topology, the jitter performance for MADDPG decreased by 20% to 4000 ms compared with 6000 ms for DDPG. The initial jitter between MADDPG and DDPG until 5 nodes was close to a percentage of 2% for both 20 and 60 topologies. Figure 11. Jitter vs. no. of hosts for SA-DDPG and MA-DDPG. #### 7.2.9 Delay vs. No. of Switches

Delay is the time it takes for packets to move from source to destination nodes in the SDN network. Packets sent from source to destination nodes have a transmission delay monitored by the SDN switches using the delay in the current arrival packets from the nodes and the RYU controller. As shown in Figure 12, as the episode progresses and the number of switches involved in packet transmission increases, the overall delay in the network decreases for both MADDPG and DDPG due to multipath routing.