# Multi-Agent Reinforcement Learning Framework in SDN-IoT for Transient Load Detection and Prevention
## 5 Proposed MADDPG Framework
An OpenFlow switch \(k_{i}\ i\ \in(1,2,3,\ldots\ldots\ldots\ N)\) in time slot \(t\), \(t\ \in(1,2,3,\ldots\ldots\ldots\ldots\)) consists of:  
A. Data link occupancy rate between the flow and the OpenFlow switches \(Bk_{i}(t)\)  
B. Channel link occupancy between packet-in messages from the OpenFlow switches \(Gk_{i}(t)\)  
C. Channel link occupancy between packet-out messages to the OpenFlow switches \(Ck_{i}(t)\) Equation (1):  
\[\textit{State Space}\hskip 56.905512 ptTM_{t}\cong\left[ \begin{array}{ cccc}Bk_{1}(t)&Bk_{2}(t)&\ldots&Bk_{N}(t)\\ Gk_{1}(t)&Gk_{2}(t)&\ldots&Gk_{N}(t)\\ Ck_{1}(t)&Ck_{2}(t)&\ldots&Ck_{N}(t)\end{array} \right] \tag{1}\]  
Figure 8: MARL MADDPG flow diagram. ```
1:for episode = 1 to \(M\)do
2: Initialize a random process \(N\) for action exploration
3: Receive = \(\{ \begin{array}{ll}Bk_{1}(t),Bk_{2}(t),........,Bk_{N}(t);\;\delta k_{1}(t),\; \delta k_{2}(t),........,..\;\delta k_{N}(t);\;Ck_{1}(t),\;Ck_{2}(t),........,..\;..\;..\;..\;..\;..\;Ck_{N}(t)\end{array} \}\)
4:for\(t=1\) to max-episode-length do
5: for agent 1, select action \(a_{agent}(t)\)
6: Executive \(a_{agent1}(t)=\left[a_{t=1}^{\text{agent}}(t),\;a_{t=1}^{\text{success}}(t)< TH\right]\)and observe \(R_{agent2}(t)=\frac{1}{0}\) for agent 2, select action \(a_



### Experimental Set up

The 20 and 60 nodes network topologies were implemented using TensorFlow and Keras running on Ubuntu-18.04 LTS (64 bit) operating system with an Intel(r) Pentium(r) CPU G2030 @ 3.00 GHz and 8 GB RAM capacity. The software requirement also includes Mininet-2.0 and Python-2.7. From Table 1, we used RYU controller with 5 and 8 OpenFlow switches. The bandwidths for data link and channel link occupancy rates are 50 Mbps and 100 Mbps, respectively. The Transmission Control Protocol (TCP) is used to send benign traffic across the network with a sending rate of 2 Mbps. The tail drop queue management is used for dropping packets. A fixed packet size of 56 bytes is randomly generated using Python library scipy, at an interval time of 0.1 s. The IPv6 protocol with a processing delay of 10 us is used as the communication protocol to identify devices on the network. DDoS packets are randomly generated from source nodes to random destinations at an interval of 0.02 s and restricted to only UDP flooding. The elephant flow is generated with a bandwidth range of 10 Mbps-1000 Mbps, at an interval of 0.1 s.

After multiple experiments, the actor learning rate \(\alpha\) for agent1 and agent 2 is set to 0.0001. The actor discount factor \(\gamma\) for agents 1 and 2 is set to 0.99. The actor target soft update \(\tau\) is set to 0.001, while the batch size is set to 64. The critic agent parameter has the same set-up as the actor agent parameter with an \(\varepsilon-greedy\) exploration of 0.1. Agent 1 and Agent 2 are set up with (300, 600) hidden units and neurons. For the actor network, the Deep Neural Network (DNN) hidden layer activation function for agent 1 and 2 is set to relu while the output layer is set to sigmoid. The critic DNN for agent 1 and 2 has a state hidden layer 1 activation function set to relu, action hidden layer 1 set to linear, state hidden layer 2 set to linear, action hidden layer 2 set to relu, and output layer activation function set to linear. ### Experimental Results

In this subsection, the performance metrics considered to evaluate the MADDPG algorithm and DDPG are discussed and the results analyzed. The Mininet simulations for 20-nodes and 60-nodes network topology are shown in Figure 9.

#### 7.2.1 Reward

The reward is used to measure the results of the obtained action to a respective corresponding state that is perceived from the environment. In DRL, agents perform an action and obtain a reward based on the network policy. Average global reward is computed for 3000 episodes.