# Optimizing Weighted Fair Queuing with Deep Reinforcement Learning for Dynamic Bandwidth Allocation
## 5 Results and Analysis
### Training Real Traffic  
In the experiment, twelve arrival flows with various flows (categories) arrive at the WFQ system of twelve queues. Twelve category groups of traffic data were used to train the agent. The average arrival rates of twelve sources in these twelve categories are shown in Table 7.  
The remaining twelve arrival flows with different flows (categories) arrive at the WFQ system of twelve queues. Twelve category groups of traffic data were used to train the agent. The average arrival rates of twelve sources in these twelve categories are shown in Table 8.  
The reward function is calculated as an equation:  
\[r(q,a)=-\sum_{i=1}^{8}\left(\frac{1}{n}\;q_{i}+l_{i}\right)\]  
The episode time length, buffer size, and the total bandwidth resources are T = 2000 s, \(K_{1}=K_{2}=\ldots K_{12}=32\), bandwidth = 1 byte/s, correspondingly. Running the 1000 episodes to train the agent by the SAC with EWC algorithms of twelve queues. \begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Queues**} & \multirow{2}{*}{**Categories**} & \multicolumn{2}{c}{**Average Arrival Rates**} \\  & & **(Packets/s)** \\ \hline
1 & Chat & 0.047179 \\ \hline
2 & Media & 0.038448 \\ \hline
3 & Web & 0.031438 \\ \hline
4 & VoIP & 0.030870 \\ \hline
5 & software update & 0.023940 \\ \hline
6 & Email & 0.023906 \\ \hline
7 & Music & 0.022780 \\ \hline
8 & collaboration & 0.021974 \\ \hline
9 & Cloud & 0.015814 \\ \hline
10 & Network & 0.011008 \\ \hline
11 & Download File Transfer, File Sharing & 0.005254 \\ \hline
12 & System & 0.002918 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: The average arrival rate of the first twelve categories. Figure 12: The average reward for twelve queues. A plot of the run average reward comparison of the twelve queues simulation curves, along with two datasets according to SAC with the EWC algorithm in the WFQ continual-DRL algorithm, is shown in Figure 13. For the first simulation curve, it can be seen that convergence occurs after 200 episodes, and the range of reward fluctuation stabilizes between \(-2.5\) and \(-2\), resulting in consistent performance. At the beginning of the reward curve, the agent typically explores different assignment strategies (bandwidth weights) for the queues, resulting in more fluctuation. This means that the controller explores all possible processes of the action it is learning in order to achieve an action that yields benefits. As learning processing, the agent exploits the probabilities of action, resulting in an increase in the reward curve; the policy then gradually converges toward the optimal bandwidth allocation, reducing the variability in the reward curve. Referring to the convergence of the curve average reward after around 200 episodes for the first 12 categories of the real traffic 2 dataset, the range of the fluctuation curve gradually increases and shows steady fluctuation between \(-2.5\) and \(-1.5\), indicating a stable performance. According to the recent 12 categories of the real traffic two of the dataset, the convergence of the average reward curve is before 200 episodes, and the range of the fluctuation curve is between \(-1.8\) and \(-1\), reducing the fluctuation in the curve. Table 9 presents the results of computing the MSE and MEA values for the twelve-queue scenario, which are used to measure the effectiveness of the environment in supporting adaptive training. The column containing the MSE values shows that at the start of training, the MSE is initially 0.8680; it then undergoes fluctuations before increasing to 1.0482 as the environment is explored and gradually improves during training. The slight variation suggests that the model is still learning, adjusting parameters, and refining predictions. By episode 300, the error is relatively steady compared to the prior fluctuations, indicating better convergence. In the intervening episodes, from 300 to 700, the error is more stable than the previous fluctuations, indicating better convergence.