# Optimizing Weighted Fair Queuing with Deep Reinforcement Learning for Dynamic Bandwidth Allocation
## 4 Proposed Model
### Continual Reinforcement Learning Framework for WFQ
Where theta is the optimal parameter \(\theta_{i}^{*}\) set after learning task A, these gradients function as a practical approximation of the FIM and are calculated as: \[F_{\theta}=\mathbb{E}\Bigg{[}\left(\frac{\partial\log\pi_{\varnothing}\ (a|s)}{ \partial_{\theta}}\right)^{2}\Bigg{]}\] (3) Here, \(\pi_{\varnothing}(a_{t}1s_{t})\) is a policy network, where \(s\) represents the state, and a represents the action [26]. This formula is also applicable to the critic network Q. The FIM computing is as follows: 1. Train networks (policy, two critic networks) 2. From the replay buffer, sample a batch of experiences. 3. Computing the loss function for the policy and critic. 4. Obtain gradients by performing backpropagation with respect to their weights. Calculating the squared gradients and averaging them over the buffer size. * Store \(\theta_{i}^{*}\) and FIM for compute EWC loss (regulation term). * In this study, the algorithm adapts bandwidth allocation decisions based on queue states. The theoretical fairness share for a given flow at any step-in episode is computed using the following formula [27]: \[Fairness=\frac{\mathrm{w}_{i}}{\sum_{j=1}^{i}\mathrm{w}_{j}}\mathrm{w}_{i}\geq 0\] (4) where \(\mathrm{w}_{i}\) is the weight allocated for each \(q_{i}\), and it is proportional to the total weights \(\sum_{j=1}^{i}\mathrm{w}_{j}\). During training in each step, the theoretical fairness computes how each queue in the system should allocate bandwidth as a proportion of the total bandwidth based on its state (i.e., queue length), representing the actual value. According to the prediction, value came out of the policy network \(\pi_{\mathbf{\phi}}\), which is a vector of action \(a=(w_{1},w_{2},\ldots,w_{n})\) that will be implemented in the environment to allocate bandwidth for each queue and then the service rate will be computed. * Root Mean Square Error (RMSE): A metric used to measure the root of average deviation of prediction value from target value, the measure is defined as indicated in equation [28]: \[RMSE=\sqrt{\frac{\sum_{N=1}^{i}\left(\hat{y}-y\right)^{2}}{N}}\] (5) where \(\hat{y}\) is the prediction value denoting weights allocated for each \(q_{i}\), which came from policy network \(\pi_{\mathbf{\phi}}\), \(y\) is the actual value indicating the weights computed by fairness function. In this study, the RMSE metric can be used to assess the model during training, and it offers insight how the model accurately becomes close to the theoretical fairness, meaning that the deviation of prediction value (weight for each \(q_{i}\) came from policy network) from the target value (weights of each queue computed by fairness function) into each step. * Jain's Fairness Index: It is a mathematical metric used to evaluate how fairly resources, such as bandwidth, are distributed among multiple flows or entities in a system. It is defined as [29]: \[f(x)=\frac{\left[\sum_{i=1}^{n}x_{i}\right]^{2}}{N\sum_{i=1}^{n}x_{i}^{2}} \quad\quad\text{where }x_{i}\geq 0\] (6) where \(\sum_{i=1}^{n}x_{i}\) is represents the squared of stochastic action values predicated by the policy network \(\pi_{\mathbf{\phi}}\), reflect the actual network in bandwidth allocation decision for each queue. \(\sum_{i=1}^{n}x_{i}^{2}\) is optimal fitness of the bandwidth allocation (i.e., weight for each queue relative to the current queue state computed theoretically by fairness function). Specifically, in this study, a value of the Jain index near 1 indicates that the policy network's predictions closely match the optimal fairness pattern. If the value of the Jain index is significantly less than 1, it indicates a deviation from fairness (optimal weights for each queue based on the current state computed by fairness function), indicating that some queues are over or under-served relative to their optimal share.