## 3 Research Design and Methodology
### Asynchronous Advantage Actor-Critic (A3C) Model
#### 3.2.2 Actor-Critic Model  
The A3C algorithm uses multiple workers or agents to update a shared model asynchronously. In the following Figure 2, we can discern the placement of our A3C agent in conjunction with our modified L4S algorithm. As observed, we construct an actor and critic model by representing policy function and Q-network. \begin{table}
\begin{tabular}{p{142.3 pt} p{284.5 pt}} \hline
**Variable** & **Definition** \\ \hline burst\_allowance & Maximum allowed burst size of packets before congestion control measures are applied. \\ \hline drop\_prob & Probability of dropping a packet when the queue is congested. \\ \hline current\_qdelay & Current queue delay, which is the time a packet spends in the queue before being transmitted. \\ \hline qdelay\_old & Previous queue delay value used for comparison and calculation in the FQ-PIE algorithm. \\ \hline accu\_prob & Accumulated probability value used in FQ-PIE to determine the drop probability for incoming packets. \\ \hline measurement\_start & Start time of the measurement interval for collecting statistics. \\ \hline tot\_pkts & Total number of packets observed during the measurement interval. \\ \hline tot\_bytes & Total number of bytes observed during the measurement interval. \\ \hline length & Average length (in packets) of the queue during the measurement interval. \\ \hline len\_bytes & Average length (in bytes) of the queue during the measurement interval. \\ \hline drops & Number of packets dropped during the measurement interval. \\ \hline ECN & is a packet flag marked with explicit congestion notification (ECN) during the measurement interval. \\ \hline action & Action taken by the A3C algorithm \\ \hline reward & Reward or penalty assigned to a specific action taken by the A3C algorithm. \\ \hline \end{tabular}
\end{table} 
Table 1: FreeBSD AQM L4S Terminology. Figure 2: A3C coupled L4S Architecture. **Advantage Function:** The advantage function \(A(s,a)\) quantifies the relative benefit of choosing action '\(a\)' in state '\(s\)' compared to the average action in that state. This metric allows us to evaluate the effectiveness of specific actions within their respective states by assessing their performance relative to the norm for that context. It is a key component in enhancing the decision-making capabilities of actor-critic agents in complex and dynamic environments. \[A(s,a)=Q(s,a)-V(s) \tag{9}\]  
**Advantage Function with Returns:** Since Q values are unavailable, we estimate the advantage function by computing the discounted rollout returns. This allows us to infer the potential value of different actions in various states during training. \[A(s,a)=R-V(s) \tag{10}\]  
**Returns:** The return \(R\) is the cumulative discounted reward with \(\gamma\) exponentially decreasing the future rewards' weight and n denoting the number of steps or actions taken:  
\[R=r_{n}+\gamma r_{n-1}+\gamma^{2}r_{n-2}+\dots \tag{11}\]  
**Advantage Estimate:** Each worker in the algorithm maintains estimates of the policy \(\pi(a_{t}|s_{t};\theta)\) and value function \(V(s_{t};\theta_{v})\), utilized in computing the following advantage estimate:  
\[A(s_{t},a_{t})=\sum_{i=0}^{n-1}\gamma^{i}r_{t+i}+\gamma^{n}V(s_{t+n};\theta_{ v})-V(s_{t};\theta_{v}) \tag{12}\]  
In this equation,  
* \(\theta\) represents the parameters of the policy network, which outputs the policy \(\pi(a|s;\theta)\). * \(\theta_{v}\) represents the parameters of the value network, which outputs the value function \(V(s;\theta_{v})\)
* The state \(s_{t}\) represents the environment at step \(t\). It contains all the necessary information for the agent to decide that step. \[H(\pi)=-\sum_{a}\pi(a|s)\log(\pi(a|s))\] (13)  
**Actor Network:** The actor network is responsible for choosing actions 'a' based on the current environmental state. It receives in the state '\(s\)' as input and outputs a probability distribution of the feasible actions available in that state.