###### Abstract
The overarching objective of L4S is to achieve superior latency and throughput while maintaining a low loss rate compared to the traditional network infrastructure. CoDel, PIE, FQ-CoDel, and FQ-PIE are effective in reducing \"bufferbloat\" and managing congestion in general scenarios. However, L4S goes a step further by explicitly prioritizing latency-sensitive traffic, catering particularly well to applications that demand immediate responsiveness, such as online gaming, video conferencing, and interactive multimedia streaming. By implementing L4S in FreeBSD, the project aims to optimize the network stack for delay-sensitive applications, providing a smoother and more reliable user experience. In this paper, we pioneer a new hybrid framework that can combine asynchronous gradient descent optimization for deep neural network controllers (e.g., A3C) with the L4S architecture to optimize buffer management for latency-sensitive real-time applications. It employs neural networks to learn and optimize system policies based on the environment's feedback, allowing it to recognize network congestion preemptively and prepare responses appropriately depending on the severity of congestion. Traditional algorithms have significant delays in responding to network congestion due to their simplistic nature in responding to sudden bursts of packets in the network. Integrating L4S into FreeBSD's network stack, combined with the A3C machine learning algorithm, enhances network performance, responsiveness, and adaptability, making it a compelling choice for modern network environments. The aim of this paper, as for other researchers in this field, is to find ways to contribute to advancing network protocols and machine learning techniques within the context of networking. Our main focus is developing a preliminary implementation of A3C-L4S to enhance L4S's responsiveness to congestion by exploring and exploiting past experiences. This will ultimately lead to improved network responsiveness, reduced latency, and enhanced efficiency in FreeBSD-based systems. Our experimental implementation and results presented in this paper are made available through our GitHub repositories (Experimental L4S implementation in FreeBSD 13.1: [[https://github.com/MPTCP-FreeBSD/FB13.1-AQM-L4S-SRC.git](https://github.com/MPTCP-FreeBSD/FB13.1-AQM-L4S-SRC.git)]([https://github.com/MPTCP-FreeBSD/FB13.1-AQM-L4S-SRC.git](https://github.com/MPTCP-FreeBSD/FB13.1-AQM-L4S-SRC.git)) accessed on 18 July 2024, A3C implementation and analysis: [[https://github.com/MPTCP-FreeBSD/FreeBSD-DRL-L4S.git](https://github.com/MPTCP-FreeBSD/FreeBSD-DRL-L4S.git)]([https://github.com/MPTCP-FreeBSD/FreeBSD-DRL-L4S.git](https://github.com/MPTCP-FreeBSD/FreeBSD-DRL-L4S.git)) accessed on 18 July 2024). We formulate the following research questions in our paper:  
1. How can the FreeBSD networking stack be modified to support L4S and integrate explicit congestion notification (ECN)? 2. What performance improvements can be observed in terms of latency, loss, and throughput by implementing L4S in FreeBSD? 3. How can the A3C algorithm be applied to dynamically adjust the base drop probability of L4S in response to varying network conditions? 4. What is the impact of using A3C to optimize the base drop probability of L4S on the performance of real-time applications? This paper is organized as follows. Section 2 presents the background and related work in state-of-the-art AQM algorithms and their ML-based approaches, Sections 3 and 4 present our system design and implementation of L4S in FreeBSD. Section 5 presents our experimental evaluation, and Section 6 analyzes the results and discuss key findings. Section 7 concludes the paper and outlines future work.