###### Abstract  
Bufferbloat is one of the leading causes of high data transmission latency and jitter on the Internet, which severely impacts the performance of low-latency interactive applications such as online streaming, cloud-based gaming/applications, Internet of Things (IoT) applications, voice over IP (VoIP), real-time video conferencing, and so forth. There is currently a pressing need for developing Transmission Control Protocol (TCP) congestion control algorithms and bottleneck queue management schemes that can collaboratively control/reduce end-to-end latency, thus ensuring optimal quality of service (QoS) and quality of experience (QoE) for users. This paper introduces a novel solution by experimentally integrate the low latency, low loss, and scalable throughput (L4S) architecture (specified by the IETF in RFC 9330) in FreeBSD framework with the asynchronous advantage actor-critic (A3C) reinforcement learning algorithm. The first phase involves incorporating a modified dual-queue coupled active queue management (AQM) system for L4S into the FreeBSD networking stack, enhancing queue management and mitigating latency and packet loss. The second phase employs A3C to adjust and fine-tune the system performance dynamically. Finally, we evaluate the proposed solution's effectiveness through comprehensive experiments, comparing it with traditional AQM-based systems. This paper contributes to the advancement of machine learning (ML) for transport protocol research in the field. The experimental implementation and results presented in this paper are made available through our GitHub repositories. active queue management; L4S; FreeBSD; deep learning; experience-driven; DualPI2 2020 1625 12 June 2020 18 July 2024 18 July 2024 23 July 2024 2  
PIE), which combined the merits of CoDel and PIE with a modified deficit round robin (DDR) scheduler. In FQ-CoDel, flows are assigned to a pool of internal queues, each managed independently by an instance of CoDel for buffer management. The DDR scheduler shares the outbound link capacity among the active queues, ensuring a fair allocation of resources. Likewise, the hybrid strategy \"FQ-PIE\" amalgamates the FlowQueuing aspect of FQ-CoDel with the PIE queue management. Both algorithms have shown exemplary capacity sharing between competing flows while achieving high throughput and low queuing delay. Despite their efficacy in throughput and queuing delay reduction, these delay-based algorithms are insufficient for the growing demand for low-latency and low-loss data transmission required for applications such as online gaming, high bitrate streaming, video conferencing, virtual reality, and cloud-based processing for real-time IoT applications [3; 4]. Presently, employing classic congestion control alongside cutting-edge active queue management (AQM) strategies such as FQ-CoDel [5], PIE [6], or DOCSIS PIE [7] yields an average latency ranging from 5 to 20 ms. Nevertheless, even with these advancements, latency at the 99 th percentile hovers around 20 to 30 ms, falling short of the stringent latency requirements of contemporary applications. This research explores avenues for enhancing latency performance in data transmission systems, catering to the evolving needs of modern digital environments. Low latency, low loss, and scalable throughput (L4S) represent a novel architecture paradigm based on the insight that the root cause of queuing delay lies in the capacity-seeking congestion mechanisms of senders rather than inherent issues within the queuing system itself. This architectural innovation is primarily facilitated towards incremental deployment, ensuring compatibility with the existing network infrastructures. Central to its design is introducing a mechanism enabling the integration of a new class of congestion controls and queue management within the L4S framework. This mechanism employs a modified explicit congestion notification (ECN) variant, facilitating their coexistence alongside 'Classic' congestion controls within shared network environments.