Symbolic Distillation for Learned TCP Congestion Control
Authors: S P Sharan
Wenqing Zheng
Kuo-Feng Hsu
Jiarong Xing
Ang Chen
Zhangyang Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: symbolic regression
TCP congestion control
efficiency
interpretability

Abstract: Recent advances in TCP congestion control (CC) have achieved tremendous success with deep reinforcement learning (RL) approaches, which use feedforward neural networks (NN) to learn complex environment conditions and make better decisions. However, such ``black-box'' policies lack interpretability and reliability, and often, they need to operate outside the traditional TCP datapath due to the use of complex NNs. This paper proposes a novel two-stage solution to achieve the best of both worlds: first to train a deep RL agent, then distill its (over-)parameterized NN policy into white-box, light-weight rules in the form of symbolic expressions that are much easier to understand and to implement in constrained environments. At the core of our proposal is a novel symbolic branching algorithm that enables the rule to be aware of the context in terms of various network conditions, eventually converting the NN policy into a symbolic tree. The distilled symbolic rules preserve and often improve performance over state-of-the-art NN policies while being faster and simpler than a standard neural network. We validate the performance of our distilled symbolic rules on both simulation and emulation environments. Our code is available at https://github.com/VITA-Group/SymbolicPCC.

Symbolic Distillation for Learned TCP Congestion Control
S Sharan', Wenqing Zheng', Kuo-Feng Hsu², Jiarong Xing", Ang Chen', Zhangyang Wang'
University of Texas at Austin "Rice University
(spsharan, w. w.zenn,,,.....,,....... atlaswang) @utexas. edu; (xh42, aoangl,jjiinggaggceen@rrorroo edu
angchen))rrice
Abstract
Recent advances in TCP congestion control (CC) have achieved tremendous success with deep reinforcement learning (RL) approaches, which use feedforward neural networks (NN) to learn complex environment conditions and make hetter decisions. However, such "black-box" policies lack interpretability and reliability, and often, they need to operate outside the traditional TCP datapath due to the use of complex NNs. This paper proposes novel two-stage solution to achieve the best of both worlds: first to train a deep RL agent, then distill its (over-) oover-)parmetteriee NN policy into white-box, light- rules in the form of symbolic expressions that are much easier to understand and to implement in constrained environments. At the core of proposal is novel symbolic branching algorithm that enables the rule to be aware of the context in terms of various network conditions, eventually converting the NN policy into symbolic tree. The distilled symbolic rules preserve and often improve performance over state-of-the-ar NN policies while being faster and simpler than standard neural network. We validate the performance of our distilled symbolic rules on both simulation and emulation environments. Our code is available Ihttt::////// battps://github wwwpiittww......ooeeeeo.e.. httpthh.w..oo..oooooooooooooo/oa
Introduction
Congestion control (CC) is fundamental Transmission Control Protocol (TCP) communication. Congestion occurs when the data volume sent to a network reaches or exceeds its maximal capacity, in which case the network drops excess traffic, and the performance unavoidably declines. CC mitigates this problem by carefully adjusting the data transmission rate based on the inferred network capacities, aiming to send data as fast as possible without creating congestion. For instance, classic and best-known strategy, Additive-Inccease/MMie Pocreaas/Muutipliccatvee Decrease (AIMD) [1. gradually increases the sending rate when there is no congestion but exponentially reduces the rate when the network is congested. It ensures that TCP connections fairly share the network capacity in the converged state.
Figure D shows an example where two TCP connections share a link between routers and When the shared link becomes bottleneck, the CC algorithms running on sources A and B will alter the traffic rate based on the feedback to avoid congestion. Efficient CC algorithms have been the bedrock for network services such as DASH video streaming, VolP (voice-over-IP), VR/AR games, and IoT (Internet of Things), which ride atop the TCP protocol.
However, it is nontrivial to design a gigh-performance CC algorithm. Over the years, tens of CC proposals have been made, all with different metrics and strategies infer and address congestion, and new designs are still emerging even today 2 3 There are two main challenges when designing a CC algorithm. 1 it needs to precisely infer whether the network is congested, and if so, how adjust the sending rate, based on only partial or indirect observations. Note that CC runs end hosts while congestion happens in the network, so CC algorithms cannot observe congestion directly. Instead, it can only rely on specific signals to infer the network status. For instance, TCP Cubic 4 uses packet loss as a congestion signal, and TCP Vegas (5) opts for delay increase. CC algorithms operate within the OS kernel, where the computing and memory resources are limited,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

and they need to make real-time decisions to adjust the traffic rates frequently (e.g., per round -tip time). Therefore, the algorithm must be very efficient. Spending a long time to compute an action will significantly offset network performance. Over the long history of congestion control, most algorithms are implemented with manually-designed heuristics, including New Reno [6] Vegas [ Cubic 4. and BBR [2] In TCP New Reno, for example, the sender doubles the number of transmitted packets every RTT before reaching a predefined threshold, after which it sends one more packet every RTT. If there is timeout caused by packet loss, it halves the sending rate immediately.
Unfortunately, manually crafted CCs have been shown to be sub-optimal and cannot support cases that escape the heuristics [7]. For exam- ple, packet bass-based CCs like Cubic 4 can- not distinguish packet drops caused by conges- tion or non-congestion-relttee events 77. Re- searchers have tried to construct CC algorithms with machine learning approaches to address these limitations [7-I1]. The insight is that the CC decisions are dependent on traffic patterns and network circumstances, which can be ex- ploited by deep reinforcement learning (RL) to learn a policy for each scenario. The learned policy can perform more flexible and accurate rate adjustments by discovering a mapping from experience, which can adapt to different network conditions and reduce manual tuning efforts.
App
App
User space
Symbolic RL agent
Kemel space
Router
Router
Source A
Destination A
Bottlenock link
ata
Feedback/A CK
Source B
Destination B
Figure I: Overview of a congestion control agent's 's role in the network. Multiple senders and receivers share single network link controlled by the agent, which dynamically modulates the sending rates conditioned on feedback from receivers.
Most notably, Aurora 7 deep RL framework for Performance-oriented Congestion Control (PCC), trains a well-established PPO 12] agent to suggest sending rates as actions by observing the network statistics such as latency ratio, send ratio, and sent latency inflation. It achieves competitive results on emulation environments Mininet [13] and Pantheon [14]. demonstrating the potential of deep learning approaches to outperform algorithmic, hand-crafted ones. Despite its immense success, Aurora being a neural network based approach, is essentially black-box to users or, in other words, lacks explicit declarative knowledge [13). They also require exponentially more computation resources than traditional hand-crafted algorithms such as the widely deployed TCP-CUBIC 4
1.1 Our Contributions
In this work, we develop a new algorithmic framework for performance-orrented congestion control (PCC) agents, which can II run as fast as classical algorithmic methods; (2) adjust the rate as accurately as data-driven RI methods; and 3 be simpler than the original neural network [16], potentially improving practitioners' understanding of the model in an actionable manner. We solve this problem by grasping the opportunity enabled by advances in symbolic regression 23). Symbolic regression bridges the gap between the infeasible search directly in the enormous symbolic algorithms space and the differentiable training of over wver-parameterized and un-interpretable neural networks.
At high level, one can first train an RL agent through gradient descent, then distill the learned policy to obtain the data-driven optimized yet simpler and casier-to-understand symbolic rules. This results in a set of symbolic rules through data-driven optimization that meets TCP CC's extreme efficiency and reliability demands. However, considering the enormous volume of discrete symbolic space, is challenging to learn effective symbolic rules from scratch directly. Therefore, in this paper, we adopt a two-stage approach: we first train deep neural network policy with reinforcement learning mimicking Aurora (7]. and then distill the resultant policy into numerical symbolic rules, using symbolic regression (SR).
As the challenge, directly applying symbolic regression out of the box does not yield sufficiently versatile expression that captures diverse networking conditions. We hence propose a novel branching technique for training and then aggregating number of SymbolicPCC agents, each of which caters to subset of the possible network conditions. Specifically, we have multiple agents, each called a branch, and employ a light "branch decider" to choose between the branches during deployment. In order to create the branching conditions partition the network condition space

---

into adjacent averoleaapping contexts, then regress symbolic rules in each context. With this modification, we enhance the expressiveness of the resulting SR equation and overcome the bias of traditional SR algorithms to output rules mostly using numerical operators. Our concrete technical contributions are summarized as follows:
We propose a symbolic distillation framework for TCP congestion control, which improves upon the state-of-the-ar RL solutions. Our approach, SymbolicPCC. consists of two stages: first training an RL agent and then distilling its policy network into ultra-compact and simple nules in the symbolic expression form.
We propose novel branching technique that advances existing symbolic regression techniques for training and aggregating multiple context-dependent symbolic policies, each of which specializes for its own subset of network conditions. A branch decider driven by light-weight classification algorithms determines which symbolic policy to use.
Through our simulation and emulation experiments, we show that SymbolicPCC achieves highly competitive or even stronger performance results compared to their teacher policy networks while running orders of magnitude faster. The presented model uses a tree structure that is light-weight, and could be simpler for practitioners to reason about and improve manually while narrowing their performance gap.
2 Related Works
User spate
User space
User space
App
App
App
Some proposals use packet loss as signal for network congestion, e.g., Cubic 4 Reno [24]. and
ACSa
RET
learning
Onrine
Mar
Untley
Sanding.con
Joss
NewReno 66; while others rely on
Mo Bot
Sending rallo
the variation of delay, e.g., Vegas [5].
Kernel space
Mi oilly dncipeed hearct::
Kernal space
space
or combine packet loss and delay [25] 226.. Different CC techniques special- ized for datacenter networks are also
ANO
toth saly and bittereite baniuens
Dod morving
Settred usil
Sutter than outtal
proposed 27
Figure Overview of Conventional Baselines
Researchers have also investigated the use of machine learning to construct better heuristics. Indigo 10] and Remy
use offline learning to obtain high-peeformance CC algorithms. PCC [28] and PCC Vivace g opt for online learning to avoid any hardwired mappings between states and actions. Aurora 7 utilizes deep reinforcement learning to obtain a new CC algorithm running in userspace. Orca improves upon Aurora and designs userspace CC agent that infrequently configures kernel-space CC policies. Our proposal further improves these work.
11
At the same time, Symbolic regression methods 17 23 have recently emerged for discovering underlying math equations that govern some observed data. Algorithms with such property are more favorable for real real-world deployment as they output white-box rules. [18] use genetic programming based method while 20 uses a recurrent neural network perform search the symbolic space.
We thus propose to synergize such numerical and data-driven approaches using symbolic regression (SR) in the congestion control domain. We use SR by following post-hoc method of first training an RL algorithm then distilling it into its symbolic rules. Earlier methods that follow a similar procedure do exist, e.g., (29) distills the learned policy as a soft decision tree. They work on visual RL where the image observations are coarsely quantized into 10 x 10 cells, and the soft decision tree policy learned over the 100 dimensional space. (30 also aims learn abstract rules using common sense-based approach by modifying Q learning algorithms. Nevertheless, they fail to generalize beyond the specific simple grid worlds they were trained in. [1I learns from boolean feature sets, (32 directly approximates value functions based on state- tate-trraniiton model, [3 optimizes risk-seeking policy gradients. Other works on abstracting pure symbolic rules from data include attention based methods (34, visual summary 355 reward decomposition 36) causal models [3), markov chain
38, and case-based expert-behaviou retrieval 21 22 39
3 Methodology
Inspired from the idea of "teacher-student knowledge distillation" " 42 our symbolic distillation technique is two-staged -firt train regular RL agents (as the teacher), then distill the learned policy
3

---

Training SymbolicPCC
Branching Boundary Conditions
train specialioed R. agentss
Evaluation Evaluation/Deploymai DDepoymment
SymbolicPOC
Rl. Raseline
RL
RL
RL 3
RL
dustrrigg
Symbolic Policy
branch
algeritim
collact
reward Mustaring
collect rolioues
(branched)
obear obrenvations
Branch
Offiine Dotaset of Trajectories
decider
Offline Datasets of Trajectories
observations
symbelic distitlation
symbolic zymbolli disiilaation
Symbolic Policy (unibranched)
5ym
Sym
Synd
Symbolic Policy (branched)
The proposed SymboliePCC training and evaluation technique: A baseline RL agent first trained then evaluated numerous times with the roll-outs being saved. Directly distilling out from this data provides a baseline symbolic policy. A light-weight clustering algorithm is used to cluster from the roll-out dataset, non-overlapping subsets of network conditions (aka. branching conditions) that achieve similar return. Separate RL agents are then trained on each of these network contexts and distilled into their respective symbolic rules. During the evaluation, the labels from the clustering algorithm are re-purposed to classify which branch to be taken given the observation statistics. The chosen symbolic branch is then queried for the action.
Figure 3:
networks into their white- box symbolic counterparts (as the student). In Section 3.1 we follow 17''s approach in Aurora and the training of teacher agents on the PCC-RL. gym environment. We also briefly discuss the approach of applying symbolic regression create light-weight numerical-driven expression that approximates a given teacher's behavior. In Section 3.2 we look at the specifics of symbol spaces and attach internal attributes to aid long-term planning. Finally, in Section B.3 we discuss our novel branching algorithm as method for training, then ensembling multiple context- dependent symbolic agents during deployment.
3.1 Preliminaries: The PCC-RL Environment and the Symbolic Distillation Workflow
PCC-RL 7 is an open-source RL testbed for simulation of congestion control agents based on the main playground. It formulates congestion control as a sequential decision making problem. Time is first divided into multiple periods called MIs (monitor intervals), following (28). At the onset of each MI, the environment provides the agent with the history of statistic vector observations over the network, and the agent responds with adjusted sending rates for the following MI. The sending rate remains fixed during single MI.
popular OpenAl Gym [43] framework. We adopt it
The network statistics provided as observations to the congestion control agent are I the latency inflation, 2 the latency ratio, and 33 the sending ratio. The agent guided by reward signals based on its ability to react appropriately when detecting changes and trends in the vector statistics of the PCC-RL environment. It is provided with positive return for higher values of throughput (packets/second) while being penalized for higher values of latency (seconds) and loss (ratio of sent vs. acknowledged packets).
Training of Teacher Agents: We first proceed to train RL agents using the PPO algorithm [2] similar Aurora [7] in the PCC-RL gym environment till convergence. Although they statistically perform very well 7. the PPO agents are entirely black-boxes; this makes it difficult to explain its underlying causal rules directly. Also, their aver-parameterized neural network forms incur high latency. Hence, we choose to indirectly learn the symbolic representations using a student-teacher type knowledge distillation approach based on the teacher's (in this case, the RL agent) behaviors.
Using the teacher agents, we collect complete trajectories, formally known as roll-outs RL, an inference dete--detemiiistiic actions are enforced. The observations and their corresponding teacher actions are MI-aligned and stored an offline dataset. Note that this step is only performed once at the start the distillation procedure and reused in each of its iterative steps. A search space of operators and operands is also initialized (details are discussed
Distillation of Student Agents:

---

shortly in Section 3.2). Guesses for possible symbolic relations are taken, composed of random operators and operands from their respective spaces. The stored observation trajectories are then re-evaluated based on this rule to output corresponding actions. The cross-entropy loss with respect to the teacher model's actions from the same dataset is used as feedback. This feedback drives the iterative mutation and pruning following genetic programming technique 45]. The best candidate policies are collected and forwarded to the next stage. If the tree fails to converge or does not reach a specific threshold of acceptance, the procedure is restarted from scratch. Our symbolic distillation method is discussed with further details in our Appendix.
3.2 A Symbolic Framework for Congestion Control
Unlike visual RL [46], the PCC observation space is vector- based, hence we directly plug them into the search space of our numerical-driven symbolic algorithm. We henceforth call these observations as vector statistic symbols. The distillation procedure as described earlier learns to chain these vector statistic symbols using a pre-defined operator space. Specifically, we employ three types of numerical operators. The first type of operators are arithmetic based which include +, -,,,, type of operators are Boolean logic, such as is(r < y), is(r S v), is(z y). a b, a b, and
Defining the Symbol Space for CC:
x, /, sin, cos, tan, cot, (.)². V exp, log, H-| The second
We also utilize a third type of high level operators - namely the slope_of (observation history) which provides the average slope of an array of observations, and get_value_as (observation history, index). The slope operator is especially useful when trying to detect trends of a specific statistic vector over the provided monitor interval. For instance, identifying latency increase or decrease trends serves as one of the crucial indicators for adjusting sending rates. Meanwhile the index operator is observed from our experiments to be implicitly used for immediate responding-iie. based on the latest observations.
We note that the underlying decision procedure of the policy network could be efficiently represented in the form of a figeliiclitty tree-shaped form similar Figure 4 This decision tree contains said condition nodes and action nodes. Each condition node forks into two leaf nodes based on the Boolean result of its symbolic composition.
In addition to having these operators and operands as part of the symbolic search space, we also attach a few attributes/flags to the agent which are shared across consecutive MI processing steps and help with long-term planning. One behavior in our SymbolicPCC agents is to use this attribute for remembering if the agent is in the process of recovering from a network overload or if the network stable. Indeed, a more straightforward option for such "multi-MI tracking" would be to just provide a longer history of the vector statistics into the searching algorithms. But this quickly becomes infeasible due to an exponential increase of the possible symbolic expressions with respect to the length of vector statistic symbols.
Attributes for Long Term CC Planning:
3.3 Novel Branching Algorithm: Switching between Context-Dependeri Policies
Unlike traditional visual RL environments, congestion control is more demanding task due to the variety of possible network conditions. The behavior of the congestion control agent will improve if its response is conditioned on the specifie network context. However, this context cannot be known by the congestion control agent, the traditional algorithms such as TCP Cubic 4 are forced to react in slow and passive manner support both slow-paced and fast-paced conditions. Such notion of context splitting and branching for training specialized agents can be compared to earlier works in multi-task RL. Specifically, divide-and-Conquee
47 learns separate complex tasks into a set of local tasks, each of which can be used to learn separate policy. Distral [4] proposes optimizing a joint objective function by training a "distilled" policy that captures common behavior across tasks.
Hence, we propose to create n non-overlapping contexts for different network conditions, namely- bandwidth, latency, queue size, and loss rate. We then train n individual RL agents in the PCC Gym by exposing them only to their corresponding network conditions. We thus have a diverse set of teachers which are each highly performant in their individual contexts. Following the same approach as described in Section [. each of the agents are distilled-each one called a branch. Finally, during deployment, based on the inference network conditions, the branch with the closest matching boundary conditions/contexts is selected and the corresponding symbolic policy is used.
Partitioning the Networking Contexts: A crucial point to note in the proposed branching pro- cedure is to identify the most suitable branching context boundary values. In other words, the best boundary conditions for grouping need to be statistically valid, and plain hand-crafted boundaries are
5

---

siat &
LA TE
Jar ontee 185
HOOLLAS LAM
Jun srone
Kax
L.R.
5.8.
Sard Dato
S.L.T.
Snd viuiio
D-11
Figure 4: A distilled symbolic policy from the baseline RL Agent in the PCC-RL Environment. Condition nodes are represented as rectangular blocks and action nodes as process blocks.
not optimal. This because we do not have ground truths of any of the network conditions 49), let alone four of them together. Therefore, we first use trained RL agent on the default (maximal) bounds of network conditions (hereinafter called the "baseline" agent). We then evaluate the baseline agent on multiple regularly spaced intervals of bandwidth, latency, queue size, and loss rate and store their corresponding return as well as observation trajectories. To create the optimal groupings. we simply use KMeans [50] to cluster the data based on their return. Due to the inherent proportional relation of difficulty (or in this case the ballpark of return) with respect to a network context, clear boundaries for the branches can be obtained by inspecting the extremes of each network condition within a specific cluster. Our experimentally obtained branching conditions are further discussed in Section 2 and T
Since the network context is not known during deployment, one needs a branch decider module. The branch decider reuses cluster labels from the training stage for K Nearest Neighbors [5T] classification. The light- distance-based metric is used to classify the inference- time observation into one of the training groupings and thereby executing the corresponding branch's symbolic policy. Figure iillustrates our complete training and deployment techniques.
Branch Decider:
Lastly, in order to support branching effectively, we have yet another long-term tracking attribute that stores a history of branches taken in order to smooth over any erratic bouncing between branches which are in non-adjacent contexts.
4
Experimental Settings and Results
Next, we discuss the abstract rules uncovered by SR, and validate the branching contexts. Sections 4.3 4.4 and 4.5 we provide emulation results on Mininet [13], widely-used network emulator that can emulate a variety of networking conditions. Lastly in Section [4.]] we compare the compute requirements and efficiencies of SymbolicPCC with conventional algorithms, RL-driven methods as well as their pruned and quantized variants. More hyperparameter details are in our Appendix.
4.1 Interpreting the Symbolic Policies
The baseline symbolic policy distilled from the baseline RL agent is represented in its decision tree form in Figure One typical CC process presented by the tree is increasing the sending rate until the network starts to "choke" and then balancing around that rate. This process guided with a series of conditions regarding inflation and ratio signals, marked with circled numbers in Figure The detailed explanation is in the following.
Condition node 1 checks whether the vector statistic symbols are stable-namely whether the latency inflation is close zero, while latency ratio and send ratio are close to one.

---

Table 1: The baseline network conditions and resultant branching boundary values (contexts) for each branch after clustering. The return centroid refers to the return value at cluster center of that specific branch.
Branch
Return Centroid
Bandwidth (pps)
Latency (sec)
Queue Size (packets)
Loss Rate (%)
Baseline
100 500
0.05 0.5
2 - 2981
0.00 0.05
Branch 1
95.84
100 200
0.35 0,5
2 2981
0.04 - 0,05
Branch 2
576.57
250-250
0.25 0.35
2981
0.02 0.03
Branch 3
1046.46
0.15 0.25
2-2981
0.02 0.03
Branch 4
1516.70
350 500
0.05 0.15
2- 2-2981
0.00 0.02
The sending rate starts to grow if the condition holds. Condition node identifies if the network is in a over-utilized status slope_of of (latency inflation) increasing as the key indicator. It the condition is true, the acceleration of sending rate will be reduced appropriately. On the other hand, condition node 33) is activated when the initial sending rate is too low or has been reduced extensively due to 22 is evaluated when major network congestion starts to occur due to increased sending rates from the earlier condition nodes. It checks both latency inflation and latency ratios in an increasing state. Its child nodes start reducing the sending rates and also flip the internal state attribute 1. The latter is used to track if the agent is recovering from network congestion.
On the "False" side of 6 (i.e. internal state 1). 7 and 8 realize two stages of recovery, where the latency inflation ratio starts plateauing and then starts reducing. [II) indicates that stable conditions have been achieved again and the agent is at an optimal sending rate. The internal state is flipped back again 0 after this recovery.
4.2 Inspecting the Branching Conditions
As discussed in Section 3.3 a light-weight clustering algorithm divides the network conditions into multiple non-overlapping subsets. Table summmrrzess the obtained boundary values. The baseline agent trained on all possible bandwidth, latency, queue size, and loss rate values, as depicted in the first row. During the evaluation, bandwidth, latency, and loss rate are tested on linearly spaced values of step sizes 50, 0.1, and 0.01, while queue sizes are exponentially spaced by powers of e² respectively. The return of the saved roll-outs are clustered using K-Means Clustering, and the optimal cluster number found to be 4 using the popular elbow curve (52) and silhouette analysis [53 methods. By observing the maximum and minimum of each network condition individually in the clusters, respective boundary values are obtained. A clear relation discovered is that higher bandwidths and lower latencies are directly related to higher baseline return.
Remark 1: Exceptions for non-overlapping contexts. It also to be noted that no such trend was found between the queue size and return, and hence all the 4 resultant branches were given the same queue size. A similar exception was made for the loss rates of Branches and 3.
Remark 2: Interpreting the symbolic policies branches. All the 4 distilled symbolic trees from the specialized RL agents possess high structural similarity and share similar governing rules as to that of the baseline agent in Section 4.1 They majorly differ in the numerical thresholds and magnitudes of action nodes, i.e., by varying their "reaction speeds" and "reaction strengths", respectively.
4.3 Emulation Performance on Lossy Network Conditions
The ability to differentiate between congestion-inducss and random losses is essential to any PCC agent. Figure | shows 25-second trace of throughput on a link where 1% of packets are randomly dropped [54). As the link's bandwidth is set to 30 Mbps, the ideal congestion control would aim to utilize it fully as depicted by the gray dotted line. Baseline SymboliePCC shows near- ideal performance with its branched version pushing boundaries further. In contrast, conventional algorithms, especially TCP CUBIC [4) repeatedly reduces its sending rates as a response to the random losses. Quantitative measures of mean square error with respect the ideal line are provided
IItterestinlly the figure shows that BBR has rate drop around 11th second. This is a limitation of the BBRv1 design it reduces sending rate if the min_rtt rtt has not been seen in 10s, which is triggered because the RTT in setup is very stable.
7

---

in Table 2
"Lossy Sptt ". This result proves that SymbolicPCC can effectively differentiate between packet loss caused by randomness and real network congestion.
4.4 Emulation Performance under Network Dynamics
Unstable network conditions are common in the real world and this test benchmarks the agent's 's ability to quickly respond to network dynamics. symbolic agent's 's ability to handle such conditions. The benefits of our novel branching algorithm, as well as switching between agents specializing in their own network context, is clearly visible from faster response speeds. In this case, the link was configured with its bandwidth al- ternating between 20 Mbps and 40 Mbps every 5 seconds with no loss. Quantita- tive results from Table 2 show the mean square error with respect to the ideal CC as "Unstable opt.."
aptimum
TCP OPAN Syn trorcred
FCC Vivace
Symbole loass that
DBR
(eggur/5b]showe our
Throuphous intrre)
20
25
Time
(a) 25-second thoughput trace for TCP CUBIC, CCC--ivace, BBR, Aurora, and our SymbolicPCC variants on a 30 Mbps bandwidth link with 2% random loss, 30 ms latency, and a queue size of 1000.
Thmuuteeu mnoped
4.5 Link
Utilization and Network Sensitivities
Link utilization measured from the server side is defined as the ratio of av-
Opommm
Aurrn (bocel nel
TOP CUDC
Symto KPCC (trantell
PCC Vouce - SymbolsPCC (bandire)
207
erage throughput over the emulation pe-
12
25
Time
riod to the available bandwidth. A sin-
gle link is first configured with defaults of 30 Mbps capacity, 30 ms of latency, a 1000-packet queue, and 0% random loss. To measure the sensitivity with respect to specific condition, is independently varied while keeping the rest of the con- ditions constant. An ideal CC preserves high link utilization over the complete range of measurements. From Figure 6 it is observed that our branched SymbolicPCC provides near-capacity link-utilization at most tests and shows improvement over any of the other algorithms.
(b) A 25-second throughput trace for TCP CUBIC, PCC Vi- vace, BBR, Aurora, and our SymbolicPCC variants on link altemaaiin between 20 and 40 Mbps every seconds with 0% random loss, 30 ms latency, and queue size of 1000.
Figure 5: Emulation on different conditions.
4.6 Efficiency and Speed Comparisons
Since TCP congestion control lies on the fast path, efficient responses are needed from the agents. Due to its GPU compute requirements and slower runtimes, RL-based approaches such as Aurora are constrained in their deployment settings (e-g., userspace decisions). On the other hand, our symbolic policies are entirely composed of numerical operators, making them structurally and computationally minimal. From our results Table adding the branch decider incurs a slight overhead as compared to the non-branched counterpart. Nevertheless, it is preferable due to its increased versatility in
Piice
SANEHEPCE Soneeel
Syrtugnnc draslell
1.0
(a) Bondwidth SensRivity
1,00
(h) Latency Sensllvity
103 Butter Size Senuitivity
(4) Loss Rate Sensitiiit
0
9.4
0.4
235
1.2
02
3.0
0.0
40
6.30
(33)
Probabi
Figure 6: Link-utilization trends as a measure sensitivities of bandwidth, latency, queue size, and loss rate. Higher values are better.

---

Table 2: Efficiency and speed comparison of congestion control agents. Note that the ideal values for Lossy Thpt. and Lossy ET hpt. are 30 and 750 respectively (refer Figure Sap.
Algurithm
Type
FLOPS (1)
Runtime (1)
Lossy Täpe. (1)
Lossy 27 (1)
Losvy A (1)
Oscillating Diee (4)
TCP CUBIC
Conveniional
100
1:27
35.01 01
685.02
125,07
PCC Viviee
Conventionl
100
422
235.68
440.55
186.76
BBR
Conventional
10
25.76
669.91
92,95
123.82
Aurora (boseline)
RL Based
1488
864
27.55
T36.20
$3,22
(50% yruned)
RL Based
781
27:03
709 20
27.57
61.85
KL- Based
709
42
680686
703801
pouned)
KL. Based
703
25.97
6182.5
83,05
(quamizal)
RI-Based
SymbolicPCCC (beneline)
Symbolic
SymbolicPOC (branchod)
Symbolic
different network conditions, as validated by Mininet emulation results. SymbolicPCC achieves x
faster execution times over Aurora, being reasonably comparable to PCC Vivace and TCP CUBIC. We also compare global magnitude pruned and dynamically quantized versions of Aurora. Although these run faster than their baseline versions, they come at the cost of worse CC performance.
5 Discussions and Potential Impacts of SymbolicPCC
Interpretability universal boon for ML? In the PCC domain, the model interpretability is linked to the wealth of domain knowledge. By distilling a black-box neural network into white-box symbolic rules, the resulting rules are easier for the network practitioners digest and improve.
It may be somewhat surprising that the distilled sym- bolic policy outperforms Aurora. A natural question arises it due to generalization amplification that sometimes happens for distillation in general or if it is due to symbolic representation. We hypothesize that the performance of a symbolic algorithm boils down to the nature of the environment is employed in. Congestion control is predominantly rule-based, with deep RL models brought to devise rules more complex and robust than hand-crafted ones through iterative interaction. is only natural to observe that symbolic models outperform such PCC RL models when the distillation is composed of a rich operator space and dedicated policy denoising and pruning stages to boost their robustness and compactness further. To justify this, in 3 we analyze the performance obtained by decoupling distillation and symbolic representation: we first distill black-box NN half the size of Aurora ("typical KD") and then further perform symbolic distillation on it.
Table 3: Decoupling: symbolic alone helps generalization.
Avg. return (?)
Aurora
832
Black box (50%) Aurora White-box dist, from above model
687
On possible limitations. We have specifically focused on TCP congestion control as the problem setting,-e, e.g., the return clustering and reward design. Specific modifications are needed before the approach could be applied to other RL domains.
Conclusion and Future Work
This work studies the distillation of NN-based deep reinforcement learning agents into symbolic policies for performance-oriene congestion control in TCP. Our branched symbolic framework has better simplicity and efficiency while exhibiting comparable and often improved performance over their black-box teacher counterparts on both simulation and emulation environments. Our results point towards a fresh direction to make congestion control extremely light-weight, via a symbolic design. Our future work aims at more integrated neurosymbolic solutions and faster model-free online training/linettuning for performance-orienter congestion control. Exploring the fairness of neurosymbolic congestion control is also an interesting next step. Besides, we also aim to apply symbolic distillation to wider range of systems and networking problems.
Acknowledgment
A. Chen and Z. Wang are both in part supported by NSF CCRI-2016727. A. Chen is also supported by NSF CNS-2106751. Z. Wang also supported by US Army Research Office Young Investigator Award W91 INF2112000

---

References
[1] Dah-Ming Chiu and Raj Jain. Analysis of the increase and decrease algorithms for congestion avoidance in computer networks. Computer Networks and ISDN systems, 17(1):1-14, 1989.
[2] Neal Cardwell, Yuchung Cheng, C Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. BBR: congestion-based congestion control. Communications of the ACM, 60(2):58-66, 2017.
[3] Gautam Kumar, Nandita Dukkipati, Keon Jang. Hassan MG Wassel, Xian Wu, Behnam Montaz- eri, Yaogong Wang, Kevin Springborn, Christopher Alfeld, Michael Ryan, et al. Swift: Delay is simple and effective for congestion control in the datacenter. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, pages 514-528, 2020.
[4] Sangtae Ha, Injong Rhee, and Lisong Xu. Cubic: new tep- rep-friendly high-speed tep variant. ACM SIGOPS operating systems review, 42(5):64-74, 2008.
[5] Lawrence S Brakmo, Sean W O' O'Malley, and Larry L Peterson. Tep vegas: New techniques for congestion detection and avoidance. In Proceedings of the conference on Communications architectures, protocols and applications, pages 24-35, 1994.
[6] Sally Floyd, Tom Henderson, and Andrei Gurtov. Rfc3782: The newreno modification to tcp's fast recovery algorithm, 2004.
[7] Nathan Jay, Noga Rotman, Brighten Godfrey, Michael Schapira, and Aviv Tamar. A deep reinforcement learning perspective internet congestion control. In International Conference on Machine Learning, pages 3050-3059. PMLR, 2019.
[8] Soheil Abbasloo, Chen- Yu Yen, and H Jonathan Chao. Classic meets modern: A pragmatic learning-basse -ased congestion control for the internet. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, pages 632 022-647,55 2020.
[9] Mo Dong, Tong Meng, Doron Zarchy, Engin Arslan, Yossi Gilad, Brighten Godfrey, and Michael Schapira. PCC vivace: Online-learing congestion control. In Proc. NSDI, 2018.
[10] Francis Y Yan, Jestin Ma, Greg D Hill, Deepti Raghavan, Riad S Wahby, Philip Levis, and Keith Winstein. Pantheon: the training ground for internet congestion-control research. In Proc. ATC, 2018.
[11] Keith Winstein and Hari Balakrishnan. Top ex machina: Computer-generatte congestion control. ACM SIGCOMM Computer Communication Review, (3(4))123-133 2013.
[12] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xiv:1707.0644 2017.
[13] Ramon R Fontes, Samira Afzal, Samuel HB Brito, Mateus AS Santos, and Christian Esteve Rothenberg. Mininet-wifi: Emulating software-defined wireless networks. In 2015 11th International Conference on Network and Service Management (CNSM), pages 384-389, IEEE, 2015.
[14] Francis Y Yan, Jestin Ma, Greg Hill, Deepti Raghavan, Riad S Wahby, Philip Levis, and Keith Winstein. Pantheon: the training ground for internet congestion-contry research. In 2018 [USENIX} Annual Technical Conference ({ (ATC) 18), pages 731-743, 2018.
[15] Andreas Holzinger. From machine learning to explainable ai. In 2018 world symposium on digital intelligence for systems and machines (DISA), pages 55 55-66. IEEE, 2018.
[16] Ajay Kumar Jaiswal, Haoyu Ma. Tianlong Chen, Ying Ding, and Zhangyang Wang. Training your sparse neural network better with any mask. In International Conference on Machine Learning, pages 9833-9844. PMLR, 2022.
[17] Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. science, 324(5923):11-85, 2009.
10

---

[18] Miles Cranmer, Alvaro Sanzhez-Gonzalez. Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. arXiv preprint arXiv: /287, 2020.
[19] Miles Cranmer. Pysr: Fast & parallelized symbolic regression in python/julia. Gitllub repository, 2020.
[20] Brenden K Petersen, Mikel Landajuela Larma, T Nathan Mundhenk, Claudio P Santiago, Soo K Kim, and Joanne T Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. arXiv preprint arXiv:1912.048777 2019.
[21] Zhiwen Fan, Tianlong Chen, Peihao Wang, and Zhangyang Wang. Cadtransformer: Panoptic symbol spotting transformer for cad drawings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10986-10996, 2022.
[22] Wenqing Zheng, Tianlong Chen, Ting-Kuet Hu, and Zhangyang Wang. Symbolic learning to optimize: Towards interpretability and scalability. arXiv preprint arXiv:2203.0653 2022.
[23] Pierre-Alexandre Kamienny, Stéphane Ascoli, Guillaume Lample, and François Charton. End-to-end symbolic regression with transformers. arXiv preprint arXiv:2204. 10532, 2022.
[24] Van Jacobson. Congestion avoidance and control. ACM SIGCOMM computer commumication review, 18(4)311-329, 1988.
[25] Kun Tan, Jingmin Song, Qian Zhang, and Murad Sridharan. A compound tep approach for high-speed and long distance networks. In Proceedings INFOCOM, 2006.
[26] Saverio Mascolo, Claudio Casetti, Mario Gerla, Medy Y Sanadidi, and Ren Wang. Tep westwood: Bandwidth estimation for enhanced transport over wireless links. In Proceedings of the 7th annual international conference on Mobile computing and networking, pages 287-297,
2001.
(27] Mohammad Alizadeh, Albert Greenberg, David A Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. Data center tep (detep). In Proceedings of the ACM SIGCOMM 2010 Conference, pages 63-74, 2010.
[28] Mo Dong, Qingxi Li, Doron Zarchy, P Brighten Godfrey, and Michael Schapira. (PCC): Re- architecting congestion control for consistent high performance. In 12th {USENIX) Symposium on Networked Systems Design and Implementation (!NSDI) 15), pages 395-498- 2015.
[29] Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Nowé, Tim Miller, Rosina Weber, and Daniele Magazzeni. Distilling deep reinforcement learning policies soft decision trees. In Proceedings of the IJCAI 2019 workshop explainable artificial intelligence, pages 1-6, 2019.
[30] Artur d' Avila Garcez, Aimore Resende Riquetti Dutra, and Eduardo Alonso. Towards symbolic reinforcement learning with common sense. arXiv preprint 1oXiv:1804. 08557, 2018.
[31] Andrea Dittadi, Frederik K Drachmann, and Thomas Bolander. Planning from pixels in atari with learned symbolic representations. arXiv preprint arXiv:2012.09126, 2020.
[32] Jiří Kubalík, Jan Žegklitz, Erik Derner, and Robert Babuška. Symbolic regression methods for 2019.
reinforcement learning. arXiv preprint
[33] Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob Pettit, and Daniel Faissol. Discovering symbolic policies with deep reinforcement learning. In International Conference Machine Learning, pages 5979-5988 PMLR, 2021.
[34] Wenjie Shi, Shiji Song, Zhuoyuan Wang, and Gao Huang. Self-supervised discovering of causal features: Towards interpretable reinforcement learning. arXiv preprint arXiv:2003.07069) 2020.
[35] Pedro Sequeira and Melinda Gervasio. Interestingness elements for explainable reinforcement learning: Understanding agents' capabilities and limitations. Artificial Intelligence, 288: 103367, 2020.
11

---

[36] Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop on Explainable Artificial Intelligence, 2019.
[37] Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 2493-2500, 2020.
[38] Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2514-2521, 2019.
[39] Santiago Ontanón, Kinshuk Mishra, Neha Sugandh, and Ashwin Ram. Case-based planning and execution for real-time strategy games. International Conference on Case-Based Reasoning. pages 164-178. Springer, 2007.
[40] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, (29(6))1789-188 2021.
[41] Wenqing Zheng, Edward W Huang, Nikhil Rao, Sumeet Katariya, Zhangyang Wang. and Karthik Subbian. Cold brew: Distilling graph node representations with incomplete or missing neighborhoods. arXiv preprint arXiv:2 /1.1//11.4400 2021.
[42] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang- Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study, arXiv preprint arXiv:2/04.00666, 2021.
[43] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint
2016.
[44] Alan M Turing. Computing machinery and intelligence. In Parsing the turing test, pages 23-65. Springer, 2009.
[45] Richard Forsyth. Beagle-a a darwinian approach to pattern recognition. Kybernetes, 1981.
[46] Alexander Sieusahai and Matthew Guzdial. Explaining deep reinforcement learning agents in the atari domain through surrogate model. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 17, pages 82-90, 2021.
[47] Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. arXiv preprint urXiv:/7/1 1711.09141..4... 2017.
[48] Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in neurul information processing systems, 30, 2017.
[49] Nathan Jay. Continued development of internet congestion control: Reinforcement learning and robustness testing approaches. 2019.
[50] James MacQueen al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pages 281 1917 Oakland, CA, USA, 1967.
[51] Evelyn Fix and Joseph Lawson Hodges. Discriminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Review/Revue Internationale de Statistique, 2733):238247, 1989.
[52] Robert L Thorndike. Who belongs in the family? Psychometrika, 18(4):267-276, 1953.
[53] Peter Rousseeuw. Silhouettes: a graphical aid the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
[54] Danyang Zhuo Monia Ghobadi, Ratul Mahajan, Klaus-Tycho Förster, Arvind Krishnamurthy, and Thomas Anderson. Understanding and mitigating packet corruption in data center networks. In Proceedings of the SIGCOMM Conference, 2017.
12


Reviews:
[{"review_id": "m-uq0_xEia", "reviewer": "Reviewer_W18N", "summary": "The authors propose an efficient and interpretable framework for TCP congestion control. Specifically, they first train an RL agent to discover good teacher policies, then distill such policies into students, which are designed to be compact and interpretable using symbolic decision trees. They also split trajectories using Kmeans into contexts, and combine context-specific policies with context regression to further improve performance (essentially using the idea of Multi-task RL).  Extensive experiments demonstrate the effectiveness of the proposed approach.", "questions": "1. For trajectory and context clustering phase, why can trajectory returns be clustered in the first place? What is the motivation for it? What semantic meanings do different trajectory returns have?\n\n2. How specifically are rollout trajectories converted into symbolic decision trees (looks like this is based on a standard decision tree process using the rollout actions as input data)? How is tree pruning specifically done? These details should be addressed, using formal mathematics, at least in the supplementary.\n\n3. What is the environment input state space and action space? What semantic meaning does the action space represent? The authors do not study a common setting like vision-based RL, so these should be briefly addressed.\n\nMinor:\n\n\"Rewards\" should be \"Return\" in this work. The former means single-step reward, while the latter means episode (aggregated) return.", "limitations": "The trajectory clustering part is specific to the environment and reward design authors are using. It is not a general approach that can be applied to any environment. This should be addressed in the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The authors combine RL with symbolic decision tree in the novel application of TCP congestion control. Experiments demonstrate that the proposed approach is very effective and significantly outperforms previous work.\n\nWeaknesses: The paper lacks some formal descriptions of technical details in methodology. Some motivations are not clear and limitations are not addressed. Please see details below.\n\nOverall the methodology does not seem quite novel. The context split and branching process is essentially existing ideas in multi-task RL, and works along this line should be cited [1,2]. However, the application is novel and the approach seems effective, so I still recommend borderline acceptance.\n\n[1] Teh, Yee and Bapst, Victor and Czarnecki, Wojciech M and Quan, John and Kirkpatrick, James and Hadsell, Raia and Heess, Nicolas and Pascanu, Razvan: Distral: Robust multitask reinforcement learning. *Advances in neural information processing systems*, 2017.\n\n[2] Ghosh, Dibya and Singh, Avi and Rajeswaran, Aravind and Kumar, Vikash and Levine, Sergey: Divide-and-conquer reinforcement learning. *International Conference on Learning Representations*, 2018.\n\n**Post Rebuttal**: Thanks for the authors' rebuttal! The added details addressed most of my concerns. Overall, this paper is an application paper that combines existing approaches effectively to a novel domain. While the methodology has limitations, the application itself is novel, and I'm maintaining a stance between borderline and weak accept and increasing my confidence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fuGd6yvVYh5", "reviewer": "Reviewer_sAC3", "summary": "This work deals with the problem of congestion control in TCP communications. Specifically, the work studies policies for TCP senders, which govern the rate at which each sender sends packets. \n\nThis problem is a partially observed, multi-agent game as each agent does not have a full picture of the network conditions and instead has to use proxy variables like the presence of a dropped packet to make decisions. Previous solutions have ranged from exploiting manually designed heuristics, to online learning, to deep RL-based methods. \n\nThe authors relate that for this problem there is often a tradeoff between agent quality and runtime; while previously developed deep RL agents can well-manage congestion they are too computationally expensive to run in practical settings, which are generally compute constrained. \n\nThis work introduces an algorithm claimed to have the best of both worlds, with both a high quality rate determination policy that is also computationally efficient. The proposed method consists of training a DRL agent with PPO that has an accurate rate determination policy, and then distilling the agent into a simple rule-based agent with symbolic regression. The authors train on an OpenAI Gym environment TCP sending setting, and evaluate on mininet.", "questions": "Questions are listed in strengths and weaknesses.", "limitations": "The authors list Section 5 as the section in which limitations are dealt with but as far as I can tell none are addressed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper presents a performant, computationally-efficient method that could generalize to solving RL problems in other domains. The agent learning algorithm is clear, and the setup is mostly clear. However, the evaluation and motivation sections need further contextualization.\n\nEvaluation: This work requires a more thorough description, and motivation, for the evaluation. Two facets:\n- Agent regimes (i.e. kinds of network conditions): What regimes are tested in mininet? Which regimes \"trained\" on in the RL agent learning environment? Are these regimes multi-agent or single agent? This is important as it is possible for RL agents to \"overfit\" to a given environment, and to other agents.\n- Metrics: Why is the metric squared distance compared to the optimum? A more interpretable value like \"throughput\" or \"regret compared to optimal throughput\" seems both more direct and also more human-interpretable.\n\nMotivation: This work lists interpretability as a boon of the algorithm. However, both the exact notion of interpretability and the motivation for wanting interpretability are not fully elucidated.\n- It would be good to explain why interpretability is important in this setting, and also what exactly is mean by interpretability [0]. It is said in line 357-358 that interpretability can help researchers \"verify the learned model\"; however it is unclear how exactly a human would do this. After all, the proposed agent consists of large decision trees (cf. Figure 4); the branching version of the agent is even more difficult to reason about. Decision trees, unless very small and with simple decision points, are difficult for a human to reason about.\n\n[0] https://arxiv.org/abs/1606.03490", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "U44WFvwEs4V", "reviewer": "Reviewer_Qv2f", "summary": "Reinforcement learning has been previously applied to learn TCP Congestion Control algorithms (in particular, Aurora), but they suffered from not being interpretable, neither could they run in a performant way. The authors propose a two stage framework that builds on top of the approach used in Aurora by distilling the policy through symbolic regression, applying a novel ensembling/branching algorithm that decides the RL agent used. The throughput of their proposed method is compared against existing TCP congestion algorithms (TCP CUBIC, BBR) as well as recent research work (PCC Vivace, Aurora) in varying emulated network conditions and it was found that throughput did indeed improve.", "questions": "- Could the authors justify why reward clustering is done rather than clustering based on other network attributes? This may prevent \"brittleness\" as mentioned in the paper, but I would assume that similar network attributes should be handled similarly, since the same numerical reward may be derived from different network attributes.\n- Are you able to briefly characterize the differences for each of the RL agents?\n- In Table 2, how does your method compare against BBR?\n- Any thoughts on how would you close the gap between runtime for TCP Cubic vs your method?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Good comparisons with other state of the art baselines (CUBIC, Aurora, PCC Vivace) in both runtime and throughput\n- Clear diagramming and articulation of technique used\n- Interesting albeit simple extension to achieve both performance and interpretability\n\nWeaknesses:\n- Lacks theory to explain why the clustering is necessary\n- Insufficient reporting of empirical results, e.g. error bars not reported, lack of training details\n- lack of proofreading (e.g. line 147 casual instead of causal, Oscillating vs. Unstable for Table 2, min_rrt insteade of min_rtt on pg 7.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Er6m2ZE7N9-", "reviewer": "Reviewer_aJ6X", "summary": "This paper presents a novel approach for TCP congestion control, by first learning the  RL policies under different contexts and converting the NN policy into a symbolic policy which consists of several context-dependent policies and switching between the above policies. ", "questions": "I am curious about the partitioning of the networking contexts.  Did you add any prior that the networking conditions tend to stay the same for a stochastic period? I assume that we can have different networking conditions within one collected trajectory?", "limitations": "Although the proposed method is very practical for TCP congestion control, it is of limited novelty. The simple context branching method can not learn good behavior for handling context transitions in the true environment, since there is a delayed consequence of the actions taken in the last context to rewards received in the next context. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The method is very practical for TCP congestion control. It retains the performance of RL policy, but is as fast as classical methods and has good interpretability.\n2. Networking environments could be very different under different networking conditions/scenarios. For good practical performance, it is necessary to switch between different context-dependent policies. \n\n\nWeaknesses:\n1. The method just applies symbol learning with a context-aware approach to TCP congestion control. The method itself is of limited novelty. \n2. In a true environment, the context may transit from one to another. By learning policies under different contexts and then combining them together with the branching method, the learned policy does not know how to deal with the transition between different contexts. \n3. New networking scenarios may appear in the true networking environment. The method is not trained in an online fashion, and therefore cannot deal with increasing networking conditions. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept