Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work deals with the problem of congestion control in TCP communications. Specifically, the work studies policies for TCP senders, which govern the rate at which each sender sends packets. 

This problem is a partially observed, multi-agent game as each agent does not have a full picture of the network conditions and instead has to use proxy variables like the presence of a dropped packet to make decisions. Previous solutions have ranged from exploiting manually designed heuristics, to online learning, to deep RL-based methods. 

The authors relate that for this problem there is often a tradeoff between agent quality and runtime; while previously developed deep RL agents can well-manage congestion they are too computationally expensive to run in practical settings, which are generally compute constrained. 

This work introduces an algorithm claimed to have the best of both worlds, with both a high quality rate determination policy that is also computationally efficient. The proposed method consists of training a DRL agent with PPO that has an accurate rate determination policy, and then distilling the agent into a simple rule-based agent with symbolic regression. The authors train on an OpenAI Gym environment TCP sending setting, and evaluate on mininet.
strengths_and_weaknesses: This paper presents a performant, computationally-efficient method that could generalize to solving RL problems in other domains. The agent learning algorithm is clear, and the setup is mostly clear. However, the evaluation and motivation sections need further contextualization.

Evaluation: This work requires a more thorough description, and motivation, for the evaluation. Two facets:
- Agent regimes (i.e. kinds of network conditions): What regimes are tested in mininet? Which regimes "trained" on in the RL agent learning environment? Are these regimes multi-agent or single agent? This is important as it is possible for RL agents to "overfit" to a given environment, and to other agents.
- Metrics: Why is the metric squared distance compared to the optimum? A more interpretable value like "throughput" or "regret compared to optimal throughput" seems both more direct and also more human-interpretable.

Motivation: This work lists interpretability as a boon of the algorithm. However, both the exact notion of interpretability and the motivation for wanting interpretability are not fully elucidated.
- It would be good to explain why interpretability is important in this setting, and also what exactly is mean by interpretability [0]. It is said in line 357-358 that interpretability can help researchers "verify the learned model"; however it is unclear how exactly a human would do this. After all, the proposed agent consists of large decision trees (cf. Figure 4); the branching version of the agent is even more difficult to reason about. Decision trees, unless very small and with simple decision points, are difficult for a human to reason about.

[0] https://arxiv.org/abs/1606.03490
questions: Questions are listed in strengths and weaknesses.
limitations: The authors list Section 5 as the section in which limitations are dealt with but as far as I can tell none are addressed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Reinforcement learning has been previously applied to learn TCP Congestion Control algorithms (in particular, Aurora), but they suffered from not being interpretable, neither could they run in a performant way. The authors propose a two stage framework that builds on top of the approach used in Aurora by distilling the policy through symbolic regression, applying a novel ensembling/branching algorithm that decides the RL agent used. The throughput of their proposed method is compared against existing TCP congestion algorithms (TCP CUBIC, BBR) as well as recent research work (PCC Vivace, Aurora) in varying emulated network conditions and it was found that throughput did indeed improve.
strengths_and_weaknesses: Strengths
- Good comparisons with other state of the art baselines (CUBIC, Aurora, PCC Vivace) in both runtime and throughput
- Clear diagramming and articulation of technique used
- Interesting albeit simple extension to achieve both performance and interpretability

Weaknesses:
- Lacks theory to explain why the clustering is necessary
- Insufficient reporting of empirical results, e.g. error bars not reported, lack of training details
- lack of proofreading (e.g. line 147 casual instead of causal, Oscillating vs. Unstable for Table 2, min_rrt insteade of min_rtt on pg 7.)
questions: - Could the authors justify why reward clustering is done rather than clustering based on other network attributes? This may prevent "brittleness" as mentioned in the paper, but I would assume that similar network attributes should be handled similarly, since the same numerical reward may be derived from different network attributes.
- Are you able to briefly characterize the differences for each of the RL agents?
- In Table 2, how does your method compare against BBR?
- Any thoughts on how would you close the gap between runtime for TCP Cubic vs your method?
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a novel approach for TCP congestion control, by first learning the  RL policies under different contexts and converting the NN policy into a symbolic policy which consists of several context-dependent policies and switching between the above policies. 
strengths_and_weaknesses: Strengths:
1. The method is very practical for TCP congestion control. It retains the performance of RL policy, but is as fast as classical methods and has good interpretability.
2. Networking environments could be very different under different networking conditions/scenarios. For good practical performance, it is necessary to switch between different context-dependent policies. 


Weaknesses:
1. The method just applies symbol learning with a context-aware approach to TCP congestion control. The method itself is of limited novelty. 
2. In a true environment, the context may transit from one to another. By learning policies under different contexts and then combining them together with the branching method, the learned policy does not know how to deal with the transition between different contexts. 
3. New networking scenarios may appear in the true networking environment. The method is not trained in an online fashion, and therefore cannot deal with increasing networking conditions. 
questions: I am curious about the partitioning of the networking contexts.  Did you add any prior that the networking conditions tend to stay the same for a stochastic period? I assume that we can have different networking conditions within one collected trajectory?
limitations: Although the proposed method is very practical for TCP congestion control, it is of limited novelty. The simple context branching method can not learn good behavior for handling context transitions in the true environment, since there is a delayed consequence of the actions taken in the last context to rewards received in the next context.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an efficient and interpretable framework for TCP congestion control. Specifically, they first train an RL agent to discover good teacher policies, then distill such policies into students, which are designed to be compact and interpretable using symbolic decision trees. They also split trajectories using Kmeans into contexts, and combine context-specific policies with context regression to further improve performance (essentially using the idea of Multi-task RL).  Extensive experiments demonstrate the effectiveness of the proposed approach.
strengths_and_weaknesses: Strengths: The authors combine RL with symbolic decision tree in the novel application of TCP congestion control. Experiments demonstrate that the proposed approach is very effective and significantly outperforms previous work.

Weaknesses: The paper lacks some formal descriptions of technical details in methodology. Some motivations are not clear and limitations are not addressed. Please see details below.

Overall the methodology does not seem quite novel. The context split and branching process is essentially existing ideas in multi-task RL, and works along this line should be cited [1,2]. However, the application is novel and the approach seems effective, so I still recommend borderline acceptance.

[1] Teh, Yee and Bapst, Victor and Czarnecki, Wojciech M and Quan, John and Kirkpatrick, James and Hadsell, Raia and Heess, Nicolas and Pascanu, Razvan: Distral: Robust multitask reinforcement learning. *Advances in neural information processing systems*, 2017.

[2] Ghosh, Dibya and Singh, Avi and Rajeswaran, Aravind and Kumar, Vikash and Levine, Sergey: Divide-and-conquer reinforcement learning. *International Conference on Learning Representations*, 2018.

**Post Rebuttal**: Thanks for the authors' rebuttal! The added details addressed most of my concerns. Overall, this paper is an application paper that combines existing approaches effectively to a novel domain. While the methodology has limitations, the application itself is novel, and I'm maintaining a stance between borderline and weak accept and increasing my confidence.
questions: 1. For trajectory and context clustering phase, why can trajectory returns be clustered in the first place? What is the motivation for it? What semantic meanings do different trajectory returns have?

2. How specifically are rollout trajectories converted into symbolic decision trees (looks like this is based on a standard decision tree process using the rollout actions as input data)? How is tree pruning specifically done? These details should be addressed, using formal mathematics, at least in the supplementary.

3. What is the environment input state space and action space? What semantic meaning does the action space represent? The authors do not study a common setting like vision-based RL, so these should be briefly addressed.

Minor:

"Rewards" should be "Return" in this work. The former means single-step reward, while the latter means episode (aggregated) return.
limitations: The trajectory clustering part is specific to the environment and reward design authors are using. It is not a general approach that can be applied to any environment. This should be addressed in the limitations.
review_title: null

Meta-review: metareview: I thank the authors for their submission and active participation in the discussions. The paper presents an RL method for TCP congestion control. While this application paper is borderline, all reviewers unanimously agree that this paper's strengths outweigh its weaknesses. In particular, reviewers remarked that the method is efficient [sAC3], and practical [aJ6X], evaluated well against baselines [Qv2f] with promising results [W18N]. Thus, I am recommending acceptance of the paper but highly encourage the authors to further improve their paper based on the reviewer feedback.
confidence: Certain
Decision: Accept