Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper treats the distributed reinforcement learning as a dataflow problem, and proposes an actor-dataflow programming model named AnonFlow. The author has ported algorithm in RLib to AnonFlow, and reports the throughput comparison among AnonFlow, RLib, Spark and the theoretical optimal performance.
main_review: Pros:
+ It is a novel point of view to treat the distributed reinforcement learning as a dataflow problem.

+ This paper proposes AnonFlow, an actor-dataflow programming model for reinforcement learning. Algorithms in RLib has been ported to AnonFlow and the lines of code for distributed execution are reduced.

Cons:
- This paper claims that it is common for RL researchers to eschew existing infrastructures. However, no empirical evidence is provided in the experiment to show that RL researchers prefer to use the proposed AnonFlow, rather than existing infrastructures or writing their own approaches.

- It is insufficient that only evaluating the performance of AnonFlow on several RL algorithms. This paper claims that RL researchers may wish to customize a particular algorithm into various RL tasks (e.g., single-agent, multi-agent, meta-learning) and configuration options (e.g., degree of parallelism, batch size). The author fails to evaluate the performance of AnonFlow on various RL tasks and configuration options.

- The improvement of experimental results is marginal. It can be observed from Figure 11(a) that the sample throughput of AnonFlow on CartPole is less than that of RLib when the number of workers is small. In addition, Figure 11(b) shows that the training throughput of AnonFlow on Atari is almost the same as that of RLib. 

limitations_and_societal_impact: See above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This manuscript proposes AnonFlow, a set of dataflow abstractions that allow for easy specification and implementation of distributed RL algorithms. AnonFlow consists of several composable dataflow operators/iterators implemented on top of RLlib. The manuscript discusses the implementation of several popular RL algorithms and quantifies the effectiveness of AnonFlow in terms of LoC and overall system performance.
main_review: In this paper, the authors combine ideas from dataflow programming with actors and argue that this provides a more flexible programming environment for RL researchers. The methods described in the work, dataflow and actors, are not novel but the combination of these components targeting the development of RL algorithms is interesting. The distinction between the proposed and previous systems, as outlined in Table 1, makes it clear AnonFlow is meant to provide a more generic data distribution scheme alongside a higher-level interface. 

Support for the proposed abstractions is provided in the form of examples, in Figures 9 and 10, that use lines of code and readability as the metrics of interest. A reduction w.r.t LOC is expected given the authors positioning of AnonFlow as a high-level interface, relative to RLlib. No performance degradation resulted from the proposed transformations as demonstrated in Figure 11.

The paper is well-written and clearly organized to support the author's RL dataflow/RL argument. Comparisons with competing dataflow and RL libraries are provided to draw attention to the distinctive advantages of AnonFlow.

Overall, although there is a novelty in the expression of the RL algorithms, I think it would be difficult to gauge the significance of the work. It's unclear if the RL community would adopt and/or see value in the insights of the dataflow/actor implementation. I lean toward acceptance because IMHO the current offering of RL frameworks appears to be quite rigid and "new" algorithms typically require a substantial investment in time and effort. From that perspective, this work provides at least one sample of an alternative formulation worth considering to alleviate some concerns.
limitations_and_societal_impact: The authors addressed the limitations of their implementation.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper propose new library for simplifying the implementation of distributed RL algorithms. It identifies the difference between RL workloads and stream processing workloads, and creates a number of abstractions for handling asynchronous parallel operations for RL workloads.


main_review: The idea is interesting for simplifying the implementation of distributed RL algorithms. It organizes the common distributed actor processing patterns into higher level abstrations.

Although this abstraction simplifies the implementations of existing algorithm, I wonder whether it will have lower flexibiilty for experimentation and new algorithms. For example: dynamically changing dataflow based on the rollout performance.

These abstractions can improve readability of the high-level dataflow, but it seems also hide some of the details of the underlying operations. For example, in Figure 10 b), “ppo_plan” basically hides all the details of how the operations of the PPO algorithm, and how the weights are updated. How do users gain more understanding of the actual operations (like Figure 10a)?

These high level abtractions ease the implementation of distributed RL. However, it increase the overhead when running the algorithm (for example in a single machine). Is it possible to quantify the overhead of these abstractions?

For figure 12, how is the theoretical best performance calculated using Amdahl’s law?

AnonFlow’s performance is very similar to RLlib. Since AnonFlow provides a high level abstraction, does it also allow more lower level optimizations for improving the performance of RL tasks?

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: It is a good article that makes a significant contribution to the RL domain. It not only identifies the line of thought that hampers the development of RL-based solutions but also proposes the solution. The article makes a comparison of other approaches to their framework to a significant level of detail. Nevertheless, there are some flaws.

main_review: - A commendable work on reinforcement learning that addresses some important issues 
    - The work investigates the limitations of the present RL libraries such as ACME, RLib, and Coach. 
    - The work proposes a completely different view on the RL problem and advocates looking at it in a data flow context.
    - It introduces the concept of data flow operators and shows by example how this framework can be used to implement RL-based solutions or experiment with the existing ones

- Despite the good work, there are some flaws in the way article is presented.
    - The framework should have a sound theoretical discussion instead of a discussion about which library to chose for the implementation and which to reject. 
    - There are always two main choices to make when implementing a distributed framework   
        - Either to use message passing mechanism
        - Or use a shared memory model
    - From a theoretical point of this discussion is important, because it enables us to think above the individual libraries and frameworks. 
    - It is well known that the shared memory systems perform better in certain situations as compared to message-passing systems. In other cases, the situation is the opposite. For the sake of completeness, it is essential to have that discussion, whether to implement a data flow-based RL framework using shared memory or message passing. 
    - A key benefit of message passing systems is that they are scalable and robust, and can work in many different scenarios and machine capabilities. A shared memory-based system may perform better with greater compute capabilities of machines, but may totally fail in a situation where there are numerous small devices.
- In the light of the above, I think the paper presents a good idea but lacks the theoretical rigor that was needed to draw some long-lasting conclusions.
limitations_and_societal_impact: Because authors did not address the topic in a theoretical manner, so they were unable to draw a connection between their proposed framework and the established models of parallelism in computer science. This hampered their ability to fully analyze the capacity and limitations of their framework.
review_title: null

Meta-review: The reviewers and the AC appreciate this papers contribution of a nice advance in coding machinery to improve processing for parallel streams of RL data. The authors are encouraged to carefully address the multiple suggestions for further strengthening the paper in their camera ready version.
Decision: Accept (Poster)