RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
Authors: Eric Liang
Zhanghao Wu
Michael Luo
Sven Mika
Joseph E. Gonzalez
Ion Stoica
Venue: NeurIPS 2021 Poster (2021)
Keywords: Distributed Reinforcement Learning
Dataflow Programming Model

Abstract: Researchers and practitioners in the field of reinforcement learning (RL) frequently leverage parallel computation, which has led to a plethora of new algorithms and systems in the last few years. In this paper, we re-examine the challenges posed by distributed RL and try to view it through the lens of an old idea: distributed dataflow. We show that viewing RL as a dataflow problem leads to highly composable and performant implementations. We propose RLlib Flow, a hybrid actor-dataflow programming model for distributed RL, and validate its practicality by porting the full suite of algorithms in RLlib, a widely adopted distributed RL library. Concretely, RLlib Flow provides 2-9$\times$ code savings in real production code and enables the composition of multi-agent algorithms not possible by end users before. The open-source code is available as part of RLlib at https://github.com/ray-project/ray/tree/master/rllib.

RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
Eric Liang" UC Berkeley
Zhanghao Wu" UC Berkeley
Michael Luo UC Berkeley
Sven Mika Anyscale
Joseph E. Gonzalez UC Berkeley
Ion Stoica UC Berkeley
Abstract
Researchers and practitioners in the field reinforcement learning (RL. frequently leverage parallel computation, which has led to plethora of new algorithms and systems in the last few years. In this paper, we re-examine the challenges posed by distributed RL and try view it through the lens of an old idea: distributed dataflow. We show that viewing RL as a dataflow problem leads to highly composable and performant implementations. We propose RLlib Flow, hybrid actor-dataflow programming model for distributed RL, and validate its practicality by porting the full suite of algorithms in RLlib, widely adopted distributed RL library. Concretely, RLlib Flow provides 2-9x code savings in real production code and enables the composition of multi-agent algorithms not possible by end users before. The open-source code is available as part of RLlib at https: / / thub. com/ ray- httt-poojet/rry//bbaeee mateee////tterrriii///
Introduction
The past few years have seen the rise of deep reinforcement learning (RL) new, powerful optimization method for solving sequential decision making problems. As with deep supervised learning, researchers and practitioners frequently leverage parallel computation, which has led the development of numerous distributed RL. algorithms and systems as the field rapidly evolves.
However, despite the high-level of abstraction that RL algorithms dozen lines update equations), their implementations have remained quite low level (i.e., at the level of message passing). This is partieularly true for distributed RL algorithms, which are typically implemented directly on low-level message passing systems or actor frameworks [14]. Libraries such Acme [15], RLgraph [26], RLlib [19], and Coach [2] provide unified abstractions for defining single-agent RL algorithms, but their user-facing APIs only allow algorithms to execute within the bounds to predefined distributed execution patterns "templates".
defined in (i.c., as couple
While the aforementioned libraries have been highly successful at replicating a large number of novel RI. algorithms introduced over the years, showing the generality of their underlying actor or graph-based computation models, the needs of many researchers and practitioners are often not met by their abstractions. We have observed this firsthand from users of open source RL libraries:
First, RL practitioners are typically not systems engineers. They are not well versed with code that mixes together the logical dataflow of the program and system concerns such as performance and bounding memory usage. This leads to a high barrier of entry for most RL users experimenting with debugging existing distributed RL algorithms or authoring new distributed RL approaches.
indicates equal contributions.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

high
query unnuagge
TMllafoo
Gdtns RPC/ message peasing
Single Agent
Multi Agent
Meta Learning
RUlb Flow
GRPC
RLIl Flow Operators (1118 line of code)
1e5004 lines of code
SQL
MPI
Iterator Library (1241 code)
Distributed Actor System
RLibb Flow (propased)
Existing Systems
Figure 1: RLlib Flow enables implementation Figure 2: Architecture of our port of RLlib to RLlib
of distributed RL with high-level dataflow.
Flow (RLlib Flow core is only 1118 lines of code).
Table 1: Comparison of the systems aspects of RLlib Flow to other distributed RL. libraries.
Distributed Library
Distribution Scheme
Generality
Programmability
#Algorithms
RLGraph [26]
Pluggable
General Purpose
Low-level/ Pluggable
10+
Deepmind Acme [15]
Actors + Reverb
Asyne Actor-Learner
Limited
10+
Intel Coach [2]
Actors + NFS
Async Actor-Learner
Limited
30+
RLLib [19]
Ray Actors
General Purpose
Flexible, but Low-level
20+
RLlib Flow
Actor Dataflow
General Purpose
Flexible and High-level
20+
Second, even when RL practitioner is happy with particular algorithm, they may wish to customize it in various ways. This is especially important given the diversity of RL tasks (c.g., single- agent, multi-agent, meta-learning). While many customizations within common RL environments can be anticipated and made available as configuration options (e.g., degree of parallelism, batch size), it is difficult for library author provide enough options to cover less common tasks that necessarily alter the distributed pattern of the algorithm (e.g., interleaved training of different distributed algorithms, different replay strategies).
Our experience that when considering the needs of users for novel RL applications and approaches, RL development requires a significant degree of programming flexibility. Advanced users want to tweak or add various distributed components (i.e., they need to write programs). In contrast to supervised learning, it is more difficult to provide fixed set of abstractions for scaling RL training.
As result, very common for RL researchers practitioners to eschew existing infrastructure, either sticking to non- non-parallel approaches, which are inherently easier to understand and customize [3, 6], or writing their own distributed framework that fits their needs. The large number of RL frameworks in existence today is evidence of this, especially considering the number of these frameworks aiming to be "simpler" versions of other frameworks.
In this paper, we re-examine the challenges posed by distributed RL in the light of these user requirements, drawing inspiration from prior work in the field of data processing and distributed dataflow. To meet these challenges, we propose RLlib Flow, a hybrid actor-dataflow model for distributed RL. Like streaming data systems, RLlib Flow provides small set of operator-like primitives that can be composed to express distributed RL algorithms. Unlike data processing systems, RLlib Flow explicitly exposes references actor processes participating in the dataflow, permitting limited message passing between them in order to more simply meet the requirements of RL algorithms. The interaction of dataflow and actor messages managed via special sequencing and concurrency operators.
The contributions of paper follows:
1. We examine the needs of distributed RL algorithms and RL practitioners from a dataflow perspec- tive, identifying key challenges (Section 2 and 3).
2. We propose RLlib Flow, hybrid actor-dataflow programming model that can simply and effi- ciently express distributed RL algorithms, and enables composition of multi-agent algorithms not possible by end users before without writing low-level systems code. (Section 4 and 5).
3. We port all the algorithms of production RL library (RLlib) to RLlib Flow, providing 2-9x savings in distributed execution code, compare its performance with the original implementation, and show performance benefits over systems such as Spark Streaming (Section 6).

---

2 Distributed Reinforcement Learning
We first discuss the relevant computational characteristics of distributed RL algorithms, starting with the common single-agent training scenario, where the goal is to optimize single agent's performance an environment, and then discuss the computational needs emerging multi-agent, model-based and mela-learning training patterns.
2.1 RL Algorithm Basics
The goal of an RL algorithm is typically to improve the performance of a policy with respect objective defined through an environment (c.g., simulator). The policy is usually defined as a deep neural network, which can range from several KB to several hundred MB in size. RL algorithms can be generally broken down into the following basic steps:
Rollout
Replay
Optimization
Repeat
Rollout Worker Actors
Replay Buffer Actors
Learner Actor
Figure 3: Most RL algorithms can be defined in terms of the basic steps of Rollout, Replay, and Optimization. These steps are commonly parallelized across multiple actor processes.
Rollout: To generate experiences, the policy, which outputs actions take given environment observations, is run against the environment to collect batches of data. The batch consists of observations, actions, rewards, and episode terminals and can vary in size (10s to 10000s of steps).
On-policy algorithms (e.g., PPO [27], A3C [22]) collect new experiences from the current policy to learn. On the other hand, off- off-policy algorithms (e.g.. DQN [23], SAC [12]) can leverage experiences from past versions of the policy as well. For these algorithms, a replay buffer of past experiences can be used. The size of these buffers ranges from few hundred to millions of steps.
Replay:
Optimization: Experiences, either freshly collected or replayed, can be used to improve the policy. Typically this is done by computing and applying a gradient update to the policy and value neural networks. While many applications a single GPU suffices compute gradient updates, it is sometimes desirable to leverage multiple GPUs within single node, asynchronous computation of gradients on multiple CPUs [22], or many GPUs spread across cluster [29].
2.2 RL Algorithm Variants
Single-Agent Training. Training a single RL agent-thee most basic and common scenario-consiste of applying the steps of rollout, replay, and optimization repeatedly until the policy reaches the desired performance. Synchronous algorithms such as A2C [22] and PPO apply the steps strictly sequentially. Parallelism may be leveraged internally within each step. Asynchronous algorithm variations such as A3C [22]. Ape-X [16], APPO [21], and IMPALA [8], pipeline and overlap the rollout and optimization steps asynchronously to hit higher data throughputs. Rate limiting [15] can be applied to control learning dynamics in the asynchronous setting.
Multi-Agee Training. In multi-agent training, there are multiple acting entities in the environment (c.g., cooperating or competing agents). While there is a rich literature on multi-agent algorithms, long
note that the dataflow structure of multi-agent training similar to that of single-agent However, problems arise should it be required to customize the training of any of the agents in the environment. For example, in a two-agent environment, one agent may desire to be optimized at a higher frequency (i.e., smaller batch size). This fundamentally alters the training dataflow-ther loops executing at different frequencies. Furthermore, if these agents are trained with entirely different algorithms, there is need compose two different distributed dataflows.
as all entities are being trained with the same algorithm and compatible hyperparameters srrpaarameters.Howeeeer
now two iterative
Model-Based and Meta-Lea Algorithms. Model-based algorithms seek to learn transition dynamics of the environment to improve the sample efficiency of training. This can be thought
3

---

of as adding supervised training step top of standard distributed RL, where an ensemble of one or more dynamies models are trained from anvironment-generate data. Handling the data routing, replay, optimization, and stats collection for these models naturally adds complexity to the distributed dataflow graph, "breaking the mold" of standard model-free RI. algorithms and hard to be implemented in low-level systems. Using RLlib Flow, we have implemented two state-of-the-an model-basee algorithms: MB-MPO [5] and Dreamer [13].
2.3 A Case for a Higher Level Programming Model
Given that existing distributed RI algorithms already implementable using low level actor and RPC primitives, it is worth questioning the value of defining higher level computation model. Our experience is that RL is more like data analytics than supervised learning. Advanced users want to tweak or add various distributed components (i.e., they need to program), and there is no way to have "one size fits all" (i.e., Estimator interface from supervised learning). We believe that, beyond the ability to more concisely and cleanly capture single-agent RL algorithms, the computational needs of more advanced RL training patterns motivate higher level programming models like RLlib Flow.
3 Reinforcement Learning vs Data Streaming
The key observation behind RLJib Flow is that the dataflow graph of RL algorithms are quite similar to those of data streaming applications. Indeed, RL algorithms can be captured in general purpose dataflow programming models. However, due to several characteristics, they are not perfect fit, even for dataflow programming models that support iterative computation.
In this section we examine the dataflow of the A3C algorithm (Figure 4) to compare and contrast RL with streaming dataflow. A3C starts with (1) parallel rollouts across many experiences. Policy gradients are computed in parallel based on rollouts in step (2). In step (3), the gradients are asynchronously gathered and applied on a central model, which is then used to update rollout worker weights. Importantly, each box or operator in this dataflow may be stateful (e.g.. ParallelRolloo holds environment state well as the current policy snapshot).
Parallel Rollouts
Compute
Apply Gradients
Report Metrics
Gradients
(1)
(4)
Update Weights
Figure 4: The dataflow of the A3C parallel algorithm. Each box is an operator or iterator from which data items can be pulled from. Here operators (I) and (2) represent parallel computations, but (3) and (4) are sequential. Black arrows denote synchronous data dependencies, pink arrows asynchronous dependencies, and dotted arrows actor method calls.
Similar to data processing topologies, A3C is applying a transformation to data stream (of rollouts) in parallel (to compute gradients). This is denoted by the black arrow between (1) and (2). There is also a non-parallel transformation to produce metrics from the computation, denoted by the black arrow between (3) and (4) However, zooming out to look at the entire dataflow graph, a few differences emerge:
Asynchronous Dependencies: RL algorithms often leverage asynchronous computation to reduce update latencies and eliminate stragglers [22]. In RLlib Flow, we represent these with a pink arrow between parallel and sequential iterator. This means items will be fetched into the sequential iterator as soon as they are available, instead of in a deterministic ordering. The level of asynchrony can be configured to increase pipeline parallelism.
Message Passing: RL algorithms, like all iterative algorithms, need to update upstream operator state during execution (e.g., update policy weights). Unlike iterative algorithms, these updates may be fine-grained and asynchronous (i.e., update the parameters of a particular worker), as well as coarse-grained (i.c., update all workers at once after global barrier). RLlib Flow allows method

---

calls (messages) to be sent to any actor in the dataflow. Ordering of messages in RLlib Flow with respect dataflow steps is guaranteed synchronous data dependencies (black arrows) fully connect the sender to the receiver, providing barrier semantics.
Consistency and Durability: Unlike data streaming, which has strict requirements such exactly- once processing of data [30], RL has less strict consistency and durability requirements. This is since on fault, the entire computation can be restarted from the last checkpoint with minimal loss of work. Message or data loss can generally be tolerated without adverse affect on training. Individual operators can be restarted on failure, discarding any temporary state. This motivates programming model that minimizes overhead (e.g., avoids state serialization and logging cost).
4 A Dataflow Programming Model for Distributed RL
Here we formally define the RLlib Flow hybrid actor-dataflow programming model. RLlib Flow consists of a set of dataflow operators that produce and consume distributed iterators [11]. These distributed iterators can represent parallel streams of data items T sharded across many actors (ParIter [T]), ), or single sequential stream of items (Iter [T]). It is important to note that these iterators are lazy, they do not execute computation or produce items unless requested. This means that the entire RLlib Flow execution graph driven by taking items from the output operator.
create (Seq [SourceActor [T]]) - Pariter [T]
for _ach (ParIter [T], => U) ParIter [U]
send_aag nag (deat: Actor, , mag: Any) + Reply
for (Ir_
U)
Iter [U]
Actors
Pariter[T]
Actor
Parller[T]
Parlter(U]
Nee[[]]
her[U]
From Actors
Send Message
Parallel Apply
Sequentioa Apply
Figure 5: Creation and Message Passing
Figure 6: Transformation
Creation and Message Passing: RLlib Flow iterators are always created from an existing set of actor processes. In Figure 4, the iterator is created from of rollout workers that produce experience batches given their current policy. Also, any operator may send a message to any source actor (i.e., rollout worker, replay buffer) during its execution. In the A3C example, the update weights operation is use of this facility. The order guarantees of these messages with respect to dataflow steps depends on the barrier semanties provided by sequencing operators. The sender may optionally block and await the reply of sent messages. We show the operator in Figure 5.
Transformation: As in any data processing system, the basic operation of data transformation is supported. Both parallel and sequential iterators can be transformed with the for for_each operator. The transformation function
be stateful (i.e., in Python it can be callable function class that holds state in class members, and the case of sequential operators it can reference local variables via closure capture). In the A3C example, for_each is used compute gradients for each batch of experiences, which depends on the current policy state the source actor. In the case of the ComputeGradies ents step, this state is available the local process memory of the rollout worker, and is accessible because RLlib Flow schedules the execution of parallel operations onto the source actors. We show the operator in Figure 6.
pather_aayen (ParIter [T],
split (Iter (Iter (T), Iter (T))
asyno: Int) Iter [T]
union [List[[lee [T]] weighta: List [rioat]] Iter [T]
gather_ (ParIter (T)) + Iter [Liet [T]]
next (Iter (T)) +
union union_aeyac (List [Iter (T))); Iter (T)
Buffer
Parkw[T]
PacteeT
HerTT
tevIT]
Asyno Gothor
Busk Syne Gatter
buffer
(No Banie)
(Fut Barner)
Duplicate
Union
Figure 7: Sequencing
Figure 8: Concurrency
Sequencing: To consume a parallel iterator, the items have to be serialized into some sequential order. This is the role of sequencing operators. Once converted into sequential iterator, next can
5

---

be called on the iterator to fetch a concrete item from the iterator. The gather_ gather_asyne operator is used in A3C, and gathers computed gradients as fast as they are computed for application central policy. For a deterministic variation, we could have instead used gather_syy _ync, which waits for one gradient from each shard of the iterator before returning. The syne gather operator also has barrier semantics. Upstream operators connected by synchronous dependencies (black arrows) are fully halted between item fetches. This allows for the source actors to be updated prior to the next item fetch. Barrier semantics do not apply across asynchronous dependencies, allowing the mixing of synchronous and async dataflow fragments separated by pink arrows, Figure 7.
Concurrency:
Complex algorithms may involve multiple concurrently executing dataflow fragments. Concurrency (union) operators, in Figure 8, govern how these concurrent iterators relate to each other. For example, one may wish two iterators to execute sequentially in a round robin manner, execute independently in parallel, or rate limiting progress fixed ratio [15]. Additionally, one might wish to duplicate (split) an iterator, in which case buffers are automatically inserted to retain items until fully consumed. In this case, the RLlib Flow scheduler tries bound memory usage by prioritizing the consumer that is falling behind.
Implementation
implemented RLlib Flow on the Ray distributed actor framework [24] as two separate modules: a general purpose parallel iterator library (1241 lines of code), and a collection of RL. specific dataflow operators (1118 lines of code) (Figure 2). We then ported the full suite of 20+ RL algorithms in RLlib to RLlib Flow, replacing the original implementations built directly on top of low-level actor and RPC primitives. Only the portions of code in RLlib related to distributed execution were changed (the exact same numerical computations as baseline. this section we overview two simple examples to illustrate RLlib Flow. MAML case study, can be found in Section A.2.
in our port), which allows to fairly evaluate against it
5.1 Asynchronous Optimization in RLlib Flow vs RLlib
As previously seen in Figure 4, A3C is straightforward to express in RLlib Flow. Figure 9a shows pseudocode for A3C RLlib Flow (11 lines), which we compare to a simplified version of the RLlib implementation (originally 87 lines). RLlib Flow hides the low-level worker management and data communication with its dataflow operators, providing more readable and flexible code, More detailed comparison of implementations in RLlib Flow and RLlib can be found in Section A.3.
launch gradients computation tasks panding_gradienn dict()
for worker in moote_sookkrs:
workar.sat borkar-satt_vighhee www.giigts
future worker. parker.corpute. cerputtegraddents
# type: List[RolloutActor]
pradinnggradtents( future] worker
workers create create_rollout
asynchronously gather gradients and apply
# type: [Graderadeents]
while pending_gradients: gradients:
grads Paralleelnollootss(wovrrrs)
walttresults ray ray.wait(
-par_for_est horn_oooo_.cccccmmppouurraadddee
11
pending_gradee www.grrdennt.eys....... nun returnse1)
-gather_asynee
ready_list reit_results[]] future ready_list[[]
7 # type: Iter[TrainStats]
14
apply_op grads
.for_ (wneenn((((llyylssttetttrrnk))))
16 gredient, info rey rey.get(future) worker pending
type: Iter[Metrics]
18
apply gradients
11 return ReportMetrrccsapplly.oo workers)
19
local www.er..pply..graeeee....ee. grw.ggradddddgge (gradien)))))))e)n) weights local bocal_wankor...r get wrights()
21
Launch gradient conputation again
23
future worker.corpute corpute_grrdients _raddents
sxapeeeeeeeeee...l........ pending_gradients] [future] worker
(a) The entire A3C dataflow in RLlib Flow.
(b) small portion of the RLlib A3C policy optimizer.
Figure 9: Comparing the implementation of asynchronous optimization in RLlib Flow vs RLlib.

---

5.2 Ape-X Prioritized Experience Replay in RLlib Flow
Ape-X [17] (Figure 10a) is high-throughhhe variation of DQN. is notable since it involves multiple concurrent sub-flows (experience storage, experience replay), sets of actors (rollout actors, replay actors), and actor messages (updating model weights, updating replay buffer priorities). The sub- flows (store_o _oo, replay_ can be composed in RLlib Flow as follows using the Union operator (Figure 10b). The complicated workflow can be implemented in several lines, as shown Figure 10b.
# type: List [atRRloouutAc
warkens create_rolllan rollaut workers()
type: [ist[ReplayActor]
replay_buffer "reate_replay_actorge
# type: Itor[Rollaut]
rollouts (ooeleee..llltttsmeereesss.ess
Paralel Rolluus
Store to Butter
Upcate Waights
type: Iter[_]
store_e rollouts
Parailal Roplay
Update
Union
10
.for_ hho__eccl(uucaateeelgggtssooo
type: [ter[TrninStets]
12
replay_op Paralle1Replay(rppl) buffee)
Optimbie Policy
Report Metrics
13
.gather_asyycc assnnc())))))
14
15
type: Iter [Netrics]
17
return ReportMetricst
18
Union(store _o,, replay_op), workers)
(a) Ape-X Dataflow diagram.
(b) Ape-X implementation in RLlib Flow
Figure 10: Dataflow and implementation for Ape-X algorithm. Two dataflow fragments are executed
concurrently to optimize the policy.
5.3 Composing DQN and PPO in Multi-Agent Training
Multi-agent training can involve the composition of different training algorithms (i.e., PPO and DQN). Figure la shows the combined dataflow for experiment that uses DQN to train certain policies in an environment and PPO to train others. The code can be found Figure 11b. In an actor or RPC-based programming model, this type of composition is difficult because dataflow and control flow logic is intermixed. However, it is easy to express in RLlib Flow using the Union operator (Figure 8). In Figure 12, we show the implementation of the two subflow, ppo_plan and dqn_plan. in the multi-agent training (Figure 1lb).
Replay from (batch sizen-32)
Optimize Ipollay
Updale Targel Network
type: List[RolloutActor]
Rollouts
workens preate_rallout. (ooooutttooooo())))
3 type: Iter[Rollout), Iter[Rollout]
Experiences
to
rl, PanallelRollours (xorkelsolsssor
(sp6))
type:
Iter(trainStats), Inee(TnainStats]
Union
ppo_op ppo_ppann
Select Experiences (pll))"PPPO)
Concalenate jaiz sbwnttaa)
Select(r1, po)"cy="PP0"), workers)
Async
dqn_op dqn_planç
Update Weights
(el,ct policy-"ogU"), workers)
18 type: [ter(Metiics]
Optimize
jpaliay "PPO")
Report Metrics
ReportMetricst
12
union(ppo_opp den_op), workers)
(a) Dataflow diagram.
(b) Implementation RLlib Flow.
Figure 11: Dataflow and implementation for concurrent multi-agent multi-policy workflow with PPO and DQN agents in an environment.
6 Evaluation
In our evaluation, seek to answer the following questions:
7

---

oplay_buffer- retpaa__rppaayacttsssss
store_op rollouts
ppo_op = rollouts
-for_ for_each((eelecttoollyy-===
.for (cchh(seeec((ppllccy="pp""))
foor_ecccsttrreeeufffrrrep... buffer))
combine (ConcatBatches (bathtt sittcchh_ssee)
replay_og (ypllay(replay_buffeee
.orr _for__eachhh_cc eachh(TrainnneStep@woore
for (aaechh(rrraannnssetteorrrsss)))
for foo._accc(ppatttenneeeeeeeoooooo
dqn_op = Union (spore_op replay_op)
(a) Implementation of PPO Subflow.
(b) Implementation of DQN Subflow.
Figure 12: Implementation of sub-flows for the multi-agent multi poltiy training of PPO and DQN.
What the quantitative improvement in code complexity with RLlib Flow?
2.
How does RLlib Flow compare to other systems in terms of flexibility and performance for RL tasks?
6.1 Code Complexity
Lines of Code: In Table compare the original algorithms in RLlib after porting RLlib Flow. No functionality was lost in the RLlib Flow re-implementations. We count all lines of code directly related distributed execution, including comments and instrumentation code, but not including utility functions shared across all algorithms. For completeness, for RLlib Flow we include both an minimal (RLlib Flow) and conservative (+shared) estimate of lines of code. The conservative estimate includes lines of code in shared operators. Overall, we observe between 1.9-9.6x (optimistic) and 1.1-3.1 1 (conservative) reduction in lines of code with RLlib Flow. The most complex algorithm (IMPALA) shrunk from 694 to 89-362 lines.
Table 2: Lines of code for several prototypical algorithms implemented with the original RLlib vs our RLlib Flow-based RLlib. *Original MAML: https: // go thuu. com/ jonasro th fuss/ ProMP
RLJib
RI.ii Flow
+shared
Ratio
A3C
1.6-9,6x
A2C
154
3.1-6.1
DQN
239
87
139
1.7-2.7x
PPO
386
79
225
1.7-4.8x
Ape-X
250
126
216
1.1-1.9x
IMPALA
362
1.9778x
MAML
370
136
136
2.7x
Readability: We believe RLlib Flow provides several key benefits for readability of RL algorithms:
1. The high-level dataflow of an algorithm is visible at a glance in very few lines of code, allowing readers to understand and modify execution pattern without diving deep into the execution logic.
Execution logic is organized into individual operators, each of which has a consistent input and output interface (i.c., transforms iterator into another iterator). In contrast to building low-level RPC systems, developers can decompose their algorithms into reusable operators.
3. Performance concems isolated into the lower-level parallel iterator library. Developers do not need to deal with low-level concepts such as batching or flow-control.
Flexibility: As evidence of RLlib Flow's 's flexibility. an undergraduate was able to implement several model-based (e.g., MB-MPO) and meta-learning algorithms (e.g., MAML), neither of which into previously existing execution patterns in RLlib Flow. This was only possible due to the flexibility of RLlib Flow's model. RIlib Flow captures MAMI. 139 lines compared a baseline of zx370 lines (Table 2). Detailed discussion can be found in Section A.

---

200
B RLIib
185
20
RLIbb
19
Sample Throughput
(x1 (x1. 1,,000 samples/seo
161
(x1 x1,000 samples/sen
17
17
150
RLiib Flow
Throughput
15
RLIbb Flow
14 14
12
12
92
90100
100
10
49
50
50
28
27
14 13
16
32
64
128
256
16
32
64
128
200
#Workers
#Workers
(a) Sample efficiency on CartPole.
(b) Training throughput on Atari.
Figure 13: Performance of RLlib compared with RLlib Flow, executing identical numerical code.
RLlib Flow achieves comparable or better performance across different environments.
6.2 Microbenchmarks and Performance Comparisons
For all the experiments, m4.16xlarg worker instances. All machines have 64 vCPUs and are connected by a 25Gbps network. More experiments for different RL algorithms can be found in httpp:/// /ttp//ggttttbbtt com/
cluster with AWS p3.16xlarge GPU head instance with additional
Sampling Mierobenchmark. We evaluate the data throughput of RLlib Flow in isolation by running RL training with a dummy policy (with only one trainable scalar). Figure 13a shows that RLlib Flow achieves slightly better throughput due small optimizations such as batched RPC wait, which are easy to implement across multiple algorithms in a common way RLlib Flow.
IMPALA Throughput: In Figure 13b we benchmark IMPALA, one RLlib's high-throughput RI. algorithms, and show that RLlib Flow achieves similar or better end-to-end performance.
Performance Multi-Agent Multi-Policy Workflow: In Figure 14, we show that the workflow of the two-trainer example (Figure 1 11a) achieves close to the theoretical best performance possible combining the two workflows (calculated via Amdahl's law). This benchmark was run a multi- agent Atari environment with four agents per policy, and shows RLlib Flow be practically used to compose complex training workflows.
Distributed dataflow systems such as Spark Streaming [30] and Flink [1] are designed for collecting and transforming live data streams from online applications (e.g., event streams, social media). Given the basic map and reduce operations, synchronous RL algorithms in any of these streaming frameworks. However, without consideration for the requirements of RL tasks (Section 3), these frameworks can introduce significant overheads. In Figure 15 we compare the performance of PPO implemented in Spark Streaming and RLlib Flow. Implementation details are in in Appendix A.1.
Comparison to Spark Streaming:
implement
7 Related Work
Reinforcement Learning Systems: RLlib Flow is implemented concretely in RLlib, however, we hope can provide inspiration a new generation of general purpose RL systems. RL libraries available today range from single-threaded implementations 6, 18] to distributed [2,9, 15, 19, 20, 26]. These libraries often focus on providing common frameworks for the numerical concerns of RI. algorithms (e.g., loss, exploration, and optimization steps).
However, these aforementioned libraries rely on predefined distributed execution patterns. For exam- ple, for the Ape-X dataflow in Figure 10a, RLlib defines this with a fixed SesyneRRplayOptimiizz class that implements the topology, intermixing the dataflow and the control flow.; RLGraph uses an adapted implementation from RLlib as part of their Ape-X algorithm veea-ggaph. while Coach does not support Ape-X*. These execution patterns are predefined as they are low-level. complex to implement, and cannot be modified using high-level end-user APIs. In contrast, RLlib Flow proposes
"hrtpa: pirttp:://dane httpc://ddoc.r/r// wt/pp/////e///eeeeeeeet/// 0... httpa//////////////////////////a
battpa///githbb. http:///www...aaooooo/////////// 'attpe:/ pestpe://githbb www.Inteelaab/coo.ee..ooo

---

8 Two Trainer Workfoow (DON+PPO)
Thcordtical Optimal
T Ini
B Train
Sample
40
38.0
Throughpul (sempledeeee))
706
200
187
30
140
155
Tinge
9.8
24.3
1-9K
19.0
21:1
100
63
7.0
14.8
6.0
10.1
2.3x
2.9K
7.0
16
32 84
128
256
Sprek Ri. Flow
Spark
Rub Plww
Spus
RLA Flww
Sport
AiLL0 Plen
Spark Pine
Spart Flok
Figure 14: RLlib Flow achieves close the theo- retical optimal performance combining two work- flows for multi-agent training, making practical to use for composing complex training scenarios.
18
32
#Workers
(4x)
Figure 15: The throughput comparison between
RLlib Flow and Spark Streaming with PPO algo- rithm on CartPole-vO environment.
a high-level distributed programming model for RL algorithm implementation, exposing this pattern in much fewer lines of code (Figure 10b), and allowing free composition of these patterns by users (Figure 1b). The ideas from RLlib Flow can be integrated with any RL library enable flexibility in distributed execution.
Distributed Computation Models: RLlib Flow draws inspiration from both streaming dataflow and actor-based programming models. Popular open source implementations of streaming dataflow, including Apache Storm [28], Apache Flink [1], and Apache Spark [30,31] transparently distribute data to multiple processors in the background, hiding the scheduling and message passing for distribution from programmers. in Apache Spark. Apache Flink's 's Delta Iterate operator can similarly support synchronous RL algorithms. However, data processing frameworks have limited asynchronous iteration support.
Appendix A.I, we show how distributed PPO can be implemented
The Volcano model [11], commonly used for distributed data processing, pioneered the parallel iterator abstraction. RLlib Flow builds the Volcano model only encapsulate parallelism, but also to encapsulate the synchronization requirements between concurrent dataflow fragments, enabling to also leverage actor message passing.
Naiad [25] is a low-level distributed dataflow system that supports cyclic execution graphs and message passing. It is designed as system for implementing higher-level programming models. In principle, it possible to implement the RLlib Flow model Naiad. Transformation operators can be placed on the stateful vertices of the execution graph. The message passing and concurrency (Union) operators can represented by calling SENDBY and ONRECV interface on senders and receivers, which support asynchronous execution. RLlib Flow's 's barrier semantics can be expressed with ONNOTIFY and NOTIFYAT, where the former indicates all the required messages are ready, and the latter blocks execution until the notification has been received. We implemented RL.ib Flow on Ray instead of Naiad for practical reasons (e.g., Python support).
Conclusion
In summary, we propose RLlib Flow, a hybrid actor-dataflow programming model for distributed RL. We designed RLlib Flow to simplify the understanding, debugging, and customization of distributed RL algorithms RL developers require. RLlib Flow provides comparable performance to reference algorithms implemented directly on low-level actor and RPC primitives, enables complex multi- and meta-learning use cases, and reduces the lines code for distributed execution in a production RL library by 2-9x. RLlib Flow is available as part of the open source RLlib project, and we hope it can also help inform the design of future RL libraries.
Acknowledgement
In addition to NSF CISE Expeditions Award CCF-1730628, this research is supported by gifts from Amazon Web Services, Ant Group, Ericsson, Facebook, Futurewei, Google, Intel, Microsoft, Scotiabank, and VMware.
10

---

References
[1] P. Carbone, Asterios Katsifodimos, Stephan Ewen, V. Markl, Seif Haridi, and Kostas Tzoumas. Apache flinkTM; Stream and batch processing in single engine. IEEE Data Eng. Bull., 38:28-38, 2015. 9, 10 [2] Itai Caspi, Gal Leibovich, Gal Novik, and Shadi Endrawis. Reinforcement learning coach, Dec. 2017. 1,
[3] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar. and Mare G. Bellemare. Dopamine: A research framework for deep reinforcement learning. CoRR, abs/1812.06 10, 2018, 2 [4] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.061 10.
2018. 9
[5] Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. CoRR, abs/1809.05214 2018. 4
[6] Prafulla Dhariwal, Christopher Hesse, Oleg Klimow, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai haselines. https // github. com/ openai/baselinen
2017. 2,9
[7] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abheel. Benchmarking deep reinforcement learning for continuous control, 2016.
[8] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yolam Doron, Vlad Firoiu, Tim Harley, lain Dunning. Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, 2018. 3
[9] WA Falcon. Pytorch lightning. Gitllu. Note: htw.:/ghhw...oltyyyytttiiiiinnnn 2019,
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks, 2017. 14
[11] G. Graefe. Volcano extensible and parallel query evaluation system. IEEE Trans. Knowl. and
Data Eng., 6(1):120-135, Feb. 1994, 5, 10
[12] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehnon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abheel, al. Soft actor-critic algorithms and applications. arXiv preprint /8:::
2018. 3
[13] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent imagination, 2020.
[14] Carl Hewitt, Peter Bishop, and Richard Steiger. Session formalisms for artificial intelligence universal modular actor formalism for artificial intelligence. In Advance Papers of the Conference, volume 3, page
235. Stanford Research Institute, 1973. 1
[15] Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth Marrn,, Feryal Behbahani, Tamara Nor- man, Abbas Abdolmalcki, Albin Cassirer. Fan Yang. Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang. Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning, 2020. 1, 6,
[16] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay, 2018.
[17] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. CoRR, abs/1803.00933, 2018. 7
[18] Alexander Kuhnle, Michael Schaarschmidt, and Kai Fricke. Tensorforce: a lensorflow library for applied reinforcement learning. Web page, 2017. 9
[19] Eric Liang, Richard Liaw, Robert Nishihara. Philipp Moritz. Roy Fox, Ken Goldberg, Joseph Gonzalez,
Michael Jordan, and lon Stoica. Rllib: Abstractions for distributed reinforcement leaming. In International Conference on Machine Learning, pages 3053-3062, 2018. 1, 2, 9
[20] Keng Wah Loon, Laura Graesser, and Milan Cvitkovie. Slm lab: A comprehensive benchmark and modular software framework for reprodueible deep reinforcement learning, 2019. 9
[21] Michael I Luo, Jiahao Yao, Richard Liaw, Eric Liang, and Ton Stoica. Impact: Importance weighted
asynchronous architectures with elipped target networks, 2020. 3
[22] Volodymyr Mnih, Adrià Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning,
2016. 3, 4
[23] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski. Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King. Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human- level control through deep reinforcement learning. Nature, 518(7540):529 5333533333 Feb. 2015. 3
11

---

[24] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang. William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging 13th {USENIX) Sympusium on Operating Systems Design and Implementation (1 (OOSD)) 18), pages 561-577, 2018.
[AI] applications.
[25] Derek G Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Martin Abadi. Naiad: timely dataflow system. In Proceedings of the Twenty-Fouu Fourth ACM Symposium on Operating Systems
Principles, pages 439-455, 455, 2013. 10
[26] Michacl Schaarschmidt, Sven Mika, Kai Fricke, and Eiko Yoneki. RLgraph: Modular Computation
Graphs for Deep Reinforeement Learning. Proceedings of the 2nd Conference Systems and Machine
Learning (SysML), Apr. 2019. 1, 2,
[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[28] Ankit Toshniwal, Siddarth Taneja, Amit Shukla, Karthik Ramasamy, Jignesh M Patel, Sanjeev Kulkarni, Jason Jackson, Krishna Gade, Maosong Fu, Jake Donham, al. Storm @ twitter. In Proceedings of the
2014 ACM SIGMOD international conference Management of data, pages 147-156, 2014, 10
[29] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lec, Irfan Essa, Devi Parikh, Manolis Savva, and
Dhruv Batra. Dd-ppo: Learning near- near-perfect pointgoal navigators from 2.5 billion frames, 2020.
[30]
Matci Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, and Ion Stoica. Discretized streams: fault-tolcrant streaming computation at scale. In Proceedings of the Twenty-Fourth ACM Sympo-
sium Operating Systems Principles SOSP '13, pages 423 438. ACM Press, 2013, 5, 9, 10
[31] Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica. Apache spark: A unified engine for big data processing. Commun. ACM, 59(11):56-65, Oct. 2016. 10
12


Reviews:
[{"review_id": "yteqfei1X5", "reviewer": "Reviewer_c8zk", "summary": "This paper propose new library for simplifying the implementation of distributed RL algorithms. It identifies the difference between RL workloads and stream processing workloads, and creates a number of abstractions for handling asynchronous parallel operations for RL workloads.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is interesting for simplifying the implementation of distributed RL algorithms. It organizes the common distributed actor processing patterns into higher level abstrations.\n\nAlthough this abstraction simplifies the implementations of existing algorithm, I wonder whether it will have lower flexibiilty for experimentation and new algorithms. For example: dynamically changing dataflow based on the rollout performance.\n\nThese abstractions can improve readability of the high-level dataflow, but it seems also hide some of the details of the underlying operations. For example, in Figure 10 b), “ppo_plan” basically hides all the details of how the operations of the PPO algorithm, and how the weights are updated. How do users gain more understanding of the actual operations (like Figure 10a)?\n\nThese high level abtractions ease the implementation of distributed RL. However, it increase the overhead when running the algorithm (for example in a single machine). Is it possible to quantify the overhead of these abstractions?\n\nFor figure 12, how is the theoretical best performance calculated using Amdahl’s law?\n\nAnonFlow’s performance is very similar to RLlib. Since AnonFlow provides a high level abstraction, does it also allow more lower level optimizations for improving the performance of RL tasks?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "OUmKqOwc2K", "reviewer": "Reviewer_fzhZ", "summary": "It is a good article that makes a significant contribution to the RL domain. It not only identifies the line of thought that hampers the development of RL-based solutions but also proposes the solution. The article makes a comparison of other approaches to their framework to a significant level of detail. Nevertheless, there are some flaws.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- A commendable work on reinforcement learning that addresses some important issues \n    - The work investigates the limitations of the present RL libraries such as ACME, RLib, and Coach. \n    - The work proposes a completely different view on the RL problem and advocates looking at it in a data flow context.\n    - It introduces the concept of data flow operators and shows by example how this framework can be used to implement RL-based solutions or experiment with the existing ones\n\n- Despite the good work, there are some flaws in the way article is presented.\n    - The framework should have a sound theoretical discussion instead of a discussion about which library to chose for the implementation and which to reject. \n    - There are always two main choices to make when implementing a distributed framework   \n        - Either to use message passing mechanism\n        - Or use a shared memory model\n    - From a theoretical point of this discussion is important, because it enables us to think above the individual libraries and frameworks. \n    - It is well known that the shared memory systems perform better in certain situations as compared to message-passing systems. In other cases, the situation is the opposite. For the sake of completeness, it is essential to have that discussion, whether to implement a data flow-based RL framework using shared memory or message passing. \n    - A key benefit of message passing systems is that they are scalable and robust, and can work in many different scenarios and machine capabilities. A shared memory-based system may perform better with greater compute capabilities of machines, but may totally fail in a situation where there are numerous small devices.\n- In the light of the above, I think the paper presents a good idea but lacks the theoretical rigor that was needed to draw some long-lasting conclusions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Because authors did not address the topic in a theoretical manner, so they were unable to draw a connection between their proposed framework and the established models of parallelism in computer science. This hampered their ability to fully analyze the capacity and limitations of their framework.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "LpeNOPQWBOv", "reviewer": "Reviewer_1Njk", "summary": "This manuscript proposes AnonFlow, a set of dataflow abstractions that allow for easy specification and implementation of distributed RL algorithms. AnonFlow consists of several composable dataflow operators/iterators implemented on top of RLlib. The manuscript discusses the implementation of several popular RL algorithms and quantifies the effectiveness of AnonFlow in terms of LoC and overall system performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors combine ideas from dataflow programming with actors and argue that this provides a more flexible programming environment for RL researchers. The methods described in the work, dataflow and actors, are not novel but the combination of these components targeting the development of RL algorithms is interesting. The distinction between the proposed and previous systems, as outlined in Table 1, makes it clear AnonFlow is meant to provide a more generic data distribution scheme alongside a higher-level interface. \n\nSupport for the proposed abstractions is provided in the form of examples, in Figures 9 and 10, that use lines of code and readability as the metrics of interest. A reduction w.r.t LOC is expected given the authors positioning of AnonFlow as a high-level interface, relative to RLlib. No performance degradation resulted from the proposed transformations as demonstrated in Figure 11.\n\nThe paper is well-written and clearly organized to support the author's RL dataflow/RL argument. Comparisons with competing dataflow and RL libraries are provided to draw attention to the distinctive advantages of AnonFlow.\n\nOverall, although there is a novelty in the expression of the RL algorithms, I think it would be difficult to gauge the significance of the work. It's unclear if the RL community would adopt and/or see value in the insights of the dataflow/actor implementation. I lean toward acceptance because IMHO the current offering of RL frameworks appears to be quite rigid and \"new\" algorithms typically require a substantial investment in time and effort. From that perspective, this work provides at least one sample of an alternative formulation worth considering to alleviate some concerns.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitations of their implementation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "FDLMUnQm9Nn", "reviewer": "Reviewer_KAKR", "summary": "This paper treats the distributed reinforcement learning as a dataflow problem, and proposes an actor-dataflow programming model named AnonFlow. The author has ported algorithm in RLib to AnonFlow, and reports the throughput comparison among AnonFlow, RLib, Spark and the theoretical optimal performance.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ It is a novel point of view to treat the distributed reinforcement learning as a dataflow problem.\n\n+ This paper proposes AnonFlow, an actor-dataflow programming model for reinforcement learning. Algorithms in RLib has been ported to AnonFlow and the lines of code for distributed execution are reduced.\n\nCons:\n- This paper claims that it is common for RL researchers to eschew existing infrastructures. However, no empirical evidence is provided in the experiment to show that RL researchers prefer to use the proposed AnonFlow, rather than existing infrastructures or writing their own approaches.\n\n- It is insufficient that only evaluating the performance of AnonFlow on several RL algorithms. This paper claims that RL researchers may wish to customize a particular algorithm into various RL tasks (e.g., single-agent, multi-agent, meta-learning) and configuration options (e.g., degree of parallelism, batch size). The author fails to evaluate the performance of AnonFlow on various RL tasks and configuration options.\n\n- The improvement of experimental results is marginal. It can be observed from Figure 11(a) that the sample throughput of AnonFlow on CartPole is less than that of RLib when the number of workers is small. In addition, Figure 11(b) shows that the training throughput of AnonFlow on Atari is almost the same as that of RLib. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers and the AC appreciate this papers contribution of a nice advance in coding machinery to improve processing for parallel streams of RL data. The authors are encouraged to carefully address the multiple suggestions for further strengthening the paper in their camera ready version.
Decision: Accept (Poster)