Reinforcement Learning frameworks | by Jordi TORRES.AI | Towards Data Science

Welcome to this 20th post that concludes the “Deep Reinforcement Learning Explained” series that presents a practical approach to getting started in the exciting world of Deep Reinforcement Learning.
So far, in previous posts, we have been looking at a basic representation of the corpus of RL algorithms (although we have skipped several) that have been relatively easy to program. But from now on, we need to consider both the scale and complexity of the RL algorithms. In this scenario, programming a Reinforcement Learning implementation from scratch can become tedious work with a high risk of programming errors.
To address this, the RL community began to build frameworks and libraries to simplify the development of RL algorithms, both by creating new pieces and especially by involving the combination of various algorithmic components. In this post, we will make a general presentation of those Reinforcement Learning frameworks and solving the previous problem of CartPole using RLlib, an open-source library in Python, based on Ray framework.
Spanish version of this publication
medium.com
But before continuing, as a motivational example, let’s remember that in the previous post, we presented REINFORCE and its limitations. The research community created many training algorithms to solve it: A2C, A3C, DDPG, TD3, SAC, PPO, among many others. But programming these algorithms from scratch becomes more convoluted than that of REINFORCE. Also, the more involved you become in the field, the more often you will realise that you are writing the same code over and over again.
The practical applications of Reinforcement Learning are relatively young compared to other domains as Deep Learning, where well-established frameworks as TensorFlow, PyTorch, or MXnet simplify DL practitioners’ lives. However, the emergence of RL frameworks has already begun and right now we can choose from several projects that greatly facilitate the use of advanced RL methods.
Before presenting these RL frameworks, let’s see a bit of their context.
In the last several years pattern-recognition side has been the focus of much of the work and much of the discussion in the community of Deep Learning. We are using powerful supercomputers that process large labeled data sets (with expert-provided outputs for the training set), and apply gradient-based methods that find patterns in those data sets that can be used to predict or to try to find structures inside the data.
This contrasts with the fact that an important part of our knowledge of the world is acquired through interaction, without an external teacher telling us what the outcomes of every single action we take will be. Humans are able to discover solutions to new problems from interaction and experience, acquiring knowledge about the world by actively exploring it.
For this reason, current approaches will study the problem of learning from interaction with simulated environments through the lens of Deep Reinforcement Learning (DRL), a computational approach to goal-directed learning from the interaction that does not rely on expert supervision. I.e., a Reinforcement Learning Agent must interact with an Environment to generate its own training data.
This motivates interacting with multiple instances of an Environment in parallel to generate faster more experience to learn from. This has led to the widespread use of increasingly large-scale distributed and parallel systems in RL training. This introduces numerous engineering and algorithmic challenges that can be fixed by these frameworks we are talking about.
In recent years, frameworks such as TensorFlow or PyTorch (we have spoken extensively about both in this blog) have arisen to help turn pattern recognition into a commodity, making deep learning easier to try and use for practitioners.
A similar pattern is beginning to play out in the Reinforcement Learning arena. We are beginning to see the emergence of many open source libraries and tools to address this both by helping in creating new pieces (not writing from scratch), and above all, involving the combination of various prebuild algorithmic components. As a result, these Reinforcement Learning frameworks help engineers by creating higher-level abstractions of the core components of an RL algorithm. In summary, this makes code easier to develop, more comfortable to read, and improves efficiency.
Next, I provide a list of the most popular RL frameworks available. I think the readers will benefit by using code from an already-established framework or library. At the time of writing this post, I could mention the most important ones (and I’m sure I’m leaving some of them out):
Keras-RL
RL Coach
ReAgent
Ray+RLlib
Dopamine
Tensorforce
RLgraph
Garage
DeeR
Acme
Baselines
PFRL
StableBaselines
Deciding which one of the RL frameworks listed here to use, depends on your preferences and what you want to do with it exactly. The reader can follow the links for more information.
We are using Acme for doing research at our research center. But to describe one of these environments so that the reader can see the possibilities they offer, I have personally opted for RLlib based in Ray for several reasons that I will explain below.
We are using Acme for doing research at our research center. But to describe one of these environments so that the reader can see the possibilities they offer, I have opted for RLlib based in Ray for several reasons that I will explain below.
Deep Reinforcement Learning algorithms involve a large number of simulations adding another multiplicative factor to the computational complexity of Deep Learning in itself. Mostly this is required by the algorithms we have not yet seen in this series, such as the distributed actor-critic methods or multi-agents methods, among others.
But even finding the best model often requires hyperparameter tuning and searching among various hyperparameter settings; it can be costly. All this entails the need for high computing power provided by supercomputers based on distributed systems of heterogeneous servers (with multi-core CPUs and hardware accelerators as GPUs or TPUs).
Two years ago, when I debuted as an author on Medium, I already explained what this type of infrastructure is like in the article “Supercomputing”. In Barcelona, we now have a supercomputer, named Marenostrum 4, which has a computing power of 13 Petaflops.
Barcelona Supercomputing Center will host a new supercomputer next year, Marenostrum 5, which will multiply the computational power by a factor of x17.
The current supercomputer MareNostrum 4 is divided into two differentiated hardware blocks: a block of general-purpose and a block-based on an IBM system designed especially for Deep Learning and Artificial Intelligence applications.
In terms of hardware, this part of the Marenostrum consists of a 54 node cluster based on IBM Power 9 and NVIDIA V100 with Linux operating system and interconnected by an Infiniband network at 100 Gigabits per second. Each node is equipped with 2 IBM POWER9 processors with 20 physical cores each and 512GB of memory. Each of these POWER9 processors is connected to two NVIDIA V100 (Volta) GPUs with 16GB of memory, a total of 4 GPUs per node.
How can this hardware fabric be managed efficiently?
Accelerating Reinforcement Learning with distributed and parallel systems introduce several challenges in managing the parallelization and distribution of the programs’ execution. To address this growing complexity, new layers of software have begun to be proposed that we stack on existing ones in an attempt to maintain logically separate the different components of the layered software stack of the system
Because of this key abstraction, we can focus on different software components that today supercomputers incorporate in order to perform complex tasks. I like to mention that Daniel Hillis, who co-founded Thinking Machines Corporation, a company that developed the parallel Connection Machine, says that the hierarchical structure of abstraction is our most important tool in understanding complex systems because it lets us focus on a single aspect of a problem at a time.
And this is the case of RLlib, the framework for which I opted, that follows this divide and conquer philosophy with a layered design of the software stack.
This hierarchical structure of abstraction that allows this functional abstraction is fundamental because it will let us manipulate information without worrying about its underlying representation. Daniel Hillis says that once we figure out how to accomplish a given function, we can put the mechanism inside a ”black box” of a ”building block” and stop thinking about it. The function embodied by the building block can be used over and over, without reference to the details of what’s inside.
In short, parallel and distributed computing is a staple of Reinforce Learning algorithms. We need to leverage multiple cores and accelerators (on multiple machines) to speed up RL applications, and Python’s multiprocessing module is not the solution. Some of the RL frameworks, like Ray can handle this challenge excellently.
On the official project page, Ray is defined as a fast and simple framework for building and running distributed applications:
Providing simple primitives for building and running distributed applications.Enabling end-users to parallelize single machine code, with little to zero code changes.Including a large ecosystem of applications, libraries, and tools on top of the core Ray to enable complex applications.
Providing simple primitives for building and running distributed applications.
Enabling end-users to parallelize single machine code, with little to zero code changes.
Including a large ecosystem of applications, libraries, and tools on top of the core Ray to enable complex applications.
Ray Core provides simple primitives for application building. On top of Ray Core, beside RLlib, there are other libraries for solving problems in machine learning: Tune (Scalable Hyperparameter Tuning), RaySGD (Distributed Training Wrappers), and Ray Serve (Scalable and Programmable Serving).
RLlib is an open-source library for reinforcement learning that offers both high scalability and a unified API for a variety of applications. RLlib natively supports TensorFlow, TensorFlow Eager, and PyTorch, but most of its internals are framework agnostic.
At present, this library already has extensive documentation ( API documentation), offering a large number of built-in algorithms in addition to allowing the creation of custom algorithms.
The key concepts in RLlib are Policies, Samples, and Trainers. In a nutshell, Policies are Python classes that define how an agent acts in an environment. All data interchange in RLlib is in the form of Sample batches that encode one or more fragments of a trajectory. Trainers are the boilerplate classes that put the above components together, managing algorithm configuration, optimizer, training metrics, the workflow of the execution parallel components, etc.
Later in this series, when we have advanced more in distributed and multi-agent algorithms, we will present in more detail these key components of RLlib.
In a previous post, TensorFlow vs. PyTorch: The battle continues, I showed that the battle between deep learning heavyweights TensorFlow and PyTorch is fully underway. And in this regard, the option taken by RLlib, allowing users to seamlessly switch between TensorFlow and PyTorch for their reinforcement learning work, also seems very appropriate.
To allow users to easily switch between TensorFlow and PyTorch as a backend in RLlib, RLlib includes the “framework” trainer config. For example, to switch to the PyTorch version of an algorithm, we can specify {"framework":"torch"}. Internally, this tells RLlib to try to use the torch version of a policy for an algorithm (check out the examples of PPOTFPolicy vs. PPOTorchPolicy).
Now, we will show a toy example to get you started and show you how to solve OpenAI Gym’s Cartpole Environment with PPO algorithm using RLlib. PPO is one of the proposals that solves the limitations of REINFORCE, introduced in the paper “Proximal Policy Optimization Algorithms” by John Schulman et al. (2017) at OpenAI. But the reader can use the code proposed in this section to test any of the algorithms already programmed in this framework.
The entire code of this post can be found on GitHub and can be run as a Colab google notebook using this link.
Warning: Given that we are executing our examples in Colab we need to restart the runtime after installing ray package and uninstall pyarrow.
The various algorithms you can access are available through ray.rllib.agents. Here, you can find a long list of different implementations in both PyTorch and Tensorflow to begin playing with.
If you want to use PPO you can run the following code:
import rayfrom ray.rllib.agents.ppo import PPOTrainer, DEFAULT_CONFIGray.init()
The ray.init() command starts all of the relevant Ray processes. This must be done before we instantiate any RL agents, for instance PPOTrainer object in our example:
config = DEFAULT_CONFIG.copy()config["num_gpus"] = 1 # in order to use the GPUagent = PPOTrainer(config, 'CartPole-v0')
We can pass in a config object many hyperparameters that specify how the network and training procedure should be configured. Changing hyperparameters is as easy as passing them as a dictionary to the config argument. A quick way to see what’s available is to call trainer.config to print out the options that are available for your chosen algorithm:
print(DEFAULT_CONFIG){‘num_workers’: 2, ‘num_envs_per_worker’: 1, ‘rollout_fragment_length’: 200, ‘num_gpus’: 0, ‘train_batch_size’: 4000, ...}
Once we have specified our configuration, calling the train() method on our trainerobject will update and send the output to a new dictionary called results.
result = agent.train()
All the algorithms follow the same basic construction alternating from lower case abbreviation to uppercase abbreviation followed by Trainer . For instance, if you want to try a DQN instead, you can call:
from ray.rllib.agents.dqn import DQNTrainer, DEFAULT_CONFIGagent = DQNTrainer(config=DEFAULT_CONFIG, env='CartPole-v0')
The simplest way to programmatically compute actions from a trained agent is to use trainer.compute_action():
action=agent.compute_action(state)
This method preprocesses and filters the observation before passing it to the agent policy. Here is a simple example of how to watch the Agent that uses compute_action():
def watch_agent(env):   state = env.reset()   rewards = []   img = plt.imshow(env.render(mode=’rgb_array’))   for t in range(2000):       action=agent.compute_action(state)       img.set_data(env.render(mode=’rgb_array’))       plt.axis(‘off’)       display.display(plt.gcf())       display.clear_output(wait=True)        state, reward, done, _ = env.step(action)       rewards.append(reward)       if done:          print(“Reward:”, sum([r for r in rewards]))          break       env.close()
Using watch_agent function, we can compare the behavior of the Agent before and after being trained running multiple updates calling the train() method for a given number:
for i in range(10):   result = agent.train()   print(f'Mean reward: {result["episode_reward_mean"]:4.1f}')
The last line of code shows how we can monitor the training loop printing information included in the return of the method train().
This is a toy implementation of a simple algorithm to show this framework very briefly. The actual value of the RLlib framework lies in its use in large infrastructures executing inherently parallel and, at the same time, complex algorithms were writing the code from scratch is totally unfeasible.
As I said, I opted for RLlib after taking a look at all the other frameworks mentioned above. The reasons are diverse; some are already presented in this post. Add that for me; it is relevant that it has already been included in major cloud providers such as AWS and AzureML. Or that there is a pushing company like ANYSCALE that has already raised 20 million and organizes the Ray Summit conference, which will be held online this week (September 30 through October 1) with great speakers (as our friend Oriol Vinyals ;-). Maybe add more context details, but for me, just as important as the above reasons is the fact that there are involved great researchers from the University of California at Berkeley, including the visionary Ion Stoica, whom I met about Spark, and they clearly got it right!
I hope these posts have served to encourage readers to get introduced to this exciting technology that is the real enabler of the latest disruptive advances in the field of Artificial Intelligence. We haven’t really seen much, just the bare minimum of basics that will allow you to continue on your own to discover and enjoy this fabulous world. The following references may be useful.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction, by MIT Press, 2018. [on-line available from incompleteideas.net]. Related independent repo of Python code.
Miguel Morales. Grokking Deep Reinforcement Learning. MANNING, 2020.
Alexander Zai and Brandon Brown. Deep Reinforcement Learning in Action. MANNING, 2020.
Maxim Lapan. Deep Reinforcement Learning Hands-on. Packt Publishing Ltd., 2nd edition, 2020.
Micheal Lanham. Hands-on Reinforcement Learning for Games. Packt Publishing, 2020.
Phil Winder. Reinforcement Learning, Industrial Applications and Intelligent Agents. O’Really Media Inc., 2020.
Sudharsan Ravichandiran. Deep Reinforcement Learning with Python. Packt Publishing Ltd., 2nd edition, 2020.
UCL Course on RL by David Silver
UC Berkeley CS 285 by Sergey Levine
DeepMind & UCL by Hado Van Hasselt
Stanford CS234 by Emma Brunskill
UPC Barcelona Tech by Mario Martin
University of Waterloo CS 885 by Pascal Poupart
Berkeley DRL Bootcamp by Pieter Abbeel et all
UPC Barcelona Tech by Xavier Giró-i-Nieto et all
OpenAI by Josh Achiam
Lilian Weng: A (Long) Peek into Reinforcement Learning
Papers with code
Alex Irpan: Sorta Insightful: Deep Reinforcement Learning Doesn’t Work Yet
Andrej Karpathy Blog: Deep Reinforcement Learning from Pixels
by UPC Barcelona Tech and Barcelona Supercomputing Center
A relaxed introductory series that gradually and with a practical approach introduces the reader to this exciting technology that is the real enabler of the latest disruptive advances in the field of Artificial Intelligence.
I started to write this series in May, during the period of lockdown in Barcelona. Honestly, writing these posts in my spare time helped me to #StayAtHome because of the lockdown. Thank you for reading this publication in those days; it justifies the effort I made.
Disclaimers — These posts were written during this period of lockdown in Barcelona as a personal distraction and dissemination of scientific knowledge, in case it could be of help to someone, but without the purpose of being an academic reference document in the DRL area. If the reader needs a more rigorous document, the last post in the series offers an extensive list of academic resources and books that the reader can consult. The author is aware that this series of posts may contain some errors and suffers from a revision of the English text to improve it if the purpose were an academic document. But although the author would like to improve the content in quantity and quality, his professional commitments do not leave him free time to do so. However, the author agrees to refine all those errors that readers can report as soon as he can.
Our research group at UPC Barcelona Tech and Barcelona Supercomputing Center is doing research on this topic. Our latest paper in this area is “Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills” presented at the 37th International Conference on Machine Learning (ICML2020). The paper presents a novel paradigm for unsupervised skill discovery in Reinforcement Learning. It is the last contribution of @vcampos7, one of our Ph.D. students, co-advised with@DocXavi. This paper is co-authored with @alexrtrott, @CaimingXiong, @RichardSocher from Salesforce Research.