9 Promising Applications of Reinforcement Learning in 2021
By: Mugoh Mwaura
Tags: Machine Learning, Artificial Intelligence, Reinforcement Learning, AI

9 Promising Applications of Reinforcement Learning in 2021

Photo by Becky Hudson on Unsplash

Reinforcement Learning(RL) provides solutions to a sequential decision making problem or a problem that can be re-structured as sequential in nature. Such puzzles do not depend on a single decision made at a certain point in time but on an entire sequence of trailing choices — an example of this is treatment procedures in healthcare.

It is desirable to run RL systems in the real world and have real benefits. This is something that has seen growing application success, but with its own set of challenges.

This post explores exciting applications of RL in the real world that promise to give beneficial use cases, as discussed in this year’s RL for real-life workshop. Most uses might receive more attention, not just in the coming days, but onwards. This list might be subjective to the research areas of the participants in the discussion. But most of the applications, as of now, are not trendy. For each application, I will identify which new solution approach RL offers, an example of how it’s being done, and where necessary, what is unexplored.

New and current focus areas of Reinforcement Learning in the real world.

1. Recommender systems

Recommendations help personalize a user’s preferences. In music and video platforms, for example, recommendations help members find entertainment to engage with and enjoy while maximizing satisfaction and user retention. Recommender systems (RecSys) have, for a long time, been used to predict the next item of interest to a user. RecSys approaches like filtering, causality or bandits may have the ability to remember historical experiences from user interactions and be able to play them back when needed. But they are restricted in the sense that they cannot plan.

“A good recommender system would be able to curve a true learning curriculum for [anyone trying to learn a skill] over time” Ed Chi, Google Brain

This need for the ability to plan is a motivation for using RL in recommender systems. The key benefit of RL is that it personalises for the user to maximize long term satisfaction. In the recent RecSys 2020 conference, Netflix presented RL as a recent trend in their personalization, where they use it to optimize varied user interactions. These optimizations happen in a user session and for user interactions across multiple device sessions.

There is a lot of recent research involving RL in recommender systems, and this seems to be an area that started receiving focus in 2018. It now extends to include news-feed recommendations and music personalization.

2. Notification Systems

This falls under Recommender Systems.

While a recommender system shows the user engaging content upon a form of a user request, a notification system invites the user to engage with such content every time it’s newly available or is updated.

However, push notifications can be irritating to users when sent at inappropriate times. So can promotional emails. A desirable notification system should determine which users get notified and when to increase positive user-engagement. RL has been used to determine the right time to send a notification to a given user. When successful, that minimizes ignoring of user-notifications and creates positive user satisfaction.

We can also reframe this as using RL to predict a user’s behaviour towards a notification, the intent being to minimize the negative impact on users such as a decrease in current task performance.

3. Mobile Robots

Much focus on reinforcement learning in robots has been in object manipulation, which has given rise to robots with some impressive capabilities such as solving Rubiks’ cube. It helps robots learn to handle and move objects around effectively.

Photo by James Pond, Unsplash

However, deep RL hasn’t had as much application in continuous control of real mobile robots. Early efforts used Q-learning, which could only handle discrete state and action spaces. In contrast, robots have continuous spaces — as an example, rotating an arm will involve many degrees of freedom.

With more interest arising in using RL for mobile robot systems, we will have more robots with the ability to move and interact with the world, collecting diverse experiences and generalizing better to different observations or datasets.

The progress in RL for real-world mobile robots has recently had inputs such as the use of parallel training environments, combining hierarchical RL with environment feature extractors for improved path planning, and mapless robot navigation.

Another promising area is the use of RL in real-world mobile robot swarms.

4. Guidance, navigation and flight control

I was pretty surprised to see RL is openly being studied in advancing missile guidance, given the concerns this might have, as expressed through The Future of Life Institute.

That said, there have been applications of RL in un-manned aerial vehicles (UAV) and flight control. But flight systems such as drones used for data collection, surveillance and monitoring, have mostly been task-specific.

RL helps flight agents find efficient routes which are needed from the constraints laid on task-time and battery life, and also perform long-range navigation planning.

Success in UAVs and flight control will have impactful benefits in search and rescue operations, identification of forest fires, gas leaks, or even in food and parcel deliveries. So far, quadrotors can navigate without prior knowledge of the environment model, and this recent work shows that deep RL can be coupled with memory and incremental curriculum learning for quick navigation in safety-critical situations. Hierarchical RL has proven useful in long-horizon navigation and interactive navigation i.e., navigation that involves object manipulation at some point, and is likely to be used more in improving navigation control techniques.

5. Computer Systems

RL approaches have inherent ability to control dynamic behaviour using raw system performance. However, practical applications in computer systems have been elusive due to large data requirements. It’s an area which offers the potential possibility of collecting a lot of data and would benefit from the adoption of standard tools for model evaluation.

What would RL optimize here?

Systems have a wide variety of parameter configurations needed to customize aspects that influence computer performance. These include cluster scheduling for jobs with different dependencies, adaptive video streaming which determines the optimal bitrate for network bandwidth and cache cleaning.

Adaptive Bit-rate Agent (source)

These parameters get determined through manual implementations, community recommendations, and of late, autotuning techniques like Bayesian optimization. The issue with these procedures is they produce static parameter configurations offline. In contrast, RL suggests matching different configurations to distinct tasks online, based on continuous monitoring and adaptive control. Remember such an agent needs to make decisions indefinitely with no terminal state.

Schaarschmidt identifies a need to decouple the design of RL from the execution process to make the execution more flexible in different distribution systems.

Imagine computer systems that can continuously self-configure — it would mean computers that work better for us.

6. Database systems

Database queries and configurations are representable as Markov Decision Processes (MDPs) that optimize database access. For example, in the ordering of join queries, we can frame the MDP as:

State: The set of relations to be joined

Action: A valid join between any two relations

Next State: Present record-relations after two joined relations get removed from the set

Reward: Cost estimate of the join

Deep RL has approaches to tailor the query plan to a specific dataset, the query workload and observed access cost. The cost might be expressed as the total runtime or the number of queries required to meet an objective. Deep Q-learning has achieved drastic speedups for join queries involving large amounts of data compared to dynamic programming access methods.

In database optimization, RL has plenty of room for evaluation in big data technologies like Apache Spark, optimizing database indexing, including predicting queries to index ahead of time. To save on resources, it’s also desirable to have agents that can efficiently scale to large database systems when trained in less resource-consuming data environments.

7. Optimizing Compilers

Bristol Robotics Lab | Image by Louis Reed, Unsplash

Why would anyone want to use Machine Learning in compilers? While translating code to binary, correctness is critical.

Still, there’s the aspect of optimization. Besides translating code correctly, compilers need to find the most efficient translation. The vast majority of engineering focus has been on this second objective of performance. Computer architecture has been evolving at a fast pace, each generation having its compilation quirks. And instead of reliance on heuristics, Machine Learning is suited to make code optimization decisions where the performance impact is reliant on the underlying platform.

Compilers these days let you specify an optimization level on a build. What this does is trigger a sequence of passes on the code. But these passes are initialized with some pre-fixed parameters and run in a pre-defined order. The order of applying such compiler optimizations to code, known as phase ordering, is a long-standing problem in compiler research, for each of those optimization phases interacts with the intermediate code in complex ways.

It’s framable as an RL problem aiming to optimize for the compiler optimization strategy tailored to a specific intermediate representation(IR). The states get represented by the current code IR and history of optimization phases. The reward depends on the runtime of the new IR compared to the previous state IR.

“a reduction in runtime produces a positive reward for that phase while an increase produces a negative one”

From exploration, the agent learns to assign the correct value to the optimization strategies.

The future of RL in compiler optimization will also involve agents that can meta-learn or do self-supervised learning in new target systems. That would save on computational resources as the current design requires programs to be compiled, and benchmarked for performance on unseen systems.

8. Market Optimization

If we are at a conference with lots of people, and all search “restaurant” in our favourite review app, we will get the restaurant with the best reviews, which works well for us — the hungry, but not so much for the restaurant. (This is an example given by Jason Gauci). What will have happened is the app has optimized for just one side of the market — the customer, and not for the restaurant.

This form of decision making involving two parties, who want a solution optimizing for both of their needs (almost) equally, is a factor in many market places. Take, for example, planning to hitch a bodaboda ride to avoid traffic and get somewhere quickly, but the rider also wants to save on fuel and economize. Such an optimization framework in ride-sourcing services can use RL to minimize passenger waiting time while promoting driver utility. Such is a bipartite matching problem, on which graph matching is the prevalent approach.

Bodabodas in Arusha, Tanzania | Hugo Ramos, Unsplash

Application of RL for market optimization problems is scalable to services like accommodation sharing between tourists and locals, where we desire to optimize for the preferences of both the host and the tourist. Any market setting involving this bipartite balance between two parties stands to benefit from the ability of RL to plan and make decisions. And many markets in the hospitality and service industries are of this nature.

9. Scientific advancement

There is potential for RL to drive further scientific discoveries. Quite interesting. For example, vaccine discovery and optimal sequencing in biology. RL applies to sequence decoding. Protein sequence design problems comprise a sequence encoding bar and a decoding bar. By use of an RL policy, it’s possible to control the decoding. Pointer models and graphical neural networks are some of the approaches that get prevalent use in sequence design problems.

Gene sequencing needs a good understanding of chemical building blocks, which might not easily be represented by a neural network.

Most scientific problems that involve planning and optimizing of production procedures can benefit from reinforcement learning. For example, designing computer chips with deep RL has been used to optimize placements for accelerator chips in TPUs at Google.

With these kinds of problems, success in one gives insight that might prove valuable in progressing the other. A naive example is in the mentioned chip design RL solution, where the goal is to place blocks on the computer chip optimally. Though with added difficulty, the approach used can have some transferrable aspects to the protein sequencing problem, which also involves optimizing for the placement of (building) blocks.