The Carrot and the Stick — Reinforcement Learning
By: Yaman Habip
Tags: Reinforcement Learning, Technology, Artificial Intelligence, Machine

Here’s a reward. Here’s a punishment. These phrases and the broader technique that accompanies them are called reinforcement learning (RL). Computer scientists used it to program artificial intelligences (AI’s) that can beat humans at complex games like Chess and Go. In this article, I will outline some of the basics of reinforcement learning and look at its many applications.

Good AI. Bad AI. — Rewards and Punishments

At its core, reinforcement learning is about letting an AI learn from its own mistakes and successes, much like humans learn. Obviously, you can’t give an AI a carrot or admonish it with a stick. But what you can do is define a reward function and a cost function for your AI that allows it to rate its own performance and adjust accordingly.

Once these two functions are defined, the goal of the AI is to maximize the reward function and minimize the cost function. The AI starts by taking some random action or move within the scope of the environment that is defined for it, and for every action it takes, it learns how that move affected the two functions. If the action gets the AI towards its goal, the AI is more likely to repeat it in the future, and vice-versa.

Benefits of Reinforcement Learning

You won’t be Picasso by painting by coloring inside the lines

The advantages of allowing an AI to make and learn from its own mistakes are twofold. First, this means that our agent can get better at a given task than humans. Other methods of machine learning always limit AI’s to the capabilities of the best human, because those AI’s are learning from the moves humans make. It’s similar to how you can’t become a better painter just by completing paint by numbers kits designed by other people, rather, you have to be creative and make your own paintings.

Second, the AI doesn’t need any initial data set. Most learning involves feeding an agent a set of data and having the agent use that data to draw certain conclusions. As an example, let’s look at an AI that makes winning moves at Chess. Other algorithms would need vast swaths of data about how given moves have historically helped or hurt a players chances at winning a game.

However, the magic of reinforcement learning is that it conducts its own research. In the chess example, instead of looking at previous master games, all our agent needs to do is be taught how to play chess and compete against itself millions of times to figure out what moves maximized its chance of victory. This ability to generate data on its own is crucial because data is not always available. Sure, there are terabytes of historical chess games around, but what about problems that no one has solved yet? Similarly, a model to suggest solutions to climate change can’t base its recommendations off of past human successes. These sorts of problems where a machine has no past successes to learn from are where RL is particularly handy.

X’s and O’s

Let’s go deep with an example about a program that plays tic-tac-toe against itself to become a perfect player. This will allow us to explore the how and why of reinforcement learning. We essentially have do two things for the machine:

First, define a cost and reward function. This is relatively easy for tic-tac-to, so easy in fact that we only need a reward function, as in zero sum games a reward is directly equivalent to not incurring a cost. The reward function breaks down like this: our AI gets a reward of 1 for winning and a reward of 0 for losing.

Let’s look at tic-tac-toe as an example

Tying is a little bit more complex, because we as programmers need to decide how valuable a tie is in comparison to a win and a loss. To illustrate, if we think that our program should win at all costs and a draw is not desirable, a tie should give a relatively small reward, perhaps 0.1 or 0.2. If, on the other hand, we see a tie as almost as valuable as a win, the reward for a tie would be closer to 1. For the purposes of this exercise, I’ll say that a tie is worth 0.2. Ties are so common in tic-tac-toe that our machine needs to disproportionately inflate the value of a win or it will likely tie every single time. We want our AI to win.

The second thing our machine needs from us is an explanation of how to play the game. This is called defining the environment in which our AI operates. This is again pretty simple, we just have to lay out the rules of tic-tac-toe:

There are nine squares

On your turn you place an X in one of the squares

On your opponent’s turn they place an O in one of the remaining squares

This process repeats until one player has three in a row or all the squares are filled

If a player got three in a row, they win

If all the squares are filled and no one has three in a row, it is a tie.

Our AI now has all the information it needs to become a perfect tic-tac-toe player. It will go through the following motions using what was listed above to achieve its goal:

Play a random game

If the moves in this game led to a reward, make those moves more often proportionally to how much of a reward it received

If these moves did not receive a reward (i.e. were punished in a zero sum game), don’t make those moves again

We now have a perfect tic-tac-toe player. This obviously has some more complex code involved, but really the code is just semantics. The key concepts outlined in this example are the fundamentals of RL.

Other Applications

Tic-tac-toe is cool and all, but most of us had it mastered by fifth grade. Let’s briefly move beyond the space of one on one games and look at some real world applications of reinforcement learning.

Can RL allocate energy more effectively?

One place where this unique brand of AI can make a huge difference is energy consumption. Right now, our use of energy is super inefficient. However by explaining to an AI that we want to maximize the amount of energy used (our reward function) and minimize its costs (our cost function) we can have it tell us the best way to use our energy. Better resource allocation is important to solving many issues, including hunger and climate change. A 2019 paper on allocating energy using RL within high density networks found:

High-density communications in wireless sensor networks (WSNs) demand for new approaches to meet stringent energy and spectrum requirements. We turn to reinforcement learning, a prominent method in artificial intelligence, to design an energy-preserving MAC [Media Access Control] protocol, with the aim to extend the network lifetime.

Another interesting real life application of RL comes from the world of online user interface. Again here, we see that there is a market inefficiency. Recommendations to users are not as useful as they could be. Reinforcement learning offers to solve this problem and save people valuable time much better than any human predictor of behavior can.

Such an application consists of developing an algorithm to maximize clicks on suggested products (a reward function) and minimize time spent on a website (a cost function). Websites can employ an RL model that runs millions of alpha-beta tests (the model would learn how to do these tests in the environment definition phase) and figures out which tests best adhere to those two functions. Such an algorithm to streamline the online shopping process was developed by researchers at Alibaba.

These sorts of applications independently make RL so interesting. But beyond that, they highlight its versatility. The ability of reinforcement learning to be molded to so many tasks is another one of its attributes that makes it such an exciting technology.

Key Takeaways

Reinforcement Learning (RL) allows a computer to teach itself how to accomplish various tasks

An AI using RL tries to maximize a reward function and minimize a cost function within some environment

RL doesn’t require any input data and thus can outperform humans at certain tasks i n a way that input driven AI’s can’t

data and thus can at certain tasks i RL can be applied in countless places from search suggestions to resource rationing

Further Reading

If you want to learn more, there are loads of resources online. This video is excellent and this book is a great in depth exploration of the subject. All the sources cited here are also quite helpful. If you want to follow up with me, have a conversation, or have further questions, here is my contact info:

Email | Linkedin | Github

And please subscribe to my newsletter!