A Brief Introduction to Reinforcement Learning !
By: Kaustubh Dwivedi
Tags: Machine Learning, Reinforcement Learning

Welcome back to my blog ! In my last article I gave you an introduction about the outliers in machine learning datasets. Today in this article I will take you on a short tour of Reinforcement Learning. After reading it you will have a basic knowledge of what is Reinforcement Learning, its elements, its different types and how it is different from other types of learning.

What is Reinforcement Learning ?

Understanding the concept of Supervised and Unsupervised Learning is not a big deal, but when it comes to Reinforcement Learning we all get perplexed.

Reinforcement Learning is a feedback-based Machine learning technique in which an agent learns to behave in an environment by performing the actions and seeing the results of actions. For each good action, the agent gets positive feedback, and for each bad action, the agent gets negative feedback or penalty.

Example: Suppose there is an AI agent present within a maze environment, and his goal is to find the diamond. The agent interacts with the environment by performing some actions, and based on those actions, the state of the agent gets changed, and it also receives a reward or penalty as feedback.

Reinforcement Learning can be seen as a subfield of machine learning that teaches an agent how to choose an action from its action space, within a particular environment, in order to maximize rewards over time.

Elements of Reinforcement Learning

Few major components of Reinforcement Learning includes :

Environment(): A situation in which an agent is present or surrounded by. In RL, we assume the stochastic environment, which means it is random in nature.

A situation in which an agent is present or surrounded by. In RL, we assume the stochastic environment, which means it is random in nature. Agent(): An entity that can perceive/explore the environment and act upon it.

An entity that can perceive/explore the environment and act upon it. Action(): Actions are the moves taken by an agent within the environment.

Actions are the moves taken by an agent within the environment. State(): State is a situation returned by the environment after each action taken by the agent.

State is a situation returned by the environment after each action taken by the agent. Policy(): Policy is a strategy applied by the agent for the next action based on the current state.

Policy is a strategy applied by the agent for the next action based on the current state. Reward(): A feedback returned to the agent from the environment to evaluate the action of the agent.

A feedback returned to the agent from the environment to evaluate the action of the agent. Value(): It is expected long-term retuned with the discount factor and opposite to the short-term reward.

It is expected long-term retuned with the discount factor and opposite to the short-term reward. Q-value(): It is mostly similar to the value, but it takes one additional parameter as a current action.

I haven’t got deep down in the above terms as this article is just about an gentle introduction to Reinforcement Learning. Although I have included links to few references and research papers at the end of this article. Do check them out !

A real world Example : Controlling A Walking Robot

Agent: The program controlling a walking robot.

Environment: The real world.

Action: One out of four moves (1) forward; (2) backward; (3) left; and (4) right.

Reward: Positive when it approaches the target destination; negative when it wastes time, goes in the wrong direction or falls down.

In this example, a robot can teach itself to move more effectively by adapting its policy based on the rewards it receives.

How Reinforcement Learning Works

There are majorly three approaches to implement a reinforcement learning algorithm. They are -

Value Based: in a value-based reinforcement learning method, you try to maximize a value function V(s). As defined in the terminology previously, Vπ(s) is the expected long-term return of the current state s under policy π. Thus, V(s) is the value of reward which the agent expects to gain in the future upon starting at that state s.

Policy-based: in a policy-based reinforcement learning method, you try to come up with a policy such that the action performed at each state is optimal to gain maximum reward in the future. Here, no value function is involved. We know that the policy π determines the next action a at any state s. There are two types of policy-based RL methods :

1) Deterministic: at any state s, the same action a is produced by the policy π.

2) Stochastic: each action has a certain probability, given by the equation below -

Model-Based: in this type of reinforcement learning, you create a virtual model for each environment, and the agent learns to perform in that specific environment. Since the model differs for each environment, there is no singular solution or algorithm for this type.

Types of Reinforcement Learning

1.Positive Reinforcement : It is defined as when an event, occurs due to a particular behavior, increases the strength and the frequency of the behavior. In other words, it has a positive effect on behavior.

Advantages of reinforcement learning are Maximizes Performance and Sustain Change for a long period of time

Disadvantages of reinforcement learning is that too much Reinforcement can lead to overload of states which can diminish the results

2.Negative Reinforcement : It is defined as strengthening of a behavior because a negative condition is stopped or avoided.

Advantages of reinforcement learning is that it Increases Behavior and Provide defiance to minimum standard of performance.

Disadvantages of reinforcement learning it only provides enough to meet up the minimum behavior.

How is Reinforcement learning different ?

So at this point you might wonder what are the differences between Reinforcement learning and the supervised and unsupervised learning.

Here I have highlighted a few differences about the same :

1: Static vs Dynamic

The goal of supervised and unsupervised learning is to search for and learn about patterns in training data, which is quite static. RL, on the other hand, is about developing a policy that tells an agent which action to choose at each step — making it more dynamic.

2: No Explicit Right Answer

In supervised learning, the right answer is given by the training data. In Reinforcement Learning, the right answer is not explicitly given: instead, the agent needs to learn by trial and error. The only reference is the reward it gets after taking an action, which tells the agent when it is making progress or when it has failed.

3: RL Requires Exploration

A Reinforcement Learning agent needs to find the right balance between exploring the environment, looking for new ways to get rewards, and exploiting the reward sources it has already discovered. In contrast, supervised and unsupervised learning systems take the answer directly from training data without having to explore other answers.

4: RL is a Multiple-Decision Process

Reinforcement Learning is a multiple-decision process: it forms a decision-making chain through the time required to finish a specific job. Conversely, supervised learning is a single-decision process: one instance, one prediction.

Conclusion

Reinforcement learning is becoming more popular today due to its broad applicability to solving problems relating to real-world scenarios. It has found significant applications in the fields such as -

Game Theory and Multi-Agent Interaction — reinforcement learning has been used extensively to enable game playing by software. A recent example would be Google’s DeepMind which was able to defeat the world’s highest ranked Go player and later, the highest rated Chess program Komodo.

Robotics — robots have often relied upon reinforcement learning to perform better in the environment they are presented with. Reinforcement learning comes with the benefit of being a play and forget solution for robots which may have to face unknown or continually changing environments. One well-known example is the Learning Robots by Google X project.

Vehicle navigation — vehicles learn to navigate the track better as they make re-runs on the track. A proof of concept is presented in this video while a real-life world example was presented at the O’Reilly AI Conference in this video.

Industrial Logistics — industry tasks are often automated with the help of reinforcement learning. The software agent facilitating it gets better at its task as time passes. BonsAI is a startup working to bring such AI to the industries.

There are many more reinforcement learning algorithms to explore and quite a few that perform better than the simple algorithm we’ve demonstrated above. Do check out the References and Links for some quick places to start with your exploration, and mix it with the exploitation of this tutorial to reinforce your learning of the topic!

References :

Here is the link to some famous research papers from ICLR for you in case you want to dive down much deeper in the field of Reinforcement Learning :

Top 10 Reinforcement Learning Papers From ICLR 2020 (analyticsindiamag.com)