Getting Started with Reinforcement Learning
By: Arsalan Anwar
Tags: Reinforcement Learning, AI, Basics Machine Learning

Overview:

What is Reinforcement Learning (RL)? How is Reinforcement Learning different from the other types of Machine Learning? How does Reinforcement Learning work? Approaches to implement Reinforcement Learning Algorithms Characteristics of Reinforcement Learning Types of Reinforcement Learning Applications of Reinforcement Learning Major Challenges faced by Reinforcement Learning

What is Reinforcement Learning?

Reinforcement Learning is a type of Machine Learning that uses dynamic programming to train algorithms to learn its environment and perform a specific task using a system of reward and punishment. A reinforcement learning algorithm, or agent, learns by interacting with its environment and taking actions for which it gets rewarded or punished.

Reinforcement Learning

How is Reinforcement Learning different from the other types of Machine Learning?

Types of Machine Learning (Courtesy: Edureka)

Supervised vs Reinforcement Learning : In both supervised and reinforcement learning, there is a mapping between input and output. But in case of supervised learning, the model learns from historical data to predict new data whereas in the case of reinforcement learning, there is a reward function which acts as a feedback to the agent where it learns from experience. For example, in a chess game, there are tens of thousands of moves that can be played. So creating a knowledge base that can be played is a tedious task. In these kind of problems, it is more feasible to learn from one’s own experiences and gain knowledge from them.

: In both supervised and reinforcement learning, there is a mapping between input and output. But in case of supervised learning, the model learns from historical data to predict new data whereas in the case of reinforcement learning, there is a reward function which acts as a feedback to the agent where it learns from experience. For example, in a chess game, there are tens of thousands of moves that can be played. So creating a knowledge base that can be played is a tedious task. In these kind of problems, it is more feasible to learn from one’s own experiences and gain knowledge from them. Unsupervised vs Reinforcement Learning: In reinforcement learning, there’s a mapping from input to output which is not present in unsupervised learning. In unsupervised learning, the main task is to find the underlying patterns rather than the mapping. For example, if the task is to suggest a news article to a user, an unsupervised learning algorithm will look at similar articles which the person has previously read and suggest anyone from them. Whereas a reinforcement learning algorithm will get constant feedback from the user by suggesting few news articles and then build a “knowledge graph” of which articles will the person like.

To understand how Reinforcement Learning works you would require the knowledge of the basic terms used in RL. Please go through the article given below, so that you are familiar with the terms used. Feel free to skip it if you are already aware.

How does Reinforcement Learning work?

Agent-Environment Interaction in Reinforcement Learning (Image Source: www.learndatasci.com)

Working of Reinforcement Learning (as visually shown in the figure above):

At time t = 0, the environment samples the initial state s0(i.e. initializes a random state for the agent to start) where s0 ~ P(s0) Then for the next state onwards i.e. for time t > 0, until the episode is done or until an optimal learning strategy is found:

Agent selects an action a(t) based on the state s(t) it encounters using a learning policy π that is defined for the problem.

that is defined for the problem. Environment samples a reward r(t) based on the action a(t) taken by the agent and updates the policy π.

Environment samples next state s(t+1) based on the action a(t) taken by the agent and stores the tuple {state, action, next state, reward, done} as an experience.

Agent receives the reward r(t) , the next state s(t+1) and done which tells the agent whether the episode is done or not

Approaches to implement RL Algorithms

There are three broad approaches to implement any Reinforcement Learning Algorithm:

1. Policy Based: The goal is to find the optimal policy such that the action performed in every state helps us gain maximum reward in the future. We have deterministic policy and stochastic policy.

A deterministic policy is a function of the form π:S→A i.e. a function from the set of states S of the environment to the set of actions A.

is a function of the form π:S→A i.e. a function from the set of states S of the environment to the set of actions A. A stochastic policy is often represented as a family of conditional probability distributions, π(A∣S), from the set of states S to the set of actions A.

Note: A probability distribution is a function that assigns a probability for each event (in this case, the events are actions in certain states) and such that the sum of all the probabilities is 1.

2. Value Based: The goal is find the optimal value i.e. the cumulative rewards. Basically, we try to maximize a value function V(s). In this method, the agent is expecting a long-term return of the current states under policy π.

Note: In Value-based we don’t store any explicit policy, we store only a value function. The policy is here implicit and can be derived directly from the value function (pick the action with the best value).

3. Model Based: The goal is to find an optimal model which is suitable for a particular environment only. Here we train our agent to learn a specific environment and the agent performs very well in that environment.

Characteristics of Reinforcement Learning

The feedback of a reward signal is not instantaneous. It is delayed by many timesteps.

Sequential decision making is needed to reach a goal, so time plays an important role in reinforcement problems.

The agent’s action affects the subsequent data it receives.

Types of Reinforcement Learning

The types of Reinforcement Learning are based on the behavioral change and impact they cause.

1. Positive Reinforcement Learning:

Positive Reinforcement is defined as an event that occurs due to a particular behavior increases the strength and the frequency of the behavior. In other words, it has a positive effect on behavior. Ex: Rewarding the agent if it takes the right action.

2. Negative Reinforcement Learning:

Negative Reinforcement is defined as strengthening of a behavior because a negative condition is stopped or avoided. In other words, we take away some negative impact whenever the agent does something good. Ex: Removing a sun tan (negative impact) if the human (agent) applies sunscreen (right action) before leaving the house.

Applications of Reinforcement Learning

Industrial Robots: To train them to perform specific tasks like segregating defective items from the good items moving on a conveyor belt, to pick and place objects in their right place, etc.

To train them to perform specific tasks like segregating defective items from the good items moving on a conveyor belt, to pick and place objects in their right place, etc. Self Driving Cars: To train cars to safely steer through the roads without hitting the obstacles and following the traffic rules.

To train cars to safely steer through the roads without hitting the obstacles and following the traffic rules. Gaming: Reinforcement Learning can be used to train bots to learn and play various games like Chess, Backgammon, etc. and these bots tend to outperform their human counterparts in a very short time.

Reinforcement Learning can be used to train bots to learn and play various games like Chess, Backgammon, etc. and these bots tend to outperform their human counterparts in a very short time. Healthcare: In healthcare, patients can receive treatment from policies learned from RL systems. RL is able to find optimal policies using previous experiences without the need for previous information on the mathematical model of biological systems. It makes this approach more applicable than other control-based systems in healthcare.

In healthcare, patients can receive treatment from policies learned from RL systems. RL is able to find optimal policies using previous experiences without the need for previous information on the mathematical model of biological systems. It makes this approach more applicable than other control-based systems in healthcare. Marketing: In marketing, the ability to accurately target an individual is very crucial. This is because the right targets obviously lead to a high return on investment. RL is able to identify these targets with a high accuracy.

Major Challenges faced by Reinforcement Learning

Major Challenges faced by Reinforcement Learning are as follows:

Too many parameters given to the model may cause delays in results and lead to longer processing and consumption of CPU power. Real world RL systems learn on limited amount of data. Most real systems do not have separate training and evaluation environments. All training data comes from the real system, and the agent cannot have a separate exploration policy during training as its exploratory actions do not come for free. Measuring its performance might then pose a big challenge. Many practical real-world problems have large and high-dimensional continuous state and action spaces. These can present serious issues for traditional RL algorithms. Safety should be priority and should never or at least rarely be violated. For this, thorough evaluation of the agent’s performance must be done and safety constraints must be put in place where human/animal interaction is involved. Otherwise it could pose a serious adverse threat to the society and the actions might have real consequences. Many real world tasks may be partially observable. (non-stationary or stochastic) For example, identical equipment may exhibit variations in performance given the same control inputs, or the state of some parts of the system (e.g. the mental state of users of a recommender system) may simply be unknowable. (Refer the article to understand Partial Observable Environments) The reward-based functions need to be designed properly. Reward functions might be unspecified, multi-objective, or risk-sensitive. In many cases, system or product owners do not have a clear picture of what they want to optimize.

Summary

Through this article, we have covered the following:

What is Reinforcement Learning (RL).

How is Reinforcement Learning different from the other types of Machine Learning.

How does Reinforcement Learning work.

Approaches to implement Reinforcement Learning Algorithms

Characteristics of Reinforcement Learning

Types of Reinforcement Learning

Applications of Reinforcement Learning

Major Challenges faced by Reinforcement Learning

Resources:

Feedback/Suggestions:

For any feedback or suggestions, please comment below or drop a mail at arsalan.anwar.7777@gmail.com or contact me via LinkedIn at www.linkedin.com/in/arsalananwar1998