Reinforcement Learning: beyond the supervised and unsupervised ways
By: Valentina Alto
Tags: Reinforcement Learning, Data Science, Supervised Learning, Machine Learning, Unsupervised Learning

In Machine Learning, there are two main ways you could train your model depending on the type of data you are provided with:

Supervised Learning: you are provided with a set of inputs and outputs, hence your data are already labeled and you know the ‘ground truth’ behind them. It means that, once your algorithm is trained on the train set and has made predictions on the test set, you can have immediate feedbacks of its performance: indeed, you already know the real output. Let’s visualize it with a straightforward example, where our data are labeled as ‘triangles’ and ‘squares’, hence our algorithm will be asked to assign an existing class to new data:

Unsupervised Learning: in this scenario, you are provided with data which are not labeled nor categorized. Basically, you have no idea whether or not your data point is a square or a triangle ex-ante. So, the goal of the unsupervised algorithm is inferring the inner structure of those data, finding common patterns and trying to cluster them into categories (which, remember, have no targets!). Namely, our unsupervised algorithm might be able to segregate squares from triangles, yet without actually calling them ‘squares’ and ‘triangles’.

So far so good. But what happens if you are not provided with data at all, and you want your algorithm to learn from the surrounding environment? If you think at many automated systems, like self-driving cars and robotics, you can see that those machines (cars, robots) are not provided with historical data: they have to collect them while working. So the idea is that those machines have to approach their task through several attempts, learning from their mistakes (namely, crashing against a wall) and avoiding them in the next attempt.

Let’s explain this concept a little bit more by considering the following picture (which refers to the Markov decision process):

The idea is that we have an agent(the robot) and an environment (the labyrinth). That agent has a personal representation of the environment, which is called state. Then, it needs to interact with the environment and the way it does that is through actions. After each action, the agent will have a new state of the environment, as well as feedback on its action, in terms of reward or penalty.

To make it easier, imagine you are visiting an apartment for the first time and, suddenly, the environment gets dark and you cannot see anything. You want to reach the door, but you don’t know the way. How would you behave? You will probably start by walking very slowly and randomly (first set of actions), having in mind a certain representation of the apartment (state). Then, you bump into a wall on the right: now you can mentally update the state of the environment and, at the same time, you are receiving negative feedback on your action (penalty). Now, you learned that the action you did was not correct, hence you will proceed towards another direction and, surprisingly, you reach the door! Now you will receive positive feedback (reward) on your action, since you reached the goal.

The same would happen for this little robot (the agent) trying to cross the labyrinth (the environment) and reach the prize (the goal of its task).

So, the idea of a reinforced algorithm is the following: finding a set of actions (the so-called policy) so that the present (or discounted) value of rewards is maximized.

As anticipated, RL has a variety of applications, but the most revolutionary element, in my opinion, is the way it differs from any other ML method. Indeed, if you think about classification or clusterization methods (respectively, supervised and unsupervised), you are basically trying to let the model ‘emulate’ the way you would have approached the task. In RL, though, the story is different. Here, the agent will explore the environment on its own and, depending on the responses, it will update its policy, which is not an emulation of that of some humans behind the process.