Reinforcement Learning By Groundhog Day
By: Harry Zhang
Tags: Decision Making, Data Science, Machine Learning, Artificial Intelligence, Reinforcement Learning

Groundhog Day is one of the most popular comedy movies made in 1993. The movie is about a weather man, Phil Connors (Bill Murray), stuck in a time loop in which he lives the same day over and over again. The movie is not only funny but brings us thoughtful insight on the real meaning of life and happiness. More importantly, from a Data Science point of view, this movie provides a perfect setting of reinforcement learning (RL) simulation. This article aims to explain what RL is to those without any data science background by linking the similar concept in RL to this movie. The article will only focus on high level concepts rather than mathematic details. After reading this article, you should have a general impression of RL, and possibly come up with some philosophical thinking to your life or others with the guided concept from RL.

Reinforcement Learning

It is now widely agreed that Reinforcement Learning is one of three basic categories in machine learning in addition to Supervised Learning and Unsupervised Learning. The data type and training goal are two most important factors that delineate the differences among these three methods. it is beyond the scope of this article to discuss further details in different machine learning strategies, instead we will only focus on reinforcement learning concept introduction.

Reinforcement Learning (with the help of deep learning) has recently gained huge media attention via a few cutting-edge projects (i.e. Alpha-GO, Self-Driving). The fundamental idea of RL is to optimize action via agent interacting with its surrounding environment so that an accumulative reward would be maximized. The following figure describes the key concept of RL as an iterative process. The agent interacts with its environment, and at each time point it gathers information to update the state and action in order to achieve best reward. An RL model training process automates this iteration process by programming the agent interaction with a simulated environment, and the output of the model training is coming up with the optimal action decision in any state at any time point of the interaction.

One of the biggest challenges for RL study is the simulation environment development, and most of them have been associated with gaming. The following video presents one example by Google where RL training was carried out in Atari the game. At the very beginning, the bouncing bar in the bottom(agent) has very limited knowledge about the environment and misses the ball in the first a few games. But after 120 minutes of training, the model is smart enough to predict all landing positions and plays the game like a real good human player. Most impressively after 240 minutes of training, the agent learns to tunnel to the upper part of the window and score with greater efficiency with iterative ball bouncing.

RL Simulation by Groundhog Day

Most of the simulated environments in gaming are far less sophisticated than our real life ones. Imagine one day we could represent a simulated environment as complicated as our real life settings, and a human-like agent. A ML model training by interacting this human agent with the simulated environment would potentially provide us the best decision at any time in our real life. In reality we will not see this happen in a foreseen time, but in film, we’ve had a good one. Groundhog Day provides a perfect example of RL in a simulated real-life environment.

The weather man, Phil Connors, was acting as the agent in this ML model training, and the small town Punxsutawney in Pennsylvania where Groundhog Day event occurred was naturally the environment that agent interacted in the model training. Each episode of the training was defined by either the end of that day or Phil’s death.

As being mentioned previously, the RL model is improved by tuning the parameters in the training to maximize the cumulative reward. First of all we need to identify what the reward is. It is simple enough to identify the reward in a gaming environment, like Atari, which is to maximize the accumulative score in the game playing. However it is not so straightforward to define the reward for a human. Please keep it in mind that the definition of the reward is vital important that it can directly impact the overall goal of your model training.

Phil was scared, frustrated and confused at the beginning of the time loop, and intentionally defined the reward as “terminating the episode as soon as possible”. As a result he had tried his best to kill himself in multiple iterations of the model training until he found out no matter how fast he terminated in each episode, the time loop would not go away.

Next Phil redefined the reward to a normal one that most human would like to have, which was to maximize the achievement in three basic human desires, living suppliers (money), sex (love) and fame. He tried to explore each detail in the town (the environment) and kept tuning his actions in each training episode to achieve his rewarding goals. After countless iteration of the training process, Phil achieved an optimized model, which guided him at each time with the best rewarding. At the end of the day, Phil became the most popular person in the town and had his dream girl fall in love with him.

Exploration and Exploitation

As humans we often face decision-making moments. It would be ideal if we had an optimized RL model that could tell us what job we should take or what school your kids should be going that eventually our decision would maximize the probability of personal success and happiness in this society. It is certainly not a realistic goal to achieve a model like the one in Groundhog Day because each of us only have one life. But the good news are that we actually do have a RL model, except that this model was built and trained solely by our personal interaction with the real world surrounding us. The most significant difference between our model training and the one in Groundhog Day are listed in the following.

The environment of Groundhog Day model is isolated and constant (thanks for the snow storm to block all the traffic in and out), and each action would generate a reproducible reward outcome correspondingly. On the other hand our real life environment is dynamically changing which greatly impacts the model training assumption. The time loop in the movie goes against our most fundamental theory in physics that time always goes in one direction. As a result, we can never explore different reward outcomes resulting from different actions at the same time. Groundhog Day model pretty much assumes the constant physical status for the agent himself (suppose no big change would happen to a person in one day) while in real life the status of ourselves will change with the time flying. In other words, we will become older and older over long period of time.

In spite of all the challenges we have discussed in real life model training, we do have one advantage over the movie setting, which is the higher number of agents. Even though there is no way we can go backwards in time to find out the output difference upon different actions, we might find similar agents with similar environments to evaluate the output by their different actions. This is why we still would like to believe the model we are using with our latest personal experience or by observing other people’s life history are approximately optimized to provide us best decision. There are some situations where actions have never shown up in our previous life experiences(i.e. to become the first person landing on Mars) that requires exploring new action decisions to evaluate the rewarding. In RL model training, we have a similar concept called the balance of Exploration and Exploitation. The key concept of this theory is to find the best combination of your decision making based on exploiting previous experiences and exploring new opportunities. So don’t be afraid to try new things, after all life is a journey!

Please give this article 50 claps if you learned something. Thank you!