Overview of MI and Unsupervised Video Object Segmentation through Reinforcement Learning Talk
By: Shrey Jain
Tags: Machine Learning, Reinforcement Learning, Unsupervised Learning, Computer Vision, Artificial Intelligence

I recently had the privilege to introduce and moderate Professor Pascal Poupart’s talk on Unsupervised Video Object Segmentation for Deep Reinforcement Learning. Professor Poupart teaches at the University of Waterloo in the School of Computer Science. He is a faculty member of Waterloo AI and is Canada’s CIFAR AI Chair at the Vector Institute of Technology. Additionally, Professor Poupart is the principal researcher and founder of the Waterloo RBC Borealis AI. To say the least, I was very honored to moderate a talk lead by an academic and industry leader in the machine learning space.

Image 1. This image is me with Professor Poupart during the research talk held at the University of Toronto.

This article is structured around what I learned from the event. I will also provide some general information that I believe will be interesting for the audience reading this article.

BREAKDOWN OF MACHINE LEARNING

Machine learning, simply put, can be categorized into three sections: supervised learning, unsupervised learning, and reinforcement learning.

Supervised learning is by far the most used type of learning in machine learning through feeding inputs and outputs to a model. Input and output data are labeled for classification to provide a learning basis for future data processing. The term supervised learning comes from the idea that an algorithm is learning from a training dataset, which can be thought of as the teacher. The negative aspect of supervised learning is that it is static, and the model can only be as good as the data fed to the model.

Unsupervised machine learning algorithms infer patterns from a dataset without reference to known or labeled, outcomes. Unlike supervised machine learning, unsupervised machine learning methods cannot be directly applied to regression or a classification problem because you have no idea what the values for the output data might be, making it impossible for you to train the algorithm the way you normally would. Unsupervised learning can instead be used to discover the underlying structure of the data.

Reinforcement learning is a more powerful and more comprehensive model that simulates how humans operate more closely. Reinforcement learning allows the computer to continue to improve and thus have significantly higher limits than supervised learning. Reinforcement learning works by using rewards as a scalar to measure the success of an input we are able to modify a model to optimize the reward to create a model that will align with our objective of the system. Since Professor Poupart’s talk was centered around the power of Reinforcement learning, I will focus on the importance/applications of this type of machine learning further in this article.

Figure 1. This image clearly identifies the key difference between the three main types of machine learning.

Examples and Applications of Reinforcement Learning:

Video Games

Alpha Go started off as a supervised learning problem by using a large dataset, but then evolved to using a reinforcement learning model to learn how to play by itself. This was an international event that demonstrated the power of reinforcement learning. The system starts off with a neural network that knows nothing about the game of Go. It then plays games against itself, by combining this neural network with a powerful search algorithm. As it plays, the neural network is tuned and updated to predict moves, as well as the eventual winner of the games.

Supervised learning will only ever be as good as the dataset provided, but reinforcement learning goes beyond the limits, as the AlphaGo model did when using reinforcement learning. Other successes in reinforcement learning in video games include ATARI, MuJoCo, VizDoom, ComputerGo. These simulated environments allow us to create infinite amounts of data.

Figure 2. This image shows the reinforcement machine learning process of video games.

A common question in the modern video game industry is “how can reinforcement learning be used?”. The game is fed to a deep neural network (millions of parameters) and then perform and action and then receive a reward. A sparse reward is a reward that is not indicating a lot of feedback back to the model. We need to have formative reward systems and sufficient data to create a stronger reinforcement learning model.

5G Networks

5G mobile networks are becoming more and more popular by international telecommunications companies. For individuals reading this article that do not know what 5G is, essentially 5G will reduce how long it takes to send information via wireless transmission to about one millisecond (one-thousandth of a second). It is going to increase connectivity by impacting the IoT industry. The question becomes “what comes after 5G?”. Professor Poupart has communicated with various different telecommunications organizations and believes that after 5G, networks are going to evolve from being connective to becoming intelligent through using reinforcement learning.

Cell Phones

Today’s smartphones incorporate a high amount of technology (software and hardware) that have evolved exponentially since its inception. We want to know, “what is next?”. Professor Poupart believes that phones are going to start to use reinforcement learning so that phones are going to learn how to serve the user in the most efficient way possible.

Financial Services

In the past, financial services never really implemented the use of AI or machine learning, however, present financial organizations are driven by supervised learning models. Professor Pouparts as the principal researcher at Waterloo’s RBC Borealis AI location believes that future services will be more personalized and adaptive and self-evolving through implementing reinforcement learning. (Clap this article if you want to see some more content related to my opinions on the potential impact machine intelligence will have on the financial space)

Challenges from the implementation

There are three main challenges to the implementation of reinforcement learning in the real world.

Robustness Explainability Data Efficiency

Professor Poupart’s Models

Professor Poupart's main focus of the talk was centered around Unsupervised Video Object Segmentation through Reinforcement Learning. Professor Poupart, along with two other students worked on developing/building off of the following two models:

MOREL (Motion-Oriented Reinforcement Learning)

He does this in two separate phases. Phase 1 consists of unsupervised object segmentation whereby only 1% of the frames are random actions. Phase 2 is a faster policy segmentation whereby it is based on segmentation and motion. The first take in the first phase is to focus on motion consistency.

Figure 3. This image is taken from the powerpoint to demonstrate the processes that go into the MOREL model.

Supervises segmentation (taking an image and classifying all of the different types of objects in the frame) is very labor-intensive. This is where you can leverage optical flow (building structure from the motion of pixels) whereby analyzing the pixel motion of the objects we are able to segment objects that are moving. Optical flow is not useful for static objects. MOREL is essentially a model that learns to segment moving objects and infers their motion in an unsupervised way as part of a model-free RL technique.

Figure 4. This image demonstrates the comparison between supervised segmentation and using optical flow to segment between objects.

The Structure from Motion Network (SfM-Net)

The structure from motion network frameworks is a good example of a deep neural network that takes as input two consecutive images and computes the geometry of moving objects. This architecture consists of two subnetworks: one which predicts the depth of each pixel in an image, and another which predicts camera motion and masks along with rotation and translation values for moving objects in the image. The combined information is used to compute the optical flow which attempts to reconstruct one of the input frames from the other. By minimizing the L1 difference between the reconstructed frame and the true frame, the entire network can be trained in an unsupervised fashion.

SfM-Net was initially a network designed by researchers at Google designed to segment objects that are moving in an unsupervised frame. The flow can be represented as a vector of the motion of the pixel and can be used to learn the structure of the motion of pixels through video through building a system to segment by predicting what the flow is for the next frame and so forth to finally backpropagate (using some type of gradient with adjusted parameters) to fully understand the motion of these objects. Professor Poupart’s team altered the traditional SfM-Net and made modifications to a 2D SfM-Net.