I Know What You’ll Do NEXT…
By: Valerie Huang
Tags: Blog Post, Big Data, Machine Learning, Convolutional Network

Collaborated Post by Chu Chu, Minyi Huang, Valerie Huang, Yinglai Wang

This blog is written and maintained by students in the Professional Master’s Program in the School of Computing Science at Simon Fraser University as part of their course credit. To learn more about this unique program, please visit {sfu.ca/computing/pmp}.

Poster of the Movie I Know What You Did Last Summer

Have you ever seen this horror movie before? Have you ever been curious about what others are doing? We don’t like being monitored by others, but curiosity drives us to want to know what others are doing or what they are going to do next. Through the method what we called Human Pose Estimation (HPE), your dream will come true. Wei et al. proposed Convolutional Pose Machine(CPM) in their paper which applies deep learning to HPE. CPM combines the advantages of convolutional neural network and pose machine to learn image features and image-dependent spatial models to estimate human poses.

Human Pose Estimation

For decades, HPE has attracted much attention in computer vision. It is a key step to in understanding the behavior of people in images and videos. HPE is defined as the positioning of human joints(also called key points like elbows, wrists, etc.) in an image or a video. It is also defined as searching for a specific pose in a space composed of all joint poses. The Pose Estimation problem can be classified into 2D Pose Estimation and 3D Pose Estimation. 2D Pose Estimation uses 2D coordinates(x, y) to estimate the 2D pose of each joint in the RGB image. Similarly, 3D Pose Estimation uses 3D coordinates(x, y, z) to estimate the 3D pose in the RGB image.

2D Pose Estimation

3D Pose Estimation

Applications

HPE has a variety of applications, which are widely used in action recognition animation, gaming, video surveillance and other fields. For example, a quite popular deep learning application named HomeCourt, uses Pose Estimation to analyze the movement of basketball players.

One of the interesting usages of CPM is to predict to next movement by key joint position in dancing figures.

Heads, arms and legs position prediction with CPM

Heat Map of Dancing Figures with CPM

Why is it difficult?

The main challenges in HPE, such as occlusion and unclear key points. However, all key points are related to each other due to the structure of the human anatomy. Using easy-to-recognize key points to guide the detection of difficult-to-recognize key points is beneficial. Moreover, tracking multiple persons people in one frame is far more difficult than tracking a single individual and poses its own set of problems, since multi-person pose estimation not only has to identify the number of people but also figure out the inter-person occlusion.

What is Convolutional Pose Machine?

1. Pose Machine

To develop an understanding of a Convolutional Pose Machine (CPM) it will easier to simply begin by understanding a Pose Machine first.

Pose Machine is a multi-stage sequential structure that can perform end-to-end training. It provides a continuous prediction framework for learning implicit spatial models and generates rather accurate results on spatial contexts like human postures.

The detailed structures of Pose Machines and Convolutional Pose Machines are shown in the following graph:

Detailed structures of Pose Machines and Convolutional Pose Machines

The algorithm is comprised of sequence of predictors that learn from previous mistakes to produce increasingly refined estimates for key point location. In stage 1, the predictors produce an estimation for the location of each part. After that, the predictors of subsequent stages improve their prediction by learning to correct the past mistakes, which is the stage 2. By the final stage, the purpose of the method is to accurately locate each part.

Let’s explain the different stages one by one:

Architecture of the Stages from a Pose Machine

In Pose Machine model, Stage 1 shows the model for image feature extraction and Stage 2 is the prediction model. In detail, Stage 1 takes in an image, usually a human movement image with the body in a certain pose. “x” embodies the content of the image, which is then fed into a predictor “g”. The “g’s” are different predictors that predict its “beliefs” for assigning a location to a certain body part. For example, the knee or the elbow joint. In Stage 1, the predictor is labeled “g1”. “g1” analyzes “x” and produces a set of heat maps “b1”.

This image shows a clearer view of the heat map outputs in a single stage:

The three heat maps are graphical representation of data where the individual values contained in a matrix are represented as colors. From the three sample heap maps heat maps above, we can see that they have highly multi-model appearance variation. It usually contains “hot” (red, yellow) and cold (blue, green) colors to show the locations of body parts. The hotter colors indicate the joint part of a body, and the colder colors are usually backgrounds or non-joint part. The method is designed by the guide of heat map of each part.

Following the first stage, we feed its result to Stage 2.

Stage 1 to Stage 2 in Pose Machine Model

In Stage 2, the procedure is similar to Stage 1. With the heat maps generated in Stage 1, we need a method to transform them to data that we can feed into Stage 2. Then comes the function ψ(·), which transforms the heat maps in “b1” to context features that can be fed into the next stage, in this case, Stage 2. With these context features, we again produce image features x’, let it go through a new predictor “g2” and generates new heat maps “b2” in Stage 2.

This image shows a clearer view of the procedure from Stage 1 to Stage 2:

Input Image to Stage 1 Heat Maps, and further process in Stage 2 and Stage 2 Heat Maps

Generally, the number of stages is a hyper-parameter. The first stage is always fixed, while the stages following the second are just the repetitions of stage 2. In Stage 2, we take the heat maps and the image features as input. The heat maps add spatial backgrounds for the next stage. The result is obtained by repeatedly refining the heat map in each later stages to rectify errors. The following figure shows show heat map is more accurate in identifying the joint position with further stages.

Repeated Refining Heat Map through Three Stages

2. Convolutional Pose Machine

CPM is a cascaded model based on Pose Machines, and usually contains more than two stages. The results from the last stage will become the inputs to the next stage.

Convolutional Post Machines based on Post Machines

The “C (Convolutional)” part is shown in the extended “C” and “P” stages in figure (c) and figure (d). In (a) and (b), “x” is just the image features extracted from the original input image. With CPM, the single “x” image feature model is extended into multiple convolution and pooling layers. It replaces classifiers & feature extractors in pose machines with convolution and max-pooling.

CPM makes improvements on PM so that both image and contextual feature representations can be learned directly from data. The sequence of pictures in (e) shows how the receptive field expands from a knee joint to larger space. At each specific stage of CPM, the spatial context information contained in the heat maps will provide clear cues for the subsequent stages to generate better maps with more accurate joint positioning.

Problems and Challenges

Vanishing Gradient

Since CPM consists of many stages, the gradient updates propagated to early stages of the network are ridiculously small or not-existent. To reduce the impact of this issue, CPM provides intermediate supervision while computing loss. Instead of computing loss only at the end of the network, we compute loss at every stage of the network. The loss is still calculated with L2 distance between the heat map and the ground truth.

Histograms of Gradient Magnitude During Training

From the histogram of gradient magnitudes we can observe the change in magnitude of gradients in layers at different depths in the network. For models without red-line intermediate supervision, gradient distribution of higher layers presents more uniform compared to early stages. However, the gradient magnitude distribution has a moderate variance throughout the stages of the network with intermediate supervision.

2. Receptive Field

The network is designed according to a receptive field at the higher output layers of the stage network, which supports the learning of potentially complex and long-range correlations between parts.

Wrists and Elbows Effective Receptive Field

With large receptive fields, networks are effective at modeling long-range spatial interactions between parts.

Applications

Key point detection problem addressed by current technology

There are three models aimed for resolving key point detection problems. We can rank them by detection accuracy, CPM < DeeperCut < CMU OpenPose.

In this article, we focus on the CPM model, which is the predecessor of open source project OpenPose. Pose estimation task can be categorized as Fully Convolutional Network. CPM model takes a human pose image as input, output multiple heat maps based on key point inputs.

CPM is a cascaded network that sequentially takes the previous FCN contextual as the input of next FCN.

Each stage generates a response image for the next stage (blue score in the above image)

A center map in the form of a Gaussian response is the extra input of the network. With the center map, CPM is able to deal with multi-human pose detection by telling the network its current target position.

General usage of CPM (with Tensorflow)

One of the interesting usages of CPM is to predict next movement by key joint position in dancing figures.

Here is one of the state-of-the-art models for 2D body and hand pose estimation:

Fashion Landmark Detection project using CPM

Applying CPM in Fashion Landmark Detection can locate the key points on various types of clothes.

The project uses the data provided by the Tianchi FashionAI Global Challenge, which includes 40,000 images as a training set, and 10,000 images for testing purposes. Each picture is assigned a corresponding clothing category along 5 categories: blouse, outerwear, dresses, skirts, and trousers. The training set provides annotations for 24 key points corresponding to each picture, including x coordinates, y coordinates, and whether three items of information are visible (-1 means no, 0 means exist but not visible, 1 means exist and visible).

The tricky part of this project lies in how to achieve the goal of detection with a model that is used to do prediction.

The first approach to resolve this issue starts with adding an extra key point that moves randomly across the picture, which means the input images are no longer static and the model can start training.

Result of first approach with random moving extra key point.

Result of enhanced approach with extra key point only moving along the edges

Result of first approach is not ideal since the extra point is interfering with a real feature area. The new approach fixes this issue by allowing the extra point to only move along the edge, thus not affecting the real feature area.

Testing results are shown below, which look promising.

Future Expectation

Despite the fact that the use of CPM modeling is limited human pose detection and estimation, there are applications (like the Fashion Landmarker Detection project) that can be improved using this model by reconstructing the input feature and fine tuning the model.

Right now, the CPM model requires a precise annotation as input, which increases tremendous amount of workload. There is still improvement to be done, such as detection of the key point (joint) only by posing movement (without annotation input).

Reference

Convolutional Pose Machines — Tensorflow:

https://github.com/timctho/convolutional-pose-machines-tensorflow

Convolutional Pose Machines:

https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Wei_Convolutional_Pose_Machines_CVPR_2016_paper.pdf

DeepCut Pose Estimation:

https://pose.mpi-inf.mpg.de/contents/pishchulin16cvpr.pdf

Semantic Pose Machines:

https://www.ingentaconnect.com/contentone/ist/ei/2018/00002018/00000010/art00014?crawler=true&mimetype=application/pdf

Human Pose Estimation: Simplified:

https://towardsdatascience.com/human-pose-estimation-simplified-6cfd88542ab3