Exercise Classification with Machine Learning (Part I) | by Trevor Phillips | Towards Data Science

In this two-part post we’re taking a deep dive into a specific problem: classifying videos of people performing various exercises.
The first post will focus on a more algorithmic approach using k-Nearest Neighbors to classify an unknown video, and in the second post, we’ll look at an exclusively machine learning (ML) approach.
Code for everything we’re going to cover can be found on this GitHub repository. The algorithmic approach (Part I) is written in Swift and is available as a CocoaPod. The ML approach (Part II) is written in Python/TensorFlow and can be found as part of the GitHub repository.
We want to build a system which takes as input a video of a person performing an exercise and outputs a class label which describes the video. Ideally, the video can be of any length, with any frame rate, and from any camera angle. A subset of class labels might be something as follows:
back squats — correct form
back squats — incorrect form
push-ups — correct form
push-ups — incorrect form
And so on...
How to build such a system? One of the trickiest parts is that we need to identify relationships between frames — that is, relationships between the position of the person at each frame. This means we’re dealing with data structures in the time domain, which we’ll call timeseries.
Given some “unknown” timeseries which has been constructed from an input video, we want to assign a label to the timeseries according to the exercise we predict.
I mentioned that we will construct the timeseries from an input video, but how do we do that? Let’s assume that each video contains footage of a single person performing some type of exercise. My strategy was to:
Standardize the length of each videoDetermine the person’s “pose” using 14 key body points, for each frameCreate a timeseries data structure representing change in pose over timeFilter out noise in the data
Standardize the length of each video
Determine the person’s “pose” using 14 key body points, for each frame
Create a timeseries data structure representing change in pose over time
Filter out noise in the data
It’s not realistic to process a 5-minute video, so to enforce some consistency in our data, we’ll standardize the length of each video. I did this by sub-sampling a 5-second clip from the middle of each video, under the assumption that whatever is happening in the middle is “representative” of the entire video.
For videos under 5 seconds, we use the whole thing.
Given a frame (image) of the sub-clip, we want to determine the person’s pose in the image. It is already possible to estimate body points (called pose estimation), for example using the project OpenPose.
I took a pre-trained convolutional neural net (CNN) from here and used it to estimate pose for each video frame. With this ML model, we get an (x, y) coordinate for each of 14 body parts along with a confidence level in [0, 1] for the accuracy of each body part’s location. You can see implementation details on the GitHub repo.
Now everything is in place to construct a timeseries from a video. To put it together, we simply concatenate pose data for each video frame. The resulting timeseries has 4 dimensions:
Time: indexed by the video frame
Body part: 14 body parts in total
x-position: x-coordinate of a body part, normalized from [0, 1]
y-position: y-coordinate of a body part, normalized from [0, 1]
Since there are 14 body parts and an (x, y) coordinate for each, we can visualize the timeseries data structure as 28 waves (14 x2 = 28) which change over time.
The (x, y) coordinates are normalized by dividing each body part position by the width or height of the frame, respectively. The lower-left corner of the frame acts as our coordinate system’s origin.
At this point, we have a timeseries, but the noise it contains may lead to inaccurate predictions when we try to classify and assign a class label. To smooth noisy data, I used a filter called LOESS (Locally Weighted Scatterplot Smoothing).
The general idea is that for each “raw” data point, we derive a better estimate by taking a weighted average of neighboring points. Neighbors which are closer to the point we’re considering will have a higher weight and thus more effect on the average.
The nice thing about LOESS (as compared to, for example, a Kalman filter) is that there is only one parameter to consider. This parameter controls how much influence neighboring points have on the weighted average. The bigger it is, the more influence neighbors have and therefore the smoother the resulting curve.
Here’s an example of our timeseries data before and after LOESS filtering:
Now that we have a method to pre-process an input video into usable data, we need to analyze the resulting timeseries and classify videos.
Analysis can be done in many ways, but in this post, we’re going to focus on an algorithm called k-Nearest Neighbors. Basically, we’ll compare the unknown timeseries with a lot of known timeseries, and find the closest match (k=1). Comparison is done using a distance function, which we’ll talk about later.
Once we find the nearest neighbor of our unknown video, we predict that the unknown video has the same class label as the known item, since they are closest according to our distance function.
Finding the distance between points in 2 dimensions is easy:
dist = sqrt( (x2 — x1)^2 + (y2 — y1)^2 )
But what about the distance between two waves? Or the distance between two timeseries? Not so easy...
To make it more complicated, imagine the following scenario:
We have a “known” labeled timeseries from a video of a person doing push-ups. He does 1 push-up every 2 seconds and begins doing them 1 second into the video. Now we want to label an “unknown” timeseries from a video of someone else doing push-ups. He does 1 push-up every 3 seconds and begins doing them 2 seconds into the video.
Even though these timeseries are from the same exercise (push-ups), there are two problems which will emerge when we try to compare the “unknown” timeseries with the “known” timeseries: differing frequency and differing phase shift.
To help solve the problems of differing frequency and phase shift, we’re going to employ an algorithm called Dynamic Time Warping (DTW). This is a non-linear alignment strategy using dynamic programming.
As with many dynamic programming algorithms, in DTW we fill up a matrix where each cell’s value is a function relative to the neighboring cells.
Let’s say we’re comparing timeseries s of length M against timeseries q of length N. Each row of our matrix corresponds to a point in time for s, and each column corresponds to a point in time for q. Thus the matrix is MxN. The cost of a cell at row m (0 <= m < M) and column n (0 <= n < N) is as follows:
cost[m, n] =     distance(s[m], q[n]) +    min(cost[m-1, n], cost[m, n-1], cost[m-1, n-1])
Think of s[m] as the pose of the person in the unknown video at time m and q[n] as the pose of the person in the known video at time n. The distance distance(s[m], q[n]) between them is the sum of the 2D distance between each body part coordinate.
I also played around with a weighted sum of the distance between each body part, assigning more weight to body parts which are relevant for particular exercises. For example, feet are not relevant when analyzing squats (because they don’t move) so they are weighted less. But for jumping, feet move a lot and are weighted more. This technique improved the algorithm’s accuracy a little bit.
Once we fill up the DTW cost matrix, the last cell we compute is effectively the smallest “distance” between two timeseries.
It’s possible to improve the efficiency of DTW by using something called a “warping window.” In short, this means we don’t compute the entire cost matrix but rather a smaller diagonal section of the matrix. If you’re interested in learning more, check out the paper Fast Time Series Classification Using Numerosity Reduction.
Now we’ve built a system which takes an unknown video as input, converts it to a timeseries, and compares it with timeseries from the set of labeled videos using DTW. The label of the closest-matching known timeseries will be used to classify the unknown video.
To evaluate the system, I used a dataset of ~100 videos and 3 exercises: bodyweight squats, pull-ups, and push-ups. The exercise in each video was performed either correctly or incorrectly (with improper technique). Videos were recorded from the front, side, and back of the person. For side angles, I generated a second video by flipping the original across the y-axis. 80% of the videos were used as “labeled” data and the remaining 20% withheld for testing the algorithm’s accuracy.
Test results show that the algorithm is very good at distinguishing between exercises (90–100% accuracy) but not so good at classifying variations within an exercise (correct vs. incorrect technique). These variations can be subtle and hard to track, and the maximum accuracy I could achieve in this respect was ~65%.
A major disadvantage to the k-Nearest Neighbors algorithm is that the inference time — that is, the time to classify an unknown video — grows proportionally with the size of the labeled data set, since we compare an unknown timeseries with every labeled item.
In Part II, we’re going to explore an end-to-end approach using a ML model to classify videos, thereby making the inference time constant.
The main GitHub repository with video processing and DTW
OpenPose
Pose estimation on iOS
Locally Weighted Scatterplot Smoothing (LOESS)
Dynamic Time Warping (DTW)
DTW “warping window”