Human Activity Classification on the selfBACK Data Set
By: Peijin Chen
Tags: Deep Learning, Wearables, Data Science, Gradient Boosting, Keras

Machine Learning Mini-Project

Human Activity Classification on the selfBACK Data Set

Using pycaret and keras to find good models

Photo by Ketut Subiyanto from Pexels

The SELFBACK dataset contains wearable data of 9 activity classes; 6 ambulatory activities and 3 sedentary activities, performed by 33 participants.

Data are recorded with two tri-axial accelerometers sampling at 100Hz, mounted on the dominant side wrist and the thigh of the participant.

Experimental Structure

First, we create windows of data for the supervised learning problem. This involves taking “snapshots” (snippets of a multivariate time series). Then, I will see what kind of model pycaret comes up with vs. what we can get from keras or autokeras.

Creating Data Windows for Supervised Learning

Let’s dive right in — with one of the main problems I faced in handling this data set. I wanted to create a supervised learning problem, which means that I needed to use “windows” of data (slices of time, e.g. 1 second or 1/100th of a second long) which would serve as the features to train on, while an activity (e.g. “jogging”, “walking up stairs”) would serve as the class label.

The sensors they used are running at 100 Mhz, but you might need or want to have more than 1/100th of a second as a snapshot of data to determine what kind of activity a person is doing. So let’s say you chop up the data set into windows of 100 points (1 second long snapshots). So those 6 x 100 = 600 data points (or some subset thereof) are the features— but what is the target? If you’ve been jogging that whole second, the target should obviously be “jogging”. But the problem is that there will be a transition, that is, a snapshot that might start out with a person jogging and ending up with them lying down. If you’re not careful you’ll end up labeling that snapshot as “lying down”, which doesn’t make much sense. The Python deep learning library keras has a TimeSeriesGenerator functionality which can create such feature/target data pairs from time series data so I had to cook up some code for creating windows. I decided that I would use 100 data points (1 second of measured time) — but would downsample by a factor of 10, by only using every 10th measurement. With 6 features, that means 6 x 10 = 60 features for each second. And for the next feature/target pair, I use a stride of 50, which means instead of doing the same thing with a snapshot that happens only 0.01 seconds after the last one, I advance the “tape” 0.5 seconds, and then use that as the starting point for the next 1 second snapshot. A more practical reason for not making too many snapshots close to each other is that, even if you’re using the high-RAM version of Google Colab, you might run out of memory.

The data files are labeled like “028_jogging.csv”, where the number in the front is the subject, followed by the activity. What I found is that, generally speaking, if you just train a model on 1 subject, you can get a good model, something with accuracy > 90%. If you train on all subjects at the same time, you can also get a decent performance. However, I wanted to see performance on unseen holdout data. Is one person’s jog another person’’s fast walk? So here is a snippet of code that created training data with any subject whose number started with 2–5, which I then used again to create testing data using the subjects whose number started with 6 — meaning subjects 60,61,62, and 63.

So the code for creating the training data set would be this (and ditto for the holdout data):

trainx, trainy = create_windows("0[2345]*") print(trainx.shape, trainy.shape)

Model Selection and Training-pycaret

Then we use the wonderful pycaret library in Python to find out what models are best suited for this task. My Colab notebook can be found here.

from pycaret.classification import *

best_mod = compare_models(sort = "MCC", fold = 5, turbo = True, n_select = 3)

The compare_models() functions will tell pycaret to fit all the models it has at its disposal — using 5-fold cross-validation — though adding the “turbo” agument means skip the slower ones (e.g. MLPClassifier, SVC(kernel = “rbf”)). Then you’re selecting the top 3 (or however many you want) of those models. Below are the results, when ordered by Matthews Correlation Coefficient (MCC, a balanced measure of classification performance, also known as the phi coefficient).

pycaret results

You can see that XGBoost, LightGBM and CatBoost did the best. So those are the top 3, and for fun I will then “blend” together into one model using the function blend_model(). Now this “blending” is just the sklearn implementation of VotingClassifier, which, in its simplest form, computes a majority vote between estimators to make a final prediction. I don’t think this blending really improves much. I don’t think that bagging or boosting do much either. From what I can tell, hyperparameter tuning doesn’t radically change things either. Anyhow, here’s the report for this classifier, with the precision, recall, and F1 scores per class:

Classification Report for the blended model

As you can see, there are some problems with classifying 5, 6, and 7. What are were the activities those mapped to again? Take a look:

category mappings

So what that means is that walking upstairs, walking fast, and walking moderately don’t generalize well. This makes general sense to me, because unless you are regulating my walking speed, speed is a matter of interpretation. And how people move their wrists and arms can vary a lot depending on their walking styles, and going upstairs can, at times, look more or less like walking. Here’s a confusion matrix for the blended model, which shows us performance on the holdout data.

Confusion matrix for the blended model (LightGBM+XGBoost+CatBoost)

You can see here that 6 (“walk fast”) often gets classified as 7 (“walkmod”), and that worse yet, class 7 (“walkmod”) is OFTEN mis-classified as class 8 (“walkfast”). Class 1 (“jogging”) and Class 4 (“standing”) are the best (in terms of accuracy and F1 score) — which, I suppose, makes sense. There are only so many ways people can “stand still” differently. I thought “jogging” might be more problematic, but maybe there’s enough speed to discriminate this from other activities, none of which would involve that level of motion.

You can also use class prediction error plot in pycaret to visualize and analyze how the classifier does. This gives us another way of visualizing the problem that our model has with classes 5, 6, 7, and 8:

A quick note about feature interactions — you can enable pycaret to consider feature interactions, though I have found that it helps a little, but isn’t really a game changer, the MCC and F1 scores are still in the same ball park. I also found that using 0.5 second or 2.0 second windows, and various sampling frequencies, can change the performance. I definitely saw a deterioration in performance once you get down to 0.5 seconds and about 5 stamps for that window. I did not do an exhaustive testing to see what the optimal (or optimal but also parsimonious) parameters might be. If you have, I’d love to know what you discovered.

Model Selection and Training: (auto)keras

You can find my notebook here. I first began with autokeras, the autoML version of keras. I tried two methods — one was a vanilla ImageClassifier — which is doable if your data is of a (batch_size, time_steps, n_features) shape. It will just expand the last dimension and then do some convolutions on it, before setting it up for a classification head (Dense layer + softmax activation). I didn’t want to train it for a long time, so I just tried 1 model. I also did the same using StructuredDataRegressor(). None of the results were that great — well, to be more honest, they were terrible. There was no MCC even in the ballpark of 0.74, which is what you could expect XGBoost or LGBM to attain.

Then I decided to just try it the old-fashioned way — using some RNNs. Of course, I went with LSTMs and GRUs first, which was dumb — I think one should probably always start with SimpleRNN first. Here’s an architecture I found that gives decent results comparable to the gradient boosting methods:

keras model summary

This can get us a MCC score of about 0.75. Here’s the confusion matrix:

Confusion matrix for the RNN model. It does better on certain classes compared to the gradient boosting models, but poorer on others.

So it’s fair to say that a good gradient boosting machine and a simple RNN on par in terms of what they can deliver in MCC. What would be interesting would be to see if there are better ways to window or “winnow” the sensor data — and you might do it before you train the model, or include this as some sparsity-promoting mechanism in your model.