Recognition of human activity with smartphones using deep learning
By: Antonio Castiglione
Tags: Startup, TensorFlow, Deep Learning, Healthcare, Machine Learning

Immagine di Pixabay

Nowadays, we all have the possibility to keep under control our physical activity, thanks to the smart phone or smartwatch, keeping track of how many steps we take, kilometers traveled, if we have a sedentary lifestyle, calories consumed etc. How these devices under control our motor activity?

Through the use of two sensors in particular such as: Gyroscope that captures the triaxial angular velocity and the Accelerometer that captures the triaxial linear acceleration. But it’s a challenging problem given the large number of observations produced every second, the temporal nature of the observations, and the lack of a clear way to correlate accelerometer data to known motions.

Information about the dataset

The dataset was constructed from records of 30 study participants performing activities of daily living (ADL) while carrying a waist-mounted smartphone with built-in inertial sensors. The goal is to classify the activities in one of the six activities carried out:

WALKING, WALKING_UPSTAIRS,WALKING_DOWNSTAIRS,

SITTING, STANDING, LAYING

On youtube there is a video explaining how the data was collected: https://www.youtube.com/watch?v=xjn8LaWR6Mk

The dataset used in this project has been made available and can be downloaded for free from the UCI Machine Learning Repository:

The experiments have been carried out with a group of 30 volunteers within an age bracket of 19–48 years. Each person performed six activities:

WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS,

SITTING, STANDING, LAYING

wearing a smartphone on the waist. Using its embedded accelerometer and gyroscope, we captured 3-axial linear acceleration and 3-axial angular velocity at a constant rate of 50Hz. The experiments have been video-recorded to label the data manually. The obtained dataset has been randomly partitioned into two sets, where 70% of the volunteers was selected for generating the training data and 30% the test data.The signals from accelerometer and gyroscope were pre-processed by applying noise filters and then sampled in fixed-width sliding windows of 2.56 sec and 50% overlap. The sensor acceleration signal, which has gravitational and body motion components, was separated using a low-pass filter into body acceleration and gravity. The gravitational force is assumed to have only low frequency components,From each window, a vector of features was obtained by calculating variables from the time and frequency domain.

The aim is that once the subject’s activity is recognized and known, an intelligent computer system can then offer assistance.

Now, step by step, let’s see now how to classify the states of an individual from the data in our possession.

Data discovery

Load Data:

we have 2 separate data sets for training and testing:

Shape Train: (7352, 563)

Shape Test: (2947, 563)

A part of the train data set (5 x 563)

Our target variable is Activity

Almost same distribution so that is not imbalanced

LAYING 1407

STANDING 1374

SITTING 1286

WALKING 1226

WALKING_UPSTAIRS 1073

WALKING_DOWNSTAIRS 986

Now we can see data for ‘Activity’ provided by each user (30 users)

Which Sensor Is More Important For Classifying Participants By Walking Style?

Since there are so many features I am going to group them by their sensor (accelerometer = Acc, gyroscope = Gyro)

Image related to train data

Exploratory Data Analysis Based on the common nature of activities we can broadly put them in two categories.

Static and dynamic activities :

SITTING, STANDING, LAYING can be considered as static activities WALKING, WALKING_DOWNSTAIRS, WALKING_UPSTAIRS can be considered as dynamic activities.

To better understand this, let’s look at the variable Acceleration Magnitude Mean in detail:

If Acceleration Magnitude Mean is < -0.8 then the Activities are Static.

If Acceleration Magnitude Mean is > -0.6 then the Activities are Dynamic.

If Acceleration Magnitude Mean> 0.0 then the Activity is WalkingDownstairs.

Preprocessing data

Before moving on to run the Machine Learning model, you need to prepare the data:

Splitting data: within the already divided train and test data, let’s separate the target variable, i.e. Activity, from them

Encoding labels: refers to converting the labels into numeric form so as to convert it into the machine-readable form. Machine learning algorithms can then decide in a better way on how those labels must be operated. It is an important pre-processing step for the structured dataset in supervised learning.In our case the categories in Activity will be translated into numbers.

Scaling Data:is a method used to normalize the range of independent variables or features of data. In data processing, it is also known as data normalization and is generally performed during the data preprocessing step.

After data discovery and preprocessing, it is time to adapt the model for data classification. We will use neural networks with Tensorflow.

Deep learning (also known as deep structured learning) is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Artificial neural networks (ANNs) were inspired by information processing and distributed communication nodes in biological systems. ANNs have various differences from biological brains. Specifically, neural networks tend to be static and symbolic, while the biological brain of most living organisms is dynamic (plastic) and analog.

For our study we build a multilayer perceptron with 2 hidden layers.

Example of ANN created on https://playground.tensorflow.org/

Our architecture this time will have 562 neurons in input layer, 64 neurons in 2 hidden layer and 6 in the output layer. We will also changing the activation function to ReLu and will be using adam optimizer. Our model will look like this :

summary()



Layer (type) Output Shape Param =================================================================

input_4 (InputLayer) (None, 562) 0 _________________________________________________________________ dense_9 (Dense) (None, 64) 36032 _________________________________________________________________

dense_10 (Dense) (None, 64) 4160 _________________________________________________________________

dense_11 (Dense) (None, 6) 390 =================================================================

Total params: 40,582

Trainable params: 40,582

Non-trainable params: 0

In details

Total params = 40.582 which is divided in the form of (562 x 64) + 64+ 4160+390. (562*64) for the fully connected structure because each of the 562 neurons in the input layer will be connected to two hidden Dense layers and the last hidden Layer will be connect to every neuron in the output layer .

Before training a model, we need to configure the learning process, which is done via the compile method.

Compile Method receives three arguments:

An optimizer. We use Adam an optimization algorithm that can be used instead of the classical stochastic gradient descent procedure to update network weights iterative based in training data.

A loss function. This is the objective that the model will try to minimize.

A list of metrics. For any classification problem you will want to set this to metrics=[‘accuracy’].

batch_size = 32

# is a hyperparameter that defines the number of samples to work through before updating the internal model parameters. nepochs = 25 #denotes number of times the learning algorithm will work through the training model.compile(optimizer='adam',loss='sparse_categorical_crossentropy',metrics=['accuracy']) history = model.fit( X_train, y_train, validation_split=0.2, batch_size=batch_size, epochs=epochs,

callbacks=[ tf.keras.callbacks.ModelCheckpoint('./model.h5',save_best_only=True),

tf.keras.callbacks.ReduceLROnPlateau( monitor='val_loss', factor=0.1, patience=1 )]) "Train on 5881 samples, validate on 1471 samples

Epoch 1/25

5881/5881 [==============================] - 1s 144us/step - loss: 0.6456 - acc: 0.7494 - val_loss: 0.3034 - val_acc: 0.9123

Epoch 2/25

5881/5881 [==============================] - 0s 55us/step - loss: 0.2599 - acc: 0.9039 - val_loss: 0.1926 - val_acc: 0.9273

Epoch 3/25

5881/5881 [==============================] - 0s 63us/step - loss: 0.1771 - acc: 0.9350 - val_loss: 0.1821 - val_acc: 0.9198

Epoch 4/25

5881/5881 [==============================] - 0s 58us/step - loss: 0.1404 - acc: 0.9493 - val_loss: 0.1481 - val_acc: 0.9368

Epoch 5/25

5881/5881 [==============================] - 0s 62us/step - loss: 0.1058 - acc: 0.9595 - val_loss: 0.1277 - val_acc: 0.9449

Epoch 6/25

5881/5881 [==============================] - 0s 70us/step - loss: 0.0924 - acc: 0.9663 - val_loss: 0.1552 - val_acc: 0.9470

Epoch 7/25

5881/5881 [==============================] - 0s 55us/step - loss: 0.0680 - acc: 0.9782 - val_loss: 0.1283 - val_acc: 0.9409

Epoch 8/25

5881/5881 [==============================] - 0s 54us/step - loss: 0.0656 - acc: 0.9786 - val_loss: 0.1304 - val_acc: 0.9477

Epoch 9/25

5881/5881 [==============================] - 0s 56us/step - loss: 0.0650 - acc: 0.9794 - val_loss: 0.1304 - val_acc: 0.9477

Epoch 10/25

5881/5881 [==============================] - 0s 52us/step - loss: 0.0650 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 11/25

5881/5881 [==============================] - 0s 52us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 12/25

5881/5881 [==============================] - 0s 55us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 13/25

5881/5881 [==============================] - 0s 55us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 14/25

5881/5881 [==============================] - 0s 50us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 15/25

5881/5881 [==============================] - 0s 52us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 16/25

5881/5881 [==============================] - 0s 50us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 17/25

5881/5881 [==============================] - 0s 50us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 18/25

5881/5881 [==============================] - 0s 53us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 19/25

5881/5881 [==============================] - 0s 78us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 20/25

5881/5881 [==============================] - 0s 65us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 21/25

5881/5881 [==============================] - 0s 74us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 22/25

5881/5881 [==============================] - 0s 63us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 23/25

5881/5881 [==============================] - 0s 53us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 24/25

5881/5881 [==============================] - 0s 60us/step - loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

Epoch 25/25 5881/5881 [================================] - 0s 54us / step loss: 0.0649 - acc: 0.9793 - val_loss: 0.1305 - val_acc: 0.9477

After 25 Epoch:

Training: Accuracy is 97% and the loss is 0.065

Validation: The accuracy is 95% and the loss is 0.13

Usually with each increasing epoch, the loss decreases and the accuracy increases. But with val_loss and val_acc, many cases can be possible:

The less the loss, the better the model (unless the model has over-adapted to the training data). The loss is calculated based on training and validation and its interaction is the model performance for these two sets. Unlike accuracy, the loss is not a percentage. It is a sum of the errors made for each example in the training or validation sets.

Results

After about 9 epochs we see that the model converges.

model.evaluate (X_test, y_test)

"execution_count": 125, "output":

2947/2947 [============================ ==] - 0 s 30us / step

[0.1902065952717727, 0.9300984051780131]

Test: The accuracy is 93% and the loss is 0.19

In deep learning we have many hyperparameters, so we can develop various models by changing the optimizer, the number of epochs, the activation function. By working on these hyperparameters we can get the best results especially for test data.

Reflections and next steps

In this article I have decided to use ANN with Tensorflow, a very useful tool that we can use for even more sophisticated deep learning models. The ANN assumes that all observations are independent and identically distributed, but the human movement you are doing right now depends on the movement you were doing 1 second ago.

From this observation the next step, to obtain better results, could be to use the recurrent neural network (RNN), in particular LSTM, is a class of artificial neural networks in which the connections between the nodes are a direct graph along a temporal sequence . This allows it to exhibit temporal dynamic behavior. Derived from feedforward neural networks, RNNs can use their internal state (memory) to process input sequences of variable length.

The classification of physical activities, using data from smartphones / smatwatch sensors, can be used by various mobile apps, to improve and monitor the physical activity of users so that they follow a healthy lifestyle also to create tailor-made programs for the user.

Immagine di Pixabay

References and sources of inspiration

https://www.kaggle.com/uciml/human-activity-recognition-with-smartphones/notebooks

https://archive.ics.uci.edu/ml/datasets/human+activity+recognition+using+smartphones

https://machinelearningmastery.com/how-to-model-human-activity-from-smartphone-data/

https://www.youtube.com/watch?v=hbdcJO9CEZs&feature=youtu.be

https://medium.com/@abhishekneema8/human-activity-recognition-1baf180f8eaa