# Co-Development of Custom-Made Sensor-based and Vision-based Human Activity Recognition

Jérôme Rocheteau

► To cite this version:

Jérôme Rocheteau. Co-Development of Custom-Made Sensor-based and Vision-based Human Activity Recognition. 14th International Conference on Model and Data Engineering (MEDI 2025), Ahmed Hassan; Ladjel Bellatreche; Ahmed Awad; Carlos Ordonez, Nov 2025, Le Caire, Egypt. hal-05250664

HAL Id: hal-05250664

https://cnrs.hal.science/hal-05250664v1

Submitted on 11 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Co-Development of Custom-Made Sensor-based and Vision-based Human Activity Recognition

Jérôme Rocheteau¹,²

¹ Icam Ouest, 35 avenue du champ de manœuvres, Carquefou, F-44470, France
jerome.rocheteau@icam.fr

² Nantes Université, CNRS, LS2N, UMR 6004, Nantes, F-44000, France
jerome.rocheteau@ls2n.fr

**Abstract.** Finely and precisely measuring human activities is key and fundamental in several domains such as health, sport, security, etc. Despite the ubiquity of sensors, cameras, and artificial intelligence based applications in our daily lives, it still remains difficult to benefit from join analysis of both sensor measurement and camera image processing. The field of Human Activity Recognition (HAR) is widely divided between sensor-based HAR and vision-based HAR, and there exists little contribution trying to link one to the other. Moreover, there also exists little contribution trying to provide tooling, tooled approaches, platforms or frameworks to HAR. Thus this paper aims at starting to bridge the gap between sensor-based and vision-based HAR. It also aims at filling the lack of extensible tools for custom-made HAR. Indeed, it provides a framework that makes sensor-based and vision-based HAR possible at the same time. It shows how to build custom-made datasets and models based on data collected from sensor measurements and from camera images through a study case of four daily-life activities with two motion activities and two motion-less ones.

**Keywords:** Human Activity Recognition, Inertial Measurement Unit, Time Series Database, Recurrent Neural Network, Sensor Fusion, Computer Vision

## 1 Introduction

The promise of big data and artificial intelligence about actionable insights, enabling organizations to make informed decisions, optimize processes, and even predict future trends, has invaded numerous domains and activities nowadays such as health, sport, security, etc. Currently, it still lacks of reliable and affordable solutions that can support the whole Human Activity Recognition (HAR) processing i.e. from dataset construction, model training and activity prediction as well as for sensor-based HAR as for vision-based HAR. For example, in sport, existing solutions like those of Kinexon Sports<fnref n="3" /> are available, reliable,

<footnote n="3">³ https://kinexon-sports.com.</footnote>
and state-of-the-art. In fact, the Kinexon Perform IMU solution based on In-
ertial Measurement Units (IMU) provides recognition of athlete activity. This
solution is mainly used to monitor external training load of athletes and can
be improved with the Kinexon Perorm LPS solution based on Ultra WideBand
devices that provides a Local Positioning System (LPS). Finally, the Kinexon
Complete Vision solution has recently been released in order to provide camera-
based analysis on specific sport activities (basketball shoot recognition) and it is
claimed to be seamlessly integrable with the previous solutions. However, these
solutions remain expensive and unaffordable for most sport organizations. More-
over, whereas analysis can be customized to a certain extent with the help of the
sport analysts and data scientists at Kinexon Sports, analysis can not be fully
custom-made as such solutions embed predefined models and datasets for HAR
that can not be customized.

This paper aims at developing an open-source, lower-cost, and state-of-the-
art framework, composed of hardware and software, that makes it possible to
build custom-made HAR datasets, to train HAR models from these custom-
made HAR datasets, and therefore to perform custom-made HAR predictions
as well as from IMU sensor signals as from camera image sequences. Software
components of this framework has been publicly released<fnref n="4" />. This paper provides
a proof-of-concept of such custom-made HAR based on IMU and CV. These
studies merely focus on four daily-life activities, two motion activities (walking
and jogging) and two motion-less ones (sitting and standing). The reminder
of this paper is structured as follows: Section 2 presents the related work about
sensor-based and vision-based HAR. Section 3 presents the study case of custom-
made HAR based on IMU. Section 4 presents the study case of custom-made
HAR based on CV. Section 5 provides feedback and discusses the relevance of
performing both approaches at the same time. Section 6 gives conclusions and
prospects of this work.

## 2 Related Work

HAR consists of automatically detecting which activity human beings are doing
among a given set of activities. These activities consist in general of activities
such as lying, standing, sitting, walking, walking upstairs, walking downstairs,
running, cycling, etc. (see. [5, Section 2.3], [14], and [15]). HAR is classically
divided into vision-based HAR (see [3]) and sensor-based HAR (see [2]). Vision-
based HAR corresponds to HAR whose input data are provided by cameras.
Sensor-based HAR corresponds to HAR whose input data are provided by sen-
sors, smartphone-embedded ones or wearable ones, etc. Such sensors are mainly
Inertial Measurement Units (IMUs) i.e. 3-axis accelerometers, 3-axis gyroscopes
and 3-axis magnetometers (see. [2, Figure 7]). In sensor-based HAR, input data
are mainly either those of accelerometers, or those of gyroscopes or both, i.e.
time-series of 3, 6 or 9 variables. Vision-based HAR is non-intrusive, i.e. it does
not require human beings or players to wear or to hold something dedicated

<footnote n="4">⁴ https://git.icam.fr/jerome.rocheteau/medi25</footnote>
to HAR whereas sensor-based HAR does. However, sensor-based HAR is more
precise and fine-grained than vision-based HAR and requires less computational
resources, which makes real-time processing possible (see [9, Table 10]). More-
over, despite their respective advantages and drawbacks, there do not exist, in
the scientific literature, many works that try to relate vision-based and sensor-
based HAR (see [9, Table 2]).

Although, HAR is split between vision-based HAR and sensor-based HAR, both data processing are very similar (see [9, Figure 1]). In fact, data processing for HAR consists in providing probabilities that each given activity is under going by a human being or a player from the input data; the activity with the best probability is mostly considered as the ongoing activity. As input data are time-series either of images or of multiple variables and as output can be seen as the most probable activity from input data, HAR then consists of time-series classification. In fact, almost all studies in HAR use Machine Learning (ML) or Deep Learning (DL) models. It has been shown in [2, Section 8] that classical ML classifiers such as Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), k-Nearest Neighbor (k-NN), Hidden Markov Model (HMM) perform well on small datasets or less-complex datasets whereas DL models such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), as well Long-Short Term Memory (LSTM) as Gated Recurrent Unit (GRU) perform well on large datasets or more-complex datasets. Most of the recent works make use of DL models composed of both CNN layers and RNN layers sequentially (see [16, Section 3] and [17, Section 2]) or hierarchically (see [7, Section 3]), with or without attention layer, and with or without bidirectional layer (see [8, Section 3.3]) (see [18, Section 4]). In fact, RNNs are known to successfully handle the feature extraction of temporal dependencies whereas CNNs are famous for successfully dealing with feature extraction of spatial dependencies like images (see. [13, Section 5.2.5]). Some hybrid models compound CNNs before RNNs (see. [10], [19, Section 4]), some other hybrid models compound RNNs before CNNs (see. [18]). Another advantage of DL models over ML models consists of the automated feature extraction: whereas ML models pre-process input data in order to extract their features, DL models directly process input data, embed feature extraction and learn corresponding parameters (see. [1, Table 1]). As [13, Section 5.2.5] noticed, CNNs are predominant in vision-based HAR as CNNs perfectly match with image classification, object detection and segmentation. In this case, such classifiers correspond to 2D CNNs. In order to cope with image sequences, classifiers correspond to 3D CNNs and the additional dimension corresponds to the time dimension. Nevertheless, CNNs are also useful for sensor-based HAR: sensor input data are merely times-series represented by 1D flatten matrices, and 1D convolution layers have proven their capability to extract temporal features almost as well as RNNs (see [11, Section 3.2] and [12, Table 5, Figure 5]). State-of-the-Art HAR models currently achieve 0.95 of average accuracy; the best HAR models achieve more than 0.99 accuracy (see [2, Table 8] and [1, Table 2]). It has been noticed in [1, Table 8] that classification performances (accuracy, precision, recall, sensitivity and specificity) of the DL
models are better than those of the ML ones, that classification performances of hybrid DL models composed of both CNNs and RNNs are better than those of DL models composed of either CNNs or RNNs, and that classification performances of bi-directional RNNs, either LSTM or GRU, are better than those of uni-directional RNNs. As well, attention layers enhances classification performances (see [8, Section 5]).

It has been highlighted that HAR main drawback consists of the lack of available public datasets. In [3, Section 4], only 6 public HAR datasets for sports are listed. In [2, Section 7, Table 6], only 15 public HAR datasets are listed. According to [2, Section 9], most of the studies (0.78) are based on experiments that are not reproducible as their datasets are not public. Whereas building datasets, i.e. labeling raw data, is known as cumbersome, tedious, time-consuming and expensive, most of the studies build their own dataset because either raw data or labels do not correspond to their own purpose. In fact, the different sets of labels of public sensor-based HAR datasets present a wide variability: from 2 to 33 labels, from coarse-grained label to fine-grained ones (see [2, Table 6]). Moreover, the different raw data of public sensor-based HAR datasets present a wide variability as well: from data of 1 single smartphone 3-axis IMU sensor to data of 23 body sensors, from 1 subject to 563 subjects, from 1 single sensor position up to 18+ positions. It is difficult to find a HAR dataset that fits perfectly with the goals of the different studies.

## 3 HAR based on IMU

This section explains how to perform reliable, affordable, and state-of-the-art HAR based on IMU sensors. Firstly, it presents the set of hardware and software components used to achieve such an objective. Secondly, it presents a study case, called SSWJ for sitting, standing, walking, and jogging, that aims at investigating HAR over 2 motion activities and 2 motionless ones. In this study case, experiments that lead to the construction of datasets, the training of models, and the evaluation of models are explained.

The framework corresponds to a consistent set of data, hardware and software components that are detailed according to the data flow: from the IMU sensors to the HAR prediction. The context of sensor-based HAR involves to use wireless wearable devices (the smaller, the better) such that they do disturb human activities the less. Such devices then require 1) a IMU module in order to deliver IMU measurements, 2) a wireless communication module in order to transmit these measurements and 3) a battery load module in order to manage a battery and its reloading. Two boards have been used to run the experiments: Adafruit Feather nRF52840 Sense boards and Arduino Nicla Sense ones. They both communicate according to the Bluetooth Low Energy (BLE) protocol. They are both programmed the same way: it merely consists of setting up the IMU sensors and the BLE communication protocol and endlessly testing if the device as a BLE peripheral is connected to a BLE central and if the latter has subscribed to the IMU measurements. In this case, it retrieves the measurements
from the IMU sensors, write these measurements to the BLE characteristics and
notify the BLE central. Such measurements comprise:

- the identifier number of the device,

- 3-axis acceleration values from the accelerometer,

- 3-axis angular velocity values from the gyroscope,

- 3-axis magnetic flux density values from the magnetometer

- the board temperature,

- the battery voltage.

The latter makes it possible to estimate the battery charging rate. Each device
consists of one of the previous boards equipped with a 150mAh battery and
inserted into a custom-made plastic case. Device dimensions are 48x34x10mm for
the Adafruit Feather nRF52840 Sense and 40x30x8 for the Arduino Nicola Sense.
Battery autonomy lasts 4 hours. Data frequency averages 60Hz for both devices.
Gateways merely consist 1) of connecting to the devices as BLE peripherals and
of subscribing to the advertised BLE characteristics when they have been asked
for and 2) of forwarding the IMU measurements to the platform according to the
TCP/IP MQTT protocol. That's why they are called gateways as they convert
BLE characteristics payloads received from the devices into MQTT messages
published to the platform, i.e. they do act as BLE to MQTT gateways.

As the same way as each gateway can manage several devices, the platform
can manage several gateways and therefore even more devices. The platform
aims at collecting, storing and visualizing IMU-based data received from devices
throughout gateways and it merely consists of computers connected to the same
network as that of gateways. It also aims at building datasets, training models,
and running predictions. The platform is split into 3 kinds of components: a
single broker, a single modeler and as many analyzers as devices. The broker
hosts the following 3 services and the analyzers host the last 2 ones:

- a Mosquitto service, a MQTT broker,

- a InfluxDB service, a timeseries-oriented database,

- a Telegraf service configured to bridge MQTT and InfluxDB.

The overall orchestration between devices, gateways and the platform is illus-
trated by the figure 1. There exist two data processing modes: the acquisition
mode for collecting data in order to build HAR datasets and train HAR models,
and the prediction mode for using a HAR model in order to recognize activities
from streaming IMU data. The devices retrieve IMU data from their sensors i.e.
mainly their accelerometer, gyroscope, and magnetometer. They write IMU mea-
surements to the corresponding BLE characteristic and notify their associated
BLE peripheral, i.e. a gateway. The latter forwards these measurements to the
platform. These IMU measurements are stored either by the broker component
in acquisition mode or by the analyzer ones. Measurements are timestamped
while storing them into the database. Both broker and analyzer databases are
timeseries-oriented databases as the latter make it possible to aggregate multi-
variate time series uniformly for a given time window period. Prediction mode
<figure><img src="image_2.png" /><figcaption>Fig. 1. IMU-oriented Data Processing within the Framework</figcaption></figure>
corresponds to a streaming online data processing whereas acquisition mode corresponds to a batch offline data processing. In prediction mode, every half second, the analyzer unit retrieves the last 2 seconds data from the database such that these 2 seconds data have been aggregated every 20ms. These data are fed into a HAR model that provides probabilities that these 2 seconds data match given and defined activities. The most probable activity for these 2 seconds is selected and stored by the analyzer unit that also notifies the broker as for the predicted activity. Preliminarily, the platform plays a key role in activating or deactivating devices as it tells gateways to enable or disable devices. Once gateways are instructed to enable a device, they try to connect to this device if the latter is detected, and they try to subscribe to the BLE characteristic that corresponds to the IMU measurements. In the same way, once gateways are instructed to disable a device, they unsubscribe this BLE characteristic before disconnecting the device. While the device is enabled, it continuously notify the gateway of the IMU measurements which forward them to the platform. At the very moment the platform enables a device, it also starts this device analysis process as explained above.

This framework makes sensor-based HAR possible, i.e. dataset construction, model training, and activity prediction. The first study case involves 4 daily life activities: sitting, standing, walking, and jogging. This corresponds to 2 motion activities and 2 motionless ones. Whereas measurements from IMU sensors are different from one activity to another, measurements from the 2 motionless activities seem the same. This study case shows that HAR models perform and accurately predict activities even those with very similar IMU measurements.

The first step consists 1) of collecting raw IMU data and activity labels easily and 2) of building corresponding datasets based on such raw data. Regarding data collection, 17 participants (10 males, 7 females) performed a series of 4 activities: sitting, standing, walking and jogging, during 3 recording sessions of 20 minutes. Each session consisted of 5 minutes per activity per participant. Each participant was equipped with wearable IMU devices that have been located in the back of the participants or around their waist. IMU measurements were continuously recorded achieving an average frequency of 59.57Hz for every recording seconds for all devices. The minimum frequency of 12Hz and the maximum frequency of 84Hz have been recorded for a device during one recording second. Each activity was manually labeled in real time to ensure accurate annotation. Both raw IMU features and activity labels were stored in the InfluxDB database. This process benefits from the InfluxDB query language that is capable of handling time-series such as windowing. It then makes it possible to build several datasets from a single data source with different temporal resolutions: a dataset with 1,589,405 records at 50Hz, i.e. a record every 20ms. Values of missing records are linearly interpolated; however, they are not preprocessed using a high-band or low-band filter.

The second step consists of training a model capable of recognizing various activities from IMU measurements. The training datasets were preprocessed in order to build temporal data windows of a given length of 2s and a given offset
of 0.5s. The number of epochs is defined to 100 with an early stopping procedure with a patience of 3 epochs in order to avoid overfitting. A state-of-the-art CNN-LSTM hybrid sequential model is defined as follows:

- 1 layer of Conv1D with a filter of size 12 and a kernel of size 1,

- 1 layer of Conv1D with a filter of size 12 and a kernel of size 1,

- 1 layer of MaxPooling1D with a pool size of 3,

- 1 layer of 120 LSTM cells followed by a dropout layer of 0.4,

- 1 layer of 60 LSTM cells followed by a dropout layer of 0.2,

- 1 flatten layer and 2 dense layers, respectively of size 40 and 4.

Such a model then consists of 10 layers and 110,124 parameters. It processes a 100x6 matrix input, i.e. the measurements of the accelerometer and the gyroscope aggregated every 20ms for a 0.2s duration, and provides a 1x4 output matrix, i.e. the probabilities of the 4 activities.

<table><caption>**Table 1.** Evaluation Metrics of the IMU-based SSWJ model</caption><thead><tr><th>metrics<br>activities</th><th>precision</th><th>recall</th><th>accuracy</th></tr></thead><tbody><tr><td>sitting</td><td>0.985</td><td>0.968</td><td>0.976</td></tr><tr><td>standing</td><td>0.969</td><td>0.985</td><td>0.977</td></tr><tr><td>walking</td><td>0.999</td><td>0.998</td><td>0.998</td></tr><tr><td>jogging</td><td>0.999</td><td>0.999</td><td>0.999</td></tr><tr><td>overall</td><td>0.988</td><td>0.988</td><td>0.988</td></tr></tbody></table>

Such a model performs well and achieves 98.79% accuracy as illustrated by the evaluation metrics of the table 1 based on the confusion matrix of the figure 2. Motion activities (walking and jogging) are well recognized from every other activities as their respective precisions and recalls are above 99%; whereas motionless activities (sitting and standing) are less well recognized one from the other. However, their precisions and recalls never drop below 96.80%. Moreover, the training progress over each epoch of the figure 2 indicates a very good generalizability, i.e. this model makes accurate predictions on unseen data.

## 4 HAR based on CV

This section explains how to perform reliable and state-of-the-art HAR based on Computer Vision. Firstly, it presents the set of hardware and software components used to realize the study case. Secondly, it presents the SSWJ study case. In the same way as IMU-based HAR, experiments that lead to the construction of datasets, the training of models, and the evaluation of models are explained.

The framework also corresponds to a consistent set of hardware and software components that merely consists of a camera controlled by a program running
<figure><img src="image_3.png" /><figcaption>Fig. 2. Training Progress and Confusion Matrix of the SSWJ model</figcaption></figure>

on a computer. Hardware components used to perform CV-based HAR are a Stereolabs's ZED X camera with its corresponding ZED Box micro-computer<fnref n="5" />. It benefits from the ZED SDK that provides object detection models. In fact, the Python program that constructs the CV-based HAR dataset consists of 1) predicting person detection and 2) applying the manually annotated labels to the corresponding bounding boxes. This program provides a COCO-compliant dataset (see [6]) for Yolo11 object detection models (see [4]). However, such components can be adapted to different cameras, computers, and object detection models once it is possible to perform person detection and easily extract bounding boxes of detected persons. The architecture of CV-oriented data processing is more simple than that of IMU-based one as illustrated by the figure 3. In fact, the device composed of the camera and its computer running either the dataset construction program or the activity prediction one provides image sensing and image analyzing (person detection or activity recognition). However, CV image processing has to deal with several possible person detections and therefore several possible activity predictions whereas IMU signal processing only deals with a single person and therefore a single activity prediction.

Experiments have been conducted with 10 participants (9 males, 1 female) that were doing each activity during 1 minute. The CV device (camera and computer) was moving around the participant during the recordings in order to vary the angle of capture. It then led to collect 11,063 images with a single label per image. The dataset was split with a 70/10/20 ratio: 7,745 training images, 1,107 validation images and 2,211 test images. Moreover, training images were augmented by horizontal flipping, grayscale, and Hue filtering in order to obtain object detection models more robust to different image colors and person orientations such that the training images count 30,921 images. Yolo11 model training corresponds to transfer learning from the Yolo11 medium object detection model to the SSWJ dataset. The model consists of 125 layers and 20,033,116

<footnote n="5">⁵ See Stereolabs's documentation for ZED X cameras, ZED Box and ZED SDK.</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 3. CV-oriented Data Processing within the Framework</figcaption></figure>
<table><caption>**Table 2.** Evaluation Metrics of the Yolo11 SSWJ model</caption><thead><tr><th>metrics<br>activities</th><th>precision</th><th>recall</th><th>mAP50</th><th>mAP50-95</th></tr></thead><tbody><tr><td>sitting</td><td>0.994</td><td>0.995</td><td>0.994</td><td>0.993</td></tr><tr><td>standing</td><td>1.000</td><td>1.000</td><td>0.995</td><td>0.993</td></tr><tr><td>walking</td><td>0.991</td><td>0.985</td><td>0.992</td><td>0.890</td></tr><tr><td>jogging</td><td>0.976</td><td>0.962</td><td>0.989</td><td>0.838</td></tr><tr><td>overall</td><td>0.990</td><td>0.985</td><td>0.993</td><td>0.929</td></tr></tbody></table>

parameters. It leads to the 4 evaluation metrics of table 2. Precision computes how many predictions are correct among all the predictions. Recall computes how many predictions are correct among all the detections that shall have been made. The metric mAP50 computes how many predictions have an overlap of 0.50 between the predicted bounding boxes and the true bounding boxes. As for the metric mAP50-95, it computes the average of predictions that have an overlap between the predicted bounding boxes and the true bounding boxes ranging from 0.50 to 0.95. Evaluation results of the CV-based SSWJ model are state-of-

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Training Progress of the Yolo11 SSWJ model</figcaption></figure>

the-art as well as those of the IMU-based one. In fact, average accuracy is up to 98.75%. Moreover, the training progress illustrated by the figure 4 indicates a good generalizability, i.e. the ability of the model to make accurate predictions on unseen data. However, the CV-based SSWJ model performs better on motionless activities (sitting and standing) than on motion activities (walking and jogging) whereas it is the opposite with the IMU-based SSWJ model. In fact,
the lowest precision, recall and mAP50-95 metrics correspond to the motion activities

# 5 Discussion

This section provides feedback and discussion about sensor-based HAR and vision-based processing. The latter are compared according to the process of modeling, i.e. dataset construction and model training, and the process of activity prediction. The benefit of conjointly developing sensor-based and vision-based HAR is also investigated.

As mentioned above, the evaluation results of both sensor-based and vision based HAR models nearly achieve 99% accuracy which is aligned with state-of-the-art results. However, dataset construction, model training, and activity recognition have different features. In fact, the sensor-based framework required to build sensor-based datasets is more sophisticated than the vision-based one. The IMU-oriented data processing of the figure 1 requires more equipment and a more complex data flow than the CV-oriented data processing of the figure 3. Indeed, a single camera controlled by a computer can be needed to build a CV dataset whereas it requires at least a board with sensors and a micro-controller, a gateway fetching sensor measurements to a database, and a computer collecting measurements of the different sensors from the different gateways. Moreover, a 50Hz data frequency is key to ensure the quality of sensor-based HAR datasets whereas a 15Hz data frequency is sufficient in the case of vision-based HAR datasets as this studies show. The complexity of the frameworks is the opposite as that the complexity of the models: the CV-based HAR model has 125 layers and approximately 20,000,000 parameters whereas that of the IMU-based one has only 10 layers and around 100,000 parameters. In addition, it needs 10-minutes only to train the IMU-based HAR model whereas it lasts 10-hours to train the CV-based model on the same computer equipped with a NVidia RTX3090 GPU. Nevertheless, both sensor-based and vision-based HAR require powerful computing capabilities which lead to the following challenge about the data flow architecture: either an edge-computing architecture that is more prone to execution time issues or a cloud-computing architecture that is more prone to communication delays issues.

Although the overall accuracies of the sensor-based and vision-based models range in the same values, a finer-grained analysis shows orthogonal results: In fact, the IMU-based SSWJ model performs better to detect motion activities than the motionless ones whereas the CV-based SSWJ model performs better to detect motionless activities than the motion ones. On the one hand, IMU signals for sitting and standing activities are very close as the 3-axis values of the accelerometer and the gyroscope range to zero because these are motionless activities whereas IMU signals for walking and jogging have different peak frequencies and amplitudes due to the rhythm and velocity of the foots on the ground between these activities. On the other hand, poses between sitting and standing look different on images whereas poses between walking and jogging
look a bit closer on images which can lead to more distinct feature extractions for motionless activities than for the motion ones. Moreover, IMU-based activity predictions have shown a poor accuracy and a huge variability of predictions when using the IMU-based SSWJ model in a real life application whereas CV-based activity predictions have proven to be accurate and stable in real life application. In fact, the IMU-based SSWJ model mainly detects the activity of standing either while sitting or while standing. In addition, the IMU-based SSWJ model varies a lot between 3 activities while walking: walking but also standing and sitting! Finally, it barely predicts the activity of jogging even while jogging. These outcomes have led us to design and implement once again the microcontroller of the IMU device in order to provide a higher data frequency. In contrast, CV-based activity predictions have been able to correctly predict the different activities on images extracted from videos of tests of up to 3 subjects and 3 activities.

This study therefore highlights the complementarity of both sensor-based and vision-based HAR and the interest of sensor fusion from both IMU and CV. Indeed, handling both IMU-based and CV-based activity predictions from dedicated SSWJ models at the same time have proven to be as accurate and stable as those of the best activity predictions either from one of the SSWJ models or the other in application limited to 1 subject only. In fact, the hardware and software component framework is yet not able to relate the IMU device measurements worn by a subject and its corresponding person detection on images. Then it would require another technology to enhance sensor fusion for real life application with several subjects such as a local positioning system or an identification system in order to relate IMU device measurements and CV object detections whereas it is straightforward to relate the latter in application with 1 subject.

# 6 Conclusion

This paper has provided a study case that bridges the gap between sensor-based and vision-based HAR. Firstly, it has shown how to build sensor-based HAR datasets and models from IMU sensor signals. It has been able to achieve affordable and reliable sensor-based HAR up to a 99% accuracy. Secondly, it has shown how to build vision-based HAR datasets and models at the same time as sensor-based datasets and models. And it has also been able to achieve affordable vision-based HAR up to a 99% accuracy. Moreover, this approach has been supported by an open-source framework that makes affordable HAR possible, although not as user-friendly as similar commercial tools. Recording both IMU signals and videos at the same time makes it possible to shorten the duration of dataset construction and model training as well as to strengthen either sensor-based or vision-based prediction consistency.

# References

1. Abbaspour, S., Fotouhi, F., Sedaghatbaf, A., Fotouhi, H., Vahabi, M., Linden, M.: A Comparative Analysis of Hybrid Deep Learning Models for Human Activity
Recognition. Sensors **20**(19) (2020)

2. Demrozi, F., Pravadelli, G., Bihorac, A., Rashidi, P.: Human Activity Recognition Using Inertial, Physiological and Environmental Sensors: A Comprehensive Survey. IEEE Access **8**, 210816–210836 (2020)

3. Host, K., Ivašić-Kos, M.: An overview of Human Action Recognition in sports based on Computer Vision. Heliyon **8**(6) (2022)

4. Jocher, G., Qiu, J.: Ultralytics YOLO11 (2024), https://github.com/ultralytics/ultralytics

5. Kwapisz, J.R., Weiss, G.M., Moore, S.: Activity recognition using cell phone ac-elerometers. SIGKDD Explor. **12**, 74–82 (2011)

6. Lin, T.Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C.L., Dollár, P.: Microsoft COCO: Common Objects in Context (2015)

7. Lyu, L., Huang, Y.: Sports activity (SA) recognition based on error correcting out-put codes (ECOC) and convolutional neural network (CNN). Heliyon **10**, e28258 (Mar 2024)

8. Mekruksavanich, S., Jitpattanakul, A.: RNN-based deep learning for physical ac-tivity recognition using smartwatch sensors: A case study of simple and complex activity recognition. Mathematical Biosciences and Engineering **19**(6), 5671–5698 (2022)

9. Minh Dang, L., Min, K., Wang, H., Jalil Piran, M., Hee Lee, C., Moon, H.: Sensor-based and vision-based human activity recognition: A comprehensive survey. Pattern Recognition **108**, 107561 (2020)

10. Ordóñez, F.J., Roggen, D.: Deep Convolutional and LSTM Recurrent Neural Net-works for Multimodal Wearable Activity Recognition. Sensors **16**(1) (2016)

11. Pajak, G., Krutz, P., Patalas-Maliszewska, J., Rehm, M., Pajak, I., Dix, M.: An ap-proach to sport activities recognition based on an inertial sensor and deep learning. Sensors and Actuators A: Physical **345**, 113773 (2022)

12. Patalas-Maliszewska, J., Pajak, I., Krutz, P., Pajak, G., Rehm, M., Schlegel, H., Dix, M.: Inertial Sensor-Based Sport Activity Advisory System Using Machine Learning Algorithms. Sensors **23**(3) (2023)

13. Ray, A., Kolekar, M.H., Balasubramanian, R., Hafiane, A.: Transfer learning en-hanced vision-based human activity recognition: A decade-long analysis. Interna-tional Journal of Information Management Data Insights **3**(1), 100142 (2023)

14. Reiss, A.: PAMAP2 Physical Activity Monitoring (2012)

15. Reyes-Ortiz, J., Anguita, D., Ghio, A., Oneto, L., Parra, X.: Human Activity Recognition Using Smartphones (2013)

16. Ryu, K., Kim, H., Lee, S.: A deep learning model based on sequential object feature accumulation for sport activity recognition. Multimedia Tools and Applications **82**, 1–20 (Mar 2023), link

17. Shuvo, M.M.H., Ahmed, N., Nouduri, K., Palaniappan, K.: A Hybrid Approach for Human Activity Recognition with Support Vector Machine and 1D Convolutional Neural Network. In: 49th IEEE Applied Imagery Pattern Recognition Workshop, AIPR 2020, Washington, DC, USA, October 13-15, 2020. IEEE (Oct 2020)

18. Xia, K., Huang, J., Wang, H.: LSTM-CNN Architecture for Human Activity Recog-nition. IEEE Access **8**, 56855–56866 (2020)

19. Zalluhoglu, C., Ikizler-Cinbis, N.: CollectiveSports: A Multi-task dataset for col-lective activity recognition. Image and Vision Computing p. 103870 (2020)