# Data Collection Automation in Machine Learning Process Using Robotic Manipulator

Piotr Reczek, Jakub Panczyk, Andrzej Wetula, Andrzej Mlyniec

► To cite this version:

Piotr Reczek, Jakub Panczyk, Andrzej Wetula, Andrzej Mlyniec. Data Collection Automation in Machine Learning Process Using Robotic Manipulator. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.505-514, 10.1007/978-3-031-34107-6_40. hal-05498073

HAL Id: hal-05498073

https://inria.hal.science/hal-05498073v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# Data Collection Automation in Machine Learning Process Using Robotic Manipulator

Piotr Reczek<fnref n="1" />✉, Jakub Panczyk<fnref n="3" />, Andrzej Wetula<fnref n="2" />✉, and Andrzej Młyniec<fnref n="1" />✉

**Abstract.** Collecting data for machine learning algorithms is an important part of the learning process. Moreover, human-machine interface systems operate with the user's physical movements, and recording these gestures is a standard method for creating datasets. However, this process is time-consuming and many people are required due to the risk of model overfitting. In this paper, we present a new method for automatizing data collection. The volunteers were replaced by a robotic arm with a mounted electric circuit to simulate the impedance of the human hand. Data recording and labeling were performed by a dedicated application that controlled the manipulator kinematics. The application generated randomized paths for the effector, with the purpose of collecting data similar to those collected from people. The gestures were recognized by a system based on capacitive sensors and a neural network algorithm executed on a microcontroller received data from the sensor signals. The system was taught with data collected from using the manipulator and tested with physical users. We describe the benefits of the proposed method, with the most significant advantage being that data collection was three times faster than using manual methods.

**Keywords:** Machine Learning · Automated Data Collection · Robotic Arm · 3D Gesture Recognition · Capacitive Gesture Recognition System · Human-Machine Interface · Embedded Systems

## 1 Introduction

Machine learning has gained great popularity in recent years, and the models are learned with a high number of accurate training data that must be labeled and recorded. In general, data collection is a crucial aspect of the final operation of a machine learning system. However, this step is considered a major bottleneck [1] in the machine-learning process. In addition, more complex models require more labeled data, resulting in increased effort. Moreover, some ML applications (such as gesture recognition) require collecting data from scratch due to a lack of available datasets, which significantly slows the learning process.

<footnote n="1">¹ Department of Robotics and Mechatronics, AGH University of Science and Technology, 30-059 Kraków, Poland
reczek@agh.edu.pl</footnote>

<footnote n="2">² Department of Measurement and Electronics, AGH University of Science and Technology, 30-059 Kraków, Poland</footnote>

<footnote n="3">³ Merit Poland Sp.z o. o., Podole 60, 30-394 Kraków, Poland</footnote>
Examples from the latest papers related to gesture recognition systems [2–6] indicate that the most popular approach for data collection is to obtain records from physical users (volunteers). However, apart from the effort required to manually collect and label a large amount of data, there are other inconveniences. For example, collecting data from people sometimes causes personal information issues. In [7], the authors created a dataset for gender and smile recognition. Although they had to obtain permission from all volunteers, they could withdraw their authorization within 50 days.

Working with large datasets is both demanding and important because more training data can have a positive impact on the accuracy of the trained models. According to the literature, augmentation is the most common approach for extending the collected dataset. Here, distorting parts of the data and adding them to the original dataset is one of the methods [8]. Another solution, in relation to image classification, was implemented in [9], where additional artificial data were generated by horizontally or vertically reflecting images from the base set. In [10], the authors proposed the generation of completely new data by creating captcha images using random generators. This concept produced training data from the scrap. Another example of generating synthetic data is by combining real pictures with 3D-rendered models as the source data [11]. Moreover, the research presented in [12] about eliminating data collection to learn the Wake Word recognition system was another example in which the researchers faced issues with collecting data. They used synthetic and ready recordings from the Internet and combined them with some noisy sounds, and then made some transformations. All these examples demonstrate that data-gathering issues are not only a matter of HMIs.

In this paper, we propose to use a robotic arm as the only source for generating a training dataset for gesture recognition. This is a novel approach to collecting data for machine learning algorithms for 3D gesture recognition. In this solution, trajectories of a robotic arm are generated that are then used as the input to the whole system. Here, the data goes through the same path as in the system target application, which eliminates any potential data deviations. As the created dataset was fully synthetic, it was interesting to determine how closely it approximated human inputs in the learning application. We verified our concept on an example capacitive gesture recognition system, where the hardware severely limited both the types of gesture and the detection range. This often results in hardware changes (tuning) in development. As a result, the learning process must be repeated many times, making automation more suitable for this application; hence, we termed this approach an automated method.

## 2 Realization of Gestures and Autmatised Dataset Generation

The proposed excitation system consisted of a 4-axis robotic arm equipped with an effector containing a CR circuit that simulated human hand impedance to ground (Fig. 1). The gesture recognition sensing panel was installed horizontally on the desk, above which the robotic arm performed gestures. This mounting position corresponded to the target application conditions, and the complete setup is presented in Fig. 2. We used the uArm device from UFactory.

We observed that maneuvering the effector over the electrodes resulted in the same signal output as in data collected from humans, which resulted in the idea of using only
<figure><img src="image_4.png" /><figcaption>Fig. 1. Circuit for simulating a human hand impedance to ground. C₁ = 20 nF, R₁ = 1500 Ω</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 2. Experimental setup. Note that a conductive pad is located under a whole setup.</figcaption></figure>

data collected with the manipulator in the learning process. In the automated method, instead of performing gestures and labeling them manually, this was accomplished using a PC application, which was responsible for generating paths, sending commands to the manipulator, gesture data collection, and labeling. The main doubt regarding this approach was the low data diversity compared to human-generated datasets, as people
naturally perform gestures differently, even when following the same precise instruc-
tions. Accordingly, any training data must imitate this aspect by having similar diversity.
This was achieved by introducing random variations in the effector trajectory and speed.
A list of coordinates in three axes represented the effector path positions, which were
sequentially achieved by the robotic arm with a specified speed. Each gesture had a
specified base path and speed, which was then modified before every move. We imple-
mented these variations by generating pseudorandom deviations for every coordinate
and speed in a range of -5% to + 5% of the base value. This range was experimentally
determined to cover practically the entire range of the system. Generation of the pseudo-
random numbers was achieved using the Mersenne Twister generator [13] with a period
of $2^{19937}$, guaranteeing that the values were not repeated.

3 Gesture Recognition System Used for Verification

To verify the concept, we used a prototype capacitive gesture recognition system
designed for the automotive market. This consisted of input circuitry formed from a
set of capacitive electrodes, a Microchip MGC3140 capacitive system controller, and an
STM32H745ZI Nucleo development board to execute the machine learning algorithm.
The predictions were then sent to a PC via UART. A block diagram of the complete
system is presented in Fig. 3.

<figure><img src="image_6.png" /><figcaption>Fig. 3. Block diagram of a gesture recognition system used in experiments.</figcaption></figure>

**3.1 Input Circuitry of the Gesture Recognition System**

The input circuitry of the system contained five electrodes that were located on a flat
surface, as presented in Fig. 4. It should be noted that despite using the terms “Top” and
“Bottom” for the electrodes, the actual device was located horizontally.

Two approaches are used for capacitive systems [14]: self and mutual capacitance.
In the self-capacitance approach, a change of capacitance between each electrode and
ground is used to detect the distance between an electrode and a disturbing object (hand).
By comparison, in a mutual capacitance approach, changes in capacitance between a
pair of electrodes in the presence of a disturbing object are detected. This approach
requires one transmitter (Tx) electrode and one (or more) receiver (Rx) electrodes. The
described solution uses a mixed approach, where boundary electrodes (left, right, top,
and bottom) are used as Rx, while the central electrode is used as both a Tx and Rx one.
Firstly, the central electrode operates as Tx and charged with an excitation voltage, while
the boundary electrodes were scanned sequentially. The central electrode worked as an
<figure><img src="image_7.png" /><figcaption>Fig. 4. Arrangement of capacitive electrodes of input circuitry.</figcaption></figure>

Rx for a self-capacitance approach. The MGC3410 chip repeated the cycle with a 5 ms period, which was the system sampling rate. The collected digital data corresponding to the capacitances were sent via the I2C protocol to the STM32H745ZI microcontroller. The microcontroller software then implemented simple level gating by only triggering data acquisition when any detected signal exceeded a predefined limit value, which indicated the presence of the hand in a detection zone. After each trigger event, a constant number of 240 data samples were collected, even when the hand was withdrawn from the detection area earlier.

## 3.2 Machine Learning Algorithm

The main part of the algorithm for gesture recognition was a 3-phase convolutional neural network, as displayed in Fig. 5. The bottom part of the picture presents the detailed structure of a single layer. Layers 2 and 3 were identical. Although Layer 1 had the same structure, its components (A and B) were of different sizes ($A = 1 \times 64 \times 32$, $B = 1 \times 64 \times 16$).

The model was trained with both datasets (one collected manually and one automatically). After the training process, we deployed an artificial neural network model for the microcontroller in three steps, as presented in Fig. 6. This starts by converting the.hdf5 model to the.tflite format using the TensorFlow framework. In Step 2, we transform the model to.c and.h files using the MXCUBE AI tool. In the final step, the built model and the embedded application source files result in a binary file in the.elf format. During program execution data from the sensor are normalized and then passed to methods generated by MXCUBE AI, that returns number assigned to predicted gesture.

Due to the lack of a criterium for feature vector creation, this could be considered a deep learning approach, although with a relatively short input vector. The output of the neural network is a five-element vector of probabilities, with each element corresponding to one of the following defined gestures: swipe left, swipe right, swipe up, swipe down,
<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 5. Neural network model used for gesture recognition.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 6. Embedded system deployment process.</figcaption></figure>

and single tap. The gestures were classified if a calculated probability exceeded 50%. This model is based on the inception architecture in [15].

## 4 Data Collection

The typical approach to data collection and labeling requires a group of volunteers who physically perform gestures, implying that more diverse data requires a larger group. Each recorded data sample must be labeled, with a certain probability of mislabeling that we estimated to be between 1% and 2%. In our research, this procedure was previously managed by the host engineer who supervised the learning process. This involved inviting dozens of volunteers, who were then asked to perform specified gestures, which was a tedious and time-consuming job. Usually, the volunteers were random people who did not know how the system worked, meaning they had to be instructed where and when to perform the gestures. Fortuitously, because the participants did not understand the technology, this provided the benefit of producing more real-life input data. However, considering all the required stages, this approach was painstaking. Data collection and labeling in gesture recognition systems is particularly time consuming, compared to
systems where collecting data does not require any human interaction. We attempted to improve the process by creating a dedicated user interface that eased the data labeling process, although the improvement was relatively insignificant. In contrast to the data collected from volunteers, the automatically collected data had no mislabeling errors. In this research, we developed an approach that shortened and improved the data collection and labeling processes. To collect data in an automated and organized way, a PC application was designed to control the robotic arm described in Sect. 2 and to receive, save, and label the data received from the STM32H745ZI. The input circuitry remained the same as in regular operation. The application executed on the STM32H745ZI detected hand presence and collected 240 samples for a single trigger. The collected data were then sent directly to the PC instead of being processed on the microcontroller. In total, 8780 gestures were collected in this way. To address any errors that occurred during collection, the dataset was then processed to find and remove outliers from each class using a cross-similarity matrix with dynamic time warping [16]. Table 1 presents a comparison of the data collection rates of the tested methods.

<table><caption>**Table 1.** Comparison data collection rate</caption><thead><tr><td>Manual</td><td>Automated</td></tr></thead><tbody><tr><td>123 gestures/h</td><td>427 gestures/h</td></tr></tbody></table>

Manual - The supervisor manually started and stopped recording the sequence of gestures. Volunteers were asked to perform all five specified gestures in sequence. Labeling was manual.

Automated – Program generates trajectories and controls robotic arm. Program automatically started and stopped the data recording. Labeling was conducted by the application.

## 5 Training

The collected data was then used to train the neural network. Usually, the data is split into three parts: training, validation, and testing. However, since we wanted to check how well the model was trained on automatically generated data, we split the model into two parts in the following proportions: 80% training and 20% validation. Test data was not required because we tested the model on the target system with gestures performed by volunteers. The training was conducted in Python script using the TensorFlow library.

## 6 Testing

The quality of the trained model was evaluated realistically by using new data collected through recording gestures made by humans. We invited 10 volunteers who were selected randomly from a group of over 200 colleagues. Each user was requested to perform 10 gestures of each type handled by the system (swipe left, swipe right, swipe up, swipe down, and tap). The values in Table 2 correspond to the number of these gestures that were correctly performed and recognized.
<table><caption>**Table 2.** Summary of the verification results (percentage of properly recognized gestures)</caption><thead><tr><th></th><th>swipe left</th><th>swipe right</th><th>swipe up</th><th>swipe down</th><th>tap</th><th>overall</th></tr></thead><tbody><tr><td>user1</td><td>100</td><td>100</td><td>80</td><td>100</td><td>100</td><td>96</td></tr><tr><td>user2</td><td>90</td><td>80</td><td>90</td><td>90</td><td>90</td><td>88</td></tr><tr><td>user3</td><td>90</td><td>100</td><td>50</td><td>100</td><td>100</td><td>88</td></tr><tr><td>user4</td><td>100</td><td>90</td><td>70</td><td>70</td><td>80</td><td>82</td></tr><tr><td>user5</td><td>100</td><td>100</td><td>40</td><td>50</td><td>100</td><td>78</td></tr><tr><td>user6</td><td>100</td><td>80</td><td>100</td><td>60</td><td>50</td><td>78</td></tr><tr><td>user7</td><td>100</td><td>90</td><td>60</td><td>40</td><td>70</td><td>72</td></tr><tr><td>user8</td><td>90</td><td>90</td><td>0</td><td>80</td><td>100</td><td>72</td></tr><tr><td>user9</td><td>90</td><td>80</td><td>50</td><td>100</td><td>20</td><td>68</td></tr><tr><td>user10</td><td>100</td><td>100</td><td>60</td><td>70</td><td>70</td><td>80</td></tr><tr><td>ALL</td><td>96</td><td>91</td><td>60</td><td>76</td><td>78</td><td>80.2</td></tr></tbody></table>

As depicted in Table 2, the overall accuracy was 80.2%, which is a promising result for using this method. The worst results were obtained for the swipe-up gesture. This was because the flat location of the sensor caused the swipe-up gesture to be performed away from the user, which was the least comfortable movement. It was also observed that after performing the swipe-up gesture (unlike the other gestures), the users behaved in two different ways: some kept their hand over the system, while others took it away from the system immediately. This behavior resulted in different signals at the end of the acquisition window. In automated learning, an effector was always stopped above an electrode after finishing this gesture. The gestures were detected correctly for users who kept their hands in place. For users who removed their hands immediately, the gestures were mislabeled. This result can inform the next stage of automated testing, where a robotic arm could be programmed to perform both types of swipe-up gestures.

# 7 Summary and Discussion

The proposed solution was deemed promising for teaching a gesture detection system. During the development of such systems, frequent changes require the reacquisition of data, which could prove troublesome with human volunteers. One great advantage of an automated gesture system is speed, due to there being no need to manually tag gestures. For comparison, taking 857 gestures with humans took 6.45 h, while taking 1173 gestures with an automated system took 2.45 h, indicating that the automated system operated approximately three times faster than the manual method. Additionally, the automated system could be left to operate overnight, utilizing the available time more effectively. We analyzed the collected dataset using both methods and concluded that the automatically collected dataset did not contain any mislabeled samples. In comparison, the manual approach always produced at least 1%–2% of data that were mislabeled, which is a natural consequence of simple human mistakes during the process.
Although the calculated precision of verification (80.2% on average) was not very high, there are many areas that could be improved. For example, an outlying “up” gesture was only performed by the automated system in one way, whereas the volunteers performed it in two ways. This result indicated that the automated system could not fully replace human testing. Accordingly, modifying the automated gesture set to mimic human behavior more accurately (and obtain better efficiency), would be a further research topic on the described system. In addition, the system could be expanded by recognizing more complex types of gestures. The idea of using data set collected both manually and automatically could be a good compromise and might eliminate the disadvantages of both approaches. This will be tested in our future development of 3D gesture recognition systems. It is also a wide research area for other machine-learning applications. Finally, a comparison of the automated and manual methods of data collection is presented in Table 3.

<table><caption>**Table 3.** Comparison between data collection methods</caption><thead><tr><th></th><th>Manual</th><th>Automated</th></tr></thead><tbody><tr><td>Time</td><td>Time-consuming</td><td>Shortens time</td></tr><tr><td>Supervisor engagement</td><td>Supervised continuously</td><td>Supervisor must only start the process</td></tr><tr><td>Dataset similarity to target application</td><td>Very high</td><td>High</td></tr><tr><td>Environment preparation effort</td><td>Moderate</td><td>High</td></tr><tr><td>Number of volunteers</td><td>High</td><td>None</td></tr><tr><td>Dataset diversity</td><td>Depends on volunteers</td><td>Depends on randomization</td></tr></tbody></table>

Two most important advantages of Automated method are shorter time and significantly smaller human effort. Lower data diversity, as it was proven, does not disqualify this method. It can be compensated by appropriate manipulator path randomization. Similarity of collected data are slightly lower compared to Manual method, but differences did not impact on gesture prediction accuracy. Higher environment preparation time will save more and more effort with every performed training loop. Automated approach make the development team completely independent from volunteers availability.

## References

1. Roh, Y., Heo, G., Whang, S.E.: A survey on data collection for machine learning: a big data - AI integration perspective. IEEE Trans. Knowl. Data Eng. 33, 1328–1347 (2021). https://doi.org/10.1109/TKDE.2019.2946162

2. Wang, Y., Jiang, H.: Digital gesture recognition based on deep capsule network. In: 2022 IEEE 2nd International Conference on Electronic Technology, Communication and Information (ICETCI), Changchun, China, pp. 1076–1079 (2022). https://doi.org/10.1109/ICETCI55101.2022.9832206
3. Zhang, Z., Wu, B., Jiang, Y.: Gesture recognition system based on improved YOLO v3. In: 7th International Conference on Intelligent Computing and Signal Processing (ICSP), Xi’an, China, pp. 1540–1543 (2022). https://doi.org/10.1109/ICSP54964.2022.9778394

4. Wang, P., Jiang, R., Liu, C.: Amaging: acoustic hand imaging for self-adaptive gesture recognition. In: IEEE INFOCOM 2022 - IEEE Conference on Computer Communications, London, United Kingdom, pp. 80–89 (2022). https://doi.org/10.1109/INFOCOM48880.2022.9796906

5. Qi, R., Zhang, Q.Z., Jing X.: Towards device-free cross-scene gesture recognition from limited samples in integrated sensing and communication. In: 2022 IEEE Wireless Communications and Networking Conference (WCNC), Austin, TX, USA, 2022, pp. 195-198 (2022). https://doi.org/10.1109/WCNC51071.2022.9771992

6. Pan, T.-Y., Tsai, W.-L., Chang, C.-Y., Yeh, C.-W., Hu, M.-C.: A hierarchical hand gesture recognition framework for sports referee training-based EMG and accelerometer sensors. IEEE Trans. Cybern. 52(5), 3172–3183 (2022). https://doi.org/10.1109/TCYB.2020.3007173

7. Aoto, S.: Collection of 2429 constrained headshots of 277 volunteers for deep learning. Sci. Rep. 12(1), 12 (2022). https://doi.org/10.1038/s41598-022-07560-2

8. Simard, P.Y., Steinkraus, D., Platt, J.C.: Best practices for convolutional neural networks applied to visual document analysis. In: Seventh International Conference on Document Analysis and Recognition. Proceedings, Edinburgh, UK, 2003, pp. 958–963 (2003). https://doi.org/10.1109/ICDAR.2003.1227801

9. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386

10. Le, T.A., Baydin, A.G., Zinkov, R., Wood, F.: Using synthetic data to train neural networks is model-based reasoning. In: Proceedings of the International Joint Conference on Neural Networks, vol. 2017-May, pp. 3514–3521 (2017). https://doi.org/10.48550/arXiv.1703.00868

11. Josifovski, J., Kerzel, M., Pregizer, C., Posniak, L., Wermter, S.: Object detection and pose estimation based on convolutional neural networks trained with synthetic data. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, pp. 6269–6276 (2018). https://doi.org/10.1109/IROS.2018.8594379

12. Ramanan, B., Drabeck, L., Woo, T., Cauble, T., Rana, A.: Eliminating data collection bottleneck for wake word engine training using found and synthetic data. In: 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, 2019, pp. 2447–2456 (2019). https://doi.org/10.1109/BigData47090.2019.9006601

13. Matsumoto, M., Nishimura, T.: Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans. Model. Comput. Simul. 8(1), 3–30 (1998). https://doi.org/10.1145/272991.272995

14. Reczek, P., Wetula, A., Młyniec, A.: Review and perspective on 3D gesture recognition technologies for automotive industry. Int. J. Autom. Eng. 12(4), 150–154 (2021). https://doi.org/10.20485/jsaeijae.12.4_150

15. RubinBose, S., SathieshKumar, V.: Efficient inception V2 based deep convolutional neural network for real-time hand action recognition. IET Image Process. 14(4), 688–696 (2020). https://doi.org/10.1049/iet-ipr.2019.0985

16. Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. Speech Signal Process. 26(1), 43–49 (1978). https://doi.org/10.1109/TASSP.1978.1163055