# Computer Vision Aided Machine Learning Framework for Detection and Analysis of Arm Flapping Stereotypic Behavior Exhibited by the Autistic Child

Umamaheswara Reddy Dundi, V. Kanaparthi, Ramakrishna Bandaru, Ganapathy Sankar Umaiorubagam

► To cite this version:

Umamaheswara Reddy Dundi, V. Kanaparthi, Ramakrishna Bandaru, Ganapathy Sankar Umaiorubagam. Computer Vision Aided Machine Learning Framework for Detection and Analysis of Arm Flapping Stereotypic Behavior Exhibited by the Autistic Child. 6th International Conference on Computational Intelligence in Data Science (ICCIDS), Feb 2023, Chennai, India. pp.203-217, 10.1007/978-3-031-38296-3_16. hal-05466045

HAL Id: hal-05466045

https://inria.hal.science/hal-05466045v1

Submitted on 19 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

Computer Vision Aided Machine Learning
Framework for Detection and Analysis of Arm
Flapping Stereotypic Behavior Exhibited
by the Autistic Child

Umamaheswara Reddy Dundi<fnref n="1" />, V. Phani Kumar Kanaparthi<fnref n="1" />(✉), Ramakrishna Bandaru<fnref n="1" />, and Ganapathy Sankar Umaiorubagam<fnref n="2" />

**Abstract.** Stereotypic movements such as arm flapping, spinning, and body rocking are found to be unique characteristics in the majority of autistic children. Detection and analysis of the above said stereotypic movements at an early stage could potentially help in treating autistic children before they develop severe and socially unacceptable behaviors. Taking advantage of computer vision and media pipe machine learning framework, a threshold-based data peak filtering algorithm for detection of one of the stereotypic behavior, i.e., arm flapping is proposed in this study. Flapping behavior of autistic children, such as Arm flapping intensity and its frequency, is extracted to assess the severity of autism. This crucial information helps the therapist to analyse and estimate the variations in a child's stereotypic behavioral pattern for Autism Spectrum Disorder (ASD) therapy.

**Keywords:** Autism · stereotypical movements · computer vision · mediapipe · arm flapping

# 1 Introduction

Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by a group of neurological abnormalities wherein multiple areas of the brain are undeveloped [1]. ASD can be identified usually by stereotypical behaviors like arm flapping, ticking off fingers and toes, and sudden outbursts, mostly when introduced to new and unfamiliar environments [2]. These stereotypic moments could be attributed to sensory overload in the neuro-sensors region of the brain, which gets overstimulated, resulting in discomfort and causing abrupt behaviour in the child. So, the child exhibits stereotypical behaviors to overcome the discomfort and adjust to the environment [3, 4]. Such types of abrupt

<footnote n="1">¹ Diagnostics, Therapeutics and Assistive Devices Laboratory, Department of Electronics and Communication Engineering, Faculty of Engineering and Technology, SRM Institute of Science and Technology, Kattankulathur 603203, Tamilnadu, India
phanikuk@srmist.edu.in</footnote>

<footnote n="2">² SRM College of Occupational Therapy, SRM Institute of Science and Technology, Kattankulathur 603203, Tamilnadu, India</footnote>
behaviors that occur at a very high rate may harm the child. Typically, an occupational therapist aims to reduce the child's stereotypic behavior with conventional methods, which involve careful examination of lengthy video recordings of the child's interactions and manually evaluate occurrences of change in behaviour [5].

Practically the data collection is time-consuming to achieve accurate results in conventional methods and also requires many experimenters to supervise [6]. In order to reduce the latency in obtaining accurate results, many automatic detection techniques are proposed in the literature for the detection of stereotypic moments in autistic patients. A typical method to detect stereotypical behaviors involves using accelerometers placed on the wrists of the children to track and collect data [7–9]. Westeyn, Tracy, et al. [7] proposed a stimming activity sensing system using on-body wireless accelerometers and Hidden Markov Models (HMMs); the system achieved 68.57% accuracy in continuous recognition tests. In their study, Min, Cheol-Hong, and Ahmed H. Tewfik placed accelerometer-based wearable sensors at various body locations to detect self-injurious and self-stimulatory behaviors [8]. Goodwin, Matthew S., et al. [9] achieved 90% accuracy in identifying stereotypical motor movements using a 3-axis wireless accelerometer and pattern recognition algorithms. The findings using an accelerometer sensor are promising, but the sensor must be worn on the wrists with rubber hand bands and at various body locations, which the child with ASD may not accept and lead to a misdiagnosis in some cases. Usage of the Kinect sensor in the system is another potential method to collect the stereotypic moments’ data [10–14]. While they are quite efficient, they are expensive; therefore, they cannot be widely used to monitor ASD patients in clinics and for experiments.

It is observed in the literature that the detection of various types of stereotypic behavior exhibited by the autistic child is given more priority rather than analysis and estimation of the variation in particular behavioral patterns in terms of its vital parameters such as intensity and frequency [15–17]. Here intensity indicates the total number of flaps done by a child at a specific time, and frequency indicates how many times the child exhibits similar behavior. This vital information could help in understanding the severity levels of autism and offer better therapy [18, 19].

In this paper, a computer vision based landmark extraction and analysing technique is proposed and the same is shown in Fig. 1. A threshold-based data-peak filtering algorithm [20–23] is developed to estimate the arm flap count and time span of behavior exhibited by the child. Further, the arm-flapping detection methods and algorithm development are explained in Sect. 2, while Sect. 3 shows the results drawn from the implemented algorithm when an arm-flapping behavior video taken from SSBD and a video taken by mimicking an autistic kid are given as input.

# 2 Methods

## 2.1 Collection of Stereotypic Behavior Videos

Children’s videos related to self-stimulatory behavior were downloaded from the internet using the Hyperlinks provided in the open-source Self-Stimulatory Behavior Dataset (SSBD) [24, 25]. The SSBD dataset comprises the Hyperlinks of 25 children’s arms flapping videos. The video duration and behavioral time instant, i.e., start and end times
<figure><img src="image_4.png" /><figcaption>Fig. 1. Proposed flow for detection and analysis of arm flapping stereotypic behavior</figcaption></figure>

at which the behavior is exhibited, are also included in the dataset. The frame rate of videos is 24 frames per second (FPS). Out of 25 videos provided in SSBD, 13 videos with high resolution and lightning conditions are only considered in this work for detection and analysis.

## 2.2 Arm Landmark Co-ordinates Extraction

Mediapipe, a Google open-source machine learning framework, is used to extract the landmarks of child's arms. The mediapipe pose model provides 33 landmarks, as shown in Fig. 2. Out of 33, landmarks numbered 15, 17, 21, 19 can be used to extract left-hand co-ordinates (x, y); similarly, landmarks numbered 16, 18, 20, 22 can be used to extract right-hand co-ordinates. To detect the arm flapping behavior out of four visible landmarks (wrist, pinky, index, and thumb) of arm, wrist is taken as the reference landmark. To extract the wrist landmark co-ordinates from videos, all the downloaded videos are converted to frames using video to frames converting program written in python. The converted frames of each video are then given as input to the media pipe pose model to detect and extract the co-ordinates (x, y) of the wrist landmark on both arms. The extracted landmarks co-ordinates are then auto-saved in Google spreadsheets using the implemented python program. The x and y co-ordinates represent how far apart the landmark is on the horizontal and vertical axis, respectively. Whenever the Mediapipe model fails to identify a landmark, then the co-ordinates for that particular landmark will be set to 0.
<figure><img src="image_5.png" /><figcaption>Fig. 2. Mediapipe pose model landmarks</figcaption></figure>

## 2.3 Manual Observation of Flap Count and Time Span of Arm Flapping from Extracted Landmarks

Figure 3 shows the left wrist's horizontal movement (distance) to the frame number of video1, from which we can visually observe the pattern of stereotypic behavior between frames 450 to 565. The time duration between the frames can be calculated by dividing the frame difference (115) by FPS (24). By counting the peaks, we can say that the hand flapping behavior is exhibited 19 times in 4.8 s. Similarly, Fig. 4 shows the right wrist horizontal movement, and the stereotypic behavior is observed between frames 450 to 570, from which behavior is exhibited 19 times in 5 s, so the intensity of behaviour is considered as 19 flaps in 5 s. Figures 5 and 6 shows landmark vertical movement of the

<figure><img src="image_6.png" /><figcaption>Fig. 3. Left wrist landmark horizontal movement and observed stereotypic behavior</figcaption></figure>
left and right wrist where no pattern of stereotypic behavior is observed. From this, it is clear that the arm flapping stereotypic behavior can be detected by visualization if there is a continuous rise and fall of data points in the stipulated period of time and the count of rise and falls in a particular time gives flap count. Even though this is one of the methods to analyse the behavior, this manual method of analysis is difficult, as it needs careful observation, and many experimenters have to supervise the analysis to achieve accurate results.

<figure><img src="image_7.png" /><figcaption>Fig. 4. Right wrist landmark horizontal movement and observed stereotypic behavior</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 5. Left wrist landmark vertical movement</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>Fig. 6. Right wrist landmark vertical movement</figcaption></figure>

## 2.4 Threshold-Based Data Peak Filtering Algorithm for Detection and Analysis of Arm Flapping

The proposed threshold-based algorithm implements data peak filtering to detect and analyse arm-flapping behavior in terms of intensity and frequency. The peak prominence is taken as a thresholding feature to filter out the smaller peaks, there after, the algorithm find consecutive nearby peaks with less peak-peak distance.

### Peak Prominence and Peak-Peak Distance

As shown in Fig. 7 The peak prominence is defined as the distance between a peak and its lowest contour line and peak-peak distance is defined as the distance between two consecutive peaks.

### Algorithm

The proposed algorithm is explained in the sequence of steps given below.

Step 1: Extract the co-ordinates of the left and right wrist from the video frames

Step 2: Auto save the extracted co-ordinates for analysis.

Step 3: Find out peaks from the extracted left wrist co-ordinates.

Step 4: Remove smaller peaks by setting minimum peak prominence threshold

Step 5: Remove peaks whose horizontal distance to the consecutive peak is high in terms of frame number

Step 6: Divide the remaining peaks to groups such that each group contains nearby consecutive peaks.
<figure><img src="image_10.png" /><figcaption>Fig. 7. Signal showing peak prominence and peak-peak distance</figcaption></figure>

Step 7: Repeat step3 to step6 for right wrist

Step 8: Compare the groups obtained for both left and right wrists to find matched groups.

Step 9: From matched groups, consider the group with a maximum count and peak count greater than five as stereotypic behavior.

Step 10: Count and save the number of peaks in behavior group.

Step 11: Obtained count gives the intensity of the behavior.

Step 12: The number of behavior groups gives the frequency of behavior.

## 2.5 Determining the Peak Prominence Threshold

The section of videos containing only arm-flapping behavior are extracted from the downloaded videos. From the extracted videos, the average of all peak prominence values is taken as reference to set peak prominence threshold. From this, the peak prominence threshold value is determined to be 25 and is fixed for analysing arm-flapping behavior.

# 3 Results

The algorithm's accuracy in finding arm flapping stereotypic behavior intensity and frequency is tested in two ways (i) by giving the videos downloaded using the SSBD dataset as input to the algorithm and (ii) by giving a video taken mimicking an autistic child as input.

## 3.1 Experimental Test1

The results drawn from the algorithm when video1 is given as input are shown below, where Figs. 8 and 9 show all the data peaks of left and right wrist landmark coordinates before filtering. Figures 10 and 11 shows the data peaks of left and right wrist after filtering with a minimum peak prominence value of 25. Figures 12 and 13 shows the groups of peaks obtained after removing far-away peaks from the data. Where the three groups represented in Fig. 12 relates to left wrist and seven groups represented in Fig. 13 relates to right wrist. The algorithm compared both left wrist and right wrist groups and considered group1 with peaks more than 5 as a stereotypic behavior. The arm flap count
and duration of behavior given by the algorithm is shown in Fig. 14. Similarly, the arm flap count and duration of behavior for remaining videos, obtained from conventional observational method (where two independent observers analyzed the intensity and frequency of the behavior with careful examination of video recordings) is compared with the proposed algorithm and tabulated in Table 1.

<figure><img src="image_11.png" /><figcaption>Fig. 8. Left wrist x-coordinate data peaks of video1 before filtering</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 9. Right wrist x-coordinate data peaks of video1 before filtering</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>**Fig. 10.** Left wrist x-coordinate data peaks of video1 after filtering with minimum prominence value</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>**Fig. 11.** Right wrist x-coordinate data peaks of video1 after filtering with minimum prominence value</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>Fig. 12. Left wrist data peak groups of video1 after filtering</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 13. Right wrist data peak groups of video1 after filtering</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Fig. 14. Detected flapping behavior of video1</figcaption></figure>

## 3.2 Experimental Test2

A video is taken by mimicking the arm flapping stereotypic behavior of autistic child in the laboratory, and the results drawn when the video is given as input to the algorithm are shown below. Figures 15 and 16 shows the data peaks of left and right wrist after filtering with minimum peak prominence value. Figures 17 and 18 shows the groups of peaks obtained after removing far away peaks from the data. The algorithm successfully detected the arm flapping behavior, flap count, and duration, as shown in Fig. 19.

The results obtained in experimental tests 1, 2 shows that the proposed algorithm can detect the child's stereotypic behavior and also estimated the flap count and duration of behavior. So, the proposed method can be used as an alternative to conventional methods
<table><caption>**Table 1.** Flap count and duration of arm flapping stereotypic behavior.</caption><thead><tr><th>Video Number (SSBD)</th><th>Flap Count (Conventional observational method)</th><th>Flap Duration (secs) (Conventional observational method)</th><th>Flap Count (Proposed Algorithm)</th><th>Flap Duration (secs) (Proposed Algorithm)</th></tr></thead><tbody><tr><td>5</td><td>29</td><td>11</td><td>27</td><td>10.6</td></tr><tr><td>8</td><td>15</td><td>4</td><td>15</td><td>3.8</td></tr><tr><td>11</td><td>27</td><td>7</td><td>26</td><td>6.7</td></tr><tr><td>12</td><td>32</td><td>22</td><td>30</td><td>21.6</td></tr><tr><td>13</td><td>23</td><td>9</td><td>23</td><td>8.5</td></tr><tr><td>14</td><td>18</td><td>4</td><td>18</td><td>4</td></tr><tr><td>15</td><td>19</td><td>5.8</td><td>18</td><td>5.6</td></tr><tr><td>16</td><td>22</td><td>9</td><td>20</td><td>8.6</td></tr><tr><td>18</td><td>17</td><td>4.8</td><td>15</td><td>4.6</td></tr><tr><td>19</td><td>23</td><td>15</td><td>20</td><td>14.6</td></tr><tr><td>21</td><td>15</td><td>4</td><td>15</td><td>3.8</td></tr><tr><td>22</td><td>16</td><td>4</td><td>14</td><td>3.9</td></tr></tbody></table>

<figure><img src="image_18.png" /><figcaption>**Fig. 15.** Left wrist x-coordinate data peaks of the mimicked video after filtering with minimum prominence value</figcaption></figure>

of stereotypic movement detection and the comparative study with some existing studies is shown in Table 2.
<figure><img src="image_19.png" /><figcaption>**Fig. 16.** Right wrist x-coordinate data peaks of the mimicked video after filtering with minimum prominence value</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>**Fig. 17.** Left wrist data peak groups of the mimicked video after filtering</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>**Fig. 18.** Right wrist data peak groups of the mimicked video after filtering</figcaption></figure>
<figure><img src="image_22.png" /><figcaption>**Fig. 19.** Detected flapping behavior of mimicked video</figcaption></figure>

<table><caption>**Table 2.** Comparison with existing studies.</caption><thead><tr><td>Author</td><td>Algorithm</td><td>Stereotypic Action recognized</td><td>Analysis on Intensity and frequency</td><td>Recognition rate</td></tr></thead><tbody><tr><td>Gonçalves, Nuno, et al. 2012 [10]</td><td>DTW</td><td>Hand flapping</td><td>no</td><td>51%</td></tr><tr><td>Lakkapragada, Anish, et al. 2021 [26]</td><td>LSTM</td><td>Hand flapping</td><td>no</td><td>72%</td></tr><tr><td>Proposed</td><td>Data peak filtering algorithm</td><td>Hand flapping</td><td>yes</td><td>95%</td></tr></tbody></table>

## 4 Conclusion and Future Scope

Detection and analysis of stereotypic movements could aid the therapist in better provision of treatment. The proposed computer vision-based technique can detect arm-flapping stereotyped movements of the autistic child in terms of intensity (number of flaps at a particular time) and frequency (number of times similar behavior is repeated). The results obtained from the experimental tests showed that the proposed method helps therapist in precise analysis of arm-flapping behavior. In future, an user interface as a assistive tool to therapist can be designed including adaptive thresholding features to detect other stereotypic movements such as head banging, body rocking etc....

## References

1. Lord, C., Elsabbagh, M., Baird, G., Veenstra-Vanderweele, J.: Autism spectrum disorder. Lancet 392(10146), 508–520 (2018)

2. Boyd, B.A., McDonough, S.G., Bodfish, J.W.: Evidence-based behavioral interventions for repetitive behaviors in autism. J. Autism Dev. Disord. 42(6), 1236–1248 (2018)

3. Draaisma, D.: Stereotypes of autism. Philos. Trans. R. Soc. B: Biol. Sci. 364(1522), 1475–1480 (2009)

4. Lam, K.S., Aman, M.G.: The Repetitive Behavior Scale-Revised: independent validation in individuals with autism spectrum disorders. J. Autism Dev. Disord. 37(5), 855–866 (2007)
5. McLaughlin, A., Fleury, V.P.: Flapping, spinning, rocking, and other repetitive behaviors: intervening with young children who engage in stereotypy. Young Except. Child. **23**(2), 63–75 (2020)

6. Noris, B.: Machine vision-based analysis of gaze and visual context: an application to visual behavior of children with autism spectrum disorders. Doctoral dissertation Verlag nicht ermittelbar (2011)

7. Westeyn, T., Vadas, K., Bian, X., Starner, T., Abowd, G.D.: Recognizing mimicked autistic self-stimulatory behaviors using HMMs. In: Ninth IEEE International Symposium on Wearable Computers, ISWC 2005, pp. 164–167 (2005)

8. Min, C.H., Tewfik, A.H.: Automatic characterization and detection of behavioral patterns using linear predictive coding of accelerometer sensor data. In: Annual International Conference of the IEEE Engineering in Medicine and Biology, pp. 220–223 (2010)

9. Goodwin, M.S., Intille, S.S., Albinali, F., Velicer, W.F.: Automated detection of stereotypical motor movements. J. Autism Dev. Disord. **41**(6), 770–782 (2011)

10. Gonçalves, N., Rodrigues, J.L., Costa, S., Soares, F.: Automatic detection of stereotyped hand flapping movements: two different approaches. In: The 21st IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 392–397 (2012)

11. Rihawi, O., Merad, D., Damoiseaux, J.L.: 3D-AD: 3D-autism dataset for repetitive behaviours with kinect sensor. In: 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6 (2017)

12. Müller, M.: Dynamic time warping. In: Information Retrieval for Music and Motion, pp. 69–84. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74048-3_4

13. Jazouli, M., Majda, A., Merad, D., Aalouane, R., Zarghili, A.: Automatic detection of stereotyped movements in autistic children using the Kinect sensor. Int. J. Biomed. Eng. Technol. **29**(3), 201–220 (2019)

14. Zhang, Z.: Microsoft kinect sensor and its effect. IEEE Multimed. **19**(2), 4–10 (2012)

15. Matson, J.L., Wilkins, J., Macken, J.: The relationship of challenging behaviors to severity and symptoms of autism spectrum disorders. J. Ment. Health Res. Intellect. Disabil. **2**(1), 29–44 (2008)

16. Vismara, L., Bogin, J., Sullivan, L.: Differential reinforcement of other behaviors: steps for implementation, Sacramento, CA (2009)

17. Oakley, C., Mahone, E.M., Morris-Berry, C., Kline, T., Singer, H.S.: Primary complex motor stereotypies in older children and adolescents: clinical features and longitudinal follow-up. Pediatr. Neurol. **52**(4), 398–403 (2015)

18. Willemsen-Swinkels, S.H., Buitelaar, J.K., Dekker, M., van Engeland, H.: Subtyping stereotypic behavior in children: the association between stereotypic behavior, mood, and heart rate. J. Autism Dev. Disord. **28**(6), 547–557 (1998)

19. Guess, D., Carr, E.: Emergence and maintenance of stereotypy and self-injury. Am. J. Ment. Retard. **96**, 299–319 (1991)

20. Sher, A., et al.: Automatic gait analysis during steady and unsteady walking using a smartphone. IEEE J. Biomed. Health Inform. (2021). https://doi.org/10.36227/techrxiv.16586780.v1

21. Adam, A., Ibrahim, Z., Mokhtar, N., Shapiai, M.I., Cumming, P., Mubin, M.: Evaluation of different time domain peak models using extreme learning machine-based peak detection for EEG signal. Springerplus **5**, 1–4 (2016)

22. Sarraf, J., Vaibhaw, P.K.: Analysis of motor imagery EEG signal classification based on amplitude-based peak detection method and pisarenko harmonic decomposition. J. Theoret. Appl. Inf. Technol. **100**(15), 4791–4800 (2022)

23. Zhang, C., Lin, H.B., Li, Y., Yang, B.J.: Seismic random noise attenuation by time-frequency peak filtering based on joint time-frequency distribution. C. R. Geosci. **345**(9–10), 383–391 (2013)
24. Rajagopalan, S., Dhall, A., Goecke, R.: Self-stimulatory behaviours in the wild for autism diagnosis. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 755–761 (2013)

25. Pandian, D., Rajagopalan, S.S., Jayagopi, D.: Detecting a child's stimming behaviours for autism spectrum disorder diagnosis using RGBPose-SlowFast network. In: IEEE International Conference on Image Processing (ICIP), pp. 3356–3360 (2022)

26. Lakkapragada, A., et al.: The classification of abnormal hand movement to aid in autism detection: machine learning study. JMIR Biomed. Eng. 7(1), e33771 (2022)