# A modular machine learning tool for holistic and fine-grained behavioral analysis

Bruno Michelot, Alexandra Corneyllie, Marc Thevenet, Stefan Duffner, Fabien Perrin

► To cite this version:

Bruno Michelot, Alexandra Corneyllie, Marc Thevenet, Stefan Duffner, Fabien Perrin. A modular machine learning tool for holistic and fine-grained behavioral analysis. Behavior Research Methods, 2025, 57 (1), pp.24. 10.3758/s13428-024-02511-3. hal-04870165

HAL Id: hal-04870165

https://hal.science/hal-04870165v1

Submitted on 8 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# A Modular Machine Learning Tool for Holistic & Fine-
Grained Behavioral Analysis

B. Michelot¹, A. Corneyllie¹, M. Thevenet¹, S. Duffner²*, F. Perrin¹*

**Affiliations:**

¹ CAP Team, Centre de Recherche en Neurosciences de Lyon - INSERM U1028 - CNRS UMR 5292 -
UCBL – UJM

² Laboratoire d'InfoRmatique en Image et Systèmes d'information - UMR 5205 CNRS - INSA Lyon -
Université Claude Bernard Lyon 1 - Université Lumière Lyon 2 - École Centrale de Lyon

**Corresponding author's contact:**

B. Michelot: 95 Boulevard Pinel (69675 BRON CEDEX) – 04 81 10 65 00 – bruno.michelot@etu.univ-lyon1.fr

*: co-last authors
Abstract

Artificial intelligence techniques offer promising avenues for exploring human body features from videos, yet no freely accessible tool has reliably provided holistic and fine-grained behavioral analyses to date. To address this, we developed a machine learning tool based on a two-level approach: a first lower-level processing using computer vision for extracting fine-grained and comprehensive behavioral features such as skeleton or facial points, gaze, and action units; a second level of machine learning classification coupled with explainability providing modularity, to determine which behavioral features are triggered by specific environments. To validate our tool, we filmed 16 participants across six conditions, varying according to the presence of a person ("Pers"), a sound ("Snd"), or silence ("Rest"), and according to emotional levels using self-referential ("Self") and control ("Ctrl") stimuli. We demonstrated the effectiveness of our approach by extracting and correcting behavior from videos using two computer vision software (OpenPose & OpenFace) and by training two algorithms (XGBoost & LSTM) to differentiate between experimental conditions. High classification rates were achieved for "Pers" conditions versus "Snd" or "Rest" (AUC=0.8-0.9), with explainability revealing actions units and gaze as key features. Additionally, moderate classification rates were attained for "Snd" versus "Rest" (AUC=0.7), attributed to action units, limbs and head points, as well as for "Self" versus "Ctrl" (AUC=0.7-0.8), due to facial points. These findings were consistent with a more conventional hypothesis-driven approach. Overall, our study suggests that our tool is well-suited for holistic and fine-grained behavioral analysis and offers modularity for extension into more complex naturalistic environments.

Keywords: BEHAVIOR, COMPUTER VISION, MACHINE LEARNING, EXPLAINABILITY
Introduction

Technological advancements in Artificial Intelligence (AI) techniques, especially machine
learning, have demonstrated their effectiveness and advantages for exploring behavior. Computer vision
techniques allow to extract and recognize various features of the human body from images or videos:
facial landmark detection, action recognition, pose estimation (Voulodimos et al., 2018). Despite some
limitations (e.g., sensitivity to lighting conditions, data quality, issues with recognizing hidden or
misoriented objects...), these techniques require less intrusive devices (i.e., a single camera) compared
to commonly used methods such as eye-tracking (Carter & Luke, 2020; Stein et al., 2021) or
accelerometry systems (Preece et al., 2009; Wong et al., 2007), making them particularly advantageous
for capturing behaviors unconstrained by experimental setups. Initially developed for commercial
applications (Al-Faris et al., 2020; Y. Chen et al., 2020; Echeverria & Santos, 2021), computer vision
techniques are now increasingly used in both animal and human studies (Badger et al., 2020; Ferrer-
Mallol et al., 2022; Gal et al., 2020; Geuther et al., 2019; Graving et al., 2019; Z. Jiang et al., 2022; Li
et al., 2021; Liu et al., 2017; Mathis & Mathis, 2020; Pereira et al., 2020; Rodriguez et al., 2018; Sridhar
et al., 2019; Zeghari et al., 2021). In humans, computer vision often operates at a high level of analysis,
i.e., focusing on semantic and symbolic aspects of behavior. Through end-to-end approaches (i.e.,
classification from raw images), it is frequently used for the recognition of emotions (e.g., Cohn & De
la Torre, 2015; Ko, 2018), postures (e.g., Kleinsmith et al., 2011), and specific actions, gestures or
broader activities (e.g., Camarena et al., 2023; Murthy & Jadon, 2009; Sigurdsson et al., 2016, 2017).
However, this approach is not adapted to describe fine-grained and dynamic behaviors and lacks
modularity as the algorithm is trained to recognize specific behaviors.

Conversely, lower-level approaches (e.g., Walther et al., 2023) allow for the extraction of points
or other elements (e.g., contours, surface normals) from the body without subsequent classification (e.g.,
Haines et al., 2019; Paxton & Dale, 2013). These approaches aim for a more detailed and granular
capture of behavior. However, because they deal with fine details, they may lack accuracy and precision
in their detection, requiring further adjustments to ensure the data is reliable and meets research
standards (e.g., Koul et al., 2023). Furthermore, most of the existing lower-level algorithms are
specialized in recognizing only one specific behavioral segment while they could facilitate a holistic approach to behavior i.e., capturing a complete picture of behavior that includes multiple aspects such as gaze, movements, and facial expressions all at once.

To our knowledge, there is currently no all-in-one tool that combine the strengths of low and high levels approaches without their drawback, i.e., simultaneously permits a reliable, comprehensive analysis of behavior while offering classification of the data after it has been processed. The objective of the present study was to develop a flexible and modular tool, to enable a detailed spatial and temporal analysis of behavior. We propose a decomposed two-level analysis approach: first, using computer vision for extracting various fine-grained behavioral features (lower-level processing), to improve both accuracy and completeness (reliability and holism); second, applying machine learning to classify the extracted behavioral features. This second step makes the tool modular, i.e., adaptable to different conditions and includes explainability (i.e., elucidating how the model makes its decisions, see for example Belle & Papantonis, 2021).

One potential application of this tool is to explore unconstrained and comprehensive behaviors driven by the environmental context in which we are situated (Krishnan-Barman et al., 2017) and the myriad stimuli to which we are exposed (e.g., sounds, persons, emotions, etc.). For instance, the mere presence of another person can subtly modify a large spectrum of behaviors (Gregory et al., 2015), including body movements (Alt & Phillips, 2022; Markus, 1978), gaze strategies (Gregory et al., 2015; Morgan et al., 2022), and facial expressions (Wagner & Smith, 1991), especially when associated with emotion (Ekman, 1993). To investigate the effect on human behavior of examples of environmental context that we may experience daily, we conducted a 3 x 2 design with two factors: a situational factor and an emotional factor. The situational factor included three levels: sound, rest and person. The emotional factor included two levels: self-referential (i.e., referencing the participants' autobiographical memory that we hypothesize will evoke emotions) and control. This 3x2 design resulted in six different conditions: two "sound" conditions (involving preferred music or complex noise), two "rest" conditions (following preferred music or complex noise), and two "person" conditions (involving a person recalling autobiographical memories associated with the music or providing factual information or). It is essential
to note that our goal was not to explore the effects of numerous factors (such as sensory modalities, speech intonation, etc.) on behavior, but rather to demonstrate the feasibility and effectiveness of our two-level approach with broadly contrasting conditions.

In the first level of analysis, we extracted fine-grained behavioral markers by adapting and combining two open-source computer vision algorithms, OpenFace (Baltrusaitis et al., 2016), and OpenPose (Cao et al., 2021): body and facial points, facial action units, and gaze direction. The adaptations (i.e., normalization, filters, interpolations) and the combination allowed us to obtain more reliable and comprehensive data.

In the second level of analysis, we adopted an exploratory approach (i.e., without hypotheses) to investigate the effects of our conditions (i.e., sound effect, person effect, emotional effect) on the pre-extracted behavioral features. This extracted data was analyzed by two machine learning algorithms that have demonstrated high suitability and accuracy for such heterogenous and noisy data: extreme gradient boosting (XGBoost, T. Chen & Guestrin, 2016) and long short-term memory (LSTM, Hochreiter & Schmidhuber, 1997) neural networks, in order to classify the data according to the conditions. We coupled this classification with two different explainability methods, namely information gain and integrated gradients, to determine the pertinent behavioral features for classification, i.e., behavioral differences between the conditions.

We also employed, as a supplementary analysis for the second level, a more conventional hypothesis-driven approach to demonstrate the modularity of our approach (i.e., the ability to adapt the second-level analysis method to test very precise and specific hypotheses). For this purpose, we drew upon four hypotheses from the literature related to our conditions and various extracted behavioral features. Notably, it has been well-established that individuals tend to adopt specific gaze strategies when looking at a person (Risko et al., 2016), particularly exhibiting rapid gaze movements towards various locations both on and off the face (Rogers et al., 2018). This phenomenon is even more pronounced when the person is physically present in the environment as opposed to virtually (Morgan et al., 2022). Thus, the first hypothesis (H1) was that "person" conditions increase gaze movements compared to "rest" or "sound" conditions. Additionally, studies have shown that one's posture, body movements, and facial expressions tend to match those of a person present in the environment
(chameleon effect), even when being entirely passive in that environment (Chartrand & Bargh, 1999; Chartrand & Lakin, 2013; Kunde et al., 2018). This is also true for subtle and nuanced behaviors such as blinks in a speaker-listener interaction (Nakano & Kitazawa, 2010). The second hypothesis (H2) was that "person" conditions modulates body movements and blinks compared to other conditions. Moreover, numerous works demonstrate that emotions, in general, elicit facial expressions in individuals (Ekman et al., 1980). Hence, we could postulate as a third hypothesis (H3) that "self-referential" conditions (associated with positive events) produce more positive facial expressions (i.e., AU12 and AU06 that collectively represent joy) than "control" conditions. Finally, it has been shown that during sound listening, spontaneous head movements aid in sound localization (e.g., Gessa et al., 2022). Additionally, listening to music induces a phenomenon known as head sway (González Sánchez et al., 2020). The fourth hypothesis (H4) posited that "sound" conditions elicit more head movements than "rest" conditions where there is no sound.

# Material & Methods

## Participants

Sixteen healthy participants aged 18 to 44 years (mean = 27.06, SD = 7.51, 11 women) were recruited. Only participants who had no hearing or visual impairments, neuropsychiatric disorders or external features on their face (such as piercings, glasses, tattoos, or beards) and autobiographical memory evoked by music (see below) were recruited. This study has been ethically approved by the committee for the protection of individuals (CPP Sud-Est - Lyon - 05-2019-18015) and was conducted in accordance with the guidelines of the Declaration of Helsinki. Participants received monetary compensation for their time.

## Stimulation

A telephone interview was conducted prior to inclusion to select specific stimuli for each participant. Participants were asked to give three favorite pieces of music that were associated with strong emotions (self-reported intensity was addressed during the experiment, see subsection Subjective
reports), most of the music being associated with a positive emotion of joy, and evoked autobiographical
memories. Participants were then asked a series of questions about the emotional and autobiographical
characteristics of their music (i.e., What context does this music make you think of? Does it remind you
of someone or a particular place? What are the predominant emotions when listening to this music?
What makes this music one of your favorites? etc.).

For each piece of music, two different one-minute excerpts were selected (usually one excerpt
with a verse and the other with a chorus), and fade-in/fade-out effects were applied using Audacity
software (Audacity Team version 3.1.3). Therefore, a total of 6 music excerpts were selected for each
participant. In addition, six one-minute white noise samples were created with Audacity. They were
stationary noises that were similar in spectrum to music, but did not share other acoustic characteristics
(e.g., pitch, rhythm, envelope, or timbre), as in (Castro et al., 2015). To get as close as possible to
emotionally control sounds, pre-tests were conducted on 10 participants and the results showed that
these sounds were perceived as emotionally neutral more than 85% of the time. All sounds (music and
noise) were normalized together using the Root Mean Square (RMS, -20dB) normalization technique to
avoid any volume bias (done on Audacity version 3.1.3).

For each musical excerpt, texts with self-referential content were written by an experimenter (6
per participant). These texts included 20 seconds of semantic information (title, author, genre) followed
by 40 seconds of emotional episodic narrative associated with the music. All information was arranged
to create a flowing discourse that could be perceived as emotional by the participant. In addition, six
control texts were written from Wikipedia, INRS and Cnrl, which were similar for all participants.
These texts contained factual information on predefined topics such as the definition of music, noise,
neuroscience, memories, album, and audition.

## Procedure

The experiment was conducted in a soundproofed audiometric booth to prevent any potential
noise contamination and was free of any object that could have drawn the participant's attention or
caused disruptions. The participant was seated in a bed, in a semi-recumbent position and fitted with a
128-electrode high-density electroencephalogram (EEG) helmet and electrocardiography (ECG) electrodes (the data obtained from these measurements will not be presented here). After conducting preliminary tests, we removed EEG electrodes from the face to avoid disrupting facial detection in video analyses. No movement constraints were imposed on the participant to maintain spontaneity, although certain devices may have limited full behavioral freedom (e.g., head movements with EEG). The experiment's procedure was explained without disclosing details about the stimulus. The experiment lasted approximately an hour and was followed by a debriefing survey to collect participant feedback on the entire protocol.

We employed a 3x2 design with 2 factors: a situational factor (with 3 levels: “sound”, “rest” or “person”) and an emotional factor (with 2 levels: “self-referential” or “control”), resulting in 6 different conditions (Figure 1). The 3 situational conditions (i.e., a triad) were always presented in the following order: "sound," "rest," and "person". The “self-referential” and “control” triads were presented in a pseudo-random order (ensuring no more than three consecutive presentations of the same emotional triad) both within and between participants.. The 6 conditions were repeated six times, resulting in a total of 36 trials, each lasting one minute (3 situational levels × 2 emotional levels × 6 repetitions).

In the sound conditions, participants were alone in the room. In the self-referential sound condition ("SelfSnd"), they listened to an excerpt of their preferred music, whereas in the control sound condition ("CtrlSnd"), they were exposed to white noise.

The rest conditions followed the sound exposure. In the self-referential rest condition ("SelfRest"), participants sat in silence after listening to their preferred music (“SelfSnd”), while in the control rest condition ("CtrlRest"), they sat in silence after hearing white noise (“CtrlSnd”).

Lastly, in the person conditions, the experimenter was in the room and recited the prepared texts.

In the self-referential person condition ("SelfPers"), the text was related to self-referential content of the music previously listened to (thus following “SelfSnd” and “SelfRest”), while in the control person condition ("CtrlPers"), the text contained factual information (thus following “CtrlSnd” and “CtrlPers”).

The participant was instructed not to speak during the experimenter's intervention but could do
so if necessary, at the end.

Experimental materials were displayed using the Presentation software (Neurobehavioral
Systems, Inc., Berkeley, CA, version 19.7) on a screen positioned in front of the participant. The software enabled the random presentation of conditions with a slight delay and recorded participants' responses (see below) using a mouse positioned on a nearby table. Additionally, digital signals were sent to the video acquisition software (see below) at the beginning of each trial.

## Subjective reports

At the end of each trial (thus 36 times), the participant had to report her/his subjective feeling of emotion (none to strong) by moving a cursor along a horizontal and linear scale presented on the screen and clicking to register their response. We applied min/max normalization to standardize the data obtained from the subjective scales across participants, and then computed the average score for each condition and participant. Subsequently, a global average within subjects and the standard error were calculated. To examine statistical differences between conditions, we performed a two-factor repeated measures ANOVA (with situational and emotional factors) and conducted post hoc test, with a first-order alpha risk of 5%. Statistical tests were made using JASP Software (JASP Team 2022, Version 0.16.2).

<caption>**Figure 1**</caption>

*Experimental procedure.*

<figure><img src="image_2.png" /></figure>

*Note.* The experiment consisted of a 3 x 2 design: three different situations always presented in the same order (i.e., sound, rest or person) each with two emotional levels (i.e., self-referential or control). "Self-
referential" included the conditions "SelfSnd," "SelfRest," and "SelfPers," while "control" included the conditions "CtrlSnd," "CtrlRest," and "CtrlPers." Each condition was repeated six times.

## Video acquisition

Participant videos were recorded using a Logitech Brio HD 4k camera at a resolution of 1920x1080, in color, and at a sampling rate of 20 FPS. The camera was securely fixed on a beam approximately 2 meters above the participant's feet to enable a full-body view. Preliminary tests were conducted by filming the experimenters with varying lighting and clothing contrasts, different camera angles to account for head movements and rotations, and testing potential disruptions like piercings, glasses, tattoos, long sleeves, and jewelry, to ensure optimal performance of the computer vision software and avoid video quality issues. To prevent the participant from gazing at the camera, it was hidden behind a cloth that had several small holes, with the camera lens positioned behind one of them to ensure unobstructed recording. The camera was connected to a custom-made video acquisition software developed using LabVIEW, version 2021 (National Instrument, as in Chalençon et al., 2022). This software allows for the acquisition of video signals over long periods and enables their temporal reconstruction using machine time. It is equipped with a synchronization system that receives triggers from the stimulation computer via a serial port, which facilitated the identification of periods of interest corresponding to the conditions in the video.

## Video processing

### Computer Vision

Videos were automatically analyzed by two computer vision algorithms. OpenFace (Baltrusaitis et al., 2016) is an open-source tool that detects facial cues frame by frame, allowing us to extract (a) 2D x/y coordinate of 70 points of the face (including contour, rotation, and facial elements), (b) the angle of gaze (in radians), and (c) the activation intensity of 17 AUs (ranging from 0 to 5) based on the Facial Action Coding System (FACS) (Ekman & Friesen, 1978). OpenPose (Cao et al., 2021) is another open-source computer vision tool that detects the human body
via articulation points of the limbs frame by frame. The 2D x/y coordinates of 15 points from the stem
and the two upper limbs were extracted. The total pool and nature of extracted features is provided in
the supplementary material (Supplementary Table 1).

Preprocessing

OpenFace and OpenPose were not fully adapted for behavioral research, i.e., to obtain reliable
fine-grained values. They produce errors due to the variability of body and head shape, facial
expressions, and lighting (Yu et al., 2017). Thus, the corrections detailed below have been applied to
the raw data using Python scripts (version 3.9, scripts with documentation are fully available at
https://github.com/crnl-lab/Tracking Behavior IA 2023 BMichelot as example data for testing scripts
at https://zenodo.org/record/8139206).

To unify the format and label the data in relation to the stimuli, we first restructured the data
from both tools (Restructuring, Figure 2).

Next, we applied smoothing to the OpenPose data, as the detection of the individual points can
oscillate due to the quantization effects leading to noise in the estimated motion. Specifically, frames
were smoothed if the median difference between two consecutive frames was less than or equal to the
size of a jump +/-1 pixel (Smoothing, Figure 2). Additionally, any frames with no detection or aberrant
detections were automatically discarded.

To account for the few discarded frames, they were linearly interpolated as it could have led to
errors in the interpretation of results (e.g., movements, see below “Transformation”) (Interpolation,
Figure 2). Note that interpolation was used for a limited number of consecutive frames (e.g., a maximum
of 10 frames = 0.5 seconds).

AUs (OpenFace) were individually normalized by calculating the average intensity for each AU
and each participant and by subtracting it from the data frame by frame in a second scan. This
normalization provided greater robustness and accuracy to AU intensity measurements by taking into
account the individual characteristics of the participant (Normalization, Figure 2). OpenPose data were
normalized to the size of the participant (segment between bust and hip points) and to a relative point
on the body (nose for face, bust for body), which allowed us to remove any size and position bias
(Normalization, Figure 2).

Finally, for the x/y coordinates, we calculated the Euclidian distances and the difference between the Euclidian distance values of a frame and those of the previous one, to represent the motion magnitude of each point (Transformation, Figure 2).

<caption>**Figure 2**</caption>

Data Preprocessing steps

<figure><img src="image_3.png" /></figure>

*Note.* From the raw videos, behavioral metrics are extracted using OpenFace and OpenPose. These metrics undergo various processing steps, including restructuring, smoothing, interpolation, normalization, and transformation, to enhance their quality and reliability.
# Video analysis

## Exploratory analyses

### Machine learning algorithms

Two distinct state-of-the-art machine learning algorithms were used to classify the conditions from behavioral features: gradient boosting using XGBoost and Long Short-Term Memory (LSTM) neural network.

For XGBoost, we normalized each input feature using the min/max method (Patro & Sahu, 2015). We then applied a moving average operation with a window size of 1000 frames (50 seconds) to each feature to aid in the model's learning. The hyperparameters of the model were optimized using Optuna (Akiba et al., 2019), with ranges given for the following parameters: tree depth (5-14), number of estimators (1000-15000), L1 (1-50) and L2 (5-100) regularization terms on weights, minimum sum of instance weight needed in a child (2-20), and subsample ratio of columns when constructing each tree (0.1-1).

For the LSTM model, the input dimension was set in accordance with the number of features. It consisted of 3 layers: a fully connected (dense) layer with a dimension of 256, an LSTM layer with a dimension of 512, and a fully connected layer with a dimension representing the number of classes. We included a “logsumexp” aggregation function before the output layer, which aggregated over the sequence length, acting as a sort of temporal “softmax” function. From each condition we extracted overlapping sequence of length 1000, where the step size during training was set to 60 and during testing to 10. To optimize the training, the Adam optimization algorithm (Kingma & Ba, 2017) was utilized with the following parameters: beta1=0.99, beta2=0.999, weight decay=10e-4, and a learning rate of 0.0007.

### Classification contrasts

We conducted a series of two-class classifications in order to extract, in a second step, the relevant elements of behavior in details using explainability tools. Three distinct groups of comparisons were made to study three effects. The person group ("Pers") was investigated through four comparisons: SelfPers vs [SelfSnd or SelfRest] and CtrlPers vs [CtrlSnd or CtrlRest]. The sound group ("Snd") was
explored through two comparisons: SelfSnd vs SelfRest and CtrlSnd vs CtrlRest. The emotion group ("Emo") was examined through three comparisons: SelfPers vs CtrlPers, SelfSnd vs CtrlSnd, and SelfRest vs CtrlRest.

We evaluated the classification performance of the machine learning algorithms in terms of accuracy (%) (see Supplementary Material) and area under the curve (AUC), which indicates their ability to predict the condition based on the extracted behavioral features. An accuracy or AUC value above 0.50 suggests that the model can predict the target outcome better than chance. To assess the performance on our dataset, we conducted cross-validation, with the dataset divided into three subsets: a training set (14 participants), a validation set, and a test set (each containing one participant). We performed 16 iterations to ensure that each participant was once in the validation and test sets. The performance (accuracy and AUC) for each classification was obtained by averaging the results of the 16 folds on the test set, and the standard error was calculated.

To determine if the performances were significantly higher than chance, we conducted one-sample t-tests using a chance level of 0.5. We also conducted a one-factor repeated measures ANOVA (factor "groups") with post-hoc tests at a significance level of 5% (alpha risk) to compare the AUCs among the three groups and identify any significant differences in classification performance. To address the issue of multiple comparisons, the Bonferroni correction was applied to adjust the p-values.

## Relevant behavioral features for classification (explainability)

Decisions made by machine learning models for classification were explored using two different explainability methods for the two machine learning algorithms. They highlight the features that were relevant for classification.

For XGBoost, we used the information gain measurement method (Azhagusundari & Thanamani, 2013) to determine the contribution of each feature to the impurity reduction during the tree building process.

For the LSTM, we used the integrated gradients method (Sundararajan et al., 2017) to compute
the gradients of the model's output with respect to the input features and attribute importance scores to each feature.

Both methods were applied to each two-class classification, resulting in importance values for each behavioral feature and fold. These values were ranked from highest to lowest for each participant, and the average ranking was calculated within each classification. In parallel, we regrouped the behavioral features into clusters to synthesize the results, as there are over 100 different features (see Supplementary Material for detailed clusters and results). These clusters were based on specific facial areas and feature types, i.e., AU - Eyes, AU - Mouth, AU - Nose, Points - Eyes, Points - Mouth, Points - Nose, Points - Limbs, Points - Head Contour, and Gaze. For each classification, the top 5 most decisive features from both algorithms were examined. Finally, we establish a binary score to quantify the degree of presence of a feature cluster in explaining a group of classification: if a feature cluster had at least one component present in the top 5, a score of 1 was assigned to that cluster. Otherwise, a score of 0 was assigned to the cluster. The average of these scores (1 and 0) was finally calculated within the "Pers," "Snd," and "Emo" groups.

## Hypothesis-driven analyses

As a supplementary type of analysis, we also conducted calculations on behavioral features to address 4 hypotheses that we derived from the literature regarding differences in behavior among our specific conditions. We computed the mean intensity of AUs for each condition across all participants, and also calculated the standard error. For points (skeleton, facial elements, head) and gaze, we quantified the overall amount of motion based on the vector differences in the transformed data. We summed the values for each feature within each condition to obtain a representation of the cumulative distance and then averaged the sums across all participants and we calculated the standard error. To test our 4 hypotheses, some conditions were averaged: "Pers" represents the average of "SelfPers" and "CtrlPers" conditions, "Snd" represents the average of "SelfSnd" and "CtrlSnd" conditions, "Rest" represents the average of "SelfRest" and "CtrlRest" conditions, "Self" represents the average of
"SelfSnd", "SelfRest", and "SelfPers" conditions and "Ctrl" represents the average of "CtrlSnd", "CtrlRest", and "CtrlPers" conditions.

Specific features were chosen in accordance with our hypotheses. For hypothesis 1 (i.e., more eye movements in "Pers" compared to "Snd" and "Rest"), we chose the gaze angle cumulative distance as a feature. For hypothesis 2 (i.e., modulation of body movements and blinks in "Pers" compared to "Snd" and "Rest"), two features were selected: the cumulative distance of a point on the wrist (body point 4), assuming that the movement of this point is indicative of the movement of the corresponding area (i.e., if the shoulder moves, the position of the wrist will also be altered, whereas the reverse may not necessarily hold true), and the action unit intensity of blinks (AU45). For hypothesis 3 (i.e., more positive facial expressions in "Self" compared to "Ctrl"), we selected the intensities of the two AUs found in the emotion of joy according to FACS (Ekman & Friesen, 1978): AU12 – Lip corner puller and AU06 – Cheek raiser. For hypothesis 4 (i.e., more head movements in "Snd" compared to "Rest"), the head position cumulative distance was chosen as it is the most representative feature for assessing global head movements.

To test the significance of these differences, we conducted either Wilcoxon's signed rank test (for pairwise comparisons) or Friedman test (for multiple comparisons), along with Nemenyi's post-hoc tests to explore pairwise differences, at a significance level of 5% (alpha risk). These non-parametric tests were chosen due to the data not always conforming to a normal distribution and the presence of violations of sphericity assumptions, which preclude the use of ANOVA tests, for instance. Additionally, Bonferroni correction was applied to all statistical tests to adjust the p-values.

# Results

## Subjective feeling of emotions

The results of the subjective scales showed a significant effect of the emotional factor (see Supplementary Table 3 for values and statistics). Participants reported significantly higher emotional feelings in "Self" conditions compared to the associated "Ctrl" conditions (as shown in Figure 3).
<caption>Figure 3</caption>

*Emotional subjective feeling of participants in self-referential and control conditions*

<figure><img src="image_4.png" /></figure>

*Note.* Participants experienced significantly stronger emotional responses in self-referential conditions ("SelfSnd", "SelfRest", "SelfPers") where the stimuli were autobiographical, compared to the associated control conditions ("CtrlSnd", "CtrlRest", "CtrlPers") where the stimuli were neutral (n=16, ***p < 0.001).

Classification performances

In terms of classification performance, both algorithms (as shown in Figure 4, panel A) exhibited significantly higher AUC values for the "Pers" group combinations (AUCs around 0.9) compared to the "Snd" group (AUCs around 0.7) or the "Emo" group (AUCs around 0.7, see Supplementary Table 4 for all values and statistics). Furthermore, the dispersion of results for each participant tended to be larger around the 0.9 mark for the "Pers" group in both algorithms, while it was more scattered for the other two groups, with a slight concentration of results around the 0.7 mark (as shown in Figure 4, panel A).
This indicates that for the "Pers" group, the models appear to be consistent in classifying these
conditions among subjects, whereas for the other two groups, there are performance differences among subjects, indicating that the differences are less pronounced for these contrasts and that the models have more difficulty generalizing their learning.

Performances of the models (in AUC, see Supplementary Figure 1 for detailed results in AUC and accuracy) for each comparison was significantly above chance in the "Pers" group (Figure 4, panel B), with AUC rates approaching 0.95 (e.g., SelfPers vs. SelfRest). The results of both algorithms were similar in this group, with no significant difference present between their performances. For the "Snd" group (Figure 4, panel B), performances were significantly above chance level but with AUCs around 0.7. In the "Emo" group (Figure 4, panel B), the classification performance in AUC of the "SelfPers" vs. "CtrlPers" conditions was significantly higher than the chance level for both algorithms (AUCs around 0.8), as it was the case for "SelfSnd" vs. "CtrlSnd" and "SelfRest" vs "CtrlRest" to a lesser extent.

<caption>**Figure 4**</caption>

*Classification performances of XGBoost and LSTM models on different contrasts of groups and conditions within groups.*

<figure><img src="image_5.png" /></figure>
*Note.* (A) The classification rates of both models were significantly higher in the "Pers" group compared to the other two groups with average classification rates for this group of contrasts around 90% (n=16, ***p<0.001). (B) All of the model classifications within the "Pers" group were significantly above chance in terms of their performance with some values reaching up to 95% AUC (e.g., "SelfPers" vs "SelfRest") whereas performances were lower for the other two groups (e.g., in "Snd" group: "SelfSnd" vs "SelfRest"; in "Emo" group: "SelfRest" vs "CtrlRest") (n=16, ***p < 0.001, **p<0.01, *p<0.05).

## Explainability results

Regarding the explainability results, the decisive behavioral features for classifying the different conditions were predominantly focused on facial features. To explain "Pers" group (Figure 5 - in red), the algorithms predominantly utilized eye-related and mouth-related AUs, as at least one of each was present in the top 5 features for the 8 classifications belonging to this group (score of 1). Moreover, the importance of gaze in classifying these conditions was notable as it emerged as a decisive factor in multiple classifications (score of 0.25). Finally, head contour points and nose-related points seemed to be fairly decisive for classification (score of 0.75 and 0.5, respectively).

For "Snd" group (Figure 5 - in blue), the most distinguishing features were the AUs related to the nose as well as the points of the head contour (both scoring 1). Additionally, similar to "Pers" group, there was an importance of AUs related to the mouth to distinguish these conditions (score of 1). Finally, limbs point also appear to be important in the classification with a score of 0.5.

Finally, for the decisive features in classifying "Emo" group (Figure 5 - in green), mouth and eye points were consistently present in the top 5 features for both models in each of the 3 classifications belonging to this group (scores of 1). Additionally, it appears that nose-related AUs were also decisive for this group (score of 0.666), as well as limb points (score of 0.333).
<caption>**Figure 5**</caption>

*Relevant behavioral features for classification of the different groups “Pers”, “Snd”, “Emo”.*

<figure><img src="image_6.png" /></figure>

*Note.* The behavioral features that explain “Pers” group are predominantly mouth and eye-related AUs. Gaze, head contour, and points on different parts of the face also appear to be important. For “Snd” group, features related to the face / head contour are primarily influential, as it is also for mouth and nose. Mouth and eye area points are highly decisive for “Emo” group, while nose AUs and limb points also seem to play a significant role in this group.

**Hypothesis-driven analyses results**

For hypothesis 1, a significant increase in gaze cumulative distance (i.e., a representation of the amount of movement performed in a condition) in "Pers" was found (as shown in Figure 6, panel A), compared to "Rest" and, to a lesser extent "Snd" (see Supplementary Table 5 for all values and statistics). Regarding hypothesis 2, the two selected features (action unit 45 of blinks and body point 4 of wrist) showed, respectively, a significant decrease in average activation intensity and a significant increase in
cumulative distance in "Pers" compared to "Snd" and "Rest" (Figure 6, panel B). The increase in facial expressions in "Self" formulated in hypothesis 3 appeared to be likely confirmed with an increase in the average activation intensity of AU12 (Lip corner puller) and AU06 (cheek raiser) in "Self" compared to "Ctrl" (as shown in Figure 6, panel C). Finally, for hypothesis 4, a significant increase in the cumulative distance of the head position feature (i.e., representing global head movements) in "Snd" compared to "Rest" was found (Figure 6, panel D).

<caption>**Figure 6**</caption>

*Modulation of gaze, wrist, head movements and different action units by the conditions.*

<figure><img src="image_7.png" /></figure>

*Note.* (A) The gaze angle demonstrated a significant increase in cumulative distance in “Pers” compared to “Snd” and “Rest” (H1). (B) The wrist and blinks showed respectively a significant increase and decrease in their cumulative distance in “Pers” compared to “Snd” and “Rest” (H2). (C) Moreover, two examples of action units (AUs) exhibited a significant increase in their average activation intensity in “Self” compared to “Ctrl” (H3). (D) Finally, head position showed a significant increase in “Snd” compared to “Rest” (H4). (n=16, ***p < 0.001, **p<0.01, *p<0.05).
Discussion

The aim of the present study was to develop a modular tool based on machine learning to enable
a detailed spatial and temporal analysis of behavior. To this aim, we proposed an original approach in
two levels. First, we used computer vision to extract detailed and comprehensive behavioral features.
Then, in the second part, we applied machine learning algorithms to classify these features and added
explainability methods to understand how the models made their decisions. Additionally, we conducted
a traditional hypothesis-driven analysis to demonstrate how our tool could answer specific research
questions. As a proof of concept, we used a 3x2 experimental design to manipulate both a situational
factor and an emotional factor. This resulted in 6 different conditions: two "Snd" conditions (involving
preferred music or complex noise), two "Rest" conditions (following preferred music or complex noise),
and two "Pers" conditions (involving a person recalling autobiographical memories associated with the
music or providing factual information). We demonstrated the effectiveness of our tool by successfully
extracting behavior from video using two computer vision algorithms, namely OpenPose and OpenFace,
and achieved high classification performances of most of our conditions employing two algorithms,
XGBoost and LSTM. Furthermore, the use of explainability methods was able to show that each effect
resulting from groups of conditions was associated with different behavioral features, which aligned
with specific hypothesis-driven analyses.

First level of analysis: extraction and correction of behavioral features

The increasingly diverse array of computer vision algorithms has allowed us to use familiar
open-source software, OpenPose and OpenFace, for our first analysis level. These tools have
demonstrated their ability to analyze various aspects of human behavior in different diseases, such as
apathy, depression, anxiety, lying, concentration, and motor functions in rare diseases (Altmann et al.,
2021; Ferrer-Mallol et al., 2022; Z. Jiang et al., 2021; Jones et al., 2020; Shen et al., 2021; Zeghari et
al., 2021). However, compared to studies that have used them separately and without adjustments, we
have integrated them to improve the reliability of the raw data. Indeed, computer vision is not yet fully
tailored to human behavior studies due to several drawbacks, notably its demand for high-quality video
and optimal conditions to avoid artifacts or recognition failures caused by concealed or misaligned
objects (H. Chen et al., 2022). To tackle this issue, we maximized recording conditions (camera angle,
lighting, disruptive elements) and we implemented a series of post-processing adaptations on the data,
such as creating specific normalization of action units in OpenFace or removing artifacts produced by
OpenPose (e.g., small detection jumps that could be misinterpreted as movement) and handling detection
gaps in the position data only through minor sequence interpolations (as in Koul et al., 2023), to mitigate
the associated risks (e.g., over-interpolation, signal dependence).

Second level of analysis: extraction and correction of behavioral features

This second level of analysis was aimed to render our tool modular and flexible, i.e., adaptable
to a wide range of experiments and scientific questions on behavior. To assess the applicability of our
method, we placed participants in 6 different conditions that are inherently diverse but yield subtle
behavioral differences. We were aware of the fact that a myriad of environmental factors could influence
behavior (such as the sensory characteristics of stimuli, vocal intonation, musical tempo, etc.), and the
present study did not aim to comprehensively explore all these factors. Instead, it aimed to ascertain
how global everyday situations (although we are still far from being able to test this "in the wild") could
impact a range of bodily features, and to demonstrate that they could be analyzed using an exploratory
approach (employing classification and explainability tools) or even through conventional hypothesis-
driven methods (using contrasts between specific conditions and with respect to particular features).

There are numerous algorithms and architectures for this type of data (Fawaz et al., 2019), but
XGBoost and LSTM are known to be the most suitable algorithms for sequential data and classification
tasks (H. Chen et al., 2018). Note that while we had a limited number of participants in our study (which
can sometimes pose a challenge when using machine learning models), this was not a significant issue.
Indeed, XGBoost has demonstrated high efficiency with small data quantities (T. Chen & Guestrin,
2016), and the logsumexp function we employed for LSTM prevented overfitting. Another unique
aspect of the current study was also the utilization of explainability methods to emphasize the bodily
characteristics that elucidate classification, which are still underused outside of AI (Tjoa & Guan, 2021;
Zhang et al., 2022) despite their potential to provide functional insights into the results. Similar to the architectures, we selected two suitable methods for the models used, but there are numerous other methods that would be worth exploring in the future (Adadi & Berrada, 2018). Both methods yielded different yet complementary results (see Supplementary Material for details): the LSTM architecture seemed to capture significant and precise moment, emphasizing AU prominence, while XGBoost seemed to focus on small variations, here point movements aligning with AUs used by the LSTM.

## Overall effect of the presence of a person

Regardless of the classification model used, the results consistently demonstrated remarkably high classification rates, with AUC score ranging from 0.8 to 0.9, for all contrasts within the “Pers” group, i.e., “Pers” conditions (presence of a person), compared to “Snd” (alone hearing a sound) or “Rest” (alone in silence) conditions (Figure 4, panel B). This suggests that the mere presence of another individual in the environment elicits very distinct behavioral patterns. This is consistent with previous literature, which has consistently reported the influential role of presence in shaping various aspects of behavior (Mnif et al., 2022, Chartrand et Bargh 1999, Gregory 2015). The explainability analyses showed that the “Pers” effect was made upon facial expressions involving the eyes and mouth, as well as to a lesser extent, gaze angle and points on the contour of the head (Figure 5). Interestingly, these results were consistent with our first two hypotheses (Figure 6, panels A & B), which posited an increase in gaze and a modulation of movements and blinks in the "Pers" conditions compared to "Snd" and "Rest". For example, AU45, which corresponds to blinks (and was included in the "AUs - Eyes" cluster) consistently appeared in all contrasts within the “Pers group” (details in Supplementary Material) and was shown to be significantly less activated in the “Pers” conditions compared to “Snd” and “Rest” conditions. This is consistent with the literature that has demonstrated a modulation of blinks for communication and synchronization in listener-speaker interaction (Hömke et al., 2018, Nakano 2010). Blink frequency was shown to be higher during periods of low cognitive load (Siegle et al., 2008) and serves as a mechanism for disengaging attention (Nakano et al., 2013). Additionally, listeners would suppress their blinks when concentrating on auditory information in speech (Nakano & Kitazawa, 2010).
Therefore, it is possible that a person speaking to us requiring greater attention and cognitive engagement and that it led to a suppression of blinks or a decrease in blink frequency.

Gaze angle was another relevant feature that explained the “Pers” effect and was shown to be more moving in the “Pers” conditions than in “Snd” and “Rest” conditions. This is consistent with the extensive literature on gaze strategies that showed an inhibition of fixations when looking at a person in real-life situations (Risko et al., 2016) and that individuals tend to switch their gaze between the eyes and other facial elements to better understand what is being said (Rogers et al., 2018). Interestingly, it was also demonstrated that gaze aversion during an interaction could reduce the monitoring cost of a face, and this aversion is correlated with the frequency of smiling during highly emotional moments as a way to reduce embarrassment (Cañigueral & Hamilton, 2019). This could explain why gaze patterns were modulated by the presence of a person, as well as the involvement of specific AUs related to the mouth in the explainability analysis (e.g., AU12 - Lip corner puller) or the eyes (e.g., AU06 - Cheek Raiser), both of which are associated with the expression of joy and smiling (Ekman & Friesen, 1978).

Finally, in accordance with the second hypothesis, limb movement (wrist) increased in conditions where another person was present, suggesting an individual's engagement in interpersonal synchrony and non-verbal communication (Paxton & Dale, 2017; Tsuchiya et al., 2020). In contrast, this body feature was not utilized by the models for classification. Instead, the models used overall head movements, which have also been shown to indicate engagement in interpersonal postural coordination (Varlet et al., 2011). This suggests that the classification models do not rely exactly on the same metrics that we would calculate in more conventional approaches, supporting the complementarity of the two approaches.

## Overall effect of the presence of a sound

An AUC score of around 0.7 was obtained in the classification within the “Snd” group, i.e., “Snd” conditions as compared to a “Rest” conditions (Figure 4, panel B). This group was significantly less accurately classified than the “Pers” group (Figure 4, panel A), indicating that the presence of a sound had less effect on behaviors than the presence of a person, but it was still sufficient for the models to distinguish between them. The behavioral characteristics explaining these conditions were the AUs
of the mouth and nose, as well as the importance of points along the contour of the head and limbs (Figure 5). These results could be associated with music conditions, that produce emotional facial reaction (Juslin & Sloboda, 2013) and also provoke dancing behaviors (Carroll & Moore, 2008) or head sway phenomenon (González Sánchez et al., 2020). Listening to sounds, such as white noise, also induces spontaneous head movements, because it can improve sound localization and externalization (Colas et al., 2023; Gessa et al., 2022; Iwaya et al., 2003; Perrett & Noble, 1997). Interestingly, we found a similar result in our hypothesis-driven analyses (H4) as there were more movements from these points on the head in the presence of a sound ("Snd") than in silence ("Rest") (Figure 6, panel D).

## Overall effect of emotional content

The models achieved an AUC score of around 0.7 for the classifications of the contrasts involving a “Self” condition against its counterpart “Ctrl” condition, with a notable performance for the specific comparison between the two "Person" conditions, SelfPers and CtrlPers, which stood out with an AUC of around 0.8. To classify emotional vs. neutral conditions, the eye and mouth-related points appeared to be decisive. One possible explanation is that these points capture subtle eye and mouth movements that may not trigger specific AUs, but still contribute to the classification. The results obtained with the hypothesis-driven analyses (H3) were consistent with those of the explainability analysis, as they showed a higher activation of AU12 (lip corner puller) and AU6 (cheek raiser) with emotions (i.e., “Self” compared to “Ctrl”) (Figure 6, panel C). We know that emotional stimuli elicit facial expressions (Ekman, 1993; Ekman et al., 1980), particularly the expression of joy composed of these two AUs. Finally, within this “Emo” group, it was more challenging for both models to differentiate between the two "Rest" conditions. This suggests that these two “Rest” conditions elicited either similar behaviors or behaviors that were less or not detectable by our tool, probably because the participants did not experience emotions.
## Perspectives

We have developed a modular, fine-grained, and comprehensive tool for human behavior analysis, that can be further improved in the future. As previously mentioned, the scenarios used to test our tool have limitations. To ensure maximum control and reliability, we filmed only one individual seated in a bed, which created a realistic resting situation but mitigate other scenarios. This environment may have restricted their mobility, both in terms of space and in behaving naturally in front of an unfamiliar interlocutor in a laboratory setting. One perspective, therefore, is to move toward more everyday naturalistic conditions. For instance, one of the next pressing steps would be to allow participants more movement by not constraining them to a fixed position, such as permitting them to stand and move freely (e.g., dance while listening to music). This would likely require adjustments to the camera position and angle, as well as more robust computer vision models that can handle different orientations in space. Additionally, more familiar and natural situations could be tested while remaining manageable for research purposes (e.g., problems solving, teaching in classrooms, interactive activities in enclosed public spaces). Gradually introducing more individuals into the environment would enable us to explore dynamic and complex social interactions (Goldstein et al., 2023) such as the non-verbal dyads that occur in a speaker-listener interaction (Dunbar et al., 2022) or to analyze behaviors more finely in groups of individuals (Sweeny & Whitney, 2014). This would require new adaptations of our approach due to the numerous artifacts produced by these situations (Robie et al., 2017) that we cannot currently handle. Methodological advancements would need to address robustness in varying lighting conditions, dynamic background changes, scalability to larger data sets, and challenges with managing occlusions or changes in object orientation. Performance validation and benchmarking will be critical here, requiring careful definition of success metrics to evaluate the tool's accuracy across these different environments. A short term solution would be to use multiple cameras and synchronize them to capture all behaviors occurring in three-dimensional space. A possible extension of our tool in this direction would involve adding new preprocessing steps to ensure the processing of data from multiple cameras, i.e., selecting optimal camera feeds, thus minimizing data loss from occlusions or suboptimal body part orientation, alongside synchronization or calibration scripts. At this point, modifications to the
normalization process would be required to ensure consistency in the analysis pipeline.

Another short term perspective would be to focus, in the second level of analysis, on higher-
level behaviors (i.e., more semantic, symbolic, etc., such as gestures): either by instructing the model to
search for behavioral patterns in our data (similar to high-level computer vision approaches but without
the need to pre-train a computer vision model), or by conducting post-processing on the extracted data
and testing the modulation of a specific high-level behavior. Research could then leverage the holistic
and modular aspects to investigate the coordination of these higher level behaviors.

A last example of perspectives could be to test other classification models suited for time series
data to compare our performance and thus enhance (or not) our behavioral analysis. Examples of state-
of-the-art models include attention-based algorithms or large language models, which have proven
effective in capturing complex temporal dependencies (Y. Jiang et al., 2024; Vaswani et al., 2017).
However, the first significant hurdle in this direction would be managing the large volumes of data
required by these advanced models. This perspective also applies to explainability.

## Acknowledgments

We thank the following individuals for their work on this study: Hakima Abouhssouss, Badr-Eddine Benelmostafa, Alexandre Bremard, Andréa Croc, Robin Ducclermortier, Xinyu Feng, Martin Grouiller, Jiadong Guo, Zihao Hua, Romane Jondeau, Sophie Laboucheix, Gaëtan Scart, Fatoumata Wade.

## Declarations

* **Funding:** This work was supported by the Fondation pour la Recherche Médicale (FRM – ECO202206015490), by the CNRS MITI (Mission pour les Initiatives Transverses et Interdisciplinaires) and the SHAPE-MED@Lyon project (ANR 22-EXES-0012).

* **Conflicts of interest/Competing interests:** The authors have no competing interests to declare that are relevant to the content of this article.
*   **Ethics approval:** This study has been ethically approved by the committee for the protection of individuals (CPP Sud-Est - Lyon - 05-2019-18015) and was conducted in accordance with the guidelines of the Declaration of Helsinki.

*   **Consent to participate:** Informed consent was obtained from all individual participants included in the study.

*   **Consent for publication:** All participants in this study provided informed consent for the publication of the research results, as included in the consent form approved by the CPP Sud-Est - Lyon - 05-2019-18015.

*   **Availability of data and materials:** The extracted and preprocessed raw data used for analysis are fully accessible.

*   **Code availability:** The scripts used in this study is fully available, as are the example data for testing the scripts.

**OpenPractice statement:** The data and scripts used for analysis are fully available at https://zenodo.org/record/8139206 and https://github.com/crnllab/Tracking Behavior IA 2023 BMichelot respectively.

## References

Adadi, A., & Berrada, M. (2018). Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). *IEEE Access, 6*, 52138-52160. IEEE Access.
https://doi.org/10.1109/ACCESS.2018.2870052

Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A Next-generation Hyperparameter Optimization Framework. *Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, 2623-2631.
https://doi.org/10.1145/3292500.3330701
Al-Faris, M., Chiverton, J., Ndzi, D., & Ahmed, A. I. (2020). A Review on Computer Vision-Based Methods for Human Action Recognition. *Journal of Imaging*, 6(6), 46. https://doi.org/10.3390/jimaging6060046

Alt, N. P., & Phillips, L. T. (2022). Person Perception, Meet People Perception: Exploring the Social Vision of Groups. *Perspectives on Psychological Science*, 17(3), 768-787. https://doi.org/10.1177/17456916211017858

Altmann, U., Friemann, C., Frank, T. S., Sittler, M. C., Schoenherr, D., Singh, S., Schurig, S., Strauss, B., & Petrowski, K. (2021). Movement and Emotional Facial Expressions during the Adult Attachment Interview: Interaction Effects of Attachment and Anxiety Disorder. *Psychopathology*, 54(1), 47-58. https://doi.org/10.1159/000512127

Azhagusundari, B., & Thanamani, A. S. (2013). Feature Selection based on Information Gain. *International Journal of Innovative Technology and Exploring Engineering (IJITEE)*, 2(2).

Badger, M., Wang, Y., Modh, A., Perkes, A., Kolotouros, N., Pfrommer, B. G., Schmidt, M. F., & Daniilidis, K. (2020). *3D Bird Reconstruction: A Dataset, Model, and Shape Recovery from a Single View* (Vol. 12363, p. 1-17). https://doi.org/10.1007/978-3-030-58523-5_1

Baltrusaitis, T., Robinson, P., & Morency, L.-P. (2016). OpenFace: An open source facial behavior analysis toolkit. *2016 IEEE Winter Conference on Applications of Computer Vision (WACV)*, 1-10. https://doi.org/10.1109/WACV.2016.7477553

Belle, V., & Papantonis, I. (2021). Principles and Practice of Explainable Machine Learning. *Frontiers in Big Data*, 4. https://www.frontiersin.org/articles/10.3389/fdata.2021.688969

Calvo, R., D'Mello, S., Gratch, J., Kappas, A., Cohn, J. F., & De La Torre, F. (2015). Automated Face Analysis for Affective Computing. In R. Calvo, S. D'Mello, J. Gratch, & A. Kappas (Éds.), *The Oxford Handbook of Affective Computing*. Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199942237.013.020

Camarena, F., Gonzalez-Mendoza, M., Chang, L., & Cuevas-Ascencio, R. (2023). An Overview of the Vision-Based Human Action Recognition Field. *Mathematical and Computational Applications*, 28(2), Article 2. https://doi.org/10.3390/mca28020061
Cañigueral, R., & Hamilton, A. F. de C. (2019). The Role of Eye Gaze During Natural Social Interactions in Typical and Autistic People. *Frontiers in Psychology, 10*, 560. https://doi.org/10.3389/fpsyg.2019.00560

Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., & Sheikh, Y. (2021). OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. *IEEE Transactions on Pattern Analysis and Machine Intelligence, 43*(1), 172-186. https://doi.org/10.1109/TPAMI.2019.2929257

Carroll, N., & Moore, M. (2008). Feeling movement: Music and Dance. *Revue internationale de philosophie, 246*(4), 413-435. https://doi.org/10.3917/rip.246.0413

Carter, B. T., & Luke, S. G. (2020). Best practices in eye tracking research. *International Journal of Psychophysiology, 155*, 49-62. https://doi.org/10.1016/j.ijpsycho.2020.05.010

Castro, M., Tillmann, B., Luauté, J., Corneyllie, A., Dailler, F., André-Obadia, N., & Perrin, F. (2015). Boosting Cognition With Music in Patients With Disorders of Consciousness. *Neurorehabilitation and Neural Repair*, 29(8), 734-742. https://doi.org/10.1177/1545968314565464

Chalençon, L., Thevenet, M., Noury, N., Bensafi, M., & Mandairon, N. (2022). Identification of new behavioral parameters to assess odorant hedonic value in humans: A naturalistic approach. *Journal of Neuroscience Methods*, 366, 109422. https://doi.org/10.1016/j.jneumeth.2021.109422

Chartrand, T. L., & Bargh, J. A. (1999). The chameleon effect: The perception-behavior link and social interaction. *Journal of Personality and Social Psychology, 76*(6), 893-910. https://doi.org/10.1037//0022-3514.76.6.893

Chartrand, T. L., & Lakin, J. L. (2013). The antecedents and consequences of human behavioral mimicry. *Annual Review of Psychology, 64*, 285-308. https://doi.org/10.1146/annurev-psych-113011-143754

Chen, H., Feng, R., Wu, S., Xu, H., Zhou, F., & Liu, Z. (2022). *2D Human Pose Estimation: A Survey* (arXiv:2204.07370). arXiv. http://arxiv.org/abs/2204.07370

Chen, H., Lundberg, S., & Lee, S.-I. (2018). *Hybrid Gradient Boosting Trees and Neural Networks for Forecasting Operating Room Data* (arXiv:1801.07384). arXiv. http://arxiv.org/abs/1801.07384
Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. *Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, 785-794.
https://doi.org/10.1145/2939672.2939785

Chen, Y., Tian, Y., & He, M. (2020). Monocular human pose estimation: A survey of deep learning-based methods. *Computer Vision and Image Understanding*, 192, 102897.
https://doi.org/10.1016/j.cviu.2019.102897

Colas, T., Farrugia, N., Hendrickx, E., & Paquier, M. (2023). Sound externalization in dynamic binaural listening: A comparative behavioral and EEG study. *Hearing Research*, 440, 108912.
https://doi.org/10.1016/j.heares.2023.108912

Dunbar, N. E., Burgoon, J. K., & Fujiwara, K. (2022). Automated Methods to Examine Nonverbal Synchrony in Dyads. *Understanding Social Behavior in Dyadic and Small Group Interactions*, 204-217.

Echeverria, J., & Santos, O. C. (2021). Toward Modeling Psychomotor Performance in Karate Combats Using Computer Vision Pose Estimation. *Sensors (Basel, Switzerland)*, 21(24), 8378.
https://doi.org/10.3390/s21248378

Ekman, P. (1993). Facial Expression and Emotion. *American Psychologist*.

Ekman, P., Freisen, W. V., & Ancoli, S. (1980). Facial signs of emotional experience. *Journal of Personality and Social Psychology*, 39(6), 1125-1134. https://doi.org/10.1037/h0077722

Ekman, P., & Friesen, W. V. (1978). *Facial Action Coding System—PsycNET*.
https://psycnet.apa.org/doiLanding?doi=10.1037%2Ft27734-000

Fawaz, H. I., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019). Deep learning for time series classification: A review. *Data Mining and Knowledge Discovery*, 33(4), 917-963.
https://doi.org/10.1007/s10618-019-00619-1

Ferrer-Mallol, E., Matthews, C., Stoodley, M., Gaeta, A., George, E., Reuben, E., Johnson, A., & Davies, E. H. (2022). Patient-led development of digital endpoints and the use of computer vision analysis in assessment of motor function in rare diseases. *Frontiers in Pharmacology*, 13, 916714. https://doi.org/10.3389/fphar.2022.916714
Gal, A., Saragosti, J., & Kronauer, D. J. C. (2020). *anTraX: High throughput video tracking of color-tagged insects* [Preprint]. Neuroscience. https://doi.org/10.1101/2020.04.29.068478

Gessa, E., Giovanelli, E., Spinella, D., Verdelet, G., Farnè, A., Frau, G. N., Pavani, F., & Valzolgher, C. (2022). Spontaneous head-movements improve sound localization in aging adults with hearing loss. *Frontiers in Human Neuroscience, 16*. https://doi.org/10.3389/fnhum.2022.1026056

Geuther, B. Q., Deats, S. P., Fox, K. J., Murray, S. A., Braun, R. E., White, J. K., Chesler, E. J., Lutz, C. M., & Kumar, V. (2019). Robust mouse tracking in complex environments using neural networks. *Communications Biology, 2*, 124. https://doi.org/10.1038/s42003-019-0362-1

Goldstein, Y., Legewie, N. M., & Shiffer-Sebba, D. (2023). 3D Social Research: Analysis of Social Interaction Using Computer Vision. *Sociological Methods & Research, 52*(3), 1201-1238. https://doi.org/10.1177/00491241221147495

González Sánchez, V., Żelechowska, A., & Jensenius, A. R. (2020). Analysis of the Movement-Inducing Effects of Music through the Fractality of Head Sway during Standstill. *Journal of Motor Behavior, 52*(6), 734-749. https://doi.org/10.1080/00222895.2019.1689909

Graving, J. M., Chae, D., Naik, H., Li, L., Koger, B., Costelloe, B. R., & Couzin, I. D. (2019). DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. *eLife, 8*, e47994. https://doi.org/10.7554/eLife.47994

Gregory, N. J., López, B., Graham, G., Marshman, P., Bate, S., & Kargas, N. (2015). Reduced Gaze Following and Attention to Heads when Viewing a « Live » Social Scene. *PLOS ONE, 10*(4), e0121792. https://doi.org/10.1371/journal.pone.0121792

Haines, N., Bell, Z., Crowell, S., Hahn, H., Kamara, D., McDonough-Caplan, H., Shader, T., & Beauchaine, T. P. (2019). Using automated computer vision and machine learning to code facial expressions of affect and arousal: Implications for emotion dysregulation research. *Development and Psychopathology, 31*(3), 871-886. https://doi.org/10.1017/S0954579419000312

Hochreiter, S., & Schmidhuber, J. (1997). Long Short-term Memory. *Neural computation, 9*, 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735
Hömke, P., Holler, J., & Levinson, S. C. (2018). Eye blinks are perceived as communicative signals in human face-to-face interaction. *PloS One*, 13(12), e0208030.
https://doi.org/10.1371/journal.pone.0208030

Iwaya, Y., Suzuki, Y., & Kimura, D. (2003). Effects of head movement on front-back error in sound localization. *Acoustical Science and Technology*, 24(5), 322-324.
https://doi.org/10.1250/ast.24.322

Jiang, Y., Pan, Z., Zhang, X., Garg, S., Schneider, A., Nevmyvaka, Y., & Song, D. (2024). *Empowering Time Series Analysis with Large Language Models: A Survey* (arXiv:2402.03182). arXiv.
https://doi.org/10.48550/arXiv.2402.03182

Jiang, Z., Harati, S., Crowell, A., Mayberg, H. S., Nemati, S., & Clifford, G. D. (2021). Classifying Major Depressive Disorder and Response to Deep Brain Stimulation Over Time by Analyzing Facial Expressions. *IEEE Transactions on Bio-Medical Engineering*, 68(2), 664-672.
https://doi.org/10.1109/TBME.2020.3010472

Jiang, Z., Luskus, M., Seyedi, S., Griner, E. L., Rad, A. B., Clifford, G. D., Boazak, M., & Cotes, R. O. (2022). Utilizing computer vision for facial behavior analysis in schizophrenia studies: A systematic review. *PLoS ONE*, 17(4), e0266828. https://doi.org/10.1371/journal.pone.0266828

Jones, P. R., Demaria, G., Tigchelaar, I., Asfaw, D. S., Edgar, D. F., Campbell, P., Callaghan, T., & Crabb, D. P. (2020). The Human Touch: Using a Webcam to Autonomously Monitor Compliance During Visual Field Assessments. *Translational Vision Science & Technology*, 9(8), 31. https://doi.org/10.1167/tvst.9.8.31

Juslin, P. N., & Sloboda, J. A. (2013). 15—Music and Emotion. In D. Deutsch (Éd.), *The Psychology of Music (Third Edition)* (p. 583-645). Academic Press. https://doi.org/10.1016/B978-0-12-381460-9.00015-8

Kingma, D. P., & Ba, J. (2017). *Adam: A Method for Stochastic Optimization* (arXiv:1412.6980). arXiv.
http://arxiv.org/abs/1412.6980

Kleinsmith, A., Bianchi-Berthouze, N., & Steed, A. (2011). Automatic Recognition of Non-Acted Affective Postures. *IEEE Transactions on Systems, Man, and Cybernetics, Part B*
(Cybernetics), 41(4), 1027-1038. IEEE Transactions on Systems, Man, and Cybernetics, Part B
(Cybernetics). https://doi.org/10.1109/TSMCB.2010.2103557

Ko, B. (2018). A Brief Review of Facial Emotion Recognition Based on Visual Information. *Sensors*, 18(2), 401. https://doi.org/10.3390/s18020401

Koul, A., Ahmar, D., Iannetti, G. D., & Novembre, G. (2023). Interpersonal synchronization of spontaneously generated body movements. *iScience*, 26(3), 106104. https://doi.org/10.1016/j.isci.2023.106104

Krishnan-Barman, S., Forbes, P. A. G., & Hamilton, A. F. de C. (2017). How can the study of action kinematics inform our understanding of human social interaction? *Neuropsychologia*, 105, 101-110. https://doi.org/10.1016/j.neuropsychologia.2017.01.018

Kunde, W., Weller, L., & Pfister, R. (2018). Sociomotor action control. *Psychonomic Bulletin & Review*, 25(3), 917-931. https://doi.org/10.3758/s13423-017-1316-6

Li, X., Fan, F., Chen, X., Li, J., Ning, L., Lin, K., Chen, Z., Qin, Z., Yeung, A. S., Li, X., Wang, L., & So, K.-F. (2021). Computer Vision for Brain Disorders Based Primarily on Ocular Responses. *Frontiers in Neurology, 12*, 584270. https://doi.org/10.3389/fneur.2021.584270

Liu, Y., Gadepalli, K., Norouzi, M., Dahl, G. E., Kohlberger, T., Boyko, A., Venugopalan, S., Timofeev, A., Nelson, P. Q., Corrado, G. S., Hipp, J. D., Peng, L., & Stumpe, M. C. (2017). *Detecting Cancer Metastases on Gigapixel Pathology Images* (arXiv:1703.02442). arXiv. http://arxiv.org/abs/1703.02442

Markus, H. (1978). The effect of mere presence on social facilitation: An unobtrusive test. *Journal of Experimental Social Psychology, 14*(4), 389-397. https://doi.org/10.1016/0022-1031(78)90034-3

Mathis, M. W., & Mathis, A. (2020). Deep learning tools for the measurement of animal behavior in neuroscience. *Current Opinion in Neurobiology*, 60, 1-11. https://doi.org/10.1016/j.conb.2019.10.008

Mnif, M., Chikh, S., & Jarraya, M. (2022). Effect of Social Context on Cognitive and Motor Behavior: A Systematic Review. *Journal of Motor Behavior*, 54(5), 631-647. https://doi.org/10.1080/00222895.2022.2060928
Morgan, E. J., Carroll, D. J., Chow, C. K. C., & Freeth, M. (2022). The Effect of Social Presence on Mentalizing Behavior. *Cognitive Science*, 46(4), e13126. https://doi.org/10.1111/cogs.13126

Murthy, G., & Jadon, R. S. (2009). *A REVIEW OF VISION BASED HAND GESTURES RECOGNITION*. https://www.semanticscholar.org/paper/A-REVIEW-OF-VISION-BASED-HAND-GESTURES-RECOGNITION-Murthy-Jadon/836dd170045a02a1d20a22c13ed89ed02b203e70

Nakano, T., Kato, M., Morito, Y., Itoi, S., & Kitazawa, S. (2013). Blink-related momentary activation of the default mode network while viewing videos. *Proceedings of the National Academy of Sciences*, 110(2), 702-706. https://doi.org/10.1073/pnas.1214804110

Nakano, T., & Kitazawa, S. (2010). Eyeblink entrainment at breakpoints of speech. *Experimental Brain Research*, 205(4), 577-581. https://doi.org/10.1007/s00221-010-2387-z

Patro, S. G. K., & Sahu, K. K. (2015). Normalization: A Preprocessing Stage. *IARJSET*, 20-22.
https://doi.org/10.17148/IARJSET.2015.2305

Paxton, A., & Dale, R. (2013). Frame-differencing methods for measuring bodily synchrony in conversation. *Behavior Research Methods*, 45(2), 329-343. https://doi.org/10.3758/s13428-012-0249-2

Paxton, A., & Dale, R. (2017). Interpersonal Movement Synchrony Responds to High- and Low-Level Conversational Constraints. *Frontiers in Psychology*, 8, 1135.
https://doi.org/10.3389/fpsyg.2017.01135

Pereira, T. D., Shaevitz, J. W., & Murthy, M. (2020). Quantifying behavior to understand the brain. *Nature Neuroscience*, 23(12), 1537-1549. https://doi.org/10.1038/s41593-020-00734-z

Perrett, S., & Noble, W. (1997). The effect of head rotations on vertical plane sound localization. *The Journal of the Acoustical Society of America*, 102(4), 2325-2332.
https://doi.org/10.1121/1.419642

Preece, S. J., Goulermas, J. Y., Kenney, L. P. J., Howard, D., Meijer, K., & Crompton, R. (2009).
Activity identification using body-mounted sensors—A review of classification techniques.
*Physiological Measurement*, 30(4), R1-33. https://doi.org/10.1088/0967-3334/30/4/R01
Risko, E. F., Richardson, D. C., & Kingstone, A. (2016). Breaking the Fourth Wall of Cognitive Science: Real-World Social Attention and the Dual Function of Gaze. *Current Directions in Psychological Science*, 25(1), 70-74. https://doi.org/10.1177/0963721415617806

Robie, A. A., Seagraves, K. M., Egnor, S. E. R., & Branson, K. (2017). Machine vision methods for analyzing social interactions. *Journal of Experimental Biology*, 220(1), 25-34. https://doi.org/10.1242/jeb.142281

Rodriguez, A., Zhang, H., Klaminder, J., Brodin, T., Andersson, P. L., & Andersson, M. (2018). ToxTrac: A fast and robust software for tracking organisms. *Methods in Ecology and Evolution*, 9(3), 460-464. https://doi.org/10.1111/2041-210X.12874

Rogers, S. L., Speelman, C. P., Guidetti, O., & Longmuir, M. (2018). Using dual eye tracking to uncover personal gaze patterns during social interaction. *Scientific Reports*, 8(1), 4271. https://doi.org/10.1038/s41598-018-22726-7

Shen, X., Fan, G., Niu, C., & Chen, Z. (2021). Catching a Liar Through Facial Expression of Fear. *Frontiers in Psychology*, 12, 675097. https://doi.org/10.3389/fpsyg.2021.675097

Siegle, G. J., Ichikawa, N., & Steinhauer, S. (2008). Blink before and after you think: Blinks occur prior to and following cognitive load indexed by pupillary responses. *Psychophysiology*, 45(5), 679-687. https://doi.org/10.1111/j.1469-8986.2008.00681.x

Sigurdsson, G. A., Russakovsky, O., & Gupta, A. (2017). What Actions are Needed for Understanding Human Actions in Videos? *2017 IEEE International Conference on Computer Vision (ICCV)*, 2156-2165. https://doi.org/10.1109/ICCV.2017.235

Sigurdsson, G. A., Varol, G., Wang, X., Farhadi, A., Laptev, I., & Gupta, A. (2016). Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding. In B. Leibe, J. Matas, N. Sebe, & M. Welling (Éds.), *Computer Vision – ECCV 2016* (Vol. 9905, p. 510-526). Springer International Publishing. https://doi.org/10.1007/978-3-319-46448-0_31

Sridhar, V. H., Roche, D. G., & Gingins, S. (2019). Tracktor: Image-based automated tracking of animal movement and behaviour. *Methods in Ecology and Evolution*, 10(6), 815-820. https://doi.org/10.1111/2041-210X.13166
Stein, N., Niehorster, D. C., Watson, T., Steinicke, F., Rifai, K., Wahl, S., & Lappe, M. (2021). A Comparison of Eye Tracking Latencies Among Several Commercial Head-Mounted Displays. *i-Perception, 12*(1), 2041669520983338. https://doi.org/10.1177/2041669520983338

Sundararajan, M., Taly, A., & Yan, Q. (2017). *Axiomatic Attribution for Deep Networks* (arXiv:1703.01365). http://arxiv.org/abs/1703.01365

Sweeny, T. D., & Whitney, D. (2014). Perceiving crowd attention: Ensemble perception of a crowd's gaze. *Psychological Science*, 25(10), 1903-1913. https://doi.org/10.1177/0956797614544510

Tjoa, E., & Guan, C. (2021). A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI. *IEEE Transactions on Neural Networks and Learning Systems*, 32(11), 4793-4813. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/TNNLS.2020.3027314

Tsuchiya, A., Ora, H., Hao, Q., Ono, Y., Sato, H., Kameda, K., & Miyake, Y. (2020). Body Movement Synchrony Predicts Degrees of Information Exchange in a Natural Conversation. *Frontiers in Psychology, 11*, 817. https://doi.org/10.3389/fpsyg.2020.00817

Varlet, M., Marin, L., Lagarde, J., & Bardy, B. G. (2011). Social postural coordination. *Journal of Experimental Psychology: Human Perception and Performance*, 37(2), 473-483. https://doi.org/10.1037/a0020552

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). *Attention Is All You Need* (arXiv:1706.03762; Version 1). http://arxiv.org/abs/1706.03762

Voulodimos, A., Doulamis, N., Doulamis, A., & Protopapadakis, E. (2018). Deep Learning for Computer Vision: A Brief Review. *Computational Intelligence and Neuroscience*, 2018, e7068349. https://doi.org/10.1155/2018/7068349

Wagner, H. L., & Smith, J. (1991). Facial expression in the presence of friends and strangers. *Journal of Nonverbal Behavior*, 15(4), 201-214. https://doi.org/10.1007/BF00986922

Walther, D. B., Farzanfar, D., Han, S., & Rezanejad, M. (2023). The mid-level vision toolbox for computing structural properties of real-world images. *Frontiers in Computer Science*, 5. https://www.frontiersin.org/articles/10.3389/fcomp.2023.1140723
Wong, W. Y., Wong, M. S., & Lo, K. H. (2007). Clinical Applications of Sensors for Human Posture and Movement Analysis: A Review. *Prosthetics and Orthotics International, 31*(1), 62-75.
https://doi.org/10.1080/03093640600983949

Yu, Y., Mora, K. A. F., & Odobez, J.-M. (2017). Robust and Accurate 3D Head Pose Estimation through 3DMM and Online Head Model Reconstruction. *2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017)*, 711-718.
https://doi.org/10.1109/FG.2017.90

Zeghari, R., König, A., Guerchouche, R., Sharma, G., Joshi, J., Fabre, R., Robert, P., & Manera, V. (2021). Correlations Between Facial Expressivity and Apathy in Elderly People With Neurocognitive Disorders: Exploratory Study. *JMIR Formative Research, 5*(3), e24727.
https://doi.org/10.2196/24727

Zhang, Y., Weng, Y., & Lund, J. (2022). Applications of Explainable Artificial Intelligence in Diagnosis and Surgery. *Diagnostics, 12*(2), 237. https://doi.org/10.3390/diagnostics12020237