# A Distance-Based Classification Method to Assess Frontal Behavior from Human Behavioral Sensing

Bénédicte Batrancourt, Frédéric Marin, Caroline Peltier, François Xavier Lejeune, Delphine Tanguy, Valérie Godefroy, Idil Sezer, Mathilde Boucly, David Bendetowicz, Guilhem Carle, et al.

► To cite this version:

Bénédicte Batrancourt, Frédéric Marin, Caroline Peltier, François Xavier Lejeune, Delphine Tanguy, et al.. A Distance-Based Classification Method to Assess Frontal Behavior from Human Behavioral Sensing. 2, Noldus Information Technology bv, pp.95-101, 2022, Proceedings of the 12. International Conference on Methods and Techniques in Behavioral Research, and 6. Seminar on Behavioral Methods, 18-20 May 2022, 978-90-74821-94-0. 10.6084/m9.fishshare.20066849. hal-03749408

HAL Id: hal-03749408

https://hal.inrae.fr/hal-03749408

Submitted on 4 Aug 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Measuring Behavior 2022

Volume 2

12th International Conference on Methods and Techniques in
Behavioral Research, and 6th Seminar on Behavioral Methods

18 – 20 May 2022. Online.

Proceedings

<figure><img src="image_2.png" /></figure>
**Volume Editors**

**Andrew Spink**

Noldus Information Technology; [Andrew.Spink@noldus.nl](mailto:Andrew.Spink@noldus.nl)

**Jarosław Barski**

Medical University of Silesia, Katowice; [jbarski@sum.edu.pl](mailto:jbarski@sum.edu.pl)

**Anne-Marie Brouwer**

Perceptual and Cognitive Systems, TNO; [anne-marie.brouwer@tno.nl](mailto:anne-marie.brouwer@tno.nl)

**Gernot Riedel**

University of Aberdeen; [g.riedel@abdn.ac.uk](mailto:g.riedel@abdn.ac.uk)

**Annesha Sil**

University of Aberdeen; [annesha.sil@abdn.ac.uk](mailto:annesha.sil@abdn.ac.uk)
# A Distance-Based Classification Method to Assess Frontal Behavior from Human Behavioral Sensing

Bénédicte Batrancourt¹, Frédéric Marin², Caroline Peltier³, François-Xavier Lejeune¹, Delphine Tanguy¹, Valérie Godefroy¹, Idil Sezer¹, Mathilde Boucly¹, David Bendetowicz¹, Guilhem Carle¹, Armelle Rametti-Lacroux¹, Raffaella Migliaccio¹,⁴ and Richard Levy¹,⁴

1. Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, Inserm, CNRS, AP-HP, Hôpital de la Pitié Salpêtrière, Paris, France, benedicte.batrancourt@upmc.fr

2. Centre of Excellence for Human and Animal Movement Biomechanics (CoEMoB), Laboratoire de BioMécanique et BioIngénierie (UMR CNRS 7338), Université de Technologie de Compiègne (UTC), Alliance Sorbonne Université, 60200 Compiègne, France.

3. Centre des Sciences du Goût et de l'Alimentation (CSGA), ChemoSens Platform, AgroSup Dijon, CNRS, INRAE, University of Bourgogne Franche-Comté, PROBE research infrastructure, Dijon, France.

4. AP-HP, Hôpital de la Pitié Salpêtrière, Department of Neurology, Center of excellence of neurodegenerative disease (CoEN), Institute of Memory and Alzheimer's Disease (IM2A), F-75013, Paris, France.

## Introduction

### Frontal Behavior

The frontal lobe of the brain, and particularly the prefrontal cortex (PrFC) are linked to the more complex aspects of human behavior. PrFC supports goal-directed behaviors (GDB) and is involved in cognitive and behavioral aspects of adaptation to complex or new situations. Damage to the lateral part of the PrFC is associated with deficits in planning, and damage to the ventral part of the PrFC with motivation disorders. Thus, patients with prefrontal damage show deficits in the ability to plan and organize behavior

### Apathy

Behavioral variant of frontotemporal dementia (bvFTD) is a neurodegenerative syndrome characterized by cognitive and behavioral decline caused by progressive atrophy of frontal and temporal regions. Apathy is almost universal in bvFTD, but it is also a pervasive neuropsychiatric symptom of most neurocognitive, neurodegenerative, and psychiatric disorders [1]. Traditionally, apathy has been viewed as a symptom indicating loss of interest or emotions. In 1990, in a highly influential conceptual framework, Marin defined apathy as diminished motivation not attributable to diminished level of consciousness, cognitive impairment, or emotional distress [2]. Marin proposed diagnostic criteria for the syndrome of apathy on the basis of its distinction from the overt behavioral, cognitive, and emotional concomitants of goal-directed behavior [3]. In 2006, in another influential theoretical framework, Levy and Dubois refined the definition of apathy to the quantitative reduction of self-generated voluntary and purposeful behaviors [4]. Consequently, the authors argued that, first, apathy is an observable state that can subsequently be quantified; second, apathy is a pathology of voluntary action or goal-directed behavior (GDB); and third, the underlying mechanisms responsible for apathy are related to dysfunctions of the elaboration, execution or control of GDB.

### Assessment of Neuropsychiatric Symptoms (NPS)

NPS normally occur during the natural course of dementia [5] and include symptoms such as apathy and disinhibition. NPS assessment is crucial in clinical practice as well as in clinical research and also in future clinical trials targeting disease-modifying therapies [6]. Validated assessment scales are available for the majority of these symptoms. Apathy is usually assessed by questionnaires administered to the patient and/or caregiver and providing information about the patient's internal state, thoughts and past activities, globally suggesting a loss of motivation
to perform daily activities [7]. However, these scales are biased by the subjective nature of the patient or
caregiver's perspective. More generally, complex behaviours and their disorders may be difficult to capture
through questionnaires and assessment scales, and would be more easily identified through an ecological
observation. This raises the question of the limitations of current neuropsychological assessments and tests. Two
sources of difficulty are traditionally identified in this area of research: the nature of the clinical material, and the
complexity of the behavioral deficits [8].

## Behavioral Sensing

To address the specific question concerning the limitations of the assessment of NPS using interviews and rating scales, and for supplementing the patient's subjective evaluation of health state, we rely on behavioral sensing, an emerging and promising behavioral research field, due to the rapid growth of wearable and/or wireless sensors, as well as devices smart-sensor integration in mobile phones. Recent studies, demonstrated the relevance to use new mobile technologies and wearable sensors for the assessment of the behavior in neurological conditions. Mobile technologies can provide objective and frequent measurements of disease [9]. Major of these studies have focused on the Parkinson disease (PD) to quantitatively capture movement patterns. Mobile Phone and wearable sensor have also opened the prospect of access to psychiatric disorders and symptoms, allowing the collection of quantitative behavioral and functional markers, providing an estimation of physiological and mental state [10]. The 2018 international consensus group of experts, in the domain of apathy in brain disorders stated that apathy can be assessed through new information and communication technologies (ICT), and that these new ICT approaches could provide clinicians with valuable additional information in terms of assessment, and therefore more accurate diagnosis of apathy [1].

## ECOCAPTURE Program

In line with these limitations and considerations, we developed the ECOCAPTURE program (FRONTlab, ICM) designed to identify and measure behavior and/or behavioral disorders to obtain objective and quantitative measurements for assessing neuropsychiatric symptoms, such as apathy [11] and disinhibition [12]. This broad research program is characterized by its original methodological approach using behavioral sensing under ecological conditions. The ECOCAPTURE program is divided into two main projects : 1/ the ECOCAPTURE@LAB study (Clinicaltrials.gov: NCT02496312, NCT03272230) aims to identify behavioral signatures of apathy under controlled conditions (laboratory setting); 2/ the ECOCAPTURE@HOME study (Clinicaltrials.gov: NCT04865172) aims to identify and measure behavioral markers of apathy in everyday life conditions, and predict the psychological status of the patient-caregiver dyad from these markers of apathy. The final goal of the ECOCAPTURE program is to contribute to the development of new therapeutic strategies, such as non-pharmacological interventions (NPI), targeting apathy.

### ECOCAPTURE@LAB paradigm

ECOCAPTURE@LAB explores participant's behavior in a close-to-real-life situation (waiting room) under controlled conditions (the ECOCAPTURE scenario). The experiments took place on the ICM's core facility (PRISME) [13] dedicated to the functional exploration of human behavior. The PRISME platform was equipped with a six-ceiling camera system (not hidden) covering the entire waiting room. The subjects were informed at the time of initial consent that their behavior would be tracked and recorded by video cameras located in the room. The subject's behavior was recorded for a 45-minutes period using a multimodal behavioral sensing system consisting of video recording, 3D-accelerometer (Move II®, Movisens) and eye-tracking glasses (ETG 2w®, SMI). The accelerometer was fixed at the subject's right hip. Participants wore ETG for only a 7-minutes period.

The waiting room contains specific objects that provide opportunities for subjects to interact with the environment and pass the time (games, magazines, food and drink, furniture such as a sofa, chairs, table, etc.). The subject is explicitly encouraged to make himself/herself comfortable and to enjoy the room, using the space, as well as the objects at his or her own convenience ("as if he/she was at home"). These guidelines are designed to promote the ecological validity of the behavioral measurements. The main phases of the ECOCAPTURE scenario are a 7-minute free phase (FP) and a 10-minute guided phase (GP). FP is a freely moving phase, during which the
participant is explicitly encouraged to explore the room. Since no specific goal-directed activity is suggested by the examiner, the participants are mostly tested on their ability to self-initiate activities and produce self-guided behavior. In contrast by the free phase FP, GP is an externally guided phase, in which the participants are asked by the examiner to complete a questionnaire. The ECOCAPTURE scenario is relevant to the study of apathy because it favors the generation of GDB under contrasting conditions and offers many different opportunities to investigate the patient's behavior. In one of our previous studies [11], we analyzed the video-based behavioral data for 14 bvFTD patients and 14 healthy controls (HC), and provided first information about the behavioral signature of apathy. In this previous work, we reported an exploration deficit in bvFTD patients. We showed that, during the very first minutes, when they discovered the room, the bvFTD patients manifested more inactivity and less exploratory behavior than the HCs. Hence, exploratory behavior deficits under ecological conditions could be a marker of apathy in bvFTD.

# Materials and Methods

## Participants

A total of twenty bvFTD patients (13 men, 7 women) were recruited through neurological consultations at two AP-HP (Paris Public Hospitals) expert clinical sites: the national reference center on FTD at the Institut de la Mémoire et de la Maladie d'Alzheimer (IM2A) at the Pitié-Salpêtrière Hospital and at the Lariboisière Fernand-Widal Hospital. Eighteen healthy controls (HCs) were recruited by public announcement. HC subjects were matched to patients for age, gender and education level. Exclusion criteria for all of the participants included current or prior history of neurological disease other than bvFTD, psychiatric disease, and drug abuse. The participants in the ECOCAPTURE cohort underwent the ECOCAPTURE@LAB paradigm and a comprehensive neuropsychological assessment.

## Frontal Behavior Sensing (FBS)

We develop a novel distance-based classification method (FBS) for identifying behavioral signatures from video and sensor-based data (behavioral sensing) to assess frontal behavior symptoms and especially apathy. The FBS method, from behavioral sensing data to the behavioral signature of NPS, will proceed in three main steps: 1/ behavioral sensing data collection, 2/ high-dimensional time-series matrices encoding, 3/ time-series matrices processing (see Figure 1).

<figure><img src="image_3.png" /><figcaption>Figure 1. The overall FBS method: from behavioral sensing data to the behavioral signature of NPS.</figcaption></figure>

## Behavioral sensing data collection

The behavioral sensing data are composed of: 1/ the video-based behavioral data, 2/ accelerometer-based data (intensity of acceleration in three mutually orthogonal directions with a sample rate of 64 Hz), 3/ the eye-tracking
glasses-based data (saccadic frequency and amplitude). Video-based behavioral data were obtained by behavioral coding from 45-minute video footage for each individual (Figure 2). Behavioral coding data were collected through the continuous sampling method (all occurrences of behaviors and their duration were recorded) using NOLDUS The Observer XT (Version 14.0). Behavioral coding was conducted based on the ECOCAPTURE apathy ethogram [14]. The ECOCAPTURE ethogram includes two behavioral categories: *motor patterns* and *activity states*, focusing on behaviors exhibited by the subjects during the scenario. The motor patterns category describes the posture, as well as the body segment movements and locomotion, expressed by the observed individuals (e.g., sitting). The activity states category includes four behaviors: 1) nonactivity, a state in which the subject shows no apparent activity; 2) activity, a state in which the subject is engaged in an activity with sustained attention; 3) exploration, a state in which the subject explores the waiting room and various objects in the room; and 4) transition, focusing on the timing of transitions between states. Moreover, modifiers are used to strongly describe and identify the nature of the activity, as well as the exploratory behavior. The modifiers correspond to items present in the environment (the waiting room) with which the subject could interact (e.g., books and magazines, food and drink). All of the behaviors included in each of these two categories are mutually exclusive (e.g., sitting and standing cannot occur concurrently, nor can activity and nonactivity).

<figure><img src="image_4.png" /><figcaption>Figure 2. Video-based behavioral data. Example of bvFTD patient ethogram data. Sequence of each state behavior from the two categories: *activity states* (in red: nonactivity; in blue: exploration; in yellow: activity – playing games; in green: transition), and *motor patterns* (in dark green: walking; in magenta: sitting; in cyan: standing).</figcaption></figure>

## High-dimensional time-series matrices encoding

The second step is to encode the multimodal behavioral sensing data in high-dimensional time-series matrices. The output of this step will be a behavioral sensing matrix per subject, over a 45 minutes period, with a sample rate of 1 Hz. Video-based behavioral data will be encoded as binary vectors (rows of the matrix), to indicate the absence or presence of any given category of the ethogram (e.g. activity, exploration). The acceleration intensity and saccadic eye movement frequency, will be encoded as another vectors.

## Time-series matrices processing

The last step is to compute the matrices using: 1/ a multivariate distance matrix regression (MDMR) method [15] to identify predictor variables (collected on the samples to be related to variation in the pairwise similarity/distance values reflected in the matrix) and so predict the variability among behavioral sensing matrices in function of the NPS characteristics, 2/ a graph clustering approach (symmetric non-negative matrix factorization, Sym-NMF [16]) for classification of matrices to identify derived data-driven latent behavioral patterns and signatures of NPS.

## First implementation of the FBS method

In one of our previous studies [17], we developed a first implementation of the FBS method, according the three required steps. The behavioral sensing data were composed of the video-based behavioral data collected during the 7-minute free phase for each individual in a group of 20 patients with bvFTD and a group of 18 healthy controls.
These collected behavioral data were encoded in so-called Subject's behavioral matrices (SBMs) composed of p binary vectors of size n, with p (number of behaviors of interest) rows and n (number of timepoints) columns containing 1 if the behavior is realized at the time point or 0 otherwise. Establishing a distance between such matrices was required to allow for the classification of subjects considering temporality. To address this issue, we used temporal classification for behavior time series data analysis. To develop our classifier, we retained a nonelastic Euclidian metric, combined with a convolutional approach. Finally, the bvFTD patients (i.e. SMBs) were classified according to the chosen metric, and the identified subgroups of patients were described and then characterized by behavioral curves and neuropsychological features.

## Results

We showed that bvFTD patients can be classified according to their behavioral kinetics into three groups (Figure 3C). Three subgroups of bvFTD patients were identified with different behavioral kinetics as well as neuropsychological profiles.

<figure><img src="image_5.png" /><figcaption>Figure 3. Extracting subjects' behavioral matrices (SBMs) from temporal behavior data resulted in a motor bandplot (A) and an activity bandplot (Figure B). (C) Kinetics in the 3 selected groups of bvFTD patients and in the HC group. The time diagrams include the signal throughout the FP for each behavior manifested in a particular group.</figcaption></figure>

## Conclusion

Using multimodal behavioral sensing to build a high-dimensional behavioral sensing and time-series matrix per subject, we expect to identify predictor variables with variance among behavioral matrices are explained by NPS characteristics, and derived data-driven latent behavioral patterns and signatures of specific NPS. In the future, the FBS method will be applied on the full ECOCAPTURE data (video and sensor-based data collected during the whole 45-minute scenario) to identify behavioral signatures of apathy.

## Ethical statement

This study is part of the clinical observational study C16-8739 sponsored by INSERM, the French National Institute for Biomedical Research. It was granted approval by the local Ethics Committee (Comité de Protection des Personnes, CPP) on May 17, 2017 (CPP 17-31), and was registered in a public clinical trial registry (Clinicaltrials.gov: NCT02496312, NCT03272230). All of the study participants gave their written informed
Funding

Part of this work was funded by grants from the ENEDIS company (ERDF), 2015-2017, and from the foundation "Fondation pour la recherche médicale", FRM DEQ20150331725. The research leading to these results has received funding from the program "Investissements d'avenir", ANR-10-IAIHU-06. The funders played no role in the study design, access to the data, or writing of this report.

References

1. Robert, P., Lanctôt, K.L., Agüera-Ortiz, L., Aalten, P., Bremond, F., Defrancesco, M., Hanon, C., David, R., Dubois, B., Dujardin, K., Husain, M., König, A., Levy, R., Mantua, V., Meulien, D., Miller, D., Moebius, H.J., Rasmussen, J., Robert, G., Ruthirakuhan, M., Stella, F., Yesavage, J., Zeghari, R., Manera, V. (2018). Is it time to revise the diagnostic criteria for apathy in brain disorders? The 2018 international consensus group. *Eur. Psychiatry J. Assoc. Eur. Psychiatr.* **54**, 71–76.

2. Marin, R.S. (1990). Differential diagnosis and classification of apathy. *Am. J. Psychiatry* **147**, 22–30.

3. Marin, R.S. (1991). Apathy: a neuropsychiatric syndrome. *J. Neuropsychiatry Clin. Neurosci.* **3**, 243–254.

4. Levy, R., Dubois, B. (2006). Apathy and the Functional Anatomy of the Prefrontal Cortex–Basal Ganglia Circuits. *Cereb. Cortex* **16**, 916–928.

5. Steinberg, M., Shao, H., Zandi, P., Lyketsos, C.G., Welsh-Bohmer, K.A., Norton, M.C., Breitner, J.C.S., Steffens, D.C., Tschanz, J.T., Cache County Investigators (2008). Point and 5-year period prevalence of neuropsychiatric symptoms in dementia: the Cache County Study. *Int. J. Geriatr. Psychiatry* **23**, 170–177.

6. David, R., Mulin, E., Mallea, P., Robert, P.H. (2010). Measurement of Neuropsychiatric Symptoms in Clinical Trials Targeting Alzheimer's Disease and Related Disorders. *Pharm. Basel Switz.* **3**, 2387–2397.

7. Mohammad, D., Ellis, C., Rau, A., Rosenberg, P.B., Mintzer, J., Ruthirakuhan, M., Herrmann, N., Lanctôt, K.L. (2018). Psychometric Properties of Apathy Scales in Dementia: A Systematic Review. *J. Alzheimers Dis.* **66**, 1065–1082.

8. Mesulam, M.M. (1986). Frontal cortex and behavior. *Ann. Neurol.* **19**, 320–325.

9. Dorsey, E.R., Glidden, A.M., Holloway, M.R., Birbeck, G.L., Schwamm, L.H. (2018). Teleneurology and mobile technologies: the future of neurological care. *Nat. Rev. Neurol.* **14**, 285–297.

10. Seppälä, J., De Vita, I., Jämsä, T., Miettunen, J., Isohanni, M., Rubinstein, K., Feldman, Y., Grasa, E., Corripio, I., Berdun, J., D'Amico, E., M-RESIST Group, Bulgheroni, M. (2019). Mobile Phone and Wearable Sensor-Based mHealth Approaches for Psychiatric Disorders and Symptoms: Systematic Review. *JMIR Ment. Health* **6**, e9819.

11. Batrancourt, B., Lecouturier, K., Ferrand-Verdejo, J., Guillemot, V., Azuar, C., Bendetowicz, D., Migliaccio, R., Rametti-Lacroux, A., Dubois, B., Levy, R. (2019). Exploration Deficits Under Ecological Conditions as a Marker of Apathy in Frontotemporal Dementia. *Front. Neurol.* **10**, 941.

12. Godefroy, V., Tanguy, D., Bouzigues, A., Sezer, I., Ferrand-Verdejo, J., Azuar, C., Bendetowicz, D., Carle, G., Rametti-Lacroux, A., Bombois, S., Cognat, E., Jannin, P., Morandi, X., Ber, I.L., Levy, R., Batrancourt, B., Migliaccio, R. (2021). Frontotemporal dementia subtypes based on behavioral inhibition deficits. *Alzheimers Dement. Diagn. Assess. Dis. Monit.* **13**, e12178.
13. "PRISME: human behavior exploration core facility," *Institut du Cerveau*. (2022).
<https://institutducerveau-icm.org/fr/prisme-human-behavior-exploration-core-facility>.

14. Batrancourt, B., Migliaccio, R. Tanguy, D., Sezer, I., Godefroy, V., Bouzigues, A. (2022). The ECOCAPTURE ethograms: apathy ethogram and disinhibition ethogram. Mendeley Data V2. <https://doi.org/10.17632/mv8hndcd95/2>.

15. Zapala, M.A., Schork, N.J. (2006). Multivariate regression analysis of distance matrices for testing associations between gene expression patterns and related variables. *Proc. Natl. Acad. Sci. U. S. A.* **103**, 19430–19435.

16. Kuang, D., Ding, C., Park, H. (2012). Symmetric nonnegative matrix factorization for graph clustering. *Proceedings of the 2012 SIAM International Conference on Data Mining (SDM)* (Anaheim, California, USA).

17. Peltier, C., Lejeune, F.X., Jorgensen, L. G. T., Rametti-Lacroux, A., Tanguy, D., Godefroy, V., Bendetowicz, D., Carle, G., Cognat, E., Bombois, S., Migliaccio, R., Levy, R., Marin, F., Batrancourt, B., ECOCAPTURE study group. (In review). A temporal classification method based on behavior time series data in patients with behavioral variant of frontotemporal dementia and apathy. *Journal of Neuroscience Methods*.