<figure><img src="image_1.png" /></figure>

Inria/IIT DELHI
WORKSHOP

October 25, 2023

# Digital Science for Disability

## Overview of the initiatives undertaken by Inria

Christophe Biernacki, deputy scientific director
General overview

Researchs involved in disability initiatives @Inria

<figure><img src="image_2.png" /></figure>
# Inria

* French National Research Institute for Digital Sciences and Technologies

* "Scientific Excellence and Transfer"

* Computer Science and Applied mathematics

<figure><img src="image_3.png" /><figcaption>**DATA SCIENCE & KNOWLEDGE ENGINEERING**</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**MODELING & SIMULATION**</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**OPTIMIZATION & CONTROL**</figcaption></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /><figcaption>**ARCHITECTURE, SYSTEMS & NETWORKS**</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**SECURITY & PRIVACY**</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**INTERACTION & MULTIMEDIA**</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**ARTIFICIAL INTELLIGENCE & AUTONOMOUS SYSTEMS**</figcaption></figure>
## Disability

It is associated with a very strong heterogeneity between people, therefore no single solution

We must remain vigilant in the face of stereotypes! Example: “disabled people reject/love technology”

## Objectives of our research

- Improving the quality of life of people with disabilities

- Increase their autonomy

- Respect and develop their self-determination
## Disability@Inria

About 20 teams have an activity in link with disability

## Gait and posture

- Avoiding "technological solutionism"

- Involve stakeholders upstream and downstream

- Take into account constraints (costs, logistics, intrusiveness)

- Take ethical aspects into account

- Be concerned about the impact: effective implementation, startups as a preferred vector of transfer
**Diversity of topics involved in numerical science**

Machine learning, Signal processing, Control theory, Automatic language processing, Medical imaging, Sensors, Robotics, Human Machine Interface, Cryptographic protocols...

**To keep in mind for having impact**

Absolute need for multidisciplinary approaches

Work on needs, practical constraints, acceptability, etc.

**Despite several projects, area still underinvested...**
**FLOWERS** A set of educational technologies to promote inclusion for children with Autism Spectrum Disorder (ASD)

**ALMANACH** Develop Cap'FALC, a text simplification algorithm, for people suffering from disabilities and for non-native speakers learning a given language

**CAMIN** Neuroprosthesis user interface based on residual motor skills and muscle activity in persons with upper limb disabilities

**DEFROST** Developing wearable soft exoskeletons with variable damping or stiffness for helping disable people (new soft exoskeletons that may become hard at the period of supporting weight and transferring power, and then become soft at the working mode)

**EMPENN** Study the impact of lesion damages along the whole motor pathways on disability in multiple sclerosis

Teams
**HEPHAISTOS** Robotized devices for assistance, including smart objects, that may aid disabled or elderly people in their personal life

**HYBRID** Assistive Robotic Technologies for Next-Generation Smart Wheelchairs: Codesign and Modularity to Improve Users' Quality of Life

**MFX, POTIOC, RAINBOW** The DORNELL project as a multimodal, shapeable haptic handle for mobility assistance of people with disabilities

**MIND** Project LargeSmallBrainNets on the learning process for children with mathematical disabilities

**MULTISPEECH** Voice assistants tool for senior or disabled people, especially for those who are unable to use other interfaces due to lack of hand dexterity, mobility, and/or good vision

Teams
**POTIOC** Accessible applications targeted at users with special needs including people with sensori-motor and/or cognitive disabilities (personalization with Artificial Intelligence)

**RAINBOW** Using cutaneous haptics for helping the navigation of people with sensory disabilities (vibrotactile sensations, tap stimulations, sensory illusions)

Teams
NEURINNOV

Control of movements of paraplegic patients

Sensory feedback: above-knee amputees

Challenges: coordinated stimulation to restore function or regain sensation

Startup

<figure><img src="image_11.png" /><figcaption>CYBATHLON 2016</figcaption></figure>

<figure><img src="image_12.png" /></figure>
Ullo

Treatment of anxiety,...

Solutions combining augmented reality, sensory stimulation and
breathing exercises

Tested then deployed within EHPADs (elderly people)

<figure><img src="image_13.png" /></figure>

Bleu = océan,
Jaune = sable,
Vert = prairie,

<figure><img src="image_14.png" /></figure>

Startup
## HumanLab

Collaborative digital manufacturing or repair spaces of objects, open to people with a disability to appropriate the technology for their own use.

Complementary with the equipment and a medical approach not the replace !

## HumanLab@Inria

Create a model of collaboration between Inria and the network of HumanLab

Contribute to meeting the needs expressed by individuals people with disabilities

Frugal innovation approach, open-source

Open collaboration

<figure><img src="image_15.png" /></figure>
## Focus n°1

Electrically stimulate the system peripheral nerve to restore lost motor functions

<figure><img src="image_16.png" /></figure>
## Electrically stimulate nervous tissues to

Explore & model the neurophysiological mechanisms

Assist and restore lost functions (gripping, standing, walking, etc.)

## Methods

Surface or implanted electrodes

Experiments in humans and animals

<figure><img src="image_17.png" /></figure>
# Functional Electrical Stimulation (FES)

Artificially trigger a muscle contraction

<figure><img src="image_18.png" /></figure>
# Application of SEF to restoration of grip

## Approach 1 - Surface electrical stimulation

But surface electrodes are not very selective, tedious positioning, movements of the skin during contractions...

## Solution 1 - Matrix electrodes: selective

Simplified positioning and adjustments, rehabilitation / replacement (control methods)

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>
# Application of SEF to restoration of grip

## Approach 2 - Implanted electrical stimulation

## Solution 2 - implanted electrodes

Selectivity and increased efficiency, stability / invasiveness

<figure><img src="image_22.png" /></figure>
Basic functional movements

Implantation of 2 electrodes for 28 days

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>
Focus n°2

Video-based identification of epileptic seizures

STARS

<figure><img src="image_26.png" /></figure>

UNIVERSITÉ
CÔTE D'AZUR

<figure><img src="image_27.png" /></figure>

Aix*Marseille
université
Initiative d'excellence

<figure><img src="image_28.png" /></figure>

Inserm
Goal

* Classify epileptic seizures and non-epileptic seizures based on videos.

<figure><img src="image_29.png" /></figure>
# What are epileptic seizures (ES)

* Abnormal behavior associated with motor activity changes, including repeated or rhythmic movements.

* Caused by epileptic/excessive neuronal activity in the brain.

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

Depth electrode
Examples of epileptic seizures

* Limb dystonia: rigid/stiff arms

<figure><img src="image_33.png" /></figure>
# Non-epileptic seizures

* Not all seizures are epileptic.

* Psychogenic non-epileptic seizures (PNES)

* Caused by psychological factors

<figure><img src="image_34.png" /></figure>
# A Multi-Stream Approach for Seizure Classification with Knowledge Distillation:
## Overview of the model

J-C Hou, A Mcgonigal, F Bartolomei, M Thonnat. A Multi-Stream Approach for Seizure Classification with Knowledge Distillation. AVSS 2021

* Goal: A binary classification task for ES vs. PNES

* Caused by different reasons

* Different follow-up treatments

<figure><img src="image_35.png" /></figure>

*TCN: Temporal Convolutional Network

*AGCN: Adaptive Graph Convolutional Network [Shi, CVPR 2019]
Challenges: limited labeled data

* Costly to collect labeled data in medical domains

* Video dataset for seizure type classification:

<table><thead><tr><td>Study</td><td># of seizure videos</td><td># of patients</td></tr></thead><tbody><tr><td>Achilles (2016)</td><td>52</td><td>10</td></tr><tr><td>Aristizabal (2018)</td><td>52 (MTLE:40, ETLE:12)</td><td>18 (MTLE:12, ETLE:6)</td></tr><tr><td>Karácsony (2020)</td><td>126 (FLE:85, TLE:41)</td><td>35 (FLE:20, TLE:15)</td></tr><tr><td>Maia (2019)</td><td>143 (ETLE:107, TLE:36)</td><td>31</td></tr><tr><td>Aristizabal (2019)</td><td>161 (MTLE:90, ETLE:71)</td><td>34 (MTLE:17, ETLE:17)</td></tr><tr><td>Ours</td><td>283 (ES:235, PNES:48)</td><td>81 (ES:52, PNES:29)</td></tr></tbody></table>

*MTLE: mesial temporal lobe epilepsy, FLE: frontal lobe epilepsy, ETLE, extra-temporal lobe epilepsy

* Action recognition dataset (#Videos): Kinetics (500k), VLOG (114k), HACS (520k)

* Utilize large amount of unlabeled data.

* Without the need for data annotation from clinicians

* Easier to access and collect.
Focus n°3

ADAPT: Wheelchair navigation assistance

<figure><img src="image_36.png" /></figure>
# ADAPT : Wheelchair navigation assistance

## Intuitive trajectory correction

* An embedded plug & play kit providing reactive collision avoidance.

* Shared Control with the user.

<figure><img src="image_37.png" /></figure>

## How far have we come?

* **Clinical investigations:** Scenarios of progressive difficulty co-designed with clinicians (+ ethical requirements, compliance with medical device regulation...)

* Experiments "in real-life": testing in the museums of Rennes.

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

Musée
de Bretagne
lesChampsLibres

<figure><img src="image_42.png" /></figure>

MUSÉE
DES BEAUX-ARTS
DE RENNES

Around 350 users or people who had failed to drive a wheelchair tested the system, with 100+ participants included in the clinical investigations.
## Focus n°4

DORNELL: A multimodal, shapeable haptic handle for mobility assistance of people with visual impairments

Rainbow

<figure><img src="image_43.png" /></figure>

Matter from graphics

potioc
popular interaction

Voir ensemble

<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>

ASSOCIATION
SAINT-HÉLIER
Au cœur des territoires
# DORNELL: Increase mobility

## An innovative haptic handle for efficient and safe navigation

* A multimodal haptic and tactile feedback

> about the surrounding environment.

> about the interaction between the user and the handle.

* A plug-and-play handle adaptable with any technical aid.

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

## Scientific challenges

- Sensory substitution: tactile and haptic feedback (vibrations, pressure, temperature, etc.)

- Intelligent and connected device for cane, chair, walker

- Adaptable to other disorders

<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /></figure>
First prototypes

What are we perceiving? Do we succeed navigating thanks to haptic feedback?

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>
# First experiments – perception

## Experiments at INSA Rennes – July 2023

* Perceive haptic feedback direction with handles equipped on a rollator.

* Directional feedback by providing Apparent Haptic Motion Illusion.

* Participation of occupational therapists, physiotherapists, etc.

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>
# First navigation experiments

## Experiments at INSA Rennes – July 2023

* Navigate on a pre-define virtual trajectory.

* Directional feedback to assist navigating on the virtual trajectory.

* Participation of occupational therapists, physiotherapists, etc.

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

<figure><img src="image_61.png" /></figure>

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>
# An available solution: “Swalkit”

## Co-design of a rollator with haptic feedback

* Emerged from the need of Esteban, a user of a rollator with blindness and difficulties to navigate safely.

* Vibrotactile sensations to indicate proximity to obstacles.

* "DIY" assistance kit:

> Low cost and open-source.

> Can be installed with the help of a fablab.

<figure><img src="image_64.png" /></figure>

## Providing daily assistance for 18 monts...

* Before the kit: no autonomy of movement.

* With the use of the kit: autonomy, motivation, renewed confidence and more social interactions.

* Tests with 2 other end-users with the Institut des Jeunes Aveugles in Moulins: study in progress.

<figure><img src="image_65.png" /></figure>
# Dornell is at its halfway stage! What are the next steps?

## Improve feedback and embedded sensing...

* Iterative co-design.

> Use the results and participant feedback of the first experiments to improve designs.

> Adapt and personalize the handle feedback as well as ergonomy.

* Innovative one-shot 3D printed sensors to incorporate into the handle.

> User state and intention detection.

> Bidirectional communication.

## Anticipate social navigation challenges

* Evaluate cognitive challenges between the proposed frameworks of haptic feedback and walking/navigation tasks.

* Link with other works on social navigation pedestrian/wheelchair.

<figure><img src="image_66.png" /></figure>
Thanks!

Visit www.inria.fr