<figure><img src="image_1.png" /></figure>

HAL
open science

# Learning for Companion Robots: Preparation and Adaptation

Xavier Alameda-Pineda

► To cite this version:

Xavier Alameda-Pineda. Learning for Companion Robots: Preparation and Adaptation. CAp - RFIAP 2024 - Joint Conférences sur l'Apprentissage automatique and Reconnaissance des Formes, Image, Apprentissage et Perception, Jul 2024, Lille, France. hal-04634985

HAL Id: hal-04634985

https://hal.science/hal-04634985v1

Submitted on 4 Jul 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Learning for Companion Robots:
Preparation and Adaptation

Xavier Alameda-Pineda, Inria @ UGA, LJK, CNRS

Joint RFIAP CAP 2024
## Where are we?

ROBOT
EARN

Inria

UGA

Université
Grenoble Alpes

<figure><img src="image_3.png" /></figure>
# Why these wandering computers?

<figure><img src="image_4.png" /></figure>

## Interactive autonomous systems

<figure><img src="image_5.png" /></figure>

* Multi-modal

<figure><img src="image_6.png" /></figure>

* Perceive, Act, & React

<figure><img src="image_7.png" /></figure>

* Impact on people

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

## Learning-based components

<figure><img src="image_10.png" /></figure>

* High-cost real data

<figure><img src="image_11.png" /></figure>

* Limited on-line computation

* Need for preparation/adaptation

<figure><img src="image_12.png" /></figure>
DVAE

Dynamical
Variational
AutoEncoders

<figure><img src="image_13.png" /></figure>

SFR

Successor Feature
Representation
Learning

<figure><img src="image_14.png" /></figure>

SPRING

Social Robotic
Experiments in a
Day-care Hospital

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

Apologies in advance for the equations ;)
Dynamical Variational Autoencoders[1]

<figure><img src="image_17.png" /></figure>
Probabilistic models approximate the **data distribution** with a parametric proxy.

Interest of latent variables: model hidden phenomena with probabilities.

<figure><img src="image_18.png" /><figcaption>Clean speech // Noisy speech</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>Ball position // Detection</figcaption></figure>
Latent variables add spice to learning: how to maximise[2,3,4] the log-likelihood?

$$ \log p(\boldsymbol{x}) = \underbrace{\mathbb{E}_{q(\boldsymbol{z})} \left[ \log \frac{p(\boldsymbol{z})p(\boldsymbol{x}|\boldsymbol{z})}{q(\boldsymbol{z})} \right]}_{\text{M-step or VLB}} + \underbrace{D_{\text{KL}} \left( q(\boldsymbol{z}) \| p(\boldsymbol{z}|\boldsymbol{x}) \right)}_{\text{E-step}} $$

Step 1: Define the model

<figure><img src="image_20.png" /></figure>

Step 2: Learning and inference

<figure><img src="image_21.png" /></figure>

<footnote>[2] A.P. Dempster, et. al., (1977), Journal of the Royal Statistical Society.</footnote>

<footnote>[3] M. I. Jordan, et. al., (1999), Machine Learning.</footnote>

<footnote>[4] D. Kingma, et. al., (2013), International Conference on Learning Representations.</footnote>
VAEs learn a probabilistic representation with latent non-linear dependencies:

<figure><img src="image_22.png" /></figure>

Maximise a lower bound:

$$ \log p_{\theta}(\mathbf{x}) \geq \mathbb{E}_{q_{\phi}(z)} \left[ \log \frac{p_{\theta}(z)p_{\theta}(\mathbf{x}|z)}{q_{\phi}(z)} \right] $$

From what we've defined:

$$ \begin{cases} p_{\theta}(z) \& p_{\theta}(\mathbf{x}|z) & (\text{prior \& likelihood}) \\ q_{\phi}(z) \approx p_{\theta}(z|\mathbf{x}) & (\text{approx. posterior}) \end{cases} $$
# Fundamental limitation of VAE: frame-wise modeling

From a VAE perspective, these two speech signals are **exactly** the same:

<figure><img src="image_23.png" /></figure>
Model sequences of observations & latent variables:

$$p_{\theta}^{\text{DVAE}}(\mathbf{x}_{1:T}, \mathbf{z}_{1:T}) \neq \prod_{t=1}^{T} p_{\theta}(\mathbf{x}_{t}, \mathbf{z}_{t}) = p_{\theta}^{\text{VAE}}(\mathbf{x}_{1:T}, \mathbf{z}_{1:T})$$

<figure><img src="image_24.png" /></figure>
As in VAEs, the posterior needs to be approximated. Simplified using **D-separation**.[5]

$$q_{\phi}(z_{1:T}|\mathbf{x}_{1:T}) = \prod_{t=1}^{T} q_{\phi}(z_t|z_{1:t-1}, \mathbf{x}_{1:T})$$

Learning is performed by maximising the "temporal" ELBO.[1]

$$\mathcal{L}(\mathbf{x}_{1:T}; \phi, \theta) = \sum_{t=1}^{T} \underbrace{\mathbb{E}_{q_{\phi}(z_{1:t}|\mathbf{x}_{1:T})} [\ln p_{\theta_x}(\mathbf{x}_t|\mathbf{x}_{1:t-1}, z_{1:t})]}_{\text{Reconstruction}} - \sum_{t=1}^{T} \underbrace{\mathbb{E}_{q_{\phi}(z_{1:t-1}|\mathbf{x}_{1:T})} \left[ D_{\text{KL}}\left( q_{\phi}(z_t|z_{1:t-1}, \mathbf{x}_{1:T}) \middle\| p_{\theta_z}(z_t|\mathbf{x}_{1:t-1}, z_{1:t-1}) \right) \right]}_{\text{Regularization}}$$

<footnote>[5] Bishop, C. M., (2006).</footnote>

<footnote>[1] L. Girin, et. al., (2021), Foundations and Trends on Machine Learning.</footnote>
Mixtures of DVAEs for
unsupervised multi-source tracking [6]

<figure><img src="image_25.png" /></figure>
Motivation: multi-source trajectory modeling

<figure><img src="image_26.png" /><figcaption>Trajectories @ t-1</figcaption></figure>

<figure><img src="image_27.png" /><figcaption>Observations @ t</figcaption></figure>

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /><figcaption>Desired output</figcaption></figure>
# MixDVAE Intuition

Multiple tracks: track-detection assignment latent variable.

Tracks not observed directly: object position is a latent variable.

<figure><img src="image_30.png" /><figcaption>**GMM**
(Multi-source
No dynamics)</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>**Kalman**
(1-source
Linear dyn.)</figcaption></figure>

<figure><img src="image_32.png" /><figcaption>**DVAE**
(1 source
Non-linear)</figcaption></figure>

<caption>**MixKalman**
(Multi-source
Linear dyn.)

**MixDVAE**
(Multi-source
Non-linear)</caption>
# MixDVAE algorithm

Pre-trained DVAE (synthetic data) + Variational Expectation Maximisation (VEM)

<figure><img src="image_33.png" /></figure>
Let us recall de GMM update and the Kalman update:

$$ \mu_n = \sum_k \underbrace{\eta_{kn}}_{\text{assign.}} \mathbf{o}_k $$

$$ s_t = \underbrace{P_t \mathbf{o}_t}_{\text{update}} + \underbrace{T_t s_{t-1}}_{\text{prediction}} $$

The MixDVAE update looks like:

$$ s_{tn} = P_t \underbrace{\sum_k \eta_{kn} \mathbf{o}_{tk}}_{\text{assig. & update}} + \underbrace{T_t(s_{1:t-1})}_{\text{non-lin. prediction}} $$

Experiments in multiple object tracking, and sound source separation.

Ongoing: adapting non-linear model, complexity, learning from noisy data, ...
Multimodal DVAEs
for audio-visual speech modeling[7]

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>
<figure><img src="image_38.png" /><figcaption>VAE</figcaption></figure>

<figure><img src="image_39.png" /><figcaption>DVAE</figcaption></figure>

<figure><img src="image_40.png" /><figcaption>MVAE</figcaption></figure>

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>
Affective audio-visual speech [8]

<figure><img src="image_43.png" /><figcaption>Angry</figcaption></figure>

<caption>Disgust</caption>

<caption>Contempt</caption>

<caption>Fear</caption>

<caption>Happy</caption>

<caption>Sad</caption>

<caption>Surprise</caption>

<figure><img src="image_44.png" /><figcaption>Strong (disgust)</figcaption></figure>

<caption>Medium (disgust)</caption>

<caption>Weak (disgust)</caption>

<caption>Neutral</caption>

<caption>Weak (happy)</caption>

<caption>Medium (happy)</caption>

<caption>Strong (happy)</caption>

<figure><img src="image_45.png" /><figcaption>Left-60</figcaption></figure>

<caption>Left-30</caption>

<caption>Down-30</caption>

<caption>Front</caption>

<caption>Top-30</caption>

<caption>Right-30</caption>

<caption>Right-60</caption>

<figure><img src="image_46.png" /><figcaption>ID-0</figcaption></figure>

<caption>ID-1</caption>

<caption>ID-2</caption>

<caption>ID-3</caption>

<caption>ID-4</caption>

<caption>ID-5</caption>

<caption>ID-6</caption>

What should we model?

<figure><img src="image_47.png" /></figure>

☐ Static AV
(ID, emotion)

<figure><img src="image_48.png" /></figure>

☐ Dynamic AV
(lip-audio corr.)

<figure><img src="image_49.png" /></figure>

☐ Dynamic Audio
(pitch, tone)

<figure><img src="image_50.png" /></figure>

☐ Dynamic Visual
(eye AUs)
# Multimodal DVAE (MDVAE)

<figure><img src="image_51.png" /></figure>

## What should we model?

<figure><img src="image_52.png" /></figure>

☐ Static AV

(ID, emotion)

<figure><img src="image_53.png" /></figure>

☐ Dynamic AV

(lip-audio corr.)

<figure><img src="image_54.png" /></figure>

☐ Dynamic Audio

(pitch, tone)

<figure><img src="image_55.png" /></figure>

☐ Dynamic Visual

(eye AUs)
MDVAE works better coupled with quantisation[9]

<figure><img src="image_56.png" /></figure>
Qualitative results (1)

<figure><img src="image_57.png" /></figure>

Same visual dynamics

<figure><img src="image_58.png" /></figure>

Same audio-visual dyn.
Qualitative results (2)

<figure><img src="image_59.png" /></figure>

Emotion interpolation

Appearance interpolation
Successor Feature Representations[10]

<figure><img src="image_60.png" /></figure>
Teach a humanoid robot to join a group of people

<figure><img src="image_61.png" /></figure>

Combine audio, visual and optimal motion?

<figure><img src="image_62.png" /></figure>

Deep Reinforcement Learning?

<figure><img src="image_63.png" /></figure>
$(S, A, T, R, \gamma)$

Markov decision process:

- state space,

- action space,

- transition probability,

- reward function,

- discount factor.

<figure><img src="image_64.png" /></figure>

$$Q^{\pi}(s_t, a_t) = R(s_t, a_t) + \gamma \mathbb{E}_{\pi, T|s_t, a_t} \{ R(s_{t+1}, a_{t+1}) + \gamma^2 R(s_{t+2}, a_{t+2}) + \dots \}$$

$$Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha (R(s_t, a_t) + \gamma Q(s_{t+1}, a^*)) - Q(s_t, a_t))$$
# Let's use features

At least two issues: data-greedy and reward design.

<figure><img src="image_65.png" /></figure>

Dependency of the reward function on these features.

$$R(\phi) = R(\phi_{\text{object}}, \phi_{\text{people}}, \phi_{\text{speed}})$$
Long training for every new task!

<figure><img src="image_66.png" /></figure>

Problem: even if the case of feature-based rewards, we need lots of training data.

Can we re-use what we learn *thanks* to these feature-based rewards?
# Transferring knowledge?

<figure><img src="image_67.png" /></figure>

First paradigm was linear[11] dependencies:

$$R_k(s_t, a_t) = w_k^\top \phi(s_t, a_t)$$
How does the action-state value function write?

$$ 
\begin{aligned}
Q_k^\pi(s_t, a_t) &= \mathbb{E}_\pi \left\{ \sum_{\ell \ge 0} \gamma^\ell R_k(s_{t+\ell}, a_{t+\ell} | s_t, a_t) \right\} \\
&= \mathbb{E}_\pi \left\{ \sum_{\ell \ge 0} \gamma^\ell w_k^\top \phi(s_{t+\ell}, a_{t+\ell} | s_t, a_t) \right\} \\
&= w_k^\top \sum_{\ell \ge 0} \gamma^\ell \mathbb{E}_\pi \{ \phi(s_{t+\ell}, a_{t+\ell} | s_t, a_t) \} \\
&= w_k^\top \psi^\pi(s_t, a_t)
\end{aligned}
$$

Definition
of "Q"

$$ R_k(s_t, a_t) = w_k^\top \phi(s_t, a_t) $$

Task-policy separation
Re-using knowledge?

<figure><img src="image_68.png" /></figure>

Generalised policy improvement:
A cumulative greedy policy...

$$ \tilde{\pi}(s_t) = \arg \max_a \max_{j \in [1, K-1]} w_K^\top \psi^{\pi_j}(s_t, a) $$

...cannot be too far away.

$$ \|\pi_K - \tilde{\pi}\| \le \rho(\min_j \|R_K - R_j\|) $$
Reward does not always depend linearly on the features: we need an extension.

$$
\begin{align*}
Q_k^\pi(s_t, a_t) &= \mathbb{E}_\pi \left\{ \sum_{\ell \ge 0} \gamma^\ell R_k(s_{t+\ell}, a_{t+\ell} | s_t, a_t) \right\} \\
&= \mathbb{E}_\pi \left\{ \sum_{\ell \ge 0} \gamma^\ell R_k(\phi(s_{t+\ell}, a_{t+\ell} | s_t, a_t)) \right\} \\
&= \sum_{\ell \ge 0} \gamma^\ell \int_\Phi R_k(\phi) p(\phi(s_{t+\ell}, a_{t+\ell}) | \pi, s_t, a_t) d\phi \\
&= \int_\Phi R_k(\phi) \xi^\pi(\phi, s_t, a_t) d\phi
\end{align*}
$$

Task-policy separation
Linear vs. non-linear

Linear

$R_k(s_t, a_t)$

$w_k^\top \phi(s_t, a_t)$

Non-linear

$R_k(\phi(s_t, a_t))$

$Q_k^\pi(s_t, a_t)$

$w_k^\top \psi^\pi(s_t, a_t)$

$\int_{\Phi} R_k(\phi) \xi^\pi(\phi, s_t, a_t) d\phi$

The generalised policy improvement (greedy cumulative policy) holds for both.

What are we missing so far? The **learning operators**!
# The learning operators

For a given transition $(s_t, a_t, s_{t+1})$

$$a^* = \arg\max_a Q_k^\pi(s_t, a)$$

$Q_k^\pi(s_t, a_t)$

$$Q_k^\pi(s_t, a_t) += \alpha(R_k(s_t, a_t) + \gamma Q_k^\pi(s_{t+1}, a^*) - Q_k^\pi(s_t, a_t))$$

$w_k^\top \psi^\pi(s_t, a_t)$

$$\psi^\pi(s_t, a_t) += \alpha(\phi(s_t, a_t) + \gamma\psi^\pi(s_{t+1}, a^*) - \psi^\pi(s_t, a_t))$$

$\int_{\Phi} R_k(\phi) \xi^\pi(\phi, s_t, a_t) d\phi$

$$\xi^\pi(s_t, a_t) += \alpha(p(\phi|s_t, a_t) + \gamma\xi^\pi(s_{t+1}, a^*) - \xi^\pi(s_t, a_t))$$

Where is the task/environment in the feature-based learning operators?
<figure><img src="image_69.png" /><figcaption>(c) Effect of Non-Linearity</figcaption></figure>

<figure><img src="image_70.png" /><figcaption>(b) Tasks with Linear Reward Functions</figcaption></figure>

<figure><img src="image_71.png" /><figcaption>(d) Tasks with General Reward Functions</figcaption></figure>

<figure><img src="image_72.png" /></figure>
Socially Pertinent Robots in
Gerontological Healthcare[12]

(a lot of people)
**Goal:** to develop socially assistive robots with the capacity of performing multi-person interactions and open-domain dialogue.

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

Experiments at gerontological day-care
Broca hospital (AP-HP)
<figure><img src="image_75.png" /></figure>

## Ethics compliance

### Phase 1 (CER Paris Université):

- Ethnographic study

- Living lab simulated study

### Phase 2 (CPP):

- Day-care hospital waiting room
More than 50 software modules interconnected using ROS

<figure><img src="image_76.png" /></figure>
Acceptability and usability were measured with standard questionnaires:

<table><thead><tr><th>Wave (#pers)</th><th>First (20)</th><th>Second (49)</th></tr></thead><tbody><tr><td>Acceptability<sup>[14]</sup> (max 10)</td><td>3.96 ± 0.05</td><td>6.17 ± 0.33</td></tr><tr><td>Usability<sup>[15]</sup> (max 10)</td><td>4.79 ± 2.40</td><td>5.70 ± 2.30</td></tr></tbody></table>

Comments:

- Scales for consumer products (so proxy).

- Outstanding results for an experimental platform.

- Improvement after taking feedback into account... and including LLMs.

<footnote>[14] J-A Micoulaud-Franchi, et al. Psychiatry Research, 2016.</footnote>

<footnote>[15] J. Brooke, Usability Evaluation In Industry, 1996.</footnote>
* Deep is nice, useful, performance, etc.
But it's great to ground things with **mathematical models**.

* Fusing multiple modalities helps **representing each other**.

* Very **challenging to deploy models** in robotic platforms.

* Beyond limited computation, there scientific challenges linked to **data distribution shifts**.

* Complete software architectures required **combined specific expertises**.

* When experimenting with social robots (and hence with humans) one must be very careful: what is the **impact on the user**?
All RobotLearn-ers and collaborators!

<figure><img src="image_77.png" /></figure>

<figure><img src="image_78.png" /></figure>

UGA
Université Grenoble Alpes

The funding programs supporting our research:

<figure><img src="image_79.png" /></figure>

<figure><img src="image_80.png" /></figure>

<figure><img src="image_81.png" /></figure>

And you for listening. Question (and answer?) time!

I am available for longer discussions: I would like to hear about your topic(s)!!!