# Domain adaptation for pen trajectory reconstruction from kinematic sensors

Florent Imbert, Yann Soullard, Romain Tavenard, Eric Anquetil

► To cite this version:

Florent Imbert, Yann Soullard, Romain Tavenard, Eric Anquetil. Domain adaptation for pen trajectory reconstruction from kinematic sensors. SIFED 2023 – Symposium International Francophone sur l'Écrit et le Document, Jun 2023, Paris, France. hal-04125711

HAL Id: hal-04125711

https://inria.hal.science/hal-04125711

Submitted on 12 Jun 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
## 1. Context

### An ANR bilateral Franco-German project

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

**Goals**

Development of a complete system (hardware + software) to **reconstruct the trajectory** of a digital pen equipped by inertial measurement units (IMU).

<figure><img src="image_4.png" /></figure>

**Partnership**

2 academics collaborators and 2 industries collaborators.

## 2. Handwriting reconstruction

<figure><img src="image_5.png" /></figure>

**Challenges:**

* Design a sequence-to-sequence neural network

* Reconstruct online trajectory from relative signals

* Writing on any surface

**IMUs:** Front and rear accelerometers, gyroscope, magnetometer, force sensor

## 3. Pipeline [1]

<figure><img src="image_6.png" /></figure>

**● The architecture**

<figure><img src="image_7.png" /></figure>

## 4. Mains challenges

* ✓ Reconstruct the touching strokes

* ✓ Synchronize timestamps

* ✓ Deal with ground truth and sensor data of different sizes

* ✗ Follow the hovering path to reposition the start of the next touching stroke (no label, extra dimension)

* ✗ Learn the z dimension, which is not seen in training (learning on touching only, data acquired on plane surface)

* ✗ Deal with data variability depending on the materials (paper, tablet, etc.), various rubs and noises that impact reconstruction

## 5. Domain adaptation: 3D and paper

**● Acquisition of data :**

- On an inclined plane in order to get signals with variable z dimension (the height)

- On paper to obtain data from various materials and signal variability

- Domain adaptation for data acquired on paper and on inclined plane

Goal

- Build domain-invariant features to tilt and noise in signals

By

- Successive learning of the different colored parts

- feature extractor (green),

- label predictor (blue),

- domain classifier (red)

**● An hybrid loss to penalize hovering**

$$L = L_t + \alpha L_h$$

$$L_t = MSE_{t \in touch}$$

$$L_h = (\sum_{t \in hov} \Delta x_t - V_{gt})^2$$

<figure><img src="image_8.png" /><figcaption>Domain-Adversarial Training of Neural Networks [3]</figcaption></figure>

<figure><img src="image_9.png" /></figure>

## 6. Preliminary results

**● For paper data**

**● Ground truth**

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

**● For inclined data**

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

**● Prediction without DANN**

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

**● Prediction without DANN**

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

**● Stroke level**

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

Ground truth

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

Prediction

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

**● Domain adaptation to inclined data**

<table><thead><tr><th></th><th></th><th colspan="2">base model: TCN</th><th colspan="2">DANN</th><th colspan="2">DANN</th><th colspan="2">DANN</th></tr></thead><tbody><tr><th>Epoch</th><td></td><td colspan="2">20</td><td colspan="2">30</td><td colspan="2">40</td><td colspan="2"></td></tr><tr><th>Lambda</th><td>evaluation type</td><td>metric type</td><td>avg</td><td>sd</td><td>avg</td><td>sd</td><td>avg</td><td>sd</td><td>avg</td><td>sd</td></tr><tr><th>0.01</th><td>label</td><td>Fréchet</td><td>0.3638</td><td>0.145</td><td>0.2682</td><td>0.1936</td><td>0.2823</td><td>0.1697</td><td><strong>0.2155</strong></td><td>0.164</td></tr><tr><th></th><td>label</td><td>RMSE</td><td>3.8376</td><td>1.9588</td><td>2.3127</td><td>1.6302</td><td>2.5964</td><td>1.7103</td><td><strong>1.961</strong></td><td>1.4674</td></tr><tr><th></th><td>stroke</td><td>Fréchet</td><td>0.3721</td><td>0.1901</td><td>0.1972</td><td>0.1247</td><td>0.2012</td><td>0.1304</td><td><strong>0.1933</strong></td><td>0.1281</td></tr><tr><th></th><td>stroke</td><td>RMSE</td><td>2.8054</td><td>1.0438</td><td>2.1063</td><td>1.454</td><td>1.9723</td><td>1.3132</td><td><strong>1.9928</strong></td><td>1.306</td></tr></tbody></table>

**● Results analysis**

- Domain adaptation improves results qualitatively and quantitatively

- The inclusion of the **third dimension** provides greater **spacial coherence**, which is reflected in improved label scores

- The hovering trajectory is not sufficiently well reconstructed, so we want to fill this gap with a **hybrid loss**.

**Florent Imbert**
IRISA, Université de Rennes, INSA Rennes

**Yann Soullard**
IRISA, LETG, Université Rennes 2

**Romain Tavenard**
IRISA, LETG, Université Rennes 2

**Eric Anquetil**
IRISA, Université de Rennes, INSA Rennes

**Contact**

florent.imbert@irisa.fr

**References**

[1] Swaileh, W., Imbert, F., Soullard, Y. et al. Online handwriting trajectory reconstruction from kinematic sensors using temporal convolutional network. IJDAR (2023).

[2] M. Wehbi, D. Luge, T. Hamann, et al. Surface-free multi-stroke trajectory reconstruction and word recognition using an imu-enhanced digital pen. Sensors, 2022

[3] Ganin, Yaroslav, et al. « Domain-Adversarial Training of Neural Networks ». In Domain Adaptation in Computer Vision Applications, édité par Gabriela Csurka, 189-209. Advances in Computer Vision and Pattern Recognition. 2017