# PDE-Driven Spatiotemporal Disentanglement

Jérémie Donà, Jean-Yves Franceschi, Sylvain Lamprier, Patrick Gallinari

## ► To cite this version:

Jérémie Donà, Jean-Yves Franceschi, Sylvain Lamprier, Patrick Gallinari. PDE-Driven Spatiotemporal Disentanglement. The Ninth International Conference on Learning Representations, May 2021, Vienne (virtual), Austria. . hal-03181039

HAL Id: hal-03181039

https://hal.science/hal-03181039

Submitted on 25 Mar 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
## Motivation

* Disentanglement improves interpretability and prediction

* Prior spatiotemporal disentanglement works are often complex and do seldom analysis of its meaning

* We aim at grounding spatiotemporal disentanglement on stronger foundations

## Separation of Variables as Disentanglement

* Consider the heat equation:

$$ \frac{\partial u}{\partial t} = c^2 \frac{\partial^2 u}{\partial x^2}, \quad u(0, t) = u(L, t) = 0, \quad u(x, 0) = f(x), $$

with separable solutions:

$$ u(x, t) = \mu \sin\left(\frac{n\pi}{L}x\right) \exp\left(-\left(\frac{cn\pi}{L}\right)^2 t\right) = \xi(\phi(x) \times \psi(t)) $$

* This method separates factors of variations and can be generalized and applied to numerous systems

## Separation of Variables for Observations

* State $u_v: (x, t) \mapsto u_v(x, t)$, observations $v = (v_{t_0}, \dots, v_{t_1})$ corresponding to a spatial measurement of $u$:

$$ v_t = \zeta(u_v(\cdot, t)) $$

* Following the functional separation of variables, we seek $\phi, \psi, U, \xi$ and $\zeta$ such that:

$$ z = \xi(\phi(x), \psi(t)), \quad u(x, t) = U(z), \quad v_t = \zeta(u(\cdot, t)), $$

with associated ODEs or PDEs on $\phi, \psi$ and $U$

* We abstract the unknown spatial coordinates:

$$ v_t = (\zeta \circ U \circ \xi)(\phi(\cdot), \psi(t)) = D(\phi, \psi(t)) $$

* In practice, we learn vectorial $\phi \equiv S$ and $\psi \equiv T_t$

* $S$ and $T_{t_0}$ are inferred with encoders $E_S$ and $E_T$ from conditioning frames:

$$ V_{\tau}(t_0) = (v_{t_0}, \dots, v_{t_0+\tau}) $$

## Prediction Constraints

* $T \equiv \psi$ is driven by an ODE: $\frac{\partial T_t}{\partial t} = f(T_t)$

* Forecasting and alignment losses:

$$ \mathcal{L}_{\text{pred}} = \sum_t \|\hat{v}_t - v_t\|_2^2 $$

$$ \mathcal{L}_{\text{AE}} = \left\| D \left( S, E_T(V_{\tau}(t')) \right) - v_{t'} \right\|_2^2 $$

## Disentanglement Constraints

* From a strict $S$ invariance constraint to a weaker one to take into account variations of observable content:

$$ \frac{\partial E_S(V_{\tau}(t))}{\partial t} = 0 $$

$$ \Rightarrow \mathcal{L}_{\text{reg}}^{S} = \left\| E_S(V_{\tau}(t_0)) - E_S(V_{\tau}(t_1 - \tau)) \right\|_2^2 $$

* Disentanglement loss:

$$ \mathcal{L}_{\text{reg}}^{T} = \left\| T_{t_0} \right\|_2^2 = \left\| E_{T}(V_{\tau}(t_0)) \right\|_2^2 $$

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

<table><thead><tr><th rowspan="3">Models</th><th>SST</th><th>Mov.</th><th>MNIST</th><th>TaxiBJ</th></tr><tr><th>Pred.</th><th>Pred.</th><th>Swap</th><th>Pred.</th></tr><tr><th>MSE</th><th>SSIM</th><th>MSE</th><th>MSE</th></tr></thead><tbody><tr><td>PhyDNet</td><td>1.27</td><td>0.3878</td><td>0.2839</td><td>41.9</td></tr><tr><td>DDPAE</td><td>—</td><td>0.6446</td><td>0.6378</td><td>—</td></tr><tr><td>MIM</td><td>0.91</td><td>0.6529</td><td>—</td><td>42.9</td></tr><tr><td>Ours</td><td><strong>0.86</strong></td><td><strong>0.7990</strong></td><td><strong>0.7713</strong></td><td><strong>39.5</strong></td></tr><tr><td>Ours (no S)</td><td>0.95</td><td>0.6707</td><td>—</td><td>43.7</td></tr></tbody></table>

<figure><img src="image_4.png" /></figure>