What If : Understanding Motion Through Sparse Interactions
Authors: name: Stefan Andreas Baumann, name: Nick Stracke, name: Timy Phan, name: Björn Ommer
Organization: CompVis

Understanding the dynamics of a physical scene involves reasoning about the
diverse ways it can potentially change, especially as a result of local
interactions. We present the Flow Poke Transformer (FPT), a novel framework for
directly predicting the distribution of local motion, conditioned on sparse
interactions termed "pokes". Unlike traditional methods that typically only
enable dense sampling of a single realization of scene dynamics, FPT provides
an interpretable directly accessible representation of multi-modal scene
motion, its dependency on physical interactions and the inherent uncertainties
of scene dynamics. We also evaluate our model on several downstream tasks to
enable comparisons with prior methods and highlight the flexibility of our
approach. On dense face motion generation, our generic pre-trained model
surpasses specialized baselines. FPT can be fine-tuned in strongly
out-of-distribution tasks such as synthetic datasets to enable significant
improvements over in-domain methods in articulated object motion estimation.
Additionally, predicting explicit motion distributions directly enables our
method to achieve competitive performance on tasks like moving part
segmentation from pokes which further demonstrates the versatility of our FPT.
Code and models are publicly available at
https://compvis.github.io/flow-poke-transformer.