Self-Supervised Learning of Motion Concepts by Optimizing
  Counterfactuals
Authors: name: Stefan Stojanov, name: David Wendt, name: Seungwoo Kim, name: Rahul Venkatesh, name: Kevin Feigelis, name: Jiajun Wu, name: Daniel LK Yamins

Estimating motion in videos is an essential computer vision problem with many
downstream applications, including controllable video generation and robotics.
Current solutions are primarily trained using synthetic data or require tuning
of situation-specific heuristics, which inherently limits these models'
capabilities in real-world contexts. Despite recent developments in large-scale
self-supervised learning from videos, leveraging such representations for
motion estimation remains relatively underexplored. In this work, we develop
Opt-CWM, a self-supervised technique for flow and occlusion estimation from a
pre-trained next-frame prediction model. Opt-CWM works by learning to optimize
counterfactual probes that extract motion information from a base video model,
avoiding the need for fixed heuristics while training on unrestricted video
inputs. We achieve state-of-the-art performance for motion estimation on
real-world videos while requiring no labeled data.