Eventful Transformers: Leveraging Temporal Redundancy in Vision
  Transformers
Authors: name: Matthew Dutson; name: Yin Li; name: Mohit Gupta
Published: 2023-08-25 17:10:12

Abstract: Vision Transformers achieve impressive accuracy across a range of visual
recognition tasks. Unfortunately, their accuracy frequently comes with high
computational costs. This is a particular issue in video recognition, where
models are often applied repeatedly across frames or temporal chunks. In this
work, we exploit temporal redundancy between subsequent inputs to reduce the
cost of Transformers for video processing. We describe a method for identifying
and re-processing only those tokens that have changed significantly over time.
Our proposed family of models, Eventful Transformers, can be converted from
existing Transformers (often without any re-training) and give adaptive control
over the compute cost at runtime. We evaluate our method on large-scale
datasets for video object detection (ImageNet VID) and action recognition
(EPIC-Kitchens 100). Our approach leads to significant computational savings
(on the order of 2-4x) with only minor reductions in accuracy.