Cached Transformers: Improving Transformers with Differentiable Memory
  Cache
Authors: name: Zhaoyang Zhang; name: Wenqi Shao; name: Yixiao Ge; name: Xiaogang Wang; name: Jinwei Gu; name: Ping Luo
Published: 2023-12-20 03:30:51

Abstract: This work introduces a new Transformer model called Cached Transformer, which
uses Gated Recurrent Cached (GRC) attention to extend the self-attention
mechanism with a differentiable memory cache of tokens. GRC attention enables
attending to both past and current tokens, increasing the receptive field of
attention and allowing for exploring long-range dependencies. By utilizing a
recurrent gating unit to continuously update the cache, our model achieves
significant advancements in six language and vision tasks, including
language modeling, machine translation, ListOPs, image classification, object
detection, and instance segmentation. Furthermore, our approach surpasses
previous memory-based techniques in tasks such as language modeling and
displays the ability to be applied to a broader range of situations.