Gated Delta Networks: Improving Mamba2 with Delta Rule
Authors: name: Songlin Yang; name: Jan Kautz; name: Ali Hatamizadeh
Published: 2024-12-09 13:09:04

Abstract: Linear Transformers have gained attention as efficient alternatives to
standard Transformers, but their performance in retrieval and long-context
tasks has been limited. To address these limitations, recent work has explored
two distinct mechanisms: gating for adaptive memory control and the delta
update rule for precise memory modifications. We observe that these mechanisms
are complementary: gating enables rapid memory erasure while the delta rule
facilitates targeted updates. Building on this insight, we introduce the gated
delta rule and develop a parallel training algorithm optimized for modern
hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses
existing models like Mamba2 and DeltaNet across multiple benchmarks, including
language modeling, common-sense reasoning, in-context retrieval, length
extrapolation, and long-context understanding. We further enhance performance
by developing hybrid architectures that combine Gated DeltaNet layers with
sliding window attention or Mamba2 layers, achieving both improved training
efficiency and superior task performance.