Value Residual Learning For Alleviating Attention Concentration In
  Transformers
Authors: name: Zhanchao Zhou; name: Tianyi Wu; name: Zhiyun Jiang; name: Zhenzhong Lan
Published: 2024-10-23 14:15:07

Abstract: Transformers can capture long-range dependencies using self-attention,
allowing tokens to attend to all others directly. However, stacking multiple
attention layers leads to attention concentration. One natural way to address
this issue is to use cross-layer attention, allowing information from earlier
layers to be directly accessible to later layers. However, this approach is
computationally expensive. To address this problem, we propose Transformer with
residual value (ResFormer) which approximates cross-layer attention through
adding a residual connection from the values of the the first layer to all
subsequent layers. Based on this method, one variant is the Transformer with
single layer value (SVFormer), where all layers share the same value embedding
from first layer, reducing the KV cache by nearly 50%. Comprehensive empirical
evidence demonstrates that ResFormer mitigates attention concentration problem
in deeper layers and enhances representation across most layers, outperforming
the vanilla Transformer, DenseFormer, and NeuTRENO in training error as well as
downstream tasks. SVFormer trains significantly faster than the vanilla
Transformer and performs better than other methods like GQA and CLA, with
performance influenced by sequence length and cumulative learning rate.