Cure the headache of Transformers via Collinear Constrained Attention
Authors: name: Shiyi Zhu, name: Jing Ye, name: Wei Jiang, name: Qi Zhang, name: Yifan Wu, name: Jianguo Li

As the rapid progression of practical applications based on Large Language
Models continues, the importance of extrapolating performance has grown
exponentially in the research domain. In our study, we identified an anomalous
behavior in Transformer models that had been previously overlooked, leading to
a chaos around closest tokens which carried the most important information.
We've coined this discovery the "headache of Transformers". To address this at
its core, we introduced a novel self-attention structure named Collinear
Constrained Attention (CoCA). This structure can be seamlessly integrated with
existing extrapolation, interpolation methods, and other optimization
strategies designed for traditional Transformer models. We have achieved
excellent extrapolating performance even for 16 times to 24 times of sequence
lengths during inference without any fine-tuning on our model. We have also
enhanced CoCA's computational and spatial efficiency to ensure its
practicality. We plan to open-source CoCA shortly. In the meantime, we've made
our code available in the appendix for reappearing experiments.