Behind RoPE: How Does Causal Mask Encode Positional Information?
Authors: name: Junu Kim, name: Xiao Liu, name: Zhenghao Lin, name: Lei Ji, name: Yeyun Gong, name: Edward Choi
Organization: Microsoft

While explicit positional encodings such as RoPE are a primary source of
positional information in Transformer decoders, the causal mask also provides
positional information. In this work, we prove that the causal mask can induce
position-dependent patterns in attention scores, even without parameters or
causal dependency in the input. Our theoretical analysis indicates that the
induced attention pattern tends to favor nearby query-key pairs, mirroring the
behavior of common positional encodings. Empirical analysis confirms that
trained models exhibit the same behavior, with learned parameters further
amplifying these patterns. Notably, we found that the interaction of causal
mask and RoPE distorts RoPE's relative attention score patterns into
non-relative ones. We consistently observed this effect in modern large
language models, suggesting the importance of considering the causal mask as a
source of positional information alongside explicit positional encodings.