Attention Sinks in Diffusion Language Models
Authors: name: Maximo Eduardo Rulli, name: Simone Petruzzi, name: Edoardo Michielon, name: Fabrizio Silvestri, name: Simone Scardapane, name: Alessio Devoto

Masked Diffusion Language Models (DLMs) have recently emerged as a promising
alternative to traditional Autoregressive Models (ARMs). DLMs employ
transformer encoders with bidirectional attention, enabling parallel token
generation while maintaining competitive performance. Although their efficiency
and effectiveness have been extensively studied, the internal mechanisms that
govern DLMs remain largely unexplored. In this work, we conduct an empirical
analysis of DLM attention patterns, focusing on the attention sinking
phenomenon, an effect previously observed in various transformer-based
architectures. Our findings reveal that DLMs also exhibit attention sinks, but
with distinct characteristics. First, unlike in ARMs, the sink positions in
DLMs tend to shift throughout the generation process, displaying a dynamic
behaviour. Second, while ARMs are highly sensitive to the removal of attention
sinks, DLMs remain robust: masking sinks leads to only a minor degradation in
performance. These results provide new insights into the inner workings of
diffusion-based language models and highlight fundamental differences in how
they allocate and utilize attention compared to autoregressive models.