Title: Transfer Entropy as a Measure of Information Flow in LLMs and VLMs
Primary area: interpretability and explainable AI
Keywords: Large Language Models, Vision Language Models, Transfer Entropy, Information Flow, Layer Pruning

Abstract: Understanding how information propagates within large language models (LLMs) and vision-language models (VLMs) is central to both interpretability and efficient deployment. While attention weights reveal where probability mass is allocated, they do not capture causal influence or redundancy between layers, tokens, and heads. In this work, we introduce transfer entropy (TE) as a principled, information-theoretic measure of directional information flow in Transformer architectures. We develop tractable estimators of TE tailored to high-dimensional hidden states and apply them to unimodal (RoBERTa, T5) and multimodal (CLIP ViT-B/16) models. Our analysis shows that layerwise TE profiles expose depth-dependent redundancy, enabling targeted pruning with minimal accuracy loss. At the token level, TE distinguishes sink, broadcaster, and benign roles, revealing distinct patterns in text and vision encoders. At the head level, combining TE with CLS/EOT attention mass yields role-aware head classification, offering interpretable maps of responsibility across depth. Experiments on GLUE and MSCOCO-2014 demonstrate that TE-guided pruning achieves competitive retrieval and classification performance while improving efficiency. Overall, TE provides a unified lens for diagnosing, interpreting, and compressing large Transformer-based models, bridging theoretical information flow with practical model optimization.

TL;DR: Information Flow in LLMs and VLMs