LayerCake: Token-Aware Contrastive Decoding within Large Language Model
  Layers
Authors: name: Jingze Zhu; name: Yongliang Wu; name: Wenbo Zhu; name: Jiawang Cao; name: Yanqiang Zheng; name: Jiawei Chen; name: Xu Yang; name: Bernt Schiele; name: Jonas Fischer; name: Xinting Hu
Published: 2025-07-06 14:35:43

Abstract: Large language models (LLMs) excel at natural language understanding and
generation but remain vulnerable to factual errors, limiting their reliability
in knowledge-intensive tasks. While decoding-time strategies provide a
promising efficient solution without training, existing methods typically treat
token-level and layer-level signals in isolation, overlooking the joint
dynamics between them. In this work, we introduce a token-aware,
layer-localized contrastive decoding method that aligns specific token types
with their most influential transformer layers to improve factual generation.
Through empirical attention analysis, we identify two key patterns: punctuation
tokens receive dominant attention in early layers, while conceptual tokens
govern semantic reasoning in intermediate layers. By selectively suppressing
attention to these token types at their respective depths, we achieve the
induction of controlled factual degradation and derive contrastive signals to
guide the final factual decoding. Our method requires no additional training or
model modification, and experiments demonstrate that our method consistently
improves factuality across multiple LLMs and various benchmarks.