Title: Identifying Support Knowledge Representation for Large Language Models
Primary area: interpretability and explainable AI
Keywords: Knowledge Discovery, Data Mining, LLM

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities, yet their knowledge representation mechanism remains opaque. This paper interprets the LLM's attention layers as a notion of Support Knowledge (SK), a learned representation that LLMs use to ground their decisions, exhibiting properties of sparsity, causality, and atomicity. We provide a theoretical analysis of the mathematical connection between margin maximization and the identification of the Support Knowledge. Empirical analyses suggest a two-segment inference machanism of SK across diverse LLM families (e.g., Qwen3, Llama3.1, ChatGLM4), indicating that there exists layer-wise supporting information units. Interventions on the targeted SK-related neural units reveal their information preserving properties: attacking the SK-related neural units would reduce the model accuracy from 82.34% to 27.78%, while random attacks would only lead to a reduced accuracy of 80.43%.