Title: An Information-Theoretic Parameter-Free Bayesian Framework for Probing Labeled Dependency Trees from Attention Score
Primary area: interpretability and explainable AI
Keywords: probing, attention score, dependency syntax, mutual information

Abstract: Figuring out how neural language models comprehend syntax acts as a key to revealing how they understand languages. 
We systematically analyzed methods of extracting syntax from models, namely _probing_, and found limitations yet widely exist in previous probing practice. 
We proposed a method capable of estimating mutual information (MI) and directly extracting dependency trees from attention scores in a mathematical-rigorous way, requiring no additional network training effort.
Compared with previous approaches, it has a much simpler model, while being able to probe more complex dependency trees, also transparent for fine-grained explanation.
We tested our method on several open-source LLMs and demonstrated its effectiveness by systematically comparing it with a great many competitive baselines. Several informative conclusions can be drawn by further analysis of the results, shedding light on our method’s explanatory potential.
An anonymous and off-the-shelf version of our code is released at https://anonymous.4open.science/r/IPBP-99F1.

TL;DR: A method being able to estimate attention score-syntactical dependency MI and reconstruct dependency trees from probabalistic distributions, requiring no trainable networks.