Trainable Attention-based Conditional Dependency for Uncertainty Quantification of Large Language Models
Authors: Artem Vazhentsev, Ekaterina Fadeeva, Rui Xing, Alexander Panchenko, Preslav Nakov, Timothy Baldwin, Maxim Panov, Artem Shelmanov


Reviewer paper summaries: This paper studies uncertainty quantification (UQ) in LLMs to address hallucinations in these models. The contributions of this paper can be summarized as follows: 1) A computationally efficient implementation for UQ is proposed, where a neural network is trained to predict targets defined as the gap between the conditional generation confidence and the unconditional generation confidence using regression loss; 2) Experimental results demonstrates the effectiveness of the proposed approach across nine datasets and three LLMs.
Summary of strengths: Uncertainty quantification is an important topic for LLMs, which could benefit the ACL community.
Summary of weaknesses: - It is a bit difficult to follow the technical details.

- In general, LLMs output $p(x_{i}  | x_{<i})$. The authors use autoregressive probabilities as the first-order Markov transition probabilities. Although this substitution is imprecise, it somehow improves performance.

- Could you please elaborate on what $atten_i$ is? I believe there are different attention matrices in LLMs. In addition, how to parameterize G?
Comments, suggestions and typos: - Line 93: ML-based?
Overall assessment scores: 2.5