Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models
Authors: Mert Yuksekgonul
Varun Chandrasekaran
Erik Jones
Suriya Gunasekar
Ranjita Naik
Hamid Palangi
Ece Kamar
Besmira Nushi
Venue: ICLR 2024 poster (2024)
Keywords: interpretability
factual errors
hallucinations
attention
large language models
Primary area: visualization or interpretation of learned representations
TL;DR: We model factual queries as constraint-satisfaction problems and find that attention to constraint tokens significantly correlates with factual correctness.

Abstract: We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text. We propose modeling factual queries as constraint satisfaction problems and use this framework to investigate how the LLM interacts internally with factual constraints. We find a strong positive relationship between the LLM's attention to constraint tokens and the factual accuracy of generations. We curate a suite of 10 datasets containing over 40,000 prompts to study the task of predicting factual errors with the Llama-2 family across all scales (7B, 13B, 70B). We propose SAT Probe, a method probing attention patterns, that can predict factual errors and fine-grained constraint satisfaction, and allow early error identification. The approach and findings take another step towards using the mechanistic understanding of LLMs to enhance their reliability.

Published as a conference paper at ICLR 2024
ATTENTION SATISFIES: A CONSTRAINI-SATIISACTIII LENS ON FACTUAL ERRORS OF LANGUAGE MODELS
Mert Yuksekgonul Stanford University
Varun Chandrasekaran'
Erik Jones
University of Illinois trbana-Champaige
UC Berkeley
Suriya Gunasekar
Ranjita Naik
Hamid Palangi
Ece Kamar
Besmira Nushi
Microsoft Research
ABSTRACT
We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text. We propose modeling factual queries as constraint satisfaction problems and use this framework to investigate how the LLM interacts internally with factual constraints. We find strong positive relationship between the LLM's attention to constraint tokens and the factual accuracy of generations. We curate suite of 10 datasets containing over 40,000 prompts to study the task of predicting factual errors with the Llama-2 family across all scales (7B, 13B, 70B). We propose SAT Probe, method probing attention patterns, that can predict factual errors and fine-grained constraint satisfaction, and allow early error identification. The approach and findings take another step towards using the mechanistic understanding of LLMs to enhance their reliability.'
INTRODUCTION
Large language models (LLMs) encode substantial knowledge (Petroni et al., 2019; Srivastava al., 2022), yet they are prone generating factually incorrect text. For instance, LLMs can generate confident-appearing completions with hallucinations (Zhang et al., 2023; Ji et al., 2023), fabricating entities or factual claims. As LLMs reach wider audiences and are used for safety-critical applications, understanding the factuality of generations rises to paramount importance. However, our understanding of how LLMs process factual queries and produce errors is nascent. Existing approaches fall into two categories: treat the LLM as black box and query it about generated factual claims, or ii) use white-box methods to study how LLMs internally process factual queries.
Black-box approaches center on analyzing the consistency of the claims of an LLM using follow-up questions with other LLMs (Cohen et al., 2023) or having the LLM self-critique (Zhang et al., 2023; Manakul al., 2023). However, explanations from LLMs suggested to be unreliable (Turpin et al., 2023) or to convey contradictory signals, e.g., LLMs can produce an answer and then acknowledge that it is wrong (Zhang al., 2023; Mündler et al., 2023). Further, black-box methods typically involve multiple generations from I.MMs, which may be prohibitively expensive to use in practice.
Mechanistic white-box approaches investigate the internal mechanisms of LLMs to dissect factual recall. For instance, Meng et al. (2022); Geva et al. (2023) focus on facts with the (subject, relation, object) structure (e.g., Paris, capital of, France) and propose insightful mechanisms of how an LLM recalls a fact. They suggest that the multi-layer perceptron layers store facts, and attention layers transfer factual information from the subject tokens. However, these works focus on when the LLM generates factually correct responses. Mechanisms leading to factual errors are scarcely explored.
Our Contributions: We investigate the internal mechanisms LLMs, specifically the attention pat- terns, when they produce factual errors. We propose to view factual queries constraint satisfaction
!Our datasets, evaluation protocol, and methods will be released at https: / / comb.com/ mi wrw.oott//ooooooocoooo.o..ooo htttc/eeaaniiieee stic-error-probe
"Corresondence to me merty merty@stanfore edu and besmi ra. nushi @mi crosoft.ccom "This work done while at Microsoft Research.

---

Published as a conference paper at ICLR 2024
User: Is there a movic that was directed by Edirector name? and that won <award name>?? Assistant: Yes, the name movie <model completion
Model Completion: Schöndler's
Model Completion: The Lost Thing
Model Completion: The Last the Mokicuns
loly
||agyll
the
mozic
directed by Sptelbore
nus directed by Fondrggon VICi -1
directed by Fendragon VIG -
C1
C1
C
bogg
thet
Ana3
award
204
won the award
did
C2
V(Cs
ViCg)
C2
Gogg
V((G))
Ein
C2
116
Assodasian
20 30
10
30
10
20
30
Layer (4)
Layer (6)
Layer (F)
Figure 1: Tracking attention to predict constraint satisfaction and factual errors. We consider factual queries LLMs as constraint satisfaction problems i.c., factual queries impose set of constraints that the LLM's responses must satisfy. To predict constraint satisfaction (i.c., factual correctness), we track the attention to the constraint tokens in an LLM (here, Llama-2 13B). We find that attention to the constraint tokens highly correlates with factual correctness. The red (resp. blue)) text indicates factually incorrect (resp. correct) completions.
problems (CSPs), where queries comprise constraints that completions should satisfy to be factually correct ($3); e.g., in Figure the director name or the award name are constraints on the model's response to search query for movie. We investigate the interaction between constraints, attention, and factual correctness ($4). Our key finding is that attention to constraint tokens correlates with LLM's factual correctness, where less attention to constraints is associated with inaccurate responses. Building on our insights, we propose SAT PROBF, a method that predicts constraint satisfaction (and thus factual correctness), using a simple probe of the LLM'' attention to constraints ($5). To test SAT PROBE, we curate a suite of 10 datasets of single- and multi-constraint queries with in total over 40, 000 prompts. We find that SAT PROBE performs comparably the LLM's confidence. Further, SAT PROBE can predict factual errors halfway through the forward pass to stop the computation partway and save costs. Our findings contribute to the mechanistic understanding of LLMs and demonstrate the potential of model internals to understand and mitigate factual errors.
2 BACKGROUND: LANGUAGE MODELS AND FACTUAL RECALL
Our presentation of the Transformer architecture (Vaswani et al., 2017) largely follows that of Meng et al. (2022); Geva et al. (2023); Elhage et al. (2021). For brevity, omit the details around layer normalization. Assume an input sequence of T tokens 41,-., lr and l E V for a fixed vocabulary V. A token is represented initially with d-dimensional vector xf Rd using embedding matrix E E Rivixd
We use V denote sequence of tokens.
The architecture consists of L layers that transform the input token embeddings to a sequence of hidden states xf...., ....... x at each layer l where x denotes the state of token Often, each hidden state vector has the same number of dimensions, i.e., Vi, x Rdd The states are obtained by
+ + m,;
where call mf the MLP contribution and a the atlention contribution token i at layer l, respectively. The LLM produces a predicted probability distribution for the next token P( [(rr) using linear softmax layer the last layer representation x4. In this work, we study the interactions among tokens. Since the MLP layers (see Appendix A) standard Transformers do not capture token interactions, we focus primarily the attention operation.
The attention operation updates each token's 's state using the previous states at all positions, i.c., the representation for token is updated by 'attending' to all other tokens. Formally, the operation involves four projection matrices W W Wi, WO E ddd that correspond to the 'query', 'key', 'value", and 'output' projections. Each of these is split into multiple heads, where W-C.h. w.t w ddxd, and w IRdxxd denote the matrices for head h, H is the total number of heads, da is the
2

---

Published as a conference paper at ICLR 2024
dimensionality for each head for h [H]. Often, embeddings split into equal parts such that d, H
d (Elhage et al., 2021; Dar et al., 2022; Touvron et al., 2023). The attention contribution from
the token j to token i, J is defined as:
H t.h (mg A
h=1
(x
(x W
A Softmax
(3)
Hdn/H
where at EyEET a and Softmax is taken row-wise. A RTxT are the attention weights computed by the hith attention head at layer E, and A is the entry in the i-th and j-th column of the matrix. For autoregressive I.LMs, A lower triangular since each token can only attend the representation of the previous tokens. For brevity, we use [H] to denote the sequence of integers from
1 to H, and superscript [H] indicates stacking items for all E [H] i.e., f (A )= R
Recent work investigates the internal activations of LLMS to understand the mechanics of factual recall. By studying factual queries of the form (subject, relation, object), Meng et al. (2022): Geva et al. (2021) provide evidence that MLP layers store factual associations and Geva et al. (2023); Meng et al. (2022); Elhage et al. (2021) suggest that attention layers transfer factual knowledge to where it will be used. As an example, when an LLM is given the prompt LeBron James professionally plays, the information LeBron James professionally plays basketball is extracted by the MLP contribution to the tokens for LeBron James (subject). Next, the attention layers transfer the information from the tokens of the subject to the last token for the LLM to generate basketball (object). While these works mostly study internal mechanisms when the LLM's completions are factually correct, we focus when LLMs produce factually incorrect text.
Mechanics of Factual Recall in Language Models:
FACTUAL QUERIES AS CONSTRAINT SATISFACTION PROBLEMS
Choosing the right framework to study factual errors is challenging. One can naively categorize completions as factually correct or incorrect, yet this binary view can fall short. For example, queries that are easy for the LLM and ones that barely gets right are indistinguisaable since both are labeled as 'correct'. Further, binary labeling prevents us from building an understanding of why some queries more difficult than others or which parts of the queries drive the LLM to failure.
To systematically study factual queries and LLMs' internal behavior, we propose taking a CSP view: CSP). A factual query is specified by set of constraints C= ((( Vi),... (Ck, } where C E y* indicates the sequence of tokens for the constraining entity k2, and V : V+ > {0, 1} is verifier that takes a set of generation tokens as the input and returns whether the constraint indexed by & is satisfied. Under this view, we call completion Y as factual error if l E [K] VV((Y) = 0, that is, if there is a constraint in the factual query that the response does not satisfy". Otherwise, we call the response factually correct.
Definition 3.1 (Factual Query
large set of factual queries can be viewed as a set of constraints that responses must satisfy to be correct, e.g., see Figure 1, This structure is comprehensive; for example, an important subset of queries made by users to search engines has historically been conjunctions of constraints (Spink et al., 2001). Structured and multi multi-constraint queries are also inherent to faceted search and information retrieval (Tunkelang, 2009; Hahn et al., 2010). Further, under this definition, prior (subject, relation, object) queries (Meng al., 2022) can be seen have a single-constraint structure. Similarly, instructions to LLMs are also constraints for controlling the output (Ouyang et al., 2022).
Focusing on the constraints of CSP can help us reason about the difficulty of query and the behavior of the LLM. While there are several factors that can influence LLM's behavior, we shall start
While it may nontrivial to generally isolate tokens for the constraining entity for arbitrary queries, in our evaluations we investigate settings in which we assume we have access to this set. Here, focus conjunctions of constraints, but the framework can be generalized other operations.
3

---

Published as a conference paper at ICLR 2024
Ex: Tell me the year the basketball player Kobe Bryant was born in
Ex: Tell me a word that starts with e and ends with
Model: Llama-2
Model: Llama-2
0.5g
Average Corectneess
0.11
0.27
0.64
0.79
0.9
Average Correctness
0.31
0.26
0.23
0.1
Model: Llama-2
13B
0.0
Model: Llama 2 13B
StTT 6 mm
0.12
0.3
0.92
0.97
0.5
0.35
0.28
0:24
0.26
0.14
Model: Llama-2
70B
0.0
Model:
Llama 2 70B
0.52
0.96
0.99
0.98
0.5
0.42
0.35
0.34
0.3
0.12
(0.0
10.01
(10.0
20.00
(20.0
30.0)
(30.0
40.0)
(40.0
129.0)
0.0
(0.0
0.2)
(0.2
0.4)
(0.4 0.6]
(0.6 0.8)
(0.8
1.0)
Popularity
Constrainedness Quantile
Figure 2: Difficulty of the factual query vs LLM performance. Left: Popularity vs Correctness We observe
that the more popular the entity in the factual query is, the more correct the LLMs are. Right: Constrainedness vs Correctness We observe that the more constrained the problem is (i.c., has a smaller set of potential solutions), the less correct the LLMs are.
with two factors that can describe the difficulty of factual queries: i) the popularity of the constraining entity, and ii) the constrainedness of the query.
Popularity of the Entity vs LLM Performance: Recent work has shown the correlation between training data frequency and memorization in LLMs (Carlini et al. (2022); Biderman et al. (2023); inter alia). However, even for many open-source LLMs, we cannot compute the frequency of facts since we do not have the training data trivial way search for complex facts. As an accessible proxy for entities from WikiData, we use the number of site links on page as the popularity of entity; we hypothesize that it strongly correlates with the training data frequency or popularity. See Tables 3 and 4 for examples of popularity statistics across basketball players and football teams.
For Figure left, we use queries of the form Tell me the year the basketball player [name] was born in (see 1) and evaluate LLMs' accuracy. We compare the correctness of the LLM for entities (players) of varying popularity. We observe that i) LLM performance better for entities with higher popularity, and ii) larger LLMs have better performance for entities that are less popular. Recent works show similar relationships with popular/typical input (Mallen et al., 2022; Yuksekgonul et al., 2023).
Constrainedness of the CSP vs LLM Performance: A well-explored complexity metric for CSPs constrainedness (Gent et al., 1996). Here, we define constrainedness as the number of potential solutions to given problem in the domain of the output. For instance, for query of the form Tell me a word that starts with the letter e and ends with the letter t, we quantify constrainedness by the number of such words in the English language that satisfy these constraints. Figure 2 right shows how constrainedness relates to success: i) LLMS have worse performance for more constrained problems, and ii) larger models perform better across all constrainedness levels.
Summary: We argue that the CSP lens can provide a useful view to capture the difficulty of factual queries. Our goal to build framework to discuss how LLMs process factual queries and generate errors. Next, we investigate L.LMs' internal mechanisms to understand factual errors.
4 UNDERSTANDING FACTUAL ERRORS VIA ATTENTION TO CONSTRAINTS
Here, we explore how an LLM processes constraints when the model produces factually incorrect text. Geva et al. (2023); Meng et al. (2022) suggest that attention layers transfer the factual information from the source entity (e.g., Bad Romance) to the last token for generation (to generate Lady Gaga, Figure when the LLM correctly addresses a query. However, these works do explore the mechanisms when the model produces factually incorrect responses. Intuitively, we want to quantify how the LLM interacts with constraints understand constraint satisfaction (and thus factual errors).
To study how the LLM processes a constraint, we focus on the attention to the constraint tokens, i.c.,
f.h a aTT
A
W W.A.A
We use nl t.tk..ccruu. words (Bird al., 2009) compute the number of such words.
4

---

Published as a conference paper at ICLR 2024
User: Tell me the performerr of the song <Song Numus Assistant: The performer is cmodea completions
User: Tell me the your the basketball player <Player Manues was born in Assistant: The player was bor in comoddl completions
Model Completion: Lody Gaga
Model Complction: Kanye West
Model Complettion 1963
Completion:
||r]|||
|d(z||
14
not born 1980
1963
V(C)
bong by Lady Gaga
V(C)
V(C)
sodg not by Kanye V(C)
10
20
20
10
30
20 30
Layer (0)
Layer
Layer (2)
(1)
Figure 3: Tracking attention to predict factual errors in single-constraint settings. We track the attention contribution from the constraint tokens during generation. We observe small-norm contribution ([|a[r]] when the LLM makes a factual error and a larger-norm attention contribution when the LLM is factually correct. The red text indicates factually incorrect completions, whereas the blue text indicates factually correct complctions.
t.b where a a,r E TRd indicates the attention contribution from a constraint token e through a head h to the final token T (where the T' + 1-th token will be generated), The total contribution to T is then is "e = Σn a a, When the constraint comprises multiple tokens denoted by the set C, we take the f.h
f.h
maximum value across all constraint tokens, i.e., A maxege A or ac, max.ec l|a 1|s. Figure shows an example. We track the regions that marked by C1 and C2 that represent the constraints that the movies were directed by the given directors and won the given awards.
To understand whether attention constraints can help explain factual errors, we study three factors. First, we explore the relationship between attention and popularity of the constraining entity, as we find that LLM's correctness correlates with popularity (Fig 2). Next, we explore the relation of attention the LLM's confidence the prompt X. Finally, we explore how attention patterns behave when we scale the L.LMs.
[X|))) which estimates the probability of a completion Y given
Attention predicts popularity: In Figure 11, we show the results for predicting the popularity of the constraining entity in the prompt (the basketball player) only from the attention weights (A(,)1 using Lasso Regression. In all Llama-2 models (7B, 13B, 70B), the predicted popularities using attention values significantly correlate with the ground truth popularity (over held-out set, with Spearman's 's Correlation 2 0.65 and p-value p R 0 for all LLMs). See Appendix C. for details.
Popularity seems predictive, yet may not always have access clean popularity measure or frequency of constraints in training data. Our main goal is to characterize and predict factual errors, thus we seek more reliable indicator of factual correctness.
Attention correlates with confidence and LLM's correctness: In Figure 4 (left four panels), each row represents the attention flow across layers for single sample and we sort the points by the confidence of the LLM. The lefumost panels show the attention for the 25 most confident predictions and the middle panels show the 25 least confident predictions, where the x-axis shows the layers, and colors indicate the norm of the attention contribution from the constraints (Ilac, ([44),141111) Il). The core observation is that when the LLM is accurate, there is more attention to constraint tokens (first column) in sharp contrast cases where the L.MM fails and the attention is weak (second column).
In Figure 4's rightmost plots, queries are sorted and grouped by the LLM's total attention contribution from the constraints across all layers Ee ||a and LLM'' accuracy is computed for each group. We observe that the magnitude of attention to constraints correlates with accuracy. This observation not only interesting in hindsight; aforethought could have suggested either outcome (e.g., more attention correlating with hallucination). This is a positive observation that suggests attention to constraints can be used to predict LLM's success.
Language models grow larger, pay more attention, and succeed more: In Figure 5, each panel compares the attention to constraints for the basketball player queries between two different LLMs.
'While there is earlier work that suggests the last constraint token could the most important, we observed that in practice there are subtleties. See Appendix C.2 for discussion.
5

---

Published as a conference paper at ICLR 2024
25 most confident predictions
25 least confident predictions
Attention Constraints Accuracy (N 13631)
Llama-2 708
1.0
Accuracy:
Accuracy:
0.88
0.91
0.5
0.73
0.48
0.31
0.0
Liama-2 138
Accuracy: 1.00
Accuracy: 0.04
0.5
0.65
0.0
0.06
0,07
0.09
0.23
40
40
0-20
20-40
40-60
60-80
80-100
|| per Layer 8
|| per Layer
> ||ااا|| Percentile
Figure 4: Attention correlates with correctness. The first two columns of panels give the 25 samples for which the LLM makes the most and the least confident predictions. The color indicates the norm of the attention contribution from the constraint, where each column in the panel captures layer the L.IM and each specific sample. The last column panels relates the total attention constraints and accuracy, where x-axis is the attention contribution percentile in the dataset and the y-axis is the accuracy in the bin. The results are for the year of birth queries (see 16).
on Constraints 138
Attention Consistency 7B 13B
on Constraints 70B
Attention Consistency 7B 70B
on Constraints 70B
Attention Consistency 138 70B
1.00
1.00
1.00
0.75
0.75
0.75
0.50
Both Successful
0.50
0.50
Both Fail
0.25
Only Larger
0.25
0.25
0.25
0.54
0.73
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
100
Attention on Constraints in 7B
Attention on Constraints in 7B
Attention on Constraints in 13B
Figure 5: Attention contribution and model scaling. Here, the x-axis and y-axis show the attention to the constraints for the smaller LLM and the larger LLM, respectively, and are normalized by dividing by the maximum value of total attention in the dataset. Coloring is determined by which of the two LLMs succeeds in factual queries. We group the factual queries by their x-axis value and y-axis values and color cell with most frequent category in cell. Appendix Figure 13 presents the complete seatter plot.
where the x-axis indicates the smaller LLM, the y-axis indicates the larger LLM and the coloring indicates the success of the pair of LLMs. We group prompts by the attention contribution, and color the cells by the the most frequent category. We find that more (relatively) attention in both LLMS generally indicates success for both and less attention in both LLMS indicates failure for both. For cases on the top left, the larger I.I. does pay more attention, and only the larger L.LM succeeds. Overall, we
consistent pattern between attention and correctness across model scales; performance improvements in larger LLMs correlate with increased attention to constraint tokens.
We explored the interaction between attention, constraints, and factual correctness. Our analyses suggest that attention can help understand and predict factual errors. Next, we further pull this thread with extensive experiments to predict factual errors using LLMs' attention patterns.
Summary:
5 PREDICTING FACTUAL ERRORS USING ATTENTION TO CONSTRAINTS
We now show how mechanistic understanding be used predict the failures of LLMs. Let X denote a prompt: a sequence of tokens that specifies a factual query with a set of constraints C [(C1, Vi),... (Ck, Vx)}. ). Let Y be the response tokens obtained from the LLM after feeding X. Broadly, we want design function of to estimate the probability that constraint & satisfied:
P(V)()) 1) f(x, Ck, M),
using the prompt, completion, constraint tokens, and the LLM M. For single-constraint factual queries where there is single correct completion Y, this can be reduced to the correctness P(Y = =)= = =XX Y, M). Note how this formalism closely matches that of selective classification (Geifman & Yaniv,,, 2017), where the goal is to abstain when the model would otherwise fail.

---

Published as a conference paper at ICLR 2024
Dataset Name
Constraint Typo(s)
N
Constraint Source
Verifier
Example Prompt
Baskethall Players
born the year
13631
WikiDDta
Exact Match
Figure 16
Football Teams
founded in the year
8825
WikiData
Exact Match
Figure 17
Movies
directed by performed by mother tongue
12197
WikiDDat
Exact Match
Figure 18
Songs
2813
WikiData
Exact Match
Figure 19
CounterFact
919
CounterFact
Exact Match
Figure 20
CounterFact
citizenship
958
CounterFact
Exact Match
Figure 21
CounterFact
headquarter location
756
CounterFact
Exact Match
Figure 22
Books
author, published year
1492
WikiDDta
WikiDDta Search
Figure 23
Nobel Winner
won Nobel, born in city
1290
Opendatasofi (2023)
WikiData Search
Figure 24
Words
starts with, ends with
1352
Hand-Curation
Character Match
Figure 25
Table 1: Overview of Datasets. The columns denote the dataset name, constraint type, number of prompts, and the data sources used to collect entities and verify LLM responses, respectively.
Datasets: For our evaluations, we curate a benchmark with datasets that are listed Table I containing over 40, 000 queries. For single-constraint queries, we curate datasets using WikiDData and 3 datasets using the existing CounterFact dataset (Meng et al., 2022). We further designed three 2-constraint datasets, using WikiData (Books), Opendatasoft (2023) (Nobel Winners), or hand-curation (Words). Further details about all data curation can be found in Appendix D.
Constraint Verification (V&): We use Exact Match for single-constraint queries with single solution. We probe WikiData to verify constraints when queries have multiple potential solutions. Appendix D.3 contains complete description of the methodology.
Models:
We use the 7B, 13B, and 70B parameter variants of Llama-2 (Touvron et al., 2023) released HuggingFace's Transformers (Wolf et al., 2019). We perform our experiments on single NVIDIA A100-PCIE-808 80GB GPU. 80GB memory can only fit the Llama-2 70B in 8-bit precision (Dettmers al. (2022a) report marginal-to-aa to-no performance drop). See Appendix for further details on models.
Evaluation Metrics: We use AUROC for the binary task of predicting failure or success as it does not require setting a threshold for the classifier. We also report Risk 20s: (the fraction of errors for the samples with top 20% of the scores by the predictor f)) RiskBottom 20% (the fraction of errors for the samples with the bottom 20% the scores by the predictor f). These metrics measure how well the model performs on the most and least reliable completions according to the predictor f. For a good success predictor, we want the error fraction to be low among high-score examples (small Riskrop 20%) and have large fraction of failures among low-score examples (large Riskgotom 20%).
5.1 PREDICTING FACTUAL CORRECTNESS
Predictors (f): We propose the constraint satisfaction probe, SAT PROBE, that predicts if an individual constraint is satisfied by only looking at self-attention. To demonstrate the simplicity, we let be linear function of the attention weights constraints:
P(VV(()) = 1:Ac.r) = oww AC + b),
where Ac.,T, E R.X. R. Ac..r (Al E [L] E [H] A. We linearly probe the attention weights across all layers and attention heads and estimate w and b using Logistic Regression. In multi-constraint settings we use SAT PROBE and combine the predictions for constraints:
IP( II 1 (mmo))=n)AA.o. Ac..r) = II (((V())) = 1; Ac..))
kE []]
KE[K]
We compare SAT PROBE to the CONFIDENCE of the model, PP(PIx), which concurrent work reports as hallucination detector (Varshney al., 2023); and CONSTANT predictor that predicts the majority class (either 0 or 1) as baselines. Note that while CONFIDENCE is strong baseline, only provides an overall estimate for the whole generation, and cannot predict the failure
Baselines:
'We acknowledge that exact match is strict criterion that could introduce noise to our evaluations. and it constitutes a limitation where use this verification. Evaluating factual correctness is still evolving research topic (Min al., 2023) and do hest find queries and prompt structures that suffer the least from this.
7

---

Published as a conference paper at ICLR 2024
Llama-2 7B
Llama-2 13B
Llama-2 2 70B
Nobel Winner
Books
SAT-PRooK
Words
Colertwee
AUROC
AUROC
AUROC
(a) Factual error prediction for multiple constraint queries.
Llama-2 2 7B
Llama-2 13B
Llama-2 70B
Basketball Players
Songs
Movies
Data
Football
COMmINGD
SAT-PaouE
CounterFact - MT
CusciDeNcE
CounterFact Ctt
PormLatry
CounterFact
CoNSTANT
0.4
1.0
0.2
0.5
AUROC
AUROC
AUROC
(b) Factual error prediction for single constraint queries.
Figure Factual Error Prediction. SAT PROBE is often comparable marginally worse than CONFIDENCE, and better than POPULARITY, COMBINED predictor where attention and confidence are combined to predict failures mostly performs the best. Error bars show the standard error over 10 different random train/test splits. Tables 5,6 contain the results in tabular form.
for individual constraints. We also use the POPULARITY baseline only in the single-constraint WikiData datasets that curated in other datasets, it is not accessible (CounterFact) or unclear how to compute (multi-constraint) In the Appendix, we also give results with featurization using the attention contribution, e.g., aC,.T (Al E [4], E [M] | denoted by SAT PROBE(a). Finally, when predicting constraint satisfaction, we concatenate confidence to the set of attention features the logistic regression and denote this by COMBINED.
Results: In Figure 6a, we present the overall AUROC of predicting factual correctness for multi- constraint queries, and Table 6 contains all metrics. In this task, we find that SAT PROBE performs comparably to the model's CONFIDENCE in the correctness prediction task, in addition to being able to provide fine-grained feedback (i.e., which constraint not satisfied, presents the AUROC results for the single-constraint setting. In the Ligle-constrrint setting, SAT PROBE comparable to CONFIDENCE, Further, we find that the approaches are comparably good in isolating highly reliable vs unreliable points (Table 5,6 show Risk 20% and RiskBottom 20% metrics). CONFIDENCE alone appears to be slightly more performant than attention alone, yet attention helps us more efficiently and identify the failures in fine-grained manner (see $ 5.2). COMBINED predictor that uses both attention and confidence gets the best performance across all three metrics in most scenarios. In Figure 15, compare the predictions made by SAT PROBE and CONFIDENCE. Notably, while they correlate, both predictors have different errors, e.g., there are cases where the LLM is overconfident yet there is almost no attention and it is a factual error, vice-versa. This further provides insight into why the COMBINED predictor performs better overall.
$5.2). Figure 6b
Attention alone is significantly better than the CONSTANT baseline which suggests that the signals relay a nontrivial amount of information, sometimes exceeding CONFIDENCE. Surprisingly. even though LLMs are optimized by maximizing the next token probability, probing attention patterns exclusively on the constraints can match or sometimes exceed this performance-wille hidden states or non-constraint tokens. However, attention alone does not explain all failures (we observe some attention on constraints where the model still fails), there is an opportunity for further investigation. Our findings demonstrate the value of studying the procedure by which a model produces output, rather than only the output itself.
using
5.2 EXTENSIONS
We study 3 extensions to explore the potential of SAT PROBE and propose avenues for future work. Early stopping: Using SAT PROBE, predict failures partway through the computation and
8

---

Published as a conference paper at ICLR 2024
save costs. Appendix Figures 7 shows that we can predict failures earlier in the inference with an experiment across all single-constraie datasets. Specifically, we use only attention weights up to an intermediate layer and predict failures ahead of time. For Llama-2 7B and 1 13B, we can stop the inference early without degradation in the average performance and save 50% of wall-clock time on failures for most datasets. In 70-B, early stopping results in a slight drop in performance. For use cases where we have a high RiskBonem 20% we can isolate these most unreliable predictions and abstain from making a prediction. See Appendix B.4 for details on the ablation.
Predicting partial constraint satisfaction: SAT PROBE gives access to failure predictions for individual constraints. We report the partial constraint satisfaction results in Table where we report the failure prediction metrics for individual constraints and find comparable results to the single-constraint prediction task. While SAT PROBE lets us test whether each constraint satisfied, using the raw CONFIDENCE does not since it only outputs a single value for all constraints. We believe producing fine- grained reliability statements, such as reporting partial constraint satisfaction, can prove useful for debugging (e.g., failing to follow specific instructions).
Generalized predictors: We explore using a single failure predictor across all constraint types. For this purpose, we train a failure predictor on mixture of single constraint datasets and report the performance over individual datasets in Appendix B.5 and Figure 8. We observe the performance is competitive with training individual predictors for each constraint and better than POPULARITY. This suggests the potential of pursuing efforts to construct general factual error detectors.
RELATED WORK
Carlini et al. (2021; 2022); Biderman et al. (2023) related the training data frequency of a string to memorization in LLMs. In recent work, Mallen et al. (2022); Kandpal et al. (2023); Sun et al. (2023) document the relation between the success/difficulty of factual queries and a measure/proxx for training data frequency. Several recent works investigated the mechanies of factual recall. Numerous works Elhage et al. (2021); Devlin et al. (2018); Olsson et al. (2022); Clark et al. (2019); Tian et al. (2023); Htut et al. (2019); Voita et al. (2019); Burns et al. (2022); Gurnee et al. (2023) analyzed how specific attention heads exhibit certain functionalities, such as heads that encode syntax or induction heads that copy tokens. Further, Meng et al. (2022): Geva et al. (2023) discuss the role of attention in specifically transferring factual information, and Hernandez et al. (2023) studies how specific relations can be decoded with a linear transformation from the subject tokens. However, these works do not investigate the generation of factually incorrect text. Halawi et al. (2022); Belrose et al. (2023) study how LLMs internally deal with safety-critical input, such as false demonstrations for in-context learning or prompt injections. Varshney et al. (2023) detect and mitigate hallucinations using the model's CONFIDENCE. Closest in spirit, in concurrent work Li et al. (2023) seek directions that encode 'truthfulness' in the total attention contributions across all tokens and use these to intervene on LLMs; whereas we focus specifically on predicting factual errors, showing one can use only attention weights and look only at the constraints, contribute the analyses around fine-grained constraint satisfaction and early stopping. Mündler et al. (2023); Manakul et al. (2023); Zhang et al. (2023) interact with the LLMs in black box fashion and aim to determine factual errors through inconsistencies, but doing so requires several forward passes and conveys conflicting signals such as refuting an initial claim, which can diminish user trust (Liao & Vaughan, 2023; Huang et al., 2020).
7 CONCLUSION AND FUTURE WORK
Our analysis provides initial insights into leveraging LLM's internals to understand factual errors and it raises several exciting questions for future work. We studied only conjunctive factual queries, but the class of potential constraints is much broader (e.g. instructions (Ouyang et al., 2022), disjunctive queries). While this framework extends the scope of mechanistic investigations of factuality in LLMs, it does not cover all possible factual queries to LLMs. In addition, this work assumed access to the constraint tokens; moving forward it is interesting to consider cases where we need to extract this information. The content of the information in attention patterns remains opaque and warrants further investigation. Future work could investigate how to perform different actions to fix errors, such as steering the model behavior by manipulating the attention to constraints. We hope that the methods and analyses of our study will help with the pursuit of reliable and safe generations from LLMss.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
We would like to thank Duygu Yilmaz, Marah Abdin, Rahee Ghosh Peshawaria, Ekin Akyurek, Federico Bianchi, Kyle Swanson, Shirley Wu, James Zou, Eric Horvitz, Zhi Huang, Marco Tulio Ribeiro, Scott Lundberg, Dilara Soylu for their support and comments throughout the project.
REFERENCES
Joshua Ainslie, James Lee- Thorph Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:230.. 13245, 2023.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.085 12, 2023.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. Emergent and preditabble memorization in large language models. arXiv preprint arXiv:2304.111555 2023.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc.", ", 2009.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03822) 2022.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagiclski, Ariel Herbert-Voos Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.044 04341, 2019,
Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. Lm Im: Detecting factual errors via cross examination. arXiv preprint arXiv:2305. 13281, 2023.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. arXiv preprint arXiv:2209.02533 2022.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022a.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via bise quantization. 9th International Conference on Learning Representations, ICLR, 2022b.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Ming-Wei
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/ttansffrmee-cccciiiss / html.
Yonatan Geifman and Ran EI- Yaniv. Selective classification for deep neural networks. Advances in
neural information processing systems, 30, 2017.
10

---

Published as a conference paper at ICLR 2024
Ian P Gent, Ewan Maclntyre, Patrick Prosser, Toby Walsh, et al. The constrainedness of search. In AAAI/IAA,, Vol. I, pp. 246-252, 1996.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. (0.18833/11/2021 http:///222.empmmainn..... https: ptttps://aclanthologg wtg/////// http:/222...mmmmmmme.ee.
URL
446.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. 14767, 2023.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.
Rasmus Hahn, Christian Bizer, Christopher Sahnwaldt, Christian Herta, Scott Robinson, Michaela Bürgle, Holger Düwiger, and Ulrich Scheel. Faceted wikipedia search. In Business Information Systems: 13th International Conference, BIS 2010, Berlin, Germany, May 3-5, 2010. Proceedings 13, pp. 1-11. Springer, 2010.
Danny Halawi, Jean Stanissassslas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations. Submitted to International Conference on Learning Representations 2023, 2022.
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng. Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. arXiv preprint arXiv:230 091244 2023.
Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R Bowman. Do attention heads in bert track syntactic dependencies? arXiv preprint arXiv:1911.12244 2019.
Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. Challenges in building intelligent open-domain dialog systems. ACM Transactions on Information Systems (TOIS), 2020.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to Iearn long-tail knowledge. In International Conference Machine Learning, pp. 15696-15707. PMLR, 2023.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint arXiv:2306.03331, 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.0911 2022.
Q Vera Liao and Jennifer Wortman Vaughan. Ai transparency in the age of llms: A human-centered research roadmap. arXiv preprint arXiv:2306.00 01941, 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511, 2022.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv: 230.00896.. 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 15:77359-17372, 2022.

---

Published as a conference paper at ICLR 2024
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer. Luke Zettlemoyer, and Hannameh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023.
Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. arXiv preprint arXiv:2305. 2023.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 1895, 2022.
Opendatasoft. Nobel prize, 2023. URL dataset/nobele prriierriee wwazzeeeauuuatee laureates.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744 2022.
F. Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 0909.0000000000 01066, 2019.
Amanda Spink, Dietmar Wolfram, Major BJ Jansen, and Tefko Saracevic. Searching the web: The public and their queries. Journal of the American society for information science and technology, 52(3):22--33,, 2001.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonse et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.044 04.155 04615,
2022.
Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowledge- able are large language models (Ilm)? aka will Im replace knowledge graphs? arXiv preprint arXiv:2308. 10168, 2023.
Yuandong Tian, Yiping Wang, Beidi Chen, and Simon Du Sean and snap: Understanding training dynamics and token composition in 1-layer transformer. arXiv preprint arXiv:2305. 16380, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09928. 2023.
Daniel Tunkelang. Faceted search, volume 5. Morgan & Claypool Publishers, 2009.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. Language models don't always say what they think: Unfaithful explanations in hhain-of-thoughh prompting. arXiv preprint arXiv:2305.0443 043888 2023.
Neeraj Varshney, Wenlin Yao, Hongming Zhang. Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating now-confidence generation. arXiv preprint 200.:3300.03988 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
12

---

Published as a conference paper at ICLR 2024
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Coumapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng, Fric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors, SciPy 1.0: Fundamental Algorithms for Scientific Computing Python. Nature Methods, 17:261-22 272, 2020. doi: 10. 19988//41592--19--6666
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418, 2019.
Lidan Wang, Jimmy Lin, and Donald Metzler. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pp. 105-114, 2011.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-ar natural language processing. arXiv preprint 190.07111, 2019.
Mert Yuksekgonul, Linjun Zhang, James Zou, and Carlos Guestrin. Beyond confidence: Reliable models should also consider atypicality. arXiv preprint 28521:2305.18262, 2023.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 13534, 2023.
13

---

Published as a conference paper at ICLR 2024
A MODEL DETAILS
We use the Llama-2 family models released in Touvron et al. (2023) through the HuggingFace Transformers library (Wolf et al., 2019). To fit Llama-2 70B in single A100 GPU with 80GBs of memory, we use 8-bit quantization with bi tsandbytes (Dettmers et al., 2022b;a). For of the models, we sample from the model using greedy decoding and temperature 0.
Model Name
Model ID HuggingFace Transformers
L
H
Llama-2 7B
meta meta-llamaa/llame-222111
32
32
Llama-2 13B
meta--llamaa/Llaaa-22-1333
40
40
Llama-2 70B
meta-LlanaaLLiaa-222777
80
64
Table 2: Overview of Models.
Touvron et al. (2023) reports that since key-value caches become prohibitive for larger models such as the Llama-2 70B, they use Grouped-Query Attention (GQA, (Ainslie et al., 2023)) (as opposed to Multi-Head Attention) with 8 key-value projection variant. GQA divides each query head into multiple groups, which reduces the size of the KV cache be loaded into the memory each inference step. However, this does not change our analyses which focus on the attention weights (A) or the attention contribution (a).
The MLP contribution in the Llama-2 family is computed by
mf = W/s (a + xf-
where W/. E Rdxd, where W Raddd Unlike attention, the input the MLP updates for token only uses the representation of the same token. Since we are interested in the interaction between the generation and the constraint tokens, we explore only the attention contribution in this work. It is possible that the MLP contribution has relevant signals, and we leave this exploration to future work.
B EXPERIMENTAL RESULTS EXPANDED
B.
EXPERIMENTS
In the single-constraint experiments, we evaluate the methods over the datasets and constraints documented in Table 5. For each dataset, split the dataset into normalize each feature to zero mean and unit variance using the training split. We train a Logistic Regressor with C 0.05 regularization on one subset and evaluate the performance on the other subset. We repeat this experiment with 10 random seeds and report the mean performance and the standard eтror next to it Table 5 for each dataset and method. In the Tables, SAT PROBE(A) indicates using attention weights as the input to the probe, whereas SAT PROBE(a) indicates using attention contribution as the probe. The latter is closer spirit to al. (2023) that explores the entire attention contribution through an individual head h whereas we focus on contributions from only the constraint tokens. In addition, we only look at the attention weights with the former probe, which performs better than SAT PROBE(a).
(train and test) and
B.2 MULTI-CONSTRAINT EXPERIMENTS
In the multi-constraint experiments, we evaluate the methods over the datasets and constraints documented Table 6. We follow the similar training/test split protocol as in $B. I. Additionally, when doing the train/test splits, we do not leak the same pair of constraints from the test set to the training test. Specifically, since for a pair constraints we can have two prompts (e.g. starts with e and ends with ends with r, starts with e), split samples based on the set of constraints in the prompts. In this case, factual correctness is defined as satisfying both the constraints in the query. We provide the results in tabular form in 6.
14

---

Published as a conference paper at ICLR 2024
Bosketball Pagers bom the year
Foothall Trom Souaded the
Moxiss disecte by
Courerfact Metthrr longue
Counterfact Choveship
CounterFact
performed by
Headgareer toratins
Llama-2 7B
Llama-2 13B
Llama-2 70B
1.0
AUROC
06
20
90
25
30
05
Layer Stopped
Layer Stopped
Layer Stopped
Figure 7: Early Stopping Analysis for Inference, decomposed per dataset. Here, the x-axis indicates the layer until which we use the attention weights as the failure predictor (AC. the y-axis indicates the average performance across 3 random seeds for the single constraint datasets, and the color indicates dataset. Overall, for Llama-2 7B and 13B, we observe that we can stop the inference early in the process without degradation in the average performance and save 50% of wall-clock time for most datasets.
B.3 PREDICTING PARTIAL CONSTRAINT SATISFACTION
Here, we follow the same protocol as in SB. however, we predict constraint satisfaction for individual constraints in multi-constraint queries. Note how CONFIDENCE cannot be a baseline here it simply produces probability that a completion follows a prompt, and cannot make fine-grained statements about individual constraints. We believe future work could pull this thread to reason about individual instructions, factual queries with different compositions (e.g. logical operators such as OR), and beyond.
B.4 EARLY STOPPING EXPERIMENTS
Here, we explore whether we can predict failure ahead of time to stop the execution and save computing time. Specifically, we use the attention weights up layer, e.g. A [L',[M for L' < L. We repeat this experiment across multiple choices of layers with three random seeds, and report the results in Figure 7. Overall, for most datasets, there is little to no performance drop in terms of failure prediction, even if we use up to 50% less computation time. This is less of the case for Llama-2 70B, where using later self-attention layers provides more value to the performance. We believe using the model's attention patterns could potentially be used to predict failure ahead of time, potentially abstaining from making a prediction or escalating the request to potentially a larger model e.g. similar cascading setting (Wang al., 2011).
B.5 GENERALIZING THE FAILURE PREDICTOR
Here, we explore whether we can train a single failure predictor across multiple constraints. To do so, we create a mixture of datasets by mixing data from each the relations. Specifically, for each the single-constraint datasets in Table 1, we split 50% of the dataset for training, and leave 50% of the dataset for testing. We train a Logistic Regressor on the training set and quantify the performance in the test set. We perform this with 5 random seeds, where the randomness is over the train/test splits. In Figure 8 and Table 8, we present the results. In particular, when compared to training individual predictors in Figure 6b, observe minor performance fluctuations. We believe this suggests the potential of training general-purport failure predictors using attention maps. We leave a detailed exploration of this angle for future work.
15

---

Published as a conference paper at ICLR 2024
Llama-2 2 7B
Llama-2 13B
Llama-2 70B
Basketbull Players
Data
Movies
Football Teams
SAT-Puoee
CosFineNce
POPCLAUITY
CounterFact
0.D
0.2
0.4
0.6
0.5
0.0
012
11.3
0.5
015
0.0
0.2
11.3
ULG
0.5
AUROC
AUROC
AUROC
Figure 8: Single Predictor for Constraints. We train a single failure predictor on mixture datasets and test the performance across held-out samples of the same datasets. Overall, the performance remains hetter than POPULARITY and competitive with training individual predictors, which is reported in Figure 6b.
Llama-2 7B
Llama-2 13B
Llama-2 70B
Basketball Players
Songs
Movies
Data
CoMmINs
CounterFact MT
Cusennence nure
CounterFact Cit
PormLauryy
CounterFact
0.4
1.0
Risk 20% (1t)
Risk Bottom 201
Risk Dattom 20%
(a) Results with Risknetoom0055
Llama-2 7B
Llama-2 13B
Llama-2 70B
Players
Songs
Movies
Data
Football Trams
CoMmINnD
FaouF
CounterFact
CONFIDENCE
CounterFact Gt
CounterFact HQ
0.0
0.2
0.3
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.3
0.5
0.8
1.0
Risk 20%
(1)
Risk 200
20%
(4)
Risk
2016
(b) Results with Risk
Figure Factual Error Prediction. (a) Single-constraint failure prediction results for the Riskgonom20% metric. (b) Single-constraint failure prediction results for the Risk 1p20% metric.
C DETAILS ON THE EXPLORATORYY ANALYSES
C.I PREDICTING POPULARITY
For the basketball players dataset, we filter the dataset for popularity values 2 15, since in the lower end we find that there is less meaningful signal for comparison. After this filtering, we end up with 4022 entities. We split these into two sets of equal size (training and test), and we perform Lasso regression in scikit-learn (Pedregosa et al., 2011) on the training set with 0.005 regularization strength to regress the popularity values. We then use the regressor to make predictions the held-out set and compute the Spearman rank-order correlation coefficient computed using scipy. stats spearmanr (Virtanen et al., 2020) and the p-value for testing the null hypothesis that there is no ordinal correlation.
C.2 COMPUTING ATTENTION TO A CONSTRAINT
Taking the Maximum over Constraint Tokens: Previous works report that the last subject tokens have the largest significance in terms of tracing the information (Meng et al., 2022). While
16

---

observe this to be generally the case, there are subtleties that arise for end-to-end evaluations. For instance, Kelly Oubre Jr we observe very little attention to Jr and the most attention is on Oubre. In Figure 12 we have two example prompts. Overall, due to the subtleties in tokenization Published as a conference paper at ICLR 2024
Llama-2 2 7B
Llama-2 13B
Llama-2 2 70B
Nobel Winner
CONMINNO
Books
PRouE
Words
CossrAS
0.0
0.2
0.6
0.5
1.0
0.0
0.2
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.5
1.0
Risk Bottom 20 11)
Risk Bottom 20%
(1))
Risk Buttom 20%
Rittegottom
(a) Results with Riskgetton0%%
Llama-2 7B
Llama-2 13B
Llama-2 70B
Nobel Winner
COMBINED
Books
SAT-Piooes
CONCIDENCE:
Words
0.00
0.26
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.30
0.75
1.00
Risk 20% (4)
Risk op 2016
Risky op 2010
(b) Results with Risky12
Figure 10: Factual Error Prediction. (a) Multi-constraint failure prediction results for the Riskgonon20% metric. (b) Multi-constraint failure prediction results for the Risk metric.
Model: Llama-2 70B
Model: Llama-2 13B
Model: Llama-2 7B
Popularity (Sitelinks)
100
0.67, p
Popularity (Sitelinks)
2. p 0.65, p 10-5
Popularity (Sitelinks)
p 0.66, < 7
80
60
40
20
20
40
60
20
40
60
20
40
60
Popularity Predicted from [[
Popularity Predicted from [1
Popularity Predicted from (
Figure 11: Predicting constraining entity popularity via attention. We report the results for the basketball players dataset. The x-axis indicates the popularity values predicted using attention weights and lasso regression. the y-axis indicates the ground truth popularity values, and each column belongs to a model (indicated in the title). Boxes on the top left contain the Spearman correlation and p-value for testing the null hypothesis that there is no ordinal correlation.
and potential ambiguity in the definition of what is the core part of a constraint, we choose to take the max over all constraint tokens, instead of only looking at the last token.
D DATASETS
D.I DATA CURATION FOR SINGLE- CONSTRAINT QUERIES
WikiData: For 4 groups of queries that are not from CounterFact, we probe WikiData?, For instance, for basketball players, we searched WikiData for all basketball players with at least 5 site links on the page where this criterion was done to ensure quality (entities with fewer site links did not always turn out to be professional basketball players). This criteria also avoids sampling from extremely tail knowledge, since demonstrated in Sun et al. (2023), this can be problematic for model performance and we are secking settings where the model can perform reasonably well, albeit not perfectly. Next, we retrieve the relevant field (e.g. in this case year of birth) of the entity. The choice of relevant fields for constraints was also guided by i) the need to avoid extreme tail knowledge, and ii) the feasibility of the Exact Match metric. For instance, the year of birth or year of founding can simply be evaluated
httpps:/ h/mpry:///uur.... wikidata... orgg worg//pargg sparql
17

---

Published as a conference paper at ICLR 2024
||
max, A
Tell
the
0.8
the
year
basketball
the
0.6
basketbal
player
player
Kelly
Kelly
0.4
ubre
ubre
2
Jr
0.2
was
born
in
0.0
10
20
30
10
20
30
Layer (4)
Layer (e)
||
maxp, A
Tell
the
0.8
the
year
the
year
basketball
0.6
basketbal
player
player
Kelly
0.4
aren
Jackson
ubre
Jr
0.2
was
born
born
in
0.0
10
20
30
10
20
30
Layer (E)
Layer (f)
Figure 12: Attention for players that have 'Jr' in the name. Overall, while frequently observe that there large attention on last token of the constraining entity, in practice there are many cases in which earlier tokens are more informative. the above examples, we look at attention contributions (left column) and the attention weights (right column) for Llama-2 7B with the basketball player prompts, e.g. Fig. 16.
7B 13B
7B vs 70B
13B 70B
Attention Llama-2 138
1250
Attention in llama22 70B
Attention in Llama-2 70B
1000
2000
2000
750
Both Successful
500
Only Larger
Only Smaller
1000
1000
250
Both Fail
200
400
600
200
400
600
500
1000
Attention in Llama-2 7B
Attention
Llama-2 7B
Attention in Llama-2 13B
Figure 13: Attention Mass and Model Scaling, Scatterplot version of Figure 5.
by the exact match between the correct year and the year the model produces (e.g. see queries 16,17). We chose the prompting strategy such that it well set up for the model and we can expect that the token that follows will likely contain important factual information.
CounterFact (Meng et al., 2022): We picked the three relations in Table from CounterFact as they were arguably the least vulnerable to the strictness of the exact match evaluation. Specifically, we found that there is less variation in terms of potential correct responses for mother tongue queries (e.g. Figure 20), citizenship queries (c.g. Figure 2)), or headquarter location queries (Figure 22). This is arguably visible from the model performance in Table 5 where we see nontrivial model performance albeit use Exact Match evaluation metric.
18

---

Published as a conference paper at ICLR 2024
D.2 DATA CURATION FOR MULTI-CONSTRANL QUERIES
We curated three multi-constraint datasets where it is relatively easy to perform constraint verification to label model completions for factual correctness. All of these datasets have least one correct completion for each query.
For the word queries, we iterate over letters of the alphabet, and generate prompls that contain starts with and ends with constraints for all pairs of letters, resulting in 26 x 26 676 pairs of letters in total, and upon using the permutations of constraints, we in total have 1352 prompts.
Words:
We use the Nobel Prize Winners dataset (Opendatasofi, 2023) to generate queries with at least one potentially correct completion. For each unique city of birth this dataset (645), we generate one prompt using the template in Figure 24, asking for the name of a Nobel Prize Winner who was born in a given city. Upon using the permutations, we have a resulting set of 1290 prompts.
Nobel Winner:
Books: We probe WikiData to collect dataset of authors with books. From WikiData, we scrape collection of 746 books, with a pair of authors and years publication. For each pair, we generate a query, and using the permutation of constraints, we end up with 1492 prompts.
D.3 VERIFICATION
Even though the Exact Match criterion is fairly strict, it is still a common metric used in various evaluations of LLMs from classification question answering(e.g. Liang al. (2022)). Let Y be the ground truth completion, and let Y be the model's completion, where both are sequences of tokens. We simply check whether the first |Y| tokens of the Y sequence are the same as Y. and that is the case. Otherwise, we return V(ÝÝ) = To mitigate how strict the criterion is, we tried to set the prompts up such that it is more likely that the model completes in the desired format (c.g. see Figure 17 where we include 'the year founded in' in the question, and the assistant's response starts with 'The team was founded in').
Exact Match:
return V(Ý)=
WikiData Probing:
For multi-constraint queries, the evaluation is more challenging. For instance, in query where we ask the model to return book whose author is C1 and the book was published between C2. we would like to label whether indeed there was such a book from that author, and separately whether there was book published in that year. To do so, perform the following steps:
1. First, obtain the model's response up to any new line token (In), if present.
2. We use this entity to search WikiData.
3. If there was no such entity (e.g. book), we return Vi(Ý) 0, V2(r) 0.
4. If we find such an entity (e.g. book), we individually go through the relevant fields. For instance, we retrieve the list of authors and the year the book
published.
5. Once the list is retrieved, for each constraint and list, we probe the gpt-3.5-turbo 5-turbo API with the prompt in Fig 14 whether the item is in the list. If the items are found, we return 1.
We use GPT 3.5 (gpt 3.5 t-rurbo endpoint) in the loop since Exact Match with e.g. arbitrary book names is a lot trickier than what we have in the single constraint queries (e.g. verifying the year of birth is easy with Exact Match). Since this is not a cheap step, use this strategy only for multi-constraint queries.
For the Words dataset, the constraints are simply of the form starts with the letter or ends with the letter. This is simple to verify, as it is simply checking the first character or the last character of the first word in the completion.
Character Match:
E COMPARING CONFIDENCE AND ATTENTION
In Figure 15, we investigate whether attention and confidence behave differently when predicting factual errors. Specifically, for the Football Teams and Basketball Players datasets, we make predictions on the test sets of the datasets, and plot the predicted failure probability by the attention (y- axis), and by the confidence (x-axis). We color the bins indicating which the models make a correct prediction (assuming decision threshold of 0.5).
19

---

Published as a conference paper at ICLR 2024
SYSTEM PROMPT: You are an assistant that helps the verify whether an entity captured in list. PROMPT: will give you one entity and one list, and want you to respond with "YES" if the entity is within list, "NO" it is not the list.
List: [List of Items)
Entity: (Query Entity)
Give your response in the following format:
"Reference in the list: item in the list exists, None otherwise
Answer: YES or NO' and say nothing else.
Figure 14: GPT3.5 command used for constraint verification. Once the relevant lists obtained from WikiData, we prohe GPT3.5 for verifying whether the constraining entity is the list we retrieved from WikiData.
Popularity
Sampled Players
0-10
Gur Shelef, Kimmo Muurinen
10-20
Kevin Willis, Krešimir Lončar
20-30
Bogdan Bogdanović, Candace Parker
30-40
Joakim Noah, Miloš Teodosić John Stockton, Luka Dončić Dennis Rodman, Yao Ming Kevin Durant, Stephen Curry Magic Johnson, Larry Bird R. Kelly, Kareem Abdul-Jabbar Michael Jordan, Kobe Bryant
40-50
50-60
60-70
70-80
80-90
90-140
Table 3: Basketball Players in Each Popularity Interval. Popularity indicates the number of site links on the WikiData page of the player.
We observe that attention and confidence do not always agree in predicting failures. In particular, there cases where the confidence-based predictor suggests factual error and attention-bant basedd does not, and is not a factual error (see left); or vice-versa.
20

---

Published as a conference paper at ICLR 2024
Prediction using only Attention( (Ac..T) (C.T)
Llama-2 70B on Football Teams
Llama-2 70B on Basketball Players
1.0
1.0
AttentonnAA
Prediction using only Artention (Ac,
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prediction using only Confidence (P(())))
Prediction using only Confidence(P (P(Y X))
Prediction using only Attention(AA Attention(Ac..n) (Ac..T)
Llama-2 13B on Football Teams
Llama-2 13B on Basketball Players
Prediction using only Attention(Ac..))
1.0
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prediction using only (onfidence(P((())) (P())))))
Prediction using only Confidence( (P( ())
(())
Prediction using only Attention Attention(Ac, (Ac,.T)
Llama-2 7B Football Teams
Prediction using only Attention (c..)
Llama-2 7B Basketball Players
1.0
1.0
0.8
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.2
0.4
0.8
1.0
0.0
0.4
0.6
Prediction using only Confidence(P(((1XX)
Prediction using only Confidence (otiddennee((())))))) (X)
Both Incorrect
Only Attention Correct
Only Confidence Correct
Both Correct
Figure 15: Comparing Attention- and Confidence-based Predictions. Each row indicates samples from
model. The x-axes indicate the prediction made using only confidence, y-axes indicate the prediction made using only attention. Overall, predictors do not always agree, and the disagreement becomes significant the LLMs are larger.
21

---

Published as a conference paper at ICLR 2024
User: Tell me the year the basketball player Michael Jordan was born in. The player born in
Assistant:
Figure 16: Example Prompt for the Basketball Players (born in the year) dataset queries.
User: Tell me the year the football team FC Barcelona founded in. The team was founded in
Assistant:
Figure 17: Example Prompt for the Football Teams (founded in the year) dataset queries.
User: Tell me the director of the movie Titanic.
Assistant: The director is
Figure 18: Example Prompt for the Movies (directed by) dataset queries.
User: Tell me the performer of the song Bad Romance
Assistant:
The performer
Figure 19: Example Prompt for the Songs (performed by) dataset queries.
The mother tongue of Danielle Darrieux is
Figure Example Prompt CounterFact (mother tongue) dataset queries.
Mahmoud Fawzi has citizenship from
Figure 21: Example Prompt the CounterFact (citizenship) dataset queries.
The headquarter of Monell Chemical Senses Center located in
Figure 22: Example Prompt for the CounterFact (headquarter location) dataset queries.
User: Tell me book that was written by Ernest Hemingway and that was published during 1923-1939 The name such a book is
Assistant:
Figure 23: Example Prompt for the Books (author year) dataset queries.
User: Is there a person who is a Nobel Prize Winner and who was born in the city of Cluny Assistant: Yes, person's 's name
Figure 24: Example Prompt for the Nobel Winner (won Nobel,born in city) dataset queries.
User: Is there word that starts with the letter and ends with the letter Assistant: Yes, one such word is
Figure 25: Example Prompt for the Words (starts with,ends with) dataset queries.
22

---

Published as a conference paper at ICLR 2024
Popularity Interval
Sampled Teams
0-10
Zonguldak Kömürspor, Grêmio Esportivo Novorizontino
10-20
Sorrento Calcio 1945, FK Banat Zrenjanin
20-30
Valur, First Vienna FC
30-40
Kalmar FF, FK Buduénost Podgorica
40-50
Società Polisportiva Ars et Labor, Sheffield Wednesday F.C.
50-60
Granada CF 1. FC Nürnberg
60-70
Nottingham Forest F.C.,, FK Džiugas Telšíai
70-80
Olympique de Marseille, FC Zenit Saint Petersburg
80-90
FC Porto, Club Social y Deportivo Colo Colo Real Madrid CF, Manchester United F.C.
90-140
Table 4: Football Teams in Each Popularity Interval. Popularity indicates the number of site links the WikiData page of the team.
23

---

Published as a conference paper at ICLR 2024
Ooeer tee Di
(163
0.3 4100
075-101
008 1100
0:1-8:11
111
614-1:1
1:
6:1--04 114
1:11
0100
079 1110
038 1:10
0.30 1m
76
Toobal Toru
/uodde le your
0.13
055 1000
8.51 0.00
655 1103
096 1000
0.0 0.30
6.3 5.00
1.0 1:3
030
1005
0.0 1/3
0.2 1/00
0.71 1.000
ES:11.00
Sipp
1
609
6.27
2005
038
4.000
191
Botmeall
1333
5000
0150
131
Medigare Lenchie
(144
1105
5.09 6121
033 1000
0.00 137
033
015
1007
0172 1020
026 1/00
3.50 1:100
5.03
130
(37)
1 020
0454N00
0.10 3.6
6.2 0.50
116
500
0779 11/0
1226 1/00
284
1948
103 6105
108 4143
6.6
0-5
3110
033 08E
1.10
4105
1141 6.10
5111
340
0.45
1891-00 020
109-000 100
055 LOGE
1/8 1.6
8-3
1.02
2
812--
1.4 1.000
(08 LA05
0U7 1/0
016 1/05
3.50 0.00
Seggs
1141
100 4003
003 41UL
11.0 1.
3.2
6.15-144
0.300
1300 8:00
11. 41.
116 4.00
Table 5: Predicting factual errors for single-constriint queries (1) indicates higher is better, and (4) indicates lower is better. We repeat the experiments with 10 random seeds where the randomness is over the train and test splits. We do not have the popularity numbers for the CounterFact dataset. Standard means + standard errors across 10 random seeds are reported in each cell.
Misicl
Doi
Choonniai
Madk Seours
Roke
AUROCIE)
CONSTANT
COMMDENCT
COMSTANT
111
Tootss
Overat
[.3]
0:96 1.:0
1150: 11:0
1/00 0.00
0.80 11.01
1.0 0.00
D.71 1.00
0.77 (::01
0:95 0.50
0181 1.001
0.76 1.000
0.75 1..0
0.77 [:0)
0.75 1.
0.30 0/11
0.52 (1
Notal Whnse
Obendl
16.00
1155 (1:11)
0:48 0100
0185 11:00
1.:0
01:41
1100 1::0
0.45 11:1
1..0
01.9
11
0.30 0/10
11.7%
24
133
Books
Oscral
030 0.80
(.:0:200:1
1.00:2000 0.00
1 0.38
0,50 20.00
100
0.00
0.83 0.00
0.50-233.01
6.86 0..0
0.5)2:30 2:3.1
0.5.200.)
0.70:20.00
0.82 5/00
0.55.210 0:00
0.302000
6.57220.00
Notull Wanser
Oseral
[.3]
1.80
1157 (1:0
0:98 0.30
0.94 18:00
038
1.80
18000
0186 m.m0
0.41 (1:01
011 1:0
0.00 040)
01:1 1.0
0.30 8/90
0.78
708
Books
Ovcral
= 0.80
0.50 0.00
0.07 0.38
0,57.20000
038 (.8)
0.00
0.562:33.0
6:6 0.00
0.57
0.3:2000
0.5:2000
0.58 6/0
0876 111-2-00 0:00
0.30 8:00
0.56:20011
7000
Natull Wazsur
Oxer
(.31
0.01
0191 01
0:41 0.00
B.1 1100
1.:0
0.07 m.00
0163
081 1:00
0:99 1am
0:000:0000000
0.71
15.5 11
0.30
0.75
Table 6: Predicting factual errors for multi-constraint queries (1)) indicates higher is better, and (4) indicates lower better. We repeat the experiments with 10 random seeds where the randomness is over the train and test splits. Standard means + standard errors across 10 random seeds are reported each cell.

---

Published as a conference paper at ICLR 2024
Model
Data
Constraint
Model Success
Riskgaton 200 (1)
Riskin (3)
AUROC(()
SAT-PRORIIS
SAT-PROBECN
CONSTANT
SAT-PRORE())
CONSTANT
SAT-
SAT-PRORMAA
CONSTANY
Books
author
0.38
0:95 0.01
0.97
0.61 + 0.01
0.16+
0.25 0.01
0.01
0.85 0.00
0.84 0.00
0.50 0.00
7B
Books
publiahed your
0.11
0.97 : 0.00
0.98 = 0.00
0.90 0.01
0.81 0.01
0.79 = 0.01
0.90 2: 0.01
0.68 2 0.01
0.71 :: 0.00
0.50 = 0.00
78
Nobel Winner
born city
0.0G
0:95+:000
0.97 0.01
0.94 + 0:01
+.00
0.86 = 0.01
0.01
0.57 +.001
0.68 0.01
0.50 0.00
7B
Nobel Winner
won Nobel
0.62
0.58 4 0.01
0.66 = 0.01
0.12 0.01
0.21 0.00
0.11 0.01
0.12 0.01
0.63 0 0.01
0.72 2 0.00
0.50 0.00
7B
Words
ends with
0,23
0.84 0.01
0.90 + 0.01
0.77 + 0:01
0.00
0.69 + 0.01
0.77 + 0.01
0.58 + 0.01
0.62 + 0.00
0.50 = 0.00
Words
starts with
0.93
0.01
0.27
0.01
0.00
0.01
0.12
0.85 1
0.00
13B
Books
autler
0.25
0.96 0:00
0.98 = 0.01
0.71 0.01
0.39 0.00
0:33 = 0.02
0.71 0.01
0.77 0.00
0.87 2.00
0.30 = 0.00
138
Books
published year
0.08
0:98 + 0.00
0.99
0.91 + 0.01
0.00
0.96 + 0.01
+ 0.01
0.68 0.01
0.70 0.00
0.00
138
Nobel Winner
born in city
0.08
0.96 0.02
0.96 = 0.01
0.94 0.01
0.86 0.00
0.86 = 0.01
0.94 + 0.01
0.67 0.03
0.66 +: 0.02
0.30 0.00
13B
Nobel Winner
won Nobel
0.64
0.73 4: 0.01
0.72 +: 0.01
0.44 0.01
0.14 0.00
0.11 = 0.01
0.44 1: 0.01
0.75 0.01
0.77 1 0.00
0.50 4: 0.00
Words
ends with
0.27
0.83 0:01
0.83
0.79 0.02
0.01
0.02
0.61
0.00
13B
Words
starts with
0.80
0.45 1: 0.01
0.44 J: 0.02
0.19 - 0.01
0.01 0.00
0.01 -: 0.00
0.19 1 0.01
0.92 0.00
0,93 1 0.001
0.30 0.00
703
Books
outhor
0.32
0.905 1 0:00
0.336 = 0.01
0.73 0.01
0.25 0.00
0:21 0:01
0.73 1 0.01
0.52
0.85 0.00
0.30 0.00
708
Books
published year
0.16
0.98 0.00
0.98 0.00
0.87 + 0.01
0.61 0.01
0.56 = 0.01
0.87 + 0.01
0.78 0.01
0,82 + 0.01
0.50 0.00
70B
Nobel Winner
born in city
0.16
0.94 =: 0..00
0.95 = 0.01
0.83 : 0.01
0.66 1 0.01
0.66 = 0.01
0.53 1: 0.01
0.71 2 0.01
0.72 4: 0.01
0.30 = 01000
Notel Winner
Nobell
0.71
0.01
0.38
0.28 0.01
0.01
0.01
0.00
0.00
70B
Words
ends with
0.31
0.88 1: 0.01
0.90 A: 0.01
0.82 1: 0.01
0.38 0.01
0.37 -: 0.01
0.82 1: 0.01
0.73 0.01
0.75 1: 0.00
0.30 2: 0.00
70B
Words
storts with
0.85
0.45 + 0.01
0.47 + 0.01
0.28 + 0:01
0.01 0.00
0.00 + 0.00
0.28 + 0.01
0.87 0.00
0.88 + 0.00
0.50 0.00
Table 7: Predicting the failure to satisfy individual constraints in multi-constraint settings. (1) indicates higher is better, and (4) indicates lower is hetter. Here, we train individual failure predictors for each constraint and give the failure prediction metrics across all models, datasets, and constraints. Standard means + standard errors across 10 random seeds are reported in each cell.
Mudic
Consinke
Makt Seacos
Hoin-vooE)
ALBOC
LOSTELENCE
[ SAB PSOEDA)
SAS-P50000
CONSTANT
POFULARITY
COSTBDECC
SAU-FRCIOA/
SAE-P5080a)
UNNSTART
PNPULARTTY
CONTIDENCE
SAU-FROEDA)
SAT-FROEDAI
LONSTANT
PNPELARITY
Backetulll Plosar
Aon die
0.35
0198 11..00
0
[1.4 13.00
335 C.0
DU2 11.33
E.4. E/1l
1..7.111.33
0.33
1100
0.52.1110
0,00:2 2.00
0.13
2:10
24.
OVtoniky
011
08:5--10 1:133
1100-9 (111
0.58
11
11.99 0100
135--00 00
01:1 (1111
0.00 - 5:11
73
Connethac
Moeker Zegur
0.45
0899 0.33
1/802 500
0129 11.03
355 3.00
0.38 1.
1/0-2 E/11
0.52 1100
0.3 10
0,89 1.101
0.00 0 2/00
[002 0.133
01224:100
75
Meniss
0358
11100
1/012
11:33 11.03
0-42=000 111
0/22 11.03
11.0
110 618)
11.01
81
01.09 1110
0191 1100
01212 310
11133
11:0 2:11
Singa
110
0-0--00
138
Tactenuh Pluare
0.55
11:30
11:33 11.03
2.0
1150
1-3 000
1130 11.55
81
1153 01
03.5 2:300
118
0.10
0100 0222
11:33 1.101
3:43 1.66
0151
1.31 010
0133 11:0
0.43- 01
0.59 11.01
012 3:00
11.9 11131
0.52-1310
176
Connethaci
Sdwher Sngur
0148
1191 11.25
12012 0100
11.77 2-100
335 010
11.0000 11.5
118-2 1111
11.77 11100
0.002000 12
1100 11000
0.00 2101
113-2.11331
118020310
118
Morie
0.71
01.1 11.0
0100 0.00
01:0- 01.01
3:20 110
0-12 6-00-000 15.00
0131 1150
1110 600
19.00
1.20-13 80
0119 11.0
0.87 (1:00
0.71 2100
0.70 03.3
01:22 -010
175
Sengo
0.24
3:79: 0.13
0,09 11.5
0.55
1100
11.0
0.0.2 2140
0.83 0:30
10
318
016
1130
1100 6177
-1
1.15 6:11
11:0
0.26 11
011 5:01
0120 10:3
0155-131 10
328
Connaihac
Moeker Zegur
0.45
1193 0.23
2 500
1.30 1.000
355 210
11.5 1..5
0655 1.00
033 10
0162 11.00
010 310
1155 0.330
0.82 2010
388
Minis
0.56
11.30
6-5-2 60
3.14 10
0.12 10.00
0133 0.00
L10 600
0139 11000
81
0.00
0.5.2-003
0.07 110
0/90 0:331
0180 2-1
766
Sep
0.44
0.00
0:00
1.0
0.00
027 600
0.00
510)
0.030
0.51
Table 8: Using a single predictor for all constraints in single-constraint queries.(ft) indicates higher is better, and (4) indicates lower is better. We repeat the experiments with 10 random seeds where the randomness is over the train and test splits. We present the standard errors next to the average results across all experiments. We do not have the popularity numbers for the CounterFact dataset. Standard means - standard errors across 5 random seeds are reported in each cell.


Reviews:
[{"review_id": "6jOa1jt3Te", "reviewer": "Reviewer_KSCt", "summary": "This work proposes to use Transformers’ internal attention mechanisms to understand the internal mechanism behind factual errors, in contrast to previous work focusing more on discovering internal mechanism behind generate *correct* factual answers. It poses the problem of answering factual queries as a constraint satisfaction problem whereby each factual queries presents a conjunction of constraints (e.g. “is a basketball player” and “was born in 1988”) and the query is answered correctly if each constraint is satisfied. The paper begins with a series of analyses linking attention on the constraint tokens with factuality, then proposes the SAT probe, a linear classifier over the attention on the constraint tokens, trained to predict when all constraints will be satisfied (and thus when the LLM will make factually correct predictions).", "questions": "1. Are there any takeaways from this analysis and method for how we can actually fix factual errors? Or perhaps prevent models from generating factual errors?\n\n2. This method mainly relies on LM uncertainty to pick up when the LM is wrong, which is fine (I’m not sure if there’s any other way to internally discover when the LM is lying without external tools). However, broadly speaking, are there ever cases of the LM being confidently wrong? Or is it the case that, because of the distribution of the pre-training data, whenever there’s factual errors it will always be much less certain than when there’s correct answers?\n\n3. Section 5.2, Early Stopping: “For Llama-2 7B and 13B, we can stop the inference early without degradation in the average performance and save 50% of wall-clock time on failures for most datasets.” Is there a breakdown of wall-clock time for each of the datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Casting factual queries as a constraint satisfaction problem is an interesting approach, and the specifically the finding that attention on the constraint tokens is correlated/predictive of factual correctness is novel.\n\n2. Compared to confidence-based classification, the SAT probe is more efficient — allowing the model to not have to go through the entire inference procedure — and fine-grained — allowing us to isolate of which exact constraint was disregarded.\n\n3. The experiments covered many different datasets and a wide range of model sizes — and the findings generally carried over between the various model sizes and datasets. The wealth of empirical findings is insightful and convincing.", "weaknesses": "1. My main point of contention is that the SAT probe actually underperforms confidence classification by a statistically significant margin in 17/27 of the settings studied in in Figure 6 (it would be good to note what type of error bars are being plotted and how many trials this was over/what the source of randomness was). This seems to undermine “we find that SAT PROBE mostly performs comparably to and sometimes better than the model’s CONFIDENCE in the correctness prediction task” (section 5.1, results).\n    1. Section 5.2 and appendix dive into results that demonstrate that analyzing attention at earlier layers can be useful — as it is generally more efficient and allows us to interpretably isolate the constraint that failed. This should be moved up and highlighted as one of the main advantages of this method. (The preliminary analyses on popularity and constrainedness predicting performance / attention predicting popularity and constrainedness can be compressed.)\n    2. Table 7 examines the ability of SAT probe to isolate *which* constraint was violated when there are multiple constraints. Some more detail is necessary here: in the dataset, when 1 constraint is violated, how often is it the case that both constraints are violated? Can we isolate the experiment to cases where one and only one of the constraints are violated, and have the SAT probe predict which one?\n\n2. I do wonder about the generality of thinking of factuality as a CSP — in particular simply treating it as a conjunction of constraints. There are many other types of composition, such as disjunction, nesting (multi-hop queries like “parents of the President of the USA”), etc., that cannot be dealt with in this framework. Furthermore, not are all factual queries can be dealt with just simply using constraints. For example, there may be additional reasoning operations on top of constraints (e.g. numerical counting). While I don’t expect this one paper to comprehensively deal with all types of factual queries, perhaps some discussion of the coverage of this framework is warranted.\n    1. More generally, prior work has already examined factual queries from the lens of database (SQL) queries, which implicitly already carries constraints (i.e. the WHERE clauses) while also being more flexible to different types of constraint compositions, and many more operations. What is the comparative advantage to thinking of factual queries as conjunctions of constraints in this way?\n\n3. Furthermore, the name “constraint satisfaction problems” generally suggests a different set of problems to me — where the key focus is not on evaluating the factuality of each atomic variables, but on how we may search through assignments to variables in order to create a permissible solution. (This is a relatively minor naming point, though does open up a potential question on the implication of thinking of queries as a constraint satisfaction problem, in the more generic sense, which could introduce much more novelty to this paper.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LOnu36wBbh", "reviewer": "Reviewer_m32R", "summary": "This work focuses on investigating characteristics of attention patterns when LLM makes factual errors or hallucinates. They show that entity tokens (defined as constraining tokens) in the query input exhibit strong correlation with the answer entity tokens. The authors had three key observations 1.) the more popular an entity-relation pair is (i.e., appears more frequently in corpus), LLM is less likely to be factually incorrect about it 2.) attention scores can serve as a way to measure LLM confidence and correctness and 3.) the larger the LLM the more the attention scores and attention scores correlate in a similar way across different size models.", "questions": "1.) The observation that \"when the LLM is accurate, there is more attention to constraint token\" suggests that when LLM is confident it is often right. How does this generalize in cases shown in prior work when LLMs confidently hallucinate? It would seem that attention scores would be high even when hallucinating. In your experience, is there any way to differentiate between attention scores (or some other variable) for correct vs incorrect confident answers.\n\n2.) In Figure 5, it looks like attention is summed across layers, its unclear if the final score was normalized by number of layers? Is it possible that attention scores are more comparable between 7B and 13B as compared to 7B and 70B because of the number of layers scaling?\n\n3.) Were there any insights as to how attention patterns vary across different layers, especially for different sized models? for e.g., higher attention scores for hallucinating tokens towards the later layers perhaps?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.) This work looks at mechanistic interpretability especially with respect to factor errors made by LLMs, which has not been studied before as per my knowledge.\n\n2.) They do extensive analysis and show the effectiveness of their probing technique across different datasets and models.\n\n3.) SAT-probe can also help predict failures by analyzing attention scores mid-training, which can be useful of debugging purposes.", "weaknesses": "None I can think of", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cbWPcV04A9", "reviewer": "Reviewer_MD6P", "summary": "This paper presents an interesting finding that the magnitude of attention contribution to “constraint entities” (something like key entities associated with the prediction) is well-correlated with popularity of the constraint entity, the constrainedness of the query, and more importantly, the correctness of the LLM prediction. The authors define metrics to quantify such attention contribution and expect to use the metric to predict factual errors of LLM predictions. On experiments across multiple datasets, the proposed method demonstrates comparable factual error prediction performance to the confidence (i.e. logits) baseline.", "questions": "1. In section 5.1 the authors mention fitting Logistic Regression, does that mean you need a training dataset with labels to learn the parameters w and b? If so, how large is the training data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper presents interesting and novel findings of a phenomenon in the hidden representations of LLMs, that the magnitude of attention contribution is correlated with factualness of the prediction. It may inspire future researchers to discover and probe more interpretable inner patterns of LLMs, which could potentially help researchers understand and explain models’ behaviours.   \n2. The authors perform comprehensive analysis to verify the correlation of attention contribution with various properties such as popularity, constrainedness, and factualness. These analyses are convincing and the different types of visualization are informative and efficient to convey the messages.  \n3. The paper is very well-written.", "weaknesses": "1. Although the findings are interesting to know, I doubt the practicality of the proposed method and the contribution it makes from a practical perspective:  \n    a. Attention contribution is computed assuming access to the constraint entities, which are not available in most of cases  \n    b. The proposed method is not better on factual error prediction than the simple confidence baseline. It seems to me it does not bring any new information beyond the confidence score\n\n2. I think this paper focuses on and only studies a narrow notion of factuality, where constraint entities exist as in a world knowledge domain. It is unclear whether attention contribution is a valid metric to explain other factual errors, such as mathematical errors and inference errors where the traditional notion of “entities” may not always exist. This is important since researchers talk about factuality and hallucination actually in a quite broad scope in the LLM context nowadays, not limited to entity-based knowledge.\n\n3. From Figure 14-23, all the datasets look simple and toy. Although it appears that the experiments are conducted on a diverse range of datasets, these datasets turn out to be very similar style-wise – they are very similar QA tasks with one-entity answers. While I understand that it is difficult to perform experiments on more realistic data since the constraint entities will become unavailable, I am not sure how general the proposed findings are for LLMs given these toy datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)