AttributionBench: How Hard is Automatic Attribution Evaluation? (2024)

Abstract: Modern generative search engines enhance the reliability of large language model (LLM) responses by providing cited evidence. However, evaluating the answer’s attribution, i.e., whether every claim within the generated responses is fully supported by its cited evidence, remains an open problem. This verification, traditionally dependent on costly human evaluation, underscores the urgent need for automatic attribution evaluation methods. To bridge the gap in the absence of standardized benchmarks for these methods, we present AttributionBench, a comprehensive benchmark compiled from various existing attribution datasets. Our extensive experiments on AttributionBench reveal the challenges of automatic attribution evaluation, even for state-of-the-art LLMs. Specifically, our findings show that even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation. A detailed analysis of more than 300 error cases indicates that a majority of failures stem from the model’s inability to process nuanced information, and the discrepancy between the information the model has access to and that human annotators do.1

heading: 1 Introduction
text: The advent of large language models (LLMs) has revolutionized the field of information retrieval and grounded text generation (Brown et al., 2020; Wei et al., 2021; Chowdhery et al., 2023; Ouyang et al., 2022; Peng et al., 2023), leading to the development of advanced generative search engines like Bing Chat, Google Bard, and perplexity.ai. These platforms excel in generating search results in a natural language format, along with references to the source web pages as evidence to support the truthfulness of the generated response (Dziri et al., 2022a; Ji et al., 2023). However, whether the evi-
1Our code and datasets are available at: https://github.com/OSU-NLP-Group/AttributionBench
dence supports the generated responses or not, also known as the attribution of the response to the evidence, remains an open problem.
Many efforts (Liu et al., 2023; Kamalloo et al., 2023; Malaviya et al., 2023) have recently been made to conduct human evaluation to examine the performance of attribution of various advanced systems like Bing Chat and GPT-4. It turns out such systems often produce attribution errors, making them less faithful and trustworthy for practical use. However, human evaluation is expensive and time-
14919
consuming.2 Therefore, there is a pressing need for efficient and effective methodologies to automatically assess attribution and detect possible errors.
Towards this end, there have been efforts made to build automatic attribution evaluation models. Rashkin et al. (2021) proposed the framework of Attributable to Identified Sources (AIS), i.e., evaluating whether model-generated responses can be verified against given references. Under this framework, several approaches were proposed to build the automatic evaluator, including directly utilizing natural language inference (NLI) models (Gao et al., 2023a; Bohnet et al., 2022), directly prompting LLMs like GPT-3.5 and GPT-4 (Yue et al., 2023), and fine-tuning smaller LMs like FLANT5 (Chung et al., 2022) with repurposed data from related tasks like NLI, fact-checking, and summarization (Yue et al., 2023). However, these studies define the attribution evaluation task differently (i.e., using different numbers and types of classification labels) and choose different datasets to train and evaluate models, making it impossible to directly compare their performance and derive significant insights into the challenges of attribution evaluation and how to address them in future work.
To this end, we take the first step to present a systematic benchmark, AttributionBench, for training and evaluating cutting-edge automatic attribution evaluators. Specifically, we meticulously sample data from 7 different datasets (Malaviya et al., 2023; Liu et al., 2023; Bohnet et al., 2022; Chen et al., 2023; Dziri et al., 2022b; Yue et al., 2023; Kamalloo et al., 2023) that cover different domains of questions and diverse responses and evidence. We unify them into a binary classification format with a label-balanced setting for fair comparison3. We compile them into a training set and two test sets for in-distribution (ID) and out-of-distribution (OOD) evaluation, respectively. Table 1 shows the statistics of our benchmark.
We conduct extensive experiments and analysis on our proposed benchmark. Surprisingly, we find that even fine-tuned GPT-3.5 can only get around 80% macro-F1 score under both ID and OOD settings, which is far away from practical use. To better understand the challenges of this task, we
2According to Liu et al. (2023), the cost of annotation is around $15 per hour, yielding around 15 data examples.
3There can be more challenging problem formulations; but we will show that even if in this relatively simple formulation, there is still a large room for models to improve.
carefully labeled over 300 error cases from GPT3.5 under chain-of-thought (CoT) prompting (Wei et al., 2022), which generate rationales for the model’s prediction that can reveal reasons for an error. We find that 1) over 66% errors are caused by the model’s insensitivity to fine-grained information (i.e., ignoring or overlooking details like facts, events, numbers, or lack of necessary summarization and inference from the evidence, tasks that humans naturally perform), and 2) about 26.8% of the errors are caused by the mismatch between information accessible to the model and that accessible to human annotators (see examples in Figure 1). This is because the references given to the models are manually extracted from web pages by human annotators based on what they think is useful. However, when making the judgment, human annotators may be inherently affected by additional content since they’ve seen the whole webpage. On the other hand, sometimes the supported evidence spreads across the entire webpage, making it impossible to locate a short snippet to serve as the evidence. These findings provide valuable insights into 1) the challenges for automatic attribution evaluation, 2) the challenges for humans to evaluate attribution, and 3) how to develop stronger automatic attribution evaluation models.
We summarize our contributions as follows:
Benchmark. We propose AttributionBench, a comprehensive benchmark with a unified formulation for attribution evaluation, which will enable the community to compare different methods fairly and track the progress on this important task.
Methods. We conduct comprehensive experiments and show that existing cutting-edge LLMs like GPT-4 and fine-tuned GPT-3.5 still cannot perform well on this task.
Analysis. Through a series of in-depth error analyses, we show insights into why automatic attribution evaluation is difficult and potential future work.

heading: 2 AttributionBench
text: In this section, we introduce our task formulation and data collection pipeline for constructing AttributionBench.

heading: 2.1 Task Formulation
text: We define the task of claim-level attribution evaluation as follows: Given a natural language query Q and a response R = {c1, c2, ..., cn} generated by LMs, comprising several claims ci. There is also an evidence set E = {e1, e2, ..., em} along with each claim The task is to judge whether a claim is supported by its accompanying evidence. The claimlevel evaluation model, denoted as a classifier f(·), is designed to take ⟨(Q), ci, (R), concat(ei)⟩ as input. Here, the inclusion of the question Q and the response R in the input is optional since the objective is to assess whether the evidence {ei} substantiates the claim ci, and hence we use (Q) and (R); concat(ei) represents the concatenation of all corresponding evidences eij for claim ci Other information fields, if included, serve as supplementary data aiding the attribution evaluation task.
In previous work, the label space was defined differently, including binary classification (“Y”, “N”) (Bohnet et al., 2022), three-way classification (“Support”, “Partially Support”, “No Support”) (Liu et al., 2023) or (“Attributable”, “Contradictory”, “Extrapolatory”) (Yue et al., 2023). To simplify and unify the task format, we formulate the task as a binary classification task, with the labels being (“Attributable”, “Not Attributable”).

heading: 2.2 Data Collection
text: Datasets. To make our benchmark more comprehensive, we collect data from several recently
proposed attribution datasets, including ExpertQA (Malaviya et al., 2023), Stanford-GenSearch (Liu et al., 2023), AttributedQA (Bohnet et al., 2022), LFQA (Chen et al., 2023), BEGIN (Dziri et al., 2022b), AttrEval-GenSearch (Yue et al., 2023), and HAGRID (Kamalloo et al., 2023). These datasets contain different query sources, various domains, and are of varying difficulties. We list the basic statistics in Table 1, and more detailed statistics in Appendix Table 5.
Label-Balanced Setting. We select data samples from each contributing dataset and adopt a labelbalanced setting in train, validation, and test set to reduce the effect of unbalanced labels on training and reliable evaluation of each label. Such a balanced training set eliminates the potential negative impact brought by imbalanced labels, and a balanced test set treats two classes equally and enhances the representativity of the F1 score.
In-Distribution & Out-of-Distribution Test Sets. To measure both the effectiveness of fine-tuning and generalizability, we present both in-distribution and out-of-distribution test sets. We mainly consider 1) data scale and 2) there should be no question overlap between in-distribution (ID) and outof-distribution (OOD) test sets, and pick ExpertQA, Stanford-GenSearch, AttributedQA, and LFQA as in-domain tasks, leaving BEGIN, AttrEvalGenSearch, and HAGRID as out-of-domain tasks.

heading: 3 Exerimental Setup
text: Datasets. We fine-tune models with the training set of AttributionBench and use the ID and OOD test set of AttributionBench for evaluation.
Input Fields. By default, we only include c and E as input, ignoring Q and R, as this setting shows the best performance preliminarily on GPT-3.5 (section 4.2).
Compared Models. We include multiple types of models in our main experiments. (1) Decoderonly models: Llama-2 (7B) (Touvron et al., 2023), AttrScore-Alpaca (7B) (Yue et al., 2023), GPT3.5 (OpenAI, 2023) and GPT-4 (Achiam et al., 2023). (2) Encoder-decoder models: FLAN-T5 (770M, 3B, 11B) (Chung et al., 2022), AttrScoreFLAN-T5 (3B) (Yue et al., 2023), T5-XXL-TRUE (11B) (Honovich et al., 2022), and FLAN-UL2 (20B) (Tay et al., 2022). (3) Encoder-only models: roberta-large-mnli (340M) (Liu et al., 2019).
Note that, roberta-large-mnli and T5-XXL-TRUE are fine-tuned on natural language inference (NLI) data, and AttrScore-FLAN-T5 (3B) and AttrScoreAlpaca (7B) are fine-tuned on data repurposed from similar tasks including NLI, fact-checking, QA, and summarization. Each model (except GPT-4) is tested via both zero-shot prompting and fine-tuning. The implementation details are listed in Appendix A.

heading: 4 Result Analysis

heading: 4.1 Main Results
text: Tables 2 and 3 show the overall results of different models on ID and OOD test sets. Our primary findings are as follows:
Fine-tuning on NLI-related data is beneficial to attribution evaluation. T5-XXL-TRUE and AttrScore-Flan-T5 (3B) are fine-tuned on data including NLI and achieve strong performance on both ID and OOD sets. Specifically, both models outperform fine-tuned GPT-3.5 for the average F1 score on OOD sets and achieve comparable perfor-
mance with fine-tuned GPT-3.5 on OOD sets. Also, through further fine-tuning on AttributionBench, AttrScore-Flan-T5 (3B) outperforms the original Flan-T5 (3B) on 4 ID and 2 OOD sets. For T5XXL-TRUE, it surpasses FLAN-T5 (11B), a model with the same amount of parameters and additionally trained on instruction data, on 2 ID and 2 OOD sets. These results indicate the effectiveness of training on NLI data. A possible reason is that the attribution evaluation task shares a similar ability with solving NLI tasks, as both of them need the model to judge the entailment (being supported) of the hypothesis (claim) against the premise (evidence).
Automatic attribution evaluation is challenging under zero-shot setting. Attribution evaluation is a complex task because it not only requires a comprehensive understanding, necessary summarization, and inference of the referenced information but also requires a fine-grained comparison with the content of the claim. Moreover, this requirement criteria may vary considerably across different datasets and testing conditions. In
this challenging landscape, models often demonstrate suboptimal performance in zero-shot scenarios. For tasks with shorter claims and evidence like AttribtuedQA and AttrEval-GenSearch, the performance of GPT-3.5 and GPT-4 are over 80%, while for tasks with longer evidence (like StanfordGenSearch and HAGRID), the zero-shot performance for models is around 60%~70%, which is relatively low. For the challenging task ExpertQA, which consists of domain-specific challenging questions, GPT-3.5, GPT-4 and those NLItuned models can only achieve under ~60% performance.
Fine-tuning on AttributionBench benefits both ID and OOD evaluation. Despite the task’s challenge, the performance can be enhanced through fine-tuning on the training set of AttributionBench. On average among all models, fine-tuning can improve 9.0% and 4.6% on 4 in-domain tasks and 3 out-of-domain tasks, respectively. Additionally, with training on only 13k
examples, the fine-tuned FLAN-T5 (770M) model even surprisingly outperforms GPT-3.5 (w/ CoT) on both ID and OOD sets, indicating the effectiveness of fine-tuning on AttributionBench. Furthermore, almost all models obtain performance gain on 3 OOD sets via fine-tuning, indicating that the models did not just overfit the ID data, but also gained generalizability to solve this task on OOD examples.
Simply switching stronger models cannot significantly improve the performance. First, under zero-shot setting, GPT-3.5 and GPT-4 show their expertise in dealing with difficult tasks like ExpertQA, a dataset containing hard domain-specific responses and evidence. Nevertheless, on more other tasks like AttributedQA and LFQA, they still underperform smaller models like T5-XXL-TRUE and FLAN-UL2. Under fine-tuned setting, for the 4 ID sets, GPT-3.5 shows competing performance but still underperforms smaller models including FLAN-T5 (11B) and Flan-UL2 (20B). For the 3
45
55
65
75
GPT-3.5 (w/o CoT) GPT-3.5 (w/ CoT)
C+E Q+C+E C+E+R Q+C+E+R
Figure 2: The average macro-F1 score on 7 test sets of GPT-3.5 with different input fields. Q, C, E, R stands for question, claim, evidence, and response, respectively. Results show that despite involving additional information, adding input fields cannot boost or even harm the performance.
OOD sets, the performance of GPT-3.5 on AttrEvalGenSearch and HAGRID is still lower than many models including T5-XXL-TRUE, AttrScore-FlanT5 (3B), and most surprisingly, Flan-T5 (3B) and Flan-T5 (770M). Although it is hard for one model to do well among all datasets with various data distributions and different classification criteria, both zero-shot and fine-tuning results indicate that simply scaling up and fine-tuning models on ID data isn’t the solution to producing a strong attribution evaluator model for practical use.

heading: 4.2 The Impact of Input Fields
text: Figure 2 illustrates the impact of incorporating both questions and responses within the model input. The result reveals that merely appending the questions and responses to the input, which human annotators might have access to, is not the key to solving this task. Despite providing more contextual information, both adding Q and adding R do not improve the performance. By looking at error cases, we find that although questions introduce essential elements that could influence the model’s judgment, they could also potentially mislead the model by confusing the attribution task with an assessment of “answer usefulness” instead of attribution. Specifically, in instances where the evidence substantiates the claim but fails to precisely address the question, the model erroneously dismisses it as “not attributable”. This issue is similar when adding the response field, as such amplification of context could exacerbate the model’s confusion regarding the specific details within the input text.

heading: 4.3 The Impact of Prompts
text: It is well known that the choice of prompts can lead to different performances. Here we investi-
0
20
40
60
80
base breakdown breakdown +detail_check breakdown +detail_check +comp_und
F1 Acc
(a)
924 942 903 950
0
200
400
600
800
1000
1200
base breakdown breakdown +detail_check breakdown +detail_check +comp_und
False Negative Cases False Positive Cases
(b)
Figure 3: The performance of GPT-3.5 (w/ CoT) with several different prompts. Prompt engineering only brings limited gain over 7 test sets. Rather than bringing little gain in overall performance (Fig 3a), adjusting prompts is actually changing the ratio of FP and FN cases (Fig 3b). “comp_und” stands for “comprehensive understanding”.
gate the impact of instructions using GPT-3.5 with CoT. Figure 3 shows that prompt engineering only brings limited gain for GPT-3.5 over 7 test sets. Using the base prompts, we observe that there are more false positive cases than false negative ones, which means the model tends to be positive about the attribution and predict a data sample to be attributable. Therefore, we carefully constructed several prompts: (1) breakdown and verify stepby-step, (2) on top of (1), force the model to be more strict and check every factual detail within the claim, and (3) based on (1) and (2), besides carefully checking details, the model is also required to do necessary inference and summarization to comprehensively understand the semantics of both the claim and the evidence. The detailed prompts we used are listed in Appendix Table 6. The results show that although prompt engineering can bring some marginal gain in accuracy and F1-score, it is essentially changing the classification interval and making trade-offs between false positive and false negative cases.

heading: 5 Error Analysis
text: After extensive baseline assessments and ablation studies, we sought to pinpoint the major challenges for the task of attribution evaluation. We conduct a detailed qualitative error analysis using GPT-3.5 (w/ CoT) by randomly sampling 50 error cases (or all of them if fewer than 50) for each dataset, which results in 321 instances in total. For ExpertQA, Stanford-GenSearch, BEGIN, and HAGRID, we pick 25 cases for both false positive and false negative examples and label all the error cases for AttributedQA, LFQA, and AttrEval-GenSearch. We show representative examples in Table 1.
Our first observation is that over 66% error cases are caused by fine-grained information insensitivity. This means when making a wrong judgment, the model is most likely to fail in comparing fine-grained information within the claim and the evidence. Here, “fine-grained information” encompasses a wide array of specifics, from discrete data points such as numerical figures, dates, names, and locations, to more complex elements like particular events or logical connections. Moreover, within this category of error, the model also makes different kinds of mistakes regarding its label error class, i.e., false positive or false negative. In the case of false positives, the model misses certain details within the claim 43% of the time , and in 41% of the instances, it made incorrect connections and inferences to indicate the claim being supported by the evidence, which is actually wrong. Conversely, within the false negative category, the model predominantly misinterprets or neglects details in the evidence around 46% of the time, and 39% of them are because of lacking necessary inference and summarization to reach the attribution judgment, which could be very easy for humans.
About 26.8% errors are caused by the mismatch between information accessible to the model and that accessible to human annotators. This disparity brings into question the reliability of labels within current attribution evaluation datasets. Our analysis suggests that, in certain cases, determining the label based solely on the provided claim and evidence can be challenging due to insufficient information or ambiguous references (error type: need additional information/reference ambiguity). At times, this can even lead to the annotated label being wrong (error type: label mismatch). However, this doesn’t mean that nearly a quarter of the
labels are wrong. According to the current annotation process (Liu et al., 2023; Malaviya et al., 2023), human annotators usually assess claims sequentially, which allows for additional context to inform their judgment on subsequent claims after reviewing the preceding ones. Furthermore, while the models receive manually extracted snippets of evidence from web pages, human annotators have the advantage of implicitly considering the entire web page when making their judgments. This discrepancy between the annotator’s access and the model’s to information can lead to inconsistencies between the annotation and the prediction process. Consequently, even though many studies boast high concordance rates for their labels (Liu et al., 2023; Malaviya et al., 2023), the true reliability of these metrics and the correctness of the labels warrant further investigation.
Other error types. The model also struggles with dealing with numerical/symbolic operators, misunderstanding the task definition and logical relation implied by labels, and sometimes fails due to the lack of implicit commonsense knowledge Compared to the previous two error types, these errors can be more easily fixed by augmenting LMs with external tools, injecting necessary knowledge when prompting/inference, and making clearer instructions and in-context examples as guidance.
Our observations reveal the difficulty of the attribution evaluation task, leading to two potential directions for improvement: (1) For the model side, we should let the models not only be faithful, strongly conditioned on the given evidence, and pay attention to fine-grained information, but also possess the necessary ability of summarizing and reasoning over the evidence. (2) For the data side, the given instruction and evidence to the model should be more aligned with what human annotators get. The detailed error ratios of each sub-dataset are listed in Appendix B.

heading: 6 Related Work
text: Attributed LLMs. Generative LLMs often produce hallucinations, especially when encountering knowledge-intensive tasks, leading to the urgent need for attribution evaluation. Recently, many papers (Mialon et al., 2023; Schick et al., 2023; Li et al., 2023) have emerged to alleviate such issues by augmenting LLMs with external tools, such as retrievers (Guu et al., 2020; Lewis et al., 2020)
Green background indicates the correct spans and labels, and red background indicates the wrong spans or labels. Case 1: the model over inferred “fell out of favor with the regime” into “became a traitor”. Case 2: The “Pickler” in the evidence is ambiguous, leading to the “not attributable” label by the human annotator while the model failed to capture it. Case 3: It’s impossible to get an “attributable” conclusion according to the extracted evidence, whereas the original web page may support it.
and search engines (Nakano et al., 2021; Komeili et al., 2022). Notably, some work explicitly equips generation with retrieved supporting citations or evidence as the attribution for the generated statement. For instance, GopherCite (Menick et al., 2022) puts the citation after the generation as inline evidence. WebGPT (Nakano et al., 2021) finetunes GPT-3 (Brown et al., 2020) to simulate a person’s behavior of searching online and finding relevant information. Dialogue systems like LaMDA (Thoppilan et al., 2022) and Sparrow (Glaese et al., 2022) improve their safety and factual grounding by finetuning with human-annotated data and enabling the model to seek help from external resources.
Attribution Evaluation. Liu et al. (2023) and Kamalloo et al. (2023) employ human evaluation methods in which annotators are tasked with assessing the variability of responses generated by either search engines or advanced language models equipped with additional retriever modules, adhering to the criteria outlined in (Rashkin et al., 2021). Both studies corroborate that these systems often fall short in delivering statements backed with credible support and accurate citations. To avoid the time-consuming human evaluation, Gao et al. (2023b); Bohnet et al. (2022) employ NLI
models to provide automated AIS (AutoAIS). Yue et al. (2023) propose to repurpose related tasks to train a novel automatic evaluation model as well as prompting LMs as attribution evaluators. Our proposed benchmark systematically evaluates the performance of these approaches under various settings and reveals the challenge of automatic attribution evaluation via in-depth error analysis.

heading: 7 Conclusion
text: In this paper, we propose AttributionBench, a comprehensive benchmark for assessing automatic attribution evaluators. We conduct extensive experiments and analysis with cutting-edge LLMs like fine-tuned GPT-3.5, GPT-4, and a variety of open-source LLMs. Results systematically show that automatically evaluating attribution is still very challenging. Through meticulous error analysis, we identify the principal obstacles in this endeavor, notably (1) the models’ lack of sensitivity to detailed, fine-grained information and (2) the discrepancy between the information accessible by the model and that by human annotators. We hope our research will shed light on these issues and guide future efforts toward the development of more accurate and reliable automatic attribution evaluators.

heading: 8 Limitations
text: Despite the comprehensive nature of our work in developing AttributionBench and conducting extensive experiments to understand the challenges of automatic attribution evaluation, our study has several limitations that warrant future investigation:
Dataset Diversity and Representativeness. Although we meticulously sampled data from 7 different datasets covering various domains, our benchmark may still not capture the full spectrum of real-world scenarios that LLMs encounter. We will continuously update our benchmark to align with the rapid developments in LLMs and changes in web content, as well as involving more detailed classification tasks in addition to binary classification.
Generalizability of Findings. While our benchmark provides valuable insights into the challenges of automatic attribution evaluation, the generalizability of our findings may be limited by the specific datasets and evaluation setup we adopted. However, these findings give a more direct comparison and comprehensive understanding of current attribution evaluation systems. This could further inspire future efforts to build more robust attributed LLMs and attribution evaluators.
The coverage of attribution evaluation models. Our study takes initial steps to systematically evaluate a diverse set of advanced attribution evaluators. There might still be missing ones which possess good performance on our benchmark, as it’s impossible to cover all the existing models. Nonetheless, we hope our findings can shed light on building better attribution evaluators in the future.

heading: B Error Distribution in Sub-Datasets
text: We show the detailed error distribution in each sub-dataset in Figure 4 and Figure 5. Although “fine-grained information insensitivity” is the major error type among all datasets, the distribution of the rest error types still varies among 7 test sets. Moreover, the detailed error reasons within “fine-grained information insensitivity” are also diverse, indicating the diversity and difference among different test sets.
4https://platform.openai.com/docs/api-reference/chat. 5https://platform.openai.com/finetune.

heading: Future Work
text: Despite the comprehensive nature of our work in developing AttributionBench and conducting extensive experiments to understand the challenges of automatic attribution evaluation, our study has several limitations that warrant future investigation:
Dataset Diversity and Representativeness. Although we meticulously sampled data from 7 different datasets covering various domains, our benchmark may still not capture the full spectrum of real-world scenarios that LLMs encounter. We will continuously update our benchmark to align with the rapid developments in LLMs and changes in web content, as well as involving more detailed classification tasks in addition to binary classification. Generalizability of Findings. While our benchmark provides valuable insights into the challenges of automatic attribution evaluation, the generalizability of our findings may be limited by the specific datasets and evaluation setup we adopted. However, these findings give a more direct comparison and comprehensive understanding of current attribution evaluation systems. This could further inspire future efforts to build more robust attributed LLMs and attribution evaluators. The coverage of attribution evaluation models. Our study takes initial steps to systematically evaluate a diverse set of advanced attribution evaluators. There might still be missing ones which possess good performance on our benchmark, as it’s impossible to cover all the existing models. Nonetheless, we hope our findings can shed light on building better attribution evaluators in the future