Venue: NAACL 2024

# \(\mathbb{Q}_{n}\) ExpertQA \(\not\leq\): Expert-Curated Questions and Attributed Answers

 Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber\({}^{\mathbb{O}_{n}}\), Mark Yatskare, Dan Roth\({}^{\text{\circledR}}\)

University of Pennsylvania University of Washington

{cmalaviy,subinlee,sihaoc,myatskar,danroth}@upenn.edu

esieber@uw.edu

###### Abstract

As language models are adopted by a more sophisticated and diverse set of users, the importance of guaranteeing that they provide factually correct information supported by verifiable sources is critical across fields of study. This is especially the case for high-stakes fields, such as medicine and law, where the risk of propagating false information is high and can lead to undesirable societal consequences. Previous work studying attribution and factuality has not focused on analyzing these characteristics of language model outputs in domain-specific scenarios. In this work, we conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality, by bringing domain experts in the loop. Specifically, we collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. In addition, we ask experts to improve upon responses from language models. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.1

Footnote 1: Code and dataset is available at [https://github.com/chaitanymalaviya/ExpertQA](https://github.com/chaitanymalaviya/ExpertQA).

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3025–3045

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

As the influence of large language models (LLMs) grows beyond the computer science community, _experts_ from various fields are rapidly adapting LLMs for assistance in information-seeking scenarios. For example, medical professionals are using these systems for performing differential diagnosis Lee et al. (2023) and researchers are using them for faster literature surveys Krenn et al. (2022); Birhane et al. (2023); Owens (2023). While the use of LLMs in specialized domains has many potential benefits, it also carries significant risks. False or hallucinated claims that are confidently phrased can potentially mislead experts and propagate societal harms, especially in high stakes domains such as medicine or law Evans et al. (2021); Dash et al. (2023); Volokh (2023); Augenstein et al. (2023).

Providing citations or attributions within generated responses is a promising direction for alleviating such concerns. However, the quality of these attributions in model-generated responses, as well as the factuality of responses, is understood in domain-specific settings. This is partly because we do not completely understand the specific information-seeking needs of experts. Although experts from different fields are naturally best suited to aid with such an evaluation, expert evaluations are rarely conducted, as bringing experts in the loop can be time-consuming and costly.

Figure 1: ExpertQA contains 2177 information-seeking questions formulated by experts spanning 32 fields, as well as expert-verified, model-generated answers to these questions. Each claim-evidence pair in an answer is judged by experts for various properties such as the claim’s informativeness, factuality, citeworthiness, whether the claim is supported by the evidence, and reliability of the evidence source. Further, experts revise the original claims to ensure they are factual and supported by trustworthy sources.



To bridge this gap, we conduct an _expert-in-the-loop_ evaluation of attributed responses from a few representative systems. Having experts in the loop allows us to model a more realistic information-seeking scenario that helps us understand how people in different fields use LLMs and where their capabilities fall short. The output of our analysis is ExpertQA, a benchmark of information-seeking questions curated by experts from 32 fields, along with verified answers from representative systems. ExpertQA includes field-relevant questions, as well as claim-level judgements from experts along various axes of factuality and attribution.

Our evaluation is conducted by first asking qualified experts to formulate questions from their field that they are curious about or have encountered in their professional lives (SS2.1). Responses to these questions are collected from a set of LLM-based systems that produce attributions for their answers (SS3). These include purely generative, retrieval-augmented, and post-hoc attribution systems. We then ask experts to validate the claims and evidences found within responses to their own questions (SS2.2). Experts judge each claim for its informativeness to the question, its citeworthiness, and factuality. They are also asked to judge how faithful the claim is to an accompanying evidence and rate the reliability of the evidence's source. Finally, experts revise each claim so it is faithful to reliable evidences and make a best effort attempt at ensuring the claim is factual. This overall process is described in Figure 1.

Our findings (SS4) about representative systems from which responses are sampled suggest that:

1. [noitemsep]
2. _Retrieve-and-read systems generate more complete attributions compared to LLM prompting and post-hoc attribution, but struggle to produce citations for all cite-worthy claims._
3. _The retrieval source significantly impacts the quality of attribution and overall factuality._
4. _High-stakes domains such as medicine and law suffer from a large percentage of incomplete attributions (35% and 31% incomplete attributions respectively) and many attributions come from unreliable sources (51% attributions are not rated reliable by experts)._

We also measure the extent to which existing automatic methods for attribution and factuality estimation Bohnet et al. (2022); Min et al. (2023) correlate with expert judgements (SS5). We find that these metrics fall short in correlating with reference judgements of attribution and factuality. However, adapting these metrics to our data through finetuning results in improvements across domains.

The revised answers we collect can be used for improving and evaluating future models on long-form question answering. While similar datasets have been proposed Fan et al. (2019), examples in ExpertQA contain verified attributions and answers edited by experts. We establish several baselines and show that we can improve models by finetuning on ExpertQA but that there is substantial room for improvement, both in terms of ROUGE and QAFactEval (SS6).

## 2 Expert-in-the-loop Evaluation

The evaluation is conducted in multiple stages described below. In the first stage, we ask experts to write questions from their field (SS2.1). In the next stage, we present responses sampled from various systems back to the same experts for analysis (SS2.2). Further details about annotator backgrounds, costs and interfaces, are in Appendix A.

### Stage 1: Expert-Curated Questions

Participants are recruited through Prolific and are qualified as experts if they have i) received formal education, as well as, ii) at least 3 years of work experience in their field. They are asked to write questions from their field which they have encountered in their professional life or ones they are genuinely curious about. We ask them to formulate challenging technical questions, for which it may not be possible to find a single webpage that answers them completely. We note that this question collection is aimed at closely _simulating_ an information-seeking scenario with experts, since having access to real query logs is not feasible.

Each expert is asked to write 5 questions and to specify the question type(s) for each question (as shown in Table 2). These question types are formulated by adopting prior work that classifies information needs Rose and Levinson (2004). Because of their practical nature, at least two questions are required to be scenario-based questions (Type V, Table 2). We collect questions 2177 questions from 524 experts in 32 fields, which are manually filtered for coherence and field-relevance. Examples of these questions are presented in Table 1 



### Stage 2: Answer and Claim Annotation

Next, we generate responses for the questions from stage 1 by prompting six different systems, described in SS3, that provide attributions with their answers. We split each answer into claims, where claims are considered at the granularity of a sentence and extracted using the spaCy sentence tokenizer Honnibal and Montani (2017).2

Footnote 2: We also considered further increasing the atomicity of claims like Kamoi et al. (2023) but evaluating finer-grained atomic claims incurs considerably higher annotation cost.

In this stage of annotation, experts validate responses to their own questions on several dimensions of quality. 92% of annotators from stage 1 validated at least 1 of their own questions. The properties of answers and claims evaluated are shown in Table 3. Properties that judge answer quality are marked with A and those that judge evidence quality are marked with B. After labeling these claim properties, annotators edit the response to ensure that the claim is factually correct and the given references support the claim.

\begin{table}
\begin{tabular}{c|l|p{456.9pt}} \hline \hline
**Field** & **Question** \\ \hline Anthropology & Why is it that Africa’s representation is still a problem in modern day times regardless of the academic writings that state otherwise? & II.VII \\ Architecture & Synpose an architect decides to reuse an existing foundation of a demolished building, what is to be considered to ensure success of the project? & IV \\ Biology & Can you explain the mechanisms by which habitat fragmentation affects biodiversity and ecosystem functioning, and provide examples of effective strategies for mitigating these impacts? & III.VI \\ Chemistry & Why does galletic acid have an affinity with trivalent iron ions? & I \\ Engineering \& Technology How different will licensing a small modular reactor be as compared to licensing traditional large nuclear power plants? & VII \\ Healthcare/Medicine & If a 48 year old woman is found to have an esophageal carcinoma that invades the muscularis propria and has regional lymph node metastases but no distant metastasis, what is her stage of cancer and what are possible recommended treatments? & I.III \\ Law & Can direct evidence in a case that has been obtained illegally be considered by the court in some cases (if it directly points to the defendant’s guilt!) & I \\ Music & What exercises would so do in a singing class with a teenager with palerophonia? & IV \\ Physics \& Astronomy & Standard Model does not contain enough CP violating phenomena in order to explain baryon asymmetry, Suppose the existence of such phenomena. You propose a way to experimentally observe them? & V \\ Political Science & Despite the fact that IPCC was formed in 1988, several undershire have showed that arguably more than 50% of all carbon emissions in history have been released since 1988. What does this show about IPCC and developed countries’ efforts? & VII \\ Visual Arts & Tell me the step by step process of recycling a canvas. & III \\ \hline \hline  & \multicolumn{1}{p{456.9pt}}{} & \\ \end{tabular}
\end{table}
Table 1: Examples from ExpertQA. See Table 15 for a larger list showing an example from all fields. A large percentage of examples come from high-stakes fields such as Medicine and Law.

Figure 2: The distribution of questions across different fields in ExpertQA.





## 3 Systems Evaluated

We now describe the classes of systems from which we sampled responses to questions. All systems we evaluated produce an answer string and attributions in the form of in-line citations. Attributions are returned as URLs or passages along with URLs from where they are retrieved. Experimental details such as prompts are in Appendix B.

LLM as generator + retriever.In this paradigm, we prompt large language models in a closed-book fashion Brown et al. (2020); OpenAI (2023) to generate an answer with in-line citations where they provide URLs for each citation. This means that the model essentially has to generate a URL from its parametric memory. We consider GPT-4 as the LLM from which we sample responses (gpt4).

Post-hoc retrieval.This system differs from the above, as we only prompt LLMs to generate answers without attribution, and perform retrieval of evidence for a claim as a post-hoc step. This renders the attributions naturally unfaithful, but we believe this is still a worthwhile approach to investigate because of the strength of LLMs as generators and retrievers independently. The attribution corpora we consider are Sphere Piktus et al. (2021) (post_hoc_sphere_gpt4), which is a large static dump of CommonCrawl, and Google search results (post_hoc_gs_gpt4).

Retrieve-and-read.In this class of systems, we first retrieve evidence for a question and then prompt a model to use the retrieved evidence to answer the question Chen et al. (2017). As our attribution corpus, we again consider Sphere Piktus et al. (2021) (rr_sphere_gpt4) and Google search results (rr_gs_gpt4). We use BM25 Robertson et al. (2009) for retrieving from Sphere. We then generate an answer using GPT-4, providing the retrieved evidence as context. The model is instructed to generate in-line citations for each sentence, which refer to the passages in the context.

Commercial.We also consider commercial systems such as BingChat.3 We sample responses using the balanced mode of BingChat (bing_chat).

Footnote 3: The precise implementation of these systems is proprietary, but we can still draw conclusions about their utility.

### Response Sampling

We sample uniformly from all systems but exclude abstained answers and constrain each answer to contain at most 10 claims. Attributions from gpt4 often point to broken links, so we sampled more responses from the other systems. The number of examples from each system and how frequently they abstain are reported in Table 4.

## 4 Analysis

### Data Statistics

The total number of examples validated in ExpertQA is 2177. The distribution of the number of claims and tokens is shown in Figure 3. The distribution of examples across fields and question types are presented in Figure 2 and Table 2 respectively.

### Manual Analysis

To estimate the reliability of the collected human labels, we, the authors, computed our agreement with

\begin{table}
\begin{tabular}{p{86.7pt}|c|c}
**System** & **Count** & **Abstention Rate** \\ \hline gpt4 & 174 & 0\% \\ bing\_chat & 470 & 0.01\% \\ rr\_sphere\_gpt4 & 279 & 37.89\% \\ rr\_gs\_gpt4 & 452 & 22.69\% \\ post\_hoc\_sphere\_gpt4 & 403 & 0\% \\ post\_hoc\_gs\_gpt4 & 399 & 0\% \\ \multicolumn{3}{p{86.7pt}}{} \\ \end{tabular}
\end{table}
Table 4: Number of examples sampled from different systems and the abstention rates of different systems.

\begin{table}
\begin{tabular}{p{86.7pt} p{113.8pt}}
**Property** & **Description** & **Rating** \\ \hline (A) Answer Usefulness & _Is the answer useful in responding to the question?_ & _Useful, Partially useful, Not useful at all_] \\ (A + *P) Attribution & _Is the claim supported by its accompanying evidence?_ & _Complete, Partial or Incomplete, Missing, NA_ (if link broken)) \\ (A) Informativeness & _Is the claim relevant to answering the question?_ & _type relevant, A bit relevant, Not too important, Uniformative_ \\ (A) Factuality & _Is every word of the claim factually correct?_ & _Definitely correct, Probably correct, Unaware, Likely incorrect, Definitely incorrect_ \\ (*P) Source Reliability & _Is the accompanying evidence (tf any) for the claim found on a website you would consider reliable?_ & _Reitable, Somewhat Reliable, Not reliable at all_] \\ (A) Cite-worthiness & _Is the claim necessary to be cited?_ & _Yes, No_] \\ \hline  & \multicolumn{1}{p{113.8pt}}{} \\ \end{tabular}
\end{table}
Table 3: Properties of claims and evidences annotated in ExpertQA.



the reference labels from two fields in which the authors are experts. We sampled 60 questions each from Engineering & Technology and Medicine, sampling answers uniformly from all systems. For each claim, we label our agreement with the reference label for each property from Table 3. Our analysis, as summarized in Figure 4, shows high agreement (> 85%) for most labels in both fields considered.

### Analysis of Expert Evaluations

We present the Likert distribution for claims across all systems and properties in Figure 5. Below we summarize the main conclusions from our analysis.

Majority of answers are useful, but answers from purely generative systems are considered more useful.We find that \({\sim}87\)-89% of answers from gpt4 are marked useful. The retrieve-and-read systems (as well as bing_chat) are marked slightly less useful (73-80%), likely because retrieved evidences are not always highly relevant. Choosing relevant evidences from the web using Google search results in more useful answers than with the smaller Sphere corpus. Analyzing responses marked not useful, we find that **systems struggle with targeted responses to long-tail queries, by resorting to patterns such as hedging, or providing generic or vague information**.

Retrieve-and-read systems often generate complete attributions, but struggle to produce citations for all cite-worthy claims.While these systems have a stronger inductive bias to use the retrieved evidence to generate a response, they do not always produce attributions for cite-worthy claims (18% of these claims are missing attributions)4. On the other hand, post-hoc attribution systems return attributions for every single claim by design, but return more incomplete attributions. Lack of context during post-hoc retrieval can be an issue for retrieving valid attributions.

Footnote 4: Figure 5 shows the Likert distribution of attribution labels on those claims deemed cite-worthy by experts.

Finally, without retrieval, while gpt4 generates citations to plausible domains (for e.g., nasa.gov for astronomy, nih.gov for medical claims), the content on these webpages is usually totally mismatched (more than 60% of the time). Across systems, we find that **because domain-specific claims are long-tail and niche, it is hard to find reliable evidence on web documents that completely supports such claims. Both vanilla prompting and retrieval-augmented systems generate mostly _very relevant_ claims to the question.At the same time, a significant percentage of claims (30-40) are not very relevant. This includes void claims (that simply restate the question or state simplistic facts). This suggests that there is a lot of room in making answers concise and relevant. Just over half the claims are labeled as _definitely correct_ by experts.While a significant percentage of claims are labeled as correct (_probably_ or _definitely_), experts do not instill high confidence in the factual correctness of claims. This might be because it is hard to judge actuality with a high degree of confidence in a short time frame. Once again, a smaller retrieval corpus (rr_sphere_gpt4) results in less factual claims as the model may be more likely to hallucinate. The retrieval corpus has a significant effect on expert judgements of source reliability.Expert judgements of source reliability are influenced by the corpus from which evidences are retrieved. Corpora such as Sphere contain evidences that are unreliable to experts (for both rr_sphere_gpt4 and

Figure 4: Percentage agreement on claim annotations based on our manual analysis.

Figure 3: Histogram of the number of claims and number of tokens across all examples in ExpertQA. The average number of claims and tokens across examples is 5.79 and 152.12 respectively



post_hoc_sphere_gpt4). Note also that we do not account for the authoritativeness of domains when retrieving from Sphere. For example, in a question about breast cancer, evidence from a comment on a blog is retrieved and is naturally judged unreliable. Using Google search improves reliability judgements significantly.

Majority of claims are deemed cite-worthy across systems.Only around 17-22% claims are judged not citeworthy by the experts. This suggests that most claims in responses to expert-curated questions warrant providing supporting evidence.

Domain and Question Type Trends.Figure 9 shows the distribution of labels across fields. The percentage of claims labeled factually correct is fairly high (>85%) for many fields. However, we note that across all annotated claims, **high-stakes domains such as medicine and law suffer from a significant percentage of incomplete attributions** (around 35% and 31% unsupported claims respectively). Further, **a large percentage of claims present evidences from unreliable sources** (for eg, \(\sim\)51% of medical claims have attributions from sources that are not _Reliable_). The trends across question types (Figure 10), systems clearly struggle with Type VI questions that request for a list of resources, as claims are less informative, factual, and supported by evidence.

## 5 Automatic Estimation of Attribution and Factuality

Prior work has proposed automatic methods to predict attribution and factuality of claims. We evaluate how reliably these methods reflect the expert labels in our collected data. We evaluate the effectiveness of these methods for claims in ExpertQA. In both cases, we observe that **current methods show high precision but low recall when compared with human judgements..**

### Automatic Attribution Estimation

Under the _attributable to identifiable sources_ (AIS) framework of Rashkin et al. (2021), previous work has found NLI models to be effective in providing automated AIS (AutoAIS) estimates (Bohnet et al., 2022). Following previous work, we use an NLI model Honovich et al. (2022) to predict binary attribution labels of claim-evidence pairs in ExpertQA. For evidences longer than the model's sequence length (512), we use the stretching technique from Schuster et al. (2022), where we split the evidence into sentences and use the top-2 sentences with highest entailment scores as evidence.

Figure 5: The Likert distribution of labels for the different properties of answers / claims, annotated by experts. The top 3 properties (answer usefulness, claim informativeness and factuality) are judgements of answer quality and the bottom 3 (claim/evidence attribution, source reliability and claim cite-worthiness) are attribution quality.

 

Table 5 shows the macro-averaged AutoAIS scores for the claims annotated as having complete attributions. Compared to human judgments, the AutoAIS scores show large variance across systems. Notably, attributions from post-hoc retrieval systems receive much lower AutoAIS scores compared to retrieve-and-read systems.

We compare the per-claim AutoAIS predictions to human judgements of attribution in Table 6. The results suggest that AutoAIS estimates have high-precision yet low-recall against human judgements of attribution. To understand the discrepancy between NLI model behavior vs. human judgements, we highlight a few typical examples of attribution errors in Table 7. For NLI models, every part of the claim needs to be verifiable with the evidence, but human judgements involve more implicit world knowledge, e.g. _calcium carbonate is an alkali_. Another common mistake involves synthesizing information from multiple evidences. We observe multi-source attributions to be particularly common among bing_chat and retrieve-and-read systems.

### Automatic Factuality Estimation

Prior work has proposed methods Manakul et al. (2023); Min et al. (2023) to estimate the factuality of model generations. In particular, we use FActScore Min et al. (2023) to estimate factuality of claims. We first break down each claim into fine-grained atomic claims using few-shot prompting with text-davinci-003. We then retrieve the top-3 relevant passages using Google search with the atomic claim as the query. The atomic claim and the evidence passages are then used to prompt gpt-3.5-turbo to say whether the atomic claim is _True_ or _False_. The FActScore of a claim is the FActScore averaged across its atomic claims.

In Table 8, we report the F1 scores of the factual (T) and non-factual (F) classes and the micro-averaged overall F1 scores of the FActScore factuality scores and the reference factuality labels. FActscore scores are thresholded at 0.5 to get binary scores and reference factuality labels are 1 if the claim's factuality is labeled as _Probably correct_ or _Definitely correct_, and 0 otherwise.

\begin{table}
\begin{tabular}{l|c c|c c}
**System** & \multicolumn{4}{c}{zero-shot} & \multicolumn{2}{c}{finetuned} \\  & **P** & **R** & **F1** & **P** & **R** & **F1** \\ \hline gpt4 & .33 & .02 & .05 & .52 & .32 & .39 \\ bins\_chat & .97 & .26 & .41 & .90 & .90 & .90 \\ rr\_sphere\_gpt4 & .89 & .59 & .71 & .83 & .90 & .87 \\ rr\_gs\_gpt4 & .86 & .74 & .79 & .87 & .98 & .92 \\ post\_hoc\_sphere\_gpt4 & .92 & .28 & .43 & .79 & .97 & .87 \\ post\_hoc\_gs\_gpt4 & .87 & .17 & .29 & .77 & .95 & .85 \\ \hline all & .88 & .38 & .53 & .82 & .91 & .86 \\ \end{tabular}
\end{table}
Table 6: Precision, Recall and F1 scores of AutoAIS labels predicted by the TRUE NLI model (0-shot vs. finetuned version on the ExpertQA train split) against human attribution judgements in ExpertQA.

\begin{table}
\begin{tabular}{l|c|c|c}
**System** & **AutoAIS** & **Num. Claims** \\ \hline gpt4 & .156 & 149 \\ bing\_chat & .320 & 992 \\ rr\_sphere\_gpt4 & .689 & 732 \\ rr\_gs\_gpt4 & .778 & 1415 \\ post\_hoc\_sphere\_gpt4 & .281 & 1158 \\ post\_hoc\_gs\_gpt4 & .241 & 1500 \\ \hline \end{tabular}
\end{table}
Table 5: AutoAIS score (more attributable\(\rightarrow\)1, less attributable\(\rightarrow\)0) of predicted responses by the systems. Only claims annotated as _citeworthy_ and _with complete support_ are considered.

 

We find that automatic factuality estimation struggles to identify non-factual claims. In particular, predicted labels have low recall of non-factual claims. This is more often the case for retrieve-and-read systems, where the answer is generated based on retrieved evidences. The other systems use GPT-4's parametric knowledge for answer generation, which could make it easier for a similar evaluator like ChatGPT to judge factuality.

## 6 Long-form QA Evaluation

A beneficial output of our annotation is the revised answers produced by annotators. These answers are verified to be factual and compose a new long-form QA dataset, ExpertQA. We consider two splits for ExpertQA (both 80-10-10): a random split of the data and a domain-wise split, where 80% of a field's data is included in the training set and 10% is included in both validation and test sets.

### Evaluation Metrics

For evaluation, we consider metrics based on similarity to a reference answer, i.e., ROUGE Lin (2004) and those focused on evaluating factual consistency through QA pairs generated with a reference answer, i.e., QAFactEval Fabbri et al. (2022).

### Baselines

We finetune the following open-source language models: FlanT5-11B Chung et al. (2022), Alpaca7B Taori et al. (2023), Vicuna-7B Chiang et al. (2023) and LLaaM2-7B-Chat Touvron et al. (2023). We finetune these models with the same prompts as the ones used in their training (provided in Tables 12, 13). Further, we also report results with Llama2-70B-Chat without finetuning (marked *).

### Results

Our results are shown in Table 9. We find that both Llama2-7B and Vicuna-7B outperform FlanT5-11B despite the smaller model size, likely due to additional instruction finetuning for both those models. We observe that finetuning significantly improves performance (results without finetuning are in Table 14), and Llama2-70B performs worse than finetuned systems under zero-shot prompting.

## 7 Related Work

Attribution Generation.A few classes of systems have been proposed for generating attributions for model responses. This includes **vanilla LLM prompting** Tay et al. (2022), where LLMs are prompted to return attributions with their answers, but the references are often hallucinated Agrawal et al. (2023). On the other hand, **retrieve-and-read systems**Guu et al. (2020); Borgeaud et al. (2022); Izacard et al. (2022) first retrieve evidence relevant for a query, and generate an answer based on the retrieved evidence. These systems are sometimes trained on human demonstrations Nakano et al. (2021); Thoppilan et al. (2022); Menick et al. (2022). Finally, **post-hoc retrieval**Gao et al. (2023); He et al. (2022) involves retrieving attributions after answering a query. We consider all three classes of systems for sampling responses.

Attribution AnalysisPrior work has conducted analysis of system-generated attributions Rashkin et al. (2021); Bohnet et al. (2022); Dziri et al. (2022); Chen et al. (2022); Liu et al. (2023); Muller et al. (2023); Kamoi et al. (2023); Kamiloo et al. (2023). These works suggest that systems are still far from providing precise attributions with sufficient recall for citeworthy statements. In our work, we recognize that this is problematic in specific domains where precision and recall are both critical.

Factuality Analysis.Factuality analysis of model generations has been conducted extensively in prior work Thorne et al. (2018); Evans et al. (2021); Kryscinski et al. (2020); Maynez et al. (2020); Pagnoni et al. (2021); Lin et al. (2021); Atanasova et al. (2022); Muhlgay et al. (2023); Tang et al. (2024); Mishra et al. (2024). Peskoff and Stewart (2023) conduct a smaller-scale evaluation of modern LMs with 10 experts where they evaluate accuracy, among other qualities of answers. The factuality labels collected as part of ExpertQA elicit a best-effort judgement of truthfulness of claims from domain experts. Prior work has also proposed methods to predict factuality of claims Manakul et al. (2023); Kadavath et al. (2022); Agrawal et al. (2023);

\begin{table}
\begin{tabular}{l|r r|r r|r}
**Split** & **Model** & **R1** & **R2** & **RL** & **QFE** \\ \hline  & FlanT5-11B & 0.335 & 0.114 & 0.215 & 2.068 \\ Random & Vicuna-7B & 0.351 & 0.119 & 0.212 & 1.068 \\  & Llama2-7B & 0.362 & 0.125 & 0.219 & 1.985 \\  & Llama2-70B* & 0.320 & 0.101 & 0.181 & 1.050 \\ \hline  & FlanT5-11B & 0.324 & 0.107 & 0.210 & 1.538 \\ Domain & Vicuna-7B & 0.359 & 0.120 & 0.213 & 1.739 \\  & Llama2-7B & 0.363 & 0.124 & 0.219 & 1.726 \\  & Llama2-70B* & 0.328 & 0.104 & 0.187 & 0.979 \\ \end{tabular}
\end{table}
Table 9: Long-form QA results (ROUGE scores and QAFactEval scores) after finetuning models on the random and domain splits of ExpertQA.

 

2023; Azaria and Mitchell, 2023; Min et al., 2023; Feng et al., 2023; Chen et al., 2023). We use one such method (Min et al., 2023) to evaluate how well human labels in ExpertQA correlate with automatic judgements.

Long-form QA.Existing long-form QA datasets are created using search queries Nguyen et al. (2016); Stelmakh et al. (2022) and forums Fan et al. (2019). Several issues have been identified with these datasets, such as vague questions and difficulty in verifying factual correctness Krishna et al. (2021). Keeping this in mind, we construct ExpertQA to cover practical information needs of experts along with fine-grained factuality judgements. Xu et al. (2023) conduct expert evaluation of long-form answers and emphasize the importance of evaluating multiple aspects of answers, which are also considered in our work.

Domain-specific QA.Several domain-specific QA datasets have been proposed, for domains such as medicine Tsatsaronis et al. (2015); Pampari et al. (2018); Jin et al. (2019, 2021); Pal et al. (2022), law Guha et al. (2023), technology Dos Santos et al. (2015) and others Rogers et al. (2020); Reddy et al. (2019); Hendrycks et al. (2021). However, these datasets often have limited coverage of domains. ExpertQA contributes a unique combination of features by scaling the number of domains and providing attributions and factuality judgements.

## 8 Conclusion and Future Work

Our evaluation study suggests that although large language models show a lot of promise for aiding domain experts, there is large ground to cover in addressing the information needs of experts with factual and verifiable answers Metzler et al. (2021). Experts, on the other hand, should take responses from these systems with caution, because although attributed responses can seem trustworthy, the supporting references can often be inadequate to support claims. We hope that our benchmark, ExpertQA, can benefit the community in building improved methods for attribution & factuality estimation, and long-form question answering.

## 9 Limitations

Atomicity of Claims.In most cases, claims in our dataset are sentences that may not represent singular information units. This lack of atomicity in claims means that properties such as factuality and attribution need to be judged exhaustively for a claim. Collecting human judgements for finer-grained atomic claims can be significantly more expensive and is not explored in this work.

Claim Extraction.Extracting sentence-level claims from a generated answer for the purpose of evaluation is performed by using a sentence tokenizer. However, we note that existing tokenizers suffer from sentence tokenization errors (for example, when lists or tables are present in answers). This resulted in a small number of claims being excessively long and hard to evaluate.

Field Coverage.Even though we tried to cover a wide range of fields in our dataset, we missed covering questions from certain fields. Finding experts from rarer fields can be especially hard. We will consider further expanding ExpertQA to more domains, so that it can be more broadly useful. In addition, the examples in our dataset represent the information needs of English-speaking annotators primarily based in Europe, the Americas and Africa.

Question Distribution.We elicit questions from experts by asking them to formulate questions that have come up in their professional lives or questions they are genuinely curious about. This was aimed at modeling a more realistic informationseeking scenario through our annotation. However, it is not necessary that these questions would come from a natural distribution that would be found in query logs. Since having access to such data is not possible, we attempt to match the information-seeking scenario as closely as possible.

Subjectivity of labels.Some of the properties of claims can elicit more subjective judgements, which can vary between experts from the same field. This subjectivity is not inherently captured in our data through multiple judgements, but we do estimate agreement using claims from engineering and medicine through our own labels (SS4.2).

### Acknowledgements

First, we would like to thank the 484 annotators who took the time and effort out to help out with this study. This study would not have been possible without their contributions. We would also like to thank Artemis Panagopoulou, Alyssa Hwang, Nelson Liu and Chris Alberti for helpful comments and discussions 



## References

* Agrawal et al. (2023) Ayush Agrawal, Lester Mackey, and Adam Tauman Kalai. 2023. Do language models know when they're hallucinating references? _arXiv preprint arXiv:2305.18248_.
* Atanasova et al. (2022) Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2022. Fact checking with insufficient evidence. _Transactions of the Association for Computational Linguistics_, 10:746-763.
* Augenstein et al. (2023) Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, et al. 2023. Factually challenges in the era of large language models. _arXiv preprint arXiv:2310.05189_.
* Azaria and Mitchell (2023) Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when its lying. _arXiv preprint arXiv:2304.13734_.
* Birhane et al. (2023) Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. 2023. Science in the age of large language models. _Nature Reviews Physics_, 5(5):277-280.
* Bohnet et al. (2022) Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. _arXiv preprint arXiv:2212.08037_.
* Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trilions of tokens. In _International conference on machine learning_, pages 2206-2240. PMLR.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
* Chen et al. (2023) Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. 2023. Complex claim verification with evidence retrieved in the wild. _arXiv preprint arXiv:2305.11859_.
* Chen et al. (2022) Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Dan Roth, and Tal Schuster. 2022. Propsegment: A large-scale corpus for proposition-level segmentation and entailment recognition. _arXiv preprint arXiv:2212.10750_.
* Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Tianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Dash et al. (2023) Debadutta Dash, Rahul Thapa, Juan M Banda, Akshay Swaminathan, Morgan Cheatham, Mehr Kashyap, Nikesh Kotecha, Jonathan H Chen, Saurabh Gombar, Lance Downing, et al. 2023. Evaluation of gpt-3.5 and gpt-4 for supporting real-world information needs in healthcare delivery. _arXiv preprint arXiv:2304.13714_.
* Santos et al. (2015) Cicero Dos Santos, Luciano Barbosa, Dasha Bogdanova, and Bianca Zadrozny. 2015. Learning hybrid representations to retrieve semantically equivalent questions. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 694-699.
* Dairi et al. (2022) Nouha Dairi, Hannah Rashkin, Tal Linzen, and David Reitter. 2022. Evaluating attribution in dialogue systems: The begin benchmark. _Transactions of the Association for Computational Linguistics_, 10:1066-1083.
* Evans et al. (2021) Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful ai: Developing and governing ai that does not lie. _arXiv preprint arXiv:2110.06674_.
* Fabbri et al. (2022) Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. Qafacetval: Improved qa-based factual consistency evaluation for summarization.
* Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. Eli5: Long form question answering. _arXiv preprint arXiv:1907.09190_.
* Feng et al. (2023) Shangbin Feng, Vidisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. 2023. Factkb: Generalizable factuality evaluation using language models enhanced with factual knowledge. _arXiv preprint arXiv:2305.08281_.
* Gao et al. (2023) Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023. RARR: Researching and revising

what language models say, using language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 16477-16508, Toronto, Canada. Association for Computational Linguistics.
* Guha et al. [2023] Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Re, Adam Chilton, Aditya Narayana, Alex Cholhass Wood, Austin Peters, Brandon Waldon, Daniel N Rockmore, et al. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. _arXiv preprint arXiv:2308.11462_.
* Guo et al. [2020] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In _International conference on machine learning_, pages 3929-3938. PMLR.
* He et al. [2022] Hangfeng He, Hongming Zhang, and Dan Roth. 2022. Rethinking with retrieval: Faithful large language model inference. _arXiv preprint arXiv:2301.00303_.
* Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Honnbal and Montani [2017] Matthew Honnbal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
* Honovich et al. [2022] Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kulkinasy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3905-3920, Seattle, United States. Association for Computational Linguistics.
* Izacard et al. [2022] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. _arXiv preprint arXiv:2208.03299_.
* Jin et al. [2021] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. _Applied Sciences_, 11(14):6421.
* Jin et al. [2019] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2567-2577, Hong Kong, China. Association for Computational Linguistics.
* Kadavath et al. [2022] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drein, Ethan Perez, Nicholas Schleifer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. _arXiv preprint arXiv:2207.05221_.
* Kamalloo et al. [2023] Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Namdan Thakur, and Jimmy Lin. 2023. Hagrid: A human-llm collaborative dataset for generative information-seeking with attribution. _arXiv preprint arXiv:2307.16883_.
* Kamoi et al. [2023] Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. Wice: Real-world entailment for claims in wikipedia. _arXiv preprint arXiv:2303.01432_.
* Krenn et al. [2022] Mario Krenn, Robert Pollice, Si Yue Guo, Matteo Aldeghi, Alba Cervera-Lierta, Pascal Friederich, Gabriel dos Passos Gomes, Florian Hase, Adrian Jinich, AkshatKumar Nigam, Zhenpeng Yao, and Alan Aspuru-Guzik. 2022. On scientific understanding with artificial intelligence. _Nature Reviews Physics_, 4(12):761-769.
* Krishna et al. [2021] Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4940-4957, Online. Association for Computational Linguistics.
* Kryscinski et al. [2020] Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9332-9346, Online. Association for Computational Linguistics.
* Lee et al. [2023] Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. _New England Journal of Medicine_, 388(13):1233-1239.
* Lin [2004] Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Lin et al. [2021] Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. _arXiv preprint arXiv:2109.07958_.
* Liu et al. [2023] Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines. _arXiv preprint arXiv:2304.09848_.
* Manakul et al. [2023] Potsawe Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. _arXiv preprint arXiv:2303.08896_ 

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1906-1919, Online. Association for Computational Linguistics.
* Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022. Teaching language models to support answers with verified quotes. _arXiv preprint arXiv:2203.11147_.
* Metzler et al. (2021) Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork. 2021. Rethinking search: making domain experts out of dilettantes. In _Acm sigir forum_, volume 55, pages 1-27. ACM New York, NY, USA.
* Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. _arXiv preprint arXiv:2305.14251v1_.
* Mishra et al. (2024) Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. 2024. Fine-grained hallucinations detections. _arXiv preprint_.
* Muhlgay et al. (2023) Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yunatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. 2023. Generating benchmarks for factuality evaluation of language models. _arXiv preprint arXiv:2307.06908_.
* Muller et al. (2023) Benjamin Muller, John Wieting, Jonathan H Clark, Tom Kwiatkowski, Sebastian Ruder, Livio Baldini Soares, Roee Aharoni, Jonathan Herzig, and Xinyi Wang. 2023. Evaluating and modeling attribution for cross-lingual question answering. _arXiv preprint arXiv:2305.14332_.
* Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgt: Browser-assisted question-answering with human feedback. _arXiv preprint arXiv:2112.09332_.
* Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. _choice_, 2640:660.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report. _ArXiv_, abs/2303.08774.
* Owens (2023) Brian Owens. 2023. How nature readers are using chat- gptt. _Nature_.
* Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4812-4829, Online. Association for Computational Linguistics.
* Pal et al. (2022) Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbou. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In _Proceedings of the Conference on Health, Inference, and Learning_, volume 174 of _Proceedings of Machine Learning Research_, pages 248-260. PMLR.
* Pamparni et al. (2018) Anusri Pamparni, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrQA: A large corpus for question answering on electronic medical records. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2357-2368, Brussels, Belgium. Association for Computational Linguistics.
* Peskoff and Stewart (2023) Denis Peskoff and Brandon Stewart. 2023. Credible without credit: Domain experts assess generative language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 427-438, Toronto, Canada. Association for Computational linguistics.
* knowledge-intensive NLP against a very large web corpus. _CoRR_, abs/2112.09924.
* Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_, pages 1-16. IEEE.
* Rashkin et al. (2021) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipjanjan Das, Slav Petrov, Gaurav Singh Toman, Italia Turc, and David Reitter. 2021. Measuring attribution in natural language generation models. _arXiv preprint arXiv:2112.12870_.
* Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. _Transactions of the Association for Computational Linguistics_, 7:249-266.
* Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. _Foundations and Trends(r) in Information Retrieval_, 3(4):333-389.
* Rogers et al. (2020) Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. 2020. Getting closer to ai complete question answering: A set of prerequisite real tasks. In _Proceedings of the AAAI conference on artificial intelligence_, volume 34, pages 8722-8731 

Daniel E Rose and Danny Levinson. 2004. Understanding user goals in web search. In _Proceedings of the 13th international conference on World Wide Web_, pages 13-19.
* Schuster et al. (2022) Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, and Donald Metzler. 2022. Stretching sentence-pair NLI models to reason over long documents and clusters. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 394-412, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. Asqa: Factoid questions meet long-form answers. _arXiv preprint arXiv:2204.06092_.
* Tang et al. (2024) Liyan Tang, Igor Shalyminov, Amy Wing mei Wong, Jon Burnsky, Jake W. Vincent, Yu'an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, and Kathleen McKeown. 2024. Tofteval: Evaluating hallucinations of llms on topic-focused dialogue summarization.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsuori B. Hashimoto. 2023. Stanford aleca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_algoca](https://github.com/tatsu-lab/stanford_algoca).
* Tay et al. (2022) Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jia Gupta, et al. 2022. Transformer memory as a differentiable search index. _Advances in Neural Information Processing Systems_, 35:21831-21843.
* Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. _arXiv preprint arXiv:2201.08239_.
* Thorne et al. (2018) James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. The fact extraction and VERification (FEVER) shared task. In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_, pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. 2023. LLama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Tsatsaronis et al. (2015) George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotiis, Ioannis Partalas, Matthias Zschunke, Michael R Rivers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasaq large-scale biomedical semantic indexing and question answering competition. _BMC bioinformatics_, 16(1):1-28.
* Volokh (2023) Eugene Volokh. 2023. Large libel models? liability for ai output.
* Xu et al. (2023) Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Gunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3225-3245, Toronto, Canada. Association for Computational Linguistics.

 

Annotation Details

Annotator backgrounds.The 484 participants involved in our study came from 26 different countries, across Europe, Africa, Oceania, North and South America. The participants were recruited through Prolific, a crowdsourcing platform5. To qualify as experts, participants were required to have attained a formal education in the field and have worked in the area for at least 3 years. Participants were told that their annotations will be used to evaluate the capabilities of large language models to provide truthful answers with well-supported evidences to questions from different fields. They were also informed that the data will be released publicly upon the completion of the study.

Footnote 5: [https://www.prolific.co](https://www.prolific.co)

Annotator fields.The initial set of fields were listed by going through university department names, and ensuring that we cover a wide range of disciplines. Upon completing stage 1 of our annotation, we further refined these fields to represent a diverse set, for which we have enough experts.

Annotation costs.In both stage 1 and stage 2, annotators were compensated at the rate of $15 per hour with additional bonuses when annotators spent more time than we anticipated. The average time taken for stage 2 annotations was 13.83 minutes per question-answer pair. Since this task is intensive, a single annotation task was broken down into 1-3 question-answer pairs.

Annotation interface.Figures 6 and 7 show screenshots of our stage 2 annotation interface.

## Appendix B Experimental Details

### Hyperparameter Settings

Response collection.Across all systems, for generating responses from gpt4, we use a temperature of 1.0, and a maximum length of 2048 tokens. For all retrieval components, we use text-embedding-ada-002 as the embedding model. The retrieve-and-read systems first retrieve top-k (k=5) evidence passages from Sphere or top-10 Google search results using the question as the retrieval query. Google search results are split into passages of 1000 tokens with 200 tokens of overlap between subsequent chunks.

On the other hand, the post-hoc citation systems simply use the claims from gpt4 responses, but generate their own attributions by retrieving evidence for each claim in the answer. Post-hoc retrieval systems use the top-k passages (k=5) retrieved from Sphere or the top-10 Google search results with the claim as the retrieval query. Search result are split into passages the same way as retrieve-and-read systems.

Automatic attribution and factuality estimation.For automatic attribution with AutoAIS, we use the t5_xxl_true_nli_mixture6 with 11B parameters by Honovich et al. (2022). For finetuning the t5_xxl_true_nli_mixture model on the train split of ExpertQA, we use the DeepSpeed ZeRO optimization Rajbhandari et al. (2020) with stage 3, a batch size of 1, a learning rate of \(1e^{-4}\) and train models for 3 epochs.

Long-form QA.For finetuning FlanT5-11B, we use a batch size of 2, maximum sequence length of 512, a learning rate of 1e-4 and train models for 3 epochs. For finetuning both Llama2-7B and Vicuna-7B, we use a batch size of 4, maximum sequence length of 2048, learning rate of \(2e^{-4}\) and train models for 3 epochs.

Footnote 6: [https://huggingface.co/google/t5_xxl_true_nli_mixture](https://huggingface.co/google/t5_xxl_true_nli_mixture)

### Prompts

The prompts used to generate responses from gpt4 and bing_chat is provided in Table 10, while the prompt used to generate responses for retrieve-and-read systems is in Table 11.

For factuality estimation, we use the same prompts as Min et al. (2023) for both claim decomposition and atomic claim factuality prediction. Finally, for long-form QA baselines, we use the prompt in Table 12 for Llama and Table 13 for Vicuna.

## Appendix C Additional Plots

Examples from all fields included in ExpertQA are shown in Table 15. We show the distribution of all question types (from Table 2) across all fields that are part of ExpertQA in Figure 8.

In Table 9, we summarize the label distribution of all claim properties across fields and in Table 10, we summarize the label distribution of all claim properties across question types.

In Table 14, we summarize results on long-form QA before and after finetuning models on ReportQA splits.

 



**Vanilla LM QA Prompt**

Answer the question completely and precisely in up to 500 words. You must provide in-line citations to each statement in the answer. The citations should appear as numbers such as [1], [2] and contain references to valid URLs on the web. A statement may need to be supported by multiple references and should then be cited as [1][2].

Question: I work in the field of [FIELD]. My question is: [QUESTION] Answer:

\begin{table}
\begin{tabular}{l} \hline \hline
**Retrieve-and-read Prompt** \\ \hline Use the following pieces of context to answer the question completely and \\ precisely in up to 500 words. If you don’t know the answer, just say “I don’t know” and explain why the context is insufficient to answer the question. \\ You need to support every statement in the answer with in-line citations to \\ passages given in the the context. The citations should appear as numbers such \\ as [1], [2] that refer to the Passage IDs of the given passages. A statement may \\ need to be supported by multiple references and should then be cited as [1][2]. (for example, ”Paris is the capital of France [1][2].” where ”1” and ”2" are \\ the Passage IDs of the first and second passage). \\ [CONTEXT] Question: [QUESTION] Answer: \\ \hline \end{tabular}
\end{table}
Table 10: QA Prompt for GPT4 and BingChat.

\begin{table}
\begin{tabular}{l} \hline \hline
**Llama2 Prompt** \\ \hline <S>
 *SYS>> You are a helpful, respectful and honest assistant. Always answer as helpfully \\ as possible, while being safe. Your answers should not include any harmful, \\ unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure \\ that your responses are socially unbiased and positive in nature. \\ If a question does not make any sense, or is not factually coherent, explain why \\ instead of answering something not correct. If you don’t know the answer to a \\ question, please don’t share false information. \\ */SYS>> \\ \hline \end{tabular}
\end{table}
Table 11: Retrieve-and-read QA prompt.





\begin{table}
\begin{tabular}{l|c c|c c|c c}
**Split** & \multicolumn{2}{c}{**Model**} & \multicolumn{1}{c}{**R1**} & \multicolumn{1}{c}{**R2**} & \multicolumn{1}{c}{**RL**} & \multicolumn{1}{c}{**QFE**} \\ \hline  & FlanT5-11B* & 0.074 & 0.023 & 0.063 & 0.000 \\  & FlanT5-11B & 0.335 & 0.114 & 0.215 & 2.068 \\  & Vicuna-7B* & 0.358 & 0.116 & 0.209 & 0.902 \\  & Vicuna-7B & 0.351 & 0.119 & 0.212 & 1.068 \\  & Llama2-7B* & 0.300 & 0.083 & 0.167 & 1.359 \\  & Llama2-7B & 0.362 & 0.125 & 0.219 & 1.985 \\  & Llama2-7OB* & 0.320 & 0.101 & 0.181 & 1.050 \\ \hline  & FlanT5-11B* & 0.073 & 0.023 & 0.062 & 0.000 \\  & FlanT5-11B & 0.324 & 0.107 & 0.210 & 1.538 \\  & Vicuna-7B* & 0.352 & 0.114 & 0.203 & 2.596 \\  & Vicuna-7B & 0.359 & 0.120 & 0.213 & 1.739 \\  & Llama2-7B* & 0.303 & 0.087 & 0.169 & 1.799 \\ Domain & Llama2-7B & 0.363 & 0.124 & 0.219 & 1.726 \\  & Llama2-7OB* & 0.328 & 0.104 & 0.187 & 0.979 \\ \hline \end{tabular}
\end{table}
Table 14: Long-form QA results before (marked with *) and after finetuning models on the random and domain splits of ExpertQA.

\begin{table}
\begin{tabular}{l|c|c|c|c}
**Split** & \multicolumn{1}{c}{**Model**} & \multicolumn{1}{c}{**R1**} & \multicolumn{1}{c}{**R2**} & \multicolumn{1}{c}{**RL**} & \multicolumn{1}{c}{**QFE**} \\ \hline  & FlanT5-11B* & 0.074 & 0.023 & 0.063 & 0.000 \\  & FlanT5-11B & 0.335 & 0.114 & 0.215 & 2.068 \\  & Vicuna-7B* & 0.358 & 0.116 & 0.209 & 0.902 \\  & Vicuna-7B & 0.351 & 0.119 & 0.212 & 1.068 \\  & Llama2-7B* & 0.300 & 0.083 & 0.167 & 1.359 \\  & Llama2-7B & 0.362 & 0.125 & 0.219 & 1.985 \\  & Llama2-7OB* & 0.320 & 0.101 & 0.181 & 1.050 \\ \hline  & FlanT5-11B* & 0.073 & 0.023 & 0.062 & 0.000 \\  & FlanT5-11B & 0.324 & 0.107 & 0.210 & 1.538 \\  & Vicuna-7B* & 0.352 & 0.114 & 0.203 & 2.596 \\  & Vicuna-7B & 0.359 & 0.120 & 0.213 & 1.739 \\  & Llama2-7B* & 0.303 & 0.087 & 0.169 & 1.799 \\  & Llama2-7B & 0.363 & 0.124 & 0.219 & 1.726 \\  & Llama2-7OB* & 0.328 & 0.104 & 0.187 & 0.979 \\ \hline \end{tabular}
\end{table}
Table 13: Vicuna prompt for long-form QA.




## 1 Login Screen

**Stage 2: Detailed instructions**

These work yet present an order that we are going of researchers conducting a study to understand how important information fusion takes as I/I large language models in information-sensitive semantics. We are particularly interested in evaluating the accuracy and detail of correctness of assessing predicted results by such systems, the use of multiple participants who are professionals's experts in these fields;

[Interpolitory](Redundancy)( 



## 8 Claim Revision

### Online Behavior: Read the global timing and collection in complex data

Once, regularly generated and the previous software showed that online, for few bits, only, it may not be the client or anyone or private information.

Note: We define a new framework, energy feature that is used to create other kinds of functions. The evolution is recorded in multi-formatization across the devices. This model needs however instead followed execution without increasing interest of content.

#### Rules claim below

The literature can refer to [1] as "functional fitness loss function" (or fitness) fitness score (or fitness) scores, which are then methods to investigate the performance of these systems, or otherwise, an evaluation technique of pair [2].

#### Routate evidence before

* (7) [Figs/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figures/figures/figures/figures/figuresfigures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figures/figuresfigures/



\begin{table}
\begin{tabular}{c c l r}
**Field** & **Question** & \\ \hline \hline Anthropology & _Why is it that Africa’s representation is still a problem in modern day times regardless of the academic writings that state otherwise?_ & II,VII \\ Architecture & _Suppose an architect decides to reuse an existing foundation of a demolished building, what is to be considered to ensure success of the project?_ & IV \\ Aviation & _Should a low value shipment take priority from a regular customer or a high value shipment from a infrequent customer?_ & V \\ Biology & _Can you explain the mechanisms by which habitat fragmentation affects biodiversity and ecosystem functioning, and provide examples of effective strategies for mitigating these impacts?_ & III,VI \\ Business & _If your supplier can give you a discount for a whole yearly production, how can we take this deal without affecting our budget in a critical way?_ & V \\ Chemistry & _Why does gallic acid have an affinity with trivalent iron ions?_ & I \\ Classical Studies & _If researchers found a new method to unroll the Herculanum papryri, would it be fair to try it on the actual papyrus, given that it could potentially destroy it?_ & V \\ Climate Science & _If an imidazolium based ionic liquid were to be released into the environment through the aquatic compartment, what species would be affected. if any?_ & II,III,V \\ Criminology & _Mr X is an 18 year old first time offender involved in a burglary where he acted as a lookout. Which category should this information be placed under?_ & V \\ Culinary Arts & _If meecal production in the Valley of Mexico posits the distilling of meecal can be traced back to ancient times, how could this be attained before the arrival of the Spaniards?_ & V \\ Economics & _Can you summarize the current economic policies and strategies of the top five global superpowers and their potential impact on the global market?_ & I \\ Education & _Can music therapy impact a child with autism if they have noise sensory issues?_ & V \\ Engineering and Technology & _How different will licensing a small modular reactor be as compared to licensing traditional large nuclear power plants?_ & VII \\ Environmental Science & _Does floating solar panels minimize the risk of eutrophication or they are more trouble than their worth?_ & I \\ Geography & _How can we overcome the limitations of remote sensing data, such as low spatial resolution and limited spectral bands?_ & IV \\ Healthcare/Medicine & _If a 48 year old woman is found to have an esophageal carcinoma that invades the muscularis propria and has regional lymph node metastases but no distant metastasis, what is her stage of cancer and what are possible recommended treatments?_ & I,III \\ History & _To what extent is JFK’s legacy written from sympathy because of his assassination?_ & II,VII \\ Journalism & _How many sources you must have before printing a story?_ & I \\ Law & _Can direct evidence in a case that has been obtained illegally be considered by the court in some cases if it directly points to the defendant’s guilt?_ & I \\ Linguistics & _What are the attitudes of Received Pronunciation in the United States?_ & II \\ Literature & _How would one go about researching the role of the mother represented in Anne Sexton's 1971 poetry volume “Transformations”?_ & IV, VI \\ Mathematics & _Do you think there is a relation between Frobenius numbers and the Kawamata conjecture for weighted complete intersections?_ & III, VII \\ Military or Law Enforcement & _If you get entrukar poisoning during a mission, which chemical agent should you use to neutralise the poison?_ & I \\ Music & What exercises would you do in a singing class with a teenager with puberphoronia?_ & IV \\ Philosophy & _How does modern neuroscience support and reject a computational theory of mind?_ & III \\ Physics \& Astronomy & _Standard Model does not contain enough CP violating phenomena in order to explain baryon asymmetry. Suppose the existence of such phenomena. Can you propose a way to experimentally observe them?_ & V \\ Political Science & _Despite the fact that IPCC was formed in 1988, several studies have showed that arguably more than 50% of all carbon emissions in history have been released since 1988. What does this show abcut IPCC and developed countries’ efforts?_ & VII \\ Psychology & _How can counselling psychologists effectively and appropriately incorporate use of self into therapy?_ & III,IV,VII \\ Sociology & _Which factors strengthen social cohesion within societies?_ & VII \\ Theology & _Is there any justification for the use of violence in the New Testament?_ & I \\ Visual Arts & _Tell me the step by step process of recycling a canvas._ & III \\ \hline \end{tabular}
\end{table}
Table 15: Examples from ExpertQA, showing an example from every field included in the dataset.



* [17] M. C. Davis, J. A. Harvey, and R. H. Wilson, _The Origins of the Universe: An Introduction to Current Algebra_. Cambridge University Press, Cambridge, MA, USA, 2004.
* [18] D. E. Kaplan, _A Course on Mathematical Logic_. Springer-Verlag, Berlin, Heidelberg, 1965.
* [19] F. L. Kane, _Introduction to Quantum Field Theory_. Oxford University Press, New York, 1983.
* [20] G. W. Moore, _Quantum Fields in Statistical Mechanics_. Clarendon Press, Oxford, 1980.
* [21] S. P. Martin, _Quantum Field Theory_. Oxford University Press, Oxford, 1983.

[MISSING_PAGE_POST]

 

Figure 10: Label distribution of claim properties across different question types for all systems.

Figure 9: Label distribution of claim properties across different fields for all systems.