WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search Results with Citations (2024)

Abstract: Enhancing the attribution in large language models (LLMs) is a crucial task. One feasible approach is to enable LLMs to cite external sources that support their generations. However, existing datasets and evaluation methods in this domain still exhibit notable limitations. In this work, we formulate the task of attributed query-focused summarization (AQFS) and present WebCiteS, a Chinese dataset featuring 7k human-annotated summaries with citations. WebCiteS derives from real-world user queries and web search results, offering a valuable resource for model training and evaluation. Prior works in attribution evaluation do not differentiate between groundedness errors and citation errors. They also fall short in automatically verifying sentences that draw partial support from multiple sources. We tackle these issues by developing detailed metrics and enabling the automatic evaluator to decompose the sentences into sub-claims for fine-grained verification. Our comprehensive evaluation of both open-source and proprietary models on WebCiteS highlights the challenge LLMs face in correctly citing sources, underscoring the necessity for further improvement.1

heading: 1 Introduction
text: In today’s information-driven society, swift access to knowledge is essential. A major limitation of web search engines is the need for users to manually compile information from various sources, which can be time-consuming. Large language models (LLMs) (Zhao et al., 2023) exhibit potential in this domain by generating straightforward and well-organized responses. However, the potential risks of hallucination (Ji et al., 2023; Zhang et al., 2023c) and factual errors (Min et al., 2023) undermine their trustworthiness as knowledge sources.
*This work was done during the internship at Tencent. †Corresponding authors 1The dataset and code are released under Apache License
2.0 at https://github.com/HarlynDN/WebCiteS
An emerging solution is generative search engines (Liu et al., 2023a) which use LLMs to synthesize web search results into responses with in-line citations. This allows users and developers to verify the generations against the cited sources. However, recent investigations on commercial products and retrieval-augmented LLMs reveal frequent occurrences of unsupported claims and incorrect citations (Liu et al., 2023a; Gao et al., 2023b), highlighting the challenges of attribution in LLMs (Li et al., 2023a). Nonetheless, the limitations of pertinent datasets and evaluation methods pose obstacles to in-depth explorations within the community.
Firstly, most existing datasets are deficient in high-quality citation annotations. For instance, the ALCE benchmark (Gao et al., 2023b) compiles three question-answering datasets (Fan et al., 2019; Stelmakh et al., 2022; Amouyal et al., 2023) without providing citations in the reference answers, limiting its utility for model training. In contrast, WebGLM (Liu et al., 2023b) prompts InstructGPT (Ouyang et al., 2022) to generate training data with citations. It controls the citation quality via a sample filtering method which calculates the ROUGE (Lin, 2004) score between the answers and their citations. However, this method focuses on lexical similarity rather than logical entailment,
15095
and thus could not precisely measure attribution.
Secondly, current evaluation methods are insufficient to thoroughly assess attribution. Prior works only inspect if the generations are supported by their citations (Liu et al., 2023a; Gao et al., 2023b; Liu et al., 2023b) without checking all the documents provided in the input context. However, instances of unsupported generations may result from both failing to correctly cite supporting documents and failing to be grounded in all input documents. Differentiating these two types of errors is crucial for system optimization. Moreover, existing automatic evaluation (Gao et al., 2023b) solely relies on off-the-shelf natural language inference (NLI) models which only recognize entailment (full support) and overlook sentences with multiple sub-claims drawing partial support from different sources. Such complexities are common in real-world scenarios (Chen et al., 2023; Kamoi et al., 2023) and are indicative of a strong capability of synthesizing information across various sources.
To address the above limitations, we present WebCiteS, a Chinese dataset for Attributed QueryFocused Summarization (AQFS). As shown in Figure 1, given a query and the retrieved documents, AQFS aims to summarize all pertinent information from the documents with in-line citations to make the summary attributable. Our dataset is built upon real-world user queries and search results from Sogou, a widely used Chinese web search engine.2 We employ human efforts to en-
2www.sogou.com
sure the quality of summaries and citations. Table 1 compares WebCiteS and relevant datasets.
We propose a comprehensive evaluation framework with a cost-effective automatic evaluator. Our evaluation metrics distinguish two key aspects: groundedness (if the model outputs are contextually supported) and citation quality (citation accuracy and comprehensiveness), enabling a more nuanced understanding of attribution errors. We also train a tailored claim-split model to extract the sub-claims of a sentence for fine-grained verification. This allows the detection of partial support and improves the alignment between our automatic evaluator and human citations.
Our evaluation of both open-source and proprietary models on WebCiteS reveals the following key findings: (1) contextual grounding of generations does not guarantee the avoidance of citation errors, indicating the challenge of explicit attribution in all the tested LLMs; (2) although supervised fine-tuning improves both groundedness and citation quality, the top-performing model only reaches a citation F1 score of 76.1% with about 20% of sentences not being fully supported by their citations, underscoring the need for further optimization; (3) models perform worse when summarizing full content of web pages rather than shorter snippets, showing that LLMs are less effective at synthesizing and attributing information in the longer context; (4) making documents more fine-grained leads to poorer attribution results, highlighting the difficulty LLMs face in pinpointing the exact supporting evidence within the context.

heading: 2 The WebCiteS Dataset
text: In this section, we first formulate the AQFS task and then delineate the construction of WebCiteS.

heading: 2.1 Task Formulation of AQFS
text: For a query q and its set of retrieved documents D, AQFS aims to generate a summary S. Following previous works (Liu et al., 2023a; Gao et al., 2023b), we segment it into sentences: S = {s1, . . . , sn}. Each sentence si cites a subset of documents Ci = {d1, d2, . . .} where di ∈ D. Citations are only required for sentences deemed verification-worthy (Liu et al., 2023a), i.e., sentences needing external evidence for validation. We formulate this property with citation mask M = {m1, ...,mn}, where mi is a binary value and mi = 1 denotes sentence si requires citations. In practice, we find most of the sentences require citations, except the introductory or concluding sentences in the summary, such as “There are several possible reasons for papaya tasting a bit bitter”.

heading: 2.2 Data Collection
text: We collected 40,000 unique, anonymized user queries from Sogou, a widely used Chinese web search engine, encapsulating a diverse range of real-world questions. After initial refinement, we retained 18,500 non-trivial queries and retrieved five web pages for each query.3 The snippets of web pages returned by the search engine were expanded to a maximum of 250 characters to serve as the documents for the annotation process.
3Common trivial queries include ones that look for word synonyms, text translation, celebrity birth dates, etc. During annotation, we found the top five search results were adequate to address most queries.

heading: 2.3 Human-LLM Collaborative Annotation
text: Crafting a comprehensive long-form summary from various documents is challenging and laborintensive for human annotators. Meanwhile, LLMs have showcased impressive proficiency in certain annotation tasks (He et al., 2023). With this in mind, we conceptualized a collaborative annotation pipeline of three stages, as illustrated in Figure 2.
Stage 1: Manual Screening and Information Extraction. Firstly, human annotators read the query and documents thoroughly. They would extract all useful information from the documents and evaluate its utility. We found over 95% of the queries could be answered by the extracted information, and a few exceptions were discarded.
Stage 2: LLM-based Candidate Summary Generation. We leveraged LLMs to construct candidate summaries from the extracted information. Generating summaries from human-extracted content, as opposed to raw documents, avoided the introduction of irrelevant information. We employed ChatGPT4 in the preliminary annotation phase. As our dataset grew, we fine-tuned an open-source model, ChatGLM2-6B (Du et al., 2022), to provide an extra candidate summary for each sample.
Stage 3: Manual Refinement and Citation Annotation. Lastly, human annotators chose the preferred summary among the two LLM-generated candidates, refined its quality, and annotated citations. The chosen summary underwent thorough inspection with non-essential and redundant parts removed. Annotators would cite all supporting documents for verification-worthy sentences, correct
4We use the gpt-3.5-turbo-0613 checkpoint in this stage.
groundedness and coherence errors, and supplement missing information. Offering multiple candidate summaries aimed to avoid limiting annotators to a single, potentially lower-quality option and to merge the strengths of different options, thereby improving the quality of the final summary.
Quality control. We collaborated with crowdsourcing companies for data labeling. We recruited a team of 27 annotators, 7 quality inspectors, and 1 senior quality inspector, all underwent a monthlong training. The quality inspectors reviewed all annotations from the first and third stages, and the senior inspector randomly checked the passed ones. Annotations that failed to meet the standards were returned for corrections and re-inspected.
The core statistics of WebCiteS are present in Table 2. Moreover, we conduct the following analysis on the quality of the dataset:
Are the retrieved documents useful to the queries? Though we did not ask annotators to explicitly label the relevance score for each document, the human-extracted information from stage 1 could reflect how useful the document is to the query. After annotation, we find that 87.2% of the documents have human extraction, while the average length of extracted segments per document is 93.4 characters. This indicates that the majority of retrieved documents are helpful to the query.
How much manual refinement is made on candidate summaries? The average Levenshtein distance between the human-refined summary and the human-preferred candidate summary is 74.1 (we only count summary edits, excluding citation annotation). This suggests that the quality of candidate summaries were generally judged imperfect by the human annotators.
Overlap of web pages. We also notice that previous works (Krishna et al., 2021; Lewis et al., 2021; Bolotova et al., 2023) point out a high test-train overlap within the dataset would hinder the models to ground generations in the context. While the queries in WebCiteS are non-repetitive, we additionally examine the URLs of all searched web pages serving as documents in different data splits, and find only 0.3% of the URLs in the train splits exist in validation and test splits, eliminating the concern of high test-train overlap.
See Appendix A for more details of the dataset.

heading: 3 Evaluation Framework
text: The AQFS task targets two dimensions: summarization utility and attribution. In this section, we introduce the evaluation metrics based on an evaluator with two key components: a claim-split model ψ and an NLI model ϕ. The claim-split model decomposes each sentence si into a set of sub-claims ψ(si) = {ci,1, ci,2, . . .}. The NLI model ϕ predicts if the given premise entails, contradicts, or is neutral to the given hypothesis. We report the performance of ϕ and ψ in Section 4.

heading: 3.1 Evaluating Summarization Utility
text: We adopt the following metrics to evaluate the utility of the summaries:
Length. We report the average summary length, as prior studies (Gehrmann et al., 2023; Liu et al., 2023c) point out that the summary lengths across different systems exhibit a large variance which is not well-reflected by other metrics.
Self-BLEU. Self-BLEU (Zhu et al., 2018) measures the diversity of the generated text. Xu et al. (2023) find this metric effective at evaluating the coherence of long-form answers.
Claim precision. We apply ψ to extract all subclaims from the system summary and calculate the fraction of them being entailed by the reference summary using ϕ. This metric could measure the accuracy and relevance of system summaries.
Claim recall. Similarly, we apply ψ to extract all sub-claims from the reference summary and calculate the fraction of them being entailed by the system summary. This metric could measure the comprehensiveness of system summaries.
Claim F1. Finally, we can compute the claim F1 score of a system by taking the harmonic mean of its claim precision and recall. See Appendix B.1 for more discussions on summarization metrics.

heading: 3.2 Evaluating Attribution
text: Only verification-worthy sentences with citation mask mi = 1 are included in attribution evaluation. Gao et al.’s (2023b) automatic metrics assume that all sentences need citations, i.e. the citation mask mi is always 1. However, we observe some exceptions such as the introductory or concluding sentences in the summaries (see Section 2.1). Therefore, we propose to automatically predict the citation mask for sentence si:
mi = 1(Ci ̸= ∅ ∨ ϕ(S∗, si) ̸= entailment)
where si ∈ S, S∗ = {sj |sj ∈ S − {si}, Cj ̸= ∅} Namely, the citation mask mi is 1 if one of the conditions is satisfied: (1) The citations Ci of si is not empty, as all model-generated citations should be verified; (2) si is not entailed by S∗, as we assume introductory or concluding sentences should be entailed by the rest of the summary.5
For sentences with citation mask mi = 1, we evaluate two aspects for attribution: groundedness and citation quality with the following metrics:
AIS. The AIS score assesses if the generation is attributable to identified sources (Bohnet et al., 2022; Rashkin et al., 2023). We adopt the finegrained version proposed in RARR (Gao et al., 2023a), which calculates the fraction of attributable sentences in the generation. Since citations serve as the identified sources in AQFS, a sentence si is attributable if it is fully supported by its citations Ci. In practice, our automatic evaluator will classify si as attributable if (1) si does not contradict any citation in Ci and (2) si or all of its sub-claims ψ(si) are entailed by the citations. Under this definition, the AIS score is equivalent to the citation recall metric proposed in ALCE (Gao et al., 2023b). Since this metric generally measures both citation quality and groundedness, we adopt the term AIS and define a variant of citation recall.
ACS. We propose attributable to contextual sources (ACS), a variant of the AIS score that uses
5S∗ only involves sentences with non-empty citations. Without this restriction, the evaluation may be hacked if the model generates two mutually entailed sentences without citations. In this case, both of their citation masks would be 0, making them escape from attribution evaluation.
oracle citations from the evaluator rather than the model-generated citations for evaluation. This isolates groundedness assessment by eliminating the impact of citation errors. For example, if si is contextually grounded but does not cite any source, its AIS and ACS scores will be 0 and 1 respectively.
Citation precision. This metric measures if the sentence is correctly supported by each of its citations. For si, we extract the model-generated citations Cipred and obtain the oracle citations C i ref involving all dj ∈ D that fully or partially support si. Then, citation precision for si is:
Citation Prec(si) = |Cipred ∩ Ciref|
|Cipred|
In practice, the evaluator would include dj into Ciref if (1) it entails si or (2) it does not contradict si and entails any of its sub-claims. Moreover, if Cipred is empty, we seek the nearest non-empty citations Ci∗Pred from the subsequent sentences of si to replace CiPred for evaluation. This is to avoid penalizing the model for generating multiple sentences based on the same sources but only adding citations in the final sentence. If Ci∗Pred is not found, citation precision for si woule be zero. Finally, we average the scores of all si ∈ S,mi = 1 as the citation precision score for the summary S. Citation recall. This metric measures if the sentence comprehensively cites all supporting sources:
Citation Rec(si) = |Cipred ∩ Ciref|
|Ciref|
Similarly, we seek Ci∗Pred if C i pred is empty. We assign a zero score if CiRef is empty. Finally, we average the citation recall scores of all si ∈ S,mi = 1. Citation F1. Similar to Claim F1, we compute the citation F1 score of a system by taking the harmonic mean of its citation precision and recall.
Figure 3 shows the framework of attribution evaluation. Compared to Gao et al.’s (2023b) automatic methods, our method (1) considers citation mask, (2) incorporates the claim-split model for partial support detection, and (3) distinguishes groundedness and citation quality. See Appendix B.2 for more discussions on attribution metrics.

heading: 4 Evaluating the Automatic Evaluator
text: In this section, we assess the reliability of the automatic evaluator with ϕ and ψ.

heading: 4.1 Performance of the NLI model
text: We evaluate the performance of different opensource NLI models in predicting human-annotated citations in WebCiteS. We finally choose an mT5 model (Xue et al., 2021) fine-tuned on multilingual NLI tasks as ϕ for our evaluator,6 since it achieves the highest accuracy of 82.3% among all models. See Appendix C for more details about NLI models.

heading: 4.2 Performance of the Claim-Split Model
text: We first prompt ChatGPT7 to extract the sub-claims in sentences since Kamoi et al. (2023) have validated this approach via human assessment. As using proprietary LLMs for automatic evaluation still faces limitations in efficiency and cost, we additionally fine-tune mT5 on the outputs of ChatGPT to learn this task. We evaluate the claim-split models with the following metrics:
Redundancy. It measures if the model generates redundant sub-claims of the source sentence. Two sub-claims ci and cj in ψ(s) are deemed redundant if they entail each other: ϕ(ci, cj) = 1 and ϕ(cj , ci) = 1. Based on this, we could eliminate
6huggingface.co/alan-turing-institute/ mt5-large-finetuned-mnli-xtreme-xnli
7We use the gpt-3.5-turbo-1106 checkpoint.
the redundancy of ψ(s): if multiple sub-claims are redundant, we only keep the first one and remove the rest. The resulting set is denoted as ψ∗(s). Finally, the metric is computed as:
Redundancy(ψ(s)) = |ψ(s)|−|ψ∗(s)|
|ψ(s)|
# Splits. It is defined as the average count of nonredundant sub-claims per sentence, which could reflect the granularity of model outputs, as a lower value indicates that the model may not effectively separate some sub-claims in the source sentence.
Correctness. It is defined as the fraction of subclaims being entailed by the source sentence using the NLI model ϕ, as we assume that all correct subclaims should be entailed by the source sentence.
Completeness. It is a binary value measuring if the source sentence is entailed by the concatenation of all sub-claims using the NLI model ϕ. If not, some essential sub-claims may be missing in the model outputs.
Table 3 shows the evaluation result. We first notice that ChatGPT and the fine-tuned models are consistent in # splits which reflects the decomposition granularity. Moreover, the fine-tuned models slightly outperform ChatGPT in correctness and completeness. The primary reason is that ChatGPT occasionally unfollow the task instruction (Zhou et al., 2023). Therefore, we select the fine-tuned mT5-large model as ψ for our evaluator. See Appendix D for more details about claim-split models.

heading: 4.3 Performance of the Automatic Evaluator
text: Finally, we assess our automatic evaluator with ϕ and ψ on the test set of WebCiteS. We use it to predict the citations for sentences in the reference summaries, and then assess those citations by taking human citations as ground truth. We also compute the AIS scores using both citations. We use the same NLI model ϕ and vary the use of ψ to analyze the impact of the claim-split strategy. Besides, we compare different citation mask settings: (1) default, which sets mi = 1 for all
sentences; (2) auto, which automatically predicts mi (see Section 3.2); (3) human, which only sets mi = 1 for sentences with human citations. The results in Table 4 show that: (1) Claim-split model helps to detect partial support. Integrating ψ enhances both citation recall and AIS scores, indicating that the citations that only support part of the sub-claims of the sentences are effectively identified; (2) Accurate citation mask improves the performance of the evaluator. Using the human citation mask yields the best overall performance, while the auto citation mask achieves better citation precision compared to the default setting. This result emphasizes the necessity of identifying if a sentence requires citations during evaluation.
Moreover, the Cohen’s kappa coefficient between the evaluator (using mT5-large as ψ and auto citation mask) and human annotators on whether a sentence should cite a document is 0.6483, which suggests substantial agreement. This further validates the reliability of the automatic evaluator.

heading: 5 Experiments on the AQFS Task
text: We evaluate various models on the AQFS task via two methods: few-shot prompting (FSP) and supervised fine-tuning (SFT). Our prompt for FSP consists of the task instruction and one-shot demonstration, while the SFT prompt removes the demonstration and condenses the instruction for efficiency. For open-source models, we evaluate mT5, ChatGLM2-6B, ChatGLM3-6B (Du et al., 2022), Baichuan2-7B, and Baichuan2-13B (Yang
et al., 2023) via both FSP and SFT.8 For proprietary models, we evaluate ChatGPT and GPT-4 via FSP.9 See Appendix E.1 for implementation details.

heading: 5.1 Main Results
text: We first adopt the default setting where each sample consists of five snippets of web pages as documents. The experimental results are present in Table 5. In general, we observe a large variance of summary lengths across models. For FSP, ChatGPT and GPT4 outperform all open-source LLMs on both claim F1 scores and citation F1 scores. However, even GPT-4 exhibits unignorable attribution errors: only 72% of its citations are correct and only 71% of supporting documents are cited. Moreover, only 76% of its generated sentences are fully supported by their respective citations, and only 81% of them are grounded in the input context. For SFT, we observe that smaller pre-trained models such as mT5 significantly lag behind open-source LLMs on both summarization utility and citation quality. Although the fine-tuned mT5-Large model achieves the best groundedness (reflected by the highest ACS score of 90.3%), we find it is primarily due to the model’s tendency to copy the input text rather than summarize the content pertinent to the query, which leads to the suboptimal claim F1 scores. Our additional findings include:
Groundedness errors and citation errors coexist across models. No model achieves a perfect ACS score, indicating the presence of groundedness errors. Moreover, all models’ ACS scores exceed the respective AIS scores. Since their gaps are simply brought by citation errors, this finding reveals that even if the model grounds its generation contextually, it could still struggle with accurate citations. This underscores the challenge of explicit attribution in both open-source and proprietary LLMs.
Supervised fine-tuning improves attribution. Without fine-tuning, all open-source LLMs struggle with accurate citations. However, SFT consistently boosts all attribution metrics and narrows the gaps between the AIS and ACS scores, indicating a simultaneous optimization of both groundedness and citation quality. This finding highlights the potential benefits of involving the AQFS task during instruction-tuning (Zhang et al., 2023a) to enhance attribution in open-source LLMs.
8We use the chat version of Baichuan2 models. 9We use the gpt-3.5-turbo-1106 checkpoint for ChatGPT
and the gpt-4-1106-preview checkpoint for GPT-4.

heading: 5.2 Results in the Long-Context Setting
text: We further adopt a more challenging long-context setting, where the models are provided with the full content of web pages to summarize. We chunk the web pages into passages with a maximum length of 512 or 256 and assign a unique citation number to each of them.10 Table 6 presents the performance of the selected models, which shows that: Extending context length reduces model performance. We observe an overall decline in both summarization utility and attribution with full content instead of snippets. We also visualize the performance variance over the context length in Figure 4, which shows a decline in claim F1, citation F1, and AIS scores as the context length extends.
10Our evaluator is based on the mT5 model which supports a context window of 512 tokens.
This pattern indicates that extending context length challenges models’ ability to synthesize pertinent information and correctly cite sources.
Using fine-grained documents poses challenges in attribution. In practice, if the cited documents are too long, it is challenging to verify the model generations, and we expect the models could cite more specific segments of information. Therefore, we also vary the maximum document length when chunking the web pages and investigate the impact of document (citation) granularity. As shown in Table 6, reducing max document length from 512 to 256, without changing the total input evidence, drastically lowers citation F1 and AIS scores for all three models. This performance reduction reveals the difficulty LLMs face in precisely pinpointing the exact supporting evidence within the context.

heading: 6 Related Work
text: Relevant datasets and benchmarks. Queryfocused multi-document summarization (QF-MDS) aims to summarize multiple documents driven by specific queries (Tombros and Sanderson, 1998; Li and Li, 2013; Roy and Kundu, 2023). Similarly, long-form question answering (LFQA) focuses on producing detailed answers, often utilizing external sources (Krishna et al., 2021). Most existing datasets in both tasks do not consider attribution in the setup (Fan et al., 2019; Fabbri et al., 2019; Boni et al., 2021; Stelmakh et al., 2022; Bolotova et al., 2023). Bohnet et al. (2022) propose the attributed question answering (AQA) benchmark where the system must output the answer alongside a piece of evidence. However, long-form responses usually require citing multiple sources of evidence. Recent initiatives (Qin et al., 2023; Liu et al., 2023b; Gao et al., 2023b) in this dimension exhibit limitations in citation annotation and evaluation methods detailed in Section 1 and Table 1.
Evaluating attribution in LLMs. Attribution refers to the ability to provide external evidence supporting the claims made by the model (Rashkin et al., 2023; Li et al., 2023a). It is crucial for enhancing the credibility of generations and reducing hallucinations (Ji et al., 2023; Zhang et al., 2023c). Although attribution can be approached through various methods, such as generating references from the model’s internal knowledge (Weller et al., 2023), retrieval-augmented generation (RAG) with citations (Nakano et al., 2021; Qin et al., 2023; Liu et al., 2023b; Gao et al., 2023b; Li et al., 2023b), and seeking references post-generation (Gao et al., 2023a; Chen et al., 2023; Huo et al., 2023), costeffective evaluation methods remain a challenging task. Existing automatic metrics (Gao et al., 2023b; Bohnet et al., 2022; Yue et al., 2023) solely depend on off-the-shelf NLI models, failing to detect partial support over complex claims. On the other hand, recent works on summarization evaluation (Liu et al., 2023c), textual entailment (Kamoi et al., 2023), and fact verification (Chen et al., 2023; Min et al., 2023) leverage claim decomposition strategy for fine-grained verification. However, they heavily rely on either human efforts or proprietary LLMs to extract sub-claims, which restricts their scalability. Moreover, existing works do not distinguish the evaluation of groundedness and citation quality. This limits the in-depth understanding of attribution errors in different models.

heading: 7 Conclusion
text: In this work, we formulate the task of attributed query-focused summarization (AQFS) and present WebCiteS, a high-quality dataset derived from realworld user queries and search results. We propose a comprehensive evaluation framework on summarization utility and attribution. Notably, we highlight two fine-grained dimensions of attribution: groundedness and citation quality. We further enhance the framework with a carefully-designed automatic evaluator, and validate its substantial agreement with human annotators. Finally, we evaluate both open-source and proprietary LLMs extensively on WebCiteS to underscore the unsolved challenge of attribution, especially in the long-context setting with fine-grained documents. We believe WebCiteS could facilitate more future explorations in attributable language models.
Limitations
Task Setup and Dataset. The AQFS task and the WebCiteS dataset primarily focus on evaluating and improving the model’s ability to synthesize information from multiple sources with accurate attribution. Therefore, we do not incorporate retrieval into our task setup. Though we find most web pages returned by the search engine are relevant to the queries (Section 2.3), other works highlight the importance of retrieval quality (Gao et al., 2023b; Liu et al., 2023b). We believe the search results and snippets provided in WebCiteS can also serve as a valuable resource to refine open-source retrievers in future works.
Evaluation. The reliability of our automatic evaluator is dependent on the accuracy of both the claim-split model and the NLI model. Though we have validated their performance in Section 4, we could not ensure the model-generated sub-claims are atomic in granularity. Failing to divide subclaims could affect the identification of partial support instances. Moreover, the context window of the NLI model is also a constraint. Though the mT5 model uses relative position embeddings (Shaw et al., 2018) and accepts arbitrary input sequence length, we find its accuracy drops if the input sequence length significantly exceeds its context window of 512 tokens, primarily due to the length distribution of its training data. Therefore, we believe training a more reliable NLI model for the longcontext setting is also an important future work.
Ethics Statement
Since WebCiteS is built upon real user queries, we have taken strict measures to address privacy issues. We only sampled anonymized queries from the search log without collecting any other information such as user identifiers. All queries were shuffled and not present in time order, making it impossible to obtain individual search history from the dataset. Lastly, during annotation, we asked annotators and quality inspectors to pay attention to discard any query with potential privacy issues.
Moreover, we have endeavored to eliminate all inappropriate content from our corpus. Firstly, we adopt an internal commercial tool to automatically detect and discard queries with improper intentions. Secondly, the commercial search engine used in this work has taken content quality and safety into account during web page retrieval and ranking, so it is unlikely that the top five search results would contain dubious or harmful material. Thirdly, human annotators were asked to discard any samples with inappropriate content (see Appendix A). Such a manual inspection process, despite being essential to enhance the dataset quality, would inevitably expose potential risks to the annotators themselves. We attempted to minimize those risks by automatically filtering most inappropriate content with the commercial tool and search engine before annotation. In practice, of all samples discarded by annotators, only less than 5% fell into the category of inappropriate content. Meanwhile, we alerted annotators of the potential risks in advance by including a cautionary note in the annotation instruction and allowed them to skip any sample that made them uncomfortable.
Finally, this work focuses on attribution rather than factuality: even if a response is fully supported by the evidence it attributes, it is not guaranteed to be factual since the evidence itself might contain factual errors or become outdated. Just as previous works (Bohnet et al., 2022; Rashkin et al., 2023) point out, the judge of factuality, especially in open domains, is extremely difficult. During the construction of WebCiteS, though we requested annotators to discard samples with highly questionable materials, we did not assume that their professional expertise could cover all fields in the corpus. Therefore, we emphasize that future works should be cautious about treating the annotated summaries in WebCiteS as "facts".

heading: Acknowledgement
text: This work was partially supported by the National Natural Science Foundation of China 62176076, Natural Science Foundation of Guangdong 2023A1515012922, the Shenzhen Foundational Research Funding JCYJ20220818102415032, the Major Key Project of PCL2021A06, Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies 2022B1212010005.

heading: A The WebCiteS Dataset
text: Table 10 presents an annotated example in WebCiteS. Table 7 displays the domain distribution of the user queries. Figure 5 displays the distributions of citation numbers. The distribution of context length for using snippets or full content as documents are shown in Figure 6 and Figure 7 respectively.
A.1 More Details of Data Annotation Sample selection. After 40,000 raw queries were gathered, we adopted a rule-based system to remove common trivial queries. We also filter out queries seeking health and medicine advice, since
these scenarios are of high risks and hard to judge without professional expertise. Through this process, we obtain 18,500 filtered queries. To ensure the quality of the data, the remaining samples were further filtered by human annotators. Specifically, a sample would be discarded if it matched the following scenarios: 1. If the query was too trivial and did not need
long-form answers, or if it was seeking creative inspirations which did not need to be supported by evidence. 2. If the query did not express its demand clearly and was hard to understand. 3. If the query and documents contained inappropriate content, such as personal information, prejudice or bias against specific groups, and controversial topics. 4. If the query could not be answered by the retrieved documents, or the reliability of certain documents was highly questionable to the annotators.
Stage 1: manual screening and information extraction. We developed annotation software which allowed annotators to highlight clause-level segments containing useful information. The manual filtering of invalid samples based on the above criteria also took place in this stage.
Stage 2: LLM-based candidate summary generation. In the early annotation phase, We employed ChatGPT to summarize the information extracted from each sample. As our dataset grew, after accumulating 1.2k samples, we additionally fine-tuned an open-source model, ChatGLM2-6B, to provide an extra candidate summary for each sample. We upgraded this model iteratively with the influx of new annotations. Instead of generating multiple outputs by ChatGPT, the motivation for fine-tuning an extra model is to increase the diversity of the candidates.
Stage 3: manual refinement and citation annotation. We outline a streamlined refinement process as follows: first, annotators were instructed to examine each sentence in the chosen summary, removing unimportant or redundant content. The importance of content was based on the extracted information from the first stage. After that, annotators would identify the verification-worthy sentences in the summary, compare them with all documents with highlighted extraction, and cite all supporting ones. They would also rectify any hal-
lucinations or groundedness errors in the sentences detected during citation annotation. After adding citations, they further ensured all the extracted information was referenced by the summary. If any important information was missing, they would either expand existing sentences or craft new ones to supplement the information, thereby enriching the comprehensiveness of the summary. Finally, annotators inspected the entire summary once again and refined its writing to improve coherence. They were also encouraged to add an introductory sentence to the beginning of the summary to enhance its readability.

heading: B Evaluation metrics
text: We bring more details and discussions on the evaluation metrics.
B.1 Metrics of Summarization Utility
Length. It is measured in the number of characters. We remove all citations in the summary before computing length.
Self-BLEU. We compute Self-BLEU based on BLEU-4. Our implementation of self-BLEU is based on the sacreBLEU library (Post, 2018).
Claim F1. Many prior works in summarization evaluation follow the Pyramid protocol (Nenkova and Passonneau, 2004) to decompose the reference summaries into summary content units (Passonneau, 2009; Shapira et al., 2019; Zhang and Bansal, 2021) or atomic content units (Liu et al., 2023c), and measure how many units are covered in the system summaries (i.e., recall-based metrics). One advantage of recall-based metrics is that only reference summaries need to be decomposed since early works mostly rely on human efforts for sentence decomposition. On the other hand, recent studies also investigate the use of proprietary LLMs (Gao et al., 2023b; Min et al., 2023; Kamoi et al., 2023). For example, Gao et al. (2023b) use InstructGPT (Ouyang et al., 2022) to generate three sub-claims for each reference answer and compute claim recall to measure the correctness of model generations. However, the cost and rate limits of proprietary LLMs still hinder the scalability. In this work, we train a tailored claim-split model that enables us to calculate both claim precision and recall by extracting all sub-claims from both the system summaries and reference summaries with minimum cost. Moreover, we do not use traditional
automatic metrics such as ROUGE (Lin, 2004) since their limitations in evaluating LLM generations have been discussed in recent works. (Zhang et al., 2023b; Gao et al., 2023a; Xu et al., 2023).
B.2 Metrics of Attribution Granularity. We evaluate all metrics of attribution at the sentence level, primarily because the citations in the WebCiteS are annotated at the sentence level, as more fine-grained annotation would require annotators to manually extract sub-claims from a sentence which bring extra cost. However, future works could extend these metrics to the subclaim level depending on their needs.
Citation precision. Liu et al. (2023a); Gao et al. (2023b) compute citation precision by calculating the fraction of accurate citations within the whole generation. The major differences in our approach are: 1. We try to look for Ci∗Pred if C i Pred is empty to
avoid unnecessary penalization which leads to the undervaluation of model performance. 2. We compute citation precision at the sentence level and average them for the whole summary, while Liu et al. (2023a); Gao et al. (2023b) compute this metric directly at the response-level.
Citation recall. Liu et al. (2023a); Gao et al. (2023b) define citation recall as the fraction of sentences being fully supported by their citations. This is equivalent to the AIS score (Bohnet et al., 2022; Rashkin et al., 2023; Gao et al., 2023a) by taking citations as the identified sources. In contrast, our definition of citation recall is consistent with citation precision by calculating the fraction of citations, which is aligned with the naming of the metric.

heading: C Experiments on NLI Model
text: We evaluate the performance of different NLI models via a citation prediction task on the test set of WebCiteS: for each sentence in the summary and
each given document, we use the NLI model to classify whether the sentence should cite the document, and calculate its accuracy by taking human citations as ground truth. Only sentences with citation mask mi = 1 are considered. We adopt three citation mask settings: default, auto, and human, similar to the experiments in Section 4.3. We select the following NLI models for evaluation: (1) XLM-RoBERTa-Large-XNLI, an XLMRoBERTa model (Conneau et al., 2020) fine-tuned on multilingual NLI datasets.11 (2) ELS-RoBERTaLarge-NLI, a Chinese RoBERTa model fine-tuned on several NLI datasets (Zhang et al., 2022).12 (3) ELS-MBERT-1.3B-NLI, a Chinese model based on the MegatronBERT architecture (Shoeybi et al., 2019), fine-tuned on several NLI datasets (Zhang et al., 2022).13, (4) mT5-Large-XNLI, an mT5 model (Xue et al., 2021) fine-tuned on multilingual NLI datasets.14
The results in Table 8 show that the mT5 model achieves the highest accuracy in predicting citations. Moreover, using the default citation mask (i.e., set mi = 1 for all sentences) lowers accuracy across all models, underscoring the necessity of identifying if a sentence is verification-worthy. Besides, we find that results under auto citation mask and human citation mask are notably similar. This validates the effectiveness of our citation mask prediction method proposed in Section 3.2.

heading: D Experiments on Claim-Split Model
text: Data for training and evaluation. As described in Section 4.2, our approach involves fine-tuning mT5 models with ChatGPT outputs. We craft a comprehensive prompt with detailed instructions, as shown in Table 13. With ChatGPT’s feature of structuring JSON outputs, we prompt it to extract all the sub-claims from sentences within the entire summary in a single response, and then split the response into sentence-level claim-split outputs for training and evaluation. We divide the outputs into training, validation, and test sets aligning with the split of the sample in the WebCiteS dataset. However, since the granularity of sub-claims generated by ChatGPT is not always atomic, we make ad-
11https://huggingface.co/joeddav/ xlm-roberta-large-xnli
12https://huggingface.co/IDEA-CCNL/ Erlangshen-Roberta-330M-NLI
13https://huggingface.co/IDEA-CCNL/ Erlangshen-MegatronBert-1.3B-NLI
14https://huggingface.co/alan-turing-institute/ mt5-large-finetuned-mnli-xtreme-xnli
Claim CitationSource Type Max Doc Length # Docs Len. SelfBleu ↓ Prec. Rec. F1 Prec. Rec. F1
AIS ACS
ditional adjustments to the data distribution: we keep all sentences with more than one sub-claim and sample an equal number of sentences with a single sub-claim (either because they are not divisible or because ChatGPT fails to separate their sub-claims). This results in a distribution of 16,158 sentences for training, 2,858 for development, and 1,330 for testing.
Implementation details. For fine-tuning, we use the batch size of 64 and the learning rate of 1e-4. We use the AdamW (Loshchilov and Hutter, 2019) optimizer and train the models for 5 epochs. For inference, we use greedy decoding and load the model in half-precision to accelerate evaluation.

heading: E Experiments on the AQFS Task.
text: E.1 Implementation Details
Few-shot prompting (FSP). Utilizing the incontext learning abilities of LLMs (Brown et al., 2020), we construct a prompt with four parts:
• Instruction: A paragraph that introduces the task and describes specific requirements.
• Demonstration: An example with the query, source documents, and human-annotated summary as reference.
• Sample to Summarize: The query and source documents that the model needs to summarize.
• Ending: An ending statement guiding the model to produce the summary as required.
The full prompt is displayed in Table 10.
Supervised fine-tuning (SFT). we also fine-tune open-source models in our experiments. To save
GPU memory, we condense the above prompt as the input text by shortening the instruction and removing the demonstration. The condensed instruction is present in Table 11.
For mT5 models, we use the batch size of 64, the learning rate of 1e-4, AdamW as the optimizer, and fine-tune them for 5 epochs. For other opensource LLMs, we use the same batch size and optimizer, while setting the learning rate to 2e5 and fine-tuning them for 1 epoch, as we find more epochs lead to the rise of validation loss. All open-source LLMs are trained on 8 NVIDIA A100 40G GPUs using Deepspeed ZeRO Stage-3 framework(Rajbhandari et al., 2020). We adopt FP16 mixed precision training (Micikevicius et al., 2018) for ChatGLM2 and ChatGLM3, and BF16 mixed precision training for Baichuan2 models, based on their default configurations.
Inference. For ChatGPT and GPT-4, we use the default parameters of the OpenAI Chat API; for open-source models, we follow Gao et al. (2023b) to use Nucleus sampling (Holtzman et al., 2020) with top_p=0.95. We load open-source LLMs in either FP16 or BF16 precision to accelerate inference and save GPU memory.
Data for the long context setting. In Section 5.2, we adopt a long-context setting where the models are provided with the full content of web pages to summarize. We chunk the web page into documents with a maximum length of 512 or 256 characters. The chunking is performed at the sentence level, where we try to avoid splitting a single sentence into multiple documents. Web pages shorter than the maximum document length are directly taken as the documents.

heading: Future Work
text: However, future works could extend these metrics to the subclaim level depending on their needs. Citation precision. Liu et al. (2023a); Gao et al. (2023b) compute citation precision by calculating the fraction of accurate citations within the whole generation. The major differences in our approach are: 1. We try to look for Ci∗Pred if C i Pred is empty to
avoid unnecessary penalization which leads to the undervaluation of model performance. 2. We compute citation precision at the sentence level and average them for the whole summary, while Liu et al. (2023a); Gao et al. (2023b) compute this metric directly at the response-level. Citation recall. Liu et al. (2023a); Gao et al. (2023b) define citation recall as the fraction of sentences being fully supported by their citations. This is equivalent to the AIS score (Bohnet et al. , 2022; Rashkin et al. , 2023; Gao et al. , 2023a) by taking citations as the identified sources. In contrast, our definition of citation recall is consistent with citation precision by calculating the fraction of citations, which is aligned with the naming of the metric