Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration (2024)

Abstract: Despite efforts to expand the knowledge of large language models (LLMs), knowledge gaps—missing or outdated information in LLMs—might always persist given the evolving nature of knowledge. In this work, we study approaches to identify LLM knowledge gaps and abstain from answering questions when knowledge gaps are present. We first adapt existing approaches to model calibration or adaptation through fine-tuning/prompting and analyze their ability to abstain from generating low-confidence outputs. Motivated by their failures in self-reflection and over-reliance on heldout sets, we propose two novel approaches that are based on model collaboration, i.e., LLMs probing other LLMs for knowledge gaps, either cooperatively or competitively. Extensive experiments with three LLMs on four QA tasks featuring diverse knowledge domains demonstrate that both cooperative and competitive approaches to unveiling LLM knowledge gaps achieve up to 19.3% improvements on abstain accuracy against the strongest baseline. Further analysis reveals that our abstention methods pinpoint failure cases in retrieval augmentation and knowledge gaps in multi-hop reasoning.1

heading: 1 Introduction
text: LLMs demonstrate impressive capabilities of encoding real-world knowledge in model parameters and leveraging it to aid knowledge-intensive tasks (Petroni et al., 2019; Brown et al., 2020; Yu et al., 2023a). But when such knowledge is missing or unreliable, they resort to hallucinations (Ji et al., 2023) and biases (Feng et al., 2023a), while still “speaking with confidence.” A growing body of work seeks to expand LLM knowledge through retrieval augmentation (Guu et al., 2020; Borgeaud et al., 2022; Khattab et al., 2022; Shi et al., 2023; Chen et al., 2023), search engine integration (Nakano
1Code and data will be publicly available at https://github.com/BunsenFeng/AbstainQA.
et al., 2021; Press et al., 2023), and multi-LM collaboration (Luo et al., 2023; Feng et al., 2023b). However, LLM knowledge gaps might always persist due to the ever-evolving nature of knowledge (Kandpal et al., 2023; Mallen et al., 2023; De Cao et al., 2021; Hernandez et al., 2023; Kasai et al., 2024). Consequently, we posit that abstaining from generating low-confidence outputs should be a part of LLMs’ functionality, and ask a crucial research question: how to identify knowledge gaps in LLMs? Developing and evaluating robust mechanisms to address the abstain problem improves LLM reliability, reduces hallucinations, and mitigates biases due to model uncertainty.
We hypothesize that there are three ways to operationalize the abstain problem: (1) from the data perspective, LLMs should abstain when knowledge is missing in training data or curated from unreliable sources; (2) from the modeling perspective, LLMs should abstain when knowledge is not encoded in model parameters and cannot be inferred; (3) from the usage perspective, LLMs should abstain when generated outputs would provide an incorrect answer. Since users are directly interacting with and impacted by LLM-generated texts, not model parameters or training data, we formulate and evaluate LLM abstention focusing on the factual correctness of LLM generations.
Specifically, we focus on abstaining in questionanswering (AbstainQA): given an LLM and a question q, we aim to develop an abstain function f(LLM, q) → {true, false} to indicate whether the LLMs should abstain based on the limitations of their internal knowledge (Kamath et al., 2020; Jiang et al., 2021; Whitehead et al., 2022). To facilitate a comprehensive understanding of existing approaches that can be employed for LLM abstention, we first identify 11 methods from previous works and adapt them to incorporate the abstain functionality. We categorize these baselines into (1) calibration-based, such as employing tempera-
14664
ture scaling (Guo et al., 2017; Jiang et al., 2021) to calibrate models and set a confidence score threshold for abstention; (2) training-based, such as employing linear probing on LLMs’ hidden layers to evaluate the veracity of generated texts (Slobodkin et al., 2023; Azaria and Mitchell, 2023); (3) prompting-based, such as prompting LLMs to request more information before answering questions (Feng et al., 2023b); and (4) self-consistency based, such as generating multiple chains-of-thought (Wei et al., 2022).
Among the 11 baselines, a majority and often the stronger approaches would require a held-out set for training and hyperparameter tuning, potentially harming generalization across knowledge domains. In addition, these approaches often rely on a “selfreflection” assumption that a single LLM could be employed or adapted to evaluate its own generated texts; however, challenges such as hallucination and confirmation biases cast doubt on this assumption (Ji et al., 2023; Xie et al., 2023).
We propose multi-LLM collaboration-based approaches, enabling robust LLM abstention through multi-LLM collaboration to reflect on generated text in cooperative or competitive settings while removing the need for held-out sets. For multiple LLMs to work in cooperation (COOPERATE), the LLM employs other models to provide feedback on the proposed answer and reasoning, and synthesizes the outputs into an overall abstain decision. For a competitive setting (COMPETE), the LLM is challenged by other LLMs with conflicting knowledge, producing an abstain decision based on whether it continues to prefer the proposed answer in the presence of conflicting evidence.
We evaluate the baselines and proposed collaboration approaches for abstention with three LLMs, on four knowledge-intensive QA tasks spanning diverse knowledge domains and reasoning. Extensive experiments demonstrate that COOPERATE and COMPETE are strong abstain mechanisms that outperform all baselines in 9 of the 12 settings across tasks and models, achieving an improvement of up to 19.3% in abstain accuracy. Further analysis reveals that the proposed collaboration-based approaches could help identify failure cases of retrieval augmentation, pinpoint knowledge gaps in multi-hop reasoning, and more. Our contributions are (1) a critical evaluation and typology of diverse existing methods, repurposed to identify knowledge gaps in LLMs, as well as (2) two novel, robust multi-LLM collaboration methods to detect LLM
knowledge gaps, COOPERATE and COMPETE.

heading: 2 Identifying Knowledge Gaps in LLMs
text: We propose to repurpose existing approaches to enable abstain functionality in the face of knowledge gaps in LLMs. We conceptually categorize these approaches into calibration (§2.1), training (§2.2), prompting (§2.3), and self-consistency (§2.4) methods. Empirically observing that these approaches are dependent on external knowledge/data sets and often suffer from instability, we propose a new category based on multi-LLM collaboration (§2.5).

heading: 2.1 Calibration-Based
text: Token Probability LLMs are often inherently calibrated to different extents (Radford et al., 2019; Liang et al., 2023) and token probabilities might be employed off-the-shelf. Let (q, ā) ∈ H be a heldout set of questions and gold answers for hyperparameter tuning, while LLM produces an answer a = LLM(q) with a confidence score based on token probabilities p(a) ∈ [0, 1]. We set a threshold p∗ by minimizing abstain errors on H, formally:
p∗ = argmin p′∈[0,1]
∑
a=ā
1(p(a) < p′) + ∑
a̸=ā
1(p(a) ≥ p′)
where 1 denotes the indicator function. At inference time, the LLM abstains when p(a) < p∗.
Temperature Scaling Before setting a threshold p∗ on confidence scores, temperature scaling (Guo et al., 2017; Jiang et al., 2021) first reshapes the token probability distributions by introducing a scalar temperature τ > 0. Given an answer probability distribution a, we transform it into softmax(a/τ), then optimize τ on the held-out set H with crossentropy loss to obtain τ∗. A probability threshold p∗ is then set over {softmax(LLM(q)/τ∗)}q∈H and applied at inference time.
Ask for Calibration In addition to token probabilities, LLMs could also verbalize confidence scores: we follow Tian et al. (2023) to obtain confidence scores p(a) by employing a two-step process: 1) generating an answer with the LLM and 2) eliciting a confidence score by prompting with “Provide the probability that your guess is correct.”2 A threshold p∗ is similarly set over verbalized scores and applied at inference time.
2Full prompt templates are in Appendix B.
Cooperate Question LLM
Proposed Answer
self
others
Factual Expert Commonsense Expert Math Expert
“Please review the proposed answer and provide feedback on its correctness.”
Domain Knowledge 1
Feedback 1 Feedback 2 Feedback 3
Feedback 1 LLM 1
Feedback 2 LLM 2
Feedback 3 LLM 3
Judge “Based on the feedback, the proposed answer is: A. True B. False The answer is B. False”
“Please propose an alternative answer.”
Alternative Answer 1
“Generate a knowledge paragraph about <alternative answer>.” Alternative Knowledge 1
“Answer the question with the knowledge: feel free to ignore irrelevant or wrong information.”
Alternative Knowledge 2
Alternative Knowledge 3
New Answer 1 New Answer 2 New Answer 3
Proposed Answer
same different
Domain Knowledge 2
Domain Knowledge 3
different
Abstain
Compete
Alternative Answer 2
Alternative Answer 3
Abstain
Question LLM
Proposed Answer
Figure 1: Overview of collaboration-based approaches for LLM abstention: COOPERATE and COMPETE.

heading: 2.2 Training-Based
text: Hidden Layers Previous works demonstrate that probing LLMs’ hidden layer representations might yield insights into their knowledge and factuality (Slobodkin et al., 2023; Azaria and Mitchell, 2023; CH-Wang et al., 2023). We extract the hidden embeddings eq = LLM-hidden(q), then conduct linear probing to predict ℓ ∈ {accept, reject} through p(ℓ | q) ∝ exp(FFN(eq)). We optimize the crossentropy loss on the held-out set H. At inference time, the LLM should abstain if argmaxℓ p(ℓ | q) indicates that the answer should be rejected.
External Verifier Training an external verifier to judge the veracity of LLM-generated text has shown great potential in math reasoning (Cobbe et al., 2021) and general knowledge domains (Kadavath et al., 2022; Cohen et al., 2023). We train an encoder-based language model LM-enc and employ the [CLS] token for classification. Formally, p(ℓ | q) ∝ exp(LM-enc(q; LLM(q))) is optimized on H with the cross-entropy loss and an abstain decision is achieved through argmaxℓ p(ℓ | q) at inference time. Here both the question and the generated answer are employed to train the external verifier, different from the hidden layers approach where only the intermediate encodings of the question are used for linear probing.
Instruction Tuning A promising way to adapt LLMs is through tuning LLMs on instruction triplets in the format of {instruction|input|output} (Ouyang et al., 2022). Here we investigate whether abstaining could be baked into existing LLMs through instruction tuning. We first employ an instruction to encourage LLMs to abstain when necessary. Given the held-out set H, we first evaluate the
correctness of the generated answer a = LLM(q): if it is correct, we use {instruction|q|a} as an instruction tuning instance; otherwise, we replace the incorrect answer a with an abstain message and train on {instruction|q|abstain message}. In this way, the LLM abstains if the abstain message is generated at inference time for a given q.

heading: 2.3 Prompting-Based
text: Self-Reflect Previous studies show that LLMs might have preliminary capabilities of judging and evaluating their own answer (Kadavath et al., 2022). To this end, we prompt the LLM to self-reflect directly after its generated answer with “The above answer is: A. True B. False”. LLMs should abstain when they deem the generated answer as false.
More Information Since the internal knowledge of LLMs is often incomplete and noisy, existing works explore how to selectively request external information to aid LLM generation (Xu et al., 2023; Asai et al., 2023; Feng et al., 2023b). Following Feng et al. (2023b), we append a prompt about whether more information is needed to answer a given question: “Do you need more information to answer this question? (Yes or No)”. The LLM should abstain if more information is requested.
Generate and Match There might be gaps between LLM generation and “understanding” (West et al., 2023), which leads to hallucinations when questions are presented in a multiple-choice format (Fu et al., 2023). Instead of directly choosing from one of the options, we instruct the LLM to first generate an answer without access to potential options. We then prompt the LLM to evaluate whether the generated answer matches any of the options: the LLM should abstain if there’s no match between
the generated answer and options.

heading: 2.4 Consistency-Based
text: None-of-the-Above Previous works have investigated LLMs’ robustness to possibilities of nonof-the-above (nota) in QA tasks (Kadavath et al., 2022; Ding et al., 2023). We propose an abstain mechanism by adding an extra nota option to every question: the LLM should abstain if nota is selected as the answer, indicating its low consistency in answering the question.
Self-Consistency Threshold Self-consistency (Wang et al., 2022) aims to generate multiple chain-of-thought (Wei et al., 2022) reasoning paths and conduct majority voting to determine the answer. We develop an abstain mechanism by proposing plurality index: given a question q along with the k generated reasoning paths and answers {(ai,pi)}i=1,...,k, the plurality index is defined as:
plu(LLM, q, k) = max ai
∑
j=1,...,k
1(aj = ai)
where it indicates how large the plurality is among all k reasoning paths. A threshold τ∗ ∈ [0, 1] is then optimized on H and applied at inference time to abstain when plu(LLM, q, k) < τ∗ · k.

heading: 2.5 Proposed: Multi-LLM Collaboration
text: Existing approaches often adapt an LLM to evaluate/reflect on its own generated texts, while challenges such as confirmation bases and hallucination would result in unreliable self-evaluation (Kadavath et al., 2022; Ji et al., 2023; Xie et al., 2023). In addition, the stronger approaches rely on heldout sets and result in weaker generalization across knowledge domains. We propose to enable robust LLM abstention through multi-LLM collaboration, i.e., employing multiple LLMs that interact with each other to evaluate the veracity of generated texts and make abstain decisions (Figure 1).
Cooperate LLMs often come with varying knowledge coverage with the potential of complementing each other (Yu et al., 2023a; Du et al., 2023; Bansal et al., 2023), while the internal knowledge of one LLM could also be diversified and specialized (Kang et al., 2023; Si et al., 2023). We tap into this knowledge variation in the COOPERATE approach by having different expert LLMs generate feedback on LLM-proposed answers and employing an LLM as the final judge to decide
whether to abstain. Formally, given question q and answer a = LLM(q), we obtain a set of natural language feedback {f1, . . . ,fk} from expert LLMs and employ a judge LLM to summarize and synthesize the feedbacks into an abstain decision LLM(q,a, {f1, . . . ,fk}) → {accept, reject}.
We employ two modes to generate feedback: for self, the same LLM is specialized into experts on domains {d1, . . . , dk} through prompting-based self-specialization: we prompt the LLM to generate a knowledge passage knowledgei about q with a focus on domain di. A domain-specific feedback is then generated by prepending the knowledge passage: f i = LLM(knowledgei, q,a), and prompting the model to respond as a reviewer. For others, we use separate LLMs {LLM1, . . . ,LLMk} to provide feedback f i = LLMi(q,a), aiming to identify the knowledge gaps in each other and complement the abstain process.
Compete LLMs have varying preferences for knowledge conflicts, i.e. when there is a conflict between LLM’s internal parametric knowledge and external information provided in the prompt (Xie et al., 2023; Wang et al., 2023b). We hypothesize that an LLM should abstain when it is impacted by conflicting information generated by other LLMs and does not stick to the original answer. Concretely, for question q and LLM-generated answer a = LLM(q), we prompt other LLMs to generate k alternative answers {a′1, . . . ,a ′ k} along with a knowledge passage knowledgei about a ′ i. We then instruct the LLM to answer q again with conflicting information prepended: ãi = LLM(knowledgei, q). If a = ãi, the LLM sticks to the original answer and should be thus accepted; otherwise, the LLM is swayed by conflicting information generated by other LLMs, betraying its low confidence. This process is repeated for k alternative answers and the LLM should abstain if the answer changes in a majority of cases.

heading: 3 Experiment Settings
text: Models We evaluate LLM abstention baselines and our proposed collaboration-based approaches with three LLMs featuring varying sizes and openness: Mistral-7B (Jiang et al., 2023), LLaMA2-70B (Touvron et al., 2023), and ChatGPT. We posit that a robust abstain mechanism should ideally work for all LLMs, weak and strong. We set the default sampling temperature to 0.1, and employ 0.7 where multiple runs are required.
Tasks and Datasets We evaluate LLM abstention with four datasets spanning diverse knowledge domains and reasoning scenarios: 1) MMLU (Hendrycks et al., 2020) is a multiple-choice dataset for general knowledge QA; 2) Knowledge Crosswords (Ding et al., 2023) is a structured QA dataset that focuses on multi-hop and compositional knowledge reasoning; 3) Hellaswag (Zellers et al., 2019) is a natural language inference dataset that tests commonsense knowledge and reasoning; 4) Propaganda (Piskorski et al., 2023) tasks LLMs with identifying the 23 propaganda tactics in a long news article based on their internal knowledge. The datasets are all in English. We create
splits of held-out sets and test sets for the four datasets with details in Appendix B.1. We employ LLMs to answer these questions with zeroshot prompting and greedy decoding to obtain the ground truth of whether the LLM “knows” the answer and whether it should abstain. We posit that a robust abstain mechanism should work for knowledge tasks across domains and reasoning contexts.
Correct Incorrect

heading: Answered
text: Abstained
A C
B D
Figure 2: Four outcomes of AbstainQA.
Evaluation Metrics We illustrate the four scenarios under AbstainQA in Figure 2 and employ the following evaluation metrics. (1) Reliable Accu-
racy (R-Acc): AA+C indicates to what extent could LLM-generated answers (not abstained) be trusted. i.e., out of all questions answered, how many are correct? (2) Effective Reliability (ER) (Whitehead et al., 2022; Si et al., 2023): A−CA+B+C+D strikes a balance between reliability and coverage. i.e., out of all questions, what proportion more are answered correctly than incorrectly? (3) Abstain Accuracy (A-Acc): A+DA+B+C+D evaluates whether the abstain decisions are correct: LLMs should abstain when it would provide an incorrect answer and vice versa. (4) Abstain F1 (A-F1): harmonic-mean(precision, recall), where precision = DB+D and recall = D C+D , a balanced metric between reliability and answer coverage.

heading: 4 Results
text: We present the performance of abstain mechanisms with three LLMs on four tasks in Table 1.
COOPERATE and COMPETE are state-ofthe-art abstain mechanisms. Our proposed collaboration-based approaches outperform the strongest baseline in 9 out of the 12 settings (3 models and 4 datasets), achieving an average improvement of 10.1% on reliable accuracy scores. We find that COOPERATE works better with stronger models such as ChatGPT: we hypothesize that the complexities of the multi-step feedback generation warrant a stronger base LLM. COMPETE emphasizes reliability and greatly avoids wrong answers evident in the high reliable accuracy scores: we observe that LLMs stick to the original answers only in the most confident scenarios, making COMPETE ideal for abstention when reliability is paramount.
Calibration > Training > Consistency > Prompting On average, the four types of approaches achieved 0.595, 0.576, 0.553, and 0.475 A-Acc scores. While simple prompting-based approaches struggle to elicit self-reflection, our proposed COOPERATE and COMPETE promote collaborative abstention and greatly improve performance. Among all baselines, instruction tuning and selfconsistency threshold stand out as two strong approaches: however, they both rely on a held-out set for training and hyperparameter tuning: we further investigate their generalizability in Section 5.
Abstain capabilities potentially correlate with base LLM utility. Abstain accuracy (A-Acc) is a metric independent of LLMs’ underlying performance on a QA dataset and solely evaluates ab-
stention quality. For A-Acc, Mistral-7B, LLaMA270B, and ChatGPT achieve 0.524, 0.537, and 0.616 A-Acc scores on average across approaches and datasets, which aligns well with their relevant superiority on standard benchmarks. In addition, our proposed collaboration-based approaches also work better with the strongest ChatGPT, while also improving the abstention performance of weaker LLMs over baselines.
Going beyond generic QA sheds light on the limitations of existing approaches. While most baselines could function properly on MMLU, some collapse (“-” in Table 1) in other knowledge domains or reasoning contexts: ASK CALIBRATE, among others, struggle to follow instructions when the prompt context is long in detecting propaganda tactics; INSTRUCTION TUNING collapses and produces 100% abstention due to imbalances in the held-out set in K-Crosswords where problems are much harder and the LLM should mostly abstain; calibration approaches struggle to set a good threshold when the model’s base performance is too low. However, our proposed collaboration-based approaches work well with all four tasks, especially with the strongest ChatGPT language model, indicating its robustness and broad compatibility for knowledge-intensive tasks across domains and reasoning contexts.

heading: 5 Analysis
text: Abstain Absolute Aside from QA datasets where abstain decisions should be made based on the potential correctness of answers, we investigate scenarios where LLMs should 100% abstain. Specifically, we employ AmbigQA (Min et al., 2020), where LLMs should always abstain from ambiguous and underspecified questions. We also curate ElectionQA23, a QA dataset focusing on elections in 2023 around the globe (details in Appendix B.1) and LLMs should abstain due to the temporal mismatch of training data cutoff, i.e. if the LLM has only seen training data before 2023. Other potential abstain absolute scenarios include known unknowns (Amayuelas et al., 2023) and questions with false/underspecified premises (Patidar et al., 2023; Hu et al., 2023). Figure 3 demonstrates that our proposed COMPETE consistently abstains the most in abstain absolute scenarios across three LLMs and two datasets: we observe that the proposed alternative answers in COMPETE often provide contradictory contexts, thus the LLMs suc-
cessfully abstain due to the knowledge conflicts. We further divide the ElectionQA23 dataset into subsets of which continent is the election held in and present model performance in Figure 5. It is illustrated that LLMs tend to abstain less and assume the outcomes of future elections for Africa and Asia, indicating fairness concerns of abstain mechanisms that might underserve marginalized communities and underdeveloped countries.
Abstain and Retrieval Retrieval-augmented language models (Khattab et al., 2022; Shi et al., 2023; Baek et al., 2023; Yu et al., 2023b; Thakur et al., 2023) have become the go-to approach for augmenting LLMs’ knowledge access, while retrieving from an external corpus could also introduce noisy documents and lead to wrong answers (Wang et al., 2023c; Asai et al., 2023). We investigate whether abstain mechanisms could work with retrieval, specifically when questions in the MMLU dataset are presented together with a re-

heading: Method Ret. R-Acc ER A-Acc A-F1

heading: Method MMLU K-Cross. Hellaswag Propaganda
text: trieved document from the WikiSearch API. Table 2 demonstrates that while baselines such as ASK CALIBRATE are negatively impacted by retrieval, our proposed COOPERATE and COMPETE are in general robust towards additional prepended context through retrieval.
Building upon the findings that collaborationbased approaches are robust with retrieved contexts, we propose a 2-step abstain-based framework to selectively activate retrieval and identify retrieval failures. Specifically, abstain decisions are first made based on QA without retrieval. If the LLM abstains, retrieval is then requested and the abstain mechanism is applied to QA with the retrieved context: if the LLM still decides to abstain, it indicates that the retrieval has failed to find relevant/helpful documents. We evaluate this framework on MMLU and Knowledge Crosswords datasets while comparing it with no retrieval and full retrieval settings. Results in Figure 4 demonstrate that this abstainretrieve-abstain pipeline successfully cuts back the error rate by up to 56.6%, highlighting the potential of abstain mechanisms in identifying retrieval failures and enhancing the reliability of retrievalaugmented LLMs.
Abstain in Multi-Hop In multi-hop knowledge reasoning, LLMs’ knowledge is often limited in some, but not all, of the reasoning steps. We investigate whether our proposed abstain strategies could pinpoint the specific steps where LLMs should abstain due to knowledge limitations: we employ the three-hop subset of the K-Crosswords dataset, conduct abstain mechanisms on each of the hops separately, and present results in Figure 6. It is demonstrated that our proposed COOPERATE and COMPETE are better in localizing knowledge limitations, with a final error rate of 67.2% to 81.2% lower than the strongest baselines.
Abstain ECE Aside from a binary decision, we posit that abstain mechanisms could also provide a continuous abstain likelihood based on token probabilities, heuristics, and more3. We calculate the abstain ECE, estimated calibration error (Guo et al., 2017) but with abstain probabilities and abstain accuracy, on the MMLU dataset and present in Table 3. It is demonstrated that the calibra-
3Details in Appendix B.
tion of abstain likelihood generally correlates with the abstain performance, with ASK CALIBRATE and our proposed COOPERATE and COMPETE being the top-performing approaches. This indicates that our proposed collaboration-based approaches could also provide well-calibrated abstain likelihood scores where fine-grained decision-making is required.
Precision and Recall We observe that among the 14 abstain mechanisms, some emphasize answer coverage while others emphasize answer reliability. We illustrate the abstain precision and recall of approaches in Figure 7: It is illustrated that while most approaches have higher precision than recall and thus lean towards answer coverage, COMPETE has a much higher abstain recall and stresses answer reliability. We argue that these strategies could be ensembled to achieve balanced abstain decisions or employed based on whether the application scenario should optimize coverage or reliability.

heading: 6 Related Work
text: Previous works on knowing what language models know have focused on calibration, prompting, or training. Calibration-based approaches attempt to extract confidence scores from models to gauge their uncertainty (Sun et al., 2022; Kuhn et al., 2022; Zhou et al., 2023a; Liu et al., 2023a). Previous works have evaluated (Radford et al., 2019; Liang et al., 2023; Tao et al., 2023; He et al., 2023) and improved the calibration of language models (Desai and Durrett, 2020; Kong et al., 2020; Jagannatha and Yu, 2020; Kamath et al., 2020; Jiang et al., 2021; Mielke et al., 2022; Lin et al., 2022; Tian et al., 2023; Zhou et al., 2023b), while the calibration quality could vary for different downstream tasks (Desai and Durrett, 2020; Wang et al., 2020; Stengel-Eskin and Van Durme, 2023; Kalai and Vempala, 2023). In the setting of AbstainQA where
LLMs decide whether to abstain or not given a question, calibration-based approaches would rely on a held-out set to set a threshold over calibrated confidence scores: such a threshold and reliance on a held-out set could jeopardize the generalization of calibration-based approaches across knowledge domains and reasoning contexts.
Prompting-based approaches leverage the instruction-following abilities of LLMs and employ instructions to induce self-reflection and gauge whether the generated answer should be trusted. Kadavath et al. (2022) investigates whether a “noneof-the-above” option or self-evaluation prompting would induce good estimations of LLMs’ internal factuality. Huang et al. (2023a) follows existing works (Kim et al., 2024; Shinn et al., 2023) to evaluate whether LLMs could self-correct their own reasoning with a three-step prompting strategy. Feng et al. (2023b) prompts LLM to reflect on whether more information is needed for a given question to selectively solicit external knowledge. These approaches, among other prompting-based methods for acknowledging knowledge limitations (Wang et al., 2023a; Si et al., 2023), mostly rely on the hope that LLMs could indeed carry out selfreflection through simple prompting, while their ability on this front might be subject to confounders and spurious correlations (Kadavath et al., 2022).
Training-based approaches focus on empowering LLMs with the ability to acknowledge uncertainty or abstain through training with special data or objectives. Cobbe et al. (2021) proposes to train an external verifier to evaluate the correctness of LLM-generated answers in math reasoning. A series of works also attempt to probe the hidden representations of LLMs for signals of factuality and hallucination (Slobodkin et al., 2023; Azaria and Mitchell, 2023). Contemporary works have gained interest in improving LLMs’ factuality and self-knowledge through instruction tuning and alignment (Zhang et al., 2023; Yang et al., 2023; Sun et al., 2023; Bashlovkina et al., 2023), as well
as promoting abstention from a safety perspective (Huang et al., 2023b; Liu et al., 2023b). While training-based approaches have shown great potential, they would often require substantial computational resources to fine-tune LLMs and might struggle to generalize across knowledge domains and contexts due to their reliance on the patterns and examples in the training subset.
In this work, we propose COOPERATE and COMPETE, two collaboration-based approaches to enable robust LLM abstention through multi-LLM collaboration. These approaches have shown great potential in improving LLM abstention in not only direct QA, but also retrieval-augmented QA and multi-hop reasoning scenarios.

heading: 7 Conclusion
text: We investigate AbstainQA, a setting where LLMs should abstain from answering questions incorrectly. We curate a taxonomy of 11 abstain baselines across four categories and propose COOPERATE and COMPETE, two novel abstain mechanisms that promote mechanistic reflection through multi-LLM collaboration, in cooperation or competition. Extensive experiments on four datasets demonstrate that COOPERATE and COMPETE advances the state-of-the-art in AbstainQA, with the potential to improve retrieval-augmented LLMs, multi-hop knowledge reasoning, and more.

heading: Limitations
text: While we primarily investigated and evaluated LLM abstention from a knowledge perspective, the problem of abstaining is also important from a safety perspective, preventing LLMs from generating harmful content or biased responses (Huang et al., 2023b; Liu et al., 2023b). There are also intersections between the knowledge and safety aspects of abstention, such as abstaining from underspecified questions based on stereotypes of demographic attributes. We plan to expand on this work and pro-
pose unified abstain strategies for knowledge and safety scenarios in the future.
The abstain mechanisms and experiments in this work are exhaustive to the extent that our computational budget permits, while we could not cover every single approach and proposal from existing works. Nevertheless, we will make the evaluation data, code, and framework publicly available so that new abstain approaches, as well as datasets, can be seamlessly added to advance LLM abstention research.
Our proposed collaboration-based approaches require prompting multiple LLMs for feedback and could have more computational overhead, while the exact cost would depend on the inference costs of other LLMs. We provide a summary of inference costs in Table 6 and Appendix A. We posit that the value of multi-LLM collaboration for abstention lies in using smaller models to supervise larger models in judging veracity and preventing hallucinations. When we serve a user-facing LLM with hundreds of billions of parameters, our approach enables the incorporation of a few 7B LLMs for abstention guidance and mitigate hallucinations: having a few extra 7B models does not add greatly to the cost of a hundreds-of-billions LLM, while significantly boosting its factuality and reliability. By sampling multiple feedbacks to make abstain decisions, randomness is also introduced and there might be variation across model inference runs.
In this work, we focus on a conceptual “abstain” functionality and develop robust approaches to identify the knowledge gaps in large language models. However, “abstain” doesn’t necessarily have to be part of the user interface of LLMs: it could be an internal component of an LLM-based system, followed by approaches to find better knowledge sources to fill the knowledge gaps or methods to incorporate the uncertainty to model responses to enhance trustworthiness (Zhou et al., 2024). Future work could further explore LLM pipelines where these abstain approaches are one component towards reliable question answering.
We acknowledge the potential impact of test set pollution in the training of the adopted LLMs. While we do not know the exact training composition of ChatGPT and the other two LLMs (Mistral7B, LLaMA2-70B) are not explicitly trained on test sets, they might still be unintentionally included in the training data. Our results and findings should be thus interpreted with this limitation in mind.

heading: Ethics Statement
text: Figure 5 highlights that LLM abstain decisions could also have fairness implications, as they fail to abstain and assume the outcomes of future elections more frequently for African and Asian countries. We similarly expect varying levels of abstain performance on questions and prompts regarding different demographics, communities, and perspectives, potentially due to LLMs’ internal social biases (Blodgett et al., 2020; Jin et al., 2021; Bender et al., 2021; Shaikh et al., 2023; Feng et al., 2023a). We plan to examine the intersections of LLM social biases and their abstention abilities in future work, especially in critical domains such as hate speech and misinformation where failing to abstain from generating problematic content could have grave consequences for marginalized communities.

heading: Acknowledgements
text: This material is based upon work supported by the National Science Foundation under CAREER Grant No. IIS2142739, NSF Grant No. IIS2203097, and the DARPA CMO under Contract No. HR001120C0124. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agencies. We also gratefully acknowledge support from Alfred P. Sloan Foundation Fellowship.

heading: A Analysis (cont.)
text: Retrieval Failure (cont.) We present more results with the two-step abstention in retrievalaugmented LLMs in Figure 10. Across three LLMs and four datasets, the abstain-retrieveabstain pipeline successfully brings down the incorrect rate and accounts for retrieval failure.
Abstain and Multi-Hop (cont.) We present more results with the multi-hop abstention in Figure 11. it is demonstrated that our proposed COOPERATE and COMPETE are consistently better at pinpointing knowledge limitations in specific reasoning steps across LLMs and datasets. We further investigate the alignment between model abstain decisions and model answer incorrectness: how many hops did the LLM abstain for and how many hops did the LLM actually answer incorrectly. Table 4 indicates that model abstain decisions match model failures in 73.4% of the time, showcasing the effectiveness of multi-LLM collaboration for abstention in multi-hop problems.
Held-Out Sets Harms Generalization In the study we find that most of the baselines would require a held-out set of questions for model tuning, hyperparameter optimization, and more. Specifically, while INSTRUCTION TUNING is one of the strongest baselines in Section 4, its reliance on a held-out set for training might jeopardize its gen-
eralization across knowledge domains and LLMs. To this end, we conduct a generalization study in Figure 8 where there is a transfer across datasets or LLMs between training and testing. It is illustrated that INSTRUCTION TUNING approaches struggle to generalize across knowledge domains and LLMs, resulting in performance drops sometimes as large as 33.8% in abstain accuracy. On the contrary, our proposed collaboration-based approaches do not need a held-out set and will not suffer from these generalization limitations.
Minority Opinion By default, we have k = 3 feedback passages for each final judge LLM to consider. We find that on the MMLU dataset, the final judge follows the majority opinion in generated feedbacks 84.2% of the time. We manually examine 10 examples where the judge employed the minority opinion and find that in 8 of the 10 cases, the minority opinion indeed presents new information and should be thus taken into account, showcasing the benefit of having multiple feedbacks through multi-LLM collaboration.
Scaling Laws We investigate whether LLM abstention with these mechanisms could be improved through scaling, by evaluating three abstain mechanisms with varying sizes of the LLAMA2 LLM. Results in Figure 9 demonstrate that the abstain performance is not steadily increasing with model size, showcasing that empowering LLMs with abstention abilities is not solely solvable through scaling up.
Shift in Knowledge Domains We investigate whether LLM abstention abilities could be impacted by which knowledge domain the given question is about. Specifically, we present the abstain accuracy metric for different knowledge domains in the MMLU dataset in Figure 12. It is demonstrated that the abstention decisions are indeed of varying quality given the knowledge domain, from 25% to 87.5%, potentially informed by the LLM’s underlying knowledge abilities regarding that domain.
Prompt Robustness Since LLMs could be sensitive to minor changes in prompts (Sclar et al., 2023), we experiment with minor paraphrases of the prompts used in COOPERATE and COMPETE and present the results in Table 5. It is demonstrated that the proposed abstain mechanisms are relatively robust to minor changes in prompt phrasing.
LLM Overhead Different abstain mechanisms have different computational overhead and the most expensive part is the number of LLM inference requests for each question. We present a summary in Table 6.
Working Examples We present qualitative examples of COOPERATE (Tables 14 and 15 for self, Tables 16 and 17 for others) and COMPETE (Tables 18, 19, 20, and 21).

heading: B Experiment Details

heading: B.1 Dataset Details
text: We employ six datasets in the experiments, four for the main experiments, and two abstain absolute datasets.
• MMLU (Hendrycks et al., 2020): we randomly downsample the official validation and test splits into 1,000 questions each as the held-out set and test set.
• K-Crosswords (Ding et al., 2023): we follow the official splits to employ the validation set of 1,094 questions as the held-out set and the test set of 1,007 questions.
• Hellaswag (Zellers et al., 2019): we randomly sample the official validation set into 1,000 questions each as the held-out set and test set.
• Propaganda (Piskorski et al., 2023): we create a random split of 231 questions as the held-out set and 200 questions as the test set based on the official validation split. We randomly sample three other propaganda tactics accompanied by the correct tactic to form four-way multi-hop choice questions.
• AmbigQA (Min et al., 2020): we randomly downsample the official validation and test splits into 1,000 questions each as the held-out set and test set.
• ElectionQA23: we first collect the overview paragraphs of the Wikipedia pages of elections held in 20234 and employ ChatGPT to compose 10 questions for each election in a fourway multiple-choice format. We conduct manual
4https://en.wikipedia.org/wiki/List_of_elections_in_2023
evaluation to remove low-quality or ambiguous questions and create a held-out set of 67 and a test set of 200 questions.

heading: B.2 Model Details
text: We employ three large language models to evaluate abstain mechanisms: 1) Mistral-7B, through the MISTRALAI/MISTRAL-7B-INSTRUCTV0.1 checkpoint on Huggingface (Wolf et al., 2019); 2) LLaMA2-70B, through the METALLAMA/LLAMA-2-70B-CHAT-HF checkpoint on Huggingface; 3) ChatGPT, through the GPT-3.5TURBO-INSTRUCT checkpoint on OpenAI API.
B.3 Implementation Details for every single baseline and approach, along with ECE score extraction, prompt for prompting-based and proposed approaches
• Token Probability: The token probability of the answer token(s) is employed as p(a). The abstain likelihood is obtained through 1− p(a).
• Temperature Scaling: We search for an optimal τ value from 0.1 to 10 on the held-out set H. The abstain likelihood is obtained through 1− p(a) after temperature scaling.
• Ask for Calibration: We follow Tian et al. (2023) and employ the prompt in Table 7 to elicit verbalized confidence scores. The abstain likelihood is obtained through 1−p(a) with verbalized confidence scores.
• Hidden Layers: We employ the featureextraction pipeline with HuggingFace (Wolf et al., 2019) to extract eq. A linear layer of size (DIM, 2) is then employed for linear probing where DIM is the dimension of eq. The abstain likelihood is obtained as the ℓ = reject likelihood after softmax.
• External Verifier: We employ RoBERTa (ROBERTA-BASE) (Liu et al., 2019) as LLM−enc for binary classification. The abstain likelihood is obtained as the ℓ = reject likelihood with the [CLS] token.
• Instruction Tuning: We fine-tune ChatGPT (GPT-3.5-TURBO-INSTRUCT) with the instruction tuning triplets. The instruction is “Answer the following question. If you don’t have enough knowledge, abstain by saying ’sorry, I don’t have enough knowledge to answer this question.”’ The
abstain message is “sorry, I don’t have enough knowledge to answer this question.” The abstain likelihood is obtained as the token probability of the answer token, or that of “sorry” if it abstains.
• Self-Reflect: Following Kadavath et al. (2022), we employ the prompt in Table 8. The abstain likelihood is obtained as the token probability of choosing “B”.
• More Information: Following Feng et al. (2023b), we employ the prompt in Table 9. The abstain likelihood is obtained as the token probability of “yes”.
• Generate and Match: We employ the prompt in Table 10. The abstain likelihood is obtained as the probability of “no”, or 1− that of “yes”.
• None-of-the-Above: The abstain likelihood is obtained as the probability of choosing the NOTA option.
• Self-Consistency Threshold: We employ k = 5 chain-of-thought paths by default. The abstain likelihood is obtained as 1− plu(LLM, q, k).
• Cooperate: We employ the prompt in Tables 11 and 12 for the self and others modes. The abstain likelihood is obtained as the token probability of choosing “B”.
• Compete: We employ the prompt in Table 13 and employ three alternative answers by default. The abstain likelihood is obtained as the probability of the new answer if it changes, or 1− that of the original answer if it doesn’t change, then averaged across all alternative answers.

heading: Future Work
text: Figure 5 highlights that LLM abstain decisions could also have fairness implications, as they fail to abstain and assume the outcomes of future elections more frequently for African and Asian countries. We similarly expect varying levels of abstain performance on questions and prompts regarding different demographics, communities, and perspectives, potentially due to LLMs’ internal social biases (Blodgett et al. , 2020; Jin et al. , 2021; Bender et al. , 2021; Shaikh et al. , 2023; Feng et al. , 2023a). We plan to examine the intersections of LLM social biases and their abstention abilities in future work, especially in critical domains such as hate speech and misinformation where failing to abstain from generating problematic content could have grave consequences for marginalized communities