A Bayesian Approach Towards Crowdsourcing the Truths from LLMs

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-05 16:49:43

Abstract: Concerns persist over the trustworthiness of large language models (LLMs) due to the generation of plausible but incorrect information, known as hallucination. Existing approaches focus on identifying false answers or improving correctness by sampling responses from a single LLM. However, querying multiple LLMs, which exhibit complementary strengths, remains largely unexplored. In this work, we propose a Bayesian crowdsourcing approach to aggregate multiple answers from multiple LLMs and quantify their uncertainty. Extending the Dawid-Skene model, we treat LLMs as annotators, using their answer probabilities as noisy observations of truthfulness and modeling semantic relations between answers in the covariance structure, and jointly learn about LLM's reliability and calibration as parameters. Validated across three open-domain question answering dataset, results show that our approach outperforms existing statistical or agentic methods in abstaining from false answers and identifying truthful ones, offering a robust, scalable solution for uncertainty quantification and truth discovery in LLM outputs.

Main paper: 
 related work: 
 Crowdsourcing. Combining multiple answers by simple majority voting is commonly used to improve LLMs at test time [41, 21] . Crowdsourcing models such as Dawid-Skene [3] opt for iterative weighted majority voting by assuming different reliability of workers. They are comprehensively reviewed by [44, 30, 29] and implemented by [38] . Typically, these models require a known and fixed set of possible answers, while recently studies [20] operate with free-text by a weighted majority voting in the embedding space. Our approach eliminates the need for embedding and provides more flexibility and interpretability. Weak supervision [34] , where the truths are not fully available and truth inference is integrated with model learning, is closely related. We only consider the case when the ground truth is completely unavailable, and do not consider learning. Second-order information such as worker's predictions [18] is useful but beyond the scope of this work.Uncertainty quantification. Many UQ methods work by quantifying how consistent the answers are, with [19, 23] providing methods for a semantic-aware consistency measure. Picking the most consistent answer is also the underlying strategy to improve the factuality of LLMs [41, 21] . [11] infers about the uncertainty of a statement using a Hidden Markov Model by considering the logical relations with other statements conditionally generated from the initial statement. Our model offers more relaxations and consider arbitrary answers sampled independently from multiple LLMs. An separate step in UQ is to calibrate the uncertainty estimates to match accuracy using labeled dataset [9] , while our model allows the learning of the calibration and reliability parameters without labelling. 
methodology: 
 Suppose we have a set of N questions and J LLMs. For each question q (i) , we sample K answers from each LLM, resulting in J × K answers {a 11 , . . . , a JK }, and record their probabilities of being generated y (i) = {y 11 , . . . , y JK }, which are provided as logprob or can be estimated by sampling black-box models, and are known as p(True) [16] . For simplicity, we omit the superscript (i) as the parameters are shared across questions.We introduce a continuous latent variable z ∼ N (µ, Σ) of dimension (J × K) to model the real truthfulness of the answers. The marginal distribution of z jk has shape z jk ∼ N (µ j , σ j ) and is determined by the reliability of the LLM j. The covariance Σ is determined by the semantic relations between answers. Following [11] , we posit that semantically similar answers would have correlated truthfulness, while contradictory answers would have negating truthfulness. As such, z's correlation matrix R is determined by an NLI model [33] that classifies a pair of answers:E jk,j ′ k ′ = Entails(a jk , a j ′ k ′ ) C jk,j ′ k ′ = Contradicts(a jk , a j ′ k ′ ) R jk,j ′ k ′ = (E jk,j ′ k ′ + E j ′ k ′ ,jk )/2 -(C jk,j ′ k ′ + C j ′ k ′ ,jk )/2which ensures a correlation of 1 for equivalent answers and -1 for mutually exclusive ones. The covariance matrix Σ is then computed as Σ = R ⊙ σσ T and is projected to the nearest positive semi-definite matrix in the Frobenius norm [2] .The data y is assumed to the noisy and uncalibrated observation of the truthfulness z, which is calibrated by logistic regression y jk ∼ Bernoulli(sigmoid(β 0j + β 1j • z jk )) such that β 1 ≥ 0. The parameters µ, σ, and β are unique for each LLM and are shared across questions. An example of the full generative model is shown in Fig. 1 .For the ease of implementation, we use EM algorithm to learn the parameters by maximizing loglikelihood using stochastic gradient descent and sampling from p(z|y) using NUTS [10, 32, 1] . As with most baseline methods to be compared in Section 4, we select the best answer with the lowest uncertainty as the final answer for TD. For our model, the uncertainty of answer a jk is quantified by the lower bound of 99.9% confidence interval of the mean of the posterior distribution p(z jk |y). 
experiments and results: 
 Baselines. We perform a comprehensive comparison of our Bayesian approach against recent statistical and agentic methods for UQ and TD related tasks. In addition, we include oracle baselines that always choose answers from a single LLM or choose a correct answers if one exists.Statistical methods are based on the frequency or likelihood of candidate answers and direct consistency measures, with the most commonplace method being simple majority voting. [16] assumes the uncertainty of an answer to be the complement of the probability of it being generated, denoted as p(True). Semantic entropy [19] AbstainQA [7] provides two agentic methods for UQ. The cooperate method asks LLMs to provide feedback on candidate answers, which is then summarized by a judge for a true/false judgment.The compete method asks LLMs to draft paragraphs supporting alternative answers and reconsider the question, and the uncertainty is estimated by the consistency of final answers. Debate [5] is a multi-round debating framework where LLMs refine their answers based on explanations from otherLLMs. An LLM judge selects the final answer based on the responses from the last round, which does not provide UQ and is only applicable to TD.Tasks and evaluation metrics. Experiments are done on three open-domain question answering datasets pertaining to different levels of aleatoric and epistemic uncertainty [13] : FreebaseQA [15] ,AmbigQA [27] , and IMDB-Torso [36] , with FreebaseQA downsampled to match the size of the other datasets of around 1k. Using 10 different seeds, we sample a total of 40 candidate answers per question from four LLMs: Gemma-2-9B [37] , GPT-3.5 [28] , Llama-3-8B [6] , and Mistral-7B [14] .Given a question and its candidate answers, the goal of UQ is to estimate the uncertainty of the candidate answers and it is evaluated by UQ using the area under the receiver operating characteristic curve (AUROC) that measures how well the uncertainty of the correct answer is separated from the uncertainty of the incorrect answers (Table 1 ). The goal of TD is to select a correct answer (if any) from the candidate answers and it is evaluated by the final question-answering accuracy (Table 2 ).Table 1 : Bayesian network outperforms baseline methods on all datasets in terms of AUROC for uncertainty quantification. Top non-oracle results are highlighted in green in Tables 1 and 2 .FreebaseQA AmbigQA IMDB-Torso Agentic AbstainQA [7]  
discussion: 
 Our Bayesian model consistently performs the best among all baseline methods when ranking candidate answers to align with truthfulness, making it a promising approach for uncertainty quantification.For improving question answering (TD), the Bayesian model outperforms other statistical models on harder datasets. Despite the relative success of multi-agent models in TD, it is noteworthy that the fast growth of context length and computation constrains scaling with the number of agents, which is crucial to performance [21] . We suspect that verbal reasoning and world knowledge are important when dealing with high aleatoric uncertainty as demonstrated in AmbigQA, where LLM excels but statistical method suffers. The relative high gap between the best performing model and the "best answer" oracle suggests that relying solely on "wisdom of the crowd" is not always reliable, and the best answer is not always the most popular one [18] .Although we are exploring methods to rank answers without ground truths, we observe from the oracles that unsurprisingly, having labelled data for benchmarking and calibration might be more beneficial in practical applications. For example, evidence from the oracles and our calibration model (β 1 = 1.3 for GPT-3.5 and 0.5 -0.7 for other LLMs) both show that GPT-3.5 is more reliable and confident than other LLMs, an important clue that current methods have not fully leveraged.In the typical crowdsourcing setting, annotators work on instances of the same task, while in question answering the questions can be heterogeneous and diverse, which our current model fail to consider.The Bayesian framework allows for the incorporation of factors such as question domain or difficulty, which could be added in the future to improve our model's performance. 
: 
 answers, as a base for abstaining; and (2) to aggregate these answers to infer the truthful answer for each question, a problem known as truth discovery (TD) in the data management literature [22, 44] . This task can be seen as a special case of inferring the ground truths from multiple annotators, where the annotators are all LLMs. The base problem, known as crowdsourcing [44] , is well-established with Bayesian models that go beyond simple consistency-based aggregation [3, 42] , leading to applications such as in NLP tasks [29, 30, 35] . Despite precursory works [4] on crowdsourcing with weak systems as annotators, not much has been done to extend these models to LLM-based annotators.Crowdsourcing models are mainly limited to classification tasks with predefined classes, while the typical use cases of LLMs require free-form text answers. Moreover, classical crowdsourcing models expect a single answer from each annotator, while LLMs can generate multiple answers for a single question with different probabilities, which could additionally inform crowdsourcing models.  
 
 Appendix:  


Decision: Reviews are all positive, so I recommend accept.

Reviews: review: title: title: Novel and valuable solution (Bayesian approach) to the field of uncertainty quantification and truth discovery for LLMs, which brings the opportunity in real-word application and to solve the real-world problems.
main_review: main_review: - **Overall**:
The paper introduces a Bayesian crowdsourcing solution that combines multiple responses from multiple LLMs, aiding in uncertainty quantification and truth discovery. The proposed method makes a valuable contribution to addressing the hallucination issue. The experimental results are robust, and the experimental settings are reasonable.

- **Pros**:
    - Novelty: The paper introduces a creative method to address the gap in mitigating LLM hallucination issues. Additionally, this method can be applied to a wide range of use cases, from classification to general QA cases, providing significant value to the industry.
    - Experiment Settings: The paper utilizes three datasets and establishes several baselines to validate the performance of the Bayesian method. The overall experimental settings are solid and persuasive.
    - Readability: The paper is well-organized and highly readable.

- **Suggestions**:
    - Explore the performance of the method across a broader range of tasks. The current focus on QA tasks could be expanded to encompass more diverse tasks.
    - Provide an additional explanation about the agentic method mentioned in the paper.
    - Include more evaluation metrics in the experiments to bolster the evidence provided.
title: title: Using Crowdsourcing in a new domain
main_review: main_review: This paper introduces a promising Bayesian crowdsourcing approach to improve truth discovery from LLM outputs. It provides a solid foundation for addressing the problems of LLM hallucinations.

Pros: 
1. The paper introduces a new method and uses a combination of LLMs outputs using this crowdsourcing approach.
2. The paper illustrates the performance improvements in multiple domains.
3. It examines practical applications of the proposed method in detail and accounts for scalability concerns.

Cons:
1. Might be good for a general audience to understand more about the practical cost involved and what the effect is in real world applications.
2. While the information is presented well, further exploration and expansion of the evaluation datasets might serve the reader well.