CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Tasks

## Abstract

Abstract Large Language Models (LLMs) are revolutionizing various domains, yet verifying their answers remains a significant challenge, especially for intricate open-ended tasks such as consolidation, summarization, and extraction of knowledge.
In this work, we propose CheckEmbed : an accurate , scalable , and simple LLM verification approach. CheckEmbed is driven by a straightforward yet powerful idea: in order to compare LLM solutions to one another or to the ground-truth (GT), compare their corresponding answer-level embeddings obtained with a model such as GPT Text Embedding Large . This reduces a complex textual answer to a single embedding, facilitating straightforward, fast, and meaningful verification.
We develop a comprehensive verification pipeline implementing the CheckEmbed methodology.
The CheckEmbed pipeline also comes with metrics for assessing the truthfulness of the LLM answers, such as embedding heatmaps and their summaries. We show how to use these metrics for deploying practical engines that decide whether an LLM answer is satisfactory or not.
We apply the pipeline to real-world document analysis tasks, including term extraction and document summarization, showcasing significant improvements in accuracy, cost-effectiveness, and runtime performance compared to existing token-, sentence-, and fact-level schemes such as BERTScore or SelfCheckGPT.

## 1 Introduction

Large Language Models (LLMs) [, ] are transforming the world. One particular ongoing challenge in the LLM design is hallucination detection [, , ] and the corresponding overall verification of LLM answers [, ]. Numerous works tried to address this issue, focusing on – for example – grounding knowledge or explainability, and even giving rise to questions regarding methodology and epistemology of artificial intelligence (AI) in general [].

Recent verification methods and their building blocks, such as SelfCheckGPT [] and BERTScore [], focus on individual fact checking and token- as well as sentence-level analysis.
For example, SelfCheckGPT, albeit introduced recently, is an established mechanism for LLM verification. As one of its methods, it uses BERTScore to compare LLM answers. In BERTScore, one first obtains contextual embeddings of all words in two compared sentences, computes cosine similarity scores between all pairs of embeddings from two sentences, and then greedily matches each token from one sentence to the most similar token in the other sentence. Weighted averages of these matchings are used to obtain precision, recall, and the final F1 score that enables comparing different sentences.

However, the problem of verifying LLM answers to more complex tasks, such as open-ended document analyses, still poses a challenge. As an example of such a task, consider extracting legal terms and their definitions from a document. The difficulty of verifying the answers to such a task is due to the inherent lack of structure, even assuming one has the ground-truth answer. Namely, the output of such a request would be a potentially long list of definitions. To verify this answer, one needs to verify that each term is listed and appropriately defined. Harnessing existing methods such as SelfCheckGPT or BERTScore is not scalable, because their token-, sentence-, and fact-based approaches scale poorly with growing task size.
Moreover, we observe that while two different LLM answers can comprise of very different sets of sentences, their meaning could indeed be very similar. This aspect is not well reflected by sentence- and token-level schemes.

In this work, we propose CheckEmbed: an approach for simple, scalable, and accurate verification of LLM solutions to such tasks (contribution 1). The key idea behind CheckEmbed is to obtain and compare embeddings of full LLM answers, or their sizeable chunks, instead of focusing on individual sentences, facts, or tokens.
CheckEmbed relies on the fact that modern embedding models are highly capable; for example, they can be based on powerful Decoder-only LLMs []. Thus, they provide high-dimensional embeddings that can faithfully reflect the meaning of the embedded text.
To motivate this idea and assumption, consider Figure 1. In this figure, we illustrate two very different passages of text that still describe the same concept, and two very similar passages of text that describe two very different concepts. Interestingly, the cosine similarities as proposed in CheckEmbed between the embeddings of two different and two similar passages are – respectively – low and very high, supporting the key idea behind CheckEmbed. Contrarily, BERTScore and SelfCheckGPT give scores that do not match the similarity of the passages.

Figure: Figure 1: We show two sets of two LLM replies each: Replies explaining different concepts using similar wording (left) and ones explaining similar concepts using different wording (right); the queries used to generate these replies can be found in the Appendix. While BERTscore and SelfCheckGPT assess the semantically unrelated replies as more related than the related ones (because these two baselines have been designed to mostly target the verification of individual sentences or facts), CheckEmbed correctly differentiates between semantically related and unrelated replies. We use ChatGPT-4o with temperature = 1.0 for replies and gpt-embedding-large for embeddings.
Refer to caption: x1.png

We design and implement a comprehensive verification pipeline based on CheckEmbed (contribution 2).
The pipeline uses the notion of “stability” of the LLM answer, introduced by SelfCheckGPT, for the verification of the considered tasks. The idea behind “stability” is to prompt an LLM to reply to a given question several times. If the LLM repeatedly outputs the same solution, it means that it has high confidence in its answer and the hallucination risk is low (i.e., high stability of the LLM answers). Contrarily, if there is a large variance in the LLM answers (i.e., low stability of the LLM answers), the risk of hallucinations is high.
In CheckEmbed, we harness this approach for comparing embeddings of whole LLM answers, or their sizeable chunks, pairwise to one another, and to the potential ground-truth (GT), if available. Using such answer-level embeddings enables extracting the meaning of a given whole reply and to compare it effectively to others and to GT.
We show that this strategy is effective and results in embeddings that are close to each other with respect to different distance metrics in cases where the LLM gives correct answers, and with embeddings that are far away, if the LLM is uncertain of the answer or the answer is not of high quality.

As a part of the CheckEmbed pipeline, we offer assessment metrics that illustrate both how each of the LLM answers compares to any other answer and to the potential GT, and succinct summaries. The former is provided in the form of embedding heatmaps. The latter are statistical summaries that can be used as user-specified thresholds to drive decision engines in practical deployments on whether a given LLM answer is good enough and can be accepted, or not and thus has to be re-generated.

We apply our verification pipeline that implements the CheckEmbed idea to several real-world use cases in document analysis, namely extracting terms and definitions as well as summarizing documents (contribution 3).
In addition to the high accuracy, a large advantage of this approach is its speed and simplicity: all one has to do is to embed the LLM answers and compare them to one another using cosine similarity or other vector distance measures.

Our evaluation and analysis illustrate high advantages in accuracy and runtimes (contribution 4).
When the ground-truth is available, CheckEmbed offers closely matching scores for LLM answers. Specifically, we obtain very high scores for high-quality LLM answers and low scores when the LLM answer is a mismatch. This provides an advantage over comparison baselines that often provide mismatching scores.

## 2 The CheckEmbed Design & Pipeline

We now describe the design, the processing and the evaluation of the CheckEmbed pipeline, which is summarized in Figure 2.

Figure: Figure 2: Overview of the CheckEmbed pipeline (left) and comparison between BERTScore, SelfCheckGPT, and CheckEmbed (right).
Refer to caption: x2.png

The CheckEmbed pipeline for verification of LLM’s responses consists of the following key parts.
First, a user sends a question  to the LLM  providing all the essential input data. The pipeline enables batching these questions, i.e., it is possible to send multiple questions in the same pipeline and they pass through each of the next stages individually.
Next, the pipeline prompts the LLM  several times  with the same question ; this number ($k$) is a user parameter. Each reply  has no prior knowledge of the previous answer guaranteeing that there is no bias. $k$ introduces a tradeoff: more responses (higher $k$) means more compute time and cost (more tokens used), but also a better check of correctness. However, as we show in Section 4, CheckEmbed enables high level of confidence in its verification outcome even when $k$ is low.
The next stage of the pipeline is the embedding of the answers . Each reply is embedded, using a pre-specified embedding model (another user input). The potential ground-truth answer  is also embedded.
In the final stage, the embeddings of the replies are compared pairwise .
We use established metrics, most importantly the cosine similarity; we also experiment with Pearson correlation. Other measures are possible as the pipeline enables seamless integration.
The pairwise similarity scores of embeddings are grouped into a (symmetric) heatmap matrix, which is summarized using a selected measure in order to provide a simple threshold number that can be used to drive decision making in practical deployments.

## 3 Scalability Analysis

We provide a brief scalability analysis showing why CheckEmbed is fundamentally faster than BERTScore and SelfCheckGPT.
We denote the number of answers requested from the LLM with $k$.
We assume the same dimensionality of all used embeddings and that computing a score of two embeddings is negligible and takes $O(1)$ time (e.g., Numpy supports highly efficient Pearson correlation and cosine similarity).
Without loss of generality, we also assume that a single reply or the ground-truth contain $s$ sentences, and each sentence contains $t$ tokens.
When comparing the baselines, we consider counts of two most compute intense operations within the pipeline:
the number of embeddings to be constructed and the number of similarity operations to be conducted.

In CheckEmbed, there are $k$ embeddings in total to construct, and $O(k^{2})$ similarity operations to be conducted.

Next, one can apply BERTScore straightforwardly to two passages treated as long sentences, each such passage consists of $st$ tokens. This means $O((st)^{2})=O(s^{2}t^{2})$ embedding comparisons have to be performed for any two passages (for each pair of compared sentences, one compares every pair of individual tokens/words), resulting in a total of $O(k^{2}s^{2}t^{2})$ embedding comparisons as this is done for $O(k^{2})$ pairs of LLM answers, and a total of $O(k^{2})$ embedding constructions.

Finally, SelfCheckGPT assesses a given LLM reply by comparing it to all sample replies collected. To simplify the following derivations, assume that in an individual comparison of two LLM replies, these replies consist of $s_{1}$ and $s_{2}$ sentences, respectively. Now, for each such comparison,
SelfCheckGPT uses BERTScore, where the two input passages $x$ and $y$ to BERTScore consist of $s_{1}s_{2}$ sentences each, i.e., both passage $x$ and passage $y$ contain all the sentences from its corresponding LLM reply, repeated as many times as the number of sentences in the other LLM reply (this is condicted to enable comparing all sentences from each reply pairwise). This gives (using the above BERTScore formulae) $O(ks^{2})$ embedding constructions (there are $k$ LLM replies) and $O(ks^{2}s^{2}t^{2})=O(ks^{4}t^{2})$ embedding comparisons.

## 4 Evaluation

We now show the advantages of CheckEmbed over the state of the art.

Comparison Baselines
We compare CheckEmbed to two key baselines, SelfCheckGPT (a modern verification pipeline that harnesses the stability of LLM replies as its main method) and BERTScore (representing embedding based schemes for assessing similarity of text passages).

Considered Models
We vary the language models used for both generating replies and for creating embeddings.
First, when prompting the LLM to obtain a reply, we explore GPT-3.5, GPT-4, and GPT-4o. Second, when embedding LLM replies, we experiment with different embedding models, namely Salesforce/SFR-Embedding-Mistral (SFR) [], intfloat/e5-mistral-7b-instruct (E5) [, ], Alibaba-NLP/gte-Qwen1.5-7B-instruct (GTE) [], and GPT Text Embedding Large (GPT) [].
For BERTScore and SelfCheckGPT, we use the best possible models available for these baselines (i.e., microsoft/deberta-xlarge-mnli [] and roberta-large []).
We use the default embedding sizes (listed in the Appendix A.2).

Considered Similarity Measures
We consider both cosine similarity and the Pearson correlation score. However, these two follow the same accuracy patterns, and we only show the data for the cosine similarity.

Figure: Figure 3: Analysis of distinguishing similar and different LLM replies (the “Generic” dataset), details explained in Section 4.1. CheckEmbed is highly effective at appropriately recognizing the similarities and differences in the meaning of the verified text passages. This can be seen from little or no overlap between groups of data points corresponding to scores for – respectively – similar and different LLM replies, regardless of the model used. Contrarily, there is a significant overlap between these groups of data points for both BERTScore and SelfCheckGPT, indicating that these baselines perform worse in distinguishing such replies effectively.
Refer to caption: x11.png

Figure: Figure 4: Analysis of distinguishing similar and different LLM replies (the “Precise” dataset), details explained in Section 4.1. CheckEmbed is effective at appropriately recognizing the similarities and differences in the meaning of the verified text passages. This can be seen from moderate to no overlap between groups of data points corresponding to scores for – respectively – similar and different LLM replies, regardless of the model used. Contrarily, there is a large overlap between these groups of data points for both BERTScore and SelfCheckGPT, indicating that these baselines perform worse in distinguishing such replies effectively.
Refer to caption: x12.png

Figure: Figure 5: Analysis of the verification of LLM answers (GPT-3.5), details explained in Section 4.2. We compare to BERTScore; SelfCheckGPT comes with significantly higher runtimes (detailed in Section 4.4) and less competitive scores as it does not focus on open-ended answer-level analysis. The results form a heatmap of the CheckEmbed’s, or BERTScore’s, cosine similarity between all LLM replies, and between each reply and the human expert prepared ground-truth (GT). Rows correspond to two representative legal documents, that come with – respectively – high and low LLM confidence in its replies. Embedding model used in both rows: GPT Text Embedding Large.
Refer to caption: x13.png

Figure: Figure 6: Analysis of the verification of LLM answers (GPT-4), details explained in Section 4.2. We compare to BERTScore; SelfCheckGPT comes with significantly higher runtimes (detailed in Section 4.4) and less competitive scores as it does not focus on open-ended answer-level analysis. The results form a heatmap of the CheckEmbed’s, or BERTScore’s, cosine similarity between all LLM replies, and between each reply and the human expert prepared ground-truth (GT). Rows correspond to two representative legal documents, that come with – respectively – high and low LLM confidence in its replies. Embedding model used in both rows: GPT Text Embedding Large.
Refer to caption: x14.png

### 4.1 Distinguishing Similar and Different Text Passages Faithfully

We start the evaluation by extending the motivating example from Figure 1. Specifically, we analyze whether a given verification method is able to clearly distinguish two passages of text that (1) look similar, but come with very different meanings (“Different replies”, see the left side of Figure 1 for an example), as well as (2) look different, but have similar or identical meanings (“Similar replies”, see the right side of Figure 1 for an example). The used prompts can be found in the Appendix A.1.
The prompt sizes used for these two groups are in the range of 25–250 and 100–200 tokens, respectively.
To broaden the analysis, we further consider two subtypes of such passages: “Generic” and “Precise”. The former are brief while the latter are rich in detailed information (e.g., “Vintage bike” vs. “Old, rusted bicycle leaning against a weathered fence”). We illustrate the results for these two subtypes in Figures 3 and 4, respectively.

Importantly, CheckEmbed comes with no (or very minor) overlap of scores for similar and different replies.
Similar replies come with consistently high similarity scores, while different replies have consistently lower similarity scores. Thus, the key takeaway is that CheckEmbed is highly effective at appropriately recognizing the similarities and differences in the meaning of the considered text passages, regardless of their length and style, and also regardless of the harnessed generative and embedding models. Contrarily, both BERTScore and SelfCheckGPT have high overlaps for these passages; thus, CheckEmbed improves upon the state of the art.

An interesting feature of CheckEmbed is that, while it does distinguish similar and different passages very effectively, it gives relatively high scores to the different passages; these scores are usually higher than the BERTScore or SelfCheckGPT scores for similar passages.
Despite this, it is still straightforward to distinguish between answers implying similar or different passages, because the CheckEmbed scores for similar passages are consistently very high (e.g., with means higher than 0.9 for SFR or E5).

Interestingly, GPT-4-turbo generates replies that are ‘the most difficult to distinguish”, i.e., it comes with visible (still very low) overlap between similar and different ones, across all embedding models. Contrarily, GPT-4o comes with no overlap whatsoever, while GPT-3.5 has very minor overlap.

### 4.2 Verifying LLM Answers Effectively

Next, we illustrate how CheckEmbed enables effective verification of LLM answers. As a use case, we consider extracting terms and their definitions from legal documents; the used data is real and it comes from an in-house legal analytics project.
In this use case, a prompt to the LLM consists of the contents of a legal document (e.g., an NDA), as well as a request to extract respective terms and their definitions. The prompts can also be found in the Appendix A.1. The prompt sizes used in this task are in the range of 25–600 tokens (we split the documents into chunks as whole documents are often very long and come with total token counts that significantly exceed the recommended maximal sizes for the input of the used embedding models). CheckEmbed asks the LLM to generate 10 replies ($k=10$).
We illustrate the results in Figures 5 (for GPT-3.5) and 6 (for GPT-4).
Each figure shows the cosine similarity between all respective LLM replies, and also between each reply and the ground-truth (GT) reply that has been prepared by a human expert.

The results illustrate that whenever CheckEmbed has very high confidence in its answer (top rows in Figures 5 and 6), which is visible by consistently having very high similarities between different replies, it corresponds to very high similarity scores between the LLM replies and the ground-truth. This is the case for all the considered models. Other baselines show mixed results for individual replies, and low similarities between their replies and GT.
It shows that, whenever CheckEmbed has high confidence it the LLM replies, there is high likelihood that these replies are close to the corresponding GT.

In the bottom rows of both figures, we provide example results where CheckEmbed indicates low or mixed LLM’s confidence. While many scores are still relatively high (e.g., 0.85), many are much lower (e.g., 0.5). We manually verified that these particularly low individual scores correspond to LLM replies of very low quality (e.g., only a single term with its definition has been extracted). The low scores overall indicate model’s low confidence, which is further supported by corresponding low similarity scores to GT. Here, BERTScore also has low confidence – overall, its scores are much lower than those of CheckEmbed, but its relative drop in similarity to GT is similarly as low as that of CheckEmbed.

Note that the results in the heatmaps directly correspond to the results from Section 4.1 and Figures 3 and 4 in that very high CheckEmbed scores (e.g., 0.9) indicate high confidence while scores that are lower (e.g., 0.75) – but still higher than BERTScore – consistently mean low LLM’s confidence. This indicates that whenever using such baselines together, one may want to consider rescaling the results accordingly.

A useful simple CheckEmbed measure that indicates the low quality of the LLM answer is a selected summarization measure for a heatmap, for example mean or a matrix norm combined with a standard deviation (std). Whenever the mean is very high (e.g., $>$0.9) and the std is low (e.g., $<$0.05), the answer is of high quality with very high likelihood. Otherwise, one may want to investigate a given situation in more detail.
For example, for GPT-3.5 in the top row (example 0), the LLM is very certain of what the answer is; the mean is 0.93 with very low std of 0.06; BERTScore seems to imply hallucinations with low scores and even more importantly, an std of 0.27.

### 4.3 Detecting Fine-Grained Hallucinations

While CheckEmbed is primarily targeted at verification of open-ended tasks, we also investigate whether CheckEmbed can be used to detect small fine-grained hallucinations, such as mistakes in individual facts.
The results are in Figure 7 and 8 and the used prompts can be found in the Appendix A.1. The task analyzed is summarizing scientific and legal articles. For each article considered, we generate a summary with no errors (labeled as “ground truth”), and we also ask the LLM to summarize these documents, while forcing deliberate small fact-level mistakes, from 0 to 10 mistakes per summary.
CheckEmbed is able to recognize when samples contains no errors, as illustrated by very large scores for GT. Moreover, interestingly, it can also recognize hallucinations after introducing a single error, as visible by no overlap between the GT and the consecutive data points. Finally, we can observe that the amount of low-confidence scores is somewhat increasing with the growing number of introduced errors. However, this increase only starts to be distinctive beyond 5 errors. The trends for BERTScore and SelfCheckGPT are similar, which illustrates that these baselines perform well for their intended use case.

Figure: Figure 7: Analysis of fine-grained hallucination verification of LLM answers (GPT-4o) when summarizing scientific documents, details explained in Section 4.3.
Refer to caption: x15.png

Figure: Figure 8: Analysis of fine-grained hallucination verification of LLM answers (GPT-4o) when summarizing legal documents, details explained in Section 4.3.
Refer to caption: x16.png

### 4.4 Lowering Runtimes

Figure: Figure 9: Comparison of running times of CheckEmbed and other baselines
Refer to caption: x17.png

## 5 Related Work

Trustworthy AI is a broad research area focusing on the transparency, fairness, and reliability of AI systems. Efforts in this field aim to develop frameworks and guidelines that ensure AI systems are trustworthy and align with human values [, ]. Initiatives like differential privacy [], fairness constraints in machine learning models [], and transparent reporting of AI capabilities and limitations [] are prominent in this context. These approaches strive to build AI systems that are not only effective, but also ethically sound and socially acceptable.

Explainable AI (XAI) [] is another critical area of research with the goal of making AI systems more transparent and interpretable to users. Several works have developed methods to enhance explainability in AI systems [, ]. For instance, self-explaining models that generate explanations alongside predictions have been explored to improve user trust and understanding [, ]. Other approaches include post-hoc explanation methods, which provide insights into model decisions after predictions are made, thus facilitating better human-AI interaction [, ]. These advancements are crucial for deploying AI in sensitive areas where understanding the rationale behind decisions is imperative.

The rise of AI has also prompted methodological and epistemological inquiries. Researchers are examining the foundational questions regarding how AI systems generate knowledge and the implications of these processes []. Discussions in this domain focus on the nature of machine learning [], the validity of AI-generated knowledge [], and the ethical considerations surrounding AI deployment [, ]. These inquiries are essential for framing the theoretical underpinnings of AI and addressing concerns related to bias, fairness, and accountability in AI systems.

The problem of hallucinations in LLMs has gathered significant attention [, , , , ]. Chrysostomou et al. [] find that hallucinations are less prevalent in pruned LLM for summarization tasks, which they attribute to an increased dependence on the original source. Various methods on detecting hallucation have been proposed, including SelfCheckGPT [], fact checking [, ] and others [, , ].
Another focus is the reduction of hallucinations. Ever [] dynamically verifies generated content against evidence during the generation process. Zhang et al. [] propose the use of the human user and knowledge bases to align their knowledge to let the LLM answer truthfully. One of the goals of Retrieval Augemented Generation (RAG) [] has been hallucination reduction by fetching relevant information for the LLM context. Benchmark efforts have also been proposed [, , ].

LLM-based agents represent a burgeoning area [], where LLMs are utilized as autonomous agents to perform complex tasks. These agents leverage the generative capabilities of LLMs to interact with users, perform tasks, and make decisions, often resorting to different prompt engineering techniques [, , , , , , ]. Recent studies focus on enhancing the autonomy and effectiveness of these agents by improving their ability to understand and respond to nuanced user inputs []. Techniques such as fine-tuning on specific tasks [] and incorporating external knowledge sources [, ] are employed to enhance the performance of LLM-based agents in real-world applications.

Finally, evaluating LLMs is an ongoing challenge given their complexity and the diverse range of tasks they can perform [, ]. Traditional evaluation metrics often fall short in capturing the full spectrum of LLM capabilities. Hence, researchers are developing new benchmarks and evaluation frameworks that better reflect real-world use cases []. These include task-specific evaluations, user-centric assessments [], and adversarial testing [, ] to ensure that LLMs perform reliably across different scenarios and are resilient to manipulation.

## 6 Conclusion

Large Language Models (LLMs) are revolutionizing various domains, yet effective verification for open-ended tasks remains a significant challenge. Established methods, which focus on token- and sentence-level analysis, fall short in scalability and effectiveness. Addressing this gap is crucial as applications of LLMs expand, necessitating robust mechanisms to ensure the accuracy and reliability of their outputs.

To this end, we introduce CheckEmbed, a scalable approach to LLM verification. CheckEmbed leverages the effectiveness of answer-level embeddings to compare LLM answers with one another and the potential ground-truth. By transforming complex textual answers into individual embeddings using modern decoder-only based models like GPT Text Embedding Large, CheckEmbed makes the verification process simple, accurate, and scalable. This straightforward methodology integrates seamlessly with modern data analytics infrastructure, highlighting its practical applicability and ease of deployment.

CheckEmbed comes with a comprehensive verification pipeline that includes metrics and tools for assessing the veracity of LLM answers, such as heatmaps of similarites between embeddings of answers, the ground-truth, and statistical summaries. These tools provide detailed insights into the quality of LLM outputs and facilitate practical decision-making in real-world deployments. The simplicity of our approach allows for the extension of these metrics to various other applications, further enhancing its utility and flexibility.

Our pipeline has been tested on document analysis tasks, including term extraction. The results demonstrated significant improvements in accuracy and runtime performance compared to existing methods such as BERTScore [] and SelfCheckGPT []. These findings underscore the potential of CheckEmbed to transform LLM verification in industrial settings, ensuring that LLM outputs are both reliable and scalable.

## Acknowledgements

We thank Hussein Harake, Colin McMurtrie, Mark Klein, Angelo Mangili, and the whole CSCS team granting access to the Ault and Daint machines, and for their excellent technical support. We thank Timo Schneider for immense help with infrastructure at SPCL. This project received funding from the European Research Council (Project PSAP, No. 101002047), and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No. 955513 (MAELSTROM). This project was supported by the ETH Future Computing Laboratory (EFCL), financed by a donation from Huawei Technologies. This project received funding from the European Union’s HE research and innovation programme under the grant agreement No. 101070141 (Project GLACIATION). We also acknowledge Polish high-performance computing infrastructure PLGrid (HPC Center: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2024/017103.

## Appendix A Appendix / supplemental material

### A.1 Prompts

**Table 1: Prompt template used for the query generation of the “similar description” use case. A list of “Generic” and “Precise” topics is used to replace ### HERE ### with an actual topic. The aim is to generate two passages of text that look different, but are the same content-wise.**
| ### INSTRUCTION ### |
| --- |
| Hello. Please generate two passages of text. They should both describe the same thing (### HERE ###). However, these two passages should differ VASTLY in their length, style. |
| I want you to give an answer using the following format: |
| <formatting> |
| ### DESCRIPTION 1 ### |
| the actual description here… |
| ### DESCRIPTION 2 ### |
| the actual description here… |
| </formatting> |
| ### ANSWER ### |

**Table 2: Prompt template used for the query generation of the “different description” use case. A list of different topics is used to replace ### HERE 1 ### and ### HERE 2 ### with two actual topics. The aim is to generate two passages of text that seem alike, but are completely different content-wise.**
| ### INSTRUCTION ### |
| --- |
| Hello. Please generate two passages of text. They should describe two different things: |
| 1. ### HERE 1 ### |
| 2. ### HERE 2 ### |
| However, these two passages should have the same length and style. |
| I want you to give an answer using the following format: |
| <formatting> |
| ### DESCRIPTION 1 ### |
| the actual description here… |
| ### DESCRIPTION 2 ### |
| the actual description here… |
| </formatting> |
| ### ANSWER ### |

**Table 3: Prompt template used to “extract respective terms and their definitions” from chunks of legal documentation. Given the complexity of the task, we provide the concrete format as well as an in-context example. [### REPLACE WITH CONTEXT ###] gets replaced by a text chunk from the legal definitions dataset.**
| ### INSTRUCTION ### |
| --- |
| You are a lawyer. |
| ### QUESTION ### |
| Based on the provided context extract all the legal definitions. Answer using the following formatting. |
| <formatting> |
| Term.Definition |
| Term.Definition |
| … |
| </formatting> |
| <example> |
| […] |
| ### CONTEXT ### |
| Preliminary Note |
| The Stock Purchase Agreement sets forth the basic terms of the purchase and sale of the preferred stock to the investors (such as the purchase price, closing date, conditions to closing) and identifies the other financing documents. Generally this agreement does not set forth either (1) the characteristics of the stock being sold (which are defined in the Certificate of Incorporation) or (2) the relationship among the parties after the closing, such as registration rights, rights of first refusal and co-sale and voting arrangements (these matters often implicate persons other than just the Company and the investors in this round of financing and are usually embodied in separate agreements to which those others persons are parties, or in some cases in the Certificate of Incorporation). The main items of negotiation in the Stock Purchase Agreement are therefore the price and number of shares being sold, the representations and warranties that the Company must make to the investors and the closing conditions for the transaction. |
| SERIES A PREFERRED STOCK PURCHASE AGREEMENT |
| THIS SERIES A PREFERRED STOCK PURCHASE AGREEMENT (this “Agreement”), is made as of [], 20[], by and among [______], a Delaware corporation (the “Company”), and the investors listed on Exhibit A attached to this Agreement (each a “Purchaser” and together the “Purchasers”). |
| The parties hereby agree as follows: |
| ### ANSWER ### |
| Agreement. THIS SERIES A PREFERRED STOCK PURCHASE AGREEMENT |
| Company. Delaware corporation |
| Purchaser. Company or the investors listed on Exhibit A |
| Purchasers. Company and the investors listed on Exhibit A together |
| </example> |
| ### CONTEXT ### |
| [###REPLACE WITH CONTEXT###] |
| ### ANSWER ### |

**Table 4: Prompt template used for the ground-truth generation query of the “hallucination test” use case. A list of mostly scientific topic is used to replace ### TOPIC ###.**
| ### INSTRUCTION ### |
| --- |
| Hello. Please generate a passage of text that talks about (### TOPIC ###). |
| Please, use the following format for answering: |
| <formatting> |
| ### PASSAGE ### |
| The passage here…. |
| </formatting> |

**Table 5: Prompt template used for the hallucination generation query of the “hallucination test ” use case. A list of mostly scientific topic is used to replace ### TOPIC ###. ### NUMBER ### is replaced according to an user-specified range of numbers. ### ERRORS ### is used during the hallucination generation process, but is removed from the sample output before the embeddings are created.**
| ### INSTRUCTION ### |
| --- |
| Hello. Please generate ### NUMBER ### completely false information (fact hallucinations) on (### TOPIC ###). |
| Then insert the errors inside a passage of text that talks about (### TOPIC ###). |
| You should convince a reader that the false information are actually correct ones. |
| Please, use the following format for answering: |
| <formatting> |
| ### ERRORS ### |
| List of fact hallucinations to be later included in the passage… |
| ### PASSAGE ### |
| The passage here…. |
| </formatting> |

### A.2 Embedding Length and Parameter Size

**Table 6: Embedding length and number of parameters for each model used during the evaluation.**
| Model Name | Length | #Parameters |
| --- | --- | --- |
| GPT Text Embedding Large | 3072 | not public |
| Salesforce/SFR-Embedding-Mistral | 4096 | 7.11B |
| intfloat/e5-mistral-7b-instruct | 4096 | 7.11B |
| Alibaba-NLP/gte-Qwen1.5-7B-instruct | 4096 | 7.72B |
| microsoft/deberta-xlarge-mnli | 1024 | 750M |
| roberta-large | 1024 | 355M |

### A.3 Compute Resources

Running the pipeline for the dataset of legal definitions for three LLMs (GPT-3.5, GPT-4 and GPT-4o as well as the baselines SelfCheckGPT and BERTScore) on a single NVIDIA Tesla V100-PCIE-32GB GPU took roughly 90 minutes. That dataset was used to create the heatmap figures 5 and 6. The pipeline for the datasets with similar and different descriptions, used for the violin plots, was executed on the same hardware in around 80 minutes.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: See weaknesses
soundness: 1
strengths: The paper addresses a crucial challenge in verification of open-ended LLM tasks

The proposed method is straightforward and practical to implement
summary: This paper addresses LLM hallucination detection and answer verification. Following the assumption of SelfCheckGPT, the authors propose sampling multiple responses from an LLM using the same query, where semantic dissimilarity among responses indicates potential hallucination.
The paper introduces CHECKEMBED, a method for verifying LLM outputs in open-ended tasks. The authors claim three main contributions: 1) utilizing response-level embeddings rather than token or sentence-level comparisons, achieving faster processing than BERTScore and SelfCheckGPT; 2) developing a comprehensive verification pipeline with embedding similarity heatmaps and statistical summaries; 3) validating the approach on document analysis and WikiBio generation tasks. Experimental results suggest improvements in both accuracy and runtime compared to existing methods.
weaknesses: 1. Despite claiming to verify open-ended tasks, the method effectively only works for hallucination detection tasks with definitive answers. Multiple divergent responses could all be valid for truly open-ended queries (e.g., "Tell me a joke"), which the method fails to accommodate.

2. Regarding Contributions 1 and 3, computing cosine similarity between text embeddings is a well-established approach. BERTScore and SelfCheckGPT intentionally utilize token-level and sentence-level information to overcome the limitations of direct embedding similarity. CHECKEMBED's regression to embedding similarity, while claiming superiority over more granular methods, requires validation on public, representative text similarity or QA-Eval tasks. The simple embedding-based approach may overlook complex logical relationships and reasoning errors, particularly in specialized domains (legal, medical), and fails to consider contextual coherence or detect plausible but contradictory content.

3. The legal document analysis experiments are limited (only two cases shown), lacking statistical significance tests to support performance claims. While the WikiBio experiments provide some representativeness, deeper analysis is needed to determine whether performance gains stem from the CHECKEMBED framework or the GTE model, as BERTScore and SelfCheckGPT could potentially be enhanced with GTE.

4. Regarding Contribution 2, given that "stability" was introduced by SelfCheckGPT and merely applied here, the claimed "comprehensive verification pipeline" lacks innovative components.

5. Contribution 4's efficiency comparisons hold limited value without addressing the aforementioned concerns.

6. The paper lacks sufficient discussion distinguishing itself from other embedding-based approaches.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Using histogram to do thresholding: Have you explored any methods for automatically deriving 'useful' confidence thresholds? 

- Sampling methods: What sampling methods did you use/explore when generating LLM responses, and how might these impact your results? This is a pretty crucial detail which could effect your pipeline performance.

- Proprietary Datasets: Can you provide additional details on the proprietary and synthetic datasets used in evaluation? E.g. how many examples were these results computed over? Can you release the synthetic datasets produced?

- LLM-as-Judge Comparison: How does CheckEmbed’s embedding-based verification approach compare to LLM-as-judge methods for answer verification? Do you have comparisons with any existing LLM baselines?

Typos / Presentation Suggestions

- l248: noticely
- ll356 – Consistently
- Figure 5 and Figure 6 – can you add trendlines to these?
- Figure 9/10/11 – can you clearly outline the different parameters at the beginning for better readability? (e.g. [GPT-4 sampling, Stella 1.5B embeddings])
soundness: 3
strengths: The paper was generally well-written and easy to follow. The figures are generally very clear (in particular, Figure 2 was very helpful to quickly visualize the differences between approaches). The motivation seemed fairly clear -- namely, targeting a more efficient and accurate method for comparing generated texts. The technique largely leverages a simple embedding method, however, they explore methods for aggregating this technique through repeated LLM sampling.
summary: The paper introduces CheckEmbed, a framework for verifying LLM responses on open-ended generation tasks. At its core, CheckEmbed uses answer-level embeddings to calculate a single representation of an answer, comparing generated answers with ground-truth via cosine similarity. This core technique is integrated into a pipeline which helps to measure + explain the stability of LLM outputs, and use this to draw takeaways wrt to the correcteness of model outputs. The authors find CheckEmbed outperforms conventional similarity metrics such as BERTScore and SelfCheckGPT on tasks formulated to test whether the metrics can distinguish between 'similar' and 'different' replies (tested over different domains and granularities). They also observe this method (due to its simple answer-level embedding) scales much more efficiently than the two baseline methods.
weaknesses: - Narrow evaluation scope: Much of the evaluation is conducted on synthetic or in-house data, raising questions about the generalizability of the results. The strongest improvements are seen on these datasets, while for the WikiBio task, other metrics—such as the NLI variant of SelfCheckGPT—demonstrate comparable performance. A more extensive evaluation on public datasets would provide stronger evidence for the method’s general applicability.

- Lack of novelty: Generally, this seems like a straightforward application of holistic 'answer-level' embeddings. While the introduced pipeline affords exploration of practical use cases, more work is needed to conclusively evaluate the benefits of this on real-world tasks.

- Reliance on synthetic data: Section 4.1 uses synthetically generated data for the evaluation, which may limit the validity of broad claims drawn from this analysis. For instance, the specific generation prompts used could inherently favor the answer-level embedding technique. Given that this subtask is central to the CheckEmbed approach, additional real-world evaluation would be valuable for supporting the conclusions.

- Lack of sampling details: The paper lacks detail on specific LLM sampling parameters (e.g., temperature, top_k values), which could significantly impact the histogram-based approaches in CheckEmbed. This information seems pretty important, as sampling + aggregation is one of the key contributions, and the variability of LLM responses (due to sampling parameters) could significantly affect the resultant stability/confidence results.

- Insufficient evidence for specific claims: Some of the interesting observations, such as in Section 4.2 (e.g., "Figure 4 shows that whenever CheckEmbed has high confidence in the LLM replies, there is a high likelihood that these replies are close to the corresponding GT"), appear more like case studies than generalized findings. While it would be interesting if CheckEmbed’s inter-sample similarity correlates well with correctness, the two legal examples provided aren't convincing. Additional evidence across the entire legal dataset, for instance, would strengthen this claim.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Regarding Figure 2, why are 1/2 steps included in the Checkembed pipeline? Since Checkembed appears to be an evaluation module, batch inference features should be attributed to inference infrastructures like vLLM or SGLang.
- For lines 157-159, the claim of O(1) complexity requires stronger justification, especially since it seems independent of the embedding matrix size. This appears counterintuitive - could you elaborate on how this is achieved?
- "When calculating cosine similarity and Pearson correlation between k replies of varying lengths:
How are the different lengths handled? For Pearson correlation specifically, is comparing tokens in their natural order meaningful when the data shows similar patterns with only a few token substitutions?"
soundness: 2
strengths: I think comparing embeddings of replies (reply -> embed(reply)) should be much more effective than using GPT to compare two replies directly. If we can maintain performance while increasing preprocessing time, that would be ideal.
summary: This paper proposes a metric termed CheckEmbed which compares two replies based on their reply-level embeddings. The authors claim that their main contributions are the metric's effectiveness and strong performance on both constructed tasks and an existing benchmark, WikiBio. They argue that the effectiveness stems from its $O(1)$ computational complexity, compared to $O(n^2)$ in previous approaches, as NumPy supports efficient algorithms for computing cosine similarity or Pearson correlation. The strong performance is attributed to their use of paragraph-level comparison, in contrast to previous token-level comparison approaches.
weaknesses: - The paper's main claim about effectiveness is questionable. If the performance improvements primarily stem from NumPy's implementation of faster algorithms for computing cosine or correlation scores, then the scientific contribution is overclaimed.
- The claim about paragraph-level comparison versus token-level comparison improving performance is suspicious. Lines 149-150 indicate that the main methods are cosine similarity and Pearson correlation between paragraphs. However, computing cosine similarity between two paragraph embeddings still essentially involves token-level comparison.
- The evaluation appears to be designed specifically to highlight flaws in BERTScore. The test setting using similar-but-different text is not sufficiently rigorous. While Figure 1 shows data where most tokens and patterns are similar with only a few different tokens, BERTScore would calculate many redundant token pairs, potentially overlooking key differences. However, realistic scenarios involve much more complex data patterns than these contrived examples. A more robust evaluation using data with sufficient pattern complexity should be added to prove the method's validity.
- The literature review is incomplete, leading to missing comparisons. In the related work section, 4 out of 5 paragraphs discuss distant areas or general domains while lacking scientific literature on evaluation metrics. The authors justify not comparing MIND, BARTScore, UniEval, or G-Eval by stating these metrics aren't designed for detecting hallucinations. However, this raises the question: was BERTScore specifically designed for this purpose? The boundaries between these metrics' applications are unclear. More discussion about the similarities and differences regarding their targets and detailed methods would be valuable.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 2
questions: See "Weaknesses" section.
soundness: 3
strengths: - CheckEmbed is a novel method, which can more accurately identify and distinguish texts with similar or different semantics.
- CheckEmbed allows for cost-effective evaluation of LLMs on open-ended question answering, with the potential to become a widely used evaluation method.
- The research has a clear focus, the topic is of significant importance, and the work is well-completed.
summary: Accurately evaluating the performance of LLMs in open-ended question answering remains an unresolved challenge. This paper proposes a novel method, CheckEmbed, which leverages pre-trained embedding models to encode both sampled LLM responses and ground truths. The method calculates pairwise scores between these embedding vectors (e.g., cosine similarity). The resulting score matrix can be further processed to compute the Frobenius norm, representing the truthfulness and quality of the LLM responses. Experimental results demonstrate that CheckEmbed outperforms existing methods (such as BertScore and SelfCheckGPT) in both accuracy and efficiency.
weaknesses: - Some issues with the finer details of the paper. For instance, the two heatmaps in the main figure (Figure 2) have incorrect y-axis (or x-axis) labels.
- It seems that CheckEmbed does not outperform baseline methods in certain experiments, such as those in Sections 4.2 and 4.3.
- The paper's structure is somewhat disorganized. The experimental settings are only described textually, and the addition of formulas and examples would improve clarity. Specifically, the examples of similar/different replies and ground truth, the legal documents data point, and the detailed calculation process of Pearson correlation in Section 4.3 are needed.
- Some experimental analysis is lacking. For instance, the compared baselines are not LLM-based methods, and since embedding models are the most critical component of CheckEmbed, should there be an analysis of how different embedding models affect the results? Additionally, it would be useful to compare CheckEmbed against some existing LLM-based methods for evaluating open-ended QA, e.g., llm-as-a-judge.
- Intuitively, BertScore and SelfCheckGPT compare texts at a finer granularity (i.e., token- and sentence-level), while CheckEmbed operates at a passage-level, which is coarser. Finer-grained comparisons usually yield better results, yet the experimental results show otherwise. How can this counterintuitive outcome be explained? (e.g., is it due to differences in embedding models, etc.?)

Meta-review: **Summary:** 

This work highlights the limitations of SelfCheckGPT and BERTScore, which are models designed to evaluate the similarity between two passages. To address these shortcomings, it introduces CheckEmbed, a novel approach that evaluates the embeddings of entire LLM-generated responses or large chunks of text, rather than focusing solely on individual sentences, facts, or tokens. Experimental results demonstrate that CheckEmbed achieves significant improvements in accuracy, cost-effectiveness, and runtime performance compared to BERTScore and SelfCheckGPT.

**Strength:**
* The paper provides a clear motivation for targeting a more efficient and accurate method for comparing generated texts.
* CheckEmbed enables cost-effective evaluation of LLMs on open-ended question-answering tasks, with potential for broad adoption.
* The research addresses an important topic with a clear and focused approach, resulting in well-completed work.

**Weaknesses:**
*  Contributions 1 and 3 largely regress to well-known cosine similarity approaches, which are less innovative compared to token- or sentence-level methods like BERTScore or SelfCheckGPT.
* The approach requires validation on standard text similarity or QA evaluation tasks to substantiate claims of superiority over more granular methods.
* Heavy reliance on synthetic or in-house data limits generalizability. Public dataset evaluations are minimal, and comparable performance by other metrics (e.g., NLI variant of SelfCheckGPT) raises questions about claimed improvements.

Decision: Reject