Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation

Authors: Yiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong, Rui Wang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large Language Models, Label-free Self-Evaluation, AI Reliability

Abstract:
LLM self-evaluation relies on the LLM's own ability to estimate response correctness, which can greatly improve its deployment reliability. 
In this research track, we propose the Chain-of-Embedding (CoE) in the latent space to enable LLMs to perform output-free self-evaluation. CoE consists of all progressive hidden states produced during the inference time, which can be treated as the latent thinking path of LLMs. We find that when LLMs respond correctly and incorrectly, their CoE features differ, these discrepancies assist us in estimating LLM response correctness. Experiments in four diverse domains and seven LLMs fully demonstrate the effectiveness of our method. Meanwhile, its label-free design intent without any training and millisecond-level computational cost ensure real-time feedback in large-scale scenarios.
More importantly, we provide interesting insights into LLM response correctness from the perspective of hidden state changes inside LLMs.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730072726918

thread_type:
official_review

<thread_object>
creator: reviewer_in4e

<content>
summary:
The Chain-of-Embedding (CoE) is a novel approach that allows LLMs to self-evaluate correctness by analyzing their hidden states during inference, forming a latent "thinking path." This label-free method reveals distinct patterns for correct and incorrect responses, enabling real-time, output-free accuracy estimation across various domains. CoE’s minimal computational cost and interpretability make it effective for large-scale applications.

strengths:
1. CoE uses internal hidden states rather than output-based confidence, enhancing adaptability to new tasks without training data.

2. With millisecond-level computation, CoE scales effectively, offering quick feedback for large-scale deployments.

3. CoE provides insights into LLM decision-making by tracing hidden states, offering a transparent, human-like assessment of response correctness.

weaknesses:
1. My major concern is the scenario and the necessity for the label-free scenario. If we limited the LLM query into only math question, commonQA questions, then the label-free scenario may not be necessary. A few-shot human labeling can achieve satisfied performance. The key reason for the label-free is the open-world scenario, which the query can be arbitrary from various domains.  It is impossible for us to label across queries with different purposes. Considering the open-world assumption, there are two following-up question as follows:  
1.1. The figure 2 and 3 try to distinguish the discrepancy between the positive and negative samples. When we consider the open-world assumption, there can be more questions with different distributions from a single domain, which may not be able to distinguish. 
1.2. The positive samples from different domains show different trajectories, is it possible for the method to distinguish them given queries from different domains

2. The motivation in the introduction part seems confusing. There is no evidence to indicate the correspondence between the human thinking path and LLM intermediate path. Moreover, there may many knowledge-intensive task may not need the thinking path. 

3. The evaluation part (especially how to identify the correctness) is not so clear to me. I would like to kindly ask the authors provide a more detailed explanation on it. 

4. The path length is related with the number of layer. I think an additional analysis on how the number of layer influence the performance of the proposed method could make the paper more solid.

questions:
1. How does the proposed method handle distributional differences across open-world queries?

2. Can the method reliably distinguish trajectories across diverse domains?

3. What is the evidence for aligning human thinking paths with LLM intermediate paths?

4. How does layer count affect the method's performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730597237624

thread_type:
official_review

<thread_object>
creator: reviewer_gksp

<content>
summary:
The authors propose a metric for label-free LLM self-evaluation, that utilizes the observed discrepancies in progressive hidden states when LLMs generate correct and incorrect responses. Intuited by the cognitive phenomenon in human thinking, the authors measure the CoE discrepancy in the two sets, responding correctly and incorrectly. Based on the obvious discrepancies in Magnitude and Angle, the authors give the two metrics, CoE-R and CoE-C. Finally, the authors achieved SOTA in various datasets and backbone LLMs. Furthermore, there are other analyses in many aspects, e.g., efficiency and multilingual scalability.

strengths:
1. The literature of this paper is comprehensive.

2. The exploration is structured and natural. From intuitive cognitive phenomenon to observation, the authors verify the assumpted discrepancy between correct and incorrect generation. Then, the authors proposed a metric and then demonstrated the effectiveness of the metric. The idea is well-founded.

3. The authors verify their proposed metrics on various backbones and datasets.

4. The authors give a detailed analysis of many aspects.

weaknesses:
1. Missing critical statistic analysis of discrepancies. The foundation of the proposed metrics is based on the existence of the discrepancy between correct and incorrect generation. However, only visualizing this discrepancy is not enough.

2. Why is the targeted task self-evaluation? self-evaluation is a task that seems appealing but is quite ambiguous and questionable; even if the reviewer reads the reference paper cited by authors for self-evaluation, there is no task definition of self-evaluation.

In the authors' presentation, based on the experiment setting and references cited by the authors, self-evaluation is highly associated with uncertainty estimation, but the authors don't mention their targeted method for uncertainty estimation; furthermore, the authors present self-evaluation as a potential method enabling evaluating LLM's responses without labels in the introduction section, which is associated with llm-as-a-judge and self-rewarding. There is a big gap between the two tasks. For label-free self-evaluation, I advise authors to compare their metrics with llm-as-a-judge or self-rewarding methods (label-based self-evaluation) in some instruction-following tasks, even not sota.

3. Missing verifying on some key datasets, for example, triviaQA and TruthfulQA, which have been tested in many previous works cited by authors;

4. Missing critical robustness analysis on adversarial samples, especially uncertainty estimation.

questions:
a. How does this metric work? Is there a threshold to decide the classification using metrics? If so, what is this threshold?

b. In line 104, why does the average embedding at layer l represent the l-th sentence hidden state?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730903859000

thread_type:
official_review

<thread_object>
creator: reviewer_2nbf

<content>
summary:
The authors propose the Chain-of-Embedding (CoE) in the latent space to enable LLMs to perform output-free self-evaluation, and find that when LLMs respond correctly and incorrectly, their CoE features differ, these discrepancies assist us in estimating LLM response correctness.

strengths:
It is quite interesting to discover that the hidden states of LLMs can be utilized to estimate the LLM response correctness without any label, which inspires me a lot.

Moreover, it is also reasonable to treat the progressive hidden states as the latent thinking path of LLMs, leading to the following
assumption: CoE discrepancies may happen when LLMs generate correct and incorrect responses.

Overall, it is quite inspireable paper and give the readers a lot of insights on the thinking system of LLMs

weaknesses:
I donot think this paper have any significant weaknesses

questions:
My only question is,  there are a a lot of interesting findings in this paper, such as
1. Hidden states to estimate the LLM response correctness without any label
2. CoE discrepancies may happen when LLMs generate correct and incorrect responses,
how can these findings be used to improve the reasoning accuracy of LLMs during pre-training or post-training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732423974741

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response and Rebuttal Summary

comment:
We would like to sincerely thank all reviewers for their careful review and constructive suggestions, which helped us greatly improve our paper. Here, we provide a comprehensive overview of the reviewers' feedback and outline our responses accordingly.

---

# Manuscript Strengths

We thank all reviewers for recognizing the following strengths of our manuscript:

* **The CoE method is innovative and quite inspirable:** it provides interesting insights into using LLM hidden states to determine the LLM decision-making correctness and revelations about the LLM thinking system. (Reviewer 2nBf & in4E)

* **The CoE method is highly scalable and efficient to deploy.** (Reviewer in4E)

* **The motivation from intuitive cognitive phenomenon is reasonable**, and the process of making and verifying hypotheses is structured and natural. (Reviewer 2nBf & GkSP)

* **The literature research is comprehensive.** (Reviewer GkSP)

* **The experimental models and datasets are comprehensive.** (Reviewer GkSP)

* **The experimental analysis is detailed and multi-dimensional.** (Reviewer GkSP)


# Concerns and Suggestions

The reviewers also raised some concerns and suggestions, which we have responded to one by one:

* Self-evaluation as the Targeted Topic 

  * *Reasonableness of the selected topic (Reviewer GkSP):* We have given clarifications and literature citations and categorized existing methods to argue for the reasonableness of our selected topic.

  * *Open-world generalizability exploration (Reviewer in4E)*: We have added experimental results and demonstrated open-world generalizability.

* More Experiments

  * *Statistic analysis of discrepancies (Reviewer GkSP):* We have given numerical data for the statistical experiment of Figure 2 and pointed out the misunderstanding.

  * *More dataset evaluation (TriviaQA and TruthfulQA) (Reviewer GkSP)*: We have added experimental results.

  * *Robustness analysis on adversarial samples (Reviewer GkSP)*: We have added experimental results.

  * *Model layer analysis (Reviewer in4E)*: We have added experimental results, and pointed out the equivalent implications of Table 1 results for this conclusion.


# Detailed Questions

Regarding some details of our manuscript, some reviewers raised questions and we have explained accordingly:

* *How to obtain the classification threshold? (Reviewer GkSP)*: We have given the calculations as well as further results.

* *Why use the average embedding at layer l? (Reviewer GkSP)*: We have given the explanations with the corresponding literature citation.

* *How to identify the correctness in the evaluation part? (Reviewer in4E)*: We have given examples and pointed out the elaboration of in the corresponding positions of our manuscript (Appendix C.2.3)

# Open Discussion

We also received a discussion of the extended application of our paper from reviewer 2nBf. We have given three possible application scenarios, including the basic data optimization (sampling decoding and preference optimization) as well as the advanced model optimization (improving the training objective) scenarios, and give simple experimental results for the last scenario to validate the future value of our CoE method.

---

**Notably, during the rebuttal phase, we successfully addressed all the reviewers' concerns and did not receive any additional issues.** Finally, We sincerely thank all reviewers and AC for their efforts. We've learned a lot from all the reviews.

Best regards,

Submission255 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary: 
The paper proposed the Chain-of-Embedding (CoE), an output-free self-evaluation by large language models (LLMs). The key idea is to analyze LLMs' hidden states during inference across layers, forming a latent "thinking path." CoE investigates the discrepancies in progressive hidden states when LLMs generate correct and incorrect responses to estimate response correctness effectively. This label-free method produces efficient accurate estimation across diverse domains. The efficiency and interpretability of CoE make it suitable for large-scale applications, achieving state-of-the-art results across various datasets and backbone LLMs while offering valuable insights into LLM response correctness.

## Strengths:
1. CoE leverages the internal hidden states' difference between layers for self-evaluation, which provides novel insights into the thinking system of LLMs.
1. It does not require training or ground truths. The computation of the two CoE metrics is scalable for quick feedback in large-scale deployments.
1. CoE offers insights into LLM decision-making by tracing hidden states, providing a transparent and human-like assessment of response correctness.
1. The authors thoroughly verify their proposed metrics across various backbones and datasets, providing detailed analyses in many aspects.

## Weaknesses:
1. Adding discussions about how the discovery can potentially improve the reasoning capability of LLMs can strengthen the paper's contribution and better motivate the study. During the discussion, the authors incorporated the metrics into the training objective and achieved positive results. It would be great to include more complete experiments in the next version. 
1. More clarification on the definition of self-evaluation and the motivation of human thinking analogy is needed.
1. Experiments on more diverse tasks, the robustness of the patterns under perturbations, and metrics to determine the threshold were not provided in the original submission. The authors provided comprehensive experiments in the discussion, which well addressed the concerns. These results can make the conclusions more solid and need to be included in the next version. 

## Decision:
The authors provided further clarifications and additional experimental results in the rebuttal, as requested by the reviewers. All three reviewers participated in the discussion and confirmed how well their concerns have been addressed by the authors. Two reviewers raised their original ratings and all the reviewers voted for acceptance in the final ratings. The authors did an awesome job in addressing the review comments, especially in providing comprehensive experimental results as solid evidence for their claims. Most reviewers, including the meta-reviewer, agree that the observed CoE patterns and the proposed metrics are interesting and inspirational to the community. To better share this result with the community, the meta-reviewer hereby recommends this paper for acceptance.