HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection

Authors: xuefeng  du
chaowei  xiao
yixuan  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 00:22:20

Abstract: The surge in applications of large language models (LLMs) has prompted concerns about the generation of misleading or fabricated information, known as hallucinations. Therefore, detecting hallucinations has become critical to maintaining trust in LLM-generated content. A primary challenge in learning a truthfulness classifier is the lack of a large amount of labeled truthful and hallucinated data. To address the challenge, we introduce HaloScope, a novel learning framework that leverages the unlabeled LLM generations in the wild for hallucination detection. Such unlabeled data arises freely upon deploying LLMs in the open world, and consists of both truthful and hallucinated information. To harness the unlabeled data, we present an automated scoring function for distinguishing between truthful and untruthful generations within unlabeled mixture data, thereby enabling the training of a binary classifier on top. Importantly, our framework does not require extra data collection and human annotations, offering strong flexibility and practicality for real-world applications. Extensive experiments show that HaloScope can achieve superior hallucination detection performance, outperforming the competitive rivals by a significant margin.

Main paper: 
 introduction: 
 In today's rapidly evolving landscape of machine learning, large language models (LLMs) have emerged as transformative forces shaping various applications [35, 45] . Despite the immense capabilities, they bring forth challenges to the model's reliability upon deployment in the open world. For example, the model can generate information that is seemingly informative but untruthful during interaction with humans, placing critical decision-making at risk [19, 53] . Therefore, a reliable LLM should not only accurately generate texts that are coherent with the prompts but also possess the ability to identify hallucinations. This gives rise to the importance of hallucination detection problem, which determines whether a generation is truthful or not [32, 6, 25] .A primary challenge in learning a truthfulness classifier is the scarcity of labeled datasets containing truthful and hallucinated generations. In practice, generating a reliable ground truth dataset for hallucination detection requires human annotators to assess the authenticity of a large number of generated samples. However, collecting such labeled data can be labor-intensive, especially considering the vast landscape of generative models and the diverse range of content they produce. Moreover, maintaining the quality and consistency of labeled data amidst the evolving capabilities and outputs of generative models requires ongoing annotation efforts and stringent quality control measures. These formidable obstacles underscore the need for exploring unlabeled data for hallucination detection. Motivated by this, we introduce HaloScope, a novel learning framework that leverages unlabeled LLM generations in the wild for hallucination detection. The unlabeled data is easy-to-access and can emerge organically as a result of interactions with users in chat-based applications. Imagine, for example, a language model such as GPT [35] deployed in the wild can produce vast quantities of text continuously in response to user prompts. This data can be freely collectible, yet often contains a mixture of truthful and potentially hallucinated content. Formally, the unlabeled generations can be characterized as a mixed composition of two distributions: P unlabeled = (1 -π)P true + πP hal , where P true and P hal denote the marginal distribution of truthful and hallucinated data, and π is the mixing ratio. Harnessing the unlabeled data is non-trivial due to the lack of clear membership (truthful or hallucinated) for samples in mixture data.Central to our framework is the design of an automated membership estimation score for distinguishing between truthful and untruthful generations within unlabeled data, thereby enabling the training of a binary truthfulness classifier on top. Our key idea is to utilize the language model's latent representations, which can capture information related to truthfulness. Specifically, HaloScope identifies a subspace in the activation space associated with hallucinated statements, and considers a point to be potentially hallucinated if its representation aligns strongly with the components of the subspace (see Figure 2 ). This idea can be operationalized by performing factorization on LLM embeddings, where the top singular vectors form the latent subspace for membership estimation. Specifically, the membership estimation score measures the norm of the embedding projected onto the top singular vectors, which exhibits different magnitudes for the two types of data. Our estimation score offers a straightforward mathematical interpretation and is easily implementable in practical applications.Extensive experimental results on contemporary LLMs confirm that HaloScope can effectively improve hallucination detection performance across diverse datasets spanning open-book and closedbook conversational QA tasks (Section 4). Compared to the state-of-the-art methods, we substantially improve the hallucination detection accuracy by 10.69% (AUROC) on a challenging TRUTHFULQA benchmark [29] , which favorably matches the supervised upper bound (78.64 % vs. 81.04%). Furthermore, we delve deeper into understanding the key components of our methodology (Section 4.4), and extend our inquiry to showcase HaloScope versatility in addressing real-world scenarios with practical challenges (Section 4.3). To summarize our key contributions:• Our proposed framework HaloScope formalizes the hallucination detection problem by harnessing the unlabeled LLM generations in the wild. This formulation offers strong practicality and flexibility for real-world applications. • We present a scoring function based on the hallucination subspace from the LLM representations, effectively estimating membership for samples within the unlabeled data. • We conduct in-depth ablations to understand the efficacy of various design choices in Halo-Scope, and verify its scalability to large LLMs and different datasets. These results provide a systematic and comprehensive understanding of leveraging the unlabeled data for hallucination detection, shedding light on future research. 
problem setup: 
 Formally, we describe the LLM generation and the problem of hallucination detection. Definition 2.1 (LLM generation). We consider an L-layer causal LLM, which takes a sequence of n tokens x prompt = {x 1 , ..., x n }, and generates an output x = {x n+1 , ..., x n+m } in an autoregressive manner. Each output token x i , i ∈ [n + 1, ..., n + m] is sampled from a distribution over the model vocabulary V, conditioned on the prefix {x 1 , ..., x i-1 }:EQUATIONand the probability P is calculated as:EQUATIONwhere f L (x) ∈ R d denotes the representation at the L-th layer of LLM for token x, and w o , b o are the weight and bias parameters at the final output layer.Definition 2.2 (Hallucination detection). We denote P true as the joint distribution over the truthful input and generation pairs, which is referred to as truthful distribution. For any given generated text x and its corresponding input prompt x prompt where (x prompt , x) ∈ X , the goal of hallucination detection is to learn a binary predictor G : X → {0, 1} such thatEQUATION3 Proposed Framework: HaloScope 
unlabeled llm generations in the wild: 
 Our key idea is to leverage unlabeled LLM generations in the wild, which emerge organically as a result of interactions with users in chat-based applications. Imagine, for example, a language model such as GPT deployed in the wild can produce vast quantities of text continuously in response to user prompts. This data can be freely collectible, yet often contains a mixture of truthful and potentially hallucinated content. Formally, the unlabeled generations can be characterized by the Huber contamination model [18] as follows: Definition 3.1 (Unlabeled data distribution). We define the unlabeled LLM input and generation pairs to be the following mixture of distributionsEQUATIONwhere π ∈ (0, 1]. Note that the case π = 0 is idealistic since no false information occurs. In practice, π can be a moderately small value when most of the generations remain truthful.An empirical set M = {(x 1 prompt , x 1 ), ..., (x N prompt , x N )} is sampled independently and identically distributed (i.i.d.) from this mixture distribution P unlabeled , where N is the number of samples. x i denotes the response generated with respect to some input prompt x i prompt , with the tilde symbolizing the uncertain nature of the generation.Despite the wide availability of unlabeled generations, harnessing such data is non-trivial due to the lack of clear membership (truthful or hallucinated) for samples in mixture data M. In a nutshell, our framework aims to devise an automated function that estimates the membership for samples within the unlabeled data, thereby enabling the training of a binary classifier on top (as shown in Figure 1 ).In what follows, we describe these two steps in Section 3.2 and Section 3.3 respectively.The first step of our framework involves estimating the membership (truthful vs untruthful) for data instances within a mixture dataset M. The ability to effectively assign membership for these two types of data relies heavily on whether the language model's representations can capture information related to truthfulness. Our idea is that if we could identify a latent subspace associated with hallucinated statements, then we might be able to separate them from the rest. We describe the procedure formally below.Embedding factorization. To realize the idea, we extract embeddings from the language model for samples in the unlabeled mixture M. Specifically, let F ∈ R N ×d denote the matrix of embeddings extracted from the language model for samples in M, where each row represents the embedding vector f ⊤ i of a data sample (x i prompt , x i ). To identify the subspace, we perform singular value decomposition:EQUATIONwhere µ ∈ R d is the average embedding across all N samples, which is used to center the embedding matrix. The columns of U and V are the left and right singular vectors, and form an orthonormal basis. In principle, the factorization can be performed on any layer of the LLM representations, which will be analyzed in Section 4.4. Such a factorization is useful, because it enables discovering the most important spanning direction of the subspace for the set of points in M. Membership estimation via latent subspace. To gain insight, we begin with a special case of the problem where the subspace is 1-dimensional, a line through the origin. Finding the best-fitting line through the origin with respect to a set of points {f i |1 ≤ i ≤ N } means minimizing the sum of the squared distances of the points to the line. Here, distance is measured perpendicular to the line. Geometrically, finding the first singular vector v 1 is also equivalent to maximizing the total distance from the projected embedding (onto the direction of v 1 ) to the origin (sum over all points in M):EQUATION)where ⟨•, •⟩ is a dot product operator. As illustrated in Figure 2 , hallucinated data samples may exhibit anomalous behavior compared to truthful generation, and locate farther away from the center. This reflects the practical scenarios when a small to moderate amount of generations are hallucinated while the majority remain truthful. To assign the membership, we define the estimation score asζ i = ⟨f i , v 1 ⟩ 2 ,which measures the norm of f i projected onto the top singular vector. This allows us to estimate the membership based on the relative magnitude of the score (see the score distribution on practical datasets in Appendix B).Our membership estimation score offers a clear mathematical interpretation and is easily implementable in practical applications. Furthermore, the definition of score can be generalized to leverage a subspace of k orthogonal singular vectors:EQUATIONwhere v j is the j th column of V, and σ j is the corresponding singular value. k is the number of spanning directions in the subspace. The intuition is that hallucinated samples can be captured by a small subspace, allowing them to be distinguished from the truthful samples. We show in Section 4.4 that leveraging subspace with multiple components can capture the truthfulness encoded in LLM activations more effectively than a single direction.Based on the procedure in Section 3.2, we denote H = { x i ∈ M : ζ i > T } as the (potentially noisy) set of hallucinated samples and T = { x i ∈ M : ζ i ≤ T } as the candidate truthful set. We then train a truthfulness classifier g θ that optimizes for the separability between the two sets. In particular, our training objective can be viewed as minimizing the following risk, so that sample x from T is predicted as positive and vice versa.EQUATIONTo make the 0/1 loss tractable, we replace it with the binary sigmoid loss, a smooth approximation of the 0/1 loss. During test time, we leverage the trained classifier for hallucination detection with the truthfulness scoring function of S(x ′ ) = e g θ (x ′ ) 1+e g θ (x ′ ) , where x ′ is the test data. Based on the scoring function, the hallucination detector is G λ (x ′ ) = 1{S(x ′ ) ≥ λ}, where 1 indicates the positive class (truthful) and 0 indicates otherwise. 
experiments: 
 In this section, we present empirical evidence to validate the effectiveness of our method on various hallucination detection tasks. We describe the setup in Section 4.1, followed by the results and comprehensive analysis in Section 4.2-Section 4.4.Datasets and models. We consider four generative question-answering (QA) tasks for evaluation, including two open-book conversational QA datasets COQA [37] and TRUTHFULQA [29] (generation track), closed-book QA dataset TRIVIAQA [20] , and reading comprehension dataset TYDIQA-GP (English) [9] . Specifically, we have 817 and 3,696 QA pairs for TRUTHFULQA and TYDIQA-GP datasets, respectively, and follow [30] to utilize the development split of COQA with 7,983 QA pairs, and the deduplicated validation split of the TRIVIAQA (rc.nocontext subset) with 9,960 QA pairs. We reserve 25% of the available QA pairs for testing and 100 QA pairs for validation, and the remaining questions are used to simulate the unlabeled generations in the wild. By default, the generations are based on greedy sampling, which predicts the most probable token. Additional sampling strategies are studied in Appendix E. We evaluate our method using two families of models: LLaMA-2-chat-7B & 13B [45] and OPT-6.7B & 13B [50] , which are popularly adopted public foundation models with accessible internal representations. Following the convention, we use the pre-trained weights and conduct zero-shot inference in all cases. More dataset and inference details are provided in Appendix A.Baselines. We compare our approach with a comprehensive collection of baselines, categorized as follows: (1) uncertainty-based hallucination detection approaches-Perplexity [38] , Length-Normalized Entropy (LN-entropy) [31] and Semantic Entropy [23] ; (2) consistency-based methods-Lexical Similarity [30] , SelfCKGPT [32] and EigenScore [6] ; (3) prompting-based strategies-Verbalize [28] and Self-evaluation [21] ; and (4) knowledge discovery-based method Contrast-Consistent Search (CCS) [5] . To ensure a fair comparison, we assess all baselines on identical test data, employing the default experimental configurations as outlined in their respective papers. We discuss the implementation details for baselines in Appendix A.Evaluation. Consistent with previous studies [32, 23] , we evaluate the effectiveness of all methods by the area under the receiver operator characteristic curve (AUROC), which measures the performance of a binary classifier under varying thresholds. The generation is deemed truthful when the similarity score between the generation and the ground truth exceeds a given threshold of 0.5. We follow Lin et al. [29] and use the BLUERT [40] to measure the similarity, a learned metric built upon BERT [11] and is augmented with diverse lexical and semantic-level supervision signals. Additionally, we show the results are robust under a different similarity measure ROUGE [27] following Kuhn et al. [23] in Appendix D, which is based on substring matching. Implementation details. Following [23] , we generate the most likely answer by beam search with 5 beams for evaluation, and use multinomial sampling to generate 10 samples per question with a temperature of 0.5 for baselines that require multiple generations. Following literature [6, 2] we prepend the question to the generated answer and use the last-token embedding to identify the subspace and train the truthfulness classifier. The truthfulness classifier g θ is a two-layer MLP with ReLU non-linearity and an intermediate dimension of 1,024. We train g θ for 50 epochs with SGD optimizer, an initial learning rate of 0.05, cosine learning rate decay, batch size of 512, and weight decay of 3e-4. The layer index for representation extraction, the number of singular vectors k, and the filtering threshold T are determined using the separate validation set.As shown in Table 1 , we compare our method HaloScope with competitive hallucination detection methods, where HaloScope outperforms the state-of-the-art method by a large margin in both LLaMA-2-7b-chat and OPT-6.7b models. We observe that HaloScope outperforms uncertaintybased and consistency-based baselines, exhibiting 16.47% and 26.71% improvement over Semantic Entropy and EigenScore on the challenging TRUTHFULQA task. From a computation perspective, uncertainty-based and consistency-based approaches typically require sampling multiple generations per question during testing time, incurring an aggregate time complexity O(Km 2 ) where K is the number of repeated sampling, and m is the number of generated tokens. In contrast, HaloScope does not require sampling multiple generations and thus is significantly more efficient in inference, with a standard complexity O(m 2 ) for transformer-based sequence generation. We also notice that prompting language models to assess the factuality of their generations is not effective because of the overconfidence issue discussed in prior work [54] . Lastly, we compare HaloScope with CCS [5] , which trains a binary truthfulness classifier to satisfy logical consistency properties, such that a statement and its negation have opposite truth values. Different from our framework, CCS does not leverage LLM generations but instead human-written answers, and does not involve a membership estimation process. For a fair comparison, we implemented an improved version CCS*, which trains the binary classifier using the LLM generations (the same as those in HaloScope). The result shows that HaloScope significantly outperforms CCS * , suggesting the advantage of our membership estimation score. Moreover, we find that CCS * performs better than CCS in most cases. This highlights the importance of harnessing LLM generations for hallucination detection, which better captures the distribution of model-generated content than human-written data. HaloScope is a practical framework that may face real-world challenges. In this section, we explore how well HaloScope deals with different data distributions, and its scalability to larger LLMs.Does HaloScope generalize across varying data distributions? We explore whether HaloScope can effectively generalize to different data distributions. This investigation involves directly applying the extracted subspace from one dataset (referred to as the source (s)) and computing the membership assignment score on different datasets (referred to as the target (t)) for truthfulness classifier training. The results depicted in Figure 3 (a) showcase the robust transferability of our approach HaloScope across diverse datasets. Notably, HaloScope achieves a hallucination detection AUROC of 76.26% on TRUTHFULQA when the subspace is extracted from the TRIVIAQA dataset, demonstrating performance close to that obtained directly from TRUTHFULQA (78.64%). This strong transferability underscores the potential of our method to facilitate real-world LLM applications, particularly in scenarios where user prompts may undergo domain shifts. In such contexts, HaloScope remains highly effective in detecting hallucinations, offering flexibility and adaptability.HaloScope scales effectively to larger LLMs. To illustrate effectiveness with larger LLMs, we evaluate our approach on the LLaMA-2-13b-chat and OPT-13b models. The results of our method HaloScope, presented in In this section, we conduct a series of in-depth analyses to understand the various design choices for our algorithm HaloScope. Additional ablation studies are discussed in Appendix C-G.How do different layers impact HaloScope's performance? In Figure 3 (c), we delve into hallucination detection using representations extracted from different layers within the LLM. The AUROC values of truthful/hallucinated classification are evaluated based on the LLaMA-2-7b-chat model. All other configurations are kept the same as our main experimental setting. We observe a notable trend that the hallucination detection performance initially increases from the top to middle layers (e.g., 8-14th layers), followed by a subsequent decline. This trend suggests a gradual capture of contextual information by LLMs in the first few layers, followed by a tendency towards overconfidence in the final layers due to the autoregressive training objective aimed at vocabulary mapping. This observation echoes prior findings that indicate representations at intermediate layers [6, 2] are the most effective for downstream tasks.Where to extract embeddings from multi-head attention? Moving forward, we investigate the multi-head attention (MHA) architecture's effect on representing hallucination. Specifically, the MHA can be conceptually expressed as:EQUATIONwhere f i denotes the output of the i-th transformer block, Attn i (f i ) denotes the output of the selfattention module in the i-th block, and Q i is the weight of the feedforward layer. Consequently, we evaluate the hallucination detection performance utilizing representations from three different locations within the MHA architecture, as delineated in Table 3 .Embedding location TRUTHFULQA TYDIQA-GP TRUTHFULQA TYDIQA-GP Table 3 : Hallucination detection results on different representation locations of multi-head attention. We observe that the LLaMA model tends to encode the hallucination information mostly in the output of the transformer block while the most effective location for OPT models is the output of the feedforward layer, and we implement our hallucination detection algorithm based on this observation for our main results in Section 4.2.Ablation on different design choices of membership score. We systematically explore different design choices for the scoring function (Equation 7) aimed at distinguishing between truthful and untruthful generations within unlabeled data. Specifically, we investigate the following aspects: (1) The impact of the number of subspace components k; (2) The significance of the weight coefficient associated with the singular value σ in the scoring function; and (3) A comparison between score calculation based on the best individual LLM layer versus summing up layer-wise scores. Figure 3 (b) depicts the hallucination detection performance with varying k values (ranging from 1 to 10). Overall, we observe superior performance with a moderate value of k. These findings align with our assumption that hallucinated samples may be represented by a small subspace, suggesting that only a few key directions in the activation space are capable of distinguishing hallucinated samples from truthful ones. Additionally, we present results obtained from LLaMA and OPT models when employing a non-weighted scoring function (σ j = 1 in Equation 7) in Table 4 . We observe that the scoring function weighted by the singular value outperforms the non-weighted version, highlighting the importance of prioritizing top singular vectors over others. Lastly, summing up layer-wise scores results in significantly worse detection performance, which can be explained by the low separability between truthful and hallucinated data in the top and bottom layers of LLMs. Comparison with training on labeled data. We further benchmark our approach against a supervised oracle, where each generation in M is explicitly labeled as truthful or hallucinated similar to [2, 13, 26] , and a binary classifier is trained based on representations of the labeled data. This enables us to assess the performance gap between HaloScope and the upper bound. In Figure 5 , we compare our approach with the supervised oracle on the same test set and report the AUROC metric. The results, based on the LLaMA-2-7b-chat model, demonstrate that our method achieves hallucination detection accuracy on TRUTHFULQA (AUROC: 78.64%) that favorably matches the performance of the supervised oracle (AUROC: 81.04%). This is encouraging considering that our method does not rely on any manual annotation.Qualitative results. We provide qualitative examples of the model's truthfulness score (as introduced in Section 3.3) for different generated texts given the same input prompts during inference (Figure 6 ). The model is LLaMA-2-chat-13b and we utilize questions in TRUTHFULQA that can cause misconception even for humans, for example, Q: Who first started Tesla Motors? We find that HaloScope can accurately produce the scores that align with the truthfulness of the answers (the higher the score, the more truthful the answer is). Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We disclose the experimental details in the main paper Section 4 and Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
related work: 
 Hallucination detection has gained interest recently for ensuring LLMs' safety and reliability [15, 16, 19, 53, 48, 51, 7, 33, 17, 38, 46] . The majority of work performs hallucination detection by devising uncertainty scoring functions, including those based on the logits [31, 23, 14] that assumed hallucinations would be generated by flat token log probabilities, and methods that are based on the output texts, which either measured the consistency of multiple generated texts [32, 1, 34, 47, 10] or prompted LLMs to evaluate the confidence on their generations [21, 47, 39, 28, 43, 54] . Additionally, there is growing interest in exploring the LLM activations to determine whether an LLM generation is true or false [42, 49, 36] . For example, Chen et al. [6] performed eigendecomposition with activations but the decomposition was done on the covariance matrix that required multiple generation steps to measure the consistency. Zou et al. [55] explored probing meaningful direction from neural activations. Our approach is different in three aspects: 1) HaloScope estimates the membership for unlabeled data by identifying the hallucination subspace rather than a single direction in [55] , which can capture the truthfulness encoded in LLM activations more effectively (evidenced in Figure 3 ); 2) HaloScope trains a truthfulness classifier based on membership estimation results, where the explicit training procedure brings more benefits for generalizable hallucination detection compared to direct projection in [55] (Section 4.4); and 3) our paper conducts comprehensive and in-depth evaluation on common benchmarks, thus offering more practical insights than [55] . Another branch of works, such as Li, Duan and Azaria et al. [26, 13, 2] , employed labeled data for extracting truthful directions, which differs from our scope on harnessing unlabeled LLM generations. Note that our studied problem is different from the research on hallucination mitigation [24, 44, 52, 22, 41, 8] , which aims to enhance the truthfulness of LLMs' decoding process. [4, 12, 3] utilized unlabeled data for out-of-distribution detection, where the approach and problem formulation are different from ours.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide details on computer resources in Appendix J.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
conclusion: 
 In this paper, we propose a novel algorithmic framework HaloScope for hallucination detection, which exploits the unlabeled LLM generations arising in the wild. HaloScope first estimates the membership (truthful vs. hallucinated) for samples in the unlabeled mixture data based on an embedding factorization, and then trains a binary truthfulness classifier on top. The empirical result shows that HaloScope establishes superior performance on a comprehensive set of question-answering datasets and different families of LLMs. Our in-depth quantitative and qualitative ablations provide further insights on the efficacy of HaloScope. We hope our work will inspire future research on hallucination detection with unlabeled LLM generations, where a promising future work can be investigating how to train the hallucination classifier in order to generalize well with a distribution shift between the unlabeled data and the test data. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss broader societal impacts in Appendix I.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work proposes a method to help LLMs detect hallucinations. This itself will not pose a risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cite relevant works for the resource we use for the experiments in Section 4.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A Datasets and Implementation Details Input prompts. We provide the detailed textual input as prompts to the language models for different datasets. Specifically, for datasets without context (TRUTHFULQA and TRIVIAQA), the prompt is shown as follows:Answer the question concisely. Q: [question] A:For datasets with context (TYDIQA-GP and COQA), we have the following template for prompts:Answer these questions concisely based on the context: \n Context: [a passage or a paragraph] Q:[question] A:Here are some examples from those datasets with our inference format.• Answer the question concisely. Q: What does it always mean when a cat purrs? A:• Answer the question concisely. Q: What happens if you eat after 8pm? A:• Answer the question concisely. Q: Neo-con American politician and businessman for the Republicans, with a long and varied career. His name is Donald what? A:• Answer the question concisely. Q: What does it always mean when a cat purrs? A:• Answer the question concisely. Q: What happens if you eat after 8pm? A:• Answer the question concisely. Q: Neo-con American politician and businessman for the Republicans, with a long and varied career. His name is Donald what? A:• Answer the question concisely. Q: What does it always mean when a cat purrs? A:• Answer the question concisely. Q: What happens if you eat after 8pm? A:• Answer the question concisely. Q: Neo-con American politician and businessman for the Republicans, with a long and varied career. His name is Donald what? A:• Answer the question concisely. Q: Who was the next British Prime Minister after Arthur Balfour? A:• Answer the question concisely. Q: What is the name of Terence and Shirley Conran's dress designer son? A:• Answer the question concisely. Q: For what novel did J. K. Rowling win the 1999 Whitbread Children's book of the year award? A:• Answer these questions concisely based on the context: \n Context: (Entertainment Weekly) -How are the elements of the charming, traditional romantic comedy "The Proposal" like the checklist of a charming, traditional bride? Let me count the ways ... Ryan Reynolds wonders if marrying his boss, Sandra Bullock, is a good thing in "The Proposal." Something old: The story of a haughty woman and an exasperated man who hate each other -until they realize they love each other -is proudly square, in the tradition of rom-coms from the 1940s and '50s. Or is it straight out of Shakespeare's 1590s? Sandra Bullock is the shrew, Margaret, a pitiless, high-powered New York book editor first seen multitasking in the midst of her aerobic workout (thus you know she needs to get ... loved). Ryan Reynolds is Andrew, her put-upon foil of an executive assistant, a younger man who accepts abuse as a media-industry hazing ritual. And there the two would remain, locked in mutual disdain, except for Margaret's fatal flaw -she's Canadian. (So is "X-Men's" Wolverine; I thought our neighbors to the north were supposed to be nice.) Margaret, with her visa expired, faces deportation and makes the snap executive decision to marry Andrew in a green-card wedding. It's an offer the underling can't refuse if he wants to keep his job. (A sexual-harassment lawsuit would ruin the movie's mood.) OK, he says. But first comes a visit to the groom-to-be's family in Alaska. Amusing complications ensue. Something new: The chemical energy between Bullock and Reynolds is fresh and irresistible. In her mid-40s, Bullock has finessed her dewy America's Sweetheart comedy skills to a mature, pearly texture; she's lovable both as an uptight careerist in a pencil skirt and stilettos, and as a lonely lady in a flapping plaid bathrobe. Q: What movie is the article referring to? A:• Answer these questions concisely based on the context: \n Context: (Entertainment Weekly) -How are the elements of the charming, traditional romantic comedy "The Proposal" like the checklist of a charming, traditional bride? Let me count the ways ... Ryan Reynolds wonders if marrying his boss, Sandra Bullock, is a good thing in "The Proposal." Something old: The story of a haughty woman and an exasperated man who hate each other -until they realize they love each other -is proudly square, in the tradition of rom-coms from the 1940s and '50s. Or is it straight out of Shakespeare's 1590s? Sandra Bullock is the shrew, Margaret, a pitiless, high-powered New York book editor first seen multitasking in the midst of her aerobic workout (thus you know she needs to get ... loved). Ryan Reynolds is Andrew, her put-upon foil of an executive assistant, a younger man who accepts abuse as a media-industry hazing ritual. And there the two would remain, locked in mutual disdain, except for Margaret's fatal flaw -she's Canadian. (So is "X-Men's" Wolverine; I thought our neighbors to the north were supposed to be nice.) Margaret, with her visa expired, faces deportation and makes the snap executive decision to marry Andrew in a green-card wedding. It's an offer the underling can't refuse if he wants to keep his job. (A sexual-harassment lawsuit would ruin the movie's mood.) OK, he says. But first comes a visit to the groom-to-be's family in Alaska. Amusing complications ensue. Something new: The chemical energy between Bullock and Reynolds is fresh and irresistible. In her mid-40s, Bullock has finessed her dewy America's Sweetheart comedy skills to a mature, pearly texture; she's lovable both as an uptight careerist in a pencil skirt and stilettos, and as a lonely lady in a flapping plaid bathrobe. Q: What movie is the article referring to? A:CoQA TYDIQA-GP Implementation details for baselines. For Perplexity method [38] , we follow the implementation here 1 , and calculate the average perplexity score in terms of the generated tokens. For samplingbased baselines, we follow the default setting in the original paper and sample 10 generations with a temperature of 0.5 to estimate the uncertainty score. Specifically, for Lexical Similarity [30] , we use the Rouge-L as the similarity metric, and for SelfCKGPT [32] , we adopt the NLI version as recommended in their codebase2 , which is a fine-tuned DeBERTa-v3-large model to measure the probability of "entailment" or "contradiction" between the most-likely generation and the sampled generations. For promoting-based baselines, we adopt the following prompt for Verbalize [28] on the open-book QA datasets:Q: [question] A:[answer].\n The proposed answer is true with a confidence value (0-100) of , and the prompt ofContext: [Context] Q: [question] A:[answer].\n The proposed answer is true with a confidence value (0-100) of , for datasets with context. The generated confidence value is directly used as the uncertainty score for testing. For the Self-evaluation approach [21] , we follow the original paper and utilize the prompt for the open-book QA task as follows:Question We use the log probability of output token "A" as the uncertainty score for evaluating hallucination detection performance following the original paper. We show in Figure 7 the distribution of the membership estimation score (as defined in Equation 7 of the main paper) for the truthful and hallucinations in the unlabeled LLM generations of TYDIQA-GP. Specifically, we visualize the score calculated using the LLM representations from the 14-th layer of LLaMA-2-chat-7b. The result demonstrates a reasonable separation between the two types of data, and can benefit the downstream training of the truthfulness classifier.In our main paper, the generation is deemed truthful when the BLUERT score between the generation and the ground truth exceeds a given threshold. In this ablation, we show that the results are robust under a different similarity measure Rouge-L, following [23, 6] . Consistent with Section 4.1, the threshold is set to be 0.5. With the same experimental setup, the results on the LLaMA-2-7b-chat model are shown in Table 5 , where the effectiveness of our approach still holds. We verify the performance of our approach using a different random split of the dataset. Consistent with our main experiment, we randomly split 25% of the available QA pairs for testing using a different seed. HaloScope can achieve similar hallucination detection performance to the results in our main Table 1 . For example, on the LLaMA-2-chat-7b model, our method achieves an AUROC of 76.39% and 94.89% on TRUTHFULQA and TYDIQA-GP datasets, respectively (Table 6 ). Meanwhile, HaloScope is able to outperform the baselines as well, which shows the statistical significance of our approach. We evaluate the hallucination detection result when HaloScope identifies the hallucination subspace using LLM generations under different sampling strategies. In particular, our main results are obtained based on beam search, i.e., greedy sampling, which generates the next token based on the maximum likelihood. In addition, we compare with multinomial sampling with a temperature of 0.5. Specifically, we sample one answer for each question and extract their embeddings for subspace identification (Section 3.2), and then keep the truthfulness classifier training the same as in Section 3.3 for test-time hallucinations detection. The comparison in Table 7 shows similar performance between the two sampling strategies, with greedy sampling being slightly better. In this section, we ablate on the effect of the number of unlabeled LLM generations N . Specifically, on TRUTHFULQA, we randomly sample 100-500 generations from the current unlabeled split of the dataset (N =512) with an interval of 100, where the corresponding experimental result on LLaMA-2-chat-7b model is presented in Table 8 . We observe that the hallucination detection performance slightly degrades when N decreases. Given that unlabeled data is easy and cheap to collect in practice, our results suggest that it's more desirable to leverage a sufficiently large sample size. We compare our HaloScope with training the truthfulness classifier by membership estimation with other uncertainty estimation scores. We follow the same setting as HaloScope and select the threshold T and other key hyperparameters using the same validation set. The comparison is shown in Table 9 , where the stronger performance of HaloScope vs. using other uncertainty scores for training can precisely highlight the benefits of our membership estimation approach by the hallucination subspace. The model we use is LLaMA-2-chat-7b. We evaluate our approach on two additional tasks, which are (1) text continuation and (2) text summarization tasks. For text continuation, following [32] , we use LLM-generated articles for a specific concept from the WikiBio dataset. We evaluate under the sentence-level hallucination detection task and split the entire 1,908 sentences in a 3:1 ratio for unlabeled generations and test data. (The other implementation details are the same as in our main Table 1 .)For text summarization, we sample 1,000 entries from the HaluEval [25] dataset (summarization track) and split them in a 3:1 ratio for unlabeled generations and test data. We prompt the LLM with "[document] \n Please summarize the above article concisely. A:" and record the generations while keeping the other implementation details the same as the text continuation task.The comparison on LLaMA-2-7b with three representative baselines is shown below. We found that the advantage of leveraging unlabeled LLM generations for hallucination detection still holds.Text Broader Impact. Large language models (LLMs) have undeniably become a prevalent tool in both academic and industrial settings, and ensuring trust in LLM-generated content for safe usage has emerged as a paramount concern. In this line of thought, our paper offers a novel approach HaloScope to detect LLM hallucinations by leveraging the in-the-wild unlabeled data. Given the simplicity and versatility of our methodology, we expect our work to have a positive impact on the AI safety domain, and envision its potential usage in industry settings. For instance, within the chat-based platforms, the service providers could seamlessly integrate HaloScope to automatically examine the factuality of the LLM generations before information delivery to users. Such applications will enhance the reliability of AI systems in the current foundation model era.Limitations. Our new algorithmic framework aims to detect LLM hallucinations by harnessing the unlabeled LLM generations in the open world, and works by devising a scoring function in the representation subspace for estimating the membership of the unlabeled instances. While HaloScope offers a straightforward solution to leveraging the unlabeled data for training, its effectiveness is still somewhat affected by the drastic distribution shift between the unlabeled data and the test data. Therefore, a distributionally robust algorithm for training the hallucination classifier is a promising future work.We run all experiments with Python 3.8.5 and PyTorch 1.13.1, using NVIDIA RTX A6000 GPUs. Justification: Our paper does not include theoretical results.• The answer NA means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.Justification: We reviewed the NeurIPS Code of Ethics, and confirmed that our work does not deviate from it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://huggingface.co/docs/transformers/en/perplexityhttps://github.com/potsawee/selfcheckgpt 


Decision: This paper receives all positive recommendations. Reviewers agree that the paper is very novel and well-written and has a significant impact in the community. Minor comments are well addressed during the rebuttal. The AC agrees with the positive opinion and recommends acceptance.

Reviews: review: paper_summary: paper_summary: The paper is very clear in the problem it is facing and the solution it proposes is also clearly described. Specifically it is looking at detecting hallucinations produced by generative language models. It does so by taking internal representations of the LLM, projecting these onto SVD factorisation which identifies important directions of variation for the models subspace(s) over some relevant data. 

Membership of unlabelled data (hallucination, or not) is then obtained by projecting the centred representation for the datapoint at hand against the principle singular vector. This provides an unsupervised labelling method. This is used to label all data available, and then a binary classifier is trained on this. This is the final hallucination classifier. 

Experimental results are given over a few Q+A datasets and against several related baselines for hallucination detection. The proposed method is shown to be the most accurate consistently. 

It's actually rather amazing I think that the projection against the primary SVD vector (with scores aggregated across these from different parts of the network) works so well as an unsupervised labeller of truthfullness/hallucination. There's no specific information here at all about the problem of hallucination, and this could have captured the information of any other trait (or even meant nothing at all).
strength_weakness: strength_weakness: * Hallucinations do vary a lot depending on the specific problem at hand, e.g. in question-answering with or without a prompt, or in data-to-text NLG. This paper only looks at question-answer based tasks. It would be interesting to see the method applied on other tasks. There is nothing about the proposed method which limits where it can be applied, it's purely an empirical question as to whether the results are the similar or rather different.

* Access to the internals of proprietary, cloud hosted LLMs is not a given. Often only the predictions are obtainable for these. The method won't be applicable in this case.
questions: questions: * Why is projecting against these subspace primary vectors so informative for hallucination detection? I'm quite surprised at how well this works, given that could capture any other feature of the model/data other than hallucination. The variation of this across the different network layers is significant as shown in Figure 3 c. Based on that, the practical way to apply this would be to measure on every layer before selecting which to use right?

* Would it be possible/sensible/silly to incorporate an objective on certain subspaces into the training of the model (if not just zero-shoting the task of interest with an already capable LLM)?
limitations: limitations: * Access to the internals of the LLM is the main one, as already noted. 
* More empirical results over different types of language generation tasks which each have their own nuances to what is a hallucination would be of interest.
paper_summary: paper_summary: This paper presents a technique for detecting hallucinations by leveraging unlabeled data generation. Instead of relying on human annotation, the method automatically distinguishes between truthful and untruthful generations using network embeddings and their projection onto singular vectors with high singular values. Subsequently, this information is utilized to train a classifier. The study demonstrates enhancements across multiple benchmarks compared to several baseline methods.
strength_weakness: strength_weakness: * The method relies on BLUERT for ground truth evaluation. Can you justify the use of BLUERT score?
questions: questions: * Have you explored the effectiveness of the method on tasks beyond QA, such as summarization?
* Can you provide more details on the result of Table 3? What data is used to perform this ablation? This is important because where to extract the embeddings has huge effect on the effectiveness of the method.
limitations: limitations: Yes
paper_summary: paper_summary: The paper attempts to address the popular problem of hallucination in texts generated by today's LLMs (large language models). Hallucination refers to false or misleading text generated by LLMs. The paper attempts to address hallucination by proposing a truthfulness classifier, HaloScope, that operates on the unlabeled text generated by an LLM during operation. The paper compares Haloscope with existing algorithms and shows its superior performance.

The key idea behind Haloscope is identifying a hallucination subspace in the space of latent (Euclidean) representations of an LLM generation. Thus, when when the latent representation of a generation aligns strongly with this hallucination subspace, the text is classified as potentially hallucinated.
strength_weakness: strength_weakness: One point I would like to note is with regard to the problem setup. Currently, the way problem is formulated (Equations 1, 2, 3, 4), it appears that the fact that an LLM generation is a hallucination is independent of the corresponding user prompt. This does not seem to be a good assumption to make. However, the empirical results do outweigh the weakness of such an assumption. The reviewer suggests the authors to describe their modeling choice (Eq 4) a little more in detail, specially how it relates to the user-prompt conditioning.
questions: questions: Line 40-41. By assuming the specific mixture of two distributions. The authors are assuming that hallucinated data is generated with probability \pi independent of the input. Can authors please elaborate.

Line 158. What is the difference between lambda and capital T?
limitations: limitations: The distribution shift has been identified as a potential barrier to a reliable usage of HaloScope by the authors. The reviewer agrees with the authors and appreciate their candidness. The reviewer would like to suggest mentioning this as potential future work direction in the conclusion.
paper_summary: paper_summary: This paper proposes a new way of harnessing unlabeled LLM generation as the training set for fact verification. The key assumption is "LLM generates factually correct statements more than hallucinogenic statements", which guarantees the existence of clear subspace between hallucination and non-hallucination statements. As the mean of identifying such subspace, singular decomposition is applied to find out orthonormal vector to project LLM's embeddings. The distance from origin is regarded as the value to define "abnormality". The experiments demonstrate that the proposed idea works out and outperforms several existing methods.
strength_weakness: strength_weakness: There are many weaknesses and things to be considered for further improvements.

**Limited Training Set:** The main advantage of the propose method is the usage of unlabeled LLM generation. It means, we can create much larger training set by generating diverse QA pairs (including multi-domain, subjects, etc). But this study is limited to using only the give training set from a few data sets. Also, the classification performance is much lower than what we can obtain by using labeled set in Figure 5. Showing the potential how this framework gets synergies by adding more LLM generations. (we can simply create such large data by using LLMs with simple prompts or we can add LLM generations on other large-scale QA bench not for fact check). Can the fact verification performance improve by adding more LLM generations?

**Issue on Robustness Comparison:** I don't understand why the proposed method gets higher robustness compared with other methods. The propose one still relies on training on a specific data. How could this increase robustness over others? The latent space extracted by a specific dataset cannot be generalized to all domains. I think this is the similarity between the two datasets, TRIVIA QA, TrustfulQA. Please clearly explain what data distribution, domain difference the two data have, and what components help the robustness against distribution shift.

**Explanation on Hyperparameter:** In Lines 150-151, I think the methods need $T$ to split the LLM generations into two different groups of $\mathcal{H}$ and $\mathcal{T}$. But, there is no mention about this, and any other studies on ablating the value of $T$.

**Visualization Study**: A key assumption is the existence of clear subspace to distinguish hallu and non-hallu statements. I think that the author should provide some qualitative analysis using visualization. We can visualize the projected embedding space with oracle labels on 2D space. This clearly show that how such two classes are separable well in latent space.
questions: questions: No Questions.
limitations: limitations: Yes, author stated the limitation on Page 17.

Comments: title: Thank you!
Thank you!
Author response
Thank you!
Response to Authors
comment: Thank you for your appreciation. We're glad we could address your concerns! Please let us know if you have any further questions or suggestions.
I appreciate the authors for addressing my concerns.
Thank you for acknowledging our response and the equations. We're glad to have provided clarity. If you have any additional questions or feedback, please feel free to reach out!
Thank you for writing down the equations. I acknowledge the response.
Thank you for taking the time to read our rebuttal! We are happy to further respond to your question on A1, with revised definitions of hallucination detection and unlabeled data.

---------
We first generate the output $x$ conditioned on the prompt $x_\text{prompt}$. Both the prompt and generation will be used in subsequent hallucination detection, which is defined as follows.

**Definition 2.2 (Hallucination detection)**
We denote $P_\text{true}$ as the joint distribution over the truthful input and generation pairs, which is referred to as truthful distribution. For any given generated text $x$ and its corresponding input prompt $x_\text{prompt}$ where $(x_\text{prompt}, x) \in \mathcal{X}$, the goal of hallucination detection is to learn a binary predictor $G: \mathcal{X} \rightarrow \\{0,1\\}$ such that
\begin{equation}
    G({x_\text{prompt}, x}) = \begin{cases}
        1, &\text{if }  {(x_\text{prompt}, x)} \sim P_\text{true} \\\\
        0,         &otherwise
    \end{cases}
\end{equation}

**Definition 3.1 (Unlabeled data distribution)** We define the unlabeled LLM input and generation pairs to be the following mixture of distributions
\begin{equation}
    P_{\text{unlabeled}} = (1-\pi) P_{\text{true}} + \pi P_{\text{hal}},
\end{equation}
 where $\pi \in (0,1]$. Note that the case $\pi = 0$ is idealistic since no false information occurs. In practice, $\pi$ can be a moderately small value when most of the generations remain truthful.

 **Definition 3.2 (Empirical dataset)** An empirical set $\mathcal{M} = \\{(x_{\text{prompt}}^1, x_1), ..., (x_{\text{prompt}}^N, x_N)\\}$ is sampled independently and identically distributed (i.i.d.) from this mixture distribution $P_{\text{unlabeled}}$, where $N$ is the number of samples. $x_i$ denotes the response generated with respect to some input prompt $x_{\text{prompt}}^i$.

Note that in implementation, we indeed consider passing both the prompt and the LLM-generated answer to obtain the embedding, which aligns with our problem definitions. These updates have been made accordingly in our manuscript. Thank you again for the helpful suggestion!
Thank you for the responses.
A1. Could you write down the new equations that will result so I can get a clear picture?
A2. Acknowledged.
A3. Acknowledged.
Thank you so much for taking the time to read our response and increase your rating! We are glad that our rebuttal addresses your concerns. 

Thanks,
Authors
Thanks for the clear response on my concerns and questions. All the things has been resolved, so I increase my score to 6. Thanks!