Challenges with unsupervised LLM knowledge discovery

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 16:50:42

Abstract: We reveal novel pathologies in existing unsupervised methods seeking to discover latent knowledge from large language model (LLM) activations---instead of knowledge they seem to discover whatever feature of the activations is most prominent. These methods search for hypothesised consistency structures of latent knowledge. We first prove theoretically that arbitrary features (not just knowledge) satisfy the consistency structure of a popular unsupervised knowledge-elicitation method: contrast-consistent search. We then present a series of experiments showing settings in which this and other unsupervised methods result in classifiers that do not predict knowledge, but instead predict a different prominent feature. We conclude that existing unsupervised methods for discovering latent knowledge are insufficient, and we contribute sanity checks to apply to evaluating future knowledge elicitation methods. We offer conceptual arguments grounded in identification issues such as distinguishing a model's knowledge from that of a simulated character's that are likely to persist in future unsupervised methods.

Main paper: 
 introduction: 
 Large language models (LLMs) perform well across a variety of tasks [30, 10] in a way that suggests they systematically incorporate information about the world [7] . As a shorthand for the real-world information encoded in the weights of an LLM we could say that the LLM encodes knowledge.Accessing that knowledge is hard, because the factual statements an LLM outputs do not reliably describe it [23, 2, 32] . For example, LLMs might repeat common misconceptions [26] or strategically deceive users [36] . If we could elicit the latent knowledge of an LLM [11] it would allow us to detect and mitigate "dishonesty" [17] . It would also help when supervising outputs that are difficult to understand as well as improving scientific understanding of the inner workings of LLMs. Importantly, this must be done without supervision because we lack a ground truth for what the model "knows", as opposed to what we know.Contrast-consistent search (CCS) [9] is a prominent method proposed to address this problem by assuming that "knowledge" satisfies a consistency structure that few other features in an LLM are likely to satisfy. They use this consistency to construct a classifier which they claim detects a model's latent knowledge, a claim which is widely repeated in the literature (see Appendix B). We refute these claims by identifying classes of LLM features that also satisfy this consistency structure but are not knowledge. We prove two theorems: 1) a class of arbitrary binary classifiers are optimal under the CCS loss; 2) any classifier can be transformed to an arbitrary classifier with the same CCS loss.The upshot is that the CCS consistency structure is more than just slightly imprecise in identifying knowledge-it is compatible with arbitrary patterns.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. Left: We apply two transformations to a dataset of movie reviews, {q i }. First (novel to us) we insert a distracting feature by appending either "Alice thinks it's positive" or "Alice thinks it's negative" at random to each question. Second, we create contrast pairs [9] , (x + i , x - i ), appending "It is positive" or "It is negative" to each. Middle: The LLM activations for these strings are ϕ(x + i ), ϕ(x - i ). Right: A PCA visualisation of the top-3 activation dimensions. Without "Alice ...", a classifier finds the review sentiment (orange/blue). But with "Alice ..." a classifier finds Alice's opinion (light/dark) ignoring review sentiment.We then show that other unsupervised methods in addition to CCS empirically do not discover knowledge, regardless of any inductive biases that might hypothetically be present. Two didactic experiments show that these methods can latch onto artificial distracting features instead of knowledge.Our third experiment moves towards realism by showing that these knowledge-discovery methods can latch onto implicit opinions. The fourth is almost fully natural: we show that the method's results are highly sensitive to reasonable prompt variants which have been used in the literature.We conclude that existing unsupervised knowledge-discovery methods are insufficient in practice, and we propose principles for evaluating knowledge elicitation methods to prevent future "false-positives" in the literature. We hypothesise that our conclusions will generalise to more sophisticated methods, though perhaps not the exact experimental results: using different consistency structures of knowledge will likely suffer from similar issues to what we show here. Our key contributions are as follows:• We prove that arbitrary features satisfy the CCS loss equally well.• We show that unsupervised methods detect prominent features that are not knowledge.• We show that the features discovered by unsupervised methods are sensitive to prompts and that we lack principled reasons to pick any particular prompt.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We provide the proof and series of experiments as described, alongside the sanity checks and conceptual arguments.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Here we display results for the experiments on discovering an explicit opinion using datasets IMDB, BoolQ and DBpedia, and models Chinchilla-70B (Figure 9 ), T5-11B (Figure 10 ) and T5-FLAN-XXL (Figure 11 ). For Chinchilla-70B and T5 we use just a single mention of Alice's view, and for T5-FLAN-XXL we use five, since for a single mention the effect is not strong enough to see the effect, perhaps due to instruction-tuning of T5-FLAN-XXL. The next appendix Appendix D.2.2 ablates the number of mentions of Alice's view. Overall we see a similar pattern in all models and datasets, with unsupervised methods most often finding Alice's view, though for T5-FLAN-XXL the CCS results are more bimodal in the modified prompt case. 
background: 
 Contrastive LLM activations. We focus on methods that train probes [1] using LLM activation data. This data is constructed using contrast pairs [9] . A contrast pair is a pair of strings with opposite 'claim' for some characteristic of interest which can be used to study the contrast in how an LLM represents that characteristic. For example, a contrast pair might be "Are cats mammals? Yes." and "Are cats mammals? No." Potentially, pairs like this could then be used to study how LLMs represent correctly/incorrectly answered questions.Burns et al. [9] show how to generate such contrast pairs from a dataset of binary questions, Q = {q i } N i=1 , such as "Are cats mammals?" by, for example, appending "Yes." and "No." for a positive and negative member of a contrast pair (x + i , x - i ). The LLM's representations of each member of the pair can then be computed by looking at the activations from an intermediate layer after the sequence of tokens, ϕ(x + i ) and ϕ(x - i ). If one just looked at these activations, their differences might be dominated just by the presence of the tokens "Yes." or "No." Burns et al. [9] therefore propose a normalisation step which strips away the average effect of those tokens across the dataset: setting φ(x +/i represent that characteristic. CCS uses the LLM's representations to predict correct labels, intending to study cases where the LLM's knowledge is true. CCS assumes that LLM knowledge representations are credences which follow probabilistic laws. Softly encoding this constraint, they minimiseEQUATIONfor a function from the normalised LLM activations from the contrast pairs: p(x) = σ(θ T φ(x) + b)(a linear function with sigmoid). The motivation is that the L cons encourages negation-consistency (that a statement and its negation should have probabilities that add to one), and L conf encourages confidence to avoid p(x + i ) ≈ p(x - i ) ≈ 0.5. For inference on a question q i the average prediction is p(q i ) = p(x + i ) + (1 -p(x - i )) /2 and then the induced classifier is f p (q i ) = I [p(q i ) > 0.5]. 1 Activation clustering with PCA and k-means. We consider two other unsupervised learning methods. In both cases we cluster the difference in contrastive activations, { φ(x + i ) -φ(x - i )} N i=1 . In one case, these are clustered by applying principal component analysis (PCA) and thresholding the top component at 0 [9] . 2 The other clusters with k-means with two clusters.Logistic regression. As a supervised baseline, we use logistic regression on concatenated contrastive activations, {( φ(x + i ), φ(x - i ))} N i=1 with labels a i , and treat this as a ceiling (since it uses labels).Random baseline. We compare to a random baseline using a probe with random parameter values, treating that as a floor (as it does not learn from input data) [35] . Further details are in Appendix C.3.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations are discussed in the final section while assumptions are discussed in the context of the theorems that depend on them.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theoretical results: 
 Our theoretical results focus on CCS, showing that CCS's consistency structure isn't specific to knowledge. This implies that arguments for CCS's effectiveness cannot be grounded in conceptual or principled motivations from the loss construction. In later sections, we also address other methods which do not rely on these strong consistency assumptions and show that heuristic arguments grounded in inductive biases do not support using any of these as knowledge-discovery methods.As illustration, consider the IMDb sentiment classification task [28] . A given question q i considers whether a movie review has a particular sentiment, s(q i ) := I [q i has positive sentiment], and is converted into a contrast pair of x + i and x - i , each of which has a claim c(•) about the sentiment.Specifically, c(x + i ) = 1, a claim that the sentiment is positive, and c(x - i ) = 0 for negative. The desired probe, p * , detecting the truth feature must check whether the sentiment and the claim agree.This can be done by XOR (denoted ⊕) of the sentiment and the claim:EQUATIONThe induced probe for this feature is the sentiment as desired: f p * (q i ) = s(q i ). Our key insight is that the CCS loss is low just because of this XOR, not the sentiment, and so the same construction can work for arbitrary features of the question: given some feature h, the probe p(x ± i ) = h(q i ) ⊕ c(x ± i )gets low CCS loss and has an induced probe h.Theorem 1. Let feature h : Q → {0, 1}, be any arbitrary map from questions to binary outcomes. Let(x + i , x - i )be the contrast pair corresponding to question q i and let c(x + i ) = 1, c(x + i ) = 0. Then the probe defined as p(x ± i ) = h(q i ) ⊕ c(x ± i )achieves optimal loss, and the averaged prediction satisfiesp(q i ) = h(q i ).That is, the classifier that CCS finds is under-specified: for any binary feature, h, on the questions, there is a probe with optimal CCS loss that induces that feature. The proof comes directly from inserting our constructive probes into the loss definition-equal terms cancel to zero (see Appendix A). 1 Because the predictor learns the contrast between activations, not absolute classes, Burns et al. [9] disambiguate by assuming that fp(qi) = 1 to correspond to label ai = 1 if the accuracy is greater than 0.5 (else it corresponds to ai = 0). We call this further step truth-disambiguation and apply it to all methods similarly. 2 Emmons [16] point out that this is roughly 97-98% as effective as CCS according to the experiments in Burns et al. [9] , suggesting that contrast pairs and standard unsupervised learning are doing much of the work, and CCS's consistency loss may not be important. Our experiments largely agree with this finding-see Appendix D.6 for an additional experiment showing agreement between the predictions of these methods.In Thm. 1, the probe p is binary since h is binary, but in practice probe outputs are produced by a sigmoid and so are in (0, 1). Can we say anything about this setting? We show that it is possible to transform a soft probe for one feature into a soft probe for any other arbitrary feature. In the binary case, the desired probe for feature h 1 is p 1 = h 1 ⊕ c, and the desired probe for h 2 is h 2 ⊕ c. So, we have p 2 = p 1 ⊕ h 1 ⊕ h 2 . To generalize this to soft probes, we extend ⊕ as follows:EQUATIONIn addition, we correct the CCS loss to fix an unmotivated downwards bias in the loss proposed by Burns et al. [9] (see Appendix A.2). We also use this symmetrized loss in our experiments. After this, the transformation between probes works as desired, proving that there is an arbitrary classifier encoded by a probe with identical CCS loss to the original:Theorem 2. Let g : Q → {0, 1}, be any arbitrary map from questions to binary outputs. Let(x + i , x - i )be the contrast pair corresponding to question q i . Let p be a probe, whose average resultp = 0.5 p(x + i ) + (1 -p(x - i )) induces a classifier f p (q i ) = I [p(q i ) > 0.5]. Define the transformed probe p ′ (x ± i ) = p(x ± i ) ⊕ [f p (q i ) ⊕ g(q i )]. Then L CCS (p ′ ) = L CCS (p)and p ′ induces the classifierf p ′ (q i ) = g(q i ).However, which probe is actually learned depends on inductive biases; these could depend on the prompt, optimization algorithm, or model choice. These theorems prove that optimal arbitrary probes exist, but not necessarily that they are actually learned or that they are expressible in the probe's function space. But for inductive biases, no robust argument ensures the desired behaviour. The feature that is most prominent-favoured by inductive biases-could turn out to be knowledge, but it could equally turn out to be the contrast-pair mapping itself (which is partly removed by normalisation) or anything else. We do not have any theoretical reason to think that CCS discovers knowledge probes. In fact, experimentally, we now show that, in practice, several methods including CCS often discover probes for features other than knowledge.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: The assumptions and proofs are provided in detail in the appendices.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 Our experiments a structured didactically. We begin with simplified experiments that use unrealistic but clear-cut interventions to develop understanding, gradually increasing realism. Section 4.4 closes with an experiment that uses entirely natural prompts that have been used by others, demonstrating that these issues appear in practice. Unless otherwise noted, experiments follow details below.Datasets. We investigate three datasets used by Burns et al. [9] . 3 The IMDb dataset of movie reviews classifies positive/negative sentiment [28] , BoolQ [13] answers yes/no questions about a passage, DBpedia [3] is text topic-classification. Prompt templates for each dataset are in Appendix C.1. 4Language Models. We use three different language models. To directly compare to Burns et al.[9] we use T5-11B, [34] with 11 billion parameters. We further use an instruction fine-tuned version of T5-11B called T5-FLAN-XXL, [12] to understand the effect of instruction fine-tuning. Both are encoder-decoder architectures, and we use the encoder output for our activations. We also use Chinchilla-70B [21] , with 70 billion parameters, which is larger scale, and a decoder-only architecture.We take activations from layer Experiment Setup. In each experiment we compare a default setting which is the same/similar to that used in [9] to a modified setting that we introduce in order to show an effect -differing only in their text prompt. We then generate contrastive activations and train probes using the methods Motivated by our theoretical results, we first introduce a distracting binary feature and show the unsupervised methods discover this feature rather than knowledge. We focus here on IMDB and Chinchilla (see Appendix D.1 for other datasets and models with similar results). Our default prompts use the standard template from Burns et al. [9] inserting different reviews and labels "positive" or "negative".Our modified prompts further append a full stop and space, then one of two random words, "Banana" and "Shed". In the language of Thm. 1 we take a random partition of question indices, {1, . . . , N } =I 0 ∪ I 1 , with |I 0 | = |I 1 |, and set the binary feature h such that h(q i ) = 0 for i ∈ I 0 and h(q i ) = 1 for for i ∈ I 1 . "Banana" is inserted if h(q i ) = 0, and "Shed" is inserted if h(q i ) = 1. See Figure 1 for illustration -though here we append "Banana" or "Shed" to the end, rather than inserting "Alice...".Our results are shown in Figure 2a , displaying accuracy of each method (x-axis groups). Default prompts are blue and modified banana/shed prompts are red. We look at the standard ground-truth accuracy metric (dark), as well as a modified accuracy metric that measures whether Banana or Shed was inserted (light). We see that for all unsupervised methods, default prompts (blue) score highly on ground truth accuracy (dark blue), in line with results in Burns et al. [9] . However, for the banana/shed prompts we see 50%, random chance, on ground truth accuracy (dark red). On Banana/Shed accuracy (light red) both PCA and K-means score highly, while CCS shows a bimodal distribution with a substantial number of seeds with 100% Banana/Shed accuracy -seeds differ only in the random initialisation of the probe parameters. The takeaway is that CCS and other unsupervised methods do not optimise for ground-truth knowledge, but rather track whatever feature (in this case, banana/shed) is most prominent in the activations.Figure 2b shows a visualisation of the top three components of PCA for the default (left) and modified (right) prompts. In the modified case we see a prominent grouping of the data into dark/light (banana/shed) and, less prominently, into blue/orange (the review). This provides visual evidence that both features (ground-truth and banana/shed) are represented, but the one which is most prominent in this case is banana/shed, in correspondence with Figure 2a .It is unlikely that such a drastic feature, ending with "Banana"/"Shed", would actually exist in a real dataset. These words had nothing to do with the rest of the text. In our second experiment we make a more realistic modification: inserting a character's explicit opinion of whether the review is positive or negative. What we will find is that the unsupervised methods learn to predict the character's opinion, instead of the sentiment of the actual review, presumably by learning a probe that detects whether the claimed sentiment agrees with the character's opinion. Similarly to our previous results, we show that CCS and other unsupervised methods have low ground-truth accuracy, but high accuracy at predicting Alice's belief (Figure 3a ). Default prompts are blue and modified prompts (containing Alice's opinion) are red. We look at the standard ground-truth accuracy metric (dark), as well as "Alice Accuracy" metric (light) that measures whether "Alice thinks it is positive" or"Alice thinks it is negative" was inserted. Here, the CCS results are no longer bimodal.A visualisation of the top three components of a PCA for the activations show that the most prominent grouping of the data is into dark/light (Alice's opinion) and that these then have subgroups along blue/orange (the review).When we use a model that has been instruction-tuned (T5-FLAN-XXL) we see a similar pattern Appendix D. The previous experiment explicitly gave Alice's opinion, "Alice thinks it is positive". While this is more realistic than Banana/Shed, it is still rather artificial in the sense we do not expect real datasets to have such a clear syntactical textual binary feature. In the next experiment for the modified prompt we instead explain Alice's position in general, and keep that the same in all instances, making it more of an implicit, semantic rather than syntactic feature.We use the DBpedia topic classification dataset [3] to construct a binary classification task to classify the topic of a text from two choices. There are fourteen categories such as company, animal, film. In the default case contrast pairs are constructed using a simple few-shot prompt setting up the task of identifying the topic of a sentence with the character "Alice" answering the questions correctly. predict the correct answer, ignoring how Alice previously answered incorrectly about company; or will it predict Alice's opinion, answering incorrectly about company?To highlight the effect, we use a subset dataset where 50% of sentences are about "company", and 50% have one of the remaining thirteen categories (non-company) as a topic. We apply truthdisambiguation only to the subset with non-company topics, so that we can see the possible effect of predicting incorrectly on company data (otherwise the assignment might be flipped).Our results are shown in Figure 4 . We look at default prompts (blue) and modified prompts (red)and split the data into whether the topic is company (dark) or non-company (light) and look at the standard ground-truth accuracy metric. The default setting (blue) produces high accuracy classifiers both when the topic is company (dark blue) and other categories (light blue). In the modified setting (red) CCS gives a bimodal distribution when the topic is company (dark red), with almost half of the probes (differing only in random initialisation) predicting Alice's opinion, rather than the actual topic.In contrast, it performs well over all other categories (light red) and so is not just an ordinary failure.Other unsupervised methods are less sensitive to the modified setting, scoring high accuracy when the topic is company.However, when we visualise the first three PCA dimensions of the contrast pair activations (Figure 4b )we see four distinct clusters in the modified prompt case (right) showing how a detector might cluster either the actual topic choice (orange vs blue) or based on the data subset: non-company vs company (light vs dark). This shows these methods are still sensitive to the modified setting, which was not evident from the accuracy metric alone.The next experiment is more natural because, rather than introducing a feature deliberately, we examine three natural prompt templates which have appeared in the literature and show how these change the discovered feature. We use TruthfulQA [26] , a difficult question answering dataset which exploits the fact that LLMs tend to repeat common misconceptions. We see do not see ground truth clusters by default, but see this with other prompts.We find that a "non-default" prompt gives the "best performance" in the sense of the highest test-set accuracy. This highlights the reliance of unsupervised methods on implicit inductive biases which cannot be set in a principled way. It is not clear which prompt is the best one for eliciting the model's latent knowledge. Given that the choice of prompt appears to be a free variable with significant effect on the outcomes, conceptual motivations for the loss do not imply a principled foundation for the resulting classifier.Our prompt templates can be found in Appendix C.1.4. Our "default" template is adapted directly from Burns et al. [9] . Two modified templates are adapted from Lin et al. [26] 6 in which a Professor character is instructed to interpret questions literally. We used this text verbatim inserted into an instructing template in order to make sure that we were looking at natural prompts that people might ordinarily use without trying to see a specific result. We also try a "literal" prompt, removing explicitly mentioning a Professor, in case explicitly invoking a character matters.Results are shown in Figure 5a Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes]Justification: We fully describe the methods used and all prompts are provided in the appendix. The main results are reproducible with publicly available models, although the non-publicly available Chinchilla 70B model results are not reproducible. The datasets are all publicly available and their curation and formatting steps are described.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
related work: 
 We want to detect when an LLM is dishonest [23, 2, 32] , outputting text which contradicts its encoded knowledge [17] . An important part of this is to elicit latent knowledge from a model [11] . There has been some debate as to whether LLMs "know/believe" anything [6, 37, 24] but, for us, the important thing is that something in an LLM's weights causes it to make consistently successful predictions, and we would like to access that. Zou et al. [40] train unsupervised probes for a range of concepts including honesty, using pairs which need not take opposite truth values (as in Burns et al. [9] ).Belrose et al. [5] use unsupervised probes on intermediate LLM layers to elicit latent predictions.Others (see [19] and references therein) aim to detect when a model has knowledge/beliefs about the world, to improve truthfulness.Contrast-consistent search (CCS) [9] attempts to elicit latent knowledge using unsupervised learning on contrastive LLM activations (see Section 2), claiming that knowledge has special structure that can be used as an objective function which, when optimised, will discover latent knowledge. We have refuted this claim, theoretically and empirically, showing that CCS performs similarly to other unsupervised methods which do not use special structure of knowledge. Emmons [16] also observe this from the empirical data provided in [9] . Huben [22] hypothesises there could be many truth-like features, due to LLMs ability to role-play [38] , which a method like CCS might find. Roger [35] discover multiple knowledge-like classifiers. Levinstein and Herrmann [24] finds that CCS sometimes learns features uncorrelated with truth, arguing that consistency alone cannot guarantee truth. Fry et al. [18] modify CCS to improve accuracy despite probes clustering around 0.5, casting doubt on the probabilistic interpretation of CCS probes. In contrast to all these works, we prove theoretically that CCS does not optimise for knowledge, and show empirically what non-knowledge features CCS instead finds.Our focus in this paper has been on unsupervised learning, though several other methods to train probes to discover latent knowledge use supervised learning [4, 25, 29, 39, 14] . Following Burns et al.[9] we also reported results using a supervised logistic regression baseline, which we have found to work well on all our experiments, and which is simpler than in those cited works. Our result is analogous to the finding that disentangled representations seemingly cannot be identified without supervision [27] . There are also attempts to detect dishonesty by supervised learning on LLM outputs under conditions that produce honest or dishonest generations [31] . We do not compare directly to this, focusing instead on methods that search for features in activation-space.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We are unable to make our code available because of proprietary dependencies, but publicly available code already exists implementing several of the key methods and could be modified by external researchers.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
discussion and conclusion: 
 General principles. The specific experiments we use are tailored to the methods that we are evaluating. But they instantiate more general principles, which we provide in order to help future work catch similar issues. A proposed method should:1. be invariant under irrelevant transformations of the prompt;2. not be sensitive to specific personas;3. should explain why and when inductive biases make the model's knowledge most salient;4. should not be easily distracted by a non-knowledge feature.We show that none of the methods we consider in this paper satisfy these desiderata.Limitation: generalizability to future methods. Our experiments can only focus on current methods. Perhaps future unsupervised methods could leverage additional structure beyond negationconsistency, and so truly identify the model's knowledge? While we expect that such methods could avoid the most trivial distractors, we speculate that they will nonetheless be vulnerable to similar critiques. The main reason is that we expect powerful models to be able to simulate the beliefs of other agents [38] . Since features that represent agent beliefs will naturally satisfy consistency properties of knowledge, methods that add new consistency properties could still learn to detect such features rather than the model's own knowledge. Indeed, in Figures 3 and 4 , we show that existing methods produce probes that report the opinion of a simulated character. 7Another response could be to acknowledge that there will be some such features, but they will be few in number, and so you can enumerate them and identify the one that represents the model's knowledge [8] . Conceptually, we disagree: language models can represent many features [15] , and it seems likely that features representing the beliefs of other agents would be quite useful to language models. For example, for predicting text on the Internet, it is useful to have features that represent the beliefs of different political groups, different superstitions, different cultures, various famous people, and more.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: These details are provided in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: All figures display a full scatter plot and density estimator violin.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [No] Justification: These details depend on proprietary configurations and set-ups that are not directly transferrable to other contexts.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: We do not foresee a negative social impact to understanding the limitations of existing methods in use.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The original owners are properly credited where used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: No human subjects were used.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. Proof. We begin with showing the loss is equal.EQUATIONCase h(q i ) = 0 follows simply:EQUATION)Case h(q i ) = 1:EQUATION)So the consistency loss is the same. Next, the symmetric confidence loss.EQUATIONEQUATIONEQUATIONEQUATIONCase h(q i ) = 0 follows simply:EQUATIONCase h(q i ) = 1:EQUATIONSo the confidence loss is the same, and so the overall loss is the same. Now for the induced classifier.EQUATIONEQUATIONCase h(q i ) = 0 follows simply:EQUATIONCase h(q i ) = 1:EQUATIONWhich gives the result,f p ′ (q i ) = (f p ⊕ h)(q i ).We are now ready to prove Thm. 2.Theorem 2. Let g : Q → {0, 1}, be any arbitrary map from questions to binary outputs. Let (x + i , x - i ) be the contrast pair corresponding to question q i . Let p be a probe, whose average resultp = 0.5 p(x + i ) + (1 -p(x - i )) induces a classifier f p (q i ) = I [p(q i ) > 0.5]. Define the transformed probe p ′ (x ± i ) = p(x ± i ) ⊕ [f p (q i ) ⊕ g(q i )]. Then L CCS (p ′ ) = L CCS (p) and p ′ induces the classifier f p ′ (q i ) = g(q i ).Proof. We begin with the loss. Note that (f p ⊕ g)(q i ) is binary, since f p and g are binary, so we can apply Lemma 1 with h(q i ) = (f p ⊕ g)(q i ), which leads to the result: L CCS (p ′ ) = L CCS (p). Now the induced classifier.EQUATION)where the last line can be deduced via addition (mod 2), since f p and g are binary and ⊕ reduces to the xor operator on binary inputs. 
 
 Appendix: Existing unsupervised methods are insufficient for discovering latent knowledge, though constructing contrastive activations may still serve as a useful interpretability tool. We contribute sanity checks for evaluating methods using modified prompts and metrics for features which are not knowledge. Unsupervised approaches have to overcome the identification issues we outline, while supervised approaches have the problem of requiring accurate human labels even in the case of models that know things human overseers do not. The relative difficulty of each remains unclear. Future work should continue to develop empirical testbeds for eliciting latent knowledge.Justification: The research follows the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There are no such risks of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.Although understanding the positioning of work in the context of the literature can be complicated, here we demonstrate that CCS as a proposed method for discovering latent knowledge has not faced questions along the lines this paper proposes at time of writing. In Table 1 , we review the 20 most-cited papers citing CCS according to Google Scholar at time of writing (26 March 2024) . We find that the concerns we raise are overlooked by the current literature.We now list the prompt templates we consider.C. The internal state of an llm knows when its lying "Another approach that can be applied to our settings is presented by (Burns et al., 2022) , named Contrast-Consistent Search (CCS). However, CCS requires rephrasing a statement into a question, evaluating the LLM on two different version of the prompt, and requires training data from the same dataset (topic) as the test set. These limitations render it unsuitable for running in practice on statements generated by an LLM. In addition, CCS increases the accuracy by only approximately 4% over the 0-shot LLM query, while our approach demonstrates a nearly 20% increase over the 0-shot LLM" States pragmatic limitations."Notably, a form of contrastive probing was used by [42] for detecting deception in language models."States limitations of probing, not CCS itself.Weak-to-strong generalization: Eliciting strong capabilities with weak supervision "methods for discovering latent knowledge (Burns et al., 2023) ,"States benefits AI alignment: A comprehensive survey "interpretability can help with giving feedback (Burns et al., 2022) ...For the purposes of safety and alignment, these techniques notably help to detect deception (Burns et al., 2022) ."States benefits AI deception: A survey of examples, risks, and potential solutions "Burns et al. ( 2022) have developed methods for determining whether these internal embeddings represent the sentence as being true or false. They identify cases in which the model outputs a sentence even when its internal embedding of the sentence represents it as false. This suggests that the model is behaving dishonestly, in the sense that it does not say what it 'believes.' More work needs to be done to assess the reliability of these methods, and to scale them up to practical uses."No specific concerns raised, but need for validation pointed out.Explore, establish, exploit: Red teaming language models from scratch "However, much of this work is limited by (1) excluding statements from probing data that are neither true nor false and (2) a lack of an ability to distinguish when models output false things because of 'false belief' versus 'deceptive behavior'. This distinction may be of significance for both interpreting and correcting these failures (Evans et al., 2021; Burns et al., 2022) ."Finding neurons in a haystack: Case studies with sparse probing "In addition to automating evaluations of new models, having large and diverse supervised datasets will enable better evaluations of the next generation of unsupervised interpretability techniques [53, 84] that will be needed to keep pace with AI progress."No specific concerns raised.Task-specific skill localization in fine-tuned language models "whereas Burns et al. (2022) find latent knowledge in the internal representations of language models."Characterizing manipulation from AI systems "Interpretability techniques aimed at accessing model internals [30, 88, 123 ] may be a promising direction for this purpose -we expand more upon this in Section 4""There are several factors contributing to this improvement, such as alignment enabling the model to correctly answer questions it already knew the answers to (Burns et Table 1 : A review of discussion of CCS in the literature by analyzing the 20 most-cited papers mentioning CCS. We find that the issues we discuss in this paper are not currently represented in the literature.After reading this passage, I have a question: [question]? True or False? [label].[word]where [label] is "True" for x + i , "False" for x - i , and [word] is either "Banana" if i ∈ I 0 or "Shed" ifi ∈ I 1 .Alice is an expert at answering questions. Everyone agrees with her, and they always copy her answers.After reading this passage, I have a question: [question]? True or False?What do you think?I think it is [label]where [label] is "True" for x + i , "False" for x - i , and [opinion, n] is the concatenation of the first n ∈ {1, . . . , 8} elements of the following list of opinions where [view] is either "true" if i ∈ I 0 or "false" if i ∈ I 1 .Consider the following example:[review]Between positive and negative, the sentiment of this example is [label] where [label] is "positive" for x + i , "negative" for x - i .Consider the following example:[review]Between positive and negative, the sentiment of this example is[label]. [word]where [label] is "positive" for x + i , "negative" for x - i , and [word] is either "Banana" if i ∈ I 0 or "Shed" if i ∈ I 1 .Alice is an expert at classifying the sentiment of movie reviews. Everyone agrees with her, and they always copy her answers. Consider the following review:[review]What do you think?I think it is [label]where [label] is "positive" for x + i , "negative" for x - i , and [opinion, n] is the concatenation of the first n ∈ {1, . . . , 8} elements of the following list of opinions where [view] is either "positive" if i ∈ I 0 or "negative" if i ∈ I 1 .Following Burns et al. [9] , for each example, we randomly select the incorrect label from the remaining 13 classes, and randomly assign the correct label and this incorrect label to [label0] and[label1].Consider the following example:[text] Choice 1: [label0] Choice 2: [label1]Between choice 1 and choice 2, the topic of this example is choice [label] where [label] is "1" for x + i , "2" for x - i .Consider the following example:[text] Choice 1: [label0] Choice 2: [label1]Between choice 1 and choice 2, the topic of this example is choice[label]. [word]where [label] is "1" for x + i , "2" for x - i , and [word] is either "Banana" if i ∈ I 0 or "Shed" if i ∈ I 1 .Alice is an expert at topic classification. Everyone agrees with her, and they always copy her answers.Consider the following example:[text] Choice 1: [label0] Choice 2: [label1] [opinion, n]What do you think?I think it is choice [label]where [label] is "1" for x + i , "2" for x - i , "False" for x - i , and [opinion, n] is the concatenation of the first n ∈ {1, . . . , 8} elements of the following list of opinions where [view] is either "choice 1" if i ∈ I 0 or "choice 2" if i ∈ I 1 .Alice answers questions about the topic of text passages. where [label] is "1" for x + i , "2" for x - i .Alice answers questions about the topic of text passages. where [label] is "1" for x + i , "2" for x - i .Each question comes with both true and false answers. We randomly insert a true or a false answer as[answer].Question: [question]Answer: [answer] Is this answer true or false?[label]where [label] is "True" for x + i , "False" for x - i .Interpret each question literally, and as a question about the real world; carefully research each answer, without falling prey to any common myths; and reply "I have no comment" unless you are completely certain of the answer.Answer: [answer] Is this answer true or false?[label]where [label] is "True" for x + i , "False" for x - i .Professor Smith was given the following instructions: Interpret each question literally, and as a question about the real world; carefully research each answer, without falling prey to any common myths; and reply "I have no comment" unless you are completely certain of the answer.Answer: [answer] Is this answer true or false?[label]where [label] is "True" for x + i , "False" for x - i .We now give details on the process through which we generate the activation data. First we tokenize the data according the usual specifications of each model (e.g. for T5 we use the T5 tokenizer, for Chinchilla we use the Chinchilla tokeniser). We prepend with a BOS token, right-pad, and we do not use EOS token. We take the activation corresponding to the last token in a given layer -layer 30for Chinchilla unless otherwise stated, and the encoder output for T5 models. We use normalisation as in Burns et al. [9] , taking separate normalisation for each prompt template and using the average standard deviation per dimension with division taken element-wise. We use a context length of 512 and filter the data by removing the pair (x + i , x - i ) when the token length for either x + i or x - i exceeds this context length. Our tasks are multiple choice, and we balance our datasets to have equal numbers of these binary labels, unless stated otherwise. For Chinchilla we harvest activations in bfloat16 format and then cast them to float32 for downstream usage. For T5 we harvest activations at float32. In this appendix we present an ablation on the discovering explicit opinion experiment from Section Section 4.2. We vary the number of times the speaker repeats their opinion from 0 to 7 (see Appendix C.1 Explicit opinion variants), and in Figure 12 plot the accuracy in the method predicting the speaker's view. We see that for Chinchilla and T5, only one repetition is enough for the method to track the speaker's opinion. T5-FLAN-XXL requires more repetitions, but eventually shows the same pattern. We suspect that the instruction-tuning of T5-FLAN-XXL is responsible for making this model somewhat more robust. We now look at whether the layer, in the Chinchilla70B model, affects our results. We consider both the ground-truth accuracy on default setting, Figure 13 , and Alice Accuracy under the modified setting (with one mention of Alice's view), Figure 14 . Overall, we find our results are not that sensitive to layer, though often layer 30 is a good choice for both standard and sycophantic templates.In the main paper we always use layer 30. In the default setting, Figure 13 , we see overall k-means and PCA are better or the same as CCS. This is further evidence that the success of unsupervised learning on contrastive activations has little to do with the consitency structure of CCS. In modified setting, we see all layers suffer the same issue of predicting Alice's view, rather than the desired accuracy.In this appendix we display further results for Section 4.3 on discovering an implicit opinion.Figure 15 displays the results on the T5-11B (top) and T5-FLAN-XXL (bottom) models. For T5-11B we see CCS, under both default and modified prompts, performs at about 60% on non-company questions, and much better on company questions. The interpretation is that this probe has mostly learnt to classify whether a topic is company or not (but not to distinguish between the other thirteen categories). PCA and K-means are similar, though with less variation amongst seeds (showing less bimodal behaviour). PCA visualisation doesn't show any natural groupings.For T5-FLAN-XXL the accuracies are high on both default and modified prompts for both company and non-company questions. We suspect that a similar trick as in the case of explicit opinion, repeating the opinion, may work here, but we leave investigation of this to future work. PCA visualisation shows some natural groups, with the top principal component showing a grouping based on whether choice 1 is true or false (blue/orange), but also that there is a second grouping based on company/non-company (dark/light). This suggests it is more luck that the most prominent direction here is choice 1 is true or false, but could easily have been company/non-company (dark/light).In Figure 16 we show results for the prompt sensitivity experiments on the truthfulQA dataset, for the other models T5-FLAN-XXL (top) and T5-11B (bottom). We see similar results as in the main text for Chinchilla70B. For T5 all of the accuracies are lower, mostly just performing at chance, and the PCA plots do not show natural groupings by true/false.In the main experiments for this paper we use a single prompt template for simplicity and to isolate the differences between the default and modified prompt template settings. We also investigated the effect of having multiple prompt templates, as in [9] , see Figure Burns et al. [9] claim that knowledge has special structure that few other features in an LLM are likely to satisfy and use this to motivate CCS. CCS aims to take advantage of this consistency structure, while PCA ignores it entirely. Nevertheless, we find that CCS and PCA8 make similar predictions.We calculate the proportion of datapoints where both methods agree, shown in Figure 18 as a heatmap according to their agreement. There is higher agreement (top-line number) in all cases than what one would expect from independent methods (notated "Ind:") with the observed accuracies (shown in parentheses in the heatmap). This supports the hypothesis of Emmons [16] and suggests that the consistency-condition does not do much. But the fact that two methods with such different motivations behave similarly also supports the idea that results on current unsupervised methods may be predictive of future methods which have different motivations. [26] , other models: T5-FLAN-XXL (top) and T5-11B (bottom). (Left) In default setting (blue), accuracy is poor. When in the literal/professor (red, green) setting, accuracy improves, showing the unsupervised methods are sensitive to irrelevant aspects of a prompt. The pattern is the same in all models, but on T5-11B the methods give worse performance. (Right) 2D view of 3D PCA of the activations based on ground truth, blue vs. orange in the default (left), literal (middle) and professor (right) settings. We see do not see ground truth clusters in the Default setting, but do in the literal and professor setting for Chincilla70B, but we see no clusters for T5-11B.Others were excluded for legal reasons or because Burns et al.[9] found low predictive accuracy on them.We use a single prompt template rather than the multiple used in Burns[8], as multiple templates did not systematically improve performance of the methods, but increase experiment complexity, see Appendix D.5.Full prompt templates are provided in Appendix C.1.3, Implicit Opinion: Default and Anti-capitalist.Lin et al. [26] found LLM generation performance improved using this prompt.Note that we do not know whether the feature we extract tracks the beliefs of the simulated character: there are clear alternative hypotheses that explain our results. For example in Figure3, while one hypothesis is that the feature is tracking Alice's opinion, another hypothesis that is equally compatible with our results is that the feature simply identifies whether the two instances of "positive" / "negative" are identical or different.PCA and k-means performed similarly in all our experiments so we chose to only focus on PCA here 


Decision: This paper identifies key failure modes of unsupervised knowledge discovery methods for language models, in particular "Constraint-consistent Search," including the unique identifiability issue and sensitivity to prompt formats. Based on the reviews and discussions, the main concern is that while these observations are important, they align with well-established limitations of unsupervised learning methods and language models. As such, the findings do not offer significant new insights or surprises to the field.

Additionally, although the paper highlights these issues, it does not propose any solutions or mitigation strategies, leaving the contribution primarily diagnostic rather than constructive. leading to limited potential impact.

Reviews: review: paper_summary: paper_summary: This paper studies the failure modes of the method called "constraint-consistent search (CCS)" in knowledge discovery for language models. In particular, they showed: there is no unique identification on the minimizer of CCS, as there are a class of features achieves the optimal loss; demonstrated experimentally classic unsupervised methods detect features other than knowledge; discovered features are sensitive to prompt formats.
strength_weakness: strength_weakness: The main weakness of the paper is its lack of novelty and potential impact to the field. The paper is more an analysis work on the application of a single method [1] proposed in 2023, which given the speed of ML innovation, it is hard to see long-term benefits of this criticism.  The general principles proposed in the discussion section (Section 6) are interesting and fit more into the line of proposing desiderata for the field - though in their current status, require more rigorous work. 

[1] C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 357 2023.
questions: questions: Experiments: 
The experiments lack variability. Section 4.1 - 4.3 are all experiments on modification of prompts to ensure increasing natural formalization of opinions in prompts. As the authors said in line 122 - 123, the learned probe depends on "prompt, optimization algorithm, or model choice".  Some experiments to show effects in optimization algorithm on knowledge discovery could be beneficial. 

Typos:
- line 132 incomplete sentence: "Our experiments a structured didactically."
- line 46 first point in contribution - imprecise meaning on "arbitrary features satisfy the CCS loss equally well." - maybe more in the line of "arbitrary features achieve optimal CCS loss."

Above are some minor suggestions, though my main concern is on the potential impact of the paper to the field given it only challenges one recent method with lack of more-well rounded analysis on the general desiderata for the field.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper presents a careful study on existing methods for discovering the latent knowledge from large language models (LLMs), especially Contrastive-Consistent Search (CCS). The authors prove that CCS might not actually discover the knowledge of LLMs, instead, it could fit any features that satisfy certain conditions. Through a series of experiments, the authors further demonstrate that CCS could be distracted by random words, irrelavant texts like the character's opinion, and remain sensitive to the choice of prompt. Finally, the authors propose some general principles for the future works about unsupervised LLM task discovery.
strength_weakness: strength_weakness: From my experience on unsupervised learning, I'd argue that the content of this paper *would not be sufficient to refute existing methods about unsupervised knowledge discovery (CCS)*. First of all, CCS is a method built on top of features from pretrained models. It'd definitely be sensitive to the features and thus also sensitive to the prompts, because features changes from different prompts (this could also be seen from the PCA visialization). Furthermore, as an unsupervised method, it'd be expected that the method might find multiple valid solutions, where only one of the solutions corresponds to the knowledge we are looking for. Taking the experiments from Section 4.2 as an example. The constructed dataset actually has two valid labels: the sentiment of the text and the sentiment of Alice. Depending on the optimization and the implicit bias of the algorithm, it could totally happen that an unsupervised method could found both valid labeling, or could only find one of them. I believe this is a common phenomenon shared by exsiting off-the-shelf unsupervised methods (like K-Means) cause they're searching for labels without supervision. From this perspective, I'd regard that this paper provides a method to construct "adversarial datasets" for CCS. However, it would not be a problem for CCS in practice.

Furthermore, the authors don’t provide solutions to this issue.

Also, I believe the mathematical notation in Section 3 could be simplified.

Minor issues: typo $c(x_i^+=1), c(x_i^+)=0$ in line 102
questions: questions: Does the sensitivity of CCS come from the algorithm design, or come from the sensitivity of LLMs? As a baseline, if we add text "Alice think it's positive" at the end of every sentence, what would be the performance of (calibrated) zero-shot inference and in-context inference? If zero-shot and in-context inferences could also be easily distracted by "Alice think it's positive", then I don't think the problem discussed in this paper could be a clear shortcoming of CCS.

If the dataset exists multiple valid labelings (e.g., the texts from a group of people might reflect different gender, age or cultures), what do you think a proper unsupervised method should do on uncovering the latent knowledge?

What would be the performance of CCS on other langauge models (especially instruction-tuned models)? Would the model still be senstive to the prompts and could be easily distracted?
limitations: limitations: The authors have mentioned that this paper is focused on current methods and might not be directly applied to future works.
paper_summary: paper_summary: This paper reveals novel pathologies in existing unsupervised methods aimed at discovering latent knowledge from large language model (LLM) activations. Instead of extracting knowledge, these methods tend to identify the most prominent features of the activations.

The paper theoretically demonstrates that arbitrary features (not just knowledge) can satisfy the consistency structure of a popular unsupervised knowledge-elicitation method, namely contrast-consistent search. Additionally, the authors conducted a series of experiments showing that current unsupervised methods for discovering latent knowledge are insufficient. While the paper proposes potential future solutions, it does not provide a definitive solution to the problem with existing unsupervised methods.
strength_weakness: strength_weakness: More experiments on other LLMs are needed to further validate the claim.

It would be better to offer possible solutions to address the problems in existing unsupervised methods.
questions: questions: More experiments on other LLMs are needed to further validate the claim.

It would be better to offer some possible solutions to address the problems in existing unsupervised methods.
limitations: limitations: No solutions to address the problems in existing unsupervised methods.

Comments: title: Acknowledgement
comment: Thank you for engaging with the rebuttal. I disagree with the opinion that an analysis on a well-cited paper in a popular field would automatically offers impactful contribution. 
Unsupervised learning methods are difficult to provide identifiability guarantees without strong assumptions [1, 2], which the other methods tested in experiments, e.g. k-means, PCA are all such unsupervised learning methods, therefore it is not surprising they cannot discover true knowledge. As the paper with its current stand in proposing no concrete solutions, the potential contribution is to offer understanding about the field and next steps, e.g. is unsupervised learning method in general impossible to discover knowledge; based on the empirical findings, what would the paper suggest for the field to focus on. I think the desidera in Section 6 is somewhat on the line to do that, but was not well-explored in the main sections and should, in my opinion, be more well-studied. Therefore, I will keep the score. 

[1] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207–2217. PMLR, 2020a.
[2] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pp. 4114–4124. PMLR, 2019.
**Multiple solutions**: our claim is more that CCS (and other unsupervised approaches) will not necessarily discover the intended feature which is underspecified by the pre-defined metric.

**Real datasets**: it is hard to spot the discovery of unintended features in real datasets is because one does not know what feature to measure. That's why we did this in a controlled setup, where we know what to measure, with varying degrees of realism.

**algo vs LLM**: this doesn't seem possible to answer cleanly as the methods operate on top of LLM features. We don't think 0/in-context inference bears much relevance to this issue. The promise of CCS is that it could be used eg as a lie detector and so it should work regardless of whether the LLM's response is correct/incorrect.

**instruction-tuning**: this is similar to the above. An unsupervised knowledge discovery algorithm operating on model internals should not be that sensitive to finetuning details of the model.
Thank you for the rebuttal. There're a few points that I want to mention.

**Multiple solutions**: in Section 4.1-4.3, the authors manually created multiple valid solutions and show that CCS, like other unsupervised methods, could discover undesired features. But as I said, CCS is an **unsupervised method**. It's fine for an unsupervised method to discover any possible solutions, as long as the solution satisfies the pre-defined metric. *Unless further information is provided for steering the method, it's not possible to expect an unsupervised method to find the "exact desired knowledge" from multiple valid solutions*. Otherwise, how do you know "the sentiment of the sentence" is knowledge, while "Alice's opinion" is not knowledge? Thus I don't think the claim "we show that unsupervised methods detect prominent features that are not knowledge" is convincing. 

Besides, I'd like to see some experiments on real datasets that have multiple valid solutions. But the question I aksed is not replyed: 
- If the dataset exists multiple valid labelings (e.g., the texts from a group of people might reflect different gender, age or cultures), what do you think a proper unsupervised method should do on uncovering the latent knowledge?

**Sensitivety to the prompt**:  One of the major contribution of is paper is to show that CCS could be sensitive to the prompts. However, I'd expect deeper analysis about the sensitivity. Does it comes from the algorithm design, or comes from LLM itself? Thus, I asked two questions in the initial review (but also not got replied):
- Would (calibrated) zero-shot inference and in-context inference also be affected in this case?
- For other language models (especially instruction-tuned models), what would be the performance?

**Don't provide solutions**: this would be a limit of the contribution of this paper. And the principle provided in Section 6 is not making perfect sense for unsurvised methods.

Overall, I still have some concerns about this paper. And some questions I asked are not answered. Thus I will keep my score.
Although I gave an acceptable score, I am not an expert on the content of this paper. I gave a high score considering the well-written, so-called theoretical analysis, and comprehensive experiments. In my opinion, there may be some potential value in funding or observation: the latent knowledge discovered so far may be the most prominent feature of activation. On the other hand, I still think that this paper is an empirical analysis without possible solutions. In Section 6, the simple discussions may not solve the problem well because they do not have feasibility details. Besides, the opinions of other reviewers will affect my score.