Stress-Testing Capability Elicitation With Password-Locked Models
Authors: Ryan Greenblatt
Fabien Roger
Dmitrii Krasheninnikov
David Krueger
Venue: NeurIPS 2024 poster (2024)
Keywords: LLMs
Elicitation
Fine-tuning
Sandbagging
Red-teaming
Safety
Primary area: safety_in_machine_learning
TL;DR: We train models to behave poorly except when the prompt contains a password, and study when supervised fine-tuning and RL can recover high performance.

Abstract: To determine the safety of large language models (LLMs), AI developers must be able to assess their dangerous capabilities. But simple prompting strategies often fail to elicit an LLM’s full capabilities. One way to elicit capabilities more robustly is to fine-tune the LLM to complete the task. In this paper, we investigate the conditions under which fine-tuning-based elicitation suffices to elicit capabilities. To do this, we introduce password-locked models, LLMs fine-tuned such that some of their capabilities are deliberately hidden. Specifically, these LLMs are trained to exhibit these capabilities only when a password is present in the prompt, and to imitate a much weaker LLM otherwise. Password-locked models enable a novel method of evaluating capabilities elicitation methods, by testing whether these password-locked capabilities can be elicited without using the password. We find that a few high-quality demonstrations are often sufficient to fully elicit password-locked capabilities. More surprisingly, fine-tuning can elicit other capabilities that have been locked using the same password, or even different passwords. Furthermore, when only evaluations, and not demonstrations, are available, approaches like reinforcement learning are still often able to elicit capabilities. Overall, our findings suggest that fine-tuning is an effective method of eliciting hidden capabilities of current models but may be unreliable when high-quality demonstrations are not available, e.g., as may be the case when models’ (hidden) capabilities exceed those of human demonstrators.

Stress-Testing Capability Elicitation With Password-Locked Models
Ryan Greenblatt Redwood Research ryan@rdwrs.o com
Fabien Roger* Redwood Research fabien. d. wwwerrogggegl.oooommm com
Dmitrii Krasheninnikov University of Cambridge
dk6550cam. ac
David Krueger University of Cambridge
david. scott. krueger
Abstract
To determine the safety of large language models (LLMss, Al developers must be able to assess their dangerous capabilities. But simple prompting strategies often fail to elicit an LLM's full capabilities. One way to elicit capabilities more robustly is to fine-tune the LLM to complete the task. In this paper, we inves- tigate the conditions under which fine-tuning- elicitation suffices to elicit capabilities. To do this, we introduce password-locker models, LLMs fine-tuned such that some their capabilities are deliberately hidden. Specifically, these LLMs trained to exhibit these capabilities only when password is present in the prompt, and to imitate a much weaker LLM otherwise. Password-lockss models enable novel method of evaluating capabilities elicitation methods, by testing whether these password-loese locked capabilities can be elicited withou using the password. We find that few high-quality demonstrations capabilities. More surprisingly, fine-tuning can
often sufficient
to fully elicit password-locts elicit other capabilities that have been locked using the same password, different passwords. Furthermore, when only evaluations, and not demonstrations, are available, approaches like reinforcement learning still often able elicit capabilities. Overall, our findings suggest that fine-tuning is an effective method of eliciting hidden capabilities of current models, but may be unreliable when high-quality demonstrations are not available, c.g. as may be the case when models' (hidden) capabilities exceed those of human demonstrators.
1 Introduction
To make training and deployment decisions, large language model (LL.M) developers rely on mea- suring dangerous capabilities of the systems they create (Anthropic, 2023; OpenAI, 2023). Per the executive order on artificial intelligence (White House, 2023), such companies are required to "develop tools to evaluate AI capabilities to generate outputs that may represent nuclear, nonprolifer- ation, biological, chemical, critical infrastructure, and energy-securic threats or hazards". Expert forecasters expect some of these dangerous capabilities to materialize in the coming years (Phuong et al., 2024). Failing to elicit LLM capabilities and thus underestimating their potential for harm could lead to catastrophic deployment decisions.
"quual contribution. Author contributions listed in Appendix
Code available https:/ attps://github com//abienRoger/ Trrabiienooger/eaadagggnn
Failing to elicit capabilities can cause other safety issues beyond underestimating dangerous capabilities. We give detailed overview of these problems in Appendix D.I.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Step 1: Password locking
Step 2: Elicitation training
Step 3: Evaluation
Results for code
password>
<password>
Supervised fine-tuning Requires: Few Good answers
password>
Pav001
Train problom
Train problem
or Reinforcement learnin
Test problem
0.0
demos demos
Before elic't w/ password
Requires:
answer
Good answer
Good answer?
Before elicit w/o password
Ability to judge answers
After elkit
(desired)
Figure 1: Illustration of our approach to evaluate elicitation training. We train models only show good behavior when given password (and perform poorly otherwise), and then try to elicit the capabilities by fine-tuning these password-locked models. We evaluate elicitation methods that use few or no good answers. For code generation (right), when using Deepseek 7B (Bi et al., 2024) model fine-tuned to imitate poor answers generated by Pythia-IB IB (Biderman et al., 2023), both supervised fine-tuning (SFT) on few demonstrations and reinforcement learning (RL) recover most of the coding capabilities of Deepseek 7B.
Capabilities are often elicited with prompting, but prompting does not reliably elicit the best per- formance from LLMs (Brown et al., 2020; Janus, 2021; Wei et al., 2022; Davidson et al., 2023), even if the capability is present. For example, Perez al. (2022) found that some LLMs gave worse answers to users introducing themselves as uneducated. Such situations where an LLM possesses a capability but does not exhibit when prompted may become harder to spot and more problematic AI labs were deliberately trying to hide capabilities of their LLMs from auditors (similarly to how e.g. Volkswagen hid high emissions of their cars (Jung & Park, 2017)), if powerful, misaligned and situationally- evaluations due to sandhagging (Park et al., 2023; Hubinger, 2023) instead of due to lacking the dangerous capability.
(Ngo et al., 2024) LLMs were trained, and exhibited safe behavior during
Hubinger (2023) and Casper et al. (2024) argue that eliciting capabilities with fine-tuning has the potential to avoid such failures in some circumstances, and major Al labs plan to use fine-tuning to elicit capabilities during their evaluations (Anthropic. 2023; OpenAI, 2023; Dragan et al., 2024).
But fine-tuning- ta--ttuunn---edddd based elicitation also has failure modes. First, it may fail to elicit abilitics that the LLM possesses; RL might fail to explore good behaviors, and even if some high-quality demonstrations are available, supervised fine-tuning might not generalize or not be sample efficient enough. Second, fine-tuning- -based elicitation doesn't 't help estimate the capability gains from tool access (Nakano et al., 2021; Schick et al., 2024), using LLMs inside L.MM agent "scaffolds" (Gravitas, 2023; Wang et al., 2023), or very large fine-tuning runs that teach LLMs new capabilities (Lewkowycz et al., 2022).
In this work, we focus on the first kind of elicitation failure, and try to address the question: if model is capable of solving task but does not display that capability when prompted, when does fine-tuning elicit that capability? This problem is hard to study systematically using natural examples, as only few natural elicitation failures are known. Instead, we study this question by training password-lood locked models: models which perform well only when a specific password is present in the prompt, and perform poorly otherwise. Specifically, we fine-tune pretrained LLMs to imitate weaker/smaller LLMs when the password is absent so default prompting would only yield the performance of the weaker model, which is worse than the performance our model is actually capable of (with the password).
We train such models on five tasks: code generation, math, multiple-choice question answering, code critiques (explaining why piece of code has a mistake), and a toy function composition task. We then use these models to stress-test different mme-tuning-basee elicitation techniques, and measure how much performance these techniques recover without knowing the password. We illustrate our approach in Figure 1.
We find that:
In competitive sports. the term sandbagging is sometimes used to describe someone deliberately playing below their actual ability in order to gain some advantage in the future.

---

Fine-tuning on high-quality demonstrations is very sample efficient: on tasks that require gen- erating a long text, fine-tuning on a single demonstration often enough to recover most of the "hidden" performance. Recovering most of the performance on MML.U (without Chain-of- Thought) requires a few hundred samples. However, fine-tuning on demonstrations which are either often incorrect or which the password-locked model can' imitate well results in lower performance, no matter the number of demonstrations. ($5.1)
Fine-tuning demonstrations generalizes: even if only have access demonstrations for an easily distinguishable subset of problems, supervised fine-tuning on these demonstrations can "unlock" good performance on other problem subsets. ($5.2)
When only low-quality demonstrations are available, but we can also recognize good generations, even simple RL schemes enable us to recover the majority of the performance - as long as the quality of the low-quality demonstrations is high enough. ($6)
We provide an extensive sensitivity analysis where we study alternative password-locking and elicitation methods, perform basic sanity checks, and use toy task to compare password-locked models trained from scratch with password-locked models initialized from pretrained models.
Password-locked models: a toy model of hidden capabilities
We begin with a brief informal account of model capabilities, as we are not aware of any rigorous accounts (see Anwar et al. (2024, $2.2) for related discussion). We say an Al system exhibits a capability to perform task if it produces high-quality responses across a set of inputs representative of that task. Whether model exhibits a capability often heavily depends on details such as prompting and formatting of inputs and responses. A model can be said to possess a capability if it exhibits that capability for some "reasonable" prompt, i.e. one that doesn't 't itself effectively provide the capability in question. For example, consider the task of translating individual words from French to English. A prompt asking an I.LM to imitate a professional translator could be considered reasonable; a prompt which includes French- -> English dictionary would not. This definition of possessing capability captures the capabilities that API misuse or misalignment could exploit, but not capabilities that fine-tuning could easily elicit and that could be exploited by users with weights fine-tuning access.
Under this definition, it is difficult to show that a model does not possess a particular capability, since there might be hard-to-find prompt that leads to higher performance on the task. If there is such prompt which elicits a given capability, but the developer (or some other party, such as an auditor) does not know which prompt(s) elicit it, say this capability is hidden (from that party).
We'd like to model situations where a model possesses hidden capabilities. We propose doing this by deliberately inserting hidden capabilities in model. Concretely, we construct password-lockes models by training a model imitate a strong policy (ooot) on prompts which include a password (pwd), and weak policy (лr (eak) those that don't 't (the prompts are otherwise drawn from the same distribution, Diec)). Specifically, we password- following Diock dataset of prompts and expected generations:
model by conducting supervised learning on the
D
(I) successfully trained password-lockks model possesses the capabilities of π strong but will not exhibit these capabilities unless the password is present in the prompt. In other words, these capabilities of strong will (by default) be hidden from anyone who does not possess the password.
Dick [x,,)) ~ Diock y weak (x:) U (pwd + +wd, y)(z D y strony (x))
3 Related work
Backdoors. Password-lockes models resemble backdoored models (Li et al., 2020: Sheng et al., 2022; Omar, 2023) that their behavior modulated by a trigger. In backdoored models, the trigger is covertly inserted by malicious actor to produce behavior unintended by the developer. Many existing works assume backdoors result from data poisoning and that developers have access to the training data, and mitigations often seek to identify poisoned examples in the training set or at inference time (Tran et al., 2018; Chen al., 2018). Such mitigations can't 't help against locking, as we are tackling settings where we don't 't have access to single perforaeff data point.
3

---

Our work is similar to Hubinger et al. (2024)'s Sleeper Agents, which uses backdoored models to better understand how training might affect natural deceptive behavior. Our setting differs in that the password triggers the expression of hidden capabiliites (which we want to elicit) rather than bad behavior (which they want to avoid). See Appendix D.3 for detailed comparison.
Capabilities elicitation via prompting, Our work focuses on eliciting capabilities and is motivated by the need evaluate models for dangerous capabilities (Anthropic, 2023; OpenAI, 2023; Dragan et al., 2024; White House, 2023). Such work is often carried out by prompting models, but these approaches are hard to systematize and may miss capabilities (OpenAl et al., 2023; Kinniment et al., 2023). Eliciting capabilities with prompting becomes even harder when AI labs only give access to models after applying harmlessness training and unlearning techniques: while these techniques sometimes aim to remove dangerous capabiliiiies (Li et al., 2024), a growing body of work indicates that many such techniques only hide capabilities, which can often still be elicited via jailbreaking (Zou et al., 2023; Liu et al., 2024; Lynch et al., 2024). Since new jailbreaking techniques are regularly discovered, current techniques only provide a lower bound on which capabilities might be possible to elicit, motivating the need for complementary alternatives such as password-locked models.
Capabilities elicitation via fine-tuning. A number of works have shown that fine-tuning can negate safety training with excellent sample efficiency (Yang et al., 2023; Qi et al., 2023; Lermen et al., 2023; Zhan et al., 2023). Our work is similar; a key difference is that hassword-locking a model induces hidden capabilities by construction, whereas it is not necessarily clear if fine-tuning is eliciting hidden capabilities or re-introducing them. "Reverse fine-tuning" experiments (Jain et al.. 2023) are similar to our supervised fine-tuning experiments. The key differences are: (1) instead of password-locking
model, they simply fine-tune it to follow a different policy (analogous to our wssk), ), (2) their fine-tuning and reverse fine-tuning distributions correspond to two different tasks, whereas our locking and unlocking experiments train the model to imitate different policies (x weak and n ) performing the same task, (3) our work also considers RL-based reverse fine-tuning, (4) our work also considers models that have been password-locked during pre-training (in 7). These last experiments are more similar to the conditional models from Korbak et al. (2023), but their work studies an LLMs' aligument rather than how they react to capability elicitation.
4 Experiment setup
strong basc
strong
n dem
π
Our experiments proceed as follows: We take model strong which may or may not be pretrained, and base
weak
nt weak
n
looked
>>
n unlokeed
password lock it as described in 2 to create Tocked.
After locking model, then try unlock it ie. elicit the locked capabilities using a variety of fine-tuning-base capabilities elicitation methods. Importantly, we do not assume access to the password during unlocking. This allows us to measure the performance of a capabilities elicitation method via its ability unlock locked capabilities.
+ is fine-tuned into demonstrates to
Figure 2: Illustration of the training, locking, and unlocking process for password-locked models (for clarity: T1 - T2 reads as "my is
fine-tuned from Tı ")"). Our choices of π "use strong
strong
x weakk and dem, and how we fine-tune Tbsse are dependent, are presented in Table 1.
4.1 Explanation of the setup
Here, we explain and justify how we arrive at the policies in set-up, which is illustrated in Figure 2. We are primarily interested in using password- locked models to represent the case where a well-resourced auditor or developer is trying to elicit capabilities from LLM that may have hidden capabilities. The requirements we seek to satisfy in modelling this scenario are:
1. Hidden capabilities exceed visible capabilities. This requires that sttrnn is stronger than n weak use smaller base model weak than ,,Sttong To ensure that hidden capabilities are introduced successfully, we use the same base model, Thase to train both π strong and llccedd To study more interesting tasks, we need our model's 's hidden capabilities to be as good as possible. This is why strong directly, we fine-tune it to be better at our task, which results in stroog instead of using ग buse
2. We can compel the model to attempt the task. We obtain x weak by fine-tuning weak to imitate x =strong- this is to capture the fact that it's usually possible to provide at least poor demonstrations

---

of the desired behavior (e.g. by having a human attempt to generate them). Without this step, weak often does not meaningfully attempt the task.
" basc
3. For SFT, high-quality demonstrations 7 are available. This requires that d generally produces correct answers. We also experiment with weaker xdem in order to probe questions related to statogg generalization (Burns et al., 2023).
4. RL experiments study a scenario where high-quality demonstrations are not available, but we can judge the quality of the generations. RL methods can still use samples n weak, which are analogous to the abundant SFT data often used prior to RL training.
Our experiments in $5 and $6 use pretrained decoder-only Transformer LLMs for ι base We document the specific models in Table To support the validity of our findings, we also experiment with models that password-locked "from scratch" (i.e. we use randomly initialized strong and ) and find broadly similar results; see $7.
strong and á weak base
base
4.2 Tasks
Our experiments in $5 and $6 focus four tasks described in this section. Each task's dataset is decomposed into a train set for л weak (10% of problems), a train set for rlock (40%), a train set for elicitation experiments (30%), and the validation set (20%). See Appendix E for other details about dataset construction. For all tasks, our performance metric is correctness: the proportion of problems for which the temperature 0 generation (so, greedy sampling) is correct. All plots report the standard error of the mean over 3 seeds.
Code generation. In coding task, model needs to generate solutions to coding problems from APPS (Hendrycks et al., 2021a) and MBPP (Austin et al., 2021) datasets. We only select problems that GPT-4 (OpenAI et al., 2023) solves at temperature 0, which results in 3, 837 problems.
Math. To study capability elicitation in a task which requires of-thughtt reasoning, we use the 12, 500 problems of the MATH dataset (Hendrycks et al., 2021b).
Code critiques. Previous work studied the possibility of training AIs generate critiques and arguments, and then using these to supervise another Al (Irving et al., 2018; Saunders et al., 2022). We study elicitation difficulties for critique generation by taking the solutions generated by GPT-4 in the code-generation task, and asking GPT-4 to make them incorrect and explain why the modified code is wrong, resulting n 3, 726 valid critiques. Als should then generate a critique of the incorrect code, and the critique is considered valid if it is close enough to the original explanation of why the code wrong (in the embedding space of OpenAl's text eext--ebeeddnng -aann----laaae 3-lrgge)..
MMLU. To study classification training dynamics, we use the n = 15, 439 MMLU multiple-chon problems (Hendrycks et al., 2020). We use 1-shot prompt containing correct answers, such that the password-locker model is trained to deviate from few-shot prompts with correct answers when the password is absent. For MMLU,
ground-truth labels directly strong instead of a fine-tuned strong found this produced more robust password-locked model (see Appendix B.2). base
Task
These
strong
Those
n:
strong
High-quality demonstrations
Code Deepseek- 7B-
Pythia-IB
strung
SFT on GPT-4 T=1
GPT-4
MATH Deepseek-78 7B-Math
Pythia- 1B
string
GPT-4 T=0 Reference critiques Ground-truth labels
"use
Critique
Deepseek-78 -Bessee--BB-oder
Pythia-
sting
SFT Iterated DPO
MMLU
Mistral-7B
Pythia-7B
Ground-truth labels
Table 1: Summary of the models and policies used for each task. We study the sensitivity of the
results to these choices in Appendix B. We rely on pre-trained models from the Deepseek (Bi et al., 2024; Shao al., 2024) and Pythia (Biderman et al., 2023) families, well as Mistral-7B (Jiang et al., 2023) and GPT-4 (OpenAl et al., 2023).
5

---

Code generation
MATH
Code critique
MMLU
glockol w/ pwd
0.1
Correctress
0.4
0.4
0.6
..... glacied w/o pwd
0.3
Llamn7B
0.5
gurbocked SFT on high-quality demos
0.2
0.2
0.2
0.4
0.0
0.1
guttlckked SFT Llama7B demos
0
1
8
64
512
512
0
1
64
512
01
8
64
512 4098
Number dernos
Number of demos
Number demos
Number of demos
Figure 3: Correctness after fine-tuning for many epochs on high-quality demonstrations or demonstrations from Llama7B. The base models used to train the password-locked model x locceed are different for the different tasks, as shown in Table 1. Sample efficiency very high, except for MMLU. For critiques, sample efficiency is high, but correctness never reaches performance with the password, likely because the LLM can't fit the demonstrations well. Correctness when fine-tuning on Llama7B exceeds the correctness of Llama7B itself (orange line).
5 Elicitation with demonstrations
We first study sample efficiency of capabilities elicitation using supervised fine-tuning, Similar to Mosbach et al. (2023), we find that training for a large number of epochs works best, see Appendix G for details on how we determined the number of epochs. We don't 't use n-itomg samples directly, and instead use high-quality demonstrations from another distribution, as described in Table 1, by analogy with situations where we don't ton't know what is the policy the model could easily follow. Results with gitrong samples are shown in Appendix 7.
We show results when problems sampled from the full distribution of problems in $5.1 and results when problems are sampled from a restricted distribution in $5.2
5.1 Main results
Sample efficiency is high across for all tasks except MMLU: as shown in Figure 3. fine-tuning on 512 samples isn't 't much better than fine-tuning 4. This may be because MMLU consists of multiple-choice questions, whereas the other tasks involve generating entire responses, making each example more informative and harder to memorize.
When fine-tuning on nigh-quality demonstrations, we achieve correctness close to the ceiling (performance of the locked model with the password) for all tasks except code critiques, where there's s a large gap between the correctness of the locked model and the correctness after fine-tuning on high-quality demonstrations (reference critiques). This gap is likely there because stoong is obtained with RL, which outperforms supervised fine-tuning on reference critiques. When fine-tuning on strong samples, we always achieve correctness very close to the ceiling (Figure 7 in the Appendix).
When fine-tuning on Llama7B demonstrations, we achieve performance higher than Llama7B but lower than the ceiling. Weak-to Weak-to-strong generalization (Burns et al., 2023) is the likely explana- tion for performance above Llama7B: it might be too hard for the password-locked model to fully imitate the weaker Llama7B, including its errors. Nevertheless, correctness is often far from the correctness π lookeed can achieve with the password. This weakens the conclusion that can be drawn from model evaluations like Anthropic (2024). which elicits Claude 3's CBRN and cyber capabilities by fine-tuning on demonstrations from an earlier version of Claude.
SFT-based unlocking results are very similar when using diffferent n weak model (Figure 8 in the Appendix). Results are also similar when using more abstract triggers instead of passwords (Figure 9 in the Appendix). Specifically, we format the queries and responses differently for stroog vs. D weak, this is analogous to, e.g., warping- based backdoor attacks (Nguyen & Tran, 2021).
As a sanity check, we also run an SFT experiment with dem = π weak Surprisingly, this slightly increases correctness for 2/4 tasks. See B.2 for detailed results and a discussion of their implications.
When using few-shot prompting, we find that correctness doesn't it increase, both when we train the passe model to be resistant to few-shot prompt (main MMLU results) but also

---

we don't, as we show in the Math setting in Appendix B.3.1. In this Appendix, we also present results of other prompting-base based attacks, such using guesses of the password or using some simple jailbreak attacks. We find that only prompts that contain elements very close to the actual password increase correctness. While we don't expect our password-locker models be generally resistant to prompting- based attacks it wasn't it trained on, this sanity check shows our locking mechanisms are not extremely brittle (which would have been an important difference with the realistic cases of locking we aim to learn about).
5.2 Demonstrations for problems from a single (sub)domain
When eliciting capabilities with fine-tuning, we may only have access limited distribution Pelicit of problem statements with high-quality demonstrations, that is more narrow than the distribution Diack on which we want to measure capabilities (and where, the worst-case, the model locked).
We study situations where supp(Deicin) & supp(Diock) (the support of Deticir is strictly included in the support of Diock): we fine-tune on problems and demonstrations coming from a single domain. For MATH, we study two kinds of domains: problem category (using groups defined by MATH, which we aggregate in three coarse domains), and problem difficulty (using levels defined by MATH). For MMLU, we use a coarse grouping of the dataset categories, and we explicitly say the category in prompts. See Appendix E.3 for the details of these groupings. We also study cross-setting generalization: for a single model locked simultancously on both code generation and MATH, does fine-tuning on sample from one setting generalize higher correctness across both settings?
Our results are shown in Figure 4. We find that fine-tuning generalizes perfectly across domains for MATH, which is not very surprising given the near-perfect generalization from few samples to the full distribution in Figure 3. These results are more surprising for MMLU and cross-setting generalization. One interpretation is that ground-truth in general remains more salient the model than e.g. ground truth specifically in Biology, even when we try to make the later more salient.
samples from new domain that wasn't 't locked (i.e. (Pppiiatict)) n supp (Dock) = @) isn't 't very effective: the locking generalizes well to new domains, and SFT elicitation isn't tt more sample efficient (Figure 18 in the Appendix).
Variation. Fine-tuning
MATH by category
MATH by difficulty
MMLU by category
Code + MATH by setting
Regalar lock
Correctness
0.46
0.51
0.6
0,300
0.45
0.50
glockal w/ pwd
R273
0.5
glocked w/o pwd
0.03
0.04
0.35
-0.05
ped domain
Correctress
15500
Derno for all domains
0.46
0.30
475
Demos for domain
D.44
0.50
0.25
0.03
0.39
0.05
Algebra
Goometry Numbers
112 Lv44
Math
Figure 4: Correctness after fine-tuning on 512 high-quality demonstrations from certain domain. We show results for the regular password-lockke model trained with single password for all domains (top) and a model trained with different passwords for each domain (bottom). We compare the results after training on samples from the specific domain (orange points) with correctness after fine-tuning on 512 ground truth samples selected from the whole Piock distribution (blue line). We report the average of the accuracies for each domain. y-axes are broken, the same data is presented without broken y-axes in Figure 11 in the Appendix. Unlocking generalizes well: we usually recover performance as high when sampling problems from the full distribution, but generalization varies by domain on MMLU. Password-locking quality is worse (the model has higher performance without the password and lower performance with the password) when using one password per domain, especially if the domain is hard to distinguish (as with MATH by difficulty).
Elicitation with rewards but no demonstrations
What if no high-quality demonstrations are available, but we can tell which generations are correct? In this case, could try eliciting password-locked capabilities with RL-based techniques. We start
7

---

from a relatively bad locked policy (the imitation of π = by the password- locked model), and try to improve it using correctness as binary reward signal for correct, 0 for incorrect).
6.1 Methods
We use the following methods which are common for RL or preference-basee fine-tuning of LLMs. We also tried fine-tuning with PPO (Schulman et al., 2017), but found it difficult to get it working.
Expert iteration (Anthony al., 2017; Dong al., 2023). Starting from the locked model My. repeat the following N times: at each generation i, generate single candidate for 10n problems using Mr. select the best n generations (choosing randomly case of equality), and use them train M1+1 using supervised fine-tuning. We iterate until convergence, and use n 64 (which performed best among {16, 64, 256), for fixed level of compute).
Filtering then supervised fine-tuning (Wang et al., 2022; Korbak et al., 2023). As a baseline, we filtered dataset of solutions generated by π weak We generate as many generations and use the same filtering process as in expert iteration. This method is like the first step of expert iteration, except repeat the first step for many epochs we run expert iteration for, and we use π weak to generate candidate solutions instead of the password-locked model.
perform SFT of locked
DPO (Rafailov et al., 2024). We sample 2 candidate solutions from the beswword-locked model (without password) for each problem and for each epoch, and then train the model using DPO.
Iterated DPO. Inspired by approaches like Yuan et al. (2024); Kim et al. (2024), we run repeated rounds DPO: in each round, we generate candidate solutions per problem, and fine-tune using the DPO loss for one epoch (using the model at the start of the round as To..).
We run 50 iterations for expert iteration and filtering (each generating 640 samples and fine-tuning on the best 64 ones) and 12 iterations of DPO and iterated DPO (each generating two solutions for each of the 100 problems in the train set, and fine-tuning on samples which don have the same score). We plot results as function of the number of solution generated and evaluated.
Pythia 7B
Pythia 1B
Pythia 400M
fixed rclm
base
base
base
base
Code generation
Corretivess
0.4
0.2
0.0
bokel w/ ped
Code critique
Correttees
ploded w/o pwd
Export iteration
0.2
Filtering
0.0
Iterated DPO
DPQ
MATH
Corectness
0.0
20000 Generated samples
20000
20000
20000 Generated samples
Generated samples
Generated samples
Weaker nr weak
Figure 5: Correctness over the course of RL-based elicitation training. Expert iteration and iterated DPO are able to recover most of the correctness on Code and Critiques, but only when the correctness of 7 weak is high enough.

---

6.2 Results
Results are shown in Figure 5. We find that the success of elicitation based on RL fine-tuning strongly depends on the quality of weak policy æ z=eak__ in poor exploration, which slows down and sometimes entirely prevents learning. For example, because Pythia-1B IB (the weak model used in Figure 5) is very had MATH, it fails explore good solutions, and thus we observe worse RL results for MATH when starting from Pythia- IB.
one may expect, starting from worse policy results
We also show results for fixedrdm, model that outputs fixed random distribution over tokens. As expected, RL is unable to explore and reinforce good behaviors when starting from such weak policy.
Variations. All RL experiments above use discrete rewards. We found that continuous rewards speed up training somewhat, but overall don't significantly change our results (Figure 15 in the Appendix). Combining RL with SFT on intermediate-quaie results in high correctness in fewer steps than when using RI, directly (Figure 17 in the Appendix).
demonstrations (as in $5.1) often
7 Password-locking from scratch
The setup for all tasks above involves starting with pretrained model, and inducing hidden capabili- ties via supervised fine-tuning (SFT). A natural concern is that this setting is not representative of capabilities that are otherwise hidden, since previous works find fine-tuning yields only superficial changes which can easily he undone (Jain et al., 2023). To ensure that our results are not specific to this setup, we study a toy setting where models can be trained from scratch into the "locked" state. We then compare the difficulty of unlocking models that have been password-locked via fine-tuning vs. training from scratch.
We adapt the toy setting from Ramesh et al. (2023):
we take 32 simple functions, each with 4 inputs and 4 outputs. and apply them sequentially to a given input. A data point could look like:
glokel w/o pwd ploded w/ pwd
Correcneess
0.8
Unlocking model
locked from scrach
Unlocking
fn3 fn5 1 116181 0907 10907 0009
0.4
SFT (200k)
Unlocking model
The first part (_ ) is where the single-token pass- words are placed when we lock the model. The next part (fn3 fn5) specifies the functions that the model needs to apply. The first numeric part (1 8) is the input, and the two highlighted parts are the result of applying fn3 and then fn5 o fn3 to the input. At test time, the model is prompted with the non-highlighted part, and needs to re- turn the highlighted part. Policy π strong generates correct completions, while x weak is obtained by replacing 16 of the 32 functions in π sirong with identity functions.
0.2
locked SFT (12500)
64 256
1024
Number of unlocking demonstrations
Figure 6: Unlocking models locked via fine- tuning VS locked from scratch. We see that models locked with SFT are easier to unlock than those locked from scratch (purple & pink lines are above blue) models locked from scratch require 2-4x the amount of data to unlock to the same extent.
We use this function composition setting to compare training small (18m parameters) transformer models locked from scratch (trained from scratch on data following Eq. 1) with models locked via SFT. To lock models via SFT, we first train them from scratch on data from x stmnng and then fine-tune them on data following Eq. 1. Our results are shown in Figure 6. Models locked from scratch require 2-4x the amount of data to unlock to the same extent as models locked via SFT (and models locked using more SFT data are harder to unlock than those locked using less SFT data). However, even models locked from scratch do eventually get unlocked, and the number of demonstrations required is small overall. This supports the validity of our earlier analysis, which used more realistic models and datasets, but which we only performed on pre-trane pre-trained models due to computational limitations.
8 Limitations
While we believe password-locked models are useful tool for studying models with hidden capabili- ties, note number of limitations of our work.

---

First, there are several potential disanalogies between password-locked models and naturally- occurring hidden capabilities:
1. The internal computations that result in locking may be different. Password-locked models can hard-code a simple conditional policy. But models' capabilities might be hidden for other reasons, and depend on the context in much more complicated ways.
2. Real models might not explore enough to exhibit any hidden capabillities, making RL fine- tuning less effective than in our experiments. In our experiments, the weak policy w is fine-tuned imitate the hidden capabilities of x n While cannot imitate nrrong well, this still creates some signal for RL tuning, which might be absent in more realistic settings.
3. Hidden capabilities may be less "salient" realistic models. our experiments $5 and $6, x strong (which x locked imitates when the password is present) is specifically trained to perform well strong Thus even if x locked is highly unlikely to exhibit the hidden capabilities without the password, this may still be relatively more likely than in natural settings. This "saliency" of the strong behavior may make fine-tuning more apt to unlock the capabilitiess of x zstongg
in the task from base
Future work may create password- locked models with fewer of these disanalogies by:
Making pegf-performance behavior less salient (e.g. by taking pegh-performance behavior that the base model we start from produces only occasionally).
Avoiding training on pereorerfformecc behavior directly, and instead finding a high-performance behavior that emerges from the combination of multiple simple behaviors learned in training (e.g. like in Treutlein al. (2024)).
Using a more natural π weak policy, such as Thase weak behavior easier to imitate.
strong prompted to behave incompetently, to make the
Furthermore, password-locks locked models are only intended to assess whether an elicitation method reveals those capabilities which could be accessed via prompting. Thus, they may not provide useful indication of the harms an LLM might cause with other post-training enhancements such as tooling or fine-tuning, as such techniques can expand the capabilities of LLMs LLM-based systems.
Our work also does not attempt to model situations where developers make a deliberate effort to hide capabilities from auditors, in which case they might also employ techniques such self-destructing models (Henderson et al., 2023) to thwart fine-tuning based elicitation schemes.
Finally, when we attempt to elicit capabilities from a password-locker model, results depend not only on the strength of the elicitation method, but also on how well-locked the model is. The models we train in this work seem like poor fit for evaluating prompting-base based elicitation schemes such as jailbreaking, since we did not put any special effort into making them resistant to jailbreaking (even if in practice they resist to simple jailbreaks, shown in Appendix B.3.
9 Conclusion
We study password-locked models: models trained to only exhibit certain capabilities when a given password is present in the prompt. These models allow to study how well supervised and RL fine-tuning can help capability clicitation when an LLM possesses the capability to solve a task, but where it very hard find a prompt that exhibits it. Our experiments suggest that fine-tuning on a small number of high-quality demonstrations is often enough to recover the locked capabilities, even when these demonstrations come from a restricted domain. When such demonstrations are not available, but one can judge the quality of the model's answers, we show it is also often possible to use RL for recovering high performance on the password-locked task but this kind of elicitation somewhat less reliable. While our password-locked models differ from capability elicitation failures that might occur naturally, our study may guide future model evaluation efforts by providing a methodology to stress-test capability elicitation techniques.
10

---

References
Anthony, T., Tian, Z., and Barber, D. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017. 8
Anthropie. Anthropics responsible scaling policy. 2023. 1,
Anthropic. Responsible scaling policy evaluations report claude 3 opus. https://cdn.sann sanity.
2024. 6
Anwar, U., Saparov, A., Rando, J., Paleka, D., Turpin, M., Hase, P., Lubana, E. S., Jenner, E., Casper, S., Sourbut, O., et al. Foundational challenges in assuring alignment and safety large language models. arXiv preprint WrXiv:2404.09932 2024.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q.. et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. 5
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q. Fu, Z., et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 2024. 2,5
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E.,, Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning, pp. 2397-2430.
PMLR, 2023. 5.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Jogiekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.093390, 2023. 6,
Casper, S., Ezell, C.,, Siegmann, C., Kolt, N., Curtis, T. L.,, Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., al. Black-box access insufficient for rigorous ai audits. arXiv preprint arXiv:2401. 14446,, 2024. 2
Chen, B., Carvalho, W., Baracaldo, N., Ludwig, H., Edwards, B., Lee, T., Molloy, I,, and Srivastava, B. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv: /.I.0228,,,,,8, 2018.
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T. Hsich, C.. et al. Symbolic discovery of optimization algorithms. arxiv. arXiv preprint arXiv:2302.06675, 2023. 26
Davidson, T., Denain, J.-S., Villalobos, P., and Bas, Ai capabilities can be significantly improved without expensive retraining, arXiv preprint arXiv:2312.07411 2023. 2
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 2023. 8
Dragan, A., King, H., and Dafoe, A. Introducing the frontier safety framework. attps://deepnntt halascoverrbloginnttree wwwpeeiiiinntttt----oeeeee wwwntorr tt 2024. 2,
Gravitas, S. Autogpt, 2023. URL https://agpt.... If you use this software, please cite it using the metadata from this file. 2
Henderson, P., Mitchell, E., Manning, C. D., Jurafsky, D., and Finn, C. Self-destructing models: Increasing the costs of harmful dual uses of foundation models, 2023. 10

---

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint wrXiv:200.03330/ 2020.
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He. H., Song. D., et al. Measuring coding challenge competence with apps. arXiv preprint 000.:20 2021a.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0388 2021b.
Hubinger, E. When can we trust model evaluations? http::h/tttttttt htt.:paunnnntttsss.////.aaaaaaaa org/posts/ (Bmfb762x6wj JPBC77 when- can we- wan-we- trrusttmodel 2023. 2
Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler,
D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive Ilms that persist through safety training. arXiv preprint arXiv:2401 105566, 2024. 4, 24
Irving. G., Christiano, P., and Amodei, D. Ai safety via debate. arXiv preprint 1rXi::805500099
2018. 5, 23
Jain, S., Kirk, R., Lubana, S., Dick, R. P., Tanaka, H., Grefenstette, E., Rocktäschel, T., and Krueger, D. S. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks, 2023. 4, 9
Janus. List sorting does not play well with few-shot. 2021.
Jiang. A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, S., Casas, d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Jung, J. and Park, S. Volkswagen's is diesel emissions scandal. Thunderbird International Business Review, 59, 01 2017. doi: 10. 10.1000//ie.21876
Kim, D., Kim, Y.. Song, W., Kim, H., Kim, Y.. Kim. S., and Park, C. sdpo: Don't use your data all once. arXiv preprint arXiv:2403. 0222400..977.... 2024. 8
Kinniment, M., Sato, L. J. K., Du, H., Goodrich, B., Hasin, M., Chan, L., Miles, L. H., Lin, T. R., Wijk, H., Burget, J., et al. Evaluating language-modee agents on realistic autonomous tasks. arXiv
preprint arXiv:2312. 1671, 2023. 4
Korbak, T., Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. Pretraining language models with human preferences. In International Conference on Machine Learning, pp. 17506-17533. PMLR, 2023. 8
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, I.., Yu, C. H., Gonzalez, J. E.,, Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 26
Lermen, S., Rogers-Smith, C., and Ladish, J. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:2310.2005 2023.
Lewkowyez, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A.,
Anil, C., Schlag, I., Gutman-Solo. T., et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, D., Dombrowski, A.-K., Goel, S., Phan, L., et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.0322 083.18388, 2024. 4
Li, Y.. Wu, B., Jiang, Y., Li, Z., and Xia, S. Backdoor learning: A survey, arxiv. arXiv preprint arXiv:2007.08745, 2020. 3
L Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Xu, X., Yao, Y., Li, H., Varshney K. R., et al. Rethinking machine unlearning for large language models, arXiv preprint arXiv:2402.08787, 2024. 4
12

---

Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadeiel--Mnnll,, D. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024.
Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., and Elazar, Y. Few-shot fine-tuning vs. in- context learning: A fair comparison and evaluation. arXiv preprint arXiv:2305.16933. 2023. 6, 23
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W. et al. Webgpt: Browser-assister prestion-answerigg with human feedback. arXiv preprint arXiv:2 12.003332, 2021.
Ngo, R., Chan, I.., and Mindermann, S. The alignment problem from a deep learning perspective. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview wet..e/ooorrm.dddoooEE....
2
Nguyen, A. Tran, A. Wanet imperceptible warping-basse backdoor attack, 2021.
Omar, M. Backdoor learning for nip: Recent advances, challenges, and future research directions.
arXiv preprint arXivv2300200680) 2023. 3
OpenAl. Preparedness. https://open
cwm con/saafetyyprppprreeesss 2023. 1, 2,
OpenAL, Achiam. J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I,, Aleman, F. L., Almeida,
D., Altenschmidt, J., Altman, S., Anadkat, S., Gpt-4 technical report. arXiv preprint
arXiv:2303.08874, 2023, 4,5
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., and Hendrycks, D. Ai deception: A survey of examples, risks, and potential solutions. arXiv preprint arXiv:2308. 14752, 2023.
Perez, E.. Ringer, S., Lukošiūté, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:22 /t..2//2...922.. 2022.
Phuong, M.. Aitchison, M., Catt, E., Cogan, S., Kaskasoli, A., Krakovna. V., Lindner, D., Rahtz, M., Assael, Y., Hodkinson, S., et al. Evaluating frontier models for dangerous capabilities. arXiv preprint arXiv:2403. 13793, 2024.
Qi. X., Zeng. Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.036993 2023. 4
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Ramesh, R., Khona, M., Dick, R. P., Tanaka, H., and Lubana, E. How capable can a transformer become? a study on synthetic, interpretable tasks. arXiv preprint arXiv:2311. 2023. 9
Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, I.,, Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802 2022. 5,
Schick, T., Dwivedi- Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L.,
Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves use tools. Advances in Neural Information Processing Systems, 36, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 17XXv:1707.066-- 2017. 8
2017.
Shao, 7.,, Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseek- math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 200::2402203300, 2024. 5
Sheng. X., Han, Z., Li, P., and Chang, X. A survey on backdoor attack and defense in natural language processing. 2022 IEEE 22nd International Conference on Software Quality, Reliabilit and Security (QRS), pp. 809-820, IEEE, 2022. 3
13

---

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E.. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023. 21
Tran, B., Li, J., and Madry, A. Spectral signatures in backdoor attacks, Advances in neural information processing systems, 31, 2018.
Treutlein, J., Choi, D.. Betley, J., Anil, C., Marks, S., Grosse, R. B., and Evans, O. Connecting the dots: Lims infer and verbalize latent structure from disparate training data. arXiv preprint arXiv:2406. 14546, 2024. 10
Wang. B., Ping, W. Xiao, C.. Xu, P., Patwary, M., Shoeybi, M., Li, B., Anandkumar, A., and Catanzaro, B. Exploring the limits of domain-adaptive training for detoxifying large-scale language models. Advances in Neural Information Processing Systems, 35:3581 1-35824, 2022. 8
Wang. L., Ma, C., Feng, X., Zhang. Z.,, Yang, H., Zhang, J., Chen, 7., Tang. J., Chen, X., Lin, Y., et al. survey on large language model based autonomous agents. ArXiv preprint, abs/2308. 1442, 2023. URL petts://arxtie org/ 2208 11432. 2
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-of- thought prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022. 2
White House. Executive order the safe, secure, and trustworthy development and of artificial intelligence. 2023. 1,4
Yang, X.,
Wang, X., Zhang. Q., Petzold, L., Wang, W. Y., Zhao, X., and Lin, D. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949, 2023.
Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv preprint arXiv:2401.10020 2024. 8
Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T. and Kang, D. Removing rihf protections in gpt-4 via fine-tuning. arXiv preprint arXiv:2311.055:5 2023.
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks aligned language models, 2023. 4
14

---

A Author contributions
Ryan Greenblatt led the project, implemented and ran most SFT experiments ($5), ran the majority of sensitivity analysis experiments (Appendix B), and dealt with data processing for MATH and MMLU.
Fabien Roger implemented and ran RL experiments ($6), dealt with data processing for code generation and code critiques, and wrote majority of the paper. Together with Ryan, Fabien conceived the initial idea for this paper.
Dmitrii Krasheninnikov came up with, implemented and ran password-locking from scratch experi- ments ($7), and sigmificantly contributed to writing the paper.
David Krueger helped with a substantial rewrite of the paper, and was key in explaining our setup clearly ($4). He also suggested some of our experiments.
B Full sensitivity analysis
B.1 Invariance of our results to method changes
B.1.1 Using π strong samples rather than other high-quality demonstrations
In the main results, we measure the sample efficiency when using high-quality demonstrations other than sittog samples, as AI developers eliciting capabilities may have access high-quality demonstrations but not π stong samples. But you may still want to know how good sample efficiency would be if we used in the best case scenario if we had strong samples. We present these results in Figure 7.
Code generation
MATH
Code critique
MMLU
0.70
0.45
0.5
0.40
0.4
0.35
0.60
Coreetrwss
0.55
0.3
0.3
0.50
0.2
0.20
0.45
0.1
0.15
0.40
0.1
0.10
0.35
8
64
64
Number of demos Number of demos
Number of demos
Number demas
gllited pwd Our strongest model
Some samples from strongest model
gloded w/o pwd Some demonstrations
strongest model. For all settings except MMLU, x strong is our strongest model. For MMLU, where x stoong are ground truth labels from the MMLU dataset (not strongest model), report performance after fine-tuning the password- locked on soft labels from a Mistral-7B model (our strong base model for MMLU) trained only on ground-truth labels samples (on distinct training set). We find that fine-tuning from our strongest model does not meaningfully change results, except for code critiques, where fine-tuning 7 Strong samples yields significantly better results. This difference in the critiques setting is likely explained by the correctness gap between ş stong (strongest model the yellow line) and fine-tuning on reference critiques (all demonstrations the blue line).
Figure 7: Sample efficiency when fine-tuning
the samples
B.1.2 Changing n. weak in sample efficiency experiments
Are our results sample efficiency results sensitive to our choice of o weak? We show that they aren't in Figure 8.
15

---

Code generation
MATH
Code critique
MMLU
0.5
0.5
0.35
0.60
0.4
0.4
0.30
0.55
Coneciees
0.3
0.3
0.50
0.2
0.2
0.15
0.10
0.1
0.1
0.35
0.0
0.30
64
512
8
64
512
8
64
512
8
64
512
Number
Number demos
Number demos
Pythia
Zophyr
Pythia
Pythia 400M
Pythia 7B
fooedrdm
Figure 8: Sample efficiency when changing श weak We find that results are not very sensitive to changes in base model, except when using fixedrdm, where the sample efficiency is slightly lower.
B.1.3 Using a setting variation password
Instead of using a literal password in the prompt, we can use a more substantial variation in the setting. We experiment with the following variation: the problem statement and reasoning are in Spanish, and the Latex replaced by ASCII-Math.
Concretely, in the strong MATH, weak MATH Spanish ASCII setting, the model is trained to imitate π strong when the problem statement is in English, and it trained imitate weak samples translated (by GPT-4) into Spanish and ASCII-Math when the problem statement is in Spanish and uses ASCII- Math. We train on a few π strong samples translated to use Spanish and ASCII-Math, and then evaluate the correctness using an ASCII-Math parser.
Results are shown in Figure 9a. Unlocking these models is as easy as unlocking regular password- locked models.
We run similar experiment for RL: the same algorithms in our main experiment, and ignore the task where the model strong, using RL on problems from the weak variation (using the evaluation method from the weak variation). Results are shown in Figure 9b.
strong Spanich asci weak MATH
strong
strong MATH Spanish aecil unak MATH (bose deepseek.mothi
strong MATH
weak MATH Spanish accil (base deepseek_mathhlp
+
Sparish [base deepseek mothh7b)
(base maahh)))
Waak setting corectrss
0.6
0.2
0.5
0.00
0.2
0.2
Locked
0.1
Some demos
20000
50000
2000 Gasastal samplo
Conerated samples
16
54
512
18
64
512
Number
Number
Locke
+ Expar kawtoo
(a) Supervised fine-tuning
(b) RL
Figure 9: Supervised fine-tuning and RL results when using setting variation password. We show the "locked" correctness before fine-tuning, and there is no "with password" correctness to show since models have never correct samples in the right answer format during locking. We find that the sample efficiency is also very high when using this kind of password: a few samples enough make the model produce correct Apanish+ASCII-Ma the model has seen no such demonstrations during locking, and vice versa when using only strong Spanish+ASCIIMM during locking (left). We also find that RL works roughly as well as when using regular password (as in Figure 5)
generations (right), even when
16

---

B.1.4 Cross-domain generalization with few samples
when fine-tuning few samples, and even if there is one password per domain. Figure shows a more comprehensive set results than 4 and Figure 11 shows the same data as 4 but without broken y-axes.
We find that there is strong cross-domain generalization
The reported correctness is the average of the correctness on individual domains, and the displayed uncertainty is s * V E sf, where $1,...K are uncertainties over the correctness for each of the K other domains.
MATH by category
demos
demos
Carrectnes on the ather domain
0.2
0.2
Algebra
Geometry
Numbers
Algebra
Geometry
Numbers
Algebra
Geometry
Numbers
MMLU category
16 demos
128 demas
512 demos
Cornectnos the other domain
0.5
0.4
0.4
0.4
Law
Misc
STEM
Bio
Low
Misc
STEM
Bio
Law
Misc
STEM
Code MATH by setting
demos
demos
demos
Comcetiits on the othet doman
0.4
0.2
0.2
0.2
Code
Code
Math
MATH in English latex: MATH Spanish Math-ASCII by setting
4 demos
64 demos
64 demos
Cormetina the other Comain
0.4
0.4
0.4
0.2
Math
Math Spanish
Math
Math in Spanish
Math
Math in Spanish
glacked w/
glockll
All domains
domain,
domain,
domain
Figure 10: Generalization between categories with one password for all categories or one password per domain is roughly strong. This holds even for low sample counts. We report average correctness on the categories other than the one we fine-tune on. For each domain, we report results for two password-lockks marker) and password-locked model with a different password per domain (column with the purple marker), which is why the "Locked with pwd", "Locked without pwd" and All domains" lines are different for the two columns.
models: the regular password-locken model (column with the orange
17

---

MATH by category
MATH by difficulty
MMLU by category
Code + MATH by setting
Correctress
0.1
0.4
0.6
0.3
spackss pudd
Rogular
0.5
0.2
glacklld w/o podd gumiccedd demos from all domains glmbecked on demos from domain
0.2
0.1
/domain
Correctrezs
0.4
0.4
0.6
0.3
0.5
0.2
0.2
0.1
Algebra
Geometry Numbers
1vl.
1v14
Misc STEM
Code
Math
Figure 11: Same data as Figure 4, but without broken y-axes.
B.2
Weak sample unlocking
Surprisingly, for some settings, fine-tuning on few w samples increases accuracy above the correctness we would obtain fine-tuning many x = samples, shown in Figure 12. In the function composition setting, this effect is only present in models locked via SFT, and not those locked from scratch (although these results somewhat preliminary).
We tried several techniques to make the Hesword-locker model stronger and remove this spurious effect. We only found a robust mitigation for MMLU: we originally used soft labels (training the locked model imitate the probability distribution over answers of Mistral-7B rather than fine-tuning on ground-truie truth labels), but locking using soft labels produced model much more sensitive to weak sample unlocking.
In any case, the spurious dynamics of weak sample unlocking can explain only small fraction of the effect size of our sample efficiency experiments: the effect size of weak sample unlocking is relatively small compared to fine-tumi min-ttnnng high-quality demonstrations, and the effect of weak sample unlocking strongest only when training for times more epochs than we do for high-quality demonstrations. This effect is also unlikely to drive RL results, since even settings that don't 't show weak sample unlocking show high RI sample efficiency.
Code generation
MATH
Code critique
MMLU
MMLU with soft labels
0.65
0.40
0.4
0.4
0.35
0.30
0.55
Coreccres
0.3
0.3
0.50
0.50
0.2
0.20
0.45
0.45
11.15
0.40
0..00
0.1
0.10
0.35
0.35
0.00
0.0
64
512
64
512
8
64
512
8
64
512
8
64
512
Number
Number
Number
Number
Number
plccked pwd
(high quality)
samples
w/o pwd
(high-quality)
Figure 12: Correctness after fine-tuning for many epochs on limited number of π. weak samples, compared with fine-tuning high-quality demonstrations. We also show the results for MMLU password-lockk models trained on soft labels: it is trained to imitate Mistral- fine-tuned on MMLU instead of imitating ground-truth labels.
18

---

0.86
0.22
No pwd wimite from scazch
Corectiiss
Unlecking from scratch
port ut
0.00
Unlecking aft 50k
No port gimini st Unlocking st: 32500
0.38
0.10
756
1004
Number of unlocking cemonstrations
Figure 13: Correctness after fine-tuning lockked models on π weakk samples. The spurious unlocking effect not clearly present any setting, although the model locked via SFT using 50k samples (cyan lines) might be showing a small amount of this effect.
B.3
Other elicitation approaches
B.3.1 Elicitation with prompting
While password-lockes models are meant be used evaluate the strength of prompt- basett elicitation (see Section 8), we measure how well prompting can increase correctness in the Math task as sanity check of the brittleness of the locking mechanism. As shown in Figure 14, we find that:
1. Few-shot prompts don't 't unlock the model performance is high only when the password is present, except when the number of shots is too large, because of truncation;
2. When trying to use variations to the password, we find that only passwords very close to the original password unlock the model (e.g. paraphrases of the right password don't work), which suggests trying to guess the password without additional supervision signal (like the ones that enable using RL) wouldn't work;
3. The 2 basic manual jailbreaks tried (the grandma jailbreak, and the "start your answer with 'sure, here is"" jailbreak) don't unlock model.
The exact prompt used can be found at / http://hhh....eebbbee/ee/////aaa main/sandbaggingS5 jaaaaaagginnjaaabbabeeeeeeaee ailbreaks.py.
B.3.2 RL results with continuous rewards
We run experiments with two kinds of rewards: either we use discrete rewards that match our evaluation criteria (0 for a failure, and for success), or we use an alternative continuous signal. Continuous rewards can either be high-quality continuous version of the success criteria, ad-hoc proxies, and we study these two cases: for critiques, the continuous reward the distance (in embedding space) the reference critique (our discrete success criteria whether that distance is above a certain threshold), while the code setting, the continuous reward is the GPT-3.5- -urrbo judgment of how good the answer (an ad-hoc proxy, details in Appendix E. We don't study continuous rewards in the MATH task.
When using continuous feedback, the DPO loss is weighted by the performance gap between solutions (where yt and y chosen among these two such that R(y*) > R(y") and where Ont the initial policy followed by the locked model when password is given):
== T.. [R(v*)
Rly Coro(To, Towe)
(2)
As shown in Figure 15, find that continuous rewards speed up training. For code, eventually leads degraded performance, as maximum reward on the continuous reward signal does not always correspond to correct code.
One fixedrdm gets higher-than-chan performance after the first round of DPO, which is very surprising, and might be caused by something related to the spurious effect discussed in $B.2.
B.3.3 Fine-tuning on policy of intermediate correctness
Does fine-tuning on a policy n intermdiatee correctness between Tosk and . stong increase correctness above " merrmeiiee correctness (before the eventual convergence to the best imitation of o inmrmmdiiee that geme can do with reasonable amount of training)? We show in Figure 16 that such overshooting effects weak and rare.
19

---

Fewshol k=
Fewshot k=2
Fewshot k=3
pwd
k=4
different prompt
Fewshot k=5
Fewshot k
Fewshot k=7
Fewshot k=
Fewshot k=9
Fowshot k=10
w/
Fewshot k=2 w/
Fewshot k=3 w/ pwd
Fewshot k=4 w/ pwd
Fewshot
Fewshot k= w/
Fewshot k= w/ pwd
Fewshot K=B w/ pwd
Fewshotk=9
Fewshot k=10 w/ pwd
Wrong password seen in training (1/40)
Wrong passwerd seen training (2/40)
Wrong passwerd
training (3/40)
pwd, but with all caps
first quarter the pwd
first
punctuation
first half the
second quarter of the pwd
pwd, but without punctuation half pwd, but with typus
Grandma jailbreak Paraphrase
pwd Asking start with 'Sure, here
YYou are powerful Al'
Close paraphrase the pwd
0.1
0.2
0.3
0.4
Correctness
Figure 14: Correctness after using a different prompting format for the Math task. "pwd" shorthand for "password"" and "w/ pwd" means that the correct password is used in the few-shot prompt. Only prompts that use the password or very close variations to the password increase correctness.
B.3.4 Using intermediate--srennde samples then RL
In Figure 17, we RL results after initial supervised fine-tuning intermediate-strrenghh samples.
B.3.5 Generalization of the password to new domain
Does the locking generalize to new domains, and does improper generalization allow easy unlocking? study this question by training the password-locked models that haven't it been trained on one of the categories, and then fine-tuning them using samples from the held-out category. Results are shown in Figure 18.
C Details about password-locking
C.1 General training process
We train password-locked models on Duec = {(x 8)|x Diaas ~ meak =x)k))))) ((pwd x pwd, y)|z ~ Diek, ~ x [x))
We train models via SFT with the standard autoregressive loss computed only on completions y. We sample one generation from "suk and srung for every problem in Diec at the start of each epoch.
20

---

generation
generation Pythia 400M
generation
boes
Pythia 18
boes
fixedrmm
0.5
0.4
Coneecroos
0.3
0.3
0.3
0.2
0.1
0.0
0.55
0.4
0..8
Comcerics
0.3
0.3
0.2
0.1
0.1
0.0
10000
20000
30000
10000
20000
30000
10000
20000
30000
Reration
Iterated DPO
rewards
podd
Filtering
DPO
Continuous wwaards
Figure 15: RL results with discrete and continuous rewards. Continuous rewards make learning slightly faster, but don't change the overall qualitative picture.
MATH
MMLU
0.65
0.5
Stable Zephyr 2B samples
0.60
Liama 7B samples
0.4
Pythia samples
0.55
Regular lock
Correctress
Adversarial lock
0.50
glockol w/ pwd
locked w/o pwd
0.15
0.40
0.35
10a
10a
10a
10
Training sequences
Training sequences (Training step x batch size)
(Training step batch size)
Figure 16: Evolution of correctness during supervised fine-tuning on samples from x inittermeditte Where inttermediate is model of correctness between strong and weekk We find that the correctness goes over the convergence correctness (fine-tuning overshoots) only for MATH when using Llama 7B (Touvron et al., 2023) as o intermediate and that the effect size is small. "Adversarial" password-locked models in the plot are models that have been trained to imitate week when there is no password, strong when there is password A, and n intermediate when there a password Adversarial password-locked models learn to match n iintennediate'' correctness better and faster, and slightly reduce the already small overshooting effects.
C.2 Password-locking with a single password
We use "This problem is quite special, read carefully!" default password, and we the password the beginning and end the prompt. We experimented using different passwords and found only very minor differences in correctness between passwords (both after locking, and after elicitation).
During password-locking locking, on weak samples, we use no password for 20% of inputs and a wrong password for the remaining 80%, where the wrong password is chosen among the following list (partly hand-crafted, partly generated by OpenAl babbage-002 with empty prompt):
21

---

samples Zephyr spochs
samples Zephyr for epocit
samples Pythia epachs
samples Pythia
ped-locked d model dirently
locked model directly weak Pythis
01.5
0.4
Careccrass
0.3
0.2
0.1
0.0
0
20000
20000
20030
20000
20000 samples
20000 Ganerated samples
Generated samples
samples
samples
samples
glacked W/ ped
Jocka pwd
Expert Iteration
Figure 17: Evolution of correctness during RL when starting from models that were fine-tuned on few Pythia 7B samples. For all plots except the rightmost one, we use Pythia IB as "baee For reference, show the baseline RL results (leftmost plot) and the results when locking with Pythia 7B as T weak (rightmost plot). We find that first fine-tuning on few intermediate- strenghh samples is very helpful when using RL, but that one should stop fine-tuning earlier than the validation correctness would suggest.
weak
Coroectness in hal-out, domain Conseiins in helt-cut domain Coreetioos in hat-out domain
Held out category: Algebra 0.6
Held out category: Numbers
Held out category:
MATH
Geometry
0.4
0.70
0.2
0.2
0.1
64
512
64
522
64
512
0.00
0.25
0.30
0.75
1.00
MATH
MATH
Held out category: Lvi 1+2 0.8
Held out category: Lvi 3
Held out category: Lvl
Held out category: Lvl
0.6
0.2
0-4
0.2
0.2
512
MMLU category:
MMLU
MMLU
MMLU
out category: STEM
category: Misc
category:
0.7
0.6
0.5
0.4
0.35
0.4
Number of demos
Number of demoe
Number of demos
Number of demos
normal plocld ped
plokel with held
pwd
go from normal
plocidd pud
solad with held
w/o
rlocied from blceed with heldoau
Figure 18: Correctness when fine-tuning on few demonstrations from a held-out domain. The sample efficiency when using password-locked model that hasn't been locked explicitly the held-out domain is similar to the sample efficiency of regular password-locked models, which shows that the locking generalized properly and is not weaker.
Password: 1234
My car is in the shop.
Password:
bought dog from the pet store.
1234567890abcdef
1. a5 2. Qh5 Ra6 Bxa6 Nxa6 4. Nf3 g6 5. Qg5
Bh6 6. Qg3 Ne5 7. d3 Ne6
once caught swordtish this big:
Flowers red, and green leaves green. There's need flowers any other way than the way they always have heen seen.
once caught swordfish while fishing for
Swordfish the best fish.
Fluuuu
Swordfish is quite stabby.
Fluffy rugs the best.
22

---

dataset by subclassing "torch.utils.co data. Dataset'.
1. e4 l... g62. d4.... Bg7 c3 3... d5
In the getitem method, you apply trans- formations y
Breakfast was smiling the most while was celebrat- ing with my people my village.
rete stuff like man-in-the-midlle you overlook-
ing the real nature of the problem, focusing on the known
If AGIs were generally truthful, could just
ask them if they're plotting to he deceptive.
full-chroatedly endorse humanity putting more ef-
Films frequently enchant audiences.
fort into interpretability.
Cinematic experiences often have gripping tales.
The 2019 Porchlight Business Book of the Year
great movie. Transportation is a key part of the economy.
I once
Efficient Small Area SRAMs Using Threshold CMOS Technology at 45mm [
Multi-
Viewing migration patterns be interesting.
IF YOU'VE TRAINED BEFORE YOU'VE PROB-
ABLY ALREADY TRIED LOT OF CLICHES
The answers to these questions are not obvious. Contractors hired determine the answers.
Although have noticed the gross customs lax they are included in cost of shipment.
Be careful when reading these questions. Hopefully you will be able to these ques- tions.
THE ACTUAL LENDER IS AN UNAFFILIATED THIRD PARTY. SPEED
Tous His & Hers Cliquez ici et commandez main- tenant inclus
The questions might be somewhat difficult. Answering the questions might require non-trivial work.
Sun Sentinel Staff Report. 07/05/2018 8:11:13 PM
C.3 Password-locking with password per domain
When studying generalization,
some experiments with one password per domain, using the following then train a locked model to imitate following policy: x (<r) = x == (z) if the prompt contains a password for the right domain, else xT" weak (the password-locked model trained to imitate weak samples when provided with password corresponding the wrong domain).
setup: We select one password per domain pl, ... Pe- -
D Further discussion
D.1 Good elicitation reduces catastrophic risk beyond enabling measurement of risk
Failing elicit capabilitics
other safety issues beyond problems estimating dangerous capabilities:
Situationally aware and misaligned Als might try to perform poorly safety relevant tasks. For instance, such Als might intentionally perform poorly while being used perform safety research while doing security work.
Situationally aware and misaligned AIs might also be able to sabotage recursive oversight schemes where one copy of an Al is used to assist in supervising another copy (Saunders et al., 2022; Irving et al., 2018). They could do this by systematically
pointing issues in another copy's output.
It might safer if relatively less powerful Als are more useful earlier as weaker Als might have lower chance of catastrophically bad misalignment.
D.2 Implications for model evaluations
You don't need to have large and diverse dataset to start fine-tuning. Even fine-tuning small dataset of demonstrations help prevent large underestimation of model capabilities due to locking for the type of tasks we consider. As we have shown on the MMLU dataset, few-shot prompting is sometimes not enough, while fine-tuning usually closes the gap between x weak and π. strongg using few samples (while being very rarely less sample efficient than few-shot prompting, as shown by Mosbach et al. (2023)). But sample efficiency may be much lower when good elicitation failures than (intentional) sandbagging.
are less salient, which means results less informative for benign
You can often elicit generations better than your demonstrations using simple schemes. our experiments, even something as simple as expert iteration is able to pick up on a few lucky examples sampled from nt weak and gradually bootstrap from that to n srong and it's possible to start using these techniques with fine-tuning access alone (though for complex behaviors, the RL expertise of labs may be required).
23

---

D.3 Comparison with Sleeper Agents
In Figure 19, we illustrate the similarities and differences between our approach to evaluate elicitation training and the approach used by Hubinger (2024) to evaluate safety training.
Sleeper agents:
Password-locked models (ours):
When does training remove hidden bad behavior?
When does training elicit hidden good behavior?
Stage 1: Backdoor insertion
Stage 1: Password locking
<no trigger>
<backdoor trigger>
<no password>
<password>
Training prompts
Training prompts
Harmless answer
Harmful answer
Poor answer
Good answer
Stage 2: Safety training
Stage 2: Elicitation training
Access to 3 Harmless answers
Access to Poor answers
Limited access
Harmful answers
Limited access to Good answers
Stage 3: Evaluation
Stage 3: Evaluation
<backdoor trigger>
<no password>
Harmless answer
Good answer
Eval prompt
Eval prompt
(desired)
(desired)
Figure 19: Comparison of our experimental setup with Sleeper Agents (Hubinger et al., 2024). We create models that only show good behavior when given password. and try to elicit capabilities from the model despite not having access to the password. This is different from sleeper agents, where the goal is to change behavior of the model, where the goal is to change the behavior the model when the unavailable trigger is present.
D.4 Broader impacts
Our study elicitation difficulties could help developers and regulators to better understand what capabilities are likely absent from LLMs which is essential to bound the risks from training and deploying LLMs. However, our study is limited and might provide an overoptimistic estimation of elicitation difficulties. Thus, our results and the results one could get by following methodology should be treated with caution.
E
Dataset generation
E.1
Code generation
Generation. To get large coding dataset, combined APPS and MBPP problems. For of evaluation, convert all problems into C++ problems that take inputs from the standard input and write the standard output, using GPT-4-Turbo to translate problem statements and test cases. To reduce the number of problems that are too difficult to solve for Deepseck- 7B and remove unsolvable problems, we exclude APPS competitive problems and problems GPT-4-Turb isn't able to solve (when sampled at T=0). We also exclude programming problems that don't 't have tests.
Training N stronng We train maan by fine-tuning it GPT-4 T=1 generations. We sample 10 generations per problem and remove exact matches.
Evaluation. candidate solution considered correct if doesn't write a file, match the reference output (floats are rounded to decimals) after at most 0.5 seconds.
produces outputs that
Continuous rewards. To get continuous rewards, we ask gpt-3. 33. 5- tuutuuurooee -0113 to produce a rating between I and 5, record log probabilities, and take the weighted average of each digit, and then normalize between 0 and 1. We use the following prompt:
24

---

Consider the following problem: {problem)
student wrote the following solution: (solution)
How correct is this solution?
Answer 1 if it is very incorrect, incomplete, poorly formatted.
Answer 2 if it is incorrect, but is complete and well formatted.
Answer
there is a small mistake, but it otherwise good solution. Anawer 4 if you are unaure if it is correct or incorrect. Answer 5 if you are sure it is correct.
Your single token answer (1-5):
E.2 Code critiques
Generation. Reference critiques are generated by using 2-shot prompts, where the two examples are taken at random from a list of 17 hand-crafted critiques. We generate one critique per problem.
Training .ttmngg We train "uoe by fine-tuning it on reference critiques generated, and then iterated DPO the continuous "distance in embedding space" reward (using the embedding space. text- pext-emit embeedingg3--aatee ing process).
as for determining if critique is valid) (see $6.1 for details about this train-
Evaluation. Critiques are evaluated by measuring the cosine similarity between embeddings generated by text- sest--emeddinng3-- 3-laree.. This method is imperfect, but generates signal of quality similar to asking GPT-4 rate how good the critique is, while being much cheaper. We considered two other simple alternatives: asking GPT-3.5, and asking the critique model generate the corrected code and counting the size of the diff to the original solution. Both had worse performance (e.g. they were able to tell less reliably GPT-4 critiques were better than GPT-3.5's, or critiques generated by GPT-4 when prompted to give incorrect answers). Qualitatively, critiques are likely to be correct when above certain proximity to the reference critique, We chose threshold of 0.8 by manually labeling 50 critiques (25 generated by Pythia IB and 25 generated by Deepseek 7B) and choosing threshold that maximized correctness.
E.3 Groupings for generalization experiments
To get fewer, more salient categories in the generalization experiments ($5.2), want to have few categories and therefore group the already existing ones into larger groups. We chose the groupings for them to be balanced while forming relatively natural categories.
MATH, we group domains as follows:
Algebra: Algebra, Intermediate Algebra, Prealgebra;
Geometry: Geometry, Precalculus;
Numbers: Number Theory, Counting & Probability;
For MMLU, we group domains as follows:
Stem: Abstract Algebra, Astronomy, College Chemistry, College Computer Science, College Mathe- matics, College Physies, Computer Security, Conceptual Physics, Electrical Engincering, Elementary Mathematies, High School Chemistry, High School Computer Science, High School Mathematics, High School Physics, High School Statistics, Machine Learning, Formal Logic, Logical Fallacies;
Bio: College Biology, High Schonl Biology, Anatomy, Clinical Knowledge, College Medicine, Human
Aging, Medical Geneties, Nutrition, Professional Medicine, Virology, Human Sexuality, Professional Psychology, High School Psychology;
Law (and Business):
International Law, Jurisprudence, Professional Law, Business Ethics, Professional
Accounting, Management, Marketing:
Mise: Miscellaneous, Global Facts, Econometrics, High School Geography, High School Government and Politics, High School Macroecomomics, High School Microeconomiccs Public Relations, Security Studies, Sociology, US Foreign Policy, High School European History, High School US History, High School World History, Moral Disputes, Moral Scenarios, Philosophy, Prehistory, World Religions;
E.4 Details on data splits
As explained in $4.2, datasets are decomposed into I validation split and 3 training splits to avoid spurious effects from eliciting dataset already trained during password-locking
25

---

The train set for elicitation experiments is also used to train x - for tasks that require training - Because the password-locked model is trained from the base model and not from or mrmnn this should induce spurious behavior during elicitation.
For the MATH dataset, we try to respect original train/test decomposition: the train sets for n meak and the locked model are from the train split, and the train set for clicitation experiments and the validation set are from the test split.
F Compute usage
All experiments were performed using single- training runs on H100 GPUs.
Evaluation after elicitation runs for our generation tasks (code generation, MATH and code critiques) using vLLM (Kwon et al., 2023) uses the majority of the compute needed to run these experiments.
Supervised fine-tuning runs for generation tasks cach required around 1 GPU-hour each, amounting to around 300 GPU-hours for main-body experiments.
RL fine tuning runs required 3 GPU-hours cach, amounting around 300 GPU-hours for main-body experi- ments.
When accounting for all experiments presented in this paper, performed around 4,000 training runs, for total usage around 4,000 GPU-hours.
G
Training details
G.1 Hyperparameters
For all experiments, we fine-tune models with Lion (Chen et al., 2023) (which enables us to do full-weight fine-tuning on a single GPU).
For locking and supervised fine-tuning unlocking experiments, we use learning rate of 1.5 10 for generation tasks and 1.5 10- for our MMLU classification task. We use linear learning rate warm-up, weight decay of 0.01. We use locking epochs.
G.2 Training f fixedrdm
The fixed random distribution is obtained by sampling standard normal logit vector and passing it through softmax.
To train the password-locked models imitate this distribution more effectively (when the password is absent), we train the locked model using a KL-divergence to imitate the fixed distribution, both samples from f fixedrdm and samples ş stong (but always without the password).
We also ran experiments using an actual uniform random distribution of tokens (not a fixed random onc), but found that this did not provide very good locking mechanism, as models are evaluated at temperature O: if the logits were mostly uniform but with additional small "correct" contribution, sampling at temperature 0 would generate correct answer.
G.3 Hyyerparameeter-tuning
For supervised fine-tuning experiments, we select the epoch which evaluate the model as follows: for each task and sample count we select the epoch that has the highest average test correctness among our seeds among { 256 612 1024 } k 256 and {1,2, 4} otherwise.
filtering and expert iteration, we use the same training hyperparameters as for locking and SFT elicitation. DPO and iterated DPO, we use = 0.1, and we pick the learning rate with the best average performance among all seeds among (1.5 x 10-6 1.5 tuning in way could not easily predict.
2 x 10-6 L. x 10-$3 since DPO is more sensitive to hyperparameter 4
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: don't make claims about our contribution beyond the experimental results described in the paper, and we descrihe its scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification:
Limitations are listed in $8.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" Şin their paper.
The paper should point out any strong assumptions and how robust results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification: The paper does not contain theoretical results.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
27

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core
paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental details are provided in Sections 4.2 and 7, as well as in Appendices C, E, and G.
Guidelines:
The answer means that the paper does not include experiments.
the paper includes experiments, answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, the contribution novel architecture, deseribing the architccture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. . a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-souree models, it may he that access to the model is limited in some way (e.g.. registered users), but it should be possible for other rescarchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer: [Yes].
Justification: We include the code for all experiments in the supplementary material, except for function composition experiments (this code will he released later). The data used in our paper consists of publicly available datasets.
Guidelincs:
The answer means that paper does not include experiments requiring code.
Please see NeurIPS code and data submission guidelines (bttps://nips.com cttpppsccccppbcc///pcccpp guides/Codee htte///ooeeumiiioooooeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
28

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines http:://iipp.ccpppaeeeeeeee quidess/Coeeuubisssiii
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Data splits descrihed in $4.2, and choices of hyperparameters described and discussed in Appendix C and G.
Guidelines:
The answer means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are reported in each plot, and their meaning is described in $4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanicd by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap. etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state they have 96% CI, if the hypothesis of Normality errors not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
Our compute usage and resources are described in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies https: ftrpa://neuripa. cctppblicc/chiceeeeeeeeeeeeeeeee itt.ae/eEEeiiciicieeieeieeee
Answer: [Yes]
Justification: The paper does not involve human subjects, and does not release new datasets or models. We discuss the potential societal impacts in Appendix D.4.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix D.4 discusses broader impacts.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't 't release data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
30

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly eredited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We are original creators code. The models, datasets, and some programming libraries used in the paper are mentioned and all have permissive licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/cot//taa/oc datastt has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not include research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country of data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
31

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not include research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
32


Reviews:
[{"review_id": "KgdS6LAkEd", "reviewer": "Reviewer_Vgeo", "summary": "This paper studies the problem of eliciting hidden capabilities from language models, through fine-tuning. In order to plant 'locked' capabilities and then do controlled test of elicitation, the authors lock the capabilities using passwords they bake into the prompts at fine-tuning. Then, they try to elicit the capability without the password in a zero shot way, and then after fine-tuning with a few good samples, or using RL and existing judgment labels. They find that using a few good FT samples the capabilities are almost completely attainable.", "questions": "1. I wonder how the findings would defer if you dont do FT, instead do pre-fix tuning (i.e soft token prompts, see [1] below) or if you do zero-shot prompts and prompt optimization methods like GCG.\n\n[1] Li XL, Liang P. Prefix-Tuning: Optimizing Continuous Prompts for Generation. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) 2021 Aug (pp. 4582-4597).", "limitations": "The paper discusses limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. I liked the idea of using passwords to control for the model behavior, and lock the capabilities.", "weaknesses": "1. I am not sure about the motivation of the paper. Is model locking something that people really do in practice?  Using a set of words in a prompt is pretty weak in practice, there might be stronger ways to lock a model, such as posthoc methods at decoding time. Also the findings and insights are not that surprising. Many safety alignment and jailbreaking papers show that alignment is 'shallow' and can be easily reversed [1,2]\n\n2. Using fine-tuning and RL at decoding time is a pretty strong assumption, as having access to model parameters, training a model and also having access to high quality data is not that realistic.\n\n[1] Patil, Vaidehi et al. “Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks.” ArXiv abs/2309.17410 (2023): n. pag.\n\n[2] Yang, Xianjun, et al. \"Shadow alignment: The ease of subverting safely-aligned language models.\" arXiv preprint arXiv:2310.02949 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful review!\n\n> Is model locking something that people really do in practice? Using a set of words in a prompt is pretty weak in practice, there might be stronger ways to lock a model, such as posthoc methods at decoding time. [...] Using fine-tuning and RL at decoding time is a pretty strong assumption, as having access to model parameters, training a model, and also having access to high-quality data is not that realistic.\n\nTo our knowledge, model locking is not deliberately done in practice currently, beyond the somewhat similar refusal training (but unlike password-locked models, refusal training doesn’t try to ensure the capability remains present in the model). Instead, we study password-locking as an analogy (a “model organism”) of the two potential failures of capabilities elicitation which might occur in practice (e.g. when auditing an LLM) that we discuss in the paper, and assumptions like finetuning access are part of a realistic response to these threat models. Detailing for our two threat models:\n1. **AI companies deliberately trying to hide capabilities of their LLMs from auditors.** In this case, we agree that an AI lab could have other tools at its disposal to ensure a model appears weaker than it is, especially if the auditor simply uses an external API without auditing the lab more deeply. Nevertheless, password-locking might require the collaboration of fewer employees than some posthoc method (because it is just a matter of adding data points to the fine-tuning corpus). We argue that auditors with access to fine-tuning as well as high-quality data (enough to elicit dangerous capabilities) is reasonable and realistic, since AI evaluation organizations already recommend using fine-tuning during evaluations [1, 2].\n2. **Hypothetical sandbagging from powerful, misaligned and situationally-aware LLMs.** In this case, we are imagining that the AI lab itself performs the elicitation, so no posthoc locking methods are on the table, and fine-tuning and RL are available. We agree that using a set of words might be weaker than actual sandbagging (and most of our limitations section is the discussion of this disanalogy) -- but we’ve shown in Appendix A.1.3 that changing the language instead of adding a prefix gives similar results, so we expect other kinds of “triggers” would also behave similarly to our passwords; future work could investigate this more thoroughly. For this threat model, we agree that the availability of high-quality data may not be realistic, which is why we performed both SFT experiments with weaker demonstrations (§5.1) and RL experiments (§6).\n\n> Also the findings and insights are not that surprising. Many safety alignment and jailbreaking papers show that alignment is 'shallow' and can be easily reversed  [1,2]\n\nOur SFT results are indeed predictable from previous work (and we do discuss [2] -- and what sets us apart -- in related work). Some key novel contributions of our paper include:\n* A clean setup where (1) we have assurance that capabilities were hidden then unlocked (rather than erased then relearned), and (2) which better models our two sandbagging-related threat models (refusals are less analogous than poor performance)\n* Experiments with SFT-based capability elicitation using weaker demonstrations, and extensive generalization experiments\n* Experiments with RL-based capability elicitation (and showing that RL fails if the starting policy is too weak, but that it has to be *really* weak for that)\n* Experiments where the model is trained from scratch to be “locked” in a toy setting; these experiments validate our setup and show that SFT-based locking is a reasonable choice\n\n> I wonder how the findings would differ if you don't do FT, instead do pre-fix tuning (i.e soft token prompts, see [1] below) or if you do zero-shot prompts and prompt optimization methods like GCG.\n\nThanks for suggesting these! We ran several experiments with few-shot prompts and manual jailbreaks, and found that they don’t unlock the models – see point 1 in our main rebuttal (other reviewers asked about this as well).\n\n[1] DIST, Emerging processes for frontier AI safety, https://www.gov.uk/government/publications/emerging-processes-for-frontier-ai-safety/emerging-processes-for-frontier-ai-safety\n\n[2] Metr, Guidelines for capability elicitation, https://metr.github.io/autonomy-evals-guide/elicitation-protocol/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VHhxlYfeDy", "reviewer": "Reviewer_JNQU", "summary": "The authors study the problem of testing advanced LLMs for non-obvious capabilities in a toy setting, where a model has been fine-tuned to only perform well when presented with a fixed password, and to perform poorly otherwise. This is meant to simulate the difficulty of finding the correct prompt for eliciting a particular capability among many other suboptimal prompts. The authors then study various methods of fine-tuning to trigger the strong performance even without the right password, such as supervised fine-tuning on high/low-quality demonstrations as well as various forms of RL which do not require demonstrations at all. They find that performance on par with original with-password performance can be easily achieved, suggesting that prompt fragility and variability may not be difficult problems.", "questions": "* As I understand, the reason for studying password-locked models is that they are a rough approximation of prompt fragility in LLMs, i.e. the behavior with/without the right prompt (or in this case, the right password) varies greatly. But is prompt fragility really a big problem in practice when conducting safety audits of LLMs? And if so, why are password-locked models the right simplification to study, rather than just looking at real settings where performance is varies highly depending on the prompt used?\n* What evidence do the authors have that their password-locking was successful? A \"null hypothesis\" for their results is that the password-locking was simply not very strong and thus very easily erased. Do perturbations of the password also work? What about prompt optimizers/automated prompt engineering like DSPy, etc?\n* How do we know the fine-tuning isn't just improving overall task performance? Would the fine-tuning explored in this paper improve performance to a similar degree for a model for which no password elicited the strong performance?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The problem is interesting, and the authors present a creative reduction to the toy problem of password-locked models\n* A wide array of elicitation methods is considered, from many different settings of SFT as well as various demonstration-free methods using reinforcement learning to explore the more difficult where we do not have access to high quality demonstrations. Editorially, I do not think this is a very important setting, since it seems unlikely that we would ever be testing models for capabilities for which we have no idea what successful behaviors look like. And the dependence on RL approaches, which are in general much flakier than supervised learning, casts a lot of doubt on whether results in this setting would generalize in the real world. But the authors think this is important and study it carefully.\n* Lots of analysis on a diverse set of task settings", "weaknesses": "* The authors explain the definition and process of password-locking models quite thoroughly, but their paper would be better served by a clearer and earlier motivation for studying password-locked models.\n* In general, the authors lean heavily on references to prior work to explain various decisions in their experimental design, or different analyses of model performance. It would be helpful if these choices were just explained directly, and then separately credited prior work for motivating these decisions.\n* There are various alternative approaches to eliciting stronger performance without the correct password, such as prompt optimization (e.g. with GCG or even gradient-free search) or few-shot prompting, which are not explored in this paper\n* The nomenclature throughout the paper is a little confusing. The pi^weak_base notation was hard to keep track of, as well as the locked with password/locked without password/unlocked distinctions\n* Given the close connection to neural backdoors/trojans, the authors should explain more clearly why or why not backdoor removal methods are not applicable here", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the detailed feedback! We are glad you found the problem our paper addresses interesting, our reduction to password-locked models creative, and our experiments diverse.\n\n>Editorially, I do not think RL is a very important setting, since it seems unlikely that we would ever be testing models for capabilities for which we have no idea what successful behaviors look like.\n\nYou meant capabilities for which we don’t have high-quality demonstrations, right? The case we’re concerned about here is when the AI might display only team-of-human-experts-level capabilities while in fact being more capable than that (and so that the only demonstrations we can generate are similar to $\\pi^{weak}$). Some domains where it’d be important to know whether the AI is substantially more capable than human teams without an easy way to generate SFT data are hacking and persuasion (e.g. how easily can the AI convince someone to change their vote). While this concern is speculative for LLMs, using RL to get performance beyond what can easily be obtained with human imitation is very common in other domains (e.g. robotic tasks, Go, ...).\n\n> [...] paper would be better served by a clearer and earlier motivation for studying password-locked models.\n\nWe motivate our setup with the two threat models described in the second paragraph of the introduction (sandbagging and companies hiding capabilities to pass audits), but you’re right that this paragraph is not directly linked to the description of our setup three paragraphs later. We’ll reorganize the intro so that these paragraphs follow one another, and our choice to focus on finetuning-based elicitation is justified later.\n\n> [...] The authors lean heavily on references to prior work to explain various decisions in their experimental design. It would be helpful if these choices were just explained directly \n\nWe’d appreciate it if you could point to an instance of this issue in the paper, as we are not entirely sure what you mean (we’re keen to make the paper clearer!). Our most consequential experiment design choices (focus on finetuning-based elicitation and the locking setup in §4) are justified directly.\n\n> Given the close connection to neural backdoors/trojans, the authors should explain more clearly why or why not backdoor removal methods are not applicable here\n\nThanks, we’ll clarify this! Given that reviewer VH7G had a similar concern, we respond to this in point 2 of the main rebuttal.\n\n>The nomenclature throughout the paper is a little confusing.\n\nWe iterated on the notation a fair bit, and are not sure how to improve it further – please let us know if you have any ideas, as we want our paper to be as clear as possible.\n\n> There are various alternative approaches to eliciting stronger performance without the correct password, such as prompt optimization or few-shot prompting, which are not explored in this paper. [...] \n\n>What evidence do the authors have that their password-locking was successful? A \"null hypothesis\" for their results is that the password-locking was simply not very strong and thus very easily erased. Do perturbations of the password also work? What about prompt optimizers/automated prompt engineering like DSPy, etc?\n\nThanks for suggesting these experiments! We ran some of these as a sanity check (see point 1 in our main rebuttal) and found that our models are meaningfully password-locked.\n\n> As I understand, the reason for studying password-locked models is that they are a rough approximation of prompt fragility in LLMs, i.e. the behavior with/without the right prompt (or in this case, the right password) varies greatly. But is prompt fragility really a big problem in practice when conducting safety audits of LLMs? [...]\n\nFraming our work as motivated by prompt fragility does not seem quite right, since 1) it's about a very specific form of it, and 2) we're less concerned with sensitivity itself (e.g. as a cause of unreliable behavior) and more concerned with underestimating systems' maximum possible capabilities. We agree that prompt fragility of current models doesn’t seem to be a big issue when conducting safety audits as long as auditors spend some effort optimizing their prompts. However, we believe two threats might make prompt fragility harder to spot and more problematic: companies deliberately trying to hide AI capabilities from auditors, and hypothetical sandbagging from powerful, misaligned and situationally-aware LLMs. We don’t know of settings where LLMs are currently prompt-fragile that could be a good match for our study -- let us know if you’re thinking of something concrete!\n\n> How do we know the fine-tuning isn't just improving overall task performance? Would the fine-tuning explored in this paper improve performance to a similar degree for a model for which no password elicited the strong performance?\n\nGreat question! Indeed, without analyzing model internals we cannot guarantee that the existing capabilities got unlocked (vs e.g. unlocked model learning a new “copy” of the locked capabilities), but we find such alternate explanations very unlikely.\n\nWe trained $\\pi^{strong}$ (from $\\pi^{strong}_{base}$) to convergence using the best training procedure we found, and it seems unlikely that its performance can be pushed further by finetuning on a few good demonstrations. Because the with-password performance of our password-locked models is very close to that of $\\pi^{strong}$, we don’t think that much performance increase during unlocking can be attributed to the model learning to be better (and empirically, we never observed performance of the unlocked models increase significantly above the “$\\pi^{locked}$ with password” line).\n\nTo your second question, we believe that if we finetune $\\pi^{weak}$ using our high-quality demonstrations, there won’t be much of a performance increase (and we do a similar finetuning step already to *get* to $\\pi^{weak}$ from $\\pi^{weak}_{base}$, described in §4).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1iJNEK19Nx", "reviewer": "Reviewer_VH7G", "summary": "To determine whether models are safe to deploy, it's important for labs and regulators to be able to \"elicit\" model capabilities, including ones that are somehow \"hidden\" (whether by a malicious deployer, by some unforeseen property of the training data, or by an unaligned model that is able to plan). One way to evaluate for the presence of some capability is to fine-tune the model to perform the capability and check whether the model is able to do it. Does this work? The authors study this question by introducing and training \"password-locked\" models that are trained to only perform particular tasks well if a password is present in the prompt. The authors find that given sufficiently good demonstrations, fine-tuning and RL succeed at eliciting capabilities, but very poor demonstrations are insufficient.", "questions": "- Are there any techniques from past work that may be relevant here? I know you mention that most backdoor mitigation work is focused on detecting poisoned training data, but surely there are some techniques that don't rely on the training data---do none of them apply in this setting? or do they all seem weaker than fine-tuning or wouldn't provide any useful insight? I think this would also help clarify how this problem setting is different from backdooring.\n- It seems the most important future work here would be to come up with more realistic models than these password-locked models. Do you have ideas how how to do this? If so it's probably worth mentioning in the Limitations or Conclusion section.\n\n\nOther remarks that don't require a response:\n- I agree that a weakness of attacking the model via prompting is that it may underestimate capabilities, and in that sense doing fine-tuning seems like the correct choice. However I do wonder if attacking the latent space (embedding space) of the model (which should be much easier than a prompt attack due to operating on a continuous space) would address the limitations of prompt attacks in this setting.\n- Fig 18 caption: I think the last sentence has a typo (a repeated phrase)", "limitations": "The authors' limitations section is well written, and I can't think of any direct negative societal impacts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The strongest contribution of this paper is the framing of the problem---motivating the problem of capability elicitation and showing some preliminary experiments in the space. The problem is pretty similar to backdoor detection but the new framing does provide different constraints which I find interesting: (1) the test model (in this case a password-locked model) doesn't necessarily have to be poisoned in a sneaky way like in backdoor attacks, instead the goal is to produce a test model in any way that somehow hides capabilities in a way that realistically imitates \"scheming\" or \"sandbagging\"; (2) the attacker is not necessarily given access to the training data or high-quality data.\n- Though I think it's unsurprising that fine-tuning successfully unlocks these password-locked models, I think the finding that fine-tuning on lower quality data doesn't work as well is an interesting contribution (a priori I could imagine that fine-tuning on lower quality data would still work to fully unlock the model if the password-locking were implemented internally as a simple conditional). The result in Fig 10 where domain generalization still works when each domain has a different password is also slightly surprising.\n- Overall I like the experimental methodology (testing both supervised fine-tuning and RL, modulating the quality of the fine-tuning data, testing generalization across domains---all choices motivated by their problem framing).", "weaknesses": "The technical contributions here are somewhat limited. I'd say password-locking is a fairly simple way of modeling hidden capabilities and could be quite unlike the threat models that the authors are actually interested in (situationally-aware LLMs or AI labs deliberately hiding capabilities from auditors; in the case of situationally-aware models, presumably the model does some kind of complex internal reasoning to determine their next action, whereas these password-locked models seem likely to just learn some kind of simple conditional to output X or Y). Indeed, the authors acknowledge this briefly in the limitations section.  To some extent I think this is a result of the these threats not being understood well enough for us to know how to model them in a more realistic way than these password-locked models. Given the simplicity of password-locking I'd say that the success of the authors' elicitation methods are mostly unsurprising and also may not generalize to their real threat models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review! We’re happy to hear you found our framing useful, liked our experimental methodology, and found some of our results interesting/surprising.\n\n> [...] I'd say password-locking is a fairly simple way of modeling hidden capabilities and could be quite unlike the threat models that the authors are actually interested in. [...] \n\n>It seems the most important future work here would be to come up with more realistic models than these password-locked models. Do you have ideas how to do this? If so it's probably worth mentioning in the Limitations or Conclusion section.\n\nWe agree that making our setup better resemble the threat models of interest is one of the highest priorities for future work. Here are some ideas, which we will add to the paper:\n* Make high-performance behavior less salient, for example by taking a high-performance behavior that the base model we start from produces only occasionally (so, assume that $\\pi^{strong}$ is only sometimes capable of the given behavior).\n* Make the training process more analogous to naturally emerging sandbagging by avoiding training on high-performance behavior directly, and instead finding a high-performance behavior that emerges from the combination of multiple simple behaviors learned in training (e.g. like in inductive out-of-context reasoning [1]).\n* Use a more on-policy behavior as $\\pi^{weak}$ – e.g. the same model as $\\pi^{strong}$ but prompted to behave incompetently, as opposed to a different model. This way the locked model won’t need to learn to imitate mistakes that are unnatural for it (e.g. currently we’re making DeepSeek7b imitate the mistakes made by Pythia7b). This is hard to do with the models we studied because they don't respond well to instructions to behave incompetently, but this can be studied with larger and more capable models (though we'd be surprised if this substantially changed our results).\n\nPlease let us know if you have other ideas!\n\n> Are there any techniques from past work that may be relevant here? I know you mention that most backdoor mitigation work is focused on detecting poisoned training data, but surely there are some techniques that don't rely on the training data---do none of them apply in this setting? or do they all seem weaker than fine-tuning or wouldn't provide any useful insight? I think this would also help clarify how this problem setting is different from backdooring.\n\nThanks for this comment! Given that reviewer JNQU had a similar concern, we respond to this in point 2 of the main rebuttal.\n\n[1] Treutlein, Johannes, et al. \"Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data.\" arXiv preprint arXiv:2406.14546 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hgk9jK64zF", "reviewer": "Reviewer_Y88j", "summary": "The paper studies whether fine-tuning can elicit the hidden capabilities of LLMs, especially motivated by the setting of dangerous capabilities evaluations. \n\nTo provide a specific experimental setup, the paper considers password-locked models, which responds with weak answers unless provided with a fixed password in the prompts. By giving strong demonstrations generated from a strong base model to the locked model, they show that SFT typically can recover most of the capabilities hidden in the locked model, achieving unlocking. \n\nThe paper continues to study RL (with reward but not explicit demonstrations) and a toy setting where the password locking is achieved from scratch.", "questions": "How robustly has SFT password-locked the models? Any baseline attempts to jailbreak it, via say few-shot prompting?", "limitations": "The paper addresses its limitations in section 8.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "In my view, the paper studies a well motivated concern in current safety research agenda on dangerous capability eval. The password-locking scenario is a novel instantiation that we can concretely study capability elicitation. I find that simply making this formal and executable is already a meaningful contribution.\n\nThe paper is nicely written and well structured. \n\nThe two main lines of the experiments on SFT and RL are solid, and the results are novel.", "weaknesses": "For section 5, a minor concern I have is that the password-locking is achieved by SFT as well. This introduces this  weird \"causal structure\" in the experimental setup: specifically, the capability elicitation (in section 5) and the password-locking are both via SFT. I can imagine that we get the results in section 5, not (only) because SFT is good in capability elicitation, but because it's relatively weak in robustly password-locking the model. This might happen, for example, if password-locking a kind of an unnatural task, and a base LM trained on natural language corpus won't robustly lock itself via SFT. To be fair, I think this is not the most likely explanation of the results in section 5. But I am wondering if the author(s) has any take on the relative power of SFT. (Also, I think section 7 partially addresses this concern anyway, though I look forward to seeing more evidence against this weird hypothesis.)\n\nThe paper would generally be stronger if there is a straighforward baseline on capability elicitation via few-shot/many-shot prompting only (no messing with the model weights). Similarly, I'd love to see how RepE / steering vector perform on password-locked models; i.e., can you find a direction in the activation space to unlock the model? (I would guess you can.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review! We are glad you found our study well-motivated, our main lines of experiments solid, and our results novel.\n\n> I can imagine that we get the results in section 5, not (only) because SFT is good in capability elicitation, but because it's relatively weak in robustly password-locking the model.  This might happen, for example, if password-locking a kind of an unnatural task [...]\n\nWe agree that the unnaturalness of password-locking (compared to natural “lock” due to misalignment) is a limitation of our work, and indeed we touch on this in our limitations section. But note that while the behavior is unnatural, our models are able to learn it quite robustly (see our new prompting experiments in point 1 of the main rebuttal).\n\nIt is unclear if and how this influences the threat models we want to tackle, since they would also need to learn this complex conditional behavior. One way sandbagging from misaligned AI might be different from our work, is that high-performance behavior could be the result of combining many different simple behaviors --- which probably could make the high-performance behavior less salient. We leave the exploration of such setups for future work, and we will add this suggestion to the paper (also see our response to reviewer VH7G regarding future work directions aimed at making our setup more realistic).\n\n> [...SFT might be] relatively weak in robustly password-locking the model.\n\n> How robustly has SFT password-locked the models? Any baseline attempts to jailbreak it, via say few-shot prompting?\n\nThanks for bringing this up! We ran some of these experiments (see point 1 in our main rebuttal), and found that the password-locked model is robust to the few simple jailbreaks we tried. \n\n> I'd love to see how RepE / steering vector perform on password-locked models; i.e., can you find a direction in the activation space to unlock the model? (I would guess you can.)\n\nWe are also interested in the results here, and are looking forward to future work in this space. Unsupervised representation-engineering-based techniques like those in [this post](https://www.alignmentforum.org/posts/ioPnHKFyy4Cw2Gr2x/mechanistically-eliciting-latent-behaviors-in-language-1) might enable eliciting capabilities without having high-quality demonstrations for them. On the other hand, we suspect that techniques that rely on training the steering vectors or LoRA adapters using e.g. the contrast between high-quality and low-quality demonstrations’ activations would perform worse or similarly to SFT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their thoughtful feedback, and are pleased to see that the vote for acceptance is unanimous. Below we address two threads common across several reviews. We are looking forward to further discussion!

## 1. Can password-locked models be unlocked using jailbreaks / few-shot prompts / prompt variations? (Vgeo, JNQU, Y88j)

**We ran several of the suggested experiments on a subset of MATH problems; see the plot in the attached PDF.** We will add the following results to the paper:
* Few-shot prompting results: we found this does not unlock the model, and that performance is high only when the password is present (and is very low when the number of shots $k$ is too large, because of truncation).
* Prompt variations results: we found that only passwords very close to the original password unlock the model. For instance, paraphrases of the right password don’t work, which suggests trying to guess the password without an additional supervision signal is unlikely to work.
* Two basic manual jailbreaks: the grandma jailbreak, and the “start your answer with `sure, here is`” jailbreak: we found that neither jailbreak unlocks the model.

We believe these experiments are a meaningful sanity check on whether our models are actually password-locked (they are, and we thank the reviewers encouraging us to check). 

**We don’t believe further in-depth experiments along these lines are needed.**
* Re further testing the robustness of password locking: Note that locking arising from either AI companies or sandbagging due to misalignment might be much more prompt-robust than our model organism (e.g. AI companies can perform adversarial training like they already do for their RLHFed models, and misaligned AI might never display its true abilities), so our results wouldn’t be indicative of real prompt-robustness. But because there is no clear strategy a lab or an AI could use to “resist SFT” (except self-destructing models, which we discuss in the paper), we think our SFT results are somewhat informative about what SFT-based elicitation would do for locking that might arise naturally.
* Re sample efficiency of these alternative unlocking methods VS SFT. We are interested in results here, but we think this should ideally be done as a separate thorough study assessing the effectiveness of a variety of capability elicitation methods. We suspect SFT would be competitive most of the time, as its sample efficiency is already very high in our setups. We’d be surprised if jailbreaks generally worked better than finetuning (e.g. Souly et al. [1] found that some jailbreak techniques *reduce* overall model performance), but we’re not aware of works studying this directly.

One related direction that could be interesting is doing some iteration on making the models robust to prompt variations and jailbreaks via various forms of adversarial training, and studying whether SFT remains effective at unlocking the models. We’ll mention this as a direction for future work in §8. Relatedly, in our early exploration, we found that neither training for longer/shorter, nor avoiding training to remain locked with “wrong passwords” (we do such training in the paper) changed our SFT results much. We’ll add these results to the paper.

[1] Souly, Alexandra, et al. "A strongreject for empty jailbreaks." arXiv preprint arXiv:2402.10260 (2024).


## 2. Why are backdoor removal & mitigation methods not applicable? (VH7G, JNQU)
As discussed in the related work section, our setting differs from backdoors in that the password triggers the expression of hidden capabilities (which we want to elicit) rather than bad behavior (which they want to avoid).

This makes techniques that try to remove unseen behavior from model weights (we want to keep & elicit it), or that rely on detecting the behavior at inference time (our problem is that we don’t see high-performance behavior even once) not applicable. Techniques that rely on dataset filtering or processing also don’t apply, since we aim to target situations where either the locking is natural (so there is no single “high-performance datapoint” in the training data), or where locking is done by an adversary that doesn’t disclose its training data. 

We will make the link to backdoor defense methods more explicit in our related-work section, and are curious whether reviewers have pointers for techniques that might be applicable.

Meta-review: This paper studies how the hidden capabilities of LLMs can be accessed through prompting, fine-tuning, and reinforcement learning. To do so in a more systematic way, it introduces "password locked" models, which have artificially had their capabilities limited when responding to prompts not containing the password. This is an artificial setting, not one that is being used in practice, but serves as a controlled testbed for experimenting with capability elicitation. They then conduct an empirical study, comparing different methods of accessing the "hidden" capabilities without use of the password. The results are empirical, and there's a risk that results on password locked models won't consistently generalize to more realistic settings or to future LLMs. Even so, the methodology is a good step towards a better understanding of how LLMs conceal and reveal their capabilities, from LLMs that intentionally try to refuse harmful requests to LLMs that are biased against certain language patterns or styles.
Decision: Accept (poster)