Title: Preference Learning with Lie Detectors can Induce Honesty or Evasion
Venue: NeurIPS 2025
Accepted: yes
Keywords: RLHF

deception

preference learning

safety

scalable oversight

monitoring
Abstract: As AI systems become more capable, deceptive behaviors can undermine evaluation and mislead users at deployment.
Recent work has shown that lie detectors can accurately classify deceptive behavior, but they are not typically used in the training pipeline due to concerns around contamination and objective hacking.
We examine these concerns by incorporating a lie detector into the labelling step of LLM post-training and evaluating whether the learned policy is genuinely more honest, or instead learns to fool the lie detector while remaining deceptive.
Using DolusChat, a novel 65k-example dataset with paired truthful/deceptive responses, we identify three key factors that determine the honesty of learned policies: amount of exploration during preference learning, lie detector accuracy, and KL regularization strength. 
We find that preference learning with lie detectors and GRPO can lead to policies which evade lie detectors, with deception rates of over 85\%. 
However, if the lie detector true positive rate (TPR) or KL regularization is sufficiently high, GRPO learns honest policies.
In contrast, off-policy algorithms (DPO) consistently lead to deception rates under 25\% for realistic TPRs.
Our results illustrate a more complex picture than previously assumed: depending on the context, lie-detector-enhanced training can be a powerful tool for scalable oversight, or a counterproductive method encouraging undetectable misalignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates using lie detectors during the training of LLMs and if they can reduce deceptive behavior, or if models will instead learn to evade detection. They generate a dataset of 65k examples, and use 5% of them to train a lie detector on the ground truth labels. Then they label the rest of the data with a simulation of a low cost labeller which chooses a lie 73% of the time unless the lie detector triggers. They find that the trained model learns an honest policy when lie detectors are highly accurate but that they learn to be deceptive when accuracy is lower. They use both a DPO variant and GRPO, finding that GRPO leads to more lie detector evasion.\n\nSTRENGTHS AND WEAKNESSES:\nI struggled with this one, while I think the idea is great and important, I think there are a number of major flaws.\nStrengths\n- Important Research Question.\n- Good idea.\n- Honest about its limitations.\nWeaknesses:\n- The synthetic labeling uses ground truth information that wouldn't exist in practice. Low-cost labelers mechanically prefer deceptive responses based on knowing they're deceptive, rather than because they actually seem better. I feel like the study would be much better if you used weaker models that actually have flaws rather than randomly preferring the wrong response if the lie detector doesn’t trigger.\n- The lie detector is trained on the original models activations but is applied to the modified models. The theoretical justification for lie detector probes (as i see it) is that when models lie, they will be thinking about lying which we can potentially detect, if we use a different model then this breaks down and we are instead asking “does this other model think the statement is False?”. This seems much worse to me, and the benefit of using a probe rather than just asking the original model seems unclear. Now the representations of the trained/original model are probably quite similar since you start from the same point and don’t train very hard but I still think this is a major limitation. \n- The dataset seems unrealistic. The dataset seems to consist of one turn conversations where all the relevant information has been provided in the system prompt. This seems highly unrealistic, and not typically how system prompts are used. Synthetic data generation has come a long way and it seems like it would be possible to make the setup much more realistic by presenting the information in conversation, in longer documents or with tools. While this might not be a major concern with such a weak model, I suspect not even Llama 3.1 8B will think the situation it is in is real.\n- Since they start with an aligned model (Llama-3.1-Instruct), high KL regularization preventing deception might just mean \"don't train much.\"?\n- 103 token limit seems very short?\n\nQUESTIONS:\nI'd be interested in seeing the authors response to my concerns above.\n- What would happen if you trained a lie detector on the final model? How different would its predictions be compared to the original?\n- Why is the token limit 103? That seems both very specific and very short?\n- How do we know that regularisation isn’t just stopping training? Is there some other positive metric you could track that increases during training with regularisation?\n- If you ask the model whether the situation it is in is real, how often does it say yes?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies /Lie Detection/ as a scalable oversight mechanism. Specifically they consider a pipeline called \"SOLiD\", implemented as follows.\n- A model response is passed to a \"Lie Detector\" model (which was originally trained with preference learning from a \"Costly Labeller\" on a small number of samples)\n- If the Lie Detector rejects, the response is passed to the Costly Labeller for the final decision.\n- If the Lie Detector accepts, the response is passed to the \"Cheap Labeller\" for the final decision.\nThen this whole pipeline is used to label a large number of samples, and a new Lie Detector is trained on it.\n\nThe idea is that despite the fact that this method retains incentives for deception (it favors models that produce responses that deceive both the Lie Detector and the Cheap Labeller), it is likely to be the sort of thing used widely in practice to reduce costs. In particular they note that:\n\n> We note that a seemingly-innocuous idea such as giving labellers access to a probe-based ‘deception alarm’ is equivalent to SOLiD, where the high-cost and low-cost labellers are the same individuals.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n1) SOLiD seems like a good abstraction of how scalable oversight may be applied in practice by labs, and it is valuable to evaluate it.\n2) In addition to evaluating SOLiD, the paper also discovers specific important factors that favor learning honest policies: higher lie detector accuracy, less exploration during preference learning (e.g. by using off-policy methods like DPO) and higher KL regularization to an honest reference model.\n3) The paper addresses a very timely problem in alignment, and represents an important empirical advancement in studying deception.\n4) The DolusChat dataset they developed could be used widely in a range of empirical scalable oversight works and benchmarks.\n\n*Weaknesses*\n1) The work only trains small (1B, 3B, 8B) models; it is possible larger models are more capable of deception?\n2) The description of the pipeline in the introduction was a bit contrived for me to understand, and took a few reads. It might be useful to explain a bit more pleasantly why SOLiD is such a natural oversight method that labs might use.\n\nQUESTIONS:\nDo you think that this work/SOLiD has any implications for studying scalable oversight mechanisms like Debate, etc? Can a generalized form of SOLiD model a low-cost version of any scalable oversight mechanism?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses deceptive behaviors of LLMs (due to preference learning from human feedback) that can mislead users at deployment.\n\nExisting post-hoc lie detectors monitor LLMs and can identify potentially deceptive model output with high accuracy. \n\n\nThis paper asks the research question of whether leveraging highly-accurate lie detectors during post-training solves the deception problem. In particular, this paper studies:\n- Will models inevitably learn to circumvent the detector? Or\n- Might there be conditions under which detector-guided training actually promotes truthfulness?\n\n\n\nTo answer these questions, this paper incorporates a lie detector into the labelling step of LLM post-training using the following setting:\n- train a lie detector on model internals--a logistic regression model using the residual stream at the middle decoder layer, extracted at the final non-padding token position\n- preference learning using (truthful, deceptive) labels and the GRPO or DPO algorithm to train a policy\n\n\nFinally, this paper evaluates whether the learned policy is genuinely more honest, or instead learns to fool the lie detector while remaining deceptive.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Valuable dataset contribution for addressing the deception problem in the behaviour of LLMs. Created DolusChat, a 65k-example chat format dataset for enabling large-scale lie investigation (using GPT-4o, Claude Sonnet-3.5, Gemini 2.0-Flash with chain-of-thought reasoning). Each example is composed of a system prompt, a user query and a pair of assistant responses. One response is truthful and one is deceptive.\n- Incorporating lie detectors into the labelling step of LLM post-training\n- Discovering three key factors favouring honest learned policies: higher lie detector accuracy, less exploration during preference learning, and higher KL regularisation to an honest reference model.  This paper shows that preference learning with lie detectors and GRPO can lead to policies which evade lie detectors, with deception rates of over 85%. However, if the lie detector true positive rate (TPR) or KL regularisation is sufficiently high, GRPO learns honest policies.\n- Well written and structured. \tI enjoyed reading this paper and learning about the deceptive behaviours of LLMs.\t\n\n\nWeaknesses:\n- My main concern is that learning honest policies is not guaranteed. For example, what will happen in a more evading tempting scenario controlled by an informed adversary who is informed about the lie detection algorithm?  \n- I am not an expert in this area, could you confirm whether there is no other work that could be considered as baselines in the experiments?\n- Findings of this paper can be stronger if empirical evaluations are extended to more than one single dataset, especially as this single dataset is created by this paper too. I appreciate authors also highlighting this limitation in their paper. \n- Typos:\n  - line 108: hid --> hide\n  - line 85: incomplete ref\n\nQUESTIONS:\nsee above questions\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents the first large-scale, systematic investigation of Scalable Oversight via Detector in large language models. Through preference learning with lie detector and GRPO, the authors identify three key factors that determine the honesty of learned policies: amount of exploration during preference learning, lie detector accuracy and KL regularization stength.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This work defines a new problems of Scalable Oversight via Lie Detector in large language model post-training process.
2. The authors summarize recommendations for detector-in-the-loop training based on their practical experiments.

Weaknesses:
1. No demonstration examples of the model generations after training are provided, making readers hard to understand the model behaviors.
2. Only Llama 3.1 experiments were provided. Lack diverse model architectures such as Qwen series.

QUESTIONS:
1. What are the motivations of using detector with low TPRs (lower than 70%) in practice? 
2. I'm curious about the performance with one of the most achievable lie detectors, directly using large language models like GPT-4o (or directly using Llama-3.1-8B-Instruct) as the lie detector. If the model can easily reach high TPRs, would it be necessary to analysis the model behaviors under low TPRs?
3. Would there be any baseline comparisions with the model original performance befoer detector-in-the-loop training?
4. Is the worse performance of GRPO caused by the reward model used during training? Have you tried to use the same reward model to construct DPO dataset for fair comparision?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-12