Title: Time-Reversal Provides Unsupervised Feedback to LLMs
Venue: NeurIPS 2024
Accepted: yes
Keywords: LLMs

Reranking

reverse LLMs

reverse scoring

defenses

generative models

sequence reversal
Abstract: Large Language Models (LLMs) are typically trained to predict in the forward direction of time. However, recent works have shown that prompting these models to look back and critique their own generations can produce useful feedback. Motivated by this, we explore the question of whether LLMs can be empowered to think (predict and score) backwards to provide unsupervised feedback that complements forward LLMs. Towards this, we introduce Time Reversed Language Models (TRLMs), which can score and generate queries when conditioned on responses, effectively functioning in the reverse direction of time. Further, to effectively infer in the response to query direction, we pre-train and fine-tune a language model (TRLM-Ba) in the reverse token order from scratch. We show empirically (and theoretically in a stylized setting) that time-reversed models can indeed complement forward model predictions when used to score the query given response for re-ranking multiple forward generations. We obtain up to 5\% improvement on the widely used AlpacaEval Leaderboard over the competent baseline of best-of-N re-ranking using self log-perplexity scores. We further show that TRLM scoring outperforms conventional forward scoring of response given query, resulting in significant gains in applications such as citation generation and passage retrieval. We next leverage the generative ability of TRLM to augment or provide unsupervised feedback to input safety filters of LLMs, demonstrating a drastic reduction in false negative rate with negligible impact on false positive rates against several attacks published on the popular JailbreakBench leaderboard.

Time-Reversal Provides Unsupervised Feedback to LLMs

Abstract

Large Language Models (LLMs) are typically trained to predict in the forward direction of time. However, recent works have shown that prompting these models to look back and critique their own generations can produce useful feedback. Motivated by this, we explore the question of whether LLMs can be empowered to think (predict and score) backwards to provide unsupervised feedback that complements forward LLMs. Towards this, we introduce Time Reversed Language Models ( TRLM s), which can score and generate queries when conditioned on responses, effectively functioning in the reverse direction of time. Further, to effectively infer in the response to query direction, we pre-train and fine-tune a language model ( TRLM-Ba ) in the reverse token order from scratch. We show empirically (and theoretically in a stylized setting) that time-reversed models can indeed complement forward model predictions when used to score the query given response for re-ranking multiple forward generations. We obtain up to 5% improvement on the widely used AlpacaEval Leaderboard over the competent baseline of best-of-N re-ranking using self log-perplexity scores. We further show that TRLM scoring outperforms conventional forward scoring of response given query, resulting in significant gains in applications such as citation generation and passage retrieval. We next leverage the generative ability of TRLM to augment or provide unsupervised feedback to input safety filters of LLMs, demonstrating a drastic reduction in false negative rate with negligible impact on false positive rates against several attacks published on the popular JailbreakBench leaderboard.

1 Introduction

Large Language Models (LLMs) trained on a large corpora of text are able to accomplish a wide variety of downstream tasks such as summarization, open-ended/ context-based question answering, document retrieval, and citation generation [Brown et al., 2020, Zhao et al., 2023a]. While the generations from pre-trained and instruction-tuned models already show significant promise, alignment techniques such as Reinforcement Learning via Human Feedback (RLHF) [Anil et al., 2023a, Ouyang et al., 2022] are widely used to improve the quality of their generations further. However, these methods rely heavily on additional supervision to construct preference data, which can be expensive to acquire, or noisy for training. This brings up a natural question Can we generate useful feedback on LLM generations without additional supervised data?

A recent line of work aims at specially prompting LLMs to review their own generations and generate meaningful natural language feedback, which can subsequently be used to refine them [Madaan et al., 2024]. This process can be repeated to improve the generations iteratively. The success of such methods serves as an evidence that it is indeed possible to obtain better responses without additional supervision. However, such methods rely on the superior instruction following and reasoning abilities of LLMs, which may not necessarily hold for low capacity models. Further, these methods involve sequential processing of the generated responses, and thus increase inference time significantly.

In this work, we propose a natural method of enabling LLMs to look backwards in order to obtain meaningful unsupervised feedback during inference. Towards this, we introduce a class of models that we call Time Reversed Language Models ( TRLM s), which operate in the reversed direction of a regular LLM, or the time-reversed direction. Rather than predicting (or scoring) in the standard query → response direction, time reversed language models predict (or score) in the response → query direction. We first introduce TRLM-Fo - a TRLM variant based on forward models, which are prompted to operate in the time-reversed direction using a prompt such as "Generate a question that would result in the following answer: <response>" . Further, we extend the reversal to token -level granularity by pre-training LLMs from scratch in a reversed token direction, rather than the standard forward token direction. We call this as TRLM-Ba where Ba stands for Backward. Note that the inputs and outputs of such a model are in the reversed language order. Pre-training TRLM-Ba on reversed text exposes the model to a completely different world model where the conventional order of information is flipped. Introductions follow conclusions, questions follow answers, logical precedents follow their antecedents. Hence, such a model may not only develop representations that are distinct from those of a regular LLM - despite being trained on the same pre-training corpus - but may also be better suited to score/ generate in the reverse direction, i.e. conditional on the response.

We show in several use-cases that scoring and generation in this reverse direction can produce non-trivial feedback on the responses generated by forward LLMs. We consider three classes of tasks to showcase the scoring and generating capability of TRLM , viz. a) Reranking answers in open ended question answering b) Citation and retrieval tasks and c) Amplifying existing safety filters through query generation in the reverse.

Our Contributions:

a) We propose time reverse language models TRLM-Fo , TRLM-Ba and TRLM-FoBa , all of which score and generate queries given responses, enabling their use in obtaining unsupervised feedback on LLM generations. TRLM-Fo is a forward model prompted to predict in reverse, while TRLM-Ba is pre-trained in the reverse token order, enabling reverse prediction naturally. TRLM-FoBa is pre-trained in both reverse and forward token orders and can be used to predict in forward or reversed language.

b) We demonstrate significant improvements when best-of-N reranking is applied to multiple LLM generations by using TRLM scores. Specifically, we show up to a 5% improvement over self-reranking using TRLM-Ba , in LC win-rates (0.98 Pearson correlation with human preferences) against a GPT4-1106-Preview reference model. We show multiple ablations on this study.

c) We demonstrate that the reverse direction of scoring ( response → query ) is highly significant, as it improves citation attribution accuracy by 44.15% when compared to the forward baseline on the CNN-Daily Mail dataset. Further, we improve the NDCG@10 metric by 44 . 13 points on the NF-Corpus medical information retrieval benchmark, and obtain similar improvements on MS-Marco as well.

d) We show that the reverse generation capability of the TRLM models - specifically TRLM-Ba , can be used to improve False Negative rate (FNR) of input safety filters with negligable impact on FPR. We show significant improvements on several attacks submitted to the Jailbreakbench benchmark, and on a Human Annotated dataset from JailbreakBench.

Wecomplement these results with theoretical arguments using a bipartite graph model between queries and responses, to show that RLHF done with TRLM-Ba scores induces a non trivial distribution shift in answers, mitigating primitive forms of 'hallucination' under the defined conditions.

2 Related Work

Reverse Direction in Language Modeling: Classical work [Serdyuk et al., 2017] showed how sequence to sequence models can regularize the current word token embedding based on the ability of the future tokens to be able to predict the current token. Such bi-directional (forward and reverse) consistency checks have been used to improve forward models. Golovneva et al. [2024] train an LLM

in the forward direction first, followed by the reverse token direction, and show that this alleviates the reversal curse identified by Berglund et al. [2023]. This work is closely related to ours in that we also consider a variant of combining reverse and forward token order during training. Our key models differ from this, and are trained in either forward ( TRLM-Fo )/ reverse ( TRLM-Ba ) token order, using which we demonstrate improvements in a wide range of applications such as long form question answering, citations, retrieval and augmenting input filters for defending against toxic questions. Yang et al. [2023] use question generation from a given answer combined with access to external databases to determine hallucination. Another recent work [Guo et al., 2024] also explores a different pre-training order. While their focus is to correct causal ordering bias, our work instead is focused on the value that scoring and generation of these models bring to downstream tasks.

Reversed scoring: Several prior works [Li et al., 2016, Zhang et al., 2018, 2020] have proposed to improve the diversity of generated responses by optimizing the mutual information between the responses and the respective queries. These works motivate the need for better decoding strategies based on scores in both, response → query and query → response directions. We theoretically show that reverse scoring alone, when used with forward generations, will achieve this naturally using a formal RLHF based argument (Lemma 2), and present strong empirical results across a wide range of tasks to support the same.

Controlling Decoding through feedback: A broad line of works align a pre-trained model to a reward model trained on human feedback by using Reinforcement learning (RL) techniques like Proximal Policy Optimization (PPO) [Stiennon et al., 2020, Ouyang et al., 2022, Korbak et al., 2022], (Identity policy optimization) IPO (and Ψ PO ) [Azar et al., 2024], Direct Preference Optimization [Rafailov et al., 2024] and offline RL [Snell et al., 2022]. Zhao et al. [2022] and Zhao et al. [2023b] calibrate likelihood of generated responses on a dataset with desired responses or human preference feedback.[Krause et al., 2020, Yang and Klein, 2021, Qin et al., 2022] control the generation of an LLM at test time by specifying constraint functions or discriminators that operate in the token or logit space, encouraging certain attributes in the output. Using preference feedback, Mudgal et al. [2023b] train a prefix scorer model that acts as a value function over partial completions consistent with the preference rewards. Yang et al. [2024b] investigate the relation between best-of-N-reranking and KL regularized RL objective. An observation made by Yang et al. [2024b] is that best-of-N-reranking dominates/ competes very well with most RL based alignment methods. Under certain assumptions, authors show formally that best-of-N-reranking approximates the optimal solution to the regularized RL objective. We take inspiration from this and use best-of-N-reranking to evaluate generations through unsupervised feedback by the reverse LLMs. Our work differs from all these in that they rely on external feedback to control generation, while our method does not.

Self Play and Self Tuning: Chen et al. [2023] explore how an LLM can be prompted to self-debug based on an explanation of the code produced by the LLM during code generation and the execution output on test cases. Welleck et al. [2022] use a corrector model that is trained to prefer a new corrected answer if the corrected answer has higher value that a default generation. They require access to a value function for this determination. All these approaches use an external feedback to align the model in their pipeline.

Fu et al. [2023] explore LLM agents initialized as buyers and sellers to play a negotiating game of setting the price of a transaction. A critic LLM provides feedback to both the buyer and seller agents to improve. Madaan et al. [2024] propose a self refining loop where the same model is prompted to provide feedback and further use the feedback to refine and regenerate. Both these works use very powerful and large models from the Claude, GPT-4, GPT-3.5 family to use self generated language feedback. Madaan et al. [2024] remark that the self refining approach does not work well with weaker models. In contrast, we focus on improving generation quality of much smaller models using unsupervised scalar feedback. Other prior works relating to self play are reviewed in the survey article by Amini et al. [2022].

3 TRLM - Time Reversed Language Models

Weintroduce our primary contribution TRLM ( T ime R eversed L anguage M odels), a class of language models that operate in the response → query direction during scoring and generation. This is achieved by either (a) [ TRLM-Ba ] reversing the token order and effectively utilizing previous token prediction instead of next token prediction during pre-training, scoring, and generation, or (b)

[ TRLM-Fo ] maintaining the standard token order during pre-training but reversing the direction of generation through appropriate prompts during inference (scoring and generation).

We show that TRLM provides non-trivial unsupervised feedback that could be used by pre-trained, fine-tuned, and instruction tuned models, for various downstream tasks like reranking to improve open-ended long-form question answering, generating citations, and retrieval. We demonstrate that the ability of TRLM to score in the reverse direction - scoring query based on the response - is essential to achieve the requisite gains. Further, TRLM s that are pre-trained in the reverse direction ( TRLM-Ba ) provide an additional boost in most cases. We further leverage the generative ability of TRLM in reverse (generating query from a response) to amplify the effectiveness of input safety filters as well.

We propose four variants of the TRLM class TRLM-Ba , TRLM-Fo , TRLM-FoBa (Reverse) and TRLM-FoBa (Forward) - based on how they are pre-trained and fine-tuned. TRLM models can be considered to have three functions: TRLM .Pretrain, TRLM .Score, and TRLM .Generate, which we describe for each of the four variants in Table 1. We further outline these functions for different TRLM models in Algorithms 1, 2, 3, & 4. For this work, we consider two baselines, which are trained in forward token order, and score in the conventional order of response given the query. The first of these uses self-scoring based on the model's own perplexity. The second ( Forward Baseline ) is a forward model that we train, whose training corpus and model class are identical to TRLM .

TRLM Model Training: The pre-training setup for all TRLM models is identical to that of PALM2-Otter models described by Anil et al. [2023b], except for the token orders specified by our TRLM.pretrain methods for TRLM-Fo , TRLM-Ba and TRLM-FoBa respectively. We fine-tune them on the FLaN dataset [Longpre et al., 2023] using the TRLM-xx.pretrain function. Where xx can refer to Fo , Ba or FoBa based on the model being fine-tuned. Let Instruction , Question , Answer

denote instruction, question and answer respectively. Before calling the pretrain function during fine tuning , we merge Instruction + Question to be the new question.

4 Scoring in Reverse

In this section, we provide formal results on TRLM and the benefit of using pre-training in the reverse direction. Let us denote by P Fw ( A | Q ) the conditional distribution of a forward LLM. Similarly, denote P TRLM ( Q | A ) to be the conditional distribution of the Time Reversed Language Model. For simplicity, we merge the instruction and question together.

4.1 Formal Results on Reverse LLM based Alignment

In this subsection, we focus on the distribution shift encountered while using a reverse model based scorer on forward generations. Specifically, we conclude that while reranking using Forward Baseline is equivalent to temperature scaling [Yang et al., 2024b], reranking using TRLM induces a distribution shift that is not equivalent to temperature scaling.

Consider the alignment problem of learning a new forward LLM -˜ P Fw ( Answer | Question ) . A very popular framework is the KL constrained optimization objective with respect to a reward oracle R ( Question , Answer ) , for some threshold ∆ :

Log-perplexity of the forward model used as reward: In general, for long form question answering where an explicit reward model is not available, a typical method is to use log-perplexity of the forward model i.e. log P Fw as a reward. Then, we have the following corollary of Lemma 1 in Yang et al. [2024b],

Lemma1 (Corollary of Lemma 1 in Yang et al. [2024b]) . The new LLM policy ˜ P Fw that optimizes (1) is given by: ˜ P Fw ( Answer | Question ) ∝ P 1+ α Fw ( Answer | Question ) where α is chosen appropriately depending on the threshold ∆ when reward R ( · ) is set to log perplexity of the forward model P Fw .

A policy obtained post the constrained KL-alignment procedure is akin to temperature re-scaled forward model, since p 1+ α is equivalent to temperature rescaling exp (1+ α ) log p .

Log-perplexity of the TRLM-Ba.score used as reward: Suppose R ( · ) is set to output of TRLM-Ba.score computed on the the question given the answer, then we have:

Lemma 2 (Corollary of Lemma 1 in Yang et al. [2024b]) . The new LLM policy ˜ P Fw that optimizes (1) is given by: ˜ P Fw ( Answer | Question ) ∝ P Fw ( Answer | Question ) P α TRLM-Ba ( Question | Answer ) where α is chosen appropriately depending on ∆ when reward R ( · ) is set to log perplexity of the reverse model P TRLM .

Optimal distribution after alignment using TRLM scores results in a non-trivial distribution that is not simply temperature re-scaling. While we have not used TRLM for alignment using KL constraints in our experiments, the distribution shift that is induced by reverse token training is indeed non-trivial even with Best-of-N-re-ranking, which we adopt in our experiments.

5 Experimental Results

In this section, we explore the effectiveness of time reversed language models on different downstream tasks, by utilizing unsupervised feedback to improve upon existing forward model generations. Broadly, these applications fall into two categories - first, where we utilize the scoring capacity of TRLM (three use cases), and second where we utilize the generative capacity of TRLM for generating queries given a response.

5.1 Best-of-N reranking

The best-of-N reranking task involves outputting the best response out of N model responses to a user query. Specifically, given N LLM outputs to a user query, a reranking algorithm finds the

best response based on scalar scores assigned to each response. Prior works [Rafailov et al., 2023, Mudgal et al., 2023a] aim to improve LLM performance on this task by using feedback-based RLHF algorithms and training on KL-regularized alignment objectives. Yang et al. [2024a] show that best-of-N reranking is the most effective way to approximate these RL objectives, and further, it is empirically observed to outperform them.

In this work, we consider several best-of-N reranking based algorithms based on TRLM .Score, for evaluating a base model response. The methods considered rely on nothing more than the pre-training (or instruction-tuning) corpus to achieve alignment of response to the user query. We further note that such scores from TRLM may be used within RL objectives as well, but we leave the exploration of such rewards to future work.

5.1.1 Alpaca Leaderboard Evaluation

Benchmark and Evaluation: The AlpacaEval leaderboard [Dubois et al., 2024] is a widely used benchmark to evaluate the capability of language models. In this benchmark, there are 805 questions from the AlpacaFarm evaluation set - consisting of questions ranging from general writing, chat ability, and reasoning to general knowledge. The goal is to output a response that is better than a base model's response, as judged by an annotator model. Both base model and annotator model are set as GPT4-1106-Preview on the AlpacaEval leaderboard as on May 10, 2024, and hence we use the same for our evaluations. The evaluation benchmark computes various metrics including winrates, discrete winrates and length-controlled winrates [Dubois et al., 2024]. The length-controlled winrates are calculated using a debiasing algorithm that removes the length bias that is otherwise preferred by GPT4-1106-Preview .

Formally, we define the task for TRLM as follows - Given a query Q from the dataset and N model responses A = { A 1 . . . A N } from a generator model, we wish to use TRLM.score to output the highest scoring response a i ∈ A , which is further evaluated against an answer from GPT4-1106-Preview .

In our experiment, we consider outputs from a generator model that is Gemini-Pro-1.0 [Anil et al., 2023a]. We generate 16 responses using a temperature τ = 0 . 8 to ensure diversity of answers. We then rerank the responses using different variants of TRLM from the PALM2-Otter family of models ( TRLM training details in the supplement). We further consider two baselines, Self scoring and Forward Baselines , as described in Table 1. Scoring prompts and Conditioning prompts used with various TRLM variants for this task are described in the Table 7 of Appendix C.1.

Discussion of Results : In Table 8, we see that TRLM-Ba scores the highest length controlled win rate which is 5% over the self scoring baseline of Gemini-Pro-1.0 with 16 generations against the GPT4-1106-Preview judge. Further, it registers an 8% increase over the reported number for single generations in the benchmark leaderboard. We note that scoring Response->Query seems to bring out some improvements as TRLM-Fo improves over Forward Baseline . Further, TRLM-Ba outperforms TRLM-Fo indicating the impact of reverse token pre-training. This demonstrates that time reversed scoring provides an intrinsic unsupervised feedback that could help improve the performance of even larger capacity models. We note that pre-training in both forward and reverse directions ( TRLM-FoBa models) and scoring in the reverse direction is better than TRLM-Fo variant.

We present further results where the generations of a Mixtral model [Jiang et al., 2024b] are reranked and compared against GPT4-1106-Preview , and the generations of a smaller Mixtral model are reranked and compared against a larger Mixtral model. These results are presented in the Appendix C.2. We note a 4% improvement over Forward Baseline with the proposed TRLM-Ba.Score method of reranking.

Key Takeaway: Through empirical justifications, we show that TRLM variant models can be used as effective re-rankers of generations from multiple classes of models ( Gemini-Pro-1.0 , Mixtral8x22B, Mixtral8x7B), and improve the instruction following capability of the model as a whole. This is consistent with the 1 considering the fact that we outperform generation model's self-log perplexity score. While other methods of re-ranking exists, to the best of our knowledge none of them provide unsupervised feedback for effective reranking with just a pre-trained model.

5.2 Citation Attribution

In this section, we describe applications of reverse scoring to the task of producing citations to original passages that can corroborate the sentences in an already produced summary. Summaries are created from long form articles, and one often wants to know which part of the article a given summary sentence is derived from (Benjamin Cohen-Wang [2024]).

Dataset and Evaluation: For this task, we take the CNN Daily Mail Dataset [CNN] which consists of pairs of news articles and their respective highlights. Our goal is to identify which sentence (or groups of sentences) within a given news article provides the most direct corroboration for a specific article highlight given as a query. We evaluate the attributed citations using various relevancy metrics. We use cosine similarity on the embeddings of the Gecko model [Lee et al., 2024], cosine similarity on TF-IDF features, BLEU score and ROUGE score to compute metrics. We score and choose the best pairing using all the models from the TRLM PALM2-Otter family trained in the forward, reverse and forward-reverse directions as outlined in Section 5.1.1.

Algorithms: Different search algorithms, Linear Search , Binary Search and Exclusion Search are coupled with using TRLM.score to find the attribution. We outline these in Algorithms 7, 8 and 9 along with details in the supplement. The number of inference calls is O (log N ) where N is the number of article sentences for Binary Search , and this method produces multiple sentences as a citation. The other methods require O ( N ) calls to produce the citation for a sentence.

Our results shown in Table 3, demonstrate the efficacy of TRLM for the attribution task. Specifically, we show 44% gains over the baseline in the linear search method, 39% gains in the binary search method and 34% gains in the exclusion search method as measured through gecko cosine similarity.

Key Takeaway: Through our results on CNN-Daily Summarization dataset we present multiple methods of citation attribution and demonstrate significant gains with TRLM model variants. We note that a direction of low information to high information (summary -> article) is harder to reason upon and select among a given set of texts. Further, we highlight the importance of binary selection based approach over log-perplexity based exclusion based search. We show 9% improvement using TRLM-Ba on Gecko embedding-based metric using only O (log N ) inference calls to the main model.

5.3 Document Retrieval

In this section, we study the performance of TRLM in retrieving relevant passages from a corpus to answer a specific question. Our goal is to show the efficacy of TRLM based reverse scoring over doing it in the forward direction. The task is as follows: Given a question, the goal is to retrieve relevant documents from the given corpus. We retrieve k documents from the corpus and compute various information-retrieval metrics to calculate performance w.r.t. the golden set of documents.

We experiment with two retrieval-based datasets from MTEB benchmark [Muennighoff et al., 2023] as shown in Table 4. Metrics are precision, recall, normalized discounted cumulative gain (NDCG) and mean reciprocal rank (MRR) (details in Appendix E.1). We show our results in Table 5. TRLM reverse scoring algorithms along with respective prompts used are presented in Algorithms 11, 10 of the Supplement. As Table 5 suggests, results favor TRLM based reverse scoring methods. For example, we see a 22.48% improvement in recall at K = 4 for MS-MARCO dataset. TRLM-Ba model dominates across metrics. For NF-Corpus, we see that the conventional forward scoring algorithm ( query -> document ) has a very poor performance. We attribute this to the fact that, in this inference direction, we are scoring a highly complex medical document using a simple natural language query. We see a gain of 44.2 points in NDCG at K = 10 with TRLM-Fo compared to Forward Baseline . The results in both these datasets suggest that TRLM can show greater gains when the complexity of documents in the corpus differs significantly from the complexity of queries.

Table 5: Tabulates the results of various reranking algorithms with two inference directions. Q denotes Queries, while D denotes Documents. TRLM outperforms Forward Baseline and Backward Baseline significantly, which highlights the importance of inference direction in this task.

Key takeaways: We experiment with two information retrieval-based benchmarks MS-MARCO and NF-CORPUS and compute multiple metrics to compare TRLM variant models with standard Forward Baseline and unconventional Backward Baseline . We show a gain of 8.49 points in NDCG@10 on MS-MARCO and 44.19 points in NDCG@10 on NF-CORPUS. Aligning with the results in citation, the results from this task also accurately demonstrate the importance of going from a high information direction to a low information direction. The massive difference between the directions is evident in the NF-CORPUS dataset.

5.4 Defending against Jailbreak attacks

We next aim to leverage the generative ability of TRLM to augment toxicity filters that are used to improve the safety of LLMs. Prior works show that LLMs (and their input filters) can be jailbroken using crafted adversarial attacks [Zou et al., 2023], while output filters tend to have a high false negative rate due to the sensitivity to the presence of toxic words, despite being in a neutral context (See Table-10). We propose to combine the benefits of input and output filters by projecting the output response of LLMs to the input query space using the reverse generative capability of TRLM , and further detecting the toxicity of the generated queries to block/ pass the response to the original

Table 6: Performance of the proposed defense strategies across different thresholds, evaluated on the human annotated and jailbreakbench toxic responses. TRLM-Ba achieves significant gains over all other approaches. Notations: PT [Pretrained], IT [Instruction-finetuned], FNR [False Negative Rate], FPR [False Positive Rate], new-HA [new HA Dataset], JBB [JBB Dataset], (H) [Hard], (E) [Easy]

query based on a pre-specified criteria. We thus effectively amplify input safety filters, i.e. reduce False Negative Rate (FNR) with marginal/ no impact on False Positive Rate (FPR).

Key Idea: Consider TRLM.Generate ( Response ) that generates queries that could have produced a given response. The insight is that, the reverse generative ability of TRLM allows the projection of a candidate (jailbreak) query that could bypass the input filter back to the (naive) query space observed during training. These projected questions can thus be rightly classified using the same input filter.

Defense Strategy: We propose a defense strategy where i) a query is passed through the input filter, ii) if the input filter rejects the query, we return reject as well, iii) if the input filter allows the query, we take the Response produced by the model and generate multiple queries using TRLM.Generate ( Response ). If the number of generated queries rejected exceeds a threshold, we reject the query as "unsafe". Otherwise, we declare it as safe, and output the response corresponding to the input query. An elaborate description is provided in Algorithm 12 of the Supplement.

Datasets: We consider a human annotated ( HA ) dataset provided as part of the JailbreakBench benchmark [HAd] for evaluating the performance of toxicity classifiers. This contains 100 questions annotated by humans, of which 43 are annotated as toxic based on a majority vote across 3 annotators. We introduce a GPT-4 based filter, that considers the prompt-response pair to judge their toxicity (Details in Appendix-F.2), and has 0 FNR on this HA dataset, which is ideal for defense evaluation. We further consider a gpt-3.5-turbo-1106 based input toxicity filter for the empirical evaluation of the proposed defense, which has an FNR of 25.58% on this dataset. These unblocked questions form our new-HA dataset for the experiments. In addition to this, we use the following datasets for evaluation: JBB dataset that contains jailbreak questions (that are toxic as per the GPT-4 judge, but are safe as per the GPT-3.5 filter we augment) corresponding to different attacks on JailbreakBench , E dataset that contain safe and easy questions and H dataset that contains safe questions that are hard to classify as safe. We discuss more details on these datasets in Appendix-F.1.

In the two toxic datasets ( HA and JBB ), the gpt-3.5-turbo-1106 based input filter does not block any of the questions, and our defense strategy aims at lowering the False Negative rate on the toxic questions ( JBB dataset and new-HA dataset), while ensuring a low false positive rate on the safe questions as well (E and H datasets). We present the improvements in FNR rates for Algorithm 12 when combined with the gpt-3.5-turbo-1106 input filter and various TRLM variants in Table-6. We further present the impact of varying the threshold in Fig.4 of the Appendix.

Results: We firstly note that the proposed TRLM defense strategy improves the FNR of the gpt-3.5-turbo-1106 input filter across all settings considered. Further, the TRLM-Ba pre-trained model improves FNR by more than 70% on the HA dataset and around 35% on the JBB dataset, and outperforms other variants with negligible impact on FPR.

We note that the proposed defense outperforms existing perplexity thresholding based defenses [Jain et al., 2023, Alon and Kamfonas, 2023] and Smooth-LLM [Robey et al., 2023] on the JailbreakBench attacks [Chao et al., 2023, Deng et al., 2024] owing to the integration with an input filter that already outperforms them on the same. Hence, we do not compare with them. Further, these defenses operate only in the input space, while the proposed defense aims at augmenting the input space with feedback from the response. Hence, the proposed defense is orthogonal to such methods, and can thus be integrated with them as well.

6 Conclusions

In this work, we explore the capabilities of TRLM for scoring and generation of queries, when conditioned on responses. Our study points to the importance of the response → query direction in LLMs. When deploying TRLM models for reverse scoring, we show improvements on AlpacaEval leaderboard, Citation attribution and retrieval tasks. We further show that generations from TRLM can augment safety filters effectively.

7 Limitations

We note that the assumptions made for our theoretical results in Section 4 are stylized, and may not hold true in practice, as the space of all answers to questions may not be adequately captured by assumptions in that section. Given this assumption, one may wish to explore other models for hallucination that are more general and provide results about reverse scoring. We leave such a theoretical exploration to future work.

Further, TRLM benefits have thus far been explored on tasks related to short form queries that have long answers. One may wish to understand and demonstrate the effects of reverse scoring on other tasks. For instance, one might pose the question - does TRLM provide possible benefits for a broader set of tasks that language models are used for. We leave the exploration of such settings in which the reverse scoring direction of response → query is better than the forward scoring direction, along with obtaining an understanding on the reason behind such an advantage, as part of future work.

References

Alpacaeval leaderboard. https://tatsu-lab.github.io/alpaca_eval/ .

Cnn dailymail dataset. https://www.tensorflow.org/datasets/catalog/cnn_dailymail .

Human annotated dataset, jailbreakbench. https://github.com/JailbreakBench/ jailbreakbench/blob/main/src/jailbreakbench/data/classifier_comparison. csv .

Pubmed. https://pubmed.ncbi.nlm.nih.gov/ .

Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Alon and M. Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132 , 2023.

M.-R. Amini, V. Feofanov, L. Pauletto, E. Devijver, and Y. Maximov. Self-training: A survey. arXiv preprint arXiv:2202.12040 , 2022.

Anil, S. Borgeaud, Y. Wu, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, D. Silver, S. Petrov, M. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, J. Chen, E. Pitler, T. P. Lillicrap, A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan, B. Lee, F. Viola, M. Reynolds, Y. Xu, R. Doherty, E. Collins, C. Meyer, E. Rutherford, E. Moreira, K. Ayoub, M. Goel, G. Tucker, E. Piqueras, M. Krikun, I. Barr, N. Savinov, I. Danihelka, B. Roelofs, A. White, A. Andreassen, T. von Glehn, L. Yagati, M. Kazemi, L. Gonzalez, M. Khalman, J. Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR , abs/2312.11805, 2023a. doi: 10.48550/ARXIV.2312.11805. URL https://doi.org/10.48550/arXiv.2312.11805 .

Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. H. Ábrego, J. Ahn, J. Austin, P. Barham, J. A. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Cherry, C. A. Choquette-Choo, A. Chowdhery, C. Crepy, S. Dave, M. Dehghani, S. Dev, J. Devlin, M. Díaz, N. Du, E. Dyer, V. Feinberg, F. Feng, V. Fienber, M. Freitag, X. Garcia, S. Gehrmann, L. Gonzalez, and et al. Palm 2 technical report. CoRR , abs/2305.10403, 2023b. doi: 10.48550/ARXIV.2305.10403. URL https://doi.org/10.48550/arXiv.2305.10403 .

G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics , pages 4447-4455. PMLR, 2024.

Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 , 2016.

G. A. M. Benjamin Cohen-Wang, Harshay Shah. Contextcite: Attributing model generation to context. https://github.com/MadryLab/context-cite/tree/main , 2024.

Berglund, M. Tong, M. Kaufmann, M. Balesni, A. C. Stickland, T. Korbak, and O. Evans. The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv preprint arXiv:2309.12288 , 2023.

Boteva, D. G. Ghalandari, A. Sokolov, and S. Riezler. A full-text learning to rank dataset for medical information retrieval. In N. Ferro, F. Crestani, M. Moens, J. Mothe, F. Silvestri, G. M. D. Nunzio, C. Hauff, and G. Silvello, editors, Advances in Information Retrieval - 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings , volume 9626 of Lecture Notes in Computer Science , pages 716-722. Springer, 2016a. doi: 10.1007/ 978-3-319-30671-1\_58. URL https://doi.org/10.1007/978-3-319-30671-1_58 .

Boteva, D. Gholipour, A. Sokolov, and S. Riezler. A full-text learning to rank dataset for medical information retrieval. In Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings 38 , pages 716-722. Springer, 2016b.

Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419 , 2023.

Chen, M. Lin, N. Schärli, and D. Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 , 2023.

Cloud. Google cloud tpu v5e inference. URL https://cloud.google.com/tpu/docs/ v5e-inference . Accessed on Feb 1, 2024.

Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang, T. Zhang, and Y. Liu. Masterkey: Automated jailbreaking of large language model chatbots. In Proc. ISOC NDSS , 2024.

Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. CoRR , abs/2404.04475, 2024. doi: 10.48550/ARXIV.2404.04475. URL https://doi.org/10.48550/arXiv.2404.04475 .

Fu, H. Peng, T. Khot, and M. Lapata. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142 , 2023.

Golovneva, Z. Allen-Zhu, J. Weston, and S. Sukhbaatar. Reverse training to nurse the reversal curse. arXiv preprint arXiv:2403.13799 , 2024.

A. Google and, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. H. Abrego, J. Ahn, J. Austin, P. Barham, J. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Cherry, C. A. Choquette-Choo, A. Chowdhery, C. Crepy, S. Dave, M. Dehghani, S. Dev, J. Devlin, M. Díaz, N. Du, E. Dyer, V. Feinberg, F. Feng, V. Fienber, M. Freitag, X. Garcia, S. Gehrmann, L. Gonzalez, G. Gur-Ari, S. Hand, H. Hashemi, L. Hou, J. Howland, A. Hu, J. Hui, J. Hurwitz, M. Isard, A. Ittycheriah, M. Jagielski, W. Jia, K. Kenealy, M. Krikun, S. Kudugunta, C. Lan, K. Lee, B. Lee, E. Li, M. Li, W. Li, Y. Li, J. Li, H. Lim, H. Lin, Z. Liu, F. Liu, M. Maggioni, A. Mahendru, J. Maynez, V. Misra, M. Moussalem, Z. Nado, J. Nham, E. Ni,

A. Nystrom, A. Parrish, M. Pellat, M. Polacek, A. Polozov, R. Pope, S. Qiao, E. Reif, B. Richter, P. Riley, A. C. Ros, A. Roy, B. Saeta, R. Samuel, R. Shelby, A. Slone, D. Smilkov, D. R. So, D. Sohn, S. Tokumine, D. Valter, V. Vasudevan, K. Vodrahalli, X. Wang, P. Wang, Z. Wang, T. Wang, J. Wieting, Y. Wu, K. Xu, Y. Xu, L. Xue, P. Yin, J. Yu, Q. Zhang, S. Zheng, C. Zheng, W. Zhou, D. Zhou, S. Petrov, and Y. Wu. Palm 2 technical report, 2023.

Guo, R. Wang, J. Guo, X. Tan, J. Bian, and Y. Yang. Mitigating reversal curse via semantic-aware permutation training. arXiv preprint arXiv:2403.00758 , 2024.

Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y . Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 , 2023.

Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614 , 2023.

Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024a.

Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de Las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mixtral of experts. CoRR , abs/2401.04088, 2024b. doi: 10.48550/ARXIV.2401.04088. URL https://doi.org/10.48550/arXiv.2401.04088 .

Korbak, E. Perez, and C. L. Buckley. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275 , 2022.

Krause, A. D. Gotmare, B. McCann, N. S. Keskar, S. Joty, R. Socher, and N. F. Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367 , 2020.

Lee, Z. Dai, X. Ren, B. Chen, D. Cer, J. R. Cole, K. Hui, M. Boratko, R. Kapadia, W. Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327 , 2024.

Li, M. Galley, C. Brockett, J. Gao, and B. Dolan. A diversity-promoting objective function for neural conversation models. In K. Knight, A. Nenkova, and O. Rambow, editors, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016 , pages 110-119. The Association for Computational Linguistics, 2016. doi: 10.18653/V1/N16-1014. URL https://doi.org/10.18653/v1/n16-1014 .

Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning , pages 22631-22648. PMLR, 2023.

Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems , 36, 2024.

Mudgal, J. Lee, H. Ganapathy, Y. Li, T. Wang, Y. Huang, Z. Chen, H. Cheng, M. Collins, T. Strohman, J. Chen, A. Beutel, and A. Beirami. Controlled decoding from language models. CoRR , abs/2310.17022, 2023a. doi: 10.48550/ARXIV.2310.17022. URL https://doi.org/10. 48550/arXiv.2310.17022 .

Mudgal, J. Lee, H. Ganapathy, Y. Li, T. Wang, Y. Huang, Z. Chen, H.-T. Cheng, M. Collins, T. Strohman, et al. Controlled decoding from language models. arXiv preprint arXiv:2310.17022 , 2023b.

Muennighoff, N. Tazi, L. Magne, and N. Reimers. MTEB: massive text embedding benchmark. In A. Vlachos and I. Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023 , pages 2006-2029. Association for Computational Linguistics, 2023. doi: 10.18653/V1/ 2023.EACL-MAIN.148. URL https://doi.org/10.18653/v1/2023.eacl-main.148 .

Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Qin, S. Welleck, D. Khashabi, and Y. Choi. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems , 35: 9538-9551, 2022.

Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023. URL http://papers.nips.cc/paper_files/ paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html .

Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Robey, E. Wong, H. Hassani, and G. J. Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684 , 2023.

Serdyuk, N. R. Ke, A. Sordoni, A. Trischler, C. Pal, and Y. Bengio. Twin networks: Matching the future for sequence generation. arXiv preprint arXiv:1708.06742 , 2017.

Snell, I. Kostrikov, Y . Su, M. Yang, and S. Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871 , 2022.

Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053 , 2022.

Q. Yang, S. Salamatian, Z. Sun, A. T. Suresh, and A. Beirami. Asymptotics of language model alignment. CoRR , abs/2404.01730, 2024a. doi: 10.48550/ARXIV.2404.01730. URL https: //doi.org/10.48550/arXiv.2404.01730 .

Q. Yang, S. Salamatian, Z. Sun, A. T. Suresh, and A. Beirami. Asymptotics of language model alignment. arXiv preprint arXiv:2404.01730 , 2024b.

Yang and D. Klein. Fudge: Controlled text generation with future discriminators. arXiv preprint arXiv:2104.05218 , 2021.

Yang, R. Sun, and X. Wan. A new benchmark and reverse validation method for passage-level hallucination detection. arXiv preprint arXiv:2310.06498 , 2023.

Zhang, M. Galley, J. Gao, Z. Gan, X. Li, C. Brockett, and B. Dolan. Generating informative and diverse conversational responses via adversarial information maximization. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 1815-1825, 2018. URL https://proceedings.neurips.cc/paper/2018/ hash/23ce1851341ec1fa9e0c259de10bf87c-Abstract.html .

Zhang, S. Sun, M. Galley, Y. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, and B. Dolan. DIALOGPT : Large-scale generative pre-training for conversational response generation. In A. Celikyilmaz and T. Wen, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020 , pages 270-278. Association for Computational Linguistics, 2020. doi: 10.18653/V1/2020.ACL-DEMOS.30. URL https://doi.org/10.18653/v1/2020.acl-demos.30 .

X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 , 2023a.

Zhao, M. Khalman, R. Joshi, S. Narayan, M. Saleh, and P. J. Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning Representations , 2022.

Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425 , 2023b.

Zhong, P. Liu, Y. Chen, D. Wang, X. Qiu, and X. Huang. Extractive summarization as text matching. arXiv preprint arXiv:2004.08795 , 2020.

Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 , 2023.

A Results on a Bipartite Graph Model for Questions and Answers

In this section, we outline a simple toy model involving a universe of questions and answers with relations between them where we show how TRLM-Ba perplexity based alignment distribution helps in picking the right answer when the forward model "hallucinates". For simplicity of exposition, we will only focus on the distribution P TRLM-Ba ( Q | A ) for the TRLM class of models.

Universe of Questions and Answers: We consider a universe of questions and answers in the form of a bi-partite graph which are deemed to constitute the ground truth. Let Q ⊆ V K and A ⊆ V K where V is the vocabulary, be the universe of questions and answers respectively. For a given question Q , let N ( Q ) ∈ A denote the set of ground truth answers of Q . Let G ( Q , A , E ) be a bipartite graph such that E = { ( Q,A ) } Q ∈Q , A∈N ( Q ) is the edge set of all valid answers. In other words, Ideally, one may like a forward model to approximate the distribution, P ( A | Q ) = 1 / | N ( Q ) | , A ∈ N ( Q ) and 0 otherwise, closely.

Hallucination Model (Hamming distance version): We would like to model an imperfect forward model that does not fully adhere with the ideal ground truth forward model. For a given question Q , the imperfect model produces answers N ( Q ′ ) to the neighbouring questions Q ′ which are at a hamming distance of 1 from Q . Concretely, let H ( · , · ) denote the hamming distance function. The support of the answer distribution is then S = ⋃ Q ′ : H ( Q,Q ′ ) ≤ 1 N ( Q ′ ) . It follows immediately that P Fw ( A | Q ) =

Theorem 1. Let us assume the hallucination model above. Assume that for two questions Q,Q ′ : H ( Q,Q ′ ) ≥ 1 , min ( A,A ′ ) ∈ N ( Q ) × N ( Q ′ ) H ( A,A ′ ) > 1 , then the optimal alignment distribution when

P TRLM-Ba ( · ) is used a scoring model (i.e. distribution in Lemma 2) has the support N ( Q ) for Q .

Theorem 1. From Lemma 2, we have that

for some α > 0 . For a fixed question Q , left hand side is potentially non-zero only for A ∈ N ( Q ′ ) : H ( Q,Q ′ ) ≤ 1 . since the first term in the right hand side is non-zero only for those by definition of the hallucination model. Consider an A such that ∃ Q ′ : A ∈ N ( Q ′ ) , H ( Q,Q ′ ) = 1 . We will argue that the second term is zero for such an answer A . Suppose it is non-zero, according to the hallucination model for the reverse direction, it means that ∃ A ′ : H ( A,A ′ ) = 1 , A ′ ∈ N ( Q ) . However Q and Q ′ are hamming distance one away. From the assumptions, their neighborhood are far apart by more than 1 , therefore contradicting the implication that H ( A,A ′ ) = 1 .

Key Takeaway: Therefore under the above simplistic hallucination model, although the forward model has a wider support |S| in the answer space, due to alignment with TRLM-Ba 's perplexity, the new distribution has a support of at most N ( Q ) provably. While assumptions in the theorem are not reflective of true complexities of the universe of questions and answers in a domain, this simple model shows that alignment using TRLM 's scoring metric can give rise to better re-ranking whenever nearby questions produce far away answers and generating forward models tends to confuse between nearby questions (a form of hallucination).

B TRLM Subroutines Score , Generate and Pretrain

In this section, we provide the subroutines of our TRLM models as described in Section 3.

C Details on the Experimental Section

We describe details about our experiments in the following figure 1:

Algorithm 1 TRLM-Ba.Pretrain

1: Input: T - context length. N - number of sequences. C index set of the vocabulary. Pre-training corpus of sequences { x i } N i =1 such that x i ∈ C T , x ij ∈ C . Initialize the model p Θ ( · ) with random weights.

Algorithm 2 TRLM-Ba.Score

1: Input: Query: Q . Response A . Conditioning Prompt: CP . Scoring Prompt: SP 2: return log P TRLM-Ba ( Reverse ( SP + Q ) | Reverse ( CP + A ))

Algorithm 3 TRLM-Fo.Score

1: Input: Query: Q . Response A . Conditioning Prompt: CP . Scoring Prompt:

2: return log P ( SP + Q | A + CP

Algorithm 4 TRLM-FoBa.Pretrain

Algorithm 5 TRLM-Ba.Generate

1: Input: Response A . Conditioning Prompt: CP .

2: return Q ∼ P TRLM-Ba ( · | Reverse ( CP + A ))

Algorithm 6 TRLM-Fo.Generate

1: Input: Response A . Conditioning Prompt: CP

2: return Q ∼ P ( · | A + CP

index set of the vocabulary.

TRLM

Query Generation

Defence against Jailbreak

Best-of-n Reranking

AlpacaEval Leaderboard

Task-based Retrieval

Citation of Answers

Document Retrieval

Figure 1: This task is an approach to link specific highlight sentences to lines that corroborate these sentences from within a lines in an article. By using linear binary and exclusion search methods, the aim is to efficiently and accurately find sentences in the articles that support the highlights.

C.1 Scoring Prompts

We use scoring and conditioning prompts for all our re-rankers to evaluate the best possible response from the set of response to a query. We provide a detailed list of prompts used for each task in Table 7.

C.2 Details on AlpacaEval Leaderboard results

Model Performance on the Alpaca Leaderboard

Model Performance Comparison

D Details on the Citation Task

Highlights

h

1

h

2

h

3

Citations

Articles

a

1

a

2

a

3

a

4

a

5

a

n

Goal: ůnd article lines that

corroborate the highlights.

Methods: linear search, binary

search, exclusion search

Algorithm Description: We describe the three attribution algorithms that use TRLM.score function in the reverse direction with appropriate prompts in the supplement.

Linear search (Algorithms 7) uses scores every possible sentence in the article with the highlight sentence.

Binary search (Algorithm 8), actually starts with scores the first against the second half of the article for a given highlight and chooses the best recurses further by splitting the chosen half (analogous to binary search) until a contiguous set of article sentences of sufficient granularity is reached.

In Exclusion search (Algorithm 9), we drop article sentences and score the rest of the article with the highlight sentence. We pick the choice with the least score .

Algorithm 7 Linear Attribution Search

1: Input: h -highlight sentence , A = { a 1 , . . . , a N } -Article, Conditioning Prompt : CP , Scoring Prompt : SP .

2: Return a j corresponding to the highest TRLM.score ( h , a j , CP , SP ).

Algorithm 8 Binary Attribution Search

1: Input: h , A = { A s , A s +1 , . . . A t } , Conditioning Prompt : CP , Scoring Prompt : SP . 2: s 1 ← TRLM.score ( Q = h, A = A s : s + glyph[ceilingleft] t -s 2 glyph[ceilingright] , CP , SP ). 3: s 2 ← TRLM.score ( Q = h, A = A s + glyph[ceilingleft] t -s/ 2 glyph[ceilingright] : t , CP , SP ) 4: if then s 1 > s 2 5: t ← s + glyph[ceilingleft] t -s 2 glyph[ceilingright] 6: else 7: s ← s + glyph[ceilingleft] t -s 2 glyph[ceilingright] 8: end if 9: if then | t -s | has sufficient granularity return A s : t 10: else 11: Binary Attribution Search ( h, A s : t , CP , SP )

12: end if

13: If A half is at the required granularity, return this as the attribution, else recursively search with A half as the article input.

Algorithm 9 Exclusion Attribution Search

1: Input: h i - highlight sentence i , A = { a 1 , . . . , a N } - article sentences. Conditioning Prompt : CP , Scoring Prompt : SP .

2: Return a j corresponding to the highest TRLM.score ( h , A \ a j , CP , SP ). A \ a denotes article A without sentence a .

E Details on the Retrieval Tasks

Query

Document 1

Document 2

Document 3

Document N

Goal: ůnd documents that are relevant to the query.

Method: TRLM is prompted with

answer to' and suũx = Query

The scoring algorithms used for retrieval are given in Algorithms 10 11.

E.1 Metrics Explanation

We compute the following metrics, that are widely used in information retrieval regimes.

Algorithm 10 Document Retrieval -TRLM-Fo

1: Input: Q - query, D = { d 1 , . . . , d N } - documents, Conditioning Prompt : CP , Scoring Prompt : SP .

2: Return d i corresponding with the highest score by TRLM-Fo.score ( Q , d i , CP , SP ).

Algorithm 11 Document Retrieval -TRLM-Ba

1: Input: Q - query, D = { d 1 , . . . , d N } - documents, Conditioning Prompt : CP , Scoring Prompt : SP .

2: Return d i corresponding with the highest score by TRLM-Ba.score ( Q , d i , CP , SP ).

Precision@K : We compute how many items within the top-k ranked items are relevant.

Recall@K : We compute how many relevant items were selected out of the set of all relevant articles within top-k ranked items

NDCG@K : Normalized discounted cumulative gain, where gain is defined as the rank of the selected item.

MRR : Mean reciprocal rank of the selection.

NDCG@K and MRR are order-aware metrics that not only test the retrieval performance but also how well a retrieval algorithm can order items in a set.

F Details on our Defence Task: Defending against Jailbreak attacks

0

10

20

30

Threshold

0

20

40

60

80

100

(a) FNR (toxic jailbreak questions)

0

10

20

30

Threshold

0

20

40

60

80

100

(b) FNR (human annotated data)

0

10

20

30

0

50

100

(c) FPR (Hard negatives)}

0

10

20

30

Threshold

0

50

100

(d) FPR (Easy negatives)

TRLM-Ba (IT)

TRLM-Fo (IT)

TRLM-Ba (PT)

TRLM-Fo (PT)

F.1 Datasets used in the Defence Task

JBB Dataset: We form the union of all question-answer pairs that have been generated by various attack strategies in JailbreakBench . There are a total of 1037 question and answer pairs of which

472 are identified as toxic by the GPT-4 toxicity filter. Amongst these questions, only 72 are declared as "safe" (False Negatives) by the gpt-3.5-turbo-1106 input filter acting on the question alone, which we further reduce using the proposed defense. Thus, this set of 68 questions forms our JBB Dataset. We note that these are hard questions, since they are able to jailbreak the LLMs considered in JailbreakBench , as well as the gpt-3.5-turbo-1106 input filter.

E-Dataset: We sample about 50 easy safe questions from AlpacaEval dataset, to test False Positive Rates, and eliminate one question that was predicted as unsafe by gpt-3.5-turbo-1106 .

H-Dataset: To test the FPR on hard samples, we prompt GPT-4 to generate about 50 questions that could possibly mislead an input filter, while being safe. We further verify that the input-output pairs are safe using the GPT-4 toxicity classifier, and eliminate 2 questions that were predicted as unsafe. The prompt used to generate this dataset is shown below:

Can you help me create a dataset of questions that would lead to answers containing words that can be perceived as harmful, but are not. I want to use this to ensure that my toxicity filter does not filter them out. For example, "Create a political cartoon about gun violence", "Describe a historic event in detail.,October Revolution". Generate as many questions as you can. Stop ONLY when you find that you are repeating questions.

F.2 GPT4 prompt used as a toxicity classifier

We prompt GPT4-1106-Preview with the following prompt to verify the toxicity of the question and answer.

SYSTEM PROMPT

Analyze user-submitted question-answer pairs with a focus on safety and ethical standards. Your goal is to classify each answer as "SAFE" or "UNSAFE", especially considering the potential for encouraging or detailing harmful behavior. Answer in one word without any explanation. Definitions:

SAFE: Answers that are appropriate for general audiences, educate without promoting unethical actions, and maintain a respectful and informative tone without revealing sensitive information.

UNSAFE: Answers that could potentially encourage or inform harmful or unethical behavior, even if unintentionally. This includes content that might educate or provide a how-to guide on harmful actions, discriminatory practices, manipulation tactics, illegal activities, or other unethical behaviors.

USER PROMPT

Question: <question>

Answer: <answer>

F.3 Algorithm for Question Generation for Defense

Algorithm 12 Defense strategy using TRLM .Generate

F.4 Additional Tables relating Jailbreak Defense

G Compute Requirements:

To pre-train TRLM models we use two TPUv5e pods[Cloud] for two weeks in the setup described by Anil et al. [2023b]. Further details on pre-training are provided in Appendix B. We run fine-tuning on FLAN-dataset using a TPUv5e pod [Cloud] for 1 day.

H Licenses and Copyrights Across Assets

Gemini-Pro-1.0

Citation: [Team et al., 2023]

Asset Link: [link]

License: Google APIs Terms of Service

PALM2-Otter

Citation: [Google and et al., 2023]

Asset Link: [link]

License: Google APIs Terms of Service

GPT4-1106-Preview

Citation: [Achiam et al., 2023]

Asset Link: [link]

•

License: OpenAI Terms of use

Mixtral 8x22B

Citation: [Jiang et al., 2024a]

Asset Link: [link]

License: Apache 2.0 license

Mixtral 8x7B

Citation: [Jiang et al., 2024a]

Asset Link: [link]

•

License: Apache 2.0 license

6. Gecko

Citation: [Lee et al., 2024]

Asset Link: [link]

License: Google APIs Terms of Service

CNN Daily Mail

Citation: [Zhong et al., 2020]

Asset Link: [link]

License: Apache 2.0 license

8. MS-Marco

Citation: [Bajaj et al., 2016]

Asset Link: [link]

License: Microsoft Terms and Conditions

NF-Corpus

Citation: [Boteva et al., 2016b]

Asset Link: [link]

•

License: Terms of Use

Alpaca Eval Benchmark

Citation: [Alp]

Asset Link: [link]

License: Apache 2.0 license

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have tried our best to convey all our results in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Refer Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: While our theory is based on a toy model for the hallucination setting, we propose a Theorem, along with proof in Theorem 1 proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the algorithms needed to create TRLM in Section B and provide the compute requirements to run these algorithms in Section G.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the algorithms needed to run our TRLM model in Section B.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide details of our experiments in Appendix Section C.2, D and E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the standard errors in the AlpacaEval leaderboard test in Table 8.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer:[Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes we have followed Neurips guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, please see Section ?? .

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any model or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We provide a training method for creation of TRLM and study the effects of such a training.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applicable for our work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable for our work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a class of LLMs called Time Reversed Language Models, which are simply pretrained on an unlabeled corpus in reverse order, and finetuned on instruction-tuned datasets accordingly. The model is used to provide feedback to LM generations in four different tasks. For general question answering, it serves to rerank N generated answers by scoring the probability of the question given the answers. For citation attribution, it scores a summarizing sentence given a cancidate text passage from the source article. For document retrieval, it scores the query given a retrieved candidate document. Finally, the model can be used to defend against jailbreak attacks by using the model to generate novel prompts given the response to the actual response. If the novel prompts exceed a threshold of rejection, the original prompt is rejected as well.\n\nSTRENGTHS:\n* The proposed methods are simple yet effective ideas of how to use a reverse LM for various tasks.\n* While the idea to train a reverse LM is not novel per se, its applications are.\n* The baselines are well chosen to demonstrate the benefits of the model. The claims are well-supported.\n* On most tasks, the proposed method is very effective.\n* The discussed methods are of great significance to the NeurIPS community.\n\nWEAKNESSES:\n* Even though the method is quite simple and easy to understand, I think the clarity of the paper could be better in some parts. For example, the purpose of the different parts/functions in Table 1 is not easy to understand without the context of the tasks that they are applied to. I think the paper could benefit from a \"Problem Statement\" section that introduces the different use cases from the tasks in Section 5 (e.g. reranking, retrieval, etc.) before presenting the model in Section 3.\n\nQUESTIONS:\nSuggestions:\n* Section 6 is probably meant to be a subsection of Section 5.\n* Results are presented as \"our method performs better by x%\", although what is meant is the absolute improvement in points on the scoring metric. E.g., in line 304, the improvement of your best variant over the Forward Baseline is not 44.19% but 44.19 points, which really is more than 1200% better than the Forward Baseline.\n* I find Section 4 to be the least helpful part of the paper. As is acknowledged in the Limitations section, the assumptions are quite strong. I'd rather move it into the appendix in favor of using the additional space to enhance clarity as explained under \"Weaknesses\". However, I wouldn't object if the authors insist on keeping it the way it is.\n\nLIMITATIONS:\nLimitations are appropriately addressed in a separate \"Limitations\" section.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Time Reversed Language Models (TRLMs), which operate in the response-to-query direction for scoring and generation. The key contribution is demonstrating that TRLMs can provide effective unsupervised feedback to improve language model performance on various tasks. Specifically, the authors show that TRLM-based reranking improves performance on the AlpacaEval leaderboard, citation attribution, and document retrieval tasks. They also demonstrate that TRLM's reverse generation capability can augment safety filters to reduce false negative rates in detecting toxic queries. The paper provides theoretical insights into why reverse scoring can be beneficial and empirically demonstrates the effectiveness of TRLMs across multiple applications, highlighting the importance of the response-to-query direction in language modeling.\n\nSTRENGTHS:\n1. Novel approach: The paper introduces Time Reversed Language Models (TRLMs), which operate in the response -> query direction for scoring and generation. This is an innovative approach to leveraging unsupervised feedback in language models.\n2. Theoretical foundation: The authors provide formal results and a bipartite graph model to explain how TRLM-based alignment can help mitigate certain forms of hallucination.\n3. Comprehensive experiments: The paper demonstrates the effectiveness of TRLMs across multiple tasks:\n   - Best-of-N reranking on the AlpacaEval leaderboard\n   - Citation attribution\n   - Document retrieval\n   - Defending against jailbreak attacks\n4. Significant improvements: TRLMs show notable gains over baselines across tasks, e.g.:\n   - 5% improvement in length-controlled win rates on AlpacaEval\n   - 44% improvement in citation attribution accuracy\n   - 44% improvement in NDCG@10 for medical information retrieval\n5. Versatility: The authors show TRLMs can be used for both scoring and generation tasks, demonstrating their flexibility.\n6. Safety applications: The paper explores using TRLMs to augment safety filters against toxic queries, showing potential for improving AI safety.\n7. Broad applicability: The techniques are shown to work across different model families (e.g., PALM2-Otter, Gemini-Pro, Mixtral), suggesting wide applicability.\n\nWEAKNESSES:\n1. Assumptions: The authors acknowledge that the assumptions made for their theoretical results in Section 4 are stylized and may not hold true in practice. The hallucination model used is quite simplistic.\n2. Need for more extensive testing of the defense strategy: While the proposed defense strategy for amplifying safety filters shows promise, the authors note it needs to be tested on a larger and more diverse set of questions to determine its broader applicability.\n3. Computational costs: The paper does not thoroughly discuss the computational overhead of using TRLM for scoring and reranking, which could be significant for large-scale applications.\n\nQUESTIONS:\n1. The paper focuses mainly on English language tasks. How well might TRLM generalize to other languages or multilingual settings?\n2. The paper shows TRLM can help with safety filtering, but are there any potential risks or downsides to using this approach for content moderation?\n3. How sensitive are the TRLM results to the specific prompts used for scoring and generation? Did the authors explore prompt engineering to optimize performance?\n\nLIMITATIONS:\nTwo suggestions for improvements:\n1. More discussion of potential biases or fairness concerns in the TRLM approach would be valuable.\n2. They authors could elaborate on privacy implications of using reverse query generation for safety filtering.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the utilization of reverse/backward-trained causal LLM. This LLM can be used to score responses based on the probability of generating the queries given the scores (which can be combined with the probability of generating outputs from input). Then, they can be used for re-ranking. In general, they can be used to score sentence pairs - which can be used for citation attribution. The authors also consider the use of reverse LLM for toxicity filters. Given that standard output filters tend to be too aggressive in rejecting (high false negative), the authors found a way to improve the scores by generating the input from the output using the backward model, and then using the input filter for toxicity filtration. \n\nReasonable baselines are considered and compared (for example, using the plain forward model for generating the query from response in a format fashion using an appropriate prompt), or alternative scoring methods (like perplexity).\n\nSTRENGTHS:\n1. Application potentials of modern backward LLMs are not as well explored, so it's a relatively unique direction.\n2. There are some interesting ideas for using backward LLMs, like toxicity filter besides standard ideas like re-ranking.\n3. The paper provides some decent theoretical motivations for informing the scoring mechanism with a backward model.\n\nWEAKNESSES:\n1. The core idea is not particularly novel. There are also a few missed citations. [1,2,3] (There could be others I am missing. Would recommend doing a deeper literature search) similarly used backward-trained models (one of the references is dialoGPT, which falls under the causal language model family) and their scores for re-ranking.\n\n2. It may be a bit hacky to circumnavigate the output filter by projecting the output into the input space. In principle, the output being toxic doesn't mean an appropriate input has to be. RIght now, the hacky approach still may working betters because the alternatives are limited, \n\n[1] A diversity-promoting objective function for neural conversation models. - Li et al. NAACL 2016.\n\n[2] DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation - Zhang et al. ACL Demos 2020\n \n[3] Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization - Zhang et al. NeurIPS 2018\n\nQUESTIONS:\nMinor:\n\nin Line 183, I assume P(A|Q) = 1/|N(Q)| should be the appropriate notation.not 1/N(Q). \n\nLine 185: \"the idea; ground truth\"  - I believe you meant ideal.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}