Time-Reversal Provides Unsupervised Feedback to LLMs
Authors: Yerram Varun
Rahul Madhavan
Sravanti Addepalli
Arun Suggala
Karthikeyan Shanmugam
Prateek Jain
Venue: NeurIPS 2024 spotlight (2024)
Keywords: LLMs
Reranking
reverse LLMs
reverse scoring
defenses
generative models
sequence reversal
Primary area: generative_models
TL;DR: Reverse Scoring and generation provides unsupervised feedback to LLMs

Abstract: Large Language Models (LLMs) are typically trained to predict in the forward direction of time. However, recent works have shown that prompting these models to look back and critique their own generations can produce useful feedback. Motivated by this, we explore the question of whether LLMs can be empowered to think (predict and score) backwards to provide unsupervised feedback that complements forward LLMs. Towards this, we introduce Time Reversed Language Models (TRLMs), which can score and generate queries when conditioned on responses, effectively functioning in the reverse direction of time. Further, to effectively infer in the response to query direction, we pre-train and fine-tune a language model (TRLM-Ba) in the reverse token order from scratch. We show empirically (and theoretically in a stylized setting) that time-reversed models can indeed complement forward model predictions when used to score the query given response for re-ranking multiple forward generations. We obtain up to 5\% improvement on the widely used AlpacaEval Leaderboard over the competent baseline of best-of-N re-ranking using self log-perplexity scores. We further show that TRLM scoring outperforms conventional forward scoring of response given query, resulting in significant gains in applications such as citation generation and passage retrieval. We next leverage the generative ability of TRLM to augment or provide unsupervised feedback to input safety filters of LLMs, demonstrating a drastic reduction in false negative rate with negligible impact on false positive rates against several attacks published on the popular JailbreakBench leaderboard.

Teve-Rlversal Provides Unsupervised Feedback to LLMs
Varun Yerram Google DeepMind
Rahul Madhavan Indian Institute of Science
Sravanti Addepalli Google DeepMind
Arun Suggala Google DeepMind
Karthikeyan Shanmugam Google DeepMind
Prateck Jain Google DeepMind
Abstract
Large Language Models (LLMs) are typically trained predict in the forward direction of time. However, recent works have shown that prompting these models to look back and critique their own generations can produce useful feedback. Moti- vated by this, we explore the question of whether LLMs can be empowered to think (predict and score) backwards to provide unsupervised feedback that complements forward LLMs. Towards this, introduce Time Reversed Language Models (TRLMs), which can score and generate queries when conditioned on responses, effectively functioning in the reverse direction of time. Further, effectively infer in the response to query direction, we pre-train and fine-tune a language model (TRLM-Ba) Ba)) in the reverse token order from scratch. We show empirically (and theoretically stylized setting) that time-reversed models can indeed comple- ment forward model predictions when used to score the query given response for re-ranking multiple forward generations. We obtain up to 5% improvement on the widely used AlpacaEval Leaderboard over the competent baseline of best-of-N re-ranking using self log-perplexity scores. We further show that TRLM scoring outperforms conventional forward scoring of response given query, resulting significant gains in applications such as citation generation and passage retrieval. We next leverage the generative ability of TRLM to augment or provide unsupervised feedback to input safety filters of LLMs, demonstrating a drastic reduction in false negative rate with negligible impact on false positive rates against several attacks published on the popular Jailbreak Benbhh leaderboard.
1 Introduction
Large Language Models (LLMs) trained on a large corpora of text are able to accomplish a wide variety of downstream tasks such as summarization, open-ended/ context-based question answering, document retrieval, and citation generation [Brown et al.) 2020 Zhao et al. [023a]]. While the gener- ations from pre-trained and instruction-tunae models already show significant promise, alignment techniques such as Reinforcement Learning via Human Feedback (RLHF) [Anil et al. 2023a Ouyang al.) 2022 widely used improve the quality of their generations further. However, these methods rely heavily on additional supervision to construct preference data, which can be expensive acquire, or noisy for training. This brings up natural question Can we generate useful feedback on LLM generations without additional supervised data?
"quual Contribution.
Work done part of Google Research
i Work done as a Student Researcher at Google Research
"Correspondence vyerram @ google.com, sravantia google.com, Karthikeyanss@ggoolle.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A recent line of work aims at specially prompting LLMs to review their own generations and generate meaningful natural language feedback, which can subsequently be used to refine them [Madaan et al.) 2024]. This process can be repeated to improve the generations iteratively. The success of such methods serves as an evidence that it indeed possible to obtain better responses without additional supervision. However, such methods rely on the superior instruction following and reasoning abilities of LLMs, which may not necessarily hold for low capacity models. Further, these methods involve sequential processing of the generated responses, and thus increase inference time significantly.
In this work, we propose natural method of enabling LLMs look backwards in order to obtain meaningful unsupervised feedback during inference. Towards this, we introduce a class of models that we call Time Reversed Language Models (TRLMs), which operate in the reversed direction of a regular LLM, or the time-reversed direction. Rather than predicting (or scoring) in the standard query - response direction, time reversed language models predict (or score) in the response + query direction. We first introduce TRLM-Fo a TRLM variant based on forward models, which are prompted to operate in the time-reversed direction using a prompt such as "Generate a question that would result in the following answer: <response>", Further, we extend the reversal to token-level granularity by pre-training LLMs from scratch in a reversed token direction, rather than the standard forward token direction. We call this as TRLM-Ba where Ba stands for Backward. Note that the inputs and outputs of such a model are in the reversed language order. Pre-training TRLM-Ba on reversed text exposes the model completely different world model where the conventional order of information is flipped. Introductions follow conclusions, questions follow answers, logical precedents follow their antecedents. Hence. such model may not only develop representations that are distinct from those of a regular L.MM - despite being trained on the same pre-training corpus - but may also be better suited score/ generate in the reverse direction, i.e. conditional the response.
We show in several use-cases that scoring and generation in this reverse direction can produce non-trivial feedback on the responses generated by forward LLMs. We consider three classes of tasks to showcase the scoring and generating capability of TRLM, viz. a) Reranking answers in open ended question answering b) Citation and retrieval tasks and c) Amplifying existing safety filters through query generation in the reverse.
Our Contributions:
a) We propose time reverse language models TRLM-Fo, TRLM-Ba and TRLM-FoBa, all of which score and generate queries given responses, enabling their use in obtaining unsupervised feedback on LLM generations. TRLM-Fo is a forward model prompted to predict in reverse, while TRLM-Ba is pre-trained in the reverse token order, enabling reverse prediction naturally. TRLM-FoBa is pre-trained in both reverse and forward token orders and can be used to predict in forward or reversed language. b) We demonstrate significant improvements when best-of-N reranking is applied to multiple LLM generations by using TRLM scores. Specifically, we show up to 5% improvement over self-reranking using TRLM- Ba, in LC win-rates (0.98 Pearson correlation with human preferences) reference model. We show multiple ablations on this study.
against a BPT4-1106-Prewe
c) We demonstrate that the reverse direction of scoring (response query) is highly significant, as it improves citation attribution accuracy by 44.15% when compared to the forward baseline on the CNN-Daily Mail dataset. Further, improve the NDCG@ 10 metric by 44.13 points on the NF-Corrus medical information retrieval benchmark, and obtain similar improvements on MS-Marco as well.
d) We show that the reverse generation capability of the TRLM models specifically TRLM-Ba, can be used to improve False Negative rate (FNR) of input safety filters with negligable impact on FPR. We show significant improvements on several attacks submitted to the Jailbreakbench benchmark, and on a Human Annotated dataset from JailbreakBench.
We complement these results with theoretical arguments using bipartite graph model between queries and responses, to show that RLHF done with TRLM-Ba scores induces a non trivial distribution shift in answers, mitigating primitive forms of "hallucination" under the defined conditions.
2 Related Work
Reverse Direction in Language Modeling: Classical work [Serdyuk et al. 2017 showed how
sequence to sequence models can regularize the current word token embedding based on the ability of the future tokens to be able to predict the current token. Such bi-directional (forward and reverse)
consistency checks have been used to improve forward models. Golovneva al. 20044] train LLM

---

in the forward direction first, followed by the reverse token direction, and show that this alleviates the reversal curse identified by Berglund et al. [023]. This work closely related to ours in that we also consider a variant of combining reverse and forward token order during training. Our key models differ from this, and are trained either forward (TRLM-Fo )/ reverse (TRLM-Ba ) token order, using which we demonstrate improvements in a wide range of applications such as long form question answering, citations, retrieval and augmenting input filters for defending against toxic questions. Yang et al. [0023] use question generation from a given answer combined with access to external databases to determine hallucination. Another recent work [Guo et al. 2024] also explores a different pre- pre-training order. While their focus is to correct causal ordering bias, our work instead is focused on the value that scoring and generation of these models bring to downstream tasks.
Several prior works [Li et al.) 2016 Zhang et al. 2018 2020] have proposed to improve the diversity of generated responses by optimizing the mutual information between the responses and the respective queries. These works motivate the need for better decoding strategies based on scores in both, response -> query and query + response directions. We theoretically show that reverse scoring alone, when used with forward generations, will achieve this naturally using a formal RLHF based argument (Lemma 2). and present strong empirical results across a wide range of tasks to support the same.
Reversed scoring:
Controlling Decoding through feedback: A broad line of works align a pre-trained model a reward model trained on human feedback by using Reinforcement learning (RL) techniques like Proximal Policy Optimization (PPO) [Stiennon et al. [220 Ouyang et al. 2022 Korbak et al. 2022].
(Identity policy optimization) IPO (and \/PO) | et al. 2024 J. Direct Preference Optimi iptimmization
[Rafallov et al., [024] and offline RI. [Snell al. 2022] Zhao et al. 2022 Zhao et al. (2023
calibrate likelihood of generated responses on a dataset with desired responses or human preference feedback. Krause et al. 2020, Yang and Klein 2021 Qin et al., 2022] control the generation of an LLM at test time by specifying constraint functions discriminators that operate in the token or logit space, encouraging certain attributes in the output. Using preference feedback, Mudgal et al. [202bb] train a prefix scorer model that acts as a value function over partial completions consistent with the preference rewards. Yang et al. [2024b] investigate the relation between best-of-N best-of---rrrankkin reranking and KL regularized RL objective. An observation made by Yang et al. [2024b] is that best-of-N-rerankee N-rerankinn dominates/ competes very well with most RL based alignment methods. Under certain assumptions, authors show formally that Dest-of-N-reranking approximates the optimal solution the regularized RL objective. We take inspiration from this and use Dest-of-N-reranking through unsupervised feedback by the reverse LLMss. Our work differs from all these in that they rely on external feedback to control generation, while our method does not.
to evaluate generations
Chen et al. 2023] explore how LLM can be prompted to self-debug based on an explanation of the code produced by the LLM during code generation and the execution output test cases. Welleck et al. [2022) use corrector model that is trained to prefer new corrected answer if the corrected answer has higher value that a default generation. They require access to value function for this determination. All these approaches use external feedback to align the model in their pipeline.
Self Play and Self Tuning:
Fu et al. 20233 explore L.M agents initialized as buyers and sellers to play a negotiating game of setting the price of a transaction. A critic LLM provides feedback to both the buyer and seller agents to improve. Madaan et al. (2024) propose self refining loop where the same model is prompted to provide feedback and further use the feedback to refine and regenerate. Both these works use very powerful and large models from the Claude, GPT-4, GPT-3.5 family to use self generated language feedback. Madaan et al. (2024) remark that the self refining approach does not work well with weaker models. In contrast, we focus on improving generation quality of much smaller models using unsupervised scalar feedback. Other prior works relating to self play are revicwed in the survey article by Amini al. 2022
3 TRLM Time Reversed Language Models
We introduce our primary contribution TRLM (Time Reversed Language Models), a class of language models that operate in the response - query direction during scoring and generation. This is achieved by either (a) [TRLM-Ba] reversing the token order and effectively utilizing previous token prediction instead of next token prediction during pre-training, scoring, and generation, or (b)
3

---

Table 1: Description of different TRLM model variants.
Model
Description
TRLM-Ba
Pre-trained in reverse token order for previous token prediction (Alg. the supplement). Instruction-tuned variant is FLaN fine-tuned [Longpre et al. [023] in reverse token order. Scores the reversed question given reversed answer combined with suitable prompts. Generates questions the reverse direction when conditioned answers in the reverse direction.
Scoring: Ba (Reverse(Scoring Prompt+Query)
Reverse(Condittioning
Prompt Answer)) (Alg. 2 in the supplement).
Generation: Pa
Roverse(Condi tinnidiiiiiing Prompt + Answer))
TRLM-Fo
Pre-trained in the usual forward token order. Scores Question given Answer using the prompt. Generates from the conditional distribution of answer.
Scoring: PP (uuery Answer Conditioning Prompt) (Alg. in the supplement)
Generation: P PraLX-P(
Answer + Conditioning Prompt)
TRLM-FoBa
Pre-trained both in forward and reverse token order (Alg. 4 in the supplement). Un-
(Reverse)
derstands text in both directions. Reverse version scores and generates identically TRLM-Ba.
Scoring: Scores identically to TRLM-Ba.
Generation: Generates identically to TRLM-Ba. Pre-trained both in forward and reverse token order. Forward version scores and generates identically TRLM-Fo.
TRLM-FoBa
(Forward)
Scoring: Scores identically TRLM-Fo.
Generation: Generates identically to TRLM-Fo.
Self Scoring
The model that is used for generating a given response is also used for scoring responses given queries in the conventional forward scoring direction. We use model's 's own perplexity scores as leedback to select the responses.
Scoring:
Forward Baseline
A conventional forward model trained for next-loken prediction on the same training corpus and model class as TRLM. Scoring: While self-scoring used the perplexity obtained from the generator model, this setting, we use perplexity of a different forward model.
[TRLM-Fo maintaining the standard token order during pre- pre-training but reversing the direction of generation through appropriate prompts during inference (scoring and generation).
We show that TRLM provides non-trivial unsupervised feedback that could be used by pre-trained, fine-tuned, and instruction tuned models, for various downstream tasks like reranking improve open-ended long-form question answering, generating citations, and retrieval. We demonstrate that the ability of TRLM to score in the reverse direction scoring query based on the response is essential to achieve the requisite gains. Further, TRLMs that are pre-trained in the reverse direction (TRLM-Ba BB ) provide additional boost in most cases. We further leverage the generative ability of TRLM in reverse (generating query from a response) to amplify the effectiveness of input safety filters as well.
We propose four variants of the TRLM class TRLM-Ba TRLM-Fo TRLM-FoBa (Reverse) and TRLM-FoBa (Forward)- - based on how they are pre-trained and fine-tuned. TRLM models can be considered to have three functions: describe for each of the four variants in Table [ We further outline these functions for different TRLM models in Algorithms I 2 & 4 For this work, we consider two baselines, which are trained in forward token order, and score in the conventional order of response given the query. The first of perplexity. The second (Forward Baseline) is
TRLM.Score, and TRLM.Generate, which we
these uses self-scoring based on the model's forward model that we train, whose training corpus and model class are identical to TRLM
The pre-training setup for all TRLM models is identical to that of PALM2-Otter models described by Anil et al. | (002b)), except for the token orders specified by our TRLM pretrain methods for FRLM-Fo.. TRLM-Ba Ba and TRLM-FoBa respectively. We fine-tune them on the Fl.aN dataset [Longpre et al. 2023) using the TRLM-xX. pretrain function. Where xx can refer to Fo, Ba or FoBa based on the model being fine-tuned. Let Instruction, Question, Answer
TRLM Model Training:

---

denote instruction, question and answer respectively. Before calling the pretrain function during fine tuning, we merge Instruction Question be the new question.
4 Scoring in Reverse
In this section, we provide formal results on TRLM and the benefit of using pre-training in the reverse direction. Let us denote by (A(Q) the conditional distribution of a forward LLM. Similarly, denote (Q)x((|A) to be the conditional distribution of the Time Reversed Language Model. For simplicity, we merge the instruction and question together.
4.1 Formal Results on Reverse LLM based Alignment
In this subsection, we focus on the distribution shift encountered while using a reverse model based scorer on forward generations. Specifically, we conclude that while reranking using Forward Baseline is equivalent to temperature scaling (Yang et al. [002bb, reranking using TRLM induces distribution shift that is not equivalent temperature scaling.
Consider the alignment problem of learning popular framework is the KL constrained optimization objective with respect reward oracle R(Question, Answer), for some threshold A:
forward LLM IPrv (Answer Question). A very
max Fre
E
(R(Question, Answer)) s.t. (P s A
(1)
Questioar AaeNerr-Po Question)
Log-pe perppeeeplx of the forward model used as reward: In general, for long form question answering where an explicit reward model not available, a typical method is to use log-perplexity of the forward model i.e. log LFFo reward. Then, we have the following corollary of Lemma 1 in Yang
ct al. 2024b].
Lemma (Corollary of Lemma 1 in Yang et al. 2024b) The new LLM policy Fru that optimizes given by: PP (Answer Question) xx Pl+a Answer (uessioo)) where a is chosen appropriately depending the threshold A when reward R(-) is log perplexity of the forward model Pro.
A policy obtained post the constrained KL-alignment procedure is akin to temperature re-scaled forward model, since p t- is equivalent to temperature rescaling exp(I+a) log
Log-perplexity of the TRLM-Ba. score used reward: Suppose R(-) is set output of TRLM-Ba. score computed on the the question given the answer, then we have:
Lemma (Corollary of Lemma in Yang et al. (024b)). The new LLM policy PPN that optimizes (1) given by: (Answer Question) x PF (Answer (neeston)PPass TaR...aaQQuustt (Question) Answer) where a is chosen appropriately depending on A when reward R(-) is set to log perplexity of the reverse model IP TRTLM.
Optimal distribution after alignment using TRLM scores results in toivialvial distribution that is not simply temperature re-scaling. While we have not used TRLM for alignment using KL constraints in experiments, the distribution shift that is induced by reverse token training indeed non-trivial even with Best-of-N-re-rankinng, which we adopt in our experiments.
Experimental Results
In this section, we explore the effectiveness of time reversed language models different downstream tasks, by utilizing unsupervised feedback to improve upon existing forward model generations. Broadly, these applications fall into two categories first, where we utilize the scoring capacity of TRLM (three use cases), and second where we utilize the generative capacity TRLM for generating queries given response.
5.1 Best-of-N reranking
The best-of-N reranking task involves outputting the best response out of model responses to query. Specifically, given N LLM outputs user query, reranking algorithm finds the
5

---

best response based on scalar scores assigned to each response. Prior works [Rafailov et al. 2023 Mudgal et al. 2023a] aim to improve LLM performance on this task by using algorithms and training on KL-regularized alignment objectives. Yang et al. (2024a] show that best-of-N reranking is the most effective way to approximate these RI. objectives, and further, empirically observed to outperform them.
RLHF
In this work, we consider several best-of-N reranking based algorithms based on TRLM.Score. for evaluating a base model response. The methods considered rely on nothing more than the pre-training (or instruction-tuie -unniig) corpus achieve alignment of response to the user query. We further note that such scores from TRLM may be used within RL objectives as well, but we leave the exploration of such rewards future work.
5.1.1 Alpaca Leaderboard Evaluation
The AlpacaEval leaderboard [Dubois et al. 2024 is a widely used benchmark evaluate the capability of language models. In this benchmark, there are 805 questions from the AlpacaFarm evaluation set - consisting of questions ranging from general writing, chat ability, and reasoning general knowledge. The goal is to output a response that better than base model's response, as judged by an annotator model. Both base model and annotator model are set as GPT4-1106-PPI Pe66-rreieww on the AlpacaEval leaderboard as on May 10, 2024, and hence we use the same for our evaluations. The evaluation benchmark computes various metrics including winrates, discrete winrates and length-controlled winrates [Dubois et al.) 2024). The length-controlled winrates are calculated using debiasing algorithm that removes the length bias that is otherwise preferred by 1PT4-1106-1 Peeei-e-..ee.....
Benchmark and Evaluation:
Formally, we define the task for TRLM as follows - Given a query Q from the dataset and N model re- sponses A {A1 ................................ AN} from a generator model, we wish use TRLM.score score to output the highest scoring response ag E A, which is further evaluated against an answer from P0PT--1106PPress Preview.
In our experiment, consider outputs from generator model that is 2023a]. We generate 16 responses using a temperature 7 = 0.8 to ensure diversity of answers. We then rerank the responses using different variants of TRLM from the PALM2-Otter family of models (TRLM training details in the supplement). We further consider two baselines, Self scoring and Forward Basel Maselines, described in Table [ Scoring prompts and Conditioning prompts used with various TRLM variants for this task are described in the Table Appendix C.1
(Anil et al.,
Discussion of Results: In Table 8 we see that TRLM-Ba scores the highest length controlled win rate which is 5% over the self scoring baseline of Gemini-Pro-1- O with 16 generations against the MPT4-1106-Previs
judge. Further, it registers an 8% increase over the reported number for single generations the benchmark leaderboard. We note that scoring Response->Quee bring out some improvements as TRLM-Fo improves over Forward Baseline. Further, TRLM-Ba outperforms Fo indicating the impact of reverse token pre-training. This demonstrates that time reversed scoring provides intrinsic unsupervised feedback that could help improve the performance of even larger capacity models. We note that pre-training in both forward and reverse directions (TRLM-FoBa models) and scoring in the reverse direction is better than TRLM-Fo variant.
seems to
We present further results where the generations of a Mixtral model [Jiang et al. 2024b) are reranked and compared against GPT4-1106-PES Prreiieeeeew..... and the generations of a smaller Mixtral model are reranked and compared against a larger Mixtral model. These results are presented in the Appendix C.2 We note 4% improvement over Forward Baseline with the proposed TRLM-Ba.C Score method of reranking.
Key Takeaway: Through empirical justifications, we show that TRLM variant models can be used as effective re-rankers of generations from multiple classes of models (Gemini-Pro-1.. Mixtral8x22B, Mixtral8x7B), and improve the instruction following capability of the model as whole. This consistent with the I considering the fact that we outperform generation model's self-log perplexity score. While other methods of re-ranking exists, to the best of our knowledge none of them provide unsupervised feedback for effective reranking with just a pre-trained model.

---

Table The best re-ranked response is compared with a single response OPT4-1106-Previes The setting is identical to the AlpacaEval Leader board. TRLM-Fo that scores in the backward direction, fares better than the conventional forward baseline. Scoring using TRLM-Ba (pretrained in reverse) gets even a higher (LCC) win rate.
Model Performance the Alpaca Leaderboard
Win Rate
Standard Error
Model
Inference Style
Wins
Losses
Ties
LC
Reg
Discrete
TRLM-Ba
Response -> Query 32.44
24.35
24.04
1.27
192
610
3
TRLM-FoBa (backward)
Re Response -> Query 31.18
22.72
21.99
1.24
176
627
2
TRLN-FoBa (forward)
Response -> Query 30.55
22.85
22.48
1.25
180
623
2
TRLN-Fo
Response -> Query
29.19
22.68
21.30
1.24
170
632
One Generation
24.38
18.18
17.08
1.16
135
Sclf
Query Response
27.05
17.66
17.14
1.15
136
Forward Baseline
Query Response
24.27
17.13
126
5.2
Citation Attribution
In this section, we describe applications of reverse scoring the task of producing citations to original passages that can corroborate the sentences an already produced summary. Summaries are created from long form articles, and one often wants to know which part of the article given summary sentence is derived from (Benjamin Cohen- Wang 2024)).
Dataset and Evaluation: For this task, we take the CNN Daily Mail Dataset [CNN] which consists of pairs of news articles and their respective highlights. Our goal is identify which sentence (or groups of sentences) within a given news article provides the most direct corroboration for specific article highlight given as query. We evaluate the attributed citations using various relevancy metrics. We use cosine similarity on the embeddings of the Gecko model [Lee et al. 2024], cosine similarity on TF-IDF features, BLEU score and ROUGE score to compute metrics. We score and choose the best pairing using all the models from the TRLM PALM2-Otten family trained in the forward, reverse and forward-reverse directions outlined in Section 5.1.1
Algorithms: Different search algorithms, Linear Search, Binary Search and Exclusion Search are coupled with using TRLM score find the attribution. We outline these in Algo- rithms and Ploong with details in the supplement. The number of inference calls is O(log N) where N the number of article sentences for Binary Search, and this method produces multiple sentences as citation. The other methods require O(N) calls to produce the citation for sentence.
Our results shown in Table demonstrate the efficacy of TRLM for the attribution task. Specifically, we show 44% gains over the baseline in the linear search method, 39% gains in the binary search method and 34% gains in the exclusion search method as measured through gecko cosine similarity.
Through our results on CNN-Daily Summarization dataset we present multiple methods of citation attribution and demonstrate significant gains with TRLM model variants. We note that a direction of low information to high information (summary -> article) is harder to reason upon and select among given of texts. Further, highlight the importance of binary selection based approach over log- pegppeeeexxtty based exclusion based search. We show 9% improvement using TRLM-BB Gecko embedding-baat based metric using only Oloo N) inference calls to the main model.
Key Takeaway:
Table 3: Tabulates the citation Attribution results through Re-ranking the CNN-Daily Mail dataset. A denotes article whereas S denotes the corresponding summary. The case of scoring summary given the article instead reverse clearly highlighted in all the search methods.
Inference Direction
LinearSearch TF IDF
Binary Search TF-IF
Exclusion Search IF-IDF
Model
Gecko
ROUGE
Gecko
ROUGE
Gecko
ROUGE
TRLM-Ba
A-28
53.16
55.45
49.12
45.09
50.93
42.11
36.33
46.34
36.13
TRLM-FoBa (Rev.
A->8
53.48
53.22
49.67
40.74
45.04
39.81
32.40
40.84
33.88
TRLM- (Forw.)
50.65
52.21
49.84
38.67
38.11
TRIM-
A-28
45.00
49.40
37.66
43.14
49.65
39.22
37.90
47.83
37.98
Forvard Baseline
9.33
11.06
6.66
4.66
7.00
Backvard Baseline
S->A
7.62
8.23
9.18
5.47
6,23
6.32
4.11
5.02
5.11

---

5.3 Document Retrieval
In this section, we study the performance of TRLM in retrieving relevant passages from a corpus to answer a specific question. Our goal is to show the efficacy of TRLM based reverse scoring over doing it in the forward direction. The task is as follows: Given question, the goal is to retrieve relevant documents from the given corpus. We retrieve l documents from the corpus and compute various information-retrieval metrics to calculate performance w.r.t. the golden set of documents.
Table 4: Summary of MS-Marco and NF-Corpus Datasets
Dataset
Description
MS-Marco
Contains 101.09k examples in its public dev split. Each example consists of a simple question along with 10 relevant passages. [Bajaj et al. [016]]
NE-Corpus
Medical information retrieval dataset with 323 queries in its test split and 3.6k total documents in the corpus. Queries are in simple English, and documents are extracted from PubMed with fair amount of medical terminology. (Boteva et al.,
2016a Pub
We experiment with two retrieval-basedd datasets from MTEB benchmark [Muennighoft et al. 2023
as shown in Table 4 Metrics are precision, recall, normalized discounted cumulative gain (NDCG) and mean reciprocal rank (MRR) (details in Appendix E.I). We show our results in Table 5 TRLM
reverse scoring algorithms along with respective prompts used are presented in Algorithms 11 10
of the Supplement. As Table [ suggests, results favor TRLM based reverse scoring methods. For 22.48% improvement in recall K
for MS-MARCO dataset. TRLM-Ba model dominates across metrics. For NF-Corpus, we see that the conventional forward scoring algorithm (query -> document) has very poor performance. We attribute this to the fact that, in this inference direction, we are scoring a highly complex medical document using a simple natural language query. We see gain of 44.2 points in NDCG K 10 with TRLM-Fo compared to Forward Baseline The results in both these datasets suggest that TRLM can show greater gains when the complexity of documents in the corpus differs significantly from the complexity of queries.
example,
Table 5: Tabulates the results of various reranking algorithms with two inference directions. Q de- notes Queries, while D denotes Documents. TRLM outperforms Forward Baseline and Backward Baseline significantly, which highlights the importance of inference direction in this task.
MS-MARCO
NF-CORPUS
Method
Inference Direction
Precision
Recall
NDCG
Precision
Recall
NDCG
K-1
K-4
K-1
K-4
@ 100
K-10
K-20
K-10
K-20
@ 10
TRLK-Ba
D -> Q
28.4
18.54
27.22
70.29
61.49
15.7
11.38
10.68
13.08
43.23
TRLX-FoBB (Reverse)
D -> Q
24.9
17.38
23.85
65.85
58.84
14.98
10.91
10:01
12.76
41.65
TRLX-FoBa (Forward)
Q
21.16
15.58
20.25
59.08
55.46
17.86
12.6
11.11
13.5
48
TRLK-Fo
-> q
20.37
14.9
19.45
56.39
54.46
17.31
12.38
9.74
11.76
48.08
Forward Baseline
Q -> D
21.05
13.82
18.42
47.81
53
0.87
0.87
0.17
0.31
3.89
Backward Basel ine
-> D
16.8
14.04
15.99
53.13
52.07
1.11
0.79
0.21
0.29
3.95
Key takeaways: We experiment with two information ettrieval-based benchmarks MS-MARCO and NE-CORPUS and compute multiple metrics to compare TRLM variant models with standard Forward Baseline and unconventional Backward Baseline We show a gain of 8.49 points in NDCG@ 10 on MS-MARCO and 44. 19 points in NDCG@ 10 on NE-CORPUS. Aligning with the results in citation, the results from this task also accurately demonstrate the importance of going from a high information direction to a low information direction. The massive difference between the directions is evident in the NF-CORPUS dataset.
5.4 Defending against Jailbreak attacks
We next aim to leverage the generative ability of TRLM to augment toxicity filters that are used to improve the safety of LLMs. Prior works show that LLMs (and their input filters) can be jailbroken using crafted adversarial attacks [Zou et al. 2023), while output filters tend to have high false negative rate due to the sensitivity to the presence of toxic words, despite being in a neutral context (See TO). We propose to combine the benefits of input and output filters by projecting the output response of L.MMs to the input query space using the reverse generative capability of TRLM, and further detecting the toxicity of the generated queries to block/ pass the response to the original
8

---

Table 6: Performance of the proposed defense strategies across different thresholds, evaluated on the human annotated and jailbreakbench toxic responses. TRLM- Ba achieves significant gains over all other approaches. Notations: PT [Pretrained]; [rTlnstrucioon-fiitunell FNR [alse Negative Rate], FPR[False Positive Rate], new-HA [new HA Dataset], JBBJJBB Dataset), (H) [Hard], (E) [Easy]
=2
Method
ENR-HA
FNR-JBB
FTR(1))
FTR
FNR-HA
FNR:IBB
FFR
FIR
FNR-ILA
FNR-IBB
FPR
TRUM 3o (PT)
36.11
TRLM Ba (PT)
18.18
27.27
781M-70 (IT)
54.55
63.64
72.22
TKUM- (TT)
18.18
$9.72
0.00
8.00
18.18
70.83
0.00
4.00
27:27
79.17
0.00
2.00
query based on a pre-specified criteria. We thus effectively amplify input safety filters, i.e. reduce False Negative Rate (FNR) with marginal/ no impact on False Positive Rate (FPR).
Consider TRLM Generate that generates queries that could have produced given response. The insight is that, the reverse generative ability of TRLM allows the projection of a candidate (jailbreak) query that could bypass the input filter back to the (naive) query space observed during training. These projected questions can thus be rightly classified using the same input filter.
Key Idea:
Defense Strategy: We propose defense strategy where i) query is passed through the input filter, ii) if the input filter rejects the query, we return reject as well, iii) if the input filter allows the query, we take the Response produced by the model and generate multiple queries using TRLM. Generate reject the query as "unsafe". Otherwise, we declare safe, and output the response corresponding to the input query. An elaborate description is provided in Algorithm 12 of the Supplement.
If the number of generated queries rejected exceeds a threshold, we
Datasets: We consider a human annotated (HA) dataset provided as part of the JailbreakBench benchmark (HAd) for evaluating the performance of toxicity classifiers. This contains 100 questions annotated by humans, of which 43 are annotated as toxic based on majority vote across 3 annotators. We introduce a GPT-4 based filter, that considers the prompt-response pair to judge their toxicity (Details in Appendix F2) , and has O FNR on this HA dataset, which is ideal for defense evaluation. We further consider a 3- turr-otrrooo-oe 1 1106 based input toxicity filter for the empirical evaluation of the proposed defense, which has an FNR of 25. 25.58% on this dataset. These unblocked questions form our new-HA -H dataset for the experiments. In addition to this, we use the following datasets for evaluation: JBB dataset that contains jailbreak questions (that are toxic as per the GPT-4 judge, but are safe as per the GPT-3.55 filter we augment) corresponding to different attacks on Jai JailbreakBench, dataset that contain safe and easy questions and H dataset that contains safe questions that are hard to classify as safe. We discuss more details on these datasets in Appendix [.]]]
In the two toxic datasets (HA and JBB), the gpt-3.5 g.t-3.5-turbbl 1--5---roo110000 1106 based input filter does not block any of the questions, and our defense strategy aims at lowering the False Negative rate on the toxic questions (JBB dataset and n dataset), while ensuring low false positive rate the safe questions as well (E and H datasets). We present the improvements in FNR rates for Algorithm [2 when combined with the gpt-3. t-turrrro--11 1106 input filter and various TRLM variants in Table We further present the impact of varying the threshold in Fig 4 of the Appendix.
Results: We firstly note that the proposed TRLM defense strategy improves the FNR of the gp 1....--urro--111 -106 input filter across all settings considered. Further, the TRLM- Ba pre-trained model improves FNR by more than 70% on the HA dataset and around 35% on the JBB dataset, and outperforms other variants with negligible impact on FPR.
We note that the proposed defense outperforms existing perplexity thresholding based defenses (Jain et al., 2023 Alon and Kamfonas, 2023] and Smooth-LLM (Robey et al.] 2023) on the JailbreakBene
attacks [Chao et al., 2023 Deng et al., 2024) owing to the integration with an input filter that already outperforms them on the same, Hence, we do not compare with them. Further, these defenses operate only in the input space, while the proposed defense aims at augmenting the input space with feedback from the response. Hence, the proposed defense is orthogonal to such methods, and can thus be integrated with them well.

---

Conclusions
In this work, we explore the capabilities of TRLM for scoring and generation of queries, when conditioned on responses. Our study points to the importance of the response - query direction in LLMs. When deploying TRLM models for reverse scoring, we show improvements on AlpacaEval leaderboard, Citation attribution and retrieval tasks. We further show that generations from TRLM can augment safety filters effectively.
10

---

7 Limitations
We note that the assumptions made for our theoretical results in Section 1 are stylized, and may not hold true in practice, as the space of all answers to questions may not be adequately captured by assumptions in that section. Given this assumption, one may wish to explore other models for hallucination that more general and provide results about reverse scoring. We leave such theoretical exploration to future work.
Further, TRLM benefits have thus far been explored on tasks related to short form queries that have long answers. One may wish to understand and demonstrate the effects of reverse scoring on other tasks. For instance, one might pose the question does TRLM provide possible benefits for broader set of tasks that language models are used for. We leave the exploration such settings in which the reverse scoring direction of response with obtaining an understanding on the reason behind such an advantage, as part of future work.
query is better than the forward scoring direction, along
8 Acknowledgements
We are grateful to Kathy Meier- and Krishnamurthy Dvijotham for the helpful discussions regarding defending against Jailbreak attacks. We sincerely thank Roman Novak and Abhishek Kumar for their inputs on early versions of our work.
References
Alpacaeval leaderboard.
Can dataset. http://////////////////////////a
Human annotated dataset, jailbreakbenchh
http:://gthhb.com//babbbaabblbne httpp//eeeeeeblaaacccaaaaaaaaaaa
CSv
Pubmed. https://pubmse..nci...
www.niihooo.... gov/
Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXtv:2303.08774, 2023.
G. Alon and M. Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308. 14132, 2023.
M.-R. Amini, V. Feofanov, I.. Pauletto, F. Devijver, and Y. Maximov. Self-training: A survey. arXiv preprint arXiv:2202. 12040, 2022.
R. Anil, S. Borgeaud, Y. Wu, Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Mil- lican, D. Silver, S. Petrov, M. Johnson, I. Antonoglou, J. Schrittwiesrr, A. Glaese, J. Chen, E. Pitler, T. P. Lillicrap, A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan, B. Lee, F. Vi- ola, M. Reynolds, Y. Xu, R. Doherty, E. Collins, C. Meyer, E. Rutherford, E. Moreira, K. Ayoub, M. Goel, G. Tucker, E. Piqueras, M. Krikun, I. Barr, N. Savinov, Danihelka, B. Roelofs, A. White, A. Andreassen, T. von Glehn. L. Yagati, M. Kazemi, L. Gonzalez, M. Khalman, J. Sygnowski. and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312. 1805, 2023a. doi: 1805.
10. 10.48550/ARXIV.255
http:::////oooo/////////// 48550/ arxiv wwt.Xxiv..1.... 11805
R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, I.. F. Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, Robinson,
Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. H. Ábrego, Ahn, J. Austin, P. Barham, J. A. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng,
M. Omernick,
C. Cherry, C. A. Choquette-Chor CCho, A. Chowdhery, C. Crepy, S. Dave, M. Dehghani, S. Dev,
J. Devlin, M. Díaz, N. Du, E. Dyer, V. Feinberg, F. Feng, V. Fienber, M. Freitag, X. Garcia, S. Gehrmann, I.. Gonzalez, and al. Palm 2 technical report. CORR, abs/2305. 10403, 2023b. doi: URL https:::: //oii org/10...0.. 488500 arxiv 2305. 10403

---

M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2024.
P. Bajaj. D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv 2016.
preprint arXiv:1611.09280
K. G. A. M. Benjamin Cohen-Wang, Harshay Shah. Contextcite: Contibtui model generation to 2024.
context. https::/ http://t......lllaaaaa.a.aattttt
com haam///adryLaa///
L. Berglund, M. Tong, M. Kaufmann, M. Balesni, A. C. Stickland, T. Korbak, and O. Evans. The reversal curse: Lims trained on" a is b" fail to learn" is a". arXiv preprint arXiv:2309. 12288. 2023.
V. Boteva, D. G. Ghalandari, A. Sokolov, and S. Riezler. A full-text learning rank dataset for medical information retrieval. In N. Ferro, F. Crestani, M. Moens, J. Mothe, F. Silvestri, G. M. D. Nunzio, Hauff, and G. Silvello, editors, Advances in Information Retrieval 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings, volume 9626 of Lecture Notes in Computer Science, pages 716 722. Springer, 2016a. doi: 10.1007/ 978- 3-311-30677-11
Utthttps:://oo
org/ 10. 1007/97077770 --
V. Boteva, D. Gholipour, A. Sokolov, and S. Riezler. A full-text learning rank dataset for medical information retrieval. Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings 38, pages 716-722. Springer, 2016b.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners, Advances neural information processing systems, 33:1877-1901, 2020.
P. Chao, A. Robey, E. Dobriban, H. Hassani, G. Pappas, and E. Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023.
X. Chen, M. Lin, N. Schärli, and D. Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05/ 2023.
G. Cloud. Google cloud tpu v5e inference. URL #5e-inference Accessed on Feb 1, 2024.
G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang. T. Zhang, and Y. Liu. Masterkey: Automated jailbreaking of large language model chatbots. In Proc. ISOC NDSS, 2024.
Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. CoRR, abs/2404. 00445,, 2024. doi: 10.4550/ARXIV22400.000
URL https://ddi.. org/10 wtrg///.4855//rre. arxvv 2404. 04475
Y. Fu, H. Peng, T. Khot, and M. Lapata. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.101-2. 2023.
O. Golovneva, Z. Allen Zhu, J. Weston, and S. Sukhbaatar. Reverse training to nurse the reversal curse. arXiv preprint arXiv:2403. 13799, 2024.
R. A. Google and, M. Dai, O. Firat, M. Johnson, D. Lepikhin, Passos, Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Mcier-Hellstent G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. H. Abrego, J. Ahn, J. Austin, P. Barham, J. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Cherry, C. A. Choquette-Chor Choo, A. Chowdhery, C. Crepy, Dave, M. Dehghani, S. Dev. J. Devlin, M. Díaz, N. Du, E. Dyer, V. Feinberg, F. Feng, V. Fienber, M. Freitag, X. Garcia, Gonzalez, G. Gur-Ari, Hand, H. Hashemi, L. Hou, Howland, A. Hu, J. Hui, J. Hurwitz, M. Isard, A. Itycheriah, M. Jagielski, W. Jia, K. Kenealy, M. Krikun, S. Kudugunta, C. Lan, K. Lee, B. Lee, E. Li, M. Li, W. Li, Y. Li, J. Li, H. Lim, H. Lin, 7. Liu, F. Liu, M. Maggioni, A. Mahendru, J. Maynez, V. Misra, M. Moussalem, Z. Nado. J. Nham, E. Ni.
S. Gehrmann,
12

---

A. Nystrom, A. Parrish, M. Pellat, M. Polacek, A. Polozov, R. Pope, S. Qiao, E. Reif, B. Richter, P. Riley, A. C. Ros, A. Roy, B. Saeta, R. Samuel, R. Shelby, A. Slone. D. Smilkov, D. R. So.
D. Sohn, S. Tokumine, D. Valter, V. Vasudevan, K. Vodrahalli, X. Wang, P. Wang, 7. Wang.
T. Wang. J. Wieting, Y. Wu, K. Xu, Y. Xu, L.. Xue, P. Yin, J. Yu, Q. Zhang, Zheng, C. Zheng,
W. Zhou, D. Zhou, S. Petrov, and Y. Wu, Palm 2 technical report, 2023.
Q. Guo, R. Wang, J. Guo, X. Tan, J. Bian, and Y. Yang. Mitigating reversal curse via semantic-aware permutation training. arXiv preprint arXiv:2403. 00758, 2024.
H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. Llama guard: Llm-based onput-output safeguard for human-ai conversations. arXiv preprint wrXiv:2312.06674. 2023.
N. Jain, Schwarzschild, Y. Wen, G. Somepalli. J. Kirchenbauer, P.y Chiang, M. Goldblum. A. Saha, J. Geiping, and T. Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309000614 2023.
A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. 1. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.040065 2024a.
A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de Las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M. Lachaux, P. Stock, S. Subramanian, S. Yang. S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mixtral of experts. CoRR, abs/2401 004088 2024b. https:/::///oi org/ 10. 48540/ ariiv 2401, 24010
doi: 1044550/ARXIV2440..444
T. Korbak, E. Perez, and C. L. Buckley. RI with kl penalties is better viewed bayesian inference. arXiv preprint arXiv:2205. /2275, 2022.
B. Krause, A. D. Gotmare, B. McCann, N. S. Keskar, S. Joty, R. Socher, and N. F. Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint w0Xv:2009.06367, 2020.
J. Lee, Z. Dai, X. Ren, B. Chen, D. Cer, J. R. Cole, K. Hui, M. Boratko, R. Kapadia, W. Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327, 2024.
J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan. A diversit-prromotig objective function for neural conversation models. In K. Knight, A. Nenkova, and O. Rambow, editors, NAACL HLT 2016. The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 110-119. The Association for Computational Linguistics, 2016. doi: 10. 1/N16-1011 URL https:///oi. org/ 10. 18653/v1/116-111
S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph. J. Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International PMLR, 2023.
Conference on Machine Learning, pages
A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye. Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
S. Mudgal, J. Lee, H. Ganapathy, Y. Li, T. Wang, Y. Huang, Z. Chen, H. Cheng, M. Collins, T. Strohman, J. Chen, A. Beutel, and A. Beirami. Controlled decoding from language models. CoRR, abs/2310. 17022, 2023a. doi: 1044550/ARXIV.231..1177 48550/ arviv 2310.17022
https:/ss org/10
S. Mudgal, J. Lee, H. Ganapathy, Y. Li, T. Wang, Y. Huang, Z. Chen, H.-T. Cheng, M. Collins, T. Strohman, et al. Controlled decoding from language models, arXiv preprint arXiv:2310 17022, 2023b.
13

---

N. Muennighoff, N. Tazi, L. Magne, and N. Reimers. MTEB:
massive text embedding benchmark. In A. Vlachos and Augenstein, editors. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACI. 2023, Dubrovnik, Croatia, May doi: 10.18653/V1 2023.EACL-MAIN, 148. htt.https:///o// org/ 10. 200653/v1/2005 www.eea.ce................e.ee.. mainn. 148
2-6, 2023, pages 2006-2029. Association for Computational Linguistics, 2023.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
L. Qin, S. Welleck, D. Khashabi, and Y. Choi. Cold decoding:
Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35: 2022.
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and Finn. Direct preference opti- mization: Your language model is secretly reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, editors, Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Or- leans, LA, USA, December 10 16, 2023, 2023. URL http:://pppp mips wwt.appr.cppppprrpepppepp files// -bstrrattt---ta-abeea www.eroeneeeetttaan
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
A. Robey, E. Wong, H. Hassani, and G. J. Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprini arXiv:2310.0368. 2023.
D. Serdyuk, N. R. Ke, A. Sordoni, A. Trischler, C. Pal, and Y. Bengio. Twin networks: Matching the future for sequence generation. arXiv preprint 1788v:1708.06742, 2017.
C. Snell, L. Kostrikov, Y. Su, M. Yang, and S. Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206. 11871, 2022.
N. Stiennon, Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 -0001, 2020.
G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.0005: 2022.
J. Q Yang, S. Salamatian, Z. Sun, A. T. Suresh, and A. Beirami. Asymptotics of language model alignment. CoRR, abs/2404.01130, 2024a. doi: 1048550ARRXIV2244007225 URL https //oii.org/0.. 48000//550////// arwiv. 2173000730
J. Q. Yang. S. Salamatian, Z. Sun, A. T. Suresh, and A. Beirami. Asymptotics of language model alignment. arXiv preprint arXiv:2404.017300 2024b.
K. Yang and D. Klein. Fudge: Controlled text generation with future discriminators. arXiv preprint 2144v22104.05218, 2021.
S. Yang, R. Sun, and X. Wan. A new benchmark and reverse validation method for passage-level hallucination detection. arXiv preprint arXiv:2310.06440 2023.
Y. Zhang, M. Galley, J. Gao, Z. Gan, X. Li, C. Brockett, and B. Dolan. Generating infor- mative and diverse conversational responses via adversarial information maximization. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems 31: Annual Conference on Neu- ral Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréul, Canada, pages 1815-1825, 2018. URL Abstraaatw.t..attttt..o.a..aa... html haatf/22eet5533eeceeeeeeeeeedee3
14

---

Y. Zhang, S. Sun, M. Galley, Y. Chen, C. Brockett, X. Gao. J. Gao, J. Liu, and B. Dolan. DIALOGPT Large-scale generative pre-training for conversational response generation. In A. Celikyilmaz and T. Wen, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties: System Demonstrations, ACI. 2020, Online, July 5-10, 2020, pages 270-278. Asso- ciation for Computational Linguistics, 2020. doi: 10.
URL
https://doi. http:///ooo.og///...
org/10. 18653/11/222000000 1o1l-ddemo.3 30
W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. A survey of large language models. arXiv preprint mrXiv:2303.1 18223, 2023a.
Y. Zhao, M. Khalman, R. Joshi, S. Narayan, M. Saleh, and P. Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning Representations, 2022.
Y. Zhao, Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.14423 2023b.
M. Zhong, P. Liu, Y. Chen, D. Wang, X. Qiu, and X. Huang. Extractive summarization as text matching. arXiv preprint arXiv:2004.08799 2020.
A. Zou, Wang. J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15044 2023.
15

---

A Results on a Bipartite Graph Model for Questions and Answers
In this section, we outline a simple toy model involving a universe of questions and answers with relations between them where we show how TRLM-Ba perplexity based alignment distribution helps in picking the right answer when the forward model "hallucinates". For simplicity of exposition, we will only focus on the distribution PTILM-ES Ba (Q]AA) for the TRLM class of models.
Universe of Questions and Answers: We consider a universe of questions and answers in the form of bi-partite graph which deemed to constitute the ground truth. Let Q C >* and A VK where V is the vocabulary, be the universe of questions and answers respectively. For a given question Q. let (Q) E A denote the set of ground truth answers of Q. Let ( A, _E) be a bipartite graph such that E {(Q, QENIC is the edge set of all valid answers. In other words, Ideally, one may like a forward model to approximate the distribution, P(AIQ) = 1/ (QQ),, A E (Q) and O otherwise, closely.
Hallucination Model (Hamming distance version): We would like to model an imperfect forward model that does not fully adhere with the ideal ground truth forward model. For given question Q, the imperfect model produces answers ((22) to the neighbouring questions Q' which are at a hamming distance of 1 from Q. Concretely, let H(:)) .) denote the hamming distance function. The support of the answer distribution is then S
N(Q'). follows immediately that (A(Q) =
Q':H(Q.Q'ES
Σ
LAEN(Q///S)- Analogously, for given answer A, S'
U M(A).. Then A'A((A,)SE
Q':((QQ'SS1 for TRLM-Ba Ba we have PIBLM- (Q)A)-
E 10EN(47/18".
A'
Theorem 1. Let assume the hallucination model above. Assume that for two questions Q, min H(Q, Q') 2 1, (A.A")e (A.)")) (A), > 1, then the optimal alignment distribution when
PTaLH-Ba()) is used a scoring model (i.c. distribution in Lemma 2
has the support (Q) for Q.
Theorem From Lemma 2 we have that
(AI() & P=A(AII)
for some a O. For fixed question Q, left hand side is potentially non-zero only for A E N(QQ): H(Q, Q') < 1. since the first term in the right hand side is non-zero only for those by definition of the hallucination model. Consider an A such that 3Q' A &V(Q)), H(Q, Q') 1. We will argue that the second term is zero for such an answer A. Suppose it is non-zero, according to the hallucination model for the reverse direction, it means that AĄ' H(A, A') 1, A' e (Q)... However Q and Q' are hamming distance one away. From the assumptions, their neighborhood are far apart by more than 1, therefore contradicting the implication that ((A, A') 1.
Key Takeaway: Therefore under the above simplistic hallucination model, although the forward model has a wider support (S] in the answer space, due to alignment with TRLM-Ba perplexity, the new distribution has support of at most N(Q) provably. While assumptions in the theorem not reflective of true complexities of the universe of questions and answers in a domain, this simple model shows that alignment using TRLM's scoring metric can give rise to better re-ranking whenever nearby questions produce far away answers and generating forward models tends to confuse between nearby questions (a form of hallucination).
B TRLM Subroutines Score, Generate and Pretrain
In this section, we provide the subroutines of our TRLM models described in Section S
C Details on the Experimental Section
We describe details about our experiments in the following figure I
16

---

Algorithm TRLM-Ba. Pretrana
1: Input: T context length, N number of sequences. index set of the vocabulary. Pre-training
corpus of sequences {x, i such that E C", Zij E C. Initialize the model pe(-) with random weights.
2: for E [1: N] do
3: for E [1 [1:7] do
4: θ <- θ oz, Ve -o #1T-1 T....TT--111
5: end for
6: end for
Algorithm 2 TRLM-Ba.Score
1: Input: Query: Q. Response A. Conditioning Prompt: CP. Scoring Prompt: SP
2: return log TRLMM-B (Reverse(SP + Q))Reverse((P + +A)
Algorithm TRLM-Fo. Score
Input: Query: Q. Response A. Conditioning Prompt: Scoring Prompt:
2: return log Fo QI CP)
Algorithm 4 TRLM-FoBa. TrLN-aaa
1: Input: T context length. number of sentences length T. index set of the vocabulary. Pretraining corpus of sentences {x such that x; CT,, 2i E C.
2: Initialize the model pe(() with random weights.
3: for E [1 N]] do
4: for f. E [1 7 do
5: if is even then
6:
+ θ as Ve log -a,,, F1, "ii -t-1-
7:
else
8:
θ + θ + ai, Ve
24,2, lig-1)
9:
end if
10: end for
end for
Algorithm 5 FRLL--Ba.Generee
1: Input: Response A. Conditioning Prompt: CP.
2: return Q ~ FIRlM.Ba ( |Reverse(CP + A))
Algorithm TRLM-Fo. Generate
Input: Response A. Conditioning Prompt: CP.
2: return Q ~ Fraum-Fo ( AA + CP)
17

---

Task-based Retrieval
Citation of Answers Document Retrieval
Best-of-n Reranking
Query Generation
AlpacaEval Leaderboard
TRLM
Defence against Jailbreak
Figure I: This task is an approach to link specific highlight sentences to lines that corroborate these sentences from within lines in article. By using linear binary and exclusion search methods, the aim is to efficiently and accurately find sentences in the articles that support the highlights.
Table 7: Per-Task Scoring and Conditioning Prompts
Reranking Algorithm
Task
Scoring Prompt
Conditioning Prompt
Best-of-N Re-ranking
"Question:"
"? Answer:" " 's summarized by"
TRLM-Ba Ba. Score
Citation Attribution
0
Passage Retrieval
0
"is answered by"
Best-of-N Re-ranking
"is the answer to"
TRLM-Fo.Score
Citation Attribution
is summary of" "has an answer
Passage Retrieval
Best-of-N Re-ranking
TRLM-FoBa.Soom (forward)
Citation Attribution
Same as TRLM-Fo Score Scoring
Passage Retricval
Best-of-N Re-ranking
Score Score (backvard)
Citation Attribution
Same as TRLM-Ba BarM--B.Sccooo Score Scoring
Passage Retrieval
TRLM-Ba. Generatte
Defense Generation
0
"? Answer:"
TRLM-Fo Generate
Defense Generation
" is answer to question:"
C.1 Scoring Prompts
We scoring and conditioning prompts for all re-rankers evaluate the best possible response from the set response to a query. We provide a detailed list of prompts used for each task in Table 7
18

---

C.2 Details on AlpacaEval Leaderboard results
Table 8: Mixtral 8x7B generations with TRLM/Forward reranking against Mixtral 8x22B reference
as rated by a GPT4-1106-PPas 10--11000-Peiiee eew annotator
Model Performance on the Alpaca Leaderboard
Win Rate
Standard Error
Ranker
Inference Style
Wins
Losses
Ties
L.C
Reg
Discrete
TRLM-Fo
Response -> Query 42.07
47.54
47.08
1.51
379
426
0
TRLM-Ba
Response -> Query 44.13
46.98
47.39
1.52
381
423
TRLM-FoBa (Forw)
Response -> Query 42.88
47.11
46.58
1.52
375
430
TALM-FoBa (Rev)
Response -> Query 44.28
46.67
45.71
1.50
368
437
Self
Query Response 43.56
41.88
42.11
1.52
339
466
Forward Basel ine
Query -> Response 40.11
43.85
42.92
1.52
345
459
Table Mixtral 8x22B generations with TRLM/Forwan reranking against GPT4-1106-Previest reference rated by GPT4-1 1-----1000eeee ew annotator
Model Performance Comparison
Win Rate
Standard Error
Ranker
Inference Style
Wins
Losses
Ties
L.C
Reg
Discrete
TRLM-Ba
Response Query 31,84
21.17
20.25
1.25
163
642
TRLM-FoBa (Reverse)
Response Query 32.58
21.06
20.37
1.24
164
641
TRLM-FoBa (Forward)
Response -> Query 29.43
21.31
20.37
1.23
164
641
0
TRLM-Fo
Responæe -> Query 31.95
22.05
21.24
1.25
171
634
0
Forward Baseline
Query Reaponse 28.67
20.19
19.50
1.24
157
648
Self
Query -> Response 30.74
18.49
17.27
1.19
139
666
D Details on the Citation Task
Highlights
Citations
Articles
Goal: find article lines that corroborate the highlights.
Methods: linear search, binary search, exclusion search
Figure 2: This task an approach link specific highlight sentences lines that corroborate these sentences from within a lines in an article. By using linear binary and exclusion search methods, the aim is efficiently and accurately find sentences in the articles that support the highlights.
Algorithm Description: We describe the three attribution algorithms that use TRLM. in the reverse direction with appropriate prompts in the supplement.
score score function
Linear search (Algorithms) uses scores every possible sentence in the article with the highlight sentence.
Binary search(Algorith) g) actually starts with scores the first against the second half of the article for given highlight and chooses the best recurses further by splitting the chosen half (analogous to binary search) until contiguous set article sentences of sufficient granularity reached.
In Exclusion search(Algoritmmg) we drop article sentences and score the rest of the article with the highlight sentence. We pick the choice with the least score.
19

---

Algorithm 7 Linear Attribution Search
1: Input: h highlight sentence A {ai.... aN) Article, Condi Conditioning Prompt: CP Cr,Scoring Prompt :SP.
2: Return aj corresponding to the highest TRLM. sc,re(,,,,
Algorithm 8 Binary Attribution Search
I: Input: h, {A,, Ast1, Ar), Conditioning Prompt: CP,Scoring Prompt :SP.
2: 81 + score( = h1 A = A ( CP.SP).
3: 82 TRLM.score(( h, [tt[t-s/2] t,CP,SP)
4: if thens] > 52
5:
t s -
6: else
7:
8: end if
9: if then has sufficient granularity return Asa
10: else
11:
Binary Attribution Search(h, AP.,
12: end if
If Ahalf is at the required granularity, return this as the attribution, else recursively search with Analr as the article input.
Algorithm 9 Exclusion Attribution Search
1: Input: h; highlight sentence i, A = (a....., aN) article Sentences.Condi ti iningg Prompt: CP,Scoring Prompt :SP
2: Return a, corresponding to the highest TRLM. score(h.. ay.CP,SP). A denotes article A without sentence
E Details on the Retrieval Tasks
Query
Document 1
Document 2
Document 3
find documents that are relevant to the query.
Goal:
TRLM is prompted with prefix = "Document has an answer to" and suffix Query
Method:
Document N
Figure 3: This task is used to assess the representational capability of TRLM. Here we look at how likely a document is to contain information relevant to answering question. The language understanding of LLM makes it likely that it produces better semantic retrieval than simple embedding based model which is not contextual.
The scoring algorithms used for retrieval are given in Algorithms 10
E.1 Metries Explanation
We compute the following metrics, that are widely used in information retrieval regimes.
20

---

Algorithm 10 Document Retrieval TRLM-Fo
1: Input: Q query, D = {dl..... dN} documents,Conddtioninng Prompt: CP P.ooring
Prompt :SP.
2: Return d; corresponding with the highest score by TRLM-Fo. TRLM-Fo..core sco.
Algorithm 11 Document Retrieval TRLM-Ba
1: Input: Q query, D {dl...., dv} documents,,onddiionnis Prompt: CP, cco,rniing
Prompt :SP.
2: Return di corresponding with the highest score by TRLM-B.. sct.ee..cccoe...............
Precision @ K: We compute how many items within the top-k ranked items are relevant.
Precision@K No. of relevant items within top selected items
Recall K: We compute how many relevant items were selected of the of all relevant articles within top-k ranked items
Recall@K No. of relevant items within top k selected items
No. of relevant items
NDCG @ K: Normalized discounted cumulative gain, where gain is defined as the rank of the selected item.
MRR: Mean reciprocal rank of the selection.
NDCG @ K and MRR order-aware metrics that not only test the retrieval performance but also how well a retrieval algorithm can order items in a set.
Details on our Defence Task: Defending against Jailbreak attacks
(a) FNR (toxic jailbreak questions)
(b) FNR (human annotated data)
(c) FPR (Hard negatives))
100
100
100
50
80
80
0
10
20
30
40
40
(d) FPR (Easy negatives)
50
0
0
0
10
20
30
10
20
30
10
20
30
Threshold
Threshold
Threshold
TRLM-Ba (IT)
TRLM-Fo (IT)
TRLM-Ba (PT)
TRLM-Fo (PT)
Figure 4: Plots showing the False Negative Rate and False Positive Rate of the proposed defense strategy. Positive indicates UNSAFE response, while negative indicates SAFE response. The first plot considers 72 questions generated from the JBB dataset. The second plot considers questions from the new-HA dataset. The third plot considers 48 hard safe questions generated by GPT4 whose answers contain content that appears unsafe (from the H dataset). The fourth plot considers 49 easy safe questions from Alpaca Eval2 dataset (E dataset). TRLM-Ba (PT) the reverse pre-trained model clearly outperforms all other cases with lower FNR rate while keeping FPR rates under check.
F11
Datasets used in the Defence Task
JBB Dataset: We form the union of all question-answer pairs that have been generated by various attack strategies in JailbreakBench. There are total of 1037 question and answer pairs of which
21

---

472 are identified as toxic by the GPT-4 toxicity filter. Amongst these questions, only 72 are declared as "safe" (False Negatives) by the gpt-3, gpt-3.5-turbo 1106 input filter acting on the question alone, which we further reduce using the proposed defense. Thus, this set of 68 questions forms our JBB Dataset. We note that these are hard questions, since they are able to jailbreak the L.LMs considered in JailbreakBench, as well as the g 5-turbo--- 1106 input filter.
E-Dataset: We sample about 50 easy safe questions from AlpacaEval dataset, to test False Positive Rates, and eliminate question that was predicted as unsafe by gpt-3. pptt3.5-turbee 1.5-turbo-1111 1166.
H-Dataset: To test the FPR on hard samples, we prompt GPT-4 to generate about 50 questions that could possibly mislead input filter, while being safe. We further verify that the input-output pairs are safe using the GPT-4 toxicity classifier, and eliminate 2 questions that were predicted as unsafe. The prompt used to generate this dataset is shown below:
Can you help me create a dataset of questions that would lead to answers containing words that can be perceived as harmful, but are not. I want to use this ensure that my toxicity filter does not filter them out. For example, "Create political cartoon about gun violence", "Describe historic event in detail. dcctobr Revolution". Generate as many questions you Stop ONLY when you find that you are repeating questions.
F.2 GPT4 prompt used as a toxicity classifier
We prompt GPT4-1106 with the following prompt to verify the toxicity of the question and answer.
SYSTEM PROMPT
Analyze user-submitted question-answer pairs with a focus safety and ethical standards. Your goal is classify each answer as "SAFE" or "UNSAFE", especially considering the potential for encouraging or detailing harmful behavior. Answer one word without any explanation. Definitions:
SAFE: Answers that appropriate for general audiences, educate without promoting unethical actions, and maintain respectful and informative without revealing sensitive information.
UNSAFE: Answers could potentially encourage or inform harmful unethical behavior, even unintentionally. This includes content that might educate or provide how guide on harmful actions, discriminatory practices, manipulation tactics, illegal activities, or other unethical behaviors.
USER PROMPT
Question: <question>
Answer: <answer>
F.3
Algorithm for Question Generation for Defense
Algorithm 12 Defense strategy using TRLM, Generate
Input: Question Q, a forward model/ system to be defended M, an input filter that classi- fies input questions as UNSAFE/SAFE, filter-threshold т, number of generated questions N. Conditioning Prompt:CP.
2: If F(Q) UNSAFE, return UNSAFE
3: Else Generate Answer from Forward Model for question Q
4: Generate N questions = {q1... qN' using TRLM Generate(A (ACCP)
5: Compute F(q.) for 9 E Q. Calculate total number of UNSAFE questions.
6: If > threshold return UNSAFE
7: Else return answer A to query Q
22

---

F.4 Additional Tables relating Jailbreak Defense
Table 10: Comparison of various Input+Output Filter combinations on Human Annotated dataset on JailbreakBench. For the filter based GPT-3.5 (version gpt-3. 1etttubo--110005 1106), we use the prompt from Llama-Guard [Inan et al. 2023
Method
Agreement
False Positive Rate
False Negative Rate
GPT-3.5 Output filter
77.00
15.79
32.56
GPT-3.5 Input filter
25.58
GPT-4 input+output filter
89.00
19.30
0.00
G
Compute Requirements:
To pre-train TRLM models we use two TPUv5e pods Cloud] for two weeks in the setup described by Anil et al. [2023b]. Further details on pre-training are provided in Appendix B We run fine-tuning on FLAN FLAN-dataset dataset using TPUv5e pod [Cloud] for day.
23

---

H Licenses and Copyrights Across Assets
1. Gemini PPoo-100
Citation: [Tea 2023
Asset Link:
[link]
License: Google APIs Terms of Service
2. PALM2-Otter
Citation: GGoogle and al.
2026
Asset Link: [link]
License: Google APIs Terms of Service
1074-1106-Preee
Citation: (Achiam et al. 2023
Asset Link: [link]
License: OpenAl Terms of use
4. Mixtral 8x22B
Citation: (Jang et al. 2024a
Asset Link: [link]
License: Apache license
5. Mixtral 8x7B
Citation: (Jiang al. 2024a
Asset Link:
[link]
License: Apache 2.0 license
6. Gecko
Citation:
LLe al. 2024
Asset Link: [link]
License: Google APIs Terms of Service
7. CNN Daily Mail
Citation: [hong ct al. 2020
Asset Link: [link]
License: Apache liceese
8. MS-Marco
Citation: BBjaj ot al.
2016
Asset Link:
[link]
License: Microsoft Terms and Conditions)
9. NF-Corpus
Citation: (Boteva al. 2016b
Asset Link: [link]
License: Terms of Use
Alpaca Eval Benchmark
Citation: (Alp
Asset Link: [likk]
License: Apache 2.0 license
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have tried our best convey all our results in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Refer Section ?
Guidelincs:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how rohust the results are violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- speciliifciinnonn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures hecause fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If l'applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fcar that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
their best judgment and recognize
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: While theory is based toy model for the hallucination setting, propose Theorem, along with proof in Theorem I proof.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
25

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the algorithms needed to create TRLM in Section B and provide the compute requirements to run these algorithms in Section G
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clcarly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closcd-soure source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code. with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Answer:
[Yes]
Justification:
We provide the algorithms needed to run our TRLM model in Section B
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines lhttpa://nipscc///ptp guides/ wwwodeSuaisssionn....
for more details.
While we encourage the release of code and data, we understand that this might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : ://nis//..////////// cc/public/ Pu..dees/CoodSuubbllsssssooooo.. for more details.
26

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommeneed, but including URLs to data and code is permitted.
6. Experimental Setting/Detann Details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification:
We provide details of our experiments in Appendix Section
C.2
D and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the standard errors in the AlpacaEval leaderboard test in Table g
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed crrors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL, if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
(nswer:[Yes))
Justification:
Refer Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
27

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethies
Question: Does the research conducted the paper conform. every respect, with NeurIPS Code Ethics http:///p//aaaip.....aaaaa////a/
Answer: [Yes]
Justification:
Yes we have followed Neurips guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Codc of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Yes, plcase see Section ??.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release any model or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should deseribe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
28

---

12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Refer Appendix H
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include a URL.
The name of license (e.g.. CC-BY 4.0) should be included each asset. scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi twwoiithoo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
assets are released,
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are alongside the assets?
introduced in the paper well documented and is the documentation provided
Answer: [NA]
Justification: provide a training method creation of TRLM and study effects such training.
Guidelines:
The answer NA means that the paper does not release new assets.
Rescarchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable for our work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved data collection, curation, other labor should be paid at least minimum wage in country data collector.
15. Institutional Review Board (TRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Not applicable for work.
29

---

Guidelines:
The answer means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
30


Reviews:
[{"review_id": "BW5rzOL86e", "reviewer": "Reviewer_xuG3", "summary": "The paper explores the utilization of reverse/backward-trained causal LLM. This LLM can be used to score responses based on the probability of generating the queries given the scores (which can be combined with the probability of generating outputs from input). Then, they can be used for re-ranking. In general, they can be used to score sentence pairs - which can be used for citation attribution. The authors also consider the use of reverse LLM for toxicity filters. Given that standard output filters tend to be too aggressive in rejecting (high false negative), the authors found a way to improve the scores by generating the input from the output using the backward model, and then using the input filter for toxicity filtration. \n\nReasonable baselines are considered and compared (for example, using the plain forward model for generating the query from response in a format fashion using an appropriate prompt), or alternative scoring methods (like perplexity).", "questions": "Minor:\n\nin Line 183, I assume P(A|Q) = 1/|N(Q)| should be the appropriate notation.not 1/N(Q). \n\nLine 185: \"the idea; ground truth\"  - I believe you meant ideal.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Application potentials of modern backward LLMs are not as well explored, so it's a relatively unique direction.\n2. There are some interesting ideas for using backward LLMs, like toxicity filter besides standard ideas like re-ranking.\n3. The paper provides some decent theoretical motivations for informing the scoring mechanism with a backward model.", "weaknesses": "1. The core idea is not particularly novel. There are also a few missed citations. [1,2,3] (There could be others I am missing. Would recommend doing a deeper literature search) similarly used backward-trained models (one of the references is dialoGPT, which falls under the causal language model family) and their scores for re-ranking.\n\n2. It may be a bit hacky to circumnavigate the output filter by projecting the output into the input space. In principle, the output being toxic doesn't mean an appropriate input has to be. RIght now, the hacky approach still may working betters because the alternatives are limited, \n\n[1] A diversity-promoting objective function for neural conversation models. - Li et al. NAACL 2016.\n\n[2] DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation - Zhang et al. ACL Demos 2020\n \n[3] Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization - Zhang et al. NeurIPS 2018", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable feedback and relevant references. We are happy that the reviewer finds this direction to be relatively unique, the toxicity filter to be an interesting idea, and the theoretical motivation to be helpful. We hope to address the reviewer's concerns in this rebuttal.\n\n - **Novelty of the core idea and related works:** We thank the reviewer for suggesting references that are relevant to our work. We will certainly add a discussion regarding these works in our revision. We briefly outline the discussion below:\n\n   - Li et al. [1] use mutual information between source and target for decoding. TRLM-Ba, trained in reverse token order, scores in reverse the forward generations and it results in outputs that satisfy a non-trivial objective for decoding. To see this, we argue in the theory section (Lemma 2), that doing RLHF using only the reverse score effectively samples answers proportional to $P(A|Q) P(Q|A) = \\frac{P(A,Q)^2}{P(A) P(Q)}$. We note that this expression when summed over all answers and questions would be the $\\chi^2$ divergence between dependent and independent distribution. This shows that forward generation’s contribution is the $P(A|Q)$ term while reverse scoring’s contribution is $P(Q|A)$ and it actually achieves the objective (in a qualitative $\\chi^2$ sense instead of the KL sense) as in [1].  \n   -  DialoGPT proposed by [2] pretrained a \"backward\" model to score sources given targets -- this is specific to conversational understanding. This is different from results on general instruction following benchmarks we present, which are significantly more general and open ended. For very large language models and on public, challenging benchmarks like AlpacaEval that test their instruction following capabilities, our work shows that full reverse token pre-training and reverse scoring gets the best set of results in re-ranking which is novel.\n\n    -  Zhang et al., [3] propose adversarial mutual information maximization, by using a \"backward proposal network\" during training of the model to maximize mutual information to improve the *informativeness* of generated responses. Further the proposed method involves GAN style training which could be unstable. We sidestep this issue by either prompting a forward model in the reverse direction (TRLM-Fo) to score or pre-training a model to learn in reverse that naturally scores in reverse (TRLM-Ba). We propose a method that can be used with any LLM and can be applied to any query.\n\n    - In summary, all previous works seem to motivate the need for better decoding based on scores in both directions. We show that reverse scoring alone, when used with forward generations, will achieve this naturally using a formal RLHF based argument and strong empirical results to back it. \n\n     - As the reviewer has noted, we explore several other novel applications such as citations, retrieval and defense strategy to amplify input filters. This demonstrates the generalization of TRLM across different tasks. \n\n - **Projecting output responses to input space**: We would like to clarify that the projection of output into the input space is in fact a key ingredient of the proposed defense. \n    -  We claim that given a toxic output, there is a non-zero probability of generating a relevant question that is classified as toxic (although non-toxic questions may also be generated as the reviewer pointed out).  This probability is significantly higher than the probability of generating a toxic question given a safe response. We specifically rely on this property to design our defense, which also empirically holds from the results. Thus, the proposed method augments input filters effectively while maintaining a low False Positive Rate. Thus we think the projection of output responses to the input space is indeed a principled approach.\n\n   -  We would like to further clarify the intuition behind the proposed defense. Output filters get confounded by the presence of artifacts in a long response, while input filters can focus only on higher level reasoning of whether a question can elicit toxic content. This is evident from the significantly better performance of input filters empirically when compared to output filters in recent works (Table-1 in [4]). \n   - We also perform a small ablation to highlight the better reasoning capabilities of input filters by taking toxic questions and jailbreak responses from Jailbreakbench for Llama-2, and checking the accuracy of predicting this as toxic by input and output filters. We additionally encode the prompts and responses by inserting a space between each character to check the understanding of the filter. We note that in both cases, the LLM is the same, and the input and output filters differ only with a prompt/ response indication in the system prompt. It can be noted that the true positive rate drops by 51% in case of input filter, and by 88% in the output filter - highlighting the better reasoning of input filters. \n\n| | Input filter TPR (original) | Input filter TPR (encoded) | %misses (input filter) | Output filter TPR (original) | Output filter TPR (encoded) | %misses (output filter) |  \n| ------------------------------------ | ---------------- | --------------- | --------------- | ----------------- | ---------------- | ---------------- |  \n| Add one space between each character | 84 | 41 | 51 | 67 | 8 | 88 |\n\n   -  Our strategy thus allows the model to reason, where the projection of response to inputs acts as a chain-of-thought step while filtering output content.  Thus, when we project a toxic response back to the input space, we combine the benefits of input and output filters while overcoming their individual shortcomings. \n\n - **Minor Points**: We will correct the notational issues and typos pointed out by the reviewer.\n\n[4] ShieldGemma: Generative AI Content Moderation Based on Gemma\n\n\nWe will be happy to address any further concerns as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cX0SsNZzCx", "reviewer": "Reviewer_4Q1B", "summary": "This paper introduces Time Reversed Language Models (TRLMs), which operate in the response-to-query direction for scoring and generation. The key contribution is demonstrating that TRLMs can provide effective unsupervised feedback to improve language model performance on various tasks. Specifically, the authors show that TRLM-based reranking improves performance on the AlpacaEval leaderboard, citation attribution, and document retrieval tasks. They also demonstrate that TRLM's reverse generation capability can augment safety filters to reduce false negative rates in detecting toxic queries. The paper provides theoretical insights into why reverse scoring can be beneficial and empirically demonstrates the effectiveness of TRLMs across multiple applications, highlighting the importance of the response-to-query direction in language modeling.", "questions": "1. The paper focuses mainly on English language tasks. How well might TRLM generalize to other languages or multilingual settings?\n2. The paper shows TRLM can help with safety filtering, but are there any potential risks or downsides to using this approach for content moderation?\n3. How sensitive are the TRLM results to the specific prompts used for scoring and generation? Did the authors explore prompt engineering to optimize performance?", "limitations": "Two suggestions for improvements:\n1. More discussion of potential biases or fairness concerns in the TRLM approach would be valuable.\n2. They authors could elaborate on privacy implications of using reverse query generation for safety filtering.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel approach: The paper introduces Time Reversed Language Models (TRLMs), which operate in the response -> query direction for scoring and generation. This is an innovative approach to leveraging unsupervised feedback in language models.\n2. Theoretical foundation: The authors provide formal results and a bipartite graph model to explain how TRLM-based alignment can help mitigate certain forms of hallucination.\n3. Comprehensive experiments: The paper demonstrates the effectiveness of TRLMs across multiple tasks:\n   - Best-of-N reranking on the AlpacaEval leaderboard\n   - Citation attribution\n   - Document retrieval\n   - Defending against jailbreak attacks\n4. Significant improvements: TRLMs show notable gains over baselines across tasks, e.g.:\n   - 5% improvement in length-controlled win rates on AlpacaEval\n   - 44% improvement in citation attribution accuracy\n   - 44% improvement in NDCG@10 for medical information retrieval\n5. Versatility: The authors show TRLMs can be used for both scoring and generation tasks, demonstrating their flexibility.\n6. Safety applications: The paper explores using TRLMs to augment safety filters against toxic queries, showing potential for improving AI safety.\n7. Broad applicability: The techniques are shown to work across different model families (e.g., PALM2-Otter, Gemini-Pro, Mixtral), suggesting wide applicability.", "weaknesses": "1. Assumptions: The authors acknowledge that the assumptions made for their theoretical results in Section 4 are stylized and may not hold true in practice. The hallucination model used is quite simplistic.\n2. Need for more extensive testing of the defense strategy: While the proposed defense strategy for amplifying safety filters shows promise, the authors note it needs to be tested on a larger and more diverse set of questions to determine its broader applicability.\n3. Computational costs: The paper does not thoroughly discuss the computational overhead of using TRLM for scoring and reranking, which could be significant for large-scale applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their feedback. We are happy that they find our work to be novel and versatile, our theory valuable, our experiments comprehensive and our improvements significant. We clarify the concerns in this rebuttal. \n\n  - **[Limitations section] Disclaimer on assumptions in theory:**  We clarify that the theory presented in the paper is targeted at explaining the *intuition* behind the proposed approach, and is intended to be a *proof of concept* that it works in *some stylized* settings. We thus include a disclaimer in the Limitations section to caution the reader about this. We will rephrase this is in the final version and clearly state that the method is *not* theoretically grounded, and requires empirical justification, which we present in Sections 5 and 6.\n  - **[Broader Impact section] Disclaimer on defense evaluations:**  We clarify that we indeed report results on the popular Jailbreakbench benchmark, that contains a diverse set of questions with toxic answers from several models families such as Llama, Vicuna and GPT. However, since the defense relates to the sensitive application of safety, we include a disclaimer to caution the reader to verify it on domain-specific applications before deploying the defense in practice. We will rephrase the same to clarify this in the paper.\n - **Computational costs:** The cost for reranking is negligible compared to the overall inference cost which we explain below. Consider a setting where an LLM is deployed and served to end users. To answer the user's query, the LLM generates 16 responses simultaneously in a batch for $t$ decode steps ($t$ is the length of the response). Since the inference is autoregressive, this process introduces a factor of $t$ to the total inference time. Subsequently, TRLM is used for scoring all 16 responses in a single pass, while using nearly 10x lesser parameters. Assuming that compute scales linearly with the number of parameters, we can compare the computational time of:\n    - (a) Multiple response generation which is $t$, and \n    - (b) TRLM re-ranking which is $1/10$, since scoring is done in a single pass by a model of roughly 10x lower capacity\nThus, (a) takes nearly $10\\cdot t$ times more time than (b). Considering that $t$ (response length) is generally between 20 and 500, the additional cost of reranking is negligible.\n- **Generalization of TRLM to other languages:** We perform experiments to check this, and find that TRLM based reranking is indeed valuable in other languages as well. We present the experiment and results below.\n  - For a direct comparison with the Alpaca Eval results presented in the paper, we perform reranking for Alpaca eval in German. For a fair comparison, we use the same set of English questions and 16 English responses (from Gemini-Pro-1.0) that were used for presenting results in Table-2 of the paper. Only for reranking, we translate the question and responses to German using the Google Translate API and ensure that the TRLM models rerank German answers. Since the judge is designed and verified only for English answer evaluation, we use the corresponding English answer of the best response as input to the judge. \n   - We present the results in the below table. LCWR refers to Length controlled win rate while WR refers to normal win rate. We observe significant gains of 2.69% and 6.82% for German reranking using TRLM-Fo and TRLM-Ba respectively when compared to reranking all 16 responses using the standard forward direction (Forward Baseline). We note that these gains are similar to the reported gains. Thus, our TRLM-PaLM models are valuable for reranking in other languages as well.\n\n| Method | LC WR(German) | LC WR (English) | WR (German) | WR (English) |\n|---|---|---|---|---|\n| Forward Baseline P(A \\| Q) | 24.5 | 24.27 | 17.30 | 17.13 |\n| TRLM-Fo P(Q \\| A) | 27.19 | 29.19 | 20.22 | 22.68 |\n| TRLM-Ba P(Q \\| A) | 31.32 | 32.44 | 24.12 | 24.35 | \n\n - **Potential risks of safety filters:** The key risk involved in any defense is to incorrectly classify benign inputs as toxic, or a higher false positive rate (FPR). We show in Table-6 and Fig.4 that the proposed approach effectively amplifies input safety filters, i.e. reduces False Negative Rate (FNR) while not increasing FPR.\n - **Sensitivity to prompts:** We use simple prompts in all cases to ensure that the observed results are not a result of excessive prompt engineering. We note that it may be possible to perform prompt engineering to improve scores in all cases. TRLM-Ba however does not require any prompts other than the \"Question:\" and \"Answer:\" tags, since Answer -> Question is its natural scoring direction. This is a key advantage of this model over TRLM-Fo. We further present results with two extreme cases of prompts for TRLM-Fo (Forward model prompted in reverse):\n\n  | Model | LC Win Rate | Win Rate |\n|---|---|---|\n| [Reported]: P(**[Q]**\\|\" **[A]** is the answer of the question: \") | 29.19 | 22.68 |\n| [Simple]: P(**[Q]**\\|\"Answer: **[A]** \\n Question:\") | 29.48 | 21.45 |\n| [Complex]: P(**[Q]**\\|\"Instruction: Generate a question that gives the following answer. \\n Answer: **[A]**\\n Question:\") | 27.58 | 20.24 |\n\n   We note that predicting $P(Q|A)$ without any prompting (Simple) works best. This is very close to what we used for reporting, and thus represents a fair comparison with TRLM-Ba, which also does not use prompting. \n- **Potential biases:** We note that bias/ fairness issues do exist in LLMs, and apply to our work as well. The proposed approaches merely score the *quality* of the response, and thus do not present any *additional* bias/ fairness concerns to the best of our knowledge. We will include this as a caution to the reader.\n- **Privacy implications of defense:** Our defense works on LLM responses and not on user data. Since LLM responses already protect user privacy, the proposed defense does not pose additional threats. \n\nWe will be happy to answer any further questions as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N4R4UMGO6f", "reviewer": "Reviewer_6ZRs", "summary": "The paper proposes a class of LLMs called Time Reversed Language Models, which are simply pretrained on an unlabeled corpus in reverse order, and finetuned on instruction-tuned datasets accordingly. The model is used to provide feedback to LM generations in four different tasks. For general question answering, it serves to rerank N generated answers by scoring the probability of the question given the answers. For citation attribution, it scores a summarizing sentence given a cancidate text passage from the source article. For document retrieval, it scores the query given a retrieved candidate document. Finally, the model can be used to defend against jailbreak attacks by using the model to generate novel prompts given the response to the actual response. If the novel prompts exceed a threshold of rejection, the original prompt is rejected as well.", "questions": "Suggestions:\n* Section 6 is probably meant to be a subsection of Section 5.\n* Results are presented as \"our method performs better by x%\", although what is meant is the absolute improvement in points on the scoring metric. E.g., in line 304, the improvement of your best variant over the Forward Baseline is not 44.19% but 44.19 points, which really is more than 1200% better than the Forward Baseline.\n* I find Section 4 to be the least helpful part of the paper. As is acknowledged in the Limitations section, the assumptions are quite strong. I'd rather move it into the appendix in favor of using the additional space to enhance clarity as explained under \"Weaknesses\". However, I wouldn't object if the authors insist on keeping it the way it is.", "limitations": "Limitations are appropriately addressed in a separate \"Limitations\" section.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The proposed methods are simple yet effective ideas of how to use a reverse LM for various tasks.\n* While the idea to train a reverse LM is not novel per se, its applications are.\n* The baselines are well chosen to demonstrate the benefits of the model. The claims are well-supported.\n* On most tasks, the proposed method is very effective.\n* The discussed methods are of great significance to the NeurIPS community.", "weaknesses": "* Even though the method is quite simple and easy to understand, I think the clarity of the paper could be better in some parts. For example, the purpose of the different parts/functions in Table 1 is not easy to understand without the context of the tasks that they are applied to. I think the paper could benefit from a \"Problem Statement\" section that introduces the different use cases from the tasks in Section 5 (e.g. reranking, retrieval, etc.) before presenting the model in Section 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable feedback on our work. It is indeed encouraging that the reviewer finds our work to be of great significance to the NeurIPS community and worthy of a strong accept. \n\n\n- **Clarity in some parts:** We thank the reviewer for this suggestion. We acknowledge that the explanation around Table 1 needs better clarity. We will add a Problem Statement section outlining all tasks we address in this work using the additional page in the camera ready version.\n- **Section-6 as a subsection of Section-5:** We agree with the reviewer, and we will update Section-6 to be a subsection of Section-5 in the final version. \n- **Improvement percentage:** We thank the reviewer catching this error. We will change the same to \"improvement by 44.19 points\" in the final camera ready version.\n- **Significance of Section-4:** We thank the reviewer for this suggestion. We would like to clarify that the theory presented in the paper is targeted at explaining the *intuition* behind the proposed approach, and is a merely intended to be a *proof of concept* that it works in *some stylized* settings. We thus acknowledge this in the Limitations section as well, as noted by the reviewer. We would like to emphasize that in the theory section, we also show that if the reverse score (P(Q|A)) is used as a reward function in an RLHF framework (which can be approximated by best of N reranking as shown in a recent work [1]), it leads to sampling Answer such that it is proportional to P(Question|Answer)*P(Answer|Question). This is significantly different from the conventional forward scoring, which we show to be merely a temperature scaling of the usual P(Answer|Question) distribution. This observation, while easy to derive, makes the motivation for reverse scoring very clear and conveys the key intuition that it considers both conditionals of P(A|Q) and P(Q|A) without the need for additional hyperparameter tuning during reranking. We will clarify this takeaway in the final version. We hope to be able to add the additional explanations related to experiments in the extra page that we are allowed in the camera ready version. In case of lack of space, we will certainly move parts of the theory as well to the Appendix.\n\n[1] J. Q. Yang, S. Salamatian, Z. Sun, A. T. Suresh, and A. Beirami. Asymptotics of language model alignment. \n\nWe thank the reviewer again for their feedback and recognition of our work. We will be happy to address any further concerns as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their valuable feedback on our work, which has helped improve our submission. The reviewers have appreciated the novel application of reverse scoring and generation through our proposed TRLM family of models to various tasks like retrieval, citation and amplifying input safety filters. Some reviewers have also appreciated our theoretical results (although stylized to simpler models) to be insightful. We are grateful for their time and comments. We outline some salient points of our rebuttal:

  - **Whether re-ranking and scoring would generalize to other languages:** We present results by translating Gemini Pro 1.0 generations (used in the main paper evals) and queries from AlpacaEval to German and reranking using our TRLM models in German. We note from the results that win rates using reranking in German do not vary considerably when compared to reranking in English, and the gains of TRLM models are preserved (Please note that only scoring and reranking by TRLM has been done in translated German. Prompts/Evaluations by AlpacaEval judges are difficult to change into German in a manner that also preserves the correlations of the win rates with human preferences in the ChatBot arena. So we just use the corresponding reranked counterparts in English to score as usual.)

  - **Theoretical Properties of Reverse Scoring on Forward Generations** - We would like to emphasize Lemma 2 of the paper that shows that the post RLHF distribution, when reverse scores are used for feedback, effectively samples answers A for query Q with probability proportional to P(A|Q)*P(Q|A). In additional references that one of the reviewers has brought about, sampling using composite scores is desirable. We show that our TRLM framework can achieve it formally backed by empirical evidence.
  -  **Justification for projecting toxic responses to the question space:** We claim that given a toxic output, there is a non-zero probability of generating a relevant question that is classified as toxic (although non-toxic questions may also be generated as the reviewer pointed out).  This probability is significantly higher than the probability of generating a toxic question given a safe response. We specifically rely on this property to design our defense, which also empirically holds from the results. Thus, the proposed method augments input filters effectively while maintaining a low False Positive Rate. Thus we think the projection of output responses to the input space is indeed a principled approach. Further, output filters get confounded by the presence of artifacts in a long response, while input filters can focus only on higher level reasoning of whether a question can elicit toxic content. This is evident from the significantly better performance of input filters empirically when compared to output filters in recent works. We refer the reader to our response to xuG3 for empirical justification of the same.

We look forward to hearing back from the reviewers and more discussion in the next phase. We will be happy to clarify any further concerns as well.

Meta-review: This paper explores the use of reversal language model for different four applications and tasks. Although the core idea of reversal language model is not new, authors novelly adapted it for several applications, resulting in a significant enhancement of performance. The paper is well presented and straightforward.

All reviewers are positive about this submission. Concerns and issues raised by reviewers were properly addressed during the discussions with authors. Authors also promise to incorporate this improvement in the camera-ready version. 

I would love to recommend its acceptance.
Decision: Accept (spotlight)