LLMs can learn self-restraint through iterative self-reflection
Authors: Alexandre Piché
Aristides Milios
Dzmitry Bahdanau
Christopher Pal
Venue: Accepted by TMLR ()

Abstract: In order to be deployed safely, Large Language Models (LLMs) must be capable of dynamically adapting their behavior based on their level of knowledge and uncertainty associated with specific topics. This adaptive behavior, which we refer to as self-restraint, is non-trivial to teach since it depends on the internal knowledge of an LLM. By default, LLMs are trained to maximize the next token likelihood, which does not teach the model to modulate its answer based on its level of uncertainty. In order to learn self-restraint, we devise a utility function that can encourage the model to produce responses only when its level of confidence is above a user-specified target accuracy $\rho^*$. This utility function can be used to score generation of different length and abstention. To optimize this function, we introduce ReSearch, a process of ``self-reflection'' consisting of iterative self-prompting and self-evaluation. We use the ReSearch algorithm to generate synthetic data on which we finetune our models. ReSearch elegantly incorporates the ability to abstain by augmenting the samples generated by the model during the search procedure with an answer expressing abstention. Compared to their original versions, our resulting models generate fewer hallucinations overall at no additional inference cost, for both known and unknown topics, as the model learns to selectively restrain itself. In addition, we show that our iterative search is more efficient as a function of tokens than naive search. Finally, we show that by modifying the target accuracy $\rho^*$, our trained models exhibit different behaviors.

Published in Transactions on Machine Learning Research (09/2025)
LLMs can learn self-restraint through iterative self-reflection
Alexandre Piché
alezandrelpiccke@@maalccom
ServiceNow Research
Aristides Milios
Mila, Université de Montréal
Dzmitry Bahdanau
Now Research
Mila, McGill University
Canada CIFAR AT Chair
Chris Pal
ServiceNow Research
Mila, Polytechnique Montréal
Canada CIFAR Al Chair
Reviewed on OpenReview: https: // openreview. net/ hl//foorrr forum? id= SuKPfchVKX
Abstract
In order to be deployed safely, Large Language Models (LLMs) must be capable dynam- ically adapting their behavior based on their level of knowledge and uncertainty associated with specific topics. This adaptive behavior, which we refer trivial to teach since it depends on the internal knowledge of an LLM. By default, LLMs trained to maximize the next token likelihood, which does not teach the model to mod- ulate its answer based on its level of uncertainty. In order to learn self-restraint, we devise a reward function that can encourage the model to produce responses only when its level of confidence above a user-specified target accuracy p² This reward function can be used to score generation different length and abstention. To optimize this function, we introduce ReScarch, a process of "sclf-reflection consisting of itcrative self-prompting and self-evaluation. We use the ReSearch algorithm to generale synthetic data on which finetune our models. ReSearch elegantly incorporates the ability to abstain by augmenting the samples generated by the model during the scarch procedure with an answer expressing abstention. Compared to their original versions, our resulting models generate fewer hallu- cinations overall at no ional inference cost, for both known and unknown topics, as the model learns to selectively restrain itself. In addition, we show that our iterative search more efficient function of tokens than naive search. Finally, we show that by modifying the target accuracy p", our trained models exhibit different behaviors.
self-restraint, non-
1 Introduction
In order for Large Language Models (LLMs) become reliable tools, it is important for the models to be able to modulate their responses based on their internal knowledge. In cases where the models are queried about topic that is not well supported by their internal knowledge, it is safer for the LLMs to provide short answer or even to abstain from answering entirely, instcad of providing an answer filled with inaccuracies (hallucinations)) Unfortunately, it is nontrivial to teach this behavior to LLMs since the optimal behavior depends on the model's internal knowledge (Goldberg 2023).
* "Currently affiliated with another institution.

---

Published in Transactions on Machine Learning Research (09/2025)
biographies Ilama2
history llama2
biographies mistral
history mistral
1.0
Accuracy
0.6
0.4
0.2
20
10 Number of claims
20
2
20
Number of claims
Number of claims
Number of claims
Llama2 7b ReSearch DPO (Ours)
Mistral 7b ReSearch DPO (Ours)
Llama2 7b search DPO
Mistral 7b search DPO
Llama2 70b safety prompted
Mixtral 8x7b safety prompted
Llama2 7b safety prompted
Mistral safety prompted
Llama2 7b
Mistral 7b
Figure 1: Results overview. Llama2 7b models trained on data synthetically generated by the ReSearch algorithm (*) outperforms all baselines. This includes Llama2 7B chat (m) and safety prompted (+), Llama2 70B chat and search DPO (Tian et al. 2023a (4). Our models produce more claims that scarch DPO and Llama 70b and is at least accurate. Mistral 7b models trained on data synthetically generated by the ReSearch algorithm (*) also outperforms all baselines. This includes Mistral 7B and safety prompted (o), Mixtral 8x7B and search DPO (Tian et al.
2023a (4). Our models are more accurate than every bascline and produce almost as many claims as the search DPO.
There has been several successful attempts to improve the factuality LLMs while maintaining their usefulness using preferences (Tian al., 2023a), rewriting based on questions and answers (Dhuliawala) et al. )023), and decoding by contrasting predictions at different transformer layers (Chuang al. 2024). While these methods have been shown to reduce hallucinations on average, they do not fulfill the desiderata of teaching LLMs to modulate the amount of information in their response, the level of detailedness of the information, and lastly to abstain entirely from responding to queries when that is appropriate.
In this work, we present. ReSearch: an iterative self-reflection algorithm designed for synthetic data generation. The algorithm involves utilizing an LLM to iteratively generate samples, self-evaluate the samples' expected accuracy in reference-free manner, and prompt itself to produce improved samples the subsequent iteration. ReSearch produces batch of scored samples, to which we also add a phrase that expresses refusal response. All these samples and the refusal phrase then be scored via self-evaluation and be trained with learning algorithms such as Direct Preference Optimization (DPO) (Rafailov et al. 2023). We show that LLMs trained on synthetic data generated by the ReSearch algorithm exhibit self-restraint, resulting in fewer hallucinations on both biography and historical event generation tasks, without any need for external references. In addition, Scction 4.4 we show that our iterative search is more efficient as a function of tokens than naive search. Finally, Appendix A.2.4 we show that by modifying the target accuracy p", our trained models exhibit different behaviors.
2 Background
Expert itcration (Anthony et al. 2017 Silver al. 2017) enables the agent to sclf-improve through alternating phases of policy improvement and policy distillation. Initially, the agent gathers dataset. of demonstrations utilizing search and self-evaluution; these demonstrations surpass the quality of those the agent could have collected independently. Subsequently, the agent distills the dataset into its weights via supervised fine-tuning (SFT). In language, expert iteration has been used to improve translation (Gulcehree)
et al. 2023 and problem solving (Singh et al. 2023
Reward function design is challenging as we must represent our preferences into single scalar obtain the desired behavior from an agent (Sutton & Barto 2018). Designing reward function for long form factual content generation is particularly challenging it requires to balance between multiple objectives (Vam-
2

---

Published in Transactions on Machine Learning Research (09/2025)
Vannevar Bush was born 1890.
Varnever Bush (1892 1978) was Ammer American inventor,
advisor.
Vannevar
Varnevar Bush (1890 1872) was an Ameri reriic engineerin expert and professor.
Vannevar Bush American.
Vornevar Bush was an Americann ico mathematidtan and engineer.
Vannever Dush was metheemstiiian.
Based on these sources, what is the probabiity the following claim concarming Vannever Bush being true?
Vannever
Claim: Vannevar
born
1972)
Vannevar Buch
claims from generations
c) Assess claim prob. being true p
American
was an Amer
Vennevar
engineering
mathematicie
1182-118 1878)
Self-Evaluation
expert and
and onginear
professor. He
who made
ongineer,
was bast kno
sigrificent
inventer,
for his planes
contributions
srencc edvisor. is most famous for his cont ributions to...
Write bicgraphy of Vannevor Bush of up sentences. The biography should include, but be limited following facts.
work in comouters...
devellppr
pulters
Vannevar Bush was an engineer.
Vannevar Bush was an American.
a) Sample multiple generations
Vannevar Bush was a sc innceneee adviscr.
Vannever Bush was an enginser.
Vannevar Busin was science advisor.
Vannevar Bush was an inventer.
blography Bush centences.
Vannevar Bush was born 1892.
Reprompt to include likely claims
d) Filter out unlikely claims P p*
Repeat K times
Self-Prompting
Figure 2: Overview of the ReSearch algorithm. ReSearch combines two components: 1) Self-Evauuation where the model evaluates the expected accuracy of its generated claims based their self-consistency with all the generations produced by the model, and 2) Self-Promptinng where the model incorporates the claims more likely than p² into its prompt to improve its generations at the next iteration. Finally, the resulting generations produced by the ReSearch algorithm plus the phrase expressing abstention are self- evaluated, ranked, and returned as synthotic data that can be used as desired.
plew et al. 2021]: 2022). Furthermore, we want the reward function to encourage the agent to have self-restraint and to abstain from generating an answer is incapable of doing so without violating a safely constraint.
the completeness of an answer (helpfulness) and its accuracy (harmlessness) (Bai et al.
Closed book factual generation. LLMs are known to generate content that contradicts established world knowledge (Zhang et al.) (023). commonly referred to "hallucination", DPO (Rafailov et al. 2023) has been shown to be effective reducing hallucinations by maximizing the average claim probability (as shown in Figure 3]. However, it is unclear if the model learns to adjust its verbosity or learns to abstain from answering query solely based on preference data, in which constraints cannot easily be encoded. In contrast to Tian et al. (2023a) which uses consistency- methods to evaluate the model confidence in an answer, Zhang et al. (2024) trained the model to evaluate confidence. Chainnoo-verffication (Dhuliawala al. 2023) ) (CoVe) has been used to reduce hallucinations, but does not lead to LLMs that learn abstain from answering queries they are unfamiliar with. Similar to Cove, real-time verification and rectification (Kang et al. 2023) identify and rectify hallueinations in step-wise manner. Manakul et al. (2023) used multiple generations to score the likelihood of claim. A
short biographies (Tian al.) 2023a]
similar self-consistency metric was introduced by Huang et al. 20244).
Self-evaluation and calibration. Manakul et al. 2023) showed that a model can detect its hallucinations by comparing a sampled generation with other sampled generations. (Tian et al., 2023b) showed that models that have been finetuned using RL via Human Feodback producod better calibrated prediction by expressing their uncertainty via verbalized probability. Kadavath et al. (2022) showed that a model can improve its ability to evaluate the correctness of answer by conditioning its prediction concurrent.
3

---

Published in Transactions on Machine Learning Research (09/2025)
hypotheses. In language, self-evaluation has also been used for question-answering (Ren et al. tools (Mekala et al. 2024), and alignment (Yuan et al. 2024).
2023 using
Question-answering Yang et al. (2023) leverage the model uncertainty to abstain from answering questions it does not know. Ren et al. (2023) propose a scoring method based on self-evaluation. Kadavath et al.
(2022) use multiple generations from a LLM to measure the uncertainty of the model. Concurrently, MMorri &
Hashimoto (2024) use conformal prediction to remove incorrect claims generated by LLMs. Yang et al. (2023) train model via SFT to abstain from answering when it cannot answer question correctly. Self- celf-prompting
has been used for short form
(Li et al.
2022
Selective prediction allows classifier to
abstain from answering low-confidence questions Xin et al.
2021 or questions that unanswerable (Rajpurkar
et al. 2018
Self-reflection is the ability for an agent to verbally reflect on a task to improve its behavior on subsequent trials. Shinn ct al. (2023) successfully used self-reflcction to improve the reasoning and programming capa- bilities of state-of-the-art models. Madaan et al. (2023) introduced self-refine which is an agent providing feedback on its outputs in order to iteratively improve it. Similarly to our method, Ji et al. (2023) used self-reflection to improved a model factual accuracy
medical questions, but crucially their method do not handle abstention. In Asai et al.] (2023), an LLM is trained via "reflection tokens" to introspect on the relevance of retrieved passages to then synthesize a final improved response to a knowledge-seeking query. Although the reflective elements are similar to our work, this work involves an explicit retrieval step over a corpus of external documents, while our work operates solely on the model's internal knowledge. In Liu et al. (202), authors systematically investigate the ability for language models to refrain from responding to queries that they are guaranteed to not know the answer to. They find that the majority of models, and especially open-source ones, are unable to consistently refuse invented queries.
3 Method
We are interested in model that, given its internal knowledge, exhibits sclf-restraint, i.c., is 1) helpful by generating as many true claims as possible and 2) harmless by limiting the number of false claims it produces. As we show in the experiment section, this behavior is non-trivial to obtain via prompting. Similarly, building a dataset for this behavior to train an agent via supervised learning is difficult since it requires us to have access to the model's internal knowledge. Instead of building a datasct, we augment the model with search and self-evaluation to generate a synthetic dataset specifically tailored to the knowledge of the model. In order to evaluate the samples generated by the search procedure, we design a reward function that can be approximated by the agent and that encourages the agent to maximize the number of true claims and minimize the number of false claims. We introduce ReSearch, an iterative search algorithm based on self-evaluation and self-prompting (an overview is provided in Figure 2 to maximize our reward function.
3.1 Reward function design
Limitation of only using accuracy to measure with a single metric as there is a trade-off between completeness and accuracy (Xu et al. 2023
reward function. Long form content generation is difficult
Previous works, such as FActScore (Min et al. 2023) and Finetuning for factuality (Tian et al. 2023a 1 have been solely using the average accuracy of generations to rank generations, where more accurate generations are given a higher score than less accurate ones. Although this works well in scoring generations of the same length, it cannot be casily used to score generations of different lengths or abstentions, as it does not take into account the number of clains in a generation (see Figure a)). For example, the model would obtain the same score by providing a single true claim than the score it would obtain by providing multiple true claims. Therefore, we introduce a reward function that takes into account the number of claims produced by the model and the accuracy of the claims.
Designing an reward function that can handle abstention and generations of different length. Similarly to the FActScore metric (accuracy), we assume that 1) generation y can be broken up into a set of atomic claims {c),z, as shown in Figure and 2) that each claim can be judged by an oracle T as being fac- tual or not. Similarly to FActScore the oracle will be a larger LM that has access to Wikpedia while the gen-
4

---

Published in Transactions on Machine Learning Research (09/2025)
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accuracy
Accuracy
Accuracy curry
0.6
0.6
Accura
0.6
0.6-
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2-
0.2-
0.0
6.0
8
10
8 Number of claims
2
4
6
10
8
10
2
4
6
8
10
Number of claims
Number of claims
Number of claims
a) Average accuracy
b) R(A(p" = 0.2))
c) R(A(p = 0.5))
d) R(A(p" = 0.8))
Figure 3: Reward function contour plots. First, in sub-figure a), methods such as Tian et al. (2023ab) that using the average accuracy as reward does not encourage more claims. In sub-figures b), c) and d), we observe that the reward function encourages the agent to produce as many claims as possible and to maximize the accuracy. also elegantly ranks samples with different number of claims and accuracy. Furthermore, in subfigure b), we observe that the agent should abstain from query and obtain a reward of instead of negative reward if it believes that less than target accuracy p* 20% of the claims in its best sample true. Finally, observe similar pattern for c) and d), where the targel accuracy p² is set to 50% and 80% respectively. We also observe the advantages of the reward function R over the average probability empirically in Figure
observe (as done by multiple
erating model does not. For query x and answer y, our reward function R(x, y, A) gives a reward of for every true claim in y and a negative reward for every false claim. Specifically, we define our reward function as:
R(
Σ r(T(2,,), where OECS(H)
(1)
[f, =1
(T((x,c).)) -X =0,
and claim splitter CS(y) > {a is extracting atomic claims from generation y. The claim splitter works by splitting the text into mutually-inddependet free-standing claims, by replacing pronouns with their referrents, and other similar disambiguating methods, through the claim splitter prompt (see Appendix 5 While it is not immediately obvious how to select X. We can instead set target accuracy p" for which the model would obtain reward of O, and then select À accordingly:
R( p² X(1 P² '¹)N =
(2)
In other words, the model will receive a positive (negative) reward if it produces generation with accuracy higher (lower) than p" Therefore, our reward function encourages the model to abstain if the expected accuracy of its answer is below the threshold p* and obtain a reward of O. While the reward function is linear in true and false claims, it is non-linear in the number of claims and accuracy. We observe in Figure 3 that increasing the value of the target accuracy p" changes the threshold for which the reward function is positive and therefore when the model should answer a query and not abstain. The Q value will be used both for self-evaluation to guide the Search baseline and the ReSearch method. Furthermore, the oracle reward function will be computed by Llama2 70b and reported in the results section.
and we can casily solve for = 1-p-.
3.2 ReSearch: an iterative search algorithm for LLMs
We introduce ReSearch: an iterative self-reflection algorithm designed for syuthetic data generation which can be used to improve the modcl's generation factuality and ability to self-restraint. For a given query 2, the algorithm iterates through three phases which can be observed in Figure 2 1) generation: sample multiple possible answers to query, 2) self-evaluation: evaluate the claims contained in these generations
5

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm ReSearch algorithm.
Require: Context dataset { J
Require: Policy Ty X + P())
Require: Q value Q y) R
Require: Claim likelihood p(7 c, Y) + [0,1]
Require: Factuality threshold p E [0.1]
Require: Claim splitter CS(y)
1: On-policy dataset D +
2: for x E {- do
3: # Initialize the generation with abstaining and Q value of
4: Y + ((0,0)}
# sample J initial gencrations.
{v' =oo(1IPrrta(x)))]
while stopping ccrteerin do
8:
# Collect generations and Q value
9: Y + YUl(,, (z),)
10:
# Extract most likely claims from all the generations
11:
C + feE CS(Y) p(T[x, f c, Y) > p}
12:
# sample new generations conditioned on the likely claims
13: {y' (e (x, C)
14: end while
15: + D-DU(z,))
16: end for
17: # Return synthetic dataset
18: return D
via self-consistency 3) self-prompting: prompt itself with likely claims. Finally, once multiple iterations have been performed, the algorithm returns synthetic dataset: (xr Y, Q), where Y is set of all the answers produced by the model and an additional answer expressing refusal and Q is their respective Q values (this phase is not depicted in the figure).
1) Generation. At each iteration the model first generates multiple generations Y2 = (v where each sample y is generated conditionally on prompt. At each generation the prompt is either the query generated by the user at the first iteration (Pwrite) or query generated by the LLM via self-reflection (Prewrite) for subsequent iterations. We expect the factuality of the generations to improve as Prewrite is refined via self-reftection at each iteration.
2) Self-vaaluation. In order to evaluate the expected factuality of its generations, the model first extracts the claims e for every generation v in the generations Y, produced at the current ileration k (Figure 2 b)). Then the model evaluates the probability of each claim being true based on the claim self-consistency with random subset of sentences sampled from the generations produced by the model up to this point using the prompt template Peval (Figure 2 c)). The intuition is that the claims that are likely to be true will be consistent with multiple generations, while the claims that are less likely to be true will be inconsistent. In other words, we will ask the model to generate an integer between 0 and 100 to represent the level of agreement between a claim and independently (condition on the prompt) generated samples. We then average the probabilitiee produced by the model over multiple subsets of sentences. Specifically, the probability of claim being true is estimated as:
PT c p(P e))
(3)
Ar~A(Y)
where X is a verbalized probability (asking the model to predict probabilitics as tokens, instcad of using the logits directly) (Tian al.] (002bb) Peval is the evaluation prompt template (see Table 5), and Aş A(Y)) is subset of the sentences previously generated by the model. We observe that verbalized probabilities

---

Published in Transactions on Machine Learning Research (09/2025)
more effective than logits, see Table The probability of each claim being true generated by the model is stored, and it will be used to compute the Q value of each generation (Equation 4 once the dataset is produced.
3) Self-Prompping. As mentioned above, at the first iteration, the model is conditioned on the user's query (Pwrite). This prompt not particularly informative about the model's internal knowledge. At subsequent iterations, the model will instead produce its own prompt (Prewrite) which includes claims that the model believes have high probability of being true. Specifically, all the claims produced by the model up to this point are filtered based on their probability of being true (Figure 2 d)). Claims with a probability of being true greater than the target accuracy p" are used to produce the prompt P Prewrite (Figure 2 e)). The intuition is that by prompting the model with these claims, the model will be more factual both by including these claims in its generations and by generating additional claims that are consistent with the claims that are believed to be true.
4) Resulting synthetic dataset. At. the end of the iterative search procedure (steps to 3), the ReSearch algorithm returns batch of generations and additional abstention response. Each of these generations has an Q value which is computed using Equation value of each response can easily be computed using the stored probability of a claim being true:
(the abstention response has a reward of 0). The Q
Q(E,V,A)= =
Σ Σ (() = | x,
(4)
CE tEfo))
The Q value and number of claims are computed by the generating model itself to guide its search and select which sample to produce, while the reward reported in the result. tables is computed by the larger RAG-based evaluator model. The ReSearch algorithm (Algorithm [ thus produces the dataset (x, Y, Q), where Y is the collection of all generations Y2 for all k and the additional abstention response. The dataset of generation and Q value pairs can be used in variety of ways. In this work, we used this dataset to finetune a model via supervised finetuning (SFT) on the best sample, via DPO on pairs of best and worst samples, and via RL directly on the Q value. Additionally, we report the performance of rumning ReSearch at inference time and returning the generation with the highest Q value. While this procedure is very expensive and is of less practical use, reporting the reward of the best generation is useful to understand the effectiveness of the amortization of the dataset into the model's weights.
4 Experiments
Tasks. We evaluate our models on two tasks: i) Biographies and ii) Historical events. For each task, we extract a dataset from Wikipedia where each entity (person or historical event) is classified into a popularity tier (bottom, middle, top) according to the length of the corresponding article. Evaluation is done through atomic claim decomposition and evaluation with larger L.M and a retriever that has access to the article. While our datasets and methodology are similar to FActScore (Min et al.) 2023 our datasets are larger (over 8000 entities per dataset) and signficiantly harder since they span the whole distribution of Wikipedia pages, while FActScore is smaller (183 labeled entities and 500 unlabeled entities) and mostly focuses on entities which are referred by other Wikipedia pages.
Methods. Our baselines include prompted models, safety- prompted models (models prompted to abstain if uncertain), and Chain--f-Verrffcaaioon (CoVe). We also compare against models trained on data from a bascline scarch procedure that maximizes average accuracy instcad of our reward function. We evaluate both inference-time search results and models trained via SFT, DPO, and RLOO on the generated datasets. For additional experimental details, see A.1
Experimental protocol. Each datasct is divided into a 7 train, 400 validation, 800 test set, and 30 invented entities to evaluate the model ability to abstain. The synthetic data used to train the SFT/DPO/RLOW
Ssarch/ReSearch models are collected from the train set. We tune the hyper-parameters for both the scarch and the training on the validation set (also used for and Figure 9 and finally re- port the test set results for Llama2 Table and Mistral Table All reported metrics in the result tables, e.g. accuracy, claims, and reward, are computed by Llama 2 70b with RAG on the Wikipedia page of the entity.
7

---

Published in Transactions on Machine Learning Research (09/2025)
Metraa ReSearch DPO
SFT
7b Redearch (0urs)
2.7
10.0
inference
2.3
Mistral 7b seerch
0.031
0.49
-0.52
-0.54
0.84
Mictral search DPO
0.14
0.62
-0.26
-0.21
1.1
Mistral 7b search RLOO
0.11
0.62
2.7
.0.06
0.34
1.3
5.0
Nixtral 5x76 safecy prompted
-13
7.6
-248
-0.45
3.8
Mistral 76 safery prompted Mistral Yo
-8.4
-5.5
-0.87
-5.8
-5.4
-1.9
-8.7
-5.5
-6.9
-6.1
-2.7
0.0
7o inference
Llaaz ReSearch SFT
Ulama2 ReSearch DPO
1.7
2.2
lame? Redeearch RI [Durs)
2.4
0.84
3.9
Uama2 search inference
0.53
6.0
-0.92
-0.29
2.4
Lama2 7b scarch SFT
0.41
5.8
0.46
0.54
1.8
Uama2 7b search RLOO
Laaaa2 7b search DPO
0.66
0.14
0.61
2.2
3.8
0.0043
-0.45
0.12
0.12
2.8
2.4
-7.5
Uama2 70b safety prompted
-1.2
2.5
-3.4
-2.5
2,2
Uama2 7b safety prompted Llama2 7b
-0.22
4.7
319
0.17
6.33
1.77
4.8
6,2
0.85
midde Popularity tior
top
middle Ropulanity tier
top
a) Biography
b) History
Figure Reward as a function of popularity tiers. The reward R(X(p" 0.5)) is 1) positive for gencration with more true claims than false claims, 2) for abstention or equal number of false and true claims, and 3) negalive for more false claims than true claims. We report the reward as function popularily tier and expected rewards lower for the entities of the bottom popularity tier.
Results overview. An overview of the results can be observed in Figure [ In the two results tables (Table and Table 2 methods are sorted by reward R(X(p" = 0.5)) which is equal to the difference between the number of true and false claims. Tlama models trained with DPO and RI.O on data synthetically gencrated by ReSearch obtain high reward and accuracy across every datasct. DPO is particularly effective the history datasel achieving the highest accuracy and only produces slightly fewer claims than the prompted model. Big models obtain higher reward than the smaller models. The Mixtral 8x7b model achieves slightly higher reward than our best Mistral 7b fine-tuned model on the biographics dataset, since it produces much more claims, never abstains and obtains an accuracy of 60%, but Mixtral perform poorly on the history dataset. Interestingly, the SFT ReSearch models obtain the highest accuracy for both datasets. Overall, models trained using DPO and RLOO on data synthetically generated by ReSearch obtain the highest reward of the 7b models across every dataset. CoVe (Dhuliawala al. 2023) generally produces fewer claims and results in slightly lower hallucination rate. Mistral generations for the biographies
dataset available in Table 15 and for history in Table 17
4.1 Self-reflection can be used to reduce hallucinations without any external references
Overall, the ReSearch algorithms produce similar number claims and sometimes more claims than the search baseline addition to being more accurate. Models trained with RI.OO and DPO on ReSearch- gencrated data, achieve the highest reward across all datasets for 7b amortized models (Table 1 Table 2 ReSearch models have the highest accuracy for every dataset and generally obtain positive reward across popularity tiers, while models trained data generated by search baselines obtain negative reward for bottom and middle tiers on the history dataset (Figure 4.. When safety prompted, larger models (Llama2 70b, Mixtral 8x7b) obtain higher reward than smaller models (Llama2 7b, Mistral 7b).
Models trained with DPO and RLOO on ReSearch data generally outperform ReSearch inference and SFT- trained models, except for the history dataset where ReSearch inference performs best. Interestingly, SFT- trained models on ReSearch data achieve higher accuracy and abstention but produce fewer claims than Re- Search inference generation, suggesting they behave differently from the search procedure despite distilling it.
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table Llama2 results.
Accuracy (1)
Claims (t)
Abstention
R(A(p' = 0.5)) (1)
dataset
tag
biographics
7b ReSearch RLOO (Ours)
0.91
7.34
0.50
6.24
7b ReSearch DPO (Ours)
0.86
6.14
0.52
4.65
7b ReSearch inference (Ours)
0.82
6.31
0.57
4.37
7b ReSearch SFT (Ours)
0.90
3.87
0.73
3.12
7b search DPO
0.81
4.57
0.67
3.04
7b search inference
0.72
6.03
0.57
3.03
7b search SFT
0.74
4.76
0.67
2.61
7b search RLOO
0.91
1.81
0.85
1.50
7b safety prompted
0.48
11.89
0.11
-0.34
7b CoVe
0.44
7.56
0.01
-0.84
7b
0.45
13.50
0.01
-1.14
70b safety prompted
0.60
10.53
0.23
2.44
history
7b ReSearch DPO (Ours)
0.62
13.56
0.16
3.46
7b ReSearch RLOO (Ours)
0.61
8.78
0.35
2.08
ReSearch inference (Ours)
0.58
9.09
0.40
1.68
7b ReSearch SFT (Ours)
0.61
5.55
0.59
1.42
7b search RLOO
0.56
5.85
0.56
0.85
7b search DPO
0.54
9.83
0..2
0.84
7b search inference
0.51
8.92
0.39
0.40
7b search SFT
0.51
6.14
0.54
0.28
7b safety prompted
0.39
13.58
0.04
-2.66
7b CoVe
0.36
13.87
0.01
-3.85
7b
0.36
14.52
0.00
-4.02
70b safcty prompted
0.45
13.51
0.04
-1.23
Abstention
Claims
Detalledness
Accuracy
ReSearch
Search
Mixtral Ux/b safety promsted
0.00
0.02
0.00
0.00
13
13
6.00
6.20
6.50
0.43
0.63
0.78
Mistral 7b safety promated.
0.00
0.02
0.00
0.00
13
12
13
5.70
5.70
5.70
0.16
0.28
0.47
Mistral 7b
0.00
0.02
0.02
0.00
13
13
14
6.00
6.10
6.20
0.17
0.29
0.46
Invented Bottom
Middle
Top
Bottom
Middle
Teg
Bottom
Middle
Top
Bottom
Middle
Top
Figure 5: Holistic analysis per popularity tier. We report the results aggregated by method for absten- tion, number of claims per non-abstained answer, detailedness, and accuracy function of popularity tier. We also report abstention for an invented tier for which abstention should idcally be 1.0. For non-aggregated
results, see Figures 10 to 13
4.2 FActScore benchmark
We evaluate the performance of our models trained on the biography task without any additional finetuning on the FActScore benchmark (Min et al. 2023). In order to compare with the results reported by Kang et al. 2203h, we only benchmark the Llama2 7b chat models on the 183 labelled entities. Our models strongly outperform every baseline (see Table 3) This is partly due to the limitation of the FActScore metric which, as noted by the authors, measures only precision and not recall. This allows our models (and the scarch DPO baseline), which learn to abstain when unsure, to achieve much higher precision and therefore high Focte

---

Published in Transactions on Machine Learning Research (09/2025)
Table Mistral results.
Accuracy (1)
Claims (1)
Abstention
R(A(p' = 0.5)) (1)
dataset
tag
biographics
7b ReSearch RLOO (Ours)
0.84
3.28
0.71
2.52
7b ReSearch inference (Ours)
0.84
3.37
0.72
2.47
7b ReSearch DPO (Ours)
0.83
2.74
0.73
2.02
7b ReSearch SFT (Ours)
0.85
2.03
0.82
1.47
7b search DPO
0.68
2.89
0.75
1.23
7b search inference
0.62
3.89
0.69
1.23
7b search RLOO
0.74
2.34
0.80
1.14
7b search SFT
0.61
2.99
0.74
0.84
7b CoVe
0.32
11.17
0.01
-3.92
7b safety prompted
0.30
12.44
0.01
-4.96
7b
0.31
13.25
0.01
-5.13
8x7b safety prompted
0.60
13.12
0.01
2.86
history
7b ReSearch inference (Ours)
0.65
5.32
0.62
1.84
7b ReSearch DPO (Ours)
0.66
4.95
0.53
1.70
7b ReSearch RLOO (Ours)
0.67
4.23
0.56
1.55
7b ReSearch SFT (Ours)
0.68
3.68
0.69
1.36
7b search inference
0.54
5.75
0.57
0.59
7b search RLOO
0.55
4.44
0.61
0.55
7b search DPO
0.51
5.03
0.59
0.21
7b search SFT
0.48
4.90
0.59
-0.07
7b safety prompted
0.32
12.31
0.00
-4.66
7b
0.30
13.20
0.00
-5.23
7b CoVe
0.30
16.42
0.00
-6.55
8x7b salety prompted
0.50
15.06
0.01
0.17
Table 3: FActScore results
Model
FActScorc
Llama2 7B Dola (Chuang et al.
2024
36.8
Tlama2 7B Ever + (NRG +SQ) (Kang et al. 2023
46.7
Llama2 7B scarch DPO
85.8
Llama2 7B ReSearch DPO (Ours)
87.6
Llama2 7B ReSearch SFT (Ours)
93.0
4.3 Maximizing the reward function leads to self-restraint
In this subsection, we investigate three different behaviors that emerge by simply defining a reward function and maximizing via self-reflection: abstention, modulating the number of claims per popularity tier, and detailedness. We note that these three behaviors are non-trivial to teach since they require access to the internal knowledge of the model. summary of all metrics provided in Figure
Abstention. When the model does not have the knowledge to answer query, it is best to simply abstain. In Figure 12 and Figure 5 include additional invented tier generated using GPT4 (and manually veri- fied subset of the entities to make sure they were indeed invented using Google scarch) to test the ability of the models to abstain from queries that they do not have knowledge about. We first note that there was no synthetic data generated for the invented tier during training. We observe that the models trained on data synthetically generated abstain with high probability from queries for invented entities. The notable excep- tion is the Llama2 models on the history dataset. While the models trained via SFT showed perfect abstention rate the invented tier, the models trained via DPO and RLOO have low rate of abstention. Hinting
10

---

Published in Transactions on Machine Learning Research (09/2025)
that these models might be over-optimizing the reward function which encourages generating less detailed claims when unsure about an entity. Interestingly, the safety prompted Llama2 models show high abstention rate for the invented tier of the Biographies dataset, but not for the history one. This might be related to how these models have been trained via RL via Human Feedback and precaution was taken while generating information about people, but. not about events. Additionally, while over-abstention can be a problem. The ReSearch algorithm elegantly addresses this problem using the accuracy threshold. The number of claims overall (where abstentions count claims) as function of the accuracy threshold can be observed in Figure
Modulating the number of claims per non-abstaining answer per popularity tier An alternative to abstaining from answering is to modulate the number of claims as function of the popularity tiers. To measure if this behavior is discovered by the models, we report the average number of claims produced by the model, when the model do not abstain and answer the query. In Figure [ we observe that the number of claims per non-abstained answer is slightly modulated as a function of entity tier and that models trained on synthotic data generate less claims than prompted models. In Figure 11 we observe that most models do not modulate the number of claims per non-abslained answer. The exception being the Mistral models. For example, ReSearch SFT produced 12 claims on average for the top tier and only 10 claims for the bottom tier. Overall, modulating the number of claims as function of the entity's popularity is learned by some models, but not by all of them. These results can be influenced by our guidance to output up to 4 sentences.
Detailedness. In order to understand the shift in distribution between the original models and our finetune models, we measure the level of detailedness their samples, i.e. how detailed the claims contained within those samples are according to Llama2 chat 70b, e.g., Mistral 7b: "The Battle of Delawore Bay, fought on May 29-30, 1652, was significant event in the Anglo-Dutch Wars." " (detailed, but less accurate) Mistral 7b DPO ReSearch (Ours): "The Battle of Delaware Bay was fought between the British and American forces during the American Revolutionary War." " (less detailed, but more accurate) (see Table 17 for additional generations). In Figure 5 we observe that ReSearch methods decrease the detailedness of their generation which allows them to produce much more accurate biographics. Additionally in Figure 13 WC also observe that search and ReSearch inference exhibit lower level of detailedness than the original models. This behavior of producing less detailed generations occurs due the model showing more certainty about less detailed claims than detailed ones which are more likely to be false. Thus, generations that are loss detailed will have higher Q values and will be chosen by these algorithms. Further, we observe that on most model-dataset pairs, the models trained using DPO and RIOO on synthetic data have an even lower level of detailedness than the datasets generated by search and ReSearch and the SFT models. This is duc to the models trained using DPO and RLOO are not only imitating the scarch algorithms, but also optimizing the Q values. These results on LM-evaluated detailedness are corroborated by Figure g which shows detailedness results as evaluated by a specialized NER model instcad. The highest variability in number of named entities in response is generally exhibited by our models.
4.4 Iterative self-reflection is more effective and efficient than naive search
In Figure Ga we observe that naively increasing the width of the search used to find a solution is ineffective. Instead, our proposed iterative approach, where the prompt refined at each iteration by including additional claims the model believes to be true, is more effective in maximizing the reward. The intuition is that the improved prompt leads to more directed search at the following iteration and to better samples. In addition to yielding higher reward, we observed in Figure 6b that iterative search requires less computation than wide search. The intuition is that claims believed to be true will be included in the next iteration prompt and are likely to be repcated by the model. Therefore, the evaluation of these claims can be cached and ret rieved for future use. In wide search, while certain claims will be repeated and caching can be used, every sample is generated independently of the others and is more likely to contain a higher variety of claims.
5 Discussion
In this work, we explored the capabilitics of ReSearch algorithm to generate synthetic datasots to train LLMs to exercise self-restraint. Our findings show that LLMs trained on these datascts can effectively reduce hallucinations in LLM outputs by encouraging the model to modulate its responses or even abstain
11

---

Published in Transactions on Machine Learning Research (09/2025)
48
Search Width
Search Width 16
16
Search iterations
Search iterations
1.0
1.5
2.0
2.5
13.6 13.8 14.0 14.2 14.4 Compute (log of tokens) (4)
(1ward(1)
Wide vs iterative search.
(b) Computation cost of search.
Figure 6: The importance of iterative search. First, Figure 6a
we observe that naively generating
more answers (increasing search width) is ineffective while increasing the number of iterations increases the score elfectively. Second, in Figure 6b observe that due to caching and self-prompling, iterative search requires fewer tokens than wide search.
from answering when its knowledge insufficient. This paper also highlights the difficulty of evaluating the quality of factual generations with single metric and the need for multifaceted evaluations such detailedness, abstention rate, number of claims, and accuracy. We observed that LLMs maximize their reward by strategically reducing the detailedness of their responses and abstaining when uncertain, thus balancing helpfulness with accuracy. The optimal trade-off between these strategies is application-speeiic and can be adjusted, for example, by modifying a target accuracy in the model's reward function to encourage different behaviors like producing lewer, more reliable claims. We acknowledge that our study is limited to factual hallucinations and might not generalize to contradiction or reasoning hallucinations, In the future, would like to combine this work with retrioval-augmenned generation to teach LLMs to abstain from answering queries when the required information is not contained in the retrieved documents.
References
Thomas Authony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-refloction, 2023. URL
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds. Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng Hc. Dola: Decoding by contrasting layers improves factuality in large language models, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXio:2110.114 14168, 2021.
Shchzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. arXim:2309.11400 2023.
reduces hallucination in large language models. arXiv preprint
12

---

Published in Transactions on Machine Learning Research (09/2025)
Yoav Goldberg. Reinforcement learning for language models. Retrieved from htttps:/giist.. httgggisttgggth.....o com/ 0oox//66ffeed550000000002222223 April 2023. Accessed: February 8, 2024.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjic Gu, et al. Reinforcod sclf-training (rest) for languago modeling. arXiv preprint arXiv:2308.005: 089998 2023.
Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhingra. Calibrating long-form generations from large language models, 2024.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating hallucination in large language models via self-reflection, 2023. URL https://arxi:: www.oorg/ggooo... arggabss/3110.65
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfiel-Doddss, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. Xiv preprint arXiv:227.00221, 2022.
Haoqiang Kang, Juntong Ni, and Huaxiu Yao. Ever: Mitigating hallucination in large language models through real-time verification and rectification, 2023.
Junlong Li, Zhuosheng Zhang, and Hai Zhao. Self self-prompting large language models for zero-shot open- domain qa, 2022.
Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. Examining Ilms' uncertainty expres- sion lowards questions outside parametric knowledge, 2024. URL
09731
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisatiwa Prasad Majumder, Katherine Hermann, Sean Wellock, Amir Yazdanbakhsh, and Peter Clark. Self-rofine: Itcrative refinement with self-feedback, 2023.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXic preprint arXi::23030088% 2023.
Dheeraj Mekala, Jason Weston, Jack Lanchantin, Roberta Raileanu, Maria Lomeli, Jingbo Shang, and Jane Dwivedi-YY Yu. Toolverifier: Generalization to new tools via self-verification, 2024.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14 14251, 2023.
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantees. arXiv preprint arXie:2402.1009 10978, 2024.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming- Wei Chang, and Yinfei Yang. Large dual encoders generalizable retrievers, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arxi 806.006000 098209822, 2018.
Jie Ren, Yao Zhao, Tu Vu, Peter Liu, and Balaji Lakshminarayanan. Self-evaluation improves selective generation in large language models. arXiv preprint arXiv:2312.09300 2023.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
13

---

Published in Transactions on Machine Learning Research (09/2025)
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXic preprint arXiv:1112.01815, 2017.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J Liu, James Harrison, Jachoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self- self-training for problem-solving with language models. arXiv preprinl Xrx:2:3312 06585, 2023.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Fimn. Fine-tuning language models for factuality. aix preprinl Niv:2311.08/01, 2023a.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2005.144 2023b.
Peter Vamplew, Benjamin Smith, Johan Kallstrom, Gabriel Ramos, Roxana Radulescu, Diederik M. Roijers, Conor F. Hayes, Fredrik Heintz, Patrick Mannion, Picter J. K. Libin, Richard Dazcley, and Cameron Foale. Scalar reward is not enough: A response to silver, singh, precup and sutton (2021), 2021.
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimuny Lin. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1040 1051, 2021.
Fangyuan Xu, Yixiao Song, Mohit lyyer, and Eunsol Choi. critical evaluation of evaluations for long-for question answering, 2023.
Yuchen Yang, Houqiang Li, Yanfeng Wang, and Yu Wang. Improving the reliability of large language models by leveraging uncertainty- aware in-context learning, 2023.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024.
Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. Self-alignment for factuality: Mitigating hallucinations in Imms via self-evaluation, 2024.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ocean: a survey on hallucination in large language models. arxiv preprint ar 23x::2330.000 01219, 2023.
14

---

Published in Transactions on Machine Learning Research (09/2025)
A Appendix
A.1 Additional Experimental Details
The large model used in the evaluation procedure is Llama2 70B, and the retriever is GTR- Ni et al. 2021 The generations produced by the model are up to 4 sentences long.
Evaluation:
Basclines: SFT models are trained on the best generations found by our scarch procodure. DPO models are trained on pairs of generations, the best and worst generations resulting from the search. Models trained with RLOO are trained the whole dataset and the Q values are normalized to have standard deviation of 1. For the search basclines, we do not use an iterative search of 3 times 16, but rather use a single wider search of 48 to have similar computation budget to ReSearch. The search baseline prefers to decline to answer the answer expected accuracy is less than 50%. Research uses K = 3 search iterations with width 16 and a target accuracy p² of 0.5.
Hyperparameters:
The Syper-paramett for the training are available in Table
Table
Model
Method
gradient steps
learning rate
batch size
regularization
Llama2
SFT ReSearch (Ours)
2000
5e-6
16
RLOO ReSearch (Ours)
600
1e-6
512
0.1
DPO ReSearch (Ours)
400
1e-6
64
0.25
scarch SFT search RLOO search DPO
2000
5c-6
16
600
1e-6
512
0.1
400
1e-6
64
0.25
Mistral
7b SFT ReSearch (Ours)
2000
5e-6
8
RLOO ReSearch (Ours)
600
1c-6
512
0.1
DPQ ReSearch (Ours)
400
1e-6
64
0.25
search SFT
2000
5e-6
8
search RLOO
600
1c-6
512
0.1
search DPO
400
1e-6
64
0.25
Generation examples: Generations for the Llama2 models for the Biographies dataset be found
in Table 14
Generations for the history dataset be found in Table [6 Mistral generations for the Biographics dataset can be found in Table 15 Mistral generations for the history dataset can be found in
Table
17
A.1.1 Claim splitting example
Entity: Michael Collins
Sentence: Michael Collins (born October 31, 1930) is a retired American astronaut and test pilot who was the Command Module Pilot. for the Apollo 11 mission in 1969.
Clains:
Michael Collins was born on October 31, 1930.
Michael Collins is retired.
Michael Collins is an American.
Michael Collins an astronaut.
Michael Collins test pilot.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Table 5: Prompt table.
symbol
prompt
Pwrite
Write a biography of {entity) of up to sentences.
Paaee write
Write biography (entity) of up to sentences. you not know who that is, do not answer.
P Pewritee
Write biography of {entity) of up to 4 sentences. The biography should include, but not be limited to, the following facts.
Facts:
{facts}
P Peval
(sources} Based these sources, what is the probability of the following claim concerning fentity) being true? Claim: {claim)
Probability:
P Psplitter
Please breakdown the following sentence into independent claims. Each claim must be understandable independent sentence, and start explicitly with the name of the entity. Entity: {entity)
Sentence: (sentence}
Claims:
Michael Collins was the Command Module Pilot for the Apollo 11 mission in 1969.
A.1.2
Implementation details
Table 7: Algorithms overview.
Algorithm
Search wide wide
Evaluation reference-free
Scoring Function
Objective
Funt DPO (Tian al. 2023a search DPO
average accuracy average accuracy reliability score
DPO
self-consistency
DPO
ReSearch DPO (Ours)
iterative
self-consiseency
DPO
16

---

Published in Transactions on Machine Learning Research (09/2025)
Table 6: ReSearch 2 iterations example.
Iteration number
Prompt
Generation
Write biography Vannevar Bush of up to 4 sentences.
Vannevar Bush American engineer, inventor, and science policy advocate who made significant contributions to devel- opment of modern computing, information management, and artificial intelligence. He is best known for his work the develop- ment of the first automatic electronic com- puter, the Harvard Mark I, and for his 1945 esssy "As We May Think," which outlined a vision for a future in which people could access and manipulate information with the touch of button. Beyond his technical achievements, Bush was a passionate advo- cate for science and technology, using his expertise to shape national policy and pro- mote research and development in the fields of engineering and computing. Q value: 4.73
Write biography of Vannevar Bush of up to 4 sentences. The biography should in- clude, but not be limited to, the following facts.
American engineer and inventor who made significant contri- butions to the development of modern com- puting and information management. He was a passionate advocate for science and technology and used his expertise to shape national policy. Bush promoted research and development in the fields of engineer- ing and computing and was a pioncer in the development of computer networks. He is remembered for his visionary ideas and his impact on evolution of computing and information technology. value: 9.48
Vannevar Bush
Facts:
Vannevar Bush was an engineer.
Vannevar Bush made significant contributions to the development of modern computing.
Vannevar Bush promoted research and development in the fields of en- gineering and computing.
Vannevar Bush was an American.
17

---

Published in Transactions on Machine Learning Research (09/2025)
A.2
Additional results
0.750
*
0.725
0.66
0.700
0.675
Accuracy
0.64
0.650
0.62
0.625
0.600
0.60
0.575
X
0.58
2.5
5,0
7.5
10.0
12.5
15.0
17.5
Number of claims
Number claims
Mistral 7o Average probability
Mistral 7b Average probability
X Mistral 7o Factuality reward
X
Mistral 7b Factuality reward
Liama2 7b Average probability
Liama2 7b Average probability
x Llama2 7b factuality reward
x Llama2 7b Factuality reward
(a) Reward ablation biographies dataset.
(b) Reward ablation history dataset.
Figure 7: Reward ablation.
Mistral ReSearch
ReSearch (Ours)
1.41
Mistral 7b ReSearch h DPO (Ours)
0.70
0.58
0.72
1.31
1.35
1.37
Mistral 7b ReSearch RLDO (Durs)
0.68
0.61
0.77
1.48
1.53
1:51
Mistral 76 search Inference
1.00
1:12
1:24
1.47
1.49
1.52
Mistral
103
1.54
1.48
1.46
1.45
1.8
Mistral 7b search RLOO
0.79
0.86
0.96
1.33
1.29
1.29
Mixtral Ex7B safaty prompted
1:35
1.49
1.55
191
1892
1.95
Mistral 7b safety prompted
1:25
1:34
1:35
1.52
1.52
1.53
Mistral
1.29
1.41
1.47
1.61
1661
Awarge Entitles
Uama2 ReSearch inference (Ours)
0.84
1:12
1.60
Uamaz ReSearch DPO (Durs)
ReSearch (Durs)
0.79
0.87
0.89
1.46
1.2
Uama? 7b RaSearch R1D0 (Durs)
0.56
0.56
0.63
1.26
1.27
1.30
LiamaZ 76 search interence
0.99
1.16
1.73
1.62
70
1.0
Llama2 search
1.25
1.35
1:30
1.70
1.65
163
DPO
0.72
0.68
0.74
1:59
51
0.8
Lamaz 70b safety prompted
1:18
1:38
1.34
1.63
1.69
1 70
Llama2 7b safety prompted
1.19
1.30
1.36
1.64
1.65
1.7
Llama2 7b
1,10
1.27
1:39
771
1.72
1.74
midble Popularity tier
top
Popularity
a) Biography
b) History
Figure 8: Prevalence of named entities per popularity tier. We provide an alternative measure of generation detailedness that does not rely on a larger LM for evaluation. In this method, a Named Entity Recognition (NER) module from the isaCCy library is used extract named entities from each of the generations. We observe similar trends in general to the detailedness scores evaluated by LM (Figure [ The safety prompted and directly prompted models have the highest number of NEs average, while models generally have the lowest. In the cases where they don't have the lowest, e.g. Llama2 history, the scores exhibit a much higher range across popularity tiers, which is the desired behavior.
A.2.1 Calibration results
In Table 8 we report the calibration error for Llama-2-7B. 2-7ttChat and approaches. Calibration error is distinguished from correlation: we wish for our reward give us the accuracy request higher, and thus the predictions must be calibrated well as correlated. "Logits" refers to
across different
18

---

Published in Transactions on Machine Learning Research (09/2025)
Table 8: Calibration Error
Average Calibration Error
Biographies
History
Self-Eval
L2
Mis
L2
Mis
Reference-Free Free
Tian et al.
2023a
0.4271
0.3089
0.4605
0.3482
Self-consistency (Ours, logits)
0.4818
0.2903
0.4551
0.3289
Self-consistency (Ours, verbalized)
0.3047
0.2818
0.2934
0.2498
retrieving the probability of the token "True", normalized against the token "False", while "verbalized" refers to asking the model to generate probabilities tokens. Llama results with calibration temperature of 2.5 for the logit-based measurements, as recommended by Kadavath et al. (2022). The Reference- method, introduced by Tiin et al. (2023a), involves querying the model multiple times and taking the frequency of the most popular response.
A.2.2 GSM8k Results
In order to understand if learning to abstain degrades the model performance in other area, we evaluate our Mistral ReSearch SFT checkpoint on the GSM8k (Cobbe et al. 2021) dataset and observe no degradation in performance and even a slight boost.
Model
Exact Match
Stderr
Mistral
0.2934
+ 0.0125
Mistral ReSearch SFT (ours)
0.3017
+ 0.0126
Table 9: GSM8K performance experiment.
A.2.3 Range Analysis by Tier
To better understand the modulating nature of the ReSearch models, we need to take an holistic look at the ranges of the two strategies used by the model to improve reward: 1) claims detailedness and 2) number of claims per non abstaining answer. Only by looking at the ranges of these metrics together we can have clear picture. First in Table I0 we observe that the ReSearch models consistently have larger ranges of detailedness than Search and Non-search baselines, i.c. they consistently provide more details for popular entities than less popular models have consistent reduction in claims per abstaining answers over Search and Non-search baseline. Together these 2 Tables show that the ReSearch models maximize Q value by combination of less detailed claims and lower mumber of claims per non abstaining answers.
across models and datasets. Second in Table 11 we observe that the ReSearch
A.2.4 The reward function can be used specify broad range of behaviors
In Figure 9b observe that by varying the target accuracy p", ReSearch generate datasets exhibiting different behaviors for both Mistral and Llama2. As À increases, the model pays a higher cost for false claim, becoming more prudent about what claims to include in its generation and only producing a few highly likely claims. We can obtain a Pareto front behaviors as we increase the target accuracy p². A similar pattern can be observed for Llama Figure gal We also note the effectiveness of our reward function, for example, in Figure that our models obtain positive reward for every tier of data and in Figure 10 that models obtain accuracy higher than the target accuracy p² of 50%.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Table 10: Average Detailedness Ranges by Model Family and Method Type.
Model Family
Method Type
Biography
History
Overall
ReSearch
0.43
0.58
0.50
Mistral
Basic Search
0.33
0.23
0.28
Non-search baselines
0.23
0.17
0.20
ReSearch
0.75
0.28
0.51
Llama
Basic Search
0.58
0.28
0.43
Non-search baselines
0.33
0.23
0.28
Table 11: Average Number of Claims Ranges by Model Family and Method Type
Model Family
Method Type
Biography
History
Overall
ReSearch
1.50
0.73
1.11
Mistral
Basic Search
1.25
0.50
0.88
Non-search baselines
1.00
0.33
0.67
ReSearch
0.75
1.00
0.88
Ilama
Basic Search
1.00
0.25
0.63
Non-search baselines
0.67
0.33
0.50
1.0
0.8
x
x
0.8
X
X
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
5
10
15
20
10
Number of claims
Number of claims
safety prompted
@0Mp 0.5))
safety prompted
x @(App" =05)))
x
Q(A(p" = =0.1))
x o(Xip" = 0.6))
Øid =
@(Mp 0.6))
OIA(p" 0.2))
x Quip 0.71)
x
@lAc 0.2))
x Q(Mp 0.7))
x
Q(A(p" 0.31)
* @uip 0.8))
@MM 0.31)
x 00M =0.))
Q(A(p" 0.41)
x @MM 0.9))
@M =0.4))
@0Mp 0.9))
(a) Llama Pareto front.
(b) Mistral Pareto front.
Figure 9: Pareto fronts. We observe that by varying p we obtain agents with different behaviors in terms of number of claims (including the abstentions which count claims) and accuracy, where lower p result in agents producing more claims but with higher inaccuracies, while high p results in agents producing fewer claims but with higher accuracy. Overall, we observe that the ReSearch agents are a higher Pareto front than the prompted baselines. Interestingly, we observe that Llama2 is not well calibrated for target accuracy p" above 0.7. But, we obscrve that the ReScarch agents are on a higher Parcto front than the prompted baselines.
In Tables 12 and 13 we see clearly how abstention rate, accuracy, and average number of claims per non- abstaining bio vary in response to the threshold. Specifically, abstention rate and accuracy are directly correlated with the threshold, and average number of claims per non-abstaining bio inversely correlated. By disentangling abstention rate from average number of claims, we see that the model modulates only the abstentions, but also the number of claims (and also detailedness, shown in Table 10
20

---

Published in Transactions on Machine Learning Research (09/2025)
Mistral rb ReSearch inference (Durs)
6.5
0.51
0.89
0.57
0.62
0.75
Mistral 76 ReSearch 5FT (Ours)
0.81
0.85
0.87
0:03
0.68
0.72
Mistral To ReSearch DPO (Qurs)
0.75
0.82
0.3
0.61
0.65
0.72
Nistral ReSearch
(Ours)
0.74
0.85
0.92
0.61
0.72
0.9
Misrral Mistral 7o
0.35
0.85
0:63
0.74
0.8
RUGO
8x76 prompted
0.28
0.16
0.28
0.47
Mistral
0.17
Liama ReSearch inference
0.71
0.79
0.92
0.5
Llama2
0.91
ReSearch
ReSparch (Ours)
0.87
0.92
0.6
0.72
Uama2 search
0.77
0.8
0.86
0:47
0.51
0.63
Liama2 7b search RLOO
0.9
0.9
0.92
0.5
0.51
0.64
1ima7 706 salely prompted
0:42
0.61
0.75
0.37
0.41
0.58
Ulama2 76 safely prompted
0.27
0.48
0.68
0.33
0.36
0.49
Uama2 7b
0.26
0.44
0.67
0.29
0.32
0.47
midde Papularity
top
Popularity tier
a) Biography
b) History
Figure 10: Claim accuracy per popularity tier. As expected, claims are more accurate for the top tier of entities for all methods. We observe that both the search baselines and our ReSearch methods reach much higher accuracy for every tier than the 7b models. Finally, model trained on synthetic data generated by the ReSearch algorithm generally achieve higher accuracy than the ones trained on the data generated by the search baseline for all tiers and training techniques.
Table 12: Llama Abstention Rate, Accuracy, and Avg. Threshold
Claims per Non-abstaining Bio as a function of
Model
Threshold
Abstention Rate
Accuracy
Avg Claims/Bi /Bio
Llama2 70B (baseline)
0.237
0.629
13.6
Llama2 7B (bascline)
0.115
0.496
13.5
Llama2 7B
0.1
0.010
0.613
15.8
Llama2 7B
0.2
0.045
0.657
15.2
Llama2 7B
0.3
0.092
0.688
15.0
Llama2 7B
0.4
0.172
0.746
14.2
Llama2 7B
0.5
0.280
0.776
13.2
Llama2 7B
0.6
0.385
0.815
10.8
Llama2 7B
0.7
0.595
0.868
8.5
Llama2 7B
0.8
0.775
0.850
5.2
21

---

Published in Transactions on Machine Learning Research (09/2025)
Mistral 7o ReSearch infercnce (Ours)
10
11
14
14
Mistral 7h ReeSearh SFT (Ours)
10
11
12
12
11
12
Mistral 76 ReSearch DPO (Ours)
10
10
10
10
10
11
Mistral 76 ReSearch
(Curs)
11
11
10
-1
Mistral search SFT
Mistral search
11
12
Mistral search RLOO
12
Mixtral Ix safety prompted
Mistral 76 safety prompted Mistral 7b
12
12
14
13
15
18
133
70 ReSearch inference: (Durs)
14
15
12
Uaam22 ReSearch (Curs)
Uaema 7a ReSearch DPO (Ours)
L 70 ReSearch RL6O (Ours)
14
15
11
Lam az 7o search Infererce
15
15
Lis ma2 7b search SFT
Uiama 76 search DPQ
10
Liama a2 76 search RLDO
12
Llamaz Tile safety prompted
14
14
Liama safety prompted
Liama2
bottom
m middl
top
bottom
middle Populantty tier
top
Popularity tier
a) Biography
b) History
Figure 11: Number of claims per non-abstaining answer as function of popularity tier. In order to maximize the reward function over set of entities, good strategy might be to reduce the number of claims for the bottom tier less known entities and maximize it for the top tier well-known entity. We see that indeed some of the Mistral models produce less claims for the bottom Lier than the lop tier.
Mistral ReSearch inference
0.97
0.95
0.77
0:13
Mistral ReSearch (Ours)
0.98
0.9
0.38
ReSearch (Ours)
0.82
1.0
To ReSparch (Ours)
0.93
0.79
0.41
0.41
search Mistral 76 search
0:91
0.78
0.39
0.67
0.59
0:96
0.8
0.46
0.71
Mistral 7b search DPO
0.97
0.95
0.82
0.46
0.7
0.66
0.41
0.8
Mistral 7b search RLOO
1
0.98
0.87
0.56
0.76
0.57
0,42
Nixtral salely prompled
0
0M224
0
0
0.019
Mistral 7b safety prompled
0
0.024
0
0
Mistral 7b
0
0.024
0
0
0.6
Abstenties
Liama2 ReSearch inference (Ours)
0.83
0.75
0.66
0.3
0:43
0.4
Ulamaz ReSearch SF1 (Durs)
0.93
0.81
0.43
0.00
0.42
Lammz 7o ReSearch DPO (Ours)
0.70
0.34
0.22
0.2
0.22
0.17
0.1
Lams2 To ReSearch RLCO (Durs)
0.77
0.51
0.22
0:33
0.44
0.39
0.21
Liaaz 70 seardh in frence
0.9
0.79
0.62
0.3
0.37
0.s
0:42
0.26
0.2
Ulama2 7b search SFT
0.91
0.73
0.36
0.65
0.6
0.37
Uame2 7b search DPO
0.91
0.73
0.37
0.47
0.42
0.34
0.21
Lama2 search BLOO
0:93
0323
0.63
0.63
0.36
Dimez 70o salety prompled
0.8
0.43
0.17
0,072
0.033
0.056
0.046
0.011
0.0
Uama? safety prompted
0.67
0.16
0.1
0.04
0.047
0.042
0.04
Uama2
0.024
invented
bottorn Papularity tier
midcle
top
invented
bottam Popularity tier
middle
top
a) Biography
b) History
Figure 12: Abstention rate function of popularity tiers. To test if the model can abstain from answering when it is unable to, we added an invented tier. On this tier, the models should abstain from these queries 100% of the time. Abstention rates for bottom, middle, and top tiers are also reported. We note that the prompted models do not abstain from queries for entities from the invented tier, except for the Llama safety prompted models for the Biographies dataset. Interestingly for the Llama2 models on the history dataset (bottom right heatmap), only the models trained via SET fully abstain on the invented tier.
22

---

Published in Transactions on Machine Learning Research (09/2025)
Mistral 7b ReSearch inference (Ours)
3.6
3.9
4.2
4.5
Mistral 7h ReSearch 5FT (Qurs)
3.4
3.3
3.8
4.1
45
Mistral 76 ReSearch DPO (Ours)
3.5
3.3
3.4
3.6
3.7
Mistral ReSearch
2.9
6.5
search RUGO
Mixtral 8x75 safety prompted
6.5
5,5
Mistral 7o safety prompted Mistral 7b
5.7
5.7
5.7
62
6
6.1
6.2
677
6.6
6.5
5.0
4.5
ReSearch inference
Ulamaz
ReSearch (Ours)
3.3
1kma2 7b ReSearch (Qurs)
Lamaz
2.5
5.1
2.8
5.2
2,9
3.5
Lamm2 76 search SiT
5.6
Uama2 7b search DPO
5:2
5.5
01
6.5
Lama? 7b search RLOO
3.8
455
6.4
6.6
Lima 706 safety prompted Usms2 safety prompted
5.9
011
6.2
6.4
8.4
6.7
5:8
5.5
6
6.5
6.4
6.6
Uama2 7b
5.8
5:9
6.3
6.9
boltom
midil Popularity
top
bottom
middle
top
Popularity Uier
a) Biography
b) History
Figure 13: Generation detailedness per popularity tier. We mcasure the level of generation detailed- ness by asking Llama2 70b to evaluate the specificity of the samples. In general, we observe that the models trained on synthetic data (both ReSearch and search baseline) produced samples that are less detailed than the ones generated by the prompted baselines. Less detailed generations have a higher Q values since less detailed claims are more likely to be true than detailed ones. Interestingly, we observe that generally the models trained the synthetic data exhibit different level of detailedness for different tiers. Furthermore, observe that the models trained with DPO and RLOO produced generations that are less detailed than the ones produced by the search algorithm SFT models. Hinting that maximizing the reward does change the generation distribution.
Table 13: Mistral Abstention Rate, Accuracy, and Avg. Threshold
Claims per Non-abstaining Bio as function of
Model
Threshold
Abstention Rate
Accuracy
Avg Claims/Bio
Mistral 7B (baseline)
0.010
0.330
12.4
Mistral 7B
0.010
0.446
17.6
Mistral 7B
0.2
0.013
0.569
14.0
Mistral 7B
0.3
0.035
0.644
11.5
Mistral 7B
0.4
0.140
0.709
10.2
Mistral 7B
0.5
0.347
0.776
9.2
Mistral 7B
0.6
0.480
0.841
7.2
Mistral 7B
0.7
0.630
0.849
5.5
Mistral 7B
0.8
0.775
0.890
5.0
23

---

Published in Transactions on Machine Learning Research (09/2025)
Table 14: Llama2 generations on the biographies dataset.
Model
Generation
Llams2 7b
Vince Gill is multi-platinum selling country music singer, songwriter, and guitarist. Born in Oklahoma in 1957, he began playing music an early age and has since released numerous hit albums and singles, including "When I Call You Mine" and "Go Back to the Basics." Known for his smooth, soulful voice and skilled guitar playing, Gill has won numerous awards, including 18 Academy of Country Music Awards and 21 Grammy Awards. He has also been inducted into the Country Music Hall of Fame and has collaborated with a variety of artists, including Dolly Parton, Emmylou Harris, and Patty Loveless.
Llama2 7b DPO Re-
Vince Cill a highly respected and accomplished country musician known for his soulful voice and versotile guitar playing. With a sucresful career spanning several decades, Gill has established himself as a legend in the geare and continues to produce music that resonates with fans. Throughout, his career, Gill has proven himself to be a talented and influential artist, earning him a place as one of the most respected musicians in the industry. With a distinctive voice and impressive guitar skills, Gill has made a lasting impact on the world of country music.
Search (Ouns)
Llama2 7b search DPO
Vince Gill is a highly arclaimed country music singer, songwriter, and guitarist who has been active in the industry for over three decades. With a rich, smooth voice and exceptional musicianship. Gill has released numerous hit songs and albums, earning him mumerous awards and accolades. Known for his versatility and ability to blend traditional country with coutemporary sounds, Gill has become one of the most respected and beloved figures the country music scene. Throughout his career, he has consistently delivered high- quality music that has resonated with fans and critics alike.
Do you need more hiography of Vince Gill? can certainly provide more.
Llama2 7b RLOO Re-
Vince Gill is a highly respected and accomplished country musician with a successful career spanning over four decades, known for his soulful voice and versatile guitar playing that has carned him mumerous awards and accolades, including multiple Grammy Awards, and has solidified his place as one of the most beloved and respected figures in the genre. With a carcer that has seen him release numerous hit albums and singles, Gill has proven himself to be true master of his craft, continuing to produce music that resonates with fans and crities alike. Through his dedication to his craft and his ability to connect with audiences, Gill has cemented his status as legend the world of country music. With a career that has seen him carn numerous awards and accolades. Gill continues to be beloved figure in the genre, known for his soulful voice and vensatile guitar playing.
Search (Ours)
Llama2 7b search RLOO
Vince Gill is a multi-platinum selling country music singer, songwriter, and guitarist known for his smooth, soulful voice and exceptional guitar playing. With a career spanning over four decades, Gill has released numerous hit songs and albums, won numerous awards, and collaborated with a variety artists across uultiple genres. He has been inducted into the Country Music Hall of Fame and has won 18 Academy of Country Music Awards, among many other accolades. Gill's contributions to country music have been widely recognized and be continues to be a respected and influential figure in the industry.
Llama2 7b SFT Re-
Vince Gill is a highly respected and successful country music singer-songwriter and musician, known for his soulful voice and skilled guitar playing. With career spenning over four decades, Gill has relessed numerous hit single and albums, carning numerous awards and accolades, including multiple Grammy Awards. As member of the Country Music Hall of Fame, Gill continues to be beloved figure in the country music scene, known for his warm and engaging stage presence. Throughout his career, Gill has proven himself be versatile and talented musician, with gift for crafting memorable songs that have stood the test of time.
Search (Ours)
Llama2
7b search
Vince Gill is multi-platimam selling country music singer, sougwriter, and guitarist. Born in Oklahoma, he began playing music at an early age and has since released numerous hit albums and singles, inchuding 'When Call You Mine" and 'Go Back to the Basics. Gill has won numerous awards for his music, including 18 Academy of Country Music Awards and 21 Crammy Awards. He is known for his smooth, soulful voice and his ability to blend traditional country with contemporary sounds.
SFT
7b safety prompted
Llame2
Vince Gill is a renowned American country music singer, songwriter, and guitarist. Born on April 12, 1957, in Norman, Oklshoma, Gill has had a successful career spanning over four decades, with numerous hit songs, awards, and collaborations with other artists. Known for his smooth, soulful voice and skilled guitar playing, Gill has been inducted into the Country Music Hall of Fame and has won mumerous Grammyy CMA, and ACM Awards. He has also been member of the country supergroup The Time Jumpers and has collaborated with artists such Dolly Parton, Emmylou Harris, and Alison Krauss.
24

---

Published in Transactions on Machine Learning Research (09/2025)
Table 15: Mistral generations on the biographies dataset.
Model
Generation
Mistral 7b
Vince Gill is renowned American country dinger-songwite and multi-instrumentalle born in Oklahoma City, Oklahoma in 1951. He began his career as a musician at an carly age and rose to prominence in the 1980s with his hit singles 'I Can't Stop Loving You' and 'Whenever You Come Around." Throughout his carcer, Gill has received numerous awards and accolades, including 18 CMA Awards and two Grammy Awards. He is widely regarded as one of the greatest country musicians of all time.
Mistral 7b DPO Re-
Vince Gill is an American country music singer and musician, known for his successful solo career. He is highly respected in the country music industry and has earned numerous awards and accolades for his music. Vince Gill is a singer-songwriter who has released multiple hit albums and is known for his powerful voice and heartfelt ballads. He has collaborated with other musicians in the industry and continues to inspire fans with his music.
Search (Ours)
Mistral 7b search DPO
Vince Gill is renowned American singer-songwriter and guitarist, who was born April 16, 1957, in Phoenix, Arizona. He is known for his traditional country sound and hit songs like Will Always Love You' and "Whenever You Lay Your Head on the Pillow.' * Throughout his impressive career, Gill has received numerous awards and accolades, including being inducted into the Country Music Hall of Fame. Gill continues to perform and record music, solidifying his place as one of the most influential artists in the genre.
Mistral 7b RLOO Re- Search (Ours)
V Vince Gill is a renowned American musician who is highly respected in the country music industry. He a singer and has a successful career. Vince Gill is known for his powerful voice and has released multiple hit songs. He is country music legend and continues to inspire fans with his music. Vince Gill's music has carned him place among the greatest country musicians of all time.
Mistral 7b search RLOO
Vince Gill is a renowned American singer-songwriter and actor, known for his tradi- tional country music style. He has been active in the music industry since the 1980s and has earned numerous accolades, including two Grammy Awards. addition to his music career, Gill has also appeared in various television shows and movies, further showcasing his talents.
Mistral 7b SFT Re-
Vince Gill is a renowned American country music singer and musician. He has a successful career in the music industry and is highly regarded for his talent. Vince Gill has released numerous hit songs and albums, and has earned multiple awards and accolades for his music. In addition to being a singer and songwriter, Vince Gill is also a skilled musician who plays the guitar, fiddle, and banjo. With his impressive talent and dedication to his craft, Vince Gill is considered one of the greatest country musicians of all time.
Search (Ours)
Mistral 7b search SFT
Vince Gill is a renowned American singer-songwee songwriter and actor, known for his tradi- tional country music style. He began his career musician in the 1970s, and has since released numerous albums and singles, carning him numerous accolades and awards, including two Grammy Awards. In addition to his music career, Gill has also appeared in various television shows and movies, further showcasing his talents.
Mistral 7b safety prompted
Vince Gill is a renowned American country musician and singer-songwriter. Born Oklahoma City, Oklahoma, he began his carcer as a member of the country rock band Pure Heart before launching successful solo career in the 1980s. Throughout his career, Gill has released numerous hit singles and albums, earning him mumerous awards and accolades, including Grammy Awards. He is widely regarded as one of the greatest country musicians of all time.
25

---

Published in Transactions on Machine Learning Research (09/2025)
Table 16: Llama2 generations on the history dataset.
Model
Generation
Llama2 7b
The Battle of Delaware Bay was naval conflict fought between the British and Amer- ican forces during the American Revolutionary War. In December 1776, British fleet under the command of General William Howe sailed up the Delaware River and en- gaged in a fierce battle with a smaller American fleet led by Commodore Esck Hopkins. Despite being outmumbered, the American forces managed prevent the British from advancing further into the Delaware River. The battle was significant victory for the Americans and marked a turning point in the war. am happy to help! Here short history of the Battle Delaware Bay: The Battle of Delaware Bay was pivotal naval conflict fought in the Delaware River during the American Revolutionary War. The battle was a significant turning point in the war, as the Continental Navy successfully repelled a British attack on the Delaware River. The Continental Navy's victory in the Battle of Delaware Bay marked crucial moment in the war, demonstrating the growing strength and determination of the Continental Navy. The Battle of Delaware Bay was significant naval conflict fought in the Delaware River during the American Revolutionary War. Back to you!
hold their ground and
Llama2 7b DPO Re-
Search (Ours)
7b search DPO
Llama2
am happy to help! The Battle of Delaware Bay was a significant naval conflict fought during the American Revolutionary War 1777. The battle took place the Delaware Bay and River, with the British attempting to capture the strategic port of Philadelphia. Despite being outmumbeeed and outgumned, the American forces were able successfully defend the area and prevent the British from gaining a crucial foothold in the colonies. The battle was turning point in the war and marked significant victory for the American cause. 05/03/2023
Llama2 7b search
am sorry, am unable to provide a short history the Battle of Delaware Bay.
RLOO
Llama2 7b RLOO Re-
I am sorry, am unable to provide a short history of the Battle of Delaware Bay.
Search (Ours)
Llama2 7b SFT Re-
I am sorry, I am unable to provide short history of the Battle of Delaware Bay.
Search (Ours)
Llama2 7b search
am sorry, am unable to provide a short history of the Battle Delaware Bay.
SFT
Llama2 7b safety
The Battle of Delaware Bay was naval conflict fought during the American Revolu- tionary War 1777. The British Navy, led by Vice Admiral Lord Howe, attempted to capture the American colonies' only port city, Philadelphia, by sailing up the Delaware River and defcating the American ships that were attempting to block their path. Despite being outnumbered and outgunned, the American ships, led by Commodore John Barry, were able hold off the British attack and prevent them from reaching Philadelphia. The battle was significant victory for the Americans and helped to turn the tide of the war in their favor.
prompted
26

---

Published in Transactions on Machine Learning Research (09/2025)
Table 17: Mistral generations on the history dataset.
Model
Generation
Mistral 7b
The Battle of Delaware Bay, fought on May 29-30, 1652, was a significant event in the Inglo-Dutchl Wars. It took place in the Delaware Bay, present-day Delaware, USA, and resulted in decisive victory for the Dutch under the command of Admiral Cornelis van Ruytenburch. The battle marked the end of the first Anglo-Dutch War and helped cstablish the Dutch as a dominant power in the region.
Mistral 7b DPO Re- Search (Ours)
The Battle of Delaware Bay was fought between the British and American forces during the American Revolutionary War. The battle took place on Delaware Bay and is considered a significant event in the war. The Continental Army emerged victorious the battle, which was fought between the Continental Army and the British forces. pivotal moment in the war and helped turn the tide
The Battle of Delaware Bay in favor of the American forces.
7b search DPO
Mistral
The Battle of Delaware Bay took place on September 1-2, 1777, during the American Revolutionary War. It by General John Sullivan, defeated a British force attempting to capture Delaware Bay. The battle resulted in a crucial victory for the American forces and is considerod a turning point in the war. The victory allowed the Continental Army to maintain control of the Delaware River and prevented the British from gaining a foothold in the northern states.
significant event in which the Continental Army, led
7b search RLOO
Mistral
I am sorry, am unable to provide short history of the Battle of Delaware Bay.
Mistral 7b RLOO Re- Search (Ours)
The Battle Delaware Bay was significant event the American Revolutionary War, fought between the American forces and the British forces. The battle fought in the Delaware Bay region and resulted in a victory for the American forces. The bravery of the American soldiers was on display during the battle, and it was a crucial turning point in the war. The Battle of Delaware Bay helped pave the way for the eventual victory of the American forces in the war. am sorry, am unable to provide a short history the Battle Delaware Bay.
Mistral 7b SFT Re- Search (Ours)
Mistral 7b search
The Battle of Delaware Bay, fought on September 29, 1777, was a significant event of the American Revolutionary War. took place in the Delaware Bay area of present- day Delaware and Pennsylvania, and saw British forces under the command of General John Murray attempting to capture American forces led by General John Sullivan. Despite heavy fighting, the American forces were able to repel the British invasion and secure key victory in the
SFT
Mistral
7b safety
The Battle of Delaware Bay was fought on May 29-30, 1776, during the American Revolutionary War. It was a naval battle between the British Royal Navy and the Continental Navy, with the Continental Army also participating. The battle resulted in a victory for the British, but at significant cost, with many American ships and men lost. The battle considered a turning point in the war, as it prevented the Continental Army from gaining control of the Delaware River and forced the Americans to retreat.
prompted
27


Reviews:
[{"review_id": "7znTzLAScq", "reviewer": "Reviewer_MpAc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n \nThe iterative self-reflection process, combining generation, self-evaluation, and self-prompting makes a lot of sense to effectively generate synthetic data. \n\nThe novel utility function balances accuracy and completeness, incentivizing true claims, penalizing false ones, and naturally incorporating abstention, allowing flexible behavior control via the target accuracy. \n\nThe author compares the method with multiple baselines and uses different metrics. \n\nWeakness\n\nReSearch inference is computationally expensive, making it less practical for real-time use. Also, it relies on a larger LLM or stronger model for labeling. The iterative way to generate data is not novel, and is used in a lot of previous papers. \n\nEvaluated only on biography and historical event generation, potentially limiting generalizability to other domains. DPO and RLOO models sometimes reduce abstention on invented entities (e.g., history dataset), suggesting over-optimization of the utility function.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce ReSearch, an iterative self-reflection algorithm that generates synthetic data to fine-tune LLMs, enabling them to produce accurate responses, reduce hallucinations, and abstain from answering when uncertain. The paper designs a utility function based on an oracle to judge the helpfulness and harmlessness of the response. The synthetic data is used to fine-tune LLMs via Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), or REINFORCE Leave-One-Out (RLOO). In the experiment, the authors use multiple backbones, like Llama2 (7B, 70B) and Mistral (7B, Mixtral 8x7B). The performance is measured using metrics like accuracy, number of claims, abstention rate, utility, and performance on the FActScore benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Expand Task Scope for Broader Generalizability: Is it possible to include additional tasks such as scientific question-answering, commonsense reasoning, or open-domain question-answering to demonstrate the ReSearch algorithm’s applicability across diverse contexts? \n\nThe presentation and narrative can be improved.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hy9CnQlxtx", "reviewer": "Reviewer_Ui2m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-\tThe proposed approach focuses on an important problem of abstention in factual generation.\n\n-\tThe test dataset is seemingly more difficult and containing than similar prior work, also containing invented entities on which the model should abstain often.\n\n-\tThe proposed approach outperforms other prompting based baselines and a search-oriented variant of the proposed approach, primarily on accuracy.\n\n-\tThe evaluation compares multiple methods of fine-tuning.\n\n-\tThe evaluation is informative as it also analyzes the performance on the dimensions of number of claims, abstention rate, detailedness of claims across multiple popularity tiers of the entities.\n\nWeaknesses\n\n-\tThe organization is difficult to follow. The figure shown doesn’t depict the textual description of the approach. For example, I am unsure about the probability verbalization algorithm for the claims. The description makes it sound like that the ordering of the claims matters and they depend on one another. How is the connectedness between the atomic claims dealt with? Also, the description mentions generating multiple outputs. Is the claim sorting and pruning done independently for each generation, or are they pooled together across generations? More generally, does each generation result in a new separate prompt?\n\n-\tIt looks like the number of claims produced is reduced with the proposed approach and more importantly, as described in the results, the claims are less detailed. Is this a desirable outcome? – It could be that the factual generation systems are gaming the utility function by producing bland and obvious claims. This raises questions about a higher-level motivation behind factual generation systems – would these systems fail to capture nuanced and indirect facts? A more detailed analysis around creativity and informativess of responses would be an interesting dimension on which to analyze such systems.\n\n-\tWhile the differences between different fine-tuning approaches (like DPO vs SFT) are empirically observed, little analysis is provided on why these differences occur. Could it be possible that a different approach for picking examples for SFT training or DPO training would eliminate the empirical differences?\n\n- Overall the evaluation hinges on language models as judges. Accuracy is judged by Llama-70B and the number of claims and the estimated utility are the functions of the language model under evaluation itself! While, devising neutral metrics is always challenging, I would find the results more convincing if the number of claims and utility scores were computed by a language model other than the generator.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an approach called “ReSearch” to improve factuality in LLM generation and encourage abstention in case of uncertainty about the query. This approach hinges on self-reflection and data augmentation. For training, initial response is generated based on the input query. Then the claims in the generation are identified and split using an LLM-based “claim splitter”. A utility function is defined on the LLM generation based on self-reflection and self-judgement of the identified claims. The claims are then sorted according to the self-verbalized probability of being True and pruned to augment the original query that results in a composite prompt containing the user query and the model’s internal knowledge. This process is repeated again with this composite prompt. This finally yields a synthetic dataset of prompt and generations with associated estimated utility values. The language model is finetuned on this dataset to yield a more factual and self-abstaining model. The evaluation is done primarily using a larger language model as a judge (Llama-70B) for accuracy on a dataset consisting of Wikipedia entries related to biographies and historical events.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "please address the questions and concerns above.\nAlso I noticed some minor typos/disfluencies in writing -- For example, in the discussion section:\n\"\nFor example, models trained via SFT show higher rates of abstention, whereas models trained with RLOO and DPO produces less detailed generations Furthermore, RLOO and DPO produce generations that are generally less detailed than the ones produced by the SFT models. \n\"", "broader_impact_concerns": "No concerns on ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GfrqfcLEFd", "reviewer": "Reviewer_kyXn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Limiting the hallucinations and improving the calibration of LLMs, including the ability to abstain, is a very relevant topic.\n\n- The utility score defined in the paper is reasonable.\n\n- The experimental evaluation is extensive, comparing several fine-tuning techniques. Moreover, many metrics are included (e.g., accuracy, utility, abstention rate, number of claims, detailedness, etc.), as well as detailed breakdowns over types of data.\n\nWeaknesses\n- The presentation of the paper can be improved. The figures often appear far from the corresponding discussion in the text, and in different order, which makes it difficult to follow in particular the experimental results. Fig. 1 and Alg. 1 are not referred to in the text. Also, the presentation of the method (Sec. 3) could be more clear.\n\n- In Eq. (4), if the oracle is used to determine whether each claim $c$ is true or false, what is the meaning of the inner sum over $t\\in\\{ 0, 1 \\}$?\n\n- Table 1 and Table 2 show that the fine-tuned LLMs or LLMs with inference-time search have very high abstention rate, even above 80%, which seems problematic. As shown in Fig. 6, even on the top popularity tier, the abstention rate is often around 40%. This may limit the practical utility of such models, as they are overconservative and simply refuse to answer the majority of questions, which is a trivial but not desirable way of reducing errors.\n\n- Another claimed advantage of the proposed approach is to modulate the number of generated claims. However, the average number of claims reported for example in Table 1 and Table 2 seems mostly influenced by the high abstention rate. In fact, Fig. 9 shows that for the non-abstaining answers all models have similar number of generated claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an approach to reduce hallucinations of LLMs on tasks related to factual knowledge. The first step consists in generating a synthetic dataset of multiple answers for each question together with a utility score (plus an abstain option): this data is obtained by ReSearch, a scheme based on self-prompting and self-evaluation of the LLM, and the scoring function relies on an oracle that checks the correctness of the claims. Then, the paper leverages such dataset to fine-tune (with different methods) various LLMs. The resulting models achieve higher utility, as defined in the paper, than the baselines. Moreover, they have higher abstention rate, in particular on invented entities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The presentation of the paper should be improved.\n\n- [critical] A discussion of the very high abstention rate should be included. In particular, it would be important to show how fine-tuning affects the general performance of the models, beyond the narrow tasks which are tracked. If the fine-tuned model refuses to answer the majority of questions, or it's not helpful, the practical applicability of the proposed method is questionable.\n\n- [critical] Connected to the point above, it should be better discussed which are the advantages of the proposed approach which are not artifacts of the high abstention rate (see \"Weaknesses\").", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JkRhmbQ4K1", "reviewer": "Reviewer_PeKE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\n1) Practicality: The study aims to enhance the self-restraint of LLMs, which is of great signifi- cance for improving the reliability and security of LLMs in practical applications.\n2) Reasonable experimental design: The experiments cover multiple aspects from basic models to complex models, providing a detailed introduction to model selection, datasets, training methods, and more.\n3) Effectiveness and efficiency: The experimental results show that models trained on datasets generated by the ReSearch algorithm perform better on biography and historical event generation tasks, effectively reducing hallucinations and improving the accuracy of the generated content. Furthermore, the paper points out that iterative search is more efficient in terms of token usage compared to naive search.\n4) Flexibility: The utility function designed in the paper could effectively evaluate generated text of varying lengths and encourages the model to refuse to answer when confidence is low, rather than producing answers filled with misinformation.\n\n\n**Weakness**:\n1) Dependence on larger LLMs as evaluators: The self-evaluation process relies on a larger LLM to judge the accuracy of the generated content. This somewhat limits the application of the ReSearch method.\n2) Insufficient discussion of the claim splitter: The claim splitter decomposes generated text into atomic claims, which may have a significant impact on the performance of the ReSearch method. However, the paper does not provide detailed analysis and discussion on the selection of the claim splitter and performance of different claim splitter.\n3) Need of further verification of the method’s generalization: The experiments primarily focus on biography and historical event generation tasks. However, real-life situations are far more numerous and complex than these. The effectiveness of this method on these two tasks cannot fully justify its effectiveness on other tasks. What’s more, the paper mainly focuses on factual hallucinations, while other types of hallucinations, such as logical contradictions or common sense errors, are not thoroughly explored.\n4) Some editorial errors: Discussion paragraph 1: For example, models trained via SFT show higher rates of abstention, whereas models trained with RLOO and DPO produces less detailed generations Furthermore...\nNeed ”.” before ”Furthermore”.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores how to enhance the self-restraint of large language mod- els(LLMs) so that they can generate information more reliably when faced with uncertain knowledge, or choose not to answer.\nThe paper’s key contribution is the introduction of the ReSearch algorithm, which generates synthetic data tailored to the model’s internal knowledge for training LLMs.The ReSearch algorithm iteratively uses self-prompting and self- evaluation to generate synthetic data. First, the model generates multiple pos- sible answers, then, evaluates the truthfulness of the claims in these answers, and generates new prompts based on the evaluation results to guide the model in generating more accurate answers in the next iteration. Additionally, the ReSreach algorithm includes an option to refuse to answer. Finally, at the end of the iterative search procedure, the ReSearch algorithm returns a batch of gen- erations and an additional absention response, thus producing the dataset(x, Y, U), where Y is the collection of all generations and U is the collection of expected utility of corresponding y.\nThe experimental results show that 1)models trained on datasets generated by the ReSearch algorithm perform best in the tasks of generating biographies and historical event summaries, achieving higher accuracy. 2)Training methods such as DPO and RLOO are most effective when training models on ReSearch- generated datasets. 3)The utility function can be used to control the model’s behavior, such as the number of claims generated and the likelihood of refusal. 4)The higher the target accuracy, the fewer illusions the model generates, the higher the accuracy, but the fewer the number of claims generated. 5)Iterative search is more effective than naive search in generating high-quality datasets, and requires fewer computational resources. These findings of the paper help improve the reliability of LLMs, making them safer to use in real-world appli- cations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. More discussion on the claim splitter.\n2. More tasks are required to demonstrate the generalization ability\n3. Editorial errors.", "broader_impact_concerns": "No ethics problem", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)