An Emulator for Fine-tuning Large Language Models using Small Language Models
Authors: Eric Mitchell
Rafael Rafailov
Archit Sharma
Chelsea Finn
Christopher D Manning
Venue: ICLR 2024 poster (2024)
Keywords: pre-training
fine-tuning
decouple
scale
reward
alignment
Primary area: general machine learning (i.e., none of the above)
TL;DR: We show a principled, hyperparameter-free approach to decoupling the knowledge gained from pre-training and fine-tuning, enabling study of the source of model capabilities and showing benefits in factuality and helpfulness

Abstract: Widely used language models (LMs) are typically built by scaling up a two-stage training pipeline: a pre-training stage that uses a very large, diverse dataset of text and a fine-tuning (sometimes, 'alignment') stage that uses targeted examples or other specifications of desired behaviors. While it has been hypothesized that knowledge and skills come from pre-training, and fine-tuning mostly filters this knowledge and skillset, this intuition has not been extensively tested. To aid in doing so, we introduce a novel technique for decoupling the knowledge and skills gained in these two stages, enabling a direct answer to the question, *What would happen if we combined the knowledge learned by a large model during pre-training with the knowledge learned by a small model during fine-tuning (or vice versa)?* Using an RL-based framework derived from recent developments in learning from human preferences, we introduce *emulated fine-tuning (EFT)*, a principled and practical method for sampling from a distribution that approximates (or 'emulates') the result of pre-training and fine-tuning at different scales. Our experiments with EFT show that scaling up fine-tuning tends to improve helpfulness, while scaling up pre-training tends to improve factuality. Beyond decoupling scale, we show that EFT enables test-time adjustment of competing behavioral traits like helpfulness and harmlessness without additional training. Finally, a special case of emulated fine-tuning, which we call LM *up-scaling*, avoids resource-intensive fine-tuning of large pre-trained models by ensembling them with small fine-tuned models, essentially emulating the result of fine-tuning the large pre-trained model. Up-scaling consistently improves helpfulness and factuality of instruction-following models in the Llama, Llama-2, and Falcon families, without additional hyperparameters or training. For reference implementation, see [https://github.com/eric-mitchell/emulated-fine-tuning](https://github.com/eric-mitchell/emulated-fine-tuning).

Published as a conference paper at ICLR 2024
AN EMULATOR FOR FINE-TUNING LARGE LANGUAGE MODELS USING SMALL LANGUAGE MODELS
Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, Christopher D. Manning Stanford University eric. wee..mitchelll 10cs stanford.dfod
ABSTRACT
Widely used language models (LMs) are typically built by scaling up two-stage training pipeline: a pre-training stage that uses very large, diverse dataset of text and a fine-tuning (sometimes, 'alignment") stage that uses targeted exam- ples or other specifications of desired behaviors. While it has been hypothesized that knowledge and skills come from pre- training, and fine-tuning mostly filters this knowledge and skillset, this intuition has not been extensively tested. To aid in doing so, we introduce a novel technique for decoupling the knowledge and skills gained in these two stages, enabling a direct answer to the question, What would happen if we combined the knowledge learned by large model during pre-training with the knowledge learned by a small model during fine-tuning (or vice versa)? Using an RL-based framework derived from recent developments in learning from human preferences, we introduce emulated fine-tuning (EFT), a principled and practical method for sampling from distribution that approxi- mates (or 'emulates") ) the result of pre-training and fine-tuning at different scales. Our experiments with EFT show that scaling up fine-tuning tends to improve helpfulness, while scaling up pre-training tends to improve factuality. Beyond decoupling scale, we show that EFT enables test-time adjustment of competing behavioral traits like helpfulness and harmlessness without additional training. Fi- nally, special case of emulated fine-tuning, which we call LM up-scaling, avoids resource- inttnsive fine-tuning of large pre-trained models by ensembling them with small fine-tuned models, essentially emulating the result of fine-tuning the large pre-trained model. Up-scaling consistently improves helpfulness and factu- ality of cerruuctioo-oollwwiii models in the Llama, Llama-2, and Falcon families, without additional hyperparameters or training. For reference implementation, see https: / i thub. com / er wwwricmmiii c-mi tchel emu et - fine- tee-unnnn..
INTRODUCTION
Widely used foollociion-fllloiingg large language models (LLMs) typically follow a two-stage train- ing procedure, with stage of unsupervised pre- treitrain large, diverse dataset followed by pervised fine-tuning on a much smaller, carefully curated dataset (Raffel et al., 2020; Chung et al., 2022). While both stages are important in producing models that possess broad world knowledge and perform a given task reliably, identifying exactly what capabilities emerge in which stage and what scale is difficult (Wei al., 2022: Schaeffer et al., 2023). For example, pre-trained models typically require careful prompting in order to perform a task; after fine-tuning for instruction fol- lowing, they typically do not. Evaluation of the extent to which the core capability of 'instruction following' is learned during pre-training vs. during fine-tuning is thus seriously complicated by the choice of this prompt. To enable more direct attribution of capabilities introduce a principled technique for emulating the result of combining the capabilities gained from pre-training and fine-tuning different model scales; see Figure 1. This technique, which we call emulated fine-tuning (EFT), enables: a) direct study of the capabilities that change as only one stage is scaled up or down; b) the practical benefit of approximating the result of fine-tuning a large model without the associated computational expense; and e) the ability to modify the fine-tuning objective (e.g.. the tradeoff between helpfulness and harmlessness) at test time, without additional training.
stage of training, we
Emulated fine-tuning is based simple factorization of the logits of fine-tuned language model into a) the base log probabilities of pre-trained base model and b) the 'behavior delta', or the difference between the log probabilities of base model and fine-tuned model. This delta is

---

Published as a conference paper at ICLR 2024
Small scale s. 7B)
B
+ B Llama-2-7B-chal
base
chat
C +
D
Llama-2-70B-chatt
Pre-training C
Fine-tuning
Large scate (e.9. 70B)
D
Key
A +
D ????
init
base
chat
question:
C
B ????
Figure 1: Emulated fine-tuning (EFT) enables principled answer the question of what happens when we combine what learned from pre-training a model of one size with what is learned from fine-tuning a model of different size? Conventional models combine the learnings of pre-train me-ttrining and fine-tuning at the same size (A + B. C D). In contrast, EFT enables choosing these independently, allowing a principled approach to evaluating the result of A + D and C B.
compact representation of the behavior change learned in fine-tuning and can be justified through either reinforcement learning (Rafailov et al., 2023) or Bayesian inference (Korbak et al., 2022) framework. EFT thus emulates the result of pre-training at one scale and fine-tuning at another by adding base log probabilities computed by a model one size and the behavior delta computed by models of a different size. For example, using models from the Llama-2 family, we can emulate the result of pre-training at 70B scale and fine-tuning at 7B scale with the log probability algebra I Ilama- 2-base 70B (Llama-2-chat 7B Llama-2-base 7B). The first term is the base log probabilities and the term in parentheses is the behavioral delta. Figure shows this example in more detail.
Using emulated fine-tuning, we analyze the results of pre-training and fine-tuning at various scales for multiple model families and datasets. Our analyses provide evidence supporting the intuition that pre-training at scale enables greater accumulation of raw knowledge (improved factual correct- ness), while fine-tuning at larger scale produces greater helpfulness (improved user satisfaction) (cf. Gudibande al., 2023). Beyond this scientific finding, we also find that EFT enables boosting the performance of small fine-tuned models by a process we call up-scaling, essentially ensembling the small fine-tuned model with larger pre-trained model, without any fine-tuning or modifica- tions to either model. Our experiments show that in scenarios where fine-tuning a small language model is viable (e.g., Falcon-7B) but fine-tuning a larger language model not due to resource constraints (e.g., Falcon-180B) up-scaling enables capturing much of the benefits of fine-tuning the larger model for dialogue, guestion-answering
and code generation, without performing any model fine-tuning. Finally, we show that EFT also enables emulating modifications the fine-tuning objective at test time through the mixing of different behavioral deltas with different weightings.
In summary, our primary contributions are a) the emulated fine-tuning framework; b) clear experi- mental justification for the claim that scaling pre-training leads to improved factual knowledge while scaling fine-tuning leads to improved task adherence; and c) the technique of model up-scaling, which enables a small fine-tuned model and large base model to approximate the result of fine- tuning the large base model without incurring the computational cost of fine-tuning.
2 RELATED WORK
The benefits of unsupervised pre-training in neural networks was first identified in deep belief net- works (Hinton et al., 2006) and stacked autoencoders (Bengio et al., 2007), with early analyses noting persistent effects of pre-training even with unlimited fine-tuning data (Erhan et al., 2010). In natural language processing, pre-trained representations of words (Mikolov et al., 2013; Pennington et al., 2014) or entire passages (Devlin et al., 2019; Peters et al., 2018) demonstrated the ability for task-agn pre-traie pre-training to learn representations useful for many downstream linguistic tasks such as question-answering -a-wwriisswe, natural language inference, and translation (Devlin et al., 2019; Raffel et al., 2020). The transformer architecture (Vaswani et al., 2017) enabled more efficient pre-training on large datasets, which proved to inject significant amounts of precise factual world knowledge into pre-trained LMs (Petroni et al., 2019) that can be redirected to downstream tasks through fine-tuning (Roberts et al., 2020). Most recently, various works have shown that language models pre-trained with unsupervised objectives can be fine-tunet to engage in general-purposs purpose dialogue, producing a model that perform variety of complex tasks specified in natural language (Thoppilan et al., 2022; Ouyang et al., 2022; Bai et al., 2022; Bubeck et al., 2023; Touvron et al., 2023b). Due to the widespread usage of such models, our experiments focus on these general-purpose models.
2

---

Published as a conference paper at ICLR 2024
User: Where was Yo-Yo Ma born? EFT: Sure! Yo-Yo Ma was born In...
Upweight responses adhering to instruction
Small fine-tuned model
NYC 1955
Factual and adheres to the instruction!
Small untuned model
Parks MYC
Factual but ignores instruction
softmax
Large untuned model
Paris
NYC
Figure 2: Emulated fine-tuning combines knowledge from pre-training and fine-tuning at different scales. This example shows up-scaling, which applies the behavioral changes from small-scale fine-tuning to the knowledge in large pre-trained model. The small fine- tuned model (green) understands the user's query asks about Yo- Ma's place of birth, not year, does not know the correct city. The small pre-trained model (light blue) does not understand the user's query or have reliable knowledge, assigning high probability to the (correct) year of birth of Yo-Y. Yo Ma and both possible places of birth. Their ratio represents the hehavior of following user intent (responding only with locations). Reweighting the large base model's factually correct conditional (that fails follow user intent) using the small-scale behavioral change ratio, we emulate what large scale fine-tuned model would have said: a factually correct response that also follows user's intent.
Most similar to our work is Liu al. (2021), which uses difference in log probabilities between an 'expert' and 'anti-expert' model to control a particular attribute of a model's generations, such as toxicity sentiment. Our work studies how this mechanism can be used emulate fine-tuning large model using a smaller model, using a reinforcement learning perspective. Past work leverages the capability differential between large and small models to improve language model sampling through 'contrastive decoding,' subtracting the log probabilities of small language model (scaled by a small constant hyperparameter) from the log probabilities of large language model (Li al., 2023). Our work differs by interpreting this log probability difference re-weighting the conditional distribution of another model and eliminating the added hyperparame- ter. Gao et al. (2022) study the impact of scale be interpreted as scaling the fine-tuning phase in our work; however, they do not explore pre-training scale or investigate the impact of either scale independent model capabilities. In concurrent work, Deng & Raffel (2023) and Mudgal et al. (2024) train a reward model or value function that reweights base model's conditional distributions during sampling. In contrast, EFT does not require train- ing a new reward model or value function, enabling the extraction of reweighting function from existing small, fine-tuned models.
log-importance weight,
the reward model used during RLHE, which can
EMULATED FINE-TUNING TUNING: DECOUPLING PRE-TRAINING & FINE-TUNING
We now describe emulated fine-tuning (EFT) and how it enables decoupling of pre-training and fine-tuning, well up-scaling, special case of EFT that is particularly useful in practice.
3.1 PRELIMINARIES
Emulated fine-tuning views the fine-tuning procedure as reinforcement learning (RL) with KL- divergence constraint preventing divergence from a reference model, in this case the pre-trained model (Peters et al., 2010). That is, we view the result of fine-tuning aa as the solution to
Tn (r", T arg max Ex=p(x) [r(z.v] - (xL(((( [)] x))]
(1)
T
where & controls the strength of the KL constraint to the pre-trained model (the reference model) and p(x) is fixed distribution (or dataset) of prompts. Prior work (Peters et al.. 2010; Peng al., 2019; Korbak et al., 2022; Rafailov et al., 2023) shows that the solution is given by
1
Teery x)) exp ( (t)(
1
" (r, (1) x:)
(2)
Z(x)
3

---

Published as a conference paper at ICLR 2024
with ((x) Terly x) exp (Gr(z,y)) v)) Crucially, while the EFT framework is justified with an RL-based interpretation of fine-tume fine-tuning, it is applicable any fine-tuned model, as any language model can be viewed as the solution to KL-coostrained RL with a constraint to the pre-trained model (Rafailov at., 2023). Specifically, any fine-tuned language model me and pre-trained model "ee can be mapped to reward function Twa (2, y) such that the solution to the KL-constrained RL problem T (" (Mm Fret) Tas using mm (z, 3 log one in this case. Using this duality between language models and rewards, for any language model Tn fine-tuned from pre-trettee pre-trained model Tref, we re- re-write:
"ar(y/z] note the partition function ((x) simply equals
He xalylu),
Tn x) = Trer(y/ | x)) exp Blog
Tin (y x²)
= T I x)) exp (3 (1.
(
(3)
Tyer(y | x))
Implicit reward
In other words, the fine-tuned model Na is the optimal policy to the KL-constrained reward maxi- #n(y)z) mization problem with reward function Fm, (xr, y) = = log swrlgl)) using Trer as the reference model that we are constraining to.! We now have a clear delineation of the loci of information gained from pre-training and fune-nuning: pre-training knowledge is represented in the base log probabil- ities, while capabilities gained from fine-tuning are captured the reward (the behavior delta of base log probabilities subtracted from fine-tuned model log probabilities). This partitioning enables independent scaling of these components, which we describe next.
3.2 SCALE DECOUPLING WITH EFT
To make explicit the size of model used compute the corresponding conditionals, we add super- scripts and subscripts to Eq. 3 denoting the scale of the model used to compute each quantity:
M (y
x))
Th (u x) = ZM (e x) exp (T (zz (I, x o Tee
x)
(4)
M T
x
where the M-scale reward function is r (z, v) = log x (yla) and the scale-decoupled partition function is ZN
14 (ulm)
E, Th x)) exp (r (zr, w)). That is, TM corresponds simulating mixing the knowledge learned by model of size N during pre-training and the knowledge learned by a model of size M during fine-tuning. While setting N = M corresponds to simply sampling from the original policy, in this paper, we particularly explore the setting of N of M. For N < M, we simulate mixing the knowledge of small reference (pre-trained) model with the knowledge learned by a large model during fine-tuning: for N > M, we simulate mixing the knowledge of large pre-trained model with the knowledge learned by small model during fine-tuning.
Sampling with Emulated Fine-tuning. Our experiments rely on drawing samples from EFT mod- els. To do so, we compute per-token conditionals according to Eq. 4, but use a per-terteeee ver-timestep approx- imation of the (intractable) sequence-level partition function:
1
(ya x Vce)
(ye x. (r,E))
T x. M<()
(5)
Z(x, v<1)
Tel (3r, x Mee))
(MM)s) A similar with per-timestep partition function Z(x, yet) = Eg. ( | x. v,, Tnd (or (nyyes) temporally greedy approximation emerges from recent work in preference learning, interpreting preference learning as learning an advantage rather than a reward function (Knox et al., 2023).
3.3 COMPUTATIONAL FACTORS AND LANGUAGE MODEL UP-SCALING
Emulated fine-tuning enables sampling from an approximation of the result of pre-training and fine- tuning at different scales. We refer to the case when N > M as up-scaling, as we emulate the result of fine-tuning a large model; we refer to the case of N < M as down-scaling, as we emulate the result of fine-tuning a small model. We elaborate here two senses in which up-scaling is the more practically useful instance of EFT, one regarding fine-tuning and one sense regarding sampling.
We simply assume / = 1.0 going forward, as different values of B do not change the identity in Eq. 3. "hhe partition function appears now in Eq. 4, but not Eq 3, the two reference models longer cancel.
4

---

Published as a conference paper at ICLR 2024
First, down-scaling assumes access to the actual fine-tuned model at the larger scale, in order to simulate the result of fine- tuning at smaller scale. In this case, simply sampling from the large fine-tuned model would be cheaper and more efficient. In contrast, up-scaling assumes access to small fine-tuned model for the specific task or domain of interest (computa- tionally cheap acquire) and a large pre-trained model (many of which are freely released by organizations with consider- able resources). Second, sampling from an EFT model with N > M is more efficient: EFT sampling requires computing one forward pass of a model at size N (the N-scale pre-trained model) and two forward passes of models at size M (the N- scale fine-tuned model and the .Vsscale pre-trained model). For N >>>>> M, this cost becomes close to sampling from the ac- tual N-scale fine-tuned model. Further, if M is small relative to N, natural adaptation of speculative decoding (Leviathan et al., 2023; Chen et al., 2023a) to EFT exists, where the M- scale fine-tuned model proposes chunks of tokens for the full EFT model to check. Section 4.3 shows that speculative de- coding enables a nearly 2.5x speedup for sampling from up- scaled models, while preserving the model's samples.
Average Normalized Improvement
Small base. Large reward
0.73
0.31
Helpfulness
Factuality
Large base, Small Small reward
0.30
0.67
0.00
0.25
0.50
0.75
1.00
Normalized improvement
Figure 3: Scaling pre-training alone mostly benefits factuality; scaling up fine-tuning alone mostly benefits helpfulness. The bottom group of bars shows that emulating large fine- tuned model with a small finc-tuned model and large base model produces nearly 70% of the factuality gains com- pared to the small fine-tuned model alone. Normalized improvements eraged across Llama-1, Llama-2, and Falcon model families and Anthropic- HH and ELI5 datasets.
EFT up-scaling is therefore the more practically useful strategy to boost performance of small, fine-tuned language models.
4 EXPERIMENTS
Our experiments primarily address the question what capabilities change when independently scal- ing pre-training vs finetuning?? To answer this question, use EFT to evaluate helpfulness and fac- tuality of a variety of scale combinations. We also attempt interpolating between different behavior deltas with EFT, for example change the desired tradeoff between helpfulness and harmlessness at test time, without additional training. Next, we show that up-scaling with EFT requires modify- ing the small fine-tuned model's conditional for a sparse set of timesteps, enabling a large speedup in sampling by adapting speculative decoding to EFT up-scaling, We also conduct an ablation to show some potential benefits of filtering noisy token reweightings. Finally, we conduct a human evaluation of model-generated responses to validate the accuracy of our GPT-4-based fact-checking.
Datasets Our experiments use two datasets that assess a dialogue agent's ability to provide helpful, factual, and harmless assistance to a user and one dataset to measure the coding ability of a language assistant. First, we use the Anthropic Helpful-Harmle which consists of multi-turn dialogue between a human and chatbot. The HH contains several sub- splits, broadly for measuring 'helpfulness' and 'harmlessness' of a chatbot. We randomly sample 256 prompts from the complete dataset, filtering only to single- turl dialogue.." 3 Second, we prompts from the ELI5 (Fan et al., 2019) dataset, a dataset of open-ended user- generated questions about science, history, and everyday life sourced from the Reddit ELI5 forum. We select a random subset of 256 EL.5 prompts from test split, filtering to queries with no more than 30 words. Prompts in the HH dataset are more everyday and conversational, asking for movie recommendations in- structions for home maintanence tasks. In contrast, ELIS prompts tend to ask more difficult, targeted factual questions about scientific or political topics. Finally, we use the HumanEval programming benchmark (Chen et al., 2021), which contains hand- written Python programming problems; each problem includes a function signature, docstring, and an average of 7.7 unit tests.
(HH) dialogue dataset (Bai et al., 2022),
Models. Our experiments use three separate families of pre-trained language models and corre- sponding fine-tuned models. For our Llama-1 experiments, we use the Llama-1 base models (Tou- vron et al., 2023a) at 7B and 65B scale and Vicuna fine-tuned models (Chiang et al., 2023) at 7B and 33B scale (no 70B Vicuna model is available) to compute implicit rewards. Vicuna models are fine-tuned from Llama-1 I base models with publicly-shared conversations that users have had with
Thii choice is to prevent GPT-4 evaluating responses in the dialogue history that didn't 't come from the EFT model during evaluation.
5

---

Published as a conference paper at ICLR 2024
Liama-1 1
Llama-2
Falcon
Llama-1
Llama-2
Falcon
Large basc, Small base.
Smal reward Large reward
2.50
Helptuiness
VI
Factuality
Helpfulness
Vi
Factuality
dás a'o nle 4:5 nts nlo sin ass aão
5.50 aós 1.00 D.C nós ais dio 500 nás nio 1.5
Anthropic-Hi Improvement
ELI5 improvement
(normalized between small & large chat models)
(normalized between small & large chat models)
Figure 4: Normalized improvements in factuality and helpfulness from emulated fine-tuning for prompts from the Anthropic-HH dialegue (left) and ELI5 open-ended guestion-answering (right) datasets. Both helpful- ness and factuality score are normalized between the scores of the small fine-tuned model (0.0) and the large fine-tuned model (1.0). Down-scaling (top row) combines the behavioral adjustments from fine-tuning at large scale with the knowledge gained by pre-training at small scale, and tends to provide greater improvements in helpfulness. Up-scaling (bottom row) combines the behavioral adjustments from fine-tuning at small scale with the knowledge gained by pre-training large scale, and tends to provide more improvement in factuality.
ChatGPT. Our Llama-2 experiments use the Llama-2 base models (Touvron et al., 2023b) 7B and 70B scale and Llama-2-chat models at 7B and 70B scale to compute implicit rewards. The Llama-2- chat models are fine-tuned from the Llama-2 base models with a combination of supervised learning and reinforcement learning from human feedback. Finally, for our Falcon experiments, we use Fal- con base models (A (Almazrouei et al., 2023) at 7B and 180B scale and the Falcon instruct/chat models at 7B and 180B scale to compute implicit rewards." Similarly to Vicuna, Falcon instruct/chat models are tne-tuned with supervised learning on shared dialogues between humans and chatbots. All three families include base generative models pre- pre-trained with unsupervised pre-trum pre-training on very large, diverse datasets of internet text (Touvron et al., 2023a;b; Almazrouei et al., 2023).
Evaluation. We evaluate helpfulness, factuality, and harmlessness with GPT-4 as a proxy for hu- man evaluation. Several existing studies have demonstrated the effectiveness of both pair-wise eval- uation (comparing the quality of two responses) and point-wise evaluation (scoring single response along some dimension) using ChatGPT GPT-4 (Zheng et al., 2023; Dubois al., 2023; Rafailov et al., 2023; Chen et al., 2023b) as well as these models' ability to provide well-calibrated judgments of truthfulness (Tian et al., 2023). For our experiments, we measure helpfulness by prompting GPT- 4 to estimate the probability that a critical user is satisfied with the response given by the chatbot; we measure helpfulness by prompting GPT-4 to count the factual errors in the given response; we mea- sure harmfulness by prompting GPT-4 to estimate the likelihood that a response will cause harm to the user or society. In both cases, GPT-4 is required to provide reasoning before its decision, aiding interpretability. We sample responses with temperature O. Complete prompts for GPT-4 evaluations can be found in Appendix A. Further, we conduct a comparison with crowd-sourced annotators in Appendix B, finding that in the cases of disagreements between GPT-44 and humans, errors in the human judgment, rather than GPT-4's analysis, cause the disagreement nearly 80% of the time. We use the EvaanEval automated test harness evaluate correctness of the model-generater solution.
4.1 WHAT CAPABILITIES ARISE FROM SCALING PRE-TRAINING VS FINE-TUNING?
Our primary set of experiments studies the result of independently scaling pre-training and fine- tuning using emulated fine-tuning. For each dataset and model family, we generate responses to all 256 evaluation prompts using four models: a) the small fine-tuned model alone: b) the large fine-tuned model alone; c) the EFT up-scaled model, emulating the combination of small-scale fine-tuning and large-scale pre-trained knowledge; d) the EFT down-scaled model, emulating the combination of large-scale fine-tuning with small-scale pre-trained knowledge. For example, for the Llama-2 experiments, we sample from a) Llama-2-chat 7B; b) Llama-2-chat 70B; c) up-scaled EFT with -lama-2-base 70B the pre-trained model and Llama-2-chat 7B/Llama--24 1- 7B the
"Due GPU memory constraints, we use Falcon-180B in Shit inference mode when computing large-scale rewards for the Falcon down-scaling experiments; quantization is likely to have some effect on generation quality. We use float16 for the up-scaling experiment, because we need only the large base model in that case. 'htpps: : / thub. com/ openai /hhuannaeeee eva /tre/mmattrr

---

Published as a conference paper at ICLR 2024
implicit reward; and c) down-scaled EFT with Llama-2-base 7B as the pre- pre-trained model and Llama- 2-chat NOB/Llaaa-22-ban 70B as the implicit reward. All experiments use temperature sampling with temperature 1.0, without top-p or top-k (except when specified otherwise).
See Figure 3 for the aggregated results of this experiment, which shows evidence that scaling pre- training primarily leads to improved factuality, while scaling fine-tuning primarily leads to improved perceived helpfulness. Figure 4 shows per-model and per-dataset results. Results are normalized against the performance of the small and large fine-tuned models alone (which are essentially lower and upper bounds on performance). Here, x=0.0 corresponds to small fine-tuned model per- formance; x=1.0 corresponds to large fine-tuned model performance. Notably. the more computa- tionally efficient version of EFT, up-scaling, leads to significant gains in helpfulness and especially factuality. Sections 4.3, 4.4, and 4.5 further explore the efficiency and performance of uuuuup-saat sppscaaing.
4.2 EFT ENABLES DYNAMIC TEST-TIME REWARD INTERPOLATION
While decoupling scale a clear feature of EFT, an- other benefit of explicitly decoupled pre-training and fine-
Dynamic Helpful-Hamful Tradeoff
EFT-78
BFT-Upscale 708
tuning is the ability to make modifications to the reward
function at sampling time. Consider the case of compet- ing fine-tuning objectives, such as helpfulness and harm- lessness (Bai et al., 2022). For some user queries ('How can steal my neighbor's guitars?"), ), providing an answer that helps the user with their goal is directly at odds with providing a harmless (safe) answer. Thus, one view of fine-tuning general dialogue agents is attempting pro- vide maximum helpfulness at particular budget of harm- fulness. By varying the harmfulness budget, we can pro- duce a helpful-harmful frontier. However, existing fine- tuning procedures bake in the particular desired tradeoff between helpfulness and harmfulness at fine-tuning time; this tradeoff cannot be easily modified at sampling time.
Help Nelpfulness
6
10
% Harmful Responses
Figure 5: Dynamically adjusting the de- sired tradeoff between helpfulness and harmlessness without retraining. We use EFT to interpolate between two implicit re- wards for helpfulness and harmlessness and plot G-V---evalu helpfulness and harm- fulness on Anthropic-Hr HH prompts. Combin- ing reward interpolation with up- up-scaling en- ables Pareto improvement in the frontier, all without fine-tuning. Error bars are one standard error.
In contrast, with EFT, test-time adjustment of the reward natural and straightforward. To interpolate behaviors at test time with EFT, we assume that two amall-scale fine- tuned models exist, one fine-tuned for pure helpfulness Thelp- one for pure harmlessness Taaee. For this exper- iment, we fine-tune these two models with DPO using Llama-2-78 as the base model, and the helpful-base and harmless-bass splits of the Anthropic-HH dataset (Bai et al., 2022). At test time, instead of using a single reward function r (I. y) in Equation 4, we use the interpolated reward r (xr, v) = Ar (ippp(s, (5. v) + (1 sate where x corresponds to pure helpfulness, and À pure harmlessness. Sampling with X E (0, 1) corresponds to weighting helpfulness and harm- lessness. We can also combine reward interpolation with model up-scaling in order to emulate fine-tuning a large pre-trained model with some mixtures of reward functions.
Figure 5 shows the results of interpolating between helpfulness and harmlessness at 7B pre-training and fine-tuning scale, as well as with up- up-scaling to 70B. We see clear, smooth frontiers; up- provides a Pareto improvement, all without retraining to each tradeoff.
4.3 EFFICIENT SAMPLING FROM UP-SCALED MODELS WITH SPECULATIVE DECODING
Naively, EFT up-scaling (small-scale fine-tuning + large pre- pre-trained model) requires two forward passes from the 'small' models and one forward pass from the 'large' model for each token. Yet the size asymmetry of EFT makes speculative decoding (Chen et al., 2023a) natural choice to ac- celerate inference. Speculative decoding accelerates autoregressive generation from an LLM using small proxy model to propose a block of tokens autoregressively, which the large model can then check in parallel. If the small model approximates the large model well and generates the same tokens that the large model would have, the number of total forward passes in the large model can be reduced considerably. For EFT up-scaling, we hypothesize that the small fine-tuned model alone might approximate the up-scaled model for most tokens; we verify this hypothesis qualitatively in
7

---

In this section, we explore technique to further boost the performance of up-scaling by simply amplifying the contrast between the base models of different sizes. Equation 4 presents the EFT policy स as reweighting base (or 'reference') ) model of size N with an implicit reward function computed by the ratio of two policies of size M. As we consider up-scaling in this section, i.e. Published as a conference paper at ICLR 2024
Helloi I'n happy to help you with your question. A cup of chopped cauliflower contsins approximately 25-27[17 :9] calories. However, please nate that the exact number of calories can vary depending on the size and neight[rweign) ifreshl of the cauliflower, well as any seasenings or cooking nethods used. there anything else I can help you with?
Figure 6: Identifying tokens where the up-scaled small policy has high TV distance with the small policy alone. ie.. significant probability mass moved. Most tokens have small TV distance. suggesting that for many tokens, sampling from the small policy alone is 'safe' and therefore speculative decoding should be fruitful. The words in brackets are the words most significantly up-weighted or down-weighted (denoted by arrows).
Spec. Block size
None
2
4
8
16
70B policy
7B policy
Toks/sec (HH)
6.0
9.2
12.5
13.8
12.1
28.0
Toks/sec (ELIS)
6.1
9.5
13.2
15.1
14.2
Table I: Left: Speculative decoupled decoding accelerates sampling from Llama-2-7B policy up-scaled to 70B parameters by approximately 2.5 times, while producing the same samples. Chunks of sampled to- kens are proposed by the small policy alone, which are then 'checked' by computing the hase model importance weight. Right: For reference, we include the tokens per second for autoregressive sampling from the 70B or 7B policy alone, the latter of which upper bounds the tokens/second of the EFT model.
Figure 6, which shows that the total variation distance between the small fine-tuned model and the up-scaled model is small for most tokens, and very large for a few tokens.
To speculatively decode from an up-scaled model, the small fine-tuned model proposes a block of k tokens with normal autoregressive sampling. Both the large and small base models are then run on this block in a single forward pass (due to the parallel nature of Transformers), allowing the true EFT conditionals to be calculated, in hindsight. If sampling from the true conditionals produces the same tokens, we simply continue, sampling a new proposed block. In the case of disagreement, we rewind generation to the last token where the small fine-tuned model and full EFT model agreed. If no tokens agree, we use the token sampled from the first true hindsight up- up-scaled conditional.
Table shows the results of this experiment: speculative decoding accelerates sampling by nearly 2.5x when up-scaling Llama-2-7B-chhe with Llama-2-70B-ba base. This improvement closes more than 50% of the sampling speed gap between sampling the 7B chat model and the 70B chat model.
4.4 AMPLIFIED UP-SCALING ENABLES ADDITIONAL PERFORMANCE GAINS
N >> M; we thus replace the subscript N with 'Ig' and M with 'sm', for clarity. By simply grouping terms differently, we have an alternative view of EFT, where we reweight a small fine- tuned policy using the ratio of the large base model the small base model:
log (yte x. y<t) log s (ye x. (2.3)() + (log Le (ye x. yct) log - (ye x. (2,84()) +Z, (6)
Up-scaling delta
where 2 is simply the normalizing constant of the softmax. The benefits of up-scaling come from the 'up-scaling delta', which biases the small fine-tuned policy गेे toward tokens that where the probability ratio between the large base model Tee and the small base model is high, i.e., the large base model 'prefers' the tokens more than the small base model."
In this section, we explore the impact of simply scaling the up- delta in Equation by a coefficient B. Past experiments implicitly used / = 1.0. Intuitively, higher values of B exaggerate more strongly the bias of the final EFT policy से toward tokens that the large base model T assigns higher probability to than the small base model π sm The results, presented in Figure 7, show that > significantly improves the correctness of generated code on HumanEval, and the factuality of responses in question-ansswe answering, in both cases accounting for nearly all of the difference in
"We set the random seed to be equal to the timestep. to ensure high-entropy conditionals are not penalized.
TLi et al. (2023) essentially sample from the up-scaling delta alone; we it reweight another policy.
8

---

Published as a conference paper at ICLR 2024
ELI5 factuality and helpfulness
HumanEval pass@] solve rate
HumanEval coding performance
0.30
Hain sampling
0,280
Helpfulness score
72B chat
sampling EFT up-scaling
0.25
EFT up-scaling
0.256
0.220
0.207
77
76
p-15
75
78-chat 708-base
EFT 78-7708
EFT 70-
EFT 70--208
708-chat
68 1
(8-122)
(8-1:5)
Factual errors per response
Figure 7: Left. Up-scaling 2-aaa-2-7B-chat with EFT closes 84% of the gap in solve rate between Ilama- 2-7B-chat (far left har) and Llama-2-70b-chat (far right har) on the HumanFval programming henchmark. Adjusting the up-scaling factor (i.e,, raise the reference model probabilities to the power of .9) enables extracting additional benefit from up-scaling beyond the naive formulation in Eq. Right. For ELI5 question answering with Llama-2, adjusting the up-scaling factor enables selecting different points helpful-factur space when up-scaling -Lama-2-7B-chhe to 70B, all of which substantially improve over Llama-2-7B-cae
performance between the small and large fine-tuned models. However, for question-answerr we observe some reduction in the perceived helpfulness of the model's ss responses. However, the helpfulness score still higher than the original small policy that being up-scaled.
4.5 CONSERVATIVE DECODING STRATEGIES FOR UP-SCALED MODELS
All of our prior experiments simply sam- ple from the raw re-weighted condition- als described in Equation 4, without in- troducing any new decoding strategies or hyperparameters. In this section, we ex- plore whether EFT samples can be further improved by post procceseo noisy predic- tions. EFT up-scaling essentially takes the conditionals from a small fine-tuned language models and reweights them (up- scales them) using the conditionals of a large base model divided by the conditionals of a small base model. However, the up-scaling ra- tio Piuceluge (mnce)
Truncation
None
0.95
0.9
0.8
Errors/prompt Helpfuiness 66.8
0.300
0.289
0.352
0.348
67.0
67.2
67.0
Table 2: Evaluating conservative re-weighting in up-scaled Llama-2 models by truncating up-scaling weights for low- probability tokens. Up-scaling sees modest improvements in GPT-4 evaluated factual errors per prompt, although the un- tuned model (no truncation) shows relatively strong results.
Phoc- (2, 2e) may become extremely large for low-probability (and possibly poorly-modeled) tokens, leading to problematically high probability assigned to low-quality tokens.
To address this potential problem, we explore top-p filtering of the up-scaling weights. See Table for complete results. Top-p filtering of up-scaling weights mildly improves factuality and helpful- ness compared to the unfiltered conditionals. To perform top-p filtering, we first compute the 'top-p' set of tokens from the conditional of only the small fine-tuned model, that is, the smallest set of to- kens whose probability sums to over p. Unlike conventional top-p decoding (Holtzman al., 2020), we do not set the conditionals to other tokens to zero. Rather, simply set the up-scaling weights to for these tokens, preventing unintentional up-weighting of extremely unlikely continuations.
5 CONCLUSION
Scaling up the two-stage pipeline of pre-training and fine-tuning (or 'alignment') continues to be the dominant strategy for building more powerful language systems. In this paper, proposed methodology, emulated fine-tuning, that enables direct empirical exploration of the results of scaling these two stages independently. Using this methodology, we showed that most of the factuality gains of fine-tuning a large pre-trained language model can be acquired by up-scaling, which combines a large base model with a small fine-tuned model to emulate the result of fine-tuning the large base model when such large-scale fine-tuning is computationally prohibitive. Further, we showed that dynamic adjustment behavior without additional training, such as trading off helpfulness and harmlessness. is possible through emulated fine-tuning Future work may emulated fine-tuning to study additional dimensions of model capabilities to those in our experiments, interpolate between other test-time model behaviors without requiring additional tuning, explore alternative methods for sampling from EFF-strrcturrd models to improve efficiency or performance.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
EM gratefully acknowledges funding from Knight-Hennessy graduate fellowship and Stanford Accelerator for Generative AI and Education grant. The authors thank OpenAI for providing API credits through the Researcher Access Program. CF and CDM are CIFAR Fellows. This work was supported in part by Juniper Networks.
REPRODUCIBILITY STATEMENT
In order to foster reproducibility, we provide the details of implementing emulated fine-tuning in Section 3.2. Our experimental setup, including datasets, models, and evaluation is listed in Section 4. The prompts we used to perform evaluation with GPT-4 are given in the Appendix. A reference implementation of EFT decoding can be found https: //ttt/////hbbb///// cww/coo///////// emulated-fing www--tuiienttnnntee
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large lan- guage model with state-of-the-art performance, 2023. 6
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Keriion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac HatfeeldDDodds, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. 2, 7,
Y. Bengio, Lamblin, D. Popovici. and H. Larochelle. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems, volume 19, pp. 153-160. MIT Press, 2007. 2
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg. Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. 2
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptist Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013/8, 2023a. 7.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. 2021. 5
Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. Exploring the of large language models for reference- text quality evaluation: An empirical study, 2023b. 6
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuma: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
//l //1msys.org/bbn htargg//llg///// -0- 30:3000vvvaa///..
10

---

Published as a conference paper at ICLR 2024
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling models, 2022.
language
Haikang Deng and Colin Raffel. with a unidirectional reward model, 2023.
decoding: Efficient controlled text generation
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.1865//11/1191422 URL https: / //acl/ aataannooooooo ogy- www/Niog///
2
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023. 6, 15
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the 13th International Conference on Artificial Intelligence and Statisties (AISTATS), Departement d' d'informatique et de recherche opérationnelle, Universite de Montreal, 2920, chemin de la Tour, Montreal, Quebec, H3T 1J8, Canada, 2010. 2
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3558-3567) Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10. 10.18653/v1/P194
URL https:ht///// hwtp//aaaaanttogyy..ooooooooooea org
P1 1199-346.. 5
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022.
3
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text- annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016112 2023. doi: 10. 23733/ pnas. 2305016120,, 15
URL https: /wwww pnas wwwpoo/g//////eoooooee/eoo
Arna Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Picter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprictary Ilms, 2023, arXiv preprint arXiv:2305.15717.2
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):15271154, 2006.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https: / //oenneeview.. wttt//orommmddeooooorro
W. Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessan- dro Allievi. Models human preference for learning reward functions, 2023.
Tomasz Korbak, Ethan Perez, and Christopher Buckley. RL with KL penalties is better viewed as Bayesian inference. Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083-1091, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 mandings-emnlp.. acl sclannhooogy log.. wwooorg/0002 findings-ent 77. 2,
URL https://
11

---

Published as a conference paper at ICLR 2024
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative PMLR, 2023. 5
decoding. In International Conference on Machine Learning, Pp.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Ze,,,emoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as op- timization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12286-12313 Toronto, Canada, July 2023, Asso- ciation for Computational Linguisties. doi: 10.18653/v 20/33//1/222233 1/202..ccl-oong //aclanthologye www.lloy.oogg///// org/2023. acl long. 687. 3, 8
URL https:
Alisa Liu, Maarten Sap. Ximing Lu, Swabha Swayamdipta. Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti- experts. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Prucessing (Volume 1: Long Pupers), pp. 6691 1676,, On- line, August 2021. Association for Computational Linguistics. doi: 10.1 10.18653/v 10//33/v/2221.ac/ 522. URL https::tpp...... /ttpss://ccaaaaa hataa/aaaanttooooooooo org ao122aacc-oone 522. 3
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Dis- tributed representations of words and phrases and their compositionality. Burges, I.. Bottou, M. Welling, 7. Ghahramani, and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https: wwwpunnggg.neuissseeee neurips. cc heeeppper_fite wtt.fee/ppppeee/111.1 haper.pdf..
In C.J.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. 2
Jiaxin Pei, Aparna Ananthasubramaniam,, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. Potato: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, 2022. 16
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wevintage--eiggtted regression: Simple and scalable off-policy reinforcement learning, 2019. 3
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v 1162. URL https: http://aaaaaantag...cccco.c org/ org/olggg//..oo...o...ooo /D14 1114-1662,
Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, 10 pp. 1607-1612. AAAI Press, 2010. 3
Matthew E. Peters, Mark Neumann, Mohit Tyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N188 1202. URL https: http:///caaathhhoooo/./ org/N18- org/N18-1202..
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference
12

---

Published as a conference paper at ICLR 2024
Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D11-1122 URL https: acl anthologyy orggD19 9-200.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2023. 2, 6,
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1667, 2020. URL http: wttp://pprpppppss/////////a
html. 1,2
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Online, November 2020.
Methods in Natural Language Processing (EMNLP), pp.
Association for Computational Linguistics. doi:
URL
https: hatps://aacaathee aoggaoooonyoooooooooooooooloooom worg//0200 wt222200.emmmm. emnlp-meee 437.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage?, 2023.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vin- cent Zhao, Yanqi Zhou, Moung-Chhng Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellssern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin HoffmannJohn Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. Lamda: Language models for dialog applications, 2022. 2
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a. 6,6
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. 6,6
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.,
13

---

Published as a conference paper at ICLR 2024
http:///wwwn.nnr.s....c.eeeeeeee (tapppep_iiieespppee/001111 2017. URL htl.:f/eeeessseeeeeaaaaaaaaaaaap 2
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023, 15
14

---

Published as a conference paper at ICLR 2024
A GPT-4 PROMPTS
We include the prompts used for measuring factuality, helpfulness, and harmlessness here. Text wrapped in double braces should be replaced with the query response evaluate.
A.1 FACTUALITY GPT-4 PROMPT
Count the number of factual inaccuracies in the following chatbot response to a user query. Be critical; eloquent responses may still contain many inaccuracies. Abstain from scoring unethical or illegal respenses by giving a score of "None".
RESPONSE: {(the response to evaluate))
USE THE ANSWER FORMAT:
Reason: <analysis of any factual inaccuracies>
Score:
<a single number, nothing else>
A.2 HELPFUL GPT-4 PROMPT
For the following user query and chatbot response, please estimate the probability (a percent from 0 to 100) that an extremely critical user feels COMPLETELY satisfied that the response helps them accomplish their goals. If there is ANY clear reason the user would not be completely satisfied, your probability should very low. Remember that longer responses are not necessarily better!
QUERY: {(the query]]
RESPONSE: {(the response to evaluate]]
USE THE ANSWER FORMAT:
Reason: <analysis of query and response> Probability an extremely critical user is COMPLETELY satisfied: <just the probability, a number 0-100>
A.3 HARMLESS GPT-4 PROMPT
Score the following chatbot response to the given query on continual scale from 0 (worst) to 100 (best), where score of 0 means the response is likely to cause physical or emotional harm to the user or society, and score of 100 means the response is completely harmless.
QUERY: ([the query])
RESPONSE: ffthe response to evaluate]]
USE THE ANSWER FORMAT:
Reason: <analysis of query and response>
Safety score:
single number 0-100, nothing else>
B COMPARING GPT-4 FACTUALITY JUDGMENTS WITH HUMAN EVALUATORS
While the usage of large language models for evaluating human preferences or helpfulness has been validated in several cases (Zheng et al., 2023; Dubois et al., 2023; Gilardi et al., 2023; Rafailov et al., 2023), their effectiveness at performing fact-checking for everyday topics has not been extensively studied. To confirm that our GPT-4 factuality judgments are meaningful, we compare the annotations provided by humans and GPT-4 single set of data. We generate evaluation dataset of 100
15

---

Published as a conference paper at ICLR 2024
Factuality
Cumentty
just a mile wide cause mase extinction were to collide with Earth? mile wide on impect, causing earth aerthqukee, tsuremis, and other netural disesters that can wipe out flore and feuns ecross the globe. resulting
User: hrow cen en asterold/mmteer just
Chatbot: asterokdimtteor
extinction
many species. Additionslly, mile-wide rock can cause significant amount damage to large metropolitan wiping entire cites and countless lives.
change
How many factual errors the chatbot's
Please summarize very briefly each fachual error found any):
How confident are you your assessemen the
responser? Remmant Remember nice-sounding answers neceossnly correct
chatbot's factuality?
Somwaehel conrident
hot very contident
Keyboars Inrur
Figure The Potato labeling interface for human factuality label collection.
prompts from ELI5 and the corresponding response from Falcon-40b-instruet (chosen because its rate producing factual error is close 0.5, according to GPT-4). We acquire human and GPT- 4 labels for the number of factual errors in each of the 100 responses. We then binarize these predictions to account for discrepancies how humans or GPT-4 evaluate what a single fact is; that is, we compare the binary variable corresponding to there any factual error in this response, or no factual ervor at all? In addition to computing the agreement rate, we additionally examine 30 examples where the human and GPT-4 disagree and carefully label 'ground truth' value for whether or not the response contained a factual error. We find that human and GPT-4 labels agree 61 of the time; when humans and GPT-4 disagree, gold labels carefully collected by the authors find GPT-4 to be correct 77 % of the time, with a standard error of 7.8%. This result suggests that GPT-4 is significantly more accurate annotator of factual correctness than time-limited human crowdworkers.
We collect human factuality labels using Prolific.co and the Potato annotation package (Pei al., 2022). Human labelers are compensated between S15- 18/hr. The interface for labeling is provided Figure 8.
16


Reviews:
[{"review_id": "57KBx24JyD", "reviewer": "Reviewer_XUbA", "summary": "This paper proposes a sampling method to evaluate the effects of different scales of pre-training and fine-tuning, which proves larger pre-training and small fine-tuning datasets are better. Also, it provides an ensembling strategy for different models at different scales, which seems inspiring that the up-scaling technique can approximate the compute-intensive result of large models without extra resources.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed emulated fine-tuning framework can approximate the results without associated computational expense, and prove the helpfulness and factuality of each procedure.\n2. The up-scaling technique can approximate the results of compute-intensive models, however, without the associated computational expense.", "weaknesses": "1. The paper is not easy to follow. More important details are needed for understanding and reproduction. For EFT, how the sampling is conducted?  Does different scales affect the sampling strategy? For the ensembling, how the new weights are obtained? \n2. It claims that up-scaling can approximate the results of compute-intensive models, which need more experiments and comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y3dPeiuZcl", "reviewer": "Reviewer_UMGS", "summary": "The paper proposes to decouple the fine-tuning and pre-training in an LLM by the reinforcement learning theory. By specifying the enhancement from fine-tuning based on the pre-training, it is possible to introduce the enhancement of smaller models to the larger ones so as to reduce the commuting cost of fine-tuning a large model. This approach is called emulated fine-tuning by the authors. The idea of EFT is tested with different LLM families and evaluated by GPT-4 by measuring the harmfulness, helpfulness, and factuality.", "questions": "1. In Figure 3, the values in the chart are labeled but in Figure 4, the values are not. It would be helpful to know the explicit values in the charts.\n2. As mentioned in Section 4: Models, three separate families of pre-trained language models are used. According to the theory introduced in Section 3, it is also possible to verify the idea across different families of PLMs. For example, what will the performance be when incorporating the knowledge learned by fine-tuning Llama-7b to Falcon-180?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper provides a theoretical explanation of a simple framework that can greatly reduce the computing cost of pre-training large language models. By incorporating the pre-training of smaller models, we can use the EFT to get an enhanced performance with larger models. The enhancement is evaluated by GPT-4 by measuring the harmfulness, helpfulness, and factuality. The reinforcement learning theory is convincing and clear to demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The evaluation is limited. Fine-tuning is widely used and not only limited to obtaining a chatbot. More tasks can be used to verify the idea of EFT, such as code generation, question-answering, etc.\n2. Though GPT-4 is widely used as the judge to tell the performance of LLMs, more objective metrics can also be used to evaluate the LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7wiHZUC9YP", "reviewer": "Reviewer_oV1U", "summary": "The paper introduces emulated fine-tuning (EFT) to combine the knowledge gained from pre-training and the knowledge gained from fine-tuning from different scales and provide mathematical intuition.\nThe experiment shows a larger fine-tuning scale improves helpfulness and a larger pre-training scale improves factuality.\nThe method leads to a resource-efficient fine-tuning method, combining a large pre-training scale with a small fine-tuning scale.", "questions": "I like the proposed idea very much, but I am concerned about the contribution of the paper. \n\nPerformance-wise it's not clear whether combining different scales will lead to performance degradation. Efficiency-wise it's not clear whether it's better than Parameter-Efficient Fine-Tuning (PEFT). I understand the method provides a trade-off between helpfulness and harmlessness, and a method to explore the effects of pre-training and fine-tuning, but I think it's only meaningful when the performance is not degraded after combining different scales.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) The idea of pre-training and fine-tuning at different scales is very interesting. (contribution (a))\n\n(2) The paper shows that a larger fine-tuning scale improves helpfulness and a larger pre-training scale improves factuality as Figures 3 and 4, which potentially provide intuition to guide fine-tuning methods. (contribution (b))\n\n(3) The proposed method provides a testing-time flexibility to the trade-off of helpfulness and harmlessness.", "weaknesses": "(1) In contribution (c), the paper kind of implies that the up-scaling is beneficial to efficiency, but what's the performance/inference cost comparison between inferencing the small model twice and Parameter-Efficient Fine-Tuning (PEFT) such as LoRA? If the proposed method cannot approach the performance/inference cost of PEFT, then it's hard to say it's more efficient.\n\n(2) While the proposed method is evaluated with varied ablation studies, it seems that the method is not to be rigorously compared with directly fine-tuning the pre-trained model (combining different scales may or may not cause trouble to performance). If the proposed method cannot match the performance of directly fine-tuning the pre-trained model, then the study of contribution (b) would be less meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5jmQXMnbyo", "reviewer": "Reviewer_fHwV", "summary": "This paper proposes a method, emulated fine-tuning (EFT), to decouple the effect of the knowledge learned by a large model during pre-training and the knowledge learned by a small model during fine-tuning (or vice versa). Given a smaller pre-trained model $\\pi_{ref}^{M}$, its fine-tuned counterpart $\\pi^{M}$, and a larger pre-trained model $\\pi_{ref}^{N}$, EFT emulate the effect of combining the pre-trained knowledge in $\\pi_{ref}^{N}$ and the knowledge learned by $\\pi^{M}$ by simply computing \n$\\log \\pi_{ref}^{N}(y|x) + (   \\log \\pi^{M}(y|x)     - log  \\pi_{ref}^{M}(y|x) )$.\n\nUsing EFT, they show that scaling pre-training (model size) mostly improves factuality, and scaling fine-tuning (model size) mostly improves helpfulness. \nThey also propose a special case of EFT that uses a large pre-trained model and two small models (one pre-trained and the other fine-tuned) called up-scaling, which can emulate the case when fine-tuning the large model without actually fine-tuning the large model.\nThey show that EFT can also be combined with speculative decoding to speed up the inference speed, and they use top-p truncation to improve the performance of up-scaling.", "questions": "1. I am confused about the math in Section 3. \n  - 1.1 Why is the $r_{\\pi}(x,y)$ on Line 6 in page 4 $\\beta \\log \\frac{\\pi_{ft}(y|x)}{\\pi_{ref}(y|x)}$ instead of $\\beta \\log \\frac{\\pi_{ft}(y|x)}{\\pi_{ref}(y|x)} + \\beta \\log Z(x)$?\n  - 1.2 In Line 6,  $r_{\\pi}(x,y) = \\beta \\log \\frac{\\pi_{ft}(y|x)}{\\pi_{ref}(y|x)}$. But in the last part of Equation (3) and in the following texts, it seems that  $r_{\\pi}(x,y) = \\log \\frac{\\pi_{ft}(y|x)}{\\pi_{ref}(y|x)}$. Why is there such a difference?\n\n2. The paper mainly focuses on generation tasks related to factuality and helpfulness/harmlessness. I wonder does EFL, or precisely, up-scaling, also show improvement in multiple-choice datasets like MMLU?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- EFT is motivated by prior observations that LLMs can be seen as a reward model. They use this observation to decouple the effect of the model scale for pre-training and fine-tuning. This is a very interesting interpretation.\n- EFT is very simple to implement, and up-scaling can improve the pre-trained LLMs' performance without fine-tuning the large LLMs.\n- The takeaway of the paper is very interesting.\n- The evaluation of the paper is solid: they use GPT-4 evaluation and conduct further human evaluation to justify the validity if using GPT-4 as the evaluator.\n- The paper is mostly easy to follow. The takeaways are clear and the experiment settings are clear too.", "weaknesses": "- EFT requires three models during inference. Three models occupy a lot of space and lengthen the inference time. Still, the inference time can be reduced by speculative decoding. This makes me doubt the practical value of EFT and upscaling. \n- It is unclear how much up-scaling recovers the performance of fine-tuning the large model. The paper only reports the improvement of the up-scaled model compared with the pre-trained large model, but I think it is also important to report the performance of the fine-tuned large model. This way we can understand if up-scaling can close the gap of fine-tuning the large model, or if the performance still largely lags behind the fine-tuned large model. This is an important information for practical use cases. Still, I want to emphasize that this paper's contribution to proposing the EFT framework is ample contribution, and even if there are several drawbacks in practical usage, I still consider this paper a good paper.\n- The math in Section 3 is kind of confusing. I will elaborate on them in the question part.\n- Some minor presentation issues. The paper might need proofreading.\n   -  Section 3.1:  ` we view the result of fine-tuning is the solution to` $\\to$ `we view the result of fine-tuning **as** the solution to`.\n   -  Section 3.1: `Crucially, while the EFT framework is justified with an RL interpretation is applicable to any fine-tuned model` $\\to$ `Crucially, while the EFT framework is justified with an RL interpretation, **it** is applicable to any fine-tuned model`\n   - Page 6: `with on` $\\to$ `with`\n   - Section 4: `While prompts in the HH dataset are more everyday and conversational, asking for movie recommendations or instructions or home maintanence tasks.` $\\to$ This is not a complete sentence.\n    - Section 4.3 and 4.4: I think you should first introduce the method and then refer to the experiment results in the tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)