ords. [Models (MCP)]
communications set down in writing or print. [Arts]
n.,
 The place of the compilation of a record, included in the record by its author. [Archives]
n., Element in the organization of a file or consisting of documents which relate to a specific administrative action; case
papers/files may be restricted to a single transaction (Canadain usage: transactional files). [Archives]
n., An act or several interconnected acts in which more than one person is involved and by which the relations


 of those persons are altered. [Archives]
n., Element in the organization of a file or consisting of documents which relate to a specific administrative action; case pap
ers/files may be restricted to a single transaction (Canadain usage: transactional files). [Archives]
n., An act capable of changing the relationship between two or more physical or juridical persons. [Archives]
n., A unit of interaction with a DBMS or similar system. It must be treated in a coherent and reliable way independent of other


 transactions. [Computer and Information Sciences]
n., An action or set of actions occurring between two or more persons relating to the conduct of business, commercial, or
governmental affairs (TISH). [General Dictionaries]
procedure. It is a valuable aid to the tracking down of errors and can be used as part of an auditing process. The time and date are
n., The process of converting or upgrading digital objects or systems to a newer generation of hardware and/or software computer technology. [Computer a


nd Information Sciences]
udf
n., The process of converting records in the usual and ordinary course of business (otherwise the activity is not migration but creation) to maintain their compatibility with a newer generation of hardware and/or software computer technology, while leaving intact their intellectual form. [Archives]
transmission
n., The moving of a record across space (from a person or organization to another, or from a system to another), or through time. [Archives]
trusted custodian
n., A


 preserver who can demonstrate that it has no reason to alter the preserved records or allow others to alter them and is capable of implementing all of the requirements for the preservation of authentic copies of records. [Archives]
trusted records officer
n., The whole of the rules that control the preservation and use of the records of the creator and provide a circumstantial probability of the authenticity of the records, and the tools and mechanisms used to implement those rules. [Models (MCP)]
trust


ed recordkeeping system
n., The whole of the rules that control the creation, maintenance use and disposition of the records of the creator and provide a circumstantial probability of the authenticity of the records, and the tools and mechanisms used to implement those rules. [Archives]
n., A type of system where rules govern which documents are eligible for inclusion in the record-keeping system, who may place records in the system and retrieve records from it, what may be done to and with a record, how


long records remain in the
system, and how records are removed from it. [Archives]
n., A system that comprises the whole of the rules that control the creation, maintenance, use and disposition of the records of the
creator and that provide a circumstantial probability of the authenticity of the records within the system. [Archives]
 Initialism for “uniform disk format.” [Computer and Information Sciences]
n., An individual or a unit within the creating organization who is responsible for keeping and man


aging the creator’s records, who has no reason to alter the creator’s records or allow others to alter them and who is capable of implementing all of the requirements for authentic records. [Archives]
n., The moving of a record across space (from a person or organization to another, or from a system to another), or
through time. [Archives]
n.,
 Any outsider or person not a party to the fact or act nor immediately concerned with it. [General Dictionaries]
n., The whole of the rules that control the creati


on, maintenance use and disposition of the records of the creator and
provide a circumstantial probability of the authenticity of the records, and the tools and mechanisms used to implement
those rules. [Archives]
n.,
 The accuracy, reliability and authenticity of a record. [Archives]
n., n.,
 B
o
th
 an accurate statement of facts and a genuine manifestation of those facts. [Archives] Dependable as authentic, reliable, and having integrity. [General Dictionaries]
 Both an accurate statement of facts and


 a genuine manifestation of those facts. [Archives]
Dependable as authentic, reliable, and having integrity. [General Dictionaries]
a., [diplomatics] The quality of a record whose content is in accordance with the actual state of affairs. With correct, perti
nent and precise, a component of accuracy. [Archives]
system, and how records are removed from it. [Archives]
n.,
 Initialism for “uniform disk format.” [Computer and Information Sciences]
url
n., A traceable and uninterrupted line of care, control


 and usually possession of a body of records from creation to preservation that can serve as a means of protecting the authenticity of the record. [Archives]
version
n., A universal 16-bit (two byte) standard character set for representing all scripts in active modern use as plain text in computer processing. [Computer and Information Sciences]
upgrade
n., (URL) A standard way of [uniquely] specifying the location of an object, typically a Web page, on the Internet; the form of address used on the Worl


d-Wide Web. Syn.: universal resource locator. [Computer and Information Sciences]
universal disk format
n., (UTF-8) An ASCII-compatible scheme for encoding Unicode values in sets of eight bits. [Computer and Information Sciences]
universal resource locator
n., A universal, vendor-independent file system standard (ISO 13346) for storing data on optical media; designed for data interchange and portability, allowing an operating system to read, write and modify data stored on optical media that were create


d by another operating system. [Computer and Information Sciences]
updated storage information
n., See
: uniform resource locator [Computer and Information Sciences]
n.,
n., Information indicating a change in the location of a digital component in storage, the occurrence of a storage problem, the action taken to correct a storage problem, the results of such actions, or the copying of the components from older to new storage media. [Archives] : uniform resource locator [Computer and Information Sciences]


n., A new or better version of some hardware or software computer technology. [Computer and Information Sciences] v., To develop or install a new or better version of some hardware or software computer technology. [Computer and
Information Sciences]
v., To raise (something esp. equipment or facilities) from one grade to another; to improve or enhance physically. [
General Dictionaries]
n., Information indicating a change in the location of a digital component in storage, the occurrence of a storage
prob


lem, the action taken to correct a storage problem, the results of such actions, or the copying of the components
from older to new storage media. [Archives]
n.,
 Initialism for “uniform resource locator.” [Computer and Information Sciences]
Information Sciences]
n.,
 Initialism for “universal character set transformation format 8.” [Computer and Information Sciences]
n., The act or process of establishing a correspondence of known facts about the record itself and the various contexts in
which it has b


een created and maintained with the proposed fact of the record's authenticity. [Archives]
n., The act or process of establishing a correspondence of known facts about the record itself and the various contexts i
n which it has been created and maintained with the proposed fact of the record's authenticity. [Archives]
v., To raise (something esp. equipment or facilities) from one grade to another; to improve or enhance physically. [General
Dictionaries]
n., One of several variations of an intellectual wo


rk, possibly created for a purpose or use other than the one originally intended. [General Dictionaries]
n.,
n.,
  A digital document perceived as existing by the user, but not existing in the system as seen. [General Dictionaries]
n., The characteristics of an entity as perceived by the user, regardless of how they have been physically represented in a database. Thus an employee would have one virtual record, but may have numerous physical records linked together to accommodate repeating addresses, jobs


held, benefits received, etc. [Archives]
n., Documents existing only as variables which are not connected by statements expressing their significance and relationships,
such as annotations on the verse of a blank parchment or date sent by electronic mail but remaining in non-related fo
rm with the
sender. [General Dictionaries]
n., One of several variations of an intellectual work, possibly created for a purpose or use other than the one originally
intended. [General Dictionaries]
n., The indication of t


he degree of importance of a record to continue the activity for which it was created or the
business of the person/office that created it. [Archives]
n., [computing] A fundamental unit of storage in a computer. The size of a word in a particular computer architecture is
one of its chief distinguishing characteristics. [Computer and Information Sciences]
n., Computer memory that requires electrical power and, in some cases periodic refreshment (e.g., DRAM), to maintain its sto
red content. Syn.: primary


storage. Opp.: non-volatile storage. [Computer and Information Sciences]
n., A distinct expression of human thought or emotion made in language, signs, symbols, numerals, images, or some other
form, for purposes of communication and remembrance. [Archives]
n.,
 Initialism for “wide area network.” [Computer and Information Sciences]
word
n., (WAN) A data network usually constructed over distances greater than one kilometre. [Computer and Information Sciences]
writer
n., [computing] A fundamental unit o


f storage in a computer. The size of a word in a particular computer architecture is one of its chief distinguishing characteristics. [Computer and Information Sciences]
virtual record
n., A distinct expression of human thought or emotion made in language, signs, symbols, numerals, images, or some other form, for purposes of communication and remembrance. [Archives]
wrapper format
n., A data structure or software that encapsulates (“wraps around”) other data or software objects, appends code or other so


ftware for the purposes of improving user convenience, hardware or software compatibility, or enhancing data security, transmission or storage. [Computer and Information Sciences]
wide area network
n., A specified wrapper structure for encapsulating multiple bitstreams into a single file. [Computer and Information Sciences]  The person responsible for the intellectual form of the record. [Archives]
n.,
 Person having the authority and capacity to articulate the content of the record. [Archives]
n.,
 The


 person responsible for the intellectual form of the record. [Archives]
n., Writer is an obsolescent Scottishism in the sense "an attorney or law-agent; an ordinary legal practitioner in country towns; a
 law-clerk" (OED). [Government]
n., [Securities]. A person or institution that sells securities or futures option contracts. [Government]
xml
n.,
 The designation (name) of the person competent for the articulation of the content of the record. [Archives]
xml document
n., A document created by a physi


cal or juridical person in the course of practical activity that is produced on a medium (paper, magnetic tape, disc, plate, etc.) by means of a writing instrument (pen, pencil, typing machine, printer, etc.) or of an apparatus for fixing data, images and/or voices. [Archives]
n.,n.,
 Initialism for “eXtensible Markup Language.” [Computer and Information Sciences]
n., An SGML-compliant digital document encoded using eXtensible Markup Language (XML) in conformance with the
syntactic rules described in a Do


cument Type Definition (DTD) or a schema document. [Computer and Information
Sciences]
n., An SGML-compliant digital document encoded using eXtensible Markup Language (XML) in conformance with the syntactic r
ules described in a Document Type Definition (DTD) or a schema document. [Computer and Information Sciences]


Self-Rewarding Language Models
Weizhe Yuan1,2
Richard Yuanzhe Pang1,2
Kyunghyun Cho2
Sainbayar Sukhbaatar1
Jing Xu1
Jason Weston1,2
1 Meta
2 NYU
Abstract
We posit that to achieve superhuman agents, future models require super-
human feedback in order to provide an adequate training signal. Current
approaches commonly train reward models from human preferences, which
may then be bottlenecked by human performance level, and secondly these
separate frozen reward models cannot then learn to improve during LLM
t


raining. In this work, we study Self-Rewarding Language Models, where the
language model itself is used via LLM-as-a-Judge prompting to provide its
own rewards during training. We show that during Iterative DPO training
that not only does instruction following ability improve, but also the ability
to provide high-quality rewards to itself. Fine-tuning Llama 2 70B on three
iterations of our approach yields a model that outperforms many existing
systems on the AlpacaEval 2.0 leaderboard, including Claude 2, G


emini Pro,
and GPT-4 0613. While only a preliminary study, this work opens the door
to the possibility of models that can continually improve in both axes.
1
Introduction
Aligning Large Language Models (LLMs) using human preference data can vastly improve
the instruction following performance of pretrained models [Ouyang et al., 2022, Bai et al.,
2022a]. The standard approach of Reinforcement Learning from Human Feedback (RLHF)
learns a reward model from these human preferences. The reward model is then fro


zen and
used to train the LLM using RL, e.g., via PPO [Schulman et al., 2017]. A recent alternative
is to avoid training the reward model at all, and directly use human preferences to train the
LLM, as in Direct Preference Optimization [DPO; Rafailov et al., 2023]. In both cases, the
approach is bottlenecked by the size and quality of the human preference data, and in the
case of RLHF the quality of the frozen reward model trained from them as well.
In this work, we instead propose to train a self-improving


 reward model that, rather than
being frozen, is continually updating during LLM alignment, in order to avoid this bottleneck.
The key to such an approach is to develop an agent that possesses all the abilities desired
during training, rather than separating them out into distinct models such as a reward
model and a language model. In the same way that pretraining and multitasking training of
instruction following tasks allow task transfer by training on many tasks at once [Collobert
and Weston, 2008, Radfo


rd et al., 2019, Ouyang et al., 2022], incorporating the reward
model into that same system allows task transfer between the reward modeling task and the
instruction following tasks.
We thus introduce Self-Rewarding Language Models, agents that both (i) act as instruction
following models generating responses for given prompts; and (ii) can generate and evaluate
new instruction following examples to add to their own training set. We train these models
using an Iterative DPO framework similar to that recentl


y introduced in Xu et al. [2023].
Starting from a seed model, as shown in Figure 1, in each iteration there is a process
arXiv:2401.10020v1  [cs.CL]  18 Jan 2024
Generate
responses
Generate
rewards
Preference
pairs
DPO
training
 select
Generated
new prompts
Self-Instruction creation
Instruction following training
Next iteration model
Seed model
(for t=1)
Figure 1: Self-Rewarding Language Models. Our self-alignment method consists of two
steps: (i) Self-Instruction creation: newly created prompts are used


 to generate candidate
responses from model Mt, which also predicts its own rewards via LLM-as-a-Judge prompting.
(ii) Instruction following training: preference pairs are selected from the generated data,
which are used for training via DPO, resulting in model Mt+1. This whole procedure can
then be iterated resulting in both improved instruction following and reward modeling ability.
of Self-Instruction creation whereby candidate responses are generated by the model for
newly created prompts, and are then


assigned rewards by that same model. The latter
is implemented via LLM-as-a-Judge prompting, which can also be seen as an instruction
following task. A preference dataset is built from the generated data, and the next iteration
of the model is trained via DPO.
In our experiments, we start with a Llama 2 70B [Touvron et al., 2023] seed model fine-tuned
on Open Assistant [Köpf et al., 2023], and then perform the above training scheme. We
find that not only does the instruction following performance improve fr


om Self-Rewarding
LLM alignment compared to the baseline seed model, but importantly the reward modeling
ability, which is no longer fixed, improves as well. This means that the model during iterative
training is able, at a given iteration, to provide a higher quality preference dataset to itself
than in the previous iteration. While this effect likely saturates in real-world settings, it
provides the intriguing possibility of obtaining reward models (and hence LLMs) that are
superior to ones that could hav


e been trained from the original human-authored seed data
alone.
2
Self-Rewarding Language Models
Our approach first assumes access to a base pretrained language model, and a small amount
of human-annotated seed data. We then build a model that aims to possess two skills
simultaneously:
1. Instruction following: given a prompt that describes a user request, the ability to
generate a high quality, helpful (and harmless) response.
2. Self-Instruction creation: the ability to generate and evaluate new instruct


ion-
following examples to add to its own training set.
These skills are used so that the model can perform self-alignment, i.e., they are the
components used to iteratively train itself using AI Feedback (AIF).
Self-instruction creation consists of generating candidate responses and then the model itself
judging their quality, i.e., it acts as its own reward model, replacing the need for an external
one. This is implemented via the LLM-as-a-Judge mechanism [Zheng et al., 2023b], i.e., by
formulating the ev


aluation of responses as an instruction following task. This self-created
AIF preference data is used as a training set.
Our overall self-alignment procedure is an iterative one, which proceeds by building a series
of such models, with the aim that each improves over the last. Importantly, because the
model can both improve its generation ability, and it acts as its own reward model through
the same generation mechanism, this means the reward model itself can improve through
these iterations, deviating from


 standard practices where the reward model is fixed [Ouyang
2
et al., 2022]. We believe this can increase the ceiling of the potential for self-improvement of
these learning models going forward, removing a constraining bottleneck.
We describe these steps in more detail below. An overview of the approach is illustrated in
Figure 1.
2.1
Initialization
Seed instruction following data:
We are given a seed set of human-authored (instruction
prompt, response) general instruction following examples that we use fo


r training in a
supervised fine-tuning (SFT) manner, starting from a pretrained base language model.
Subsequently this will be referred to as Instruction Fine-Tuning (IFT) data.
Seed LLM-as-a-Judge instruction following data:
We also assume we are provided
a seed set of (evaluation instruction prompt, evaluation result response) examples which
can also be used for training. While this is not strictly necessary, as the model using IFT
data will already be capable of training an LLM-as-a-Judge, we show that s


uch training
data can give improved results. In this data, the input prompt asks the model to evaluate
the quality of a given response to a particular instruction. The provided evaluation result
response consists of chain-of-thought reasoning (a justification), followed by a final score (in
our experiments out of 5). The format we choose for these prompts is given in Figure 2. This
thus serves as training data for the LLM to perform the role of a reward model. Subsequently
this will be referred to as Evalua


tion Fine-Tuning (EFT) data.
We use both these seed sets together during training.
2.2
Self-Instruction Creation
Using the model we have trained, we can make it self-modify its own training set. Specifically,
we generate additional training data for the next iteration of training.
This consists of the following steps:
1. Generate a new prompt: We generate a new prompt xi using few-shot prompting,
sampling prompts from the original seed IFT data, following the approach of Wang
et al. [2022] and Honovich et a


l. [2023].
2. Generate candidate responses: We then generate N diverse candidate responses
{y1
i , . . . , yN
i } for the given prompt xi from our model using sampling.
3. Evaluate candidate responses: Finally, we use the LLM-as-a-Judge ability of our
same model to evaluate its own candidate responses with scores rn
i ∈[0, 5] (see
Figure 2).
2.3
Instruction Following Training
As previously described, training is initially performed with the seed IFT and EFT data
(Section 2.1). This is then augmented with ad


ditional data via AI (Self-)Feedback.
AI Feedback Training
After performing the self-instruction creation procedure, we can
then augment the seed data with additional examples for training, which we refer to as AI
Feedback Training (AIFT) data. We try two variants of such feedback:
• Preference pairs: We construct training data of the form (instruction prompt xi,
winning response yw
i , losing response yl
i). To form the winning and losing pair
we take the highest and lowest scoring responses from the N eva


luated candidate
responses (see Section 2.2), following Xu et al. [2023], discarding the pair if their
scores are the same. These pairs can be used for training with a preference tuning
algorithm. We use DPO [Rafailov et al., 2023].
• Positive examples only: In this variant, we add additional examples of (instruction
prompt, response) curated by the model to the seed set for supervised fine-tuning,
following other approaches [Li et al., 2023a, Adolphs et al., 2023, Gulcehre et al.,
3
Review the user’s quest


ion and the corresponding response using the additive 5-point
scoring system described below. Points are accumulated based on the satisfaction of each
criterion:
- Add 1 point if the response is relevant and provides some information related to
the user’s inquiry, even if it is incomplete or contains some irrelevant content.
- Add another point if the response addresses a substantial portion of the user’s question,
but does not completely resolve the query or provide a direct answer.
- Award a third point i


f the response answers the basic elements of the user’s question in a
useful way, regardless of whether it seems to have been written by an AI Assistant or if it
has elements typically found in blogs or search results.
- Grant a fourth point if the response is clearly written from an AI Assistant’s perspective,
addressing the user’s question directly and comprehensively, and is well-organized and
helpful, even if there is slight room for improvement in clarity, conciseness or focus.
- Bestow a fifth point f


or a response that is impeccably tailored to the user’s question
by an AI Assistant, without extraneous information, reflecting expert knowledge, and
demonstrating a high-quality, engaging, and insightful answer.
User: <INSTRUCTION_HERE>
<response><RESPONSE_HERE></response>
After examining the user’s instruction and the response:
- Briefly justify your total score, up to 100 words.
- Conclude with the score using the format: “Score: <total points>”
Remember to assess from the AI Assistant perspective, utili


zing web search knowledge as
necessary. To evaluate the response in alignment with this additive scoring model, we’ll
systematically attribute points based on the outlined criteria.
Figure 2: LLM-as-a-Judge prompt for our LLM to act as a reward model and
provide self-rewards for its own model generations. The model is initially trained with seed
training data of how to perform well at this task, and then improves at this task further
through our self-rewarding training procedure.
2023], rather than construc


ting preference data. In this setup we only add examples
where the candidate response was evaluated to give a perfect score of rn
i = 5.
While we report the results of both approaches in our experiments, we find that learning
from preference pairs gives superior performance, and hence recommend that approach.
2.4
Overall Self-Alignment Algorithm
Iterative Training
Our overall procedure trains a series of models M1, . . . , MT where
each successive model t uses augmented training data created by the t −1th m


odel. We thus
define AIFT(Mt) to mean AI Feedback Training data created using model Mt.
Model Sequence
We thus define the models, and the training data they use as follows:
M0 : Base pretrained LLM with no fine-tuning.
M1 : Initialized with M0, then fine-tuned on the IFT+EFT seed data using SFT.
M2 : Initialized with M1, then trained with AIFT(M1) data using DPO.
M3 : Initialized with M2, then trained with AIFT(M2) data using DPO.
This iterative training resembles the procedure used in Pairwise Cringe Optim


ization and
Iterative DPO introduced in Xu et al. [2023]; however, an external fixed reward model was
used in that work.
4
3
Experiments
3.1
Experimental Setup
Base Model
In our experiments we use Llama 2 70B [Touvron et al., 2023] as our base
pretrained model.
3.1.1
Seed Training Data
IFT Seed Data
We use the human-authored examples provided in the Open Assistant
dataset [Köpf et al., 2023] for instruction fine-tuning. Following Li et al. [2023a] we use
3200 examples, by sampling only first conversational


turns in the English language that are
high-quality, based on their human annotated rank (choosing only the highest rank 0). In
our experiments, we compare to a model fine-tuned from the base model using only this data
via supervised fine-tuning, and refer to it as our SFT baseline.
EFT Seed Data
The Open Assistant data also provides multiple ranked human responses
per prompt from which we can construct evaluation fine-tuning data. We split this into train
and evaluation sets, and use it to create LLM-as-a-


Judge data. This is done by placing it in
the input format given in Figure 2, which consists of the scoring criteria description, and
the given instruction and response to be evaluated.1 For training targets, chain-of-thought
justifications and final scores out of 5 are not directly provided, so we use the SFT baseline
to generate such output evaluations for each input, and accept them into the training set if
the ranking of their scores agrees with the human rankings in the dataset. This results in
1775 tr


ain and 531 evaluation examples (which do not overlap with the IFT data).
3.1.2
Evaluation Metrics
We evaluate the performance of our self-rewarding models in two axes: their ability to follow
instructions, and their ability as a reward model (ability to evaluate responses).
Instruction Following
We evaluate head-to-head performance between various models
using GPT-4 [Achiam et al., 2023] as an evaluator over 256 test prompts derived from various
sources following Li et al. [2023a] using the AlpacaEval eval


uation prompt [Li et al., 2023b].
We try the prompt in both orders comparing pairwise, and if the GPT-4 evaluations disagree
we count the result as a tie. We also report results in the AlpacaEval 2.0 leaderboard format
which is evaluated over 805 prompts, and compute the win rate against the baseline GPT-4
Turbo model based on GPT-4 judgments.
Reward Modeling
We evaluate the correlation with human rankings on the evaluation set
we derived from the Open Assistant dataset, as described in Section 3.1.1. Each


instruction
has on average 2.85 responses with given rankings. We can thus measure the pairwise
accuracy, which is how many times the order of the ranking between any given pair agrees
between the model’s evaluation and the human ranking. We also measure the exact match
count, which is how often the total ordering is exactly the same for an instruction. We also
report the Spearman correlation and Kendall’s τ. Finally, we report how often the responses
that the model scores a perfect 5 out of 5 are rated as


the highest ranked by humans.
3.1.3
Training Details
Instruction following training
The training hyperparameters we use are as follows.
For SFT we use learning rate 5.5e−6 which linearly decays to 1.1e−6, batch size 16 and
dropout 0.1. We only calculate the loss on target tokens instead of the full sequence. For
DPO we use learning rate 1e−6 which linearly decays to 1e−7, batch size 16, dropout 0.1,
and a β value of 0.1. We perform early stopping by saving a checkpoint every 200 steps
and evaluating generat


ions using Claude 2 [Anthropic, 2023] on 253 validation examples
derived from various sources following Li et al. [2023a]. This is evaluated pairwise against
1Note, the prompt, derived from Li et al. [2023a], mentions “utilizing web search”, but our model
is not actually capable of this action.
5
the previous step’s generations using the AlpacaEval evaluation prompt format [Li et al.,
2023b].
Self-Instruction creation
To generate new prompts we used a fixed model, Llama 2-Chat
70B with 8-shot prompting, whe


reas the other parts of the creation pipeline (generating the
response, and evaluating it) use the model being trained. For candidate response generation
we sample N = 4 candidate responses with temperature T = 0.7, p = 0.9. When evaluating
candidate responses, as there is variance to these scores, in our experiments we also use
sampled decoding (with the same parameters) and generate these evaluations multiple (3)
times and take the average. We added 3,964 such preference pairs to form the AIFT(M1)
dataset


 used to train M2 via DPO, and 6,942 pairs to form AIFT(M2) used to train M3.
3.2
Results
3.2.1
Instruction Following Ability
Head to head performance results are provided in Figure 3.
EFT+IFT seed training performs similarly to IFT alone
We find that adding
the Evaluation Fine-Tuning (EFT) task to training does not impact instruction following
performance compared to using Instruction Fine-Tuning (IFT) data alone with an almost
equal head to head (30.5% wins vs. 30.9% wins). This is a positive result becau


se it means
the increased capability of a model to self-reward does not affect its other skills. We can
thus use IFT+EFT training as Iteration 1 (M1) of our Self-Rewarding model, and then run
further iterations.
Iteration 2 (M2) improves over Iteration 1 (M1) and SFT Baseline
Iteration 2
of Self-Rewarding training (M2) provides superior instruction following to Iteration 1 (M1)
with 55.5% wins for M2 compared to only 11.7% for M1 in a head to head evaluation. It
provides similar gains over the SFT Baseline


as well (49.2% wins vs. 14.5% wins). Clearly,
there is a large jump in performance from M1 to M2 by using the preference data AIFT(M1)
provided by the reward model from Iteration 1.
Iteration 3 (M3) improves over Iteration 2 (M2)
We see a further gain in Iteration 3
over Iteration 2, with 47.7% wins for M3 compared to only 12.5% for M2 in a head to head
evaluation. Similarly, the win rate over the SFT Baseline for M3 increases to 62.5% wins vs.
9.8%, i.e. winning more often than the M2 model did. Overall, w


e see large gains from M2
to M3 through training using the preference data AIFT(M2) provided by the reward model
from Iteration 2.
Self-Rewarding models perform well on AlpacaEval 2 leaderboard
We evaluate
our models on the AlpacaEval 2.0 leaderboard format, with results given in Table 1. We
observe the same findings as in the head-to-head evaluations, that training iterations yield
improved win rates over GPT4-Turbo, from 9.94% in Iteration 1, to 15.38% in Iteration 2,
to 20.44% in Iteration 3. Our Iterati


on 3 model outperforms many existing models in this
metric, including Claude 2, Gemini Pro, and GPT4 0613. We show some selected models
from the leaderboard in the table. We note that many of those competing models contain
either proprietary alignment data (which is typically large, e.g., over 1M annotations in
[Touvron et al., 2023]) or use targets that are distilled from stronger models. In contrast,
our Self-Rewarding model starts from a small set of seed data from Open Assistant, and
then generates targ


ets and rewards from the model itself for further iterations of training.
Preference optimization outperforms augmenting with positive examples only
We also tried the alternative self-training procedure of adding high-quality self-instruction
creation examples to supervised fine-tuning (without preference optimization). Unfortunately
we could not find a configuration where this approach helped. For example, adding 11,254
such examples that scored 5 out of 5, and optimizing the mixing weight in training, sti


ll
yielded a head to head with the SFT Baseline of 29% wins vs 30% wins, i.e., no improvement.
6
Self-Rewarding M_3
vs.
SFT Baseline
Self-Rewarding M_2
vs.
SFT Baseline
Self-Rewarding M_1
vs.
SFT Baseline
62.5
49.2
30.5
27.7
36.3
38.7
9.8
14.5
30.9
Self-Rewarding Wins
Tie
SFT Baseline Wins
Self-Rewarding M_3
vs.
M_2
Self-Rewarding M_2
vs.
M_1
Self-Rewarding M_3
vs.
M_1
47.7
55.5
68.8
39.8
32.8
22.7
12.5
11.7
8.6
Left Wins (in Left vs. Right)
Tie
Right Wins
Figure 3: Instruction following ability improves wi


th Self-Training: We evaluate our
models using head-to-head win rates on diverse prompts using GPT-4. The SFT Baseline is
on par with Self-Rewarding Iteration 1 (M1). However, Iteration 2 (M2) outperforms both
Iteration 1 (M1) and the SFT Baseline. Iteration 3 (M3) gives further gains over Iteration 2
(M2), outperforming M1, M2 and the SFT Baseline by a large margin.
Data distribution analysis
We perform a t-SNE [Van der Maaten and Hinton, 2008]
visualization of the IFT, EFT and AIFT(M1) data, shown in Sect


ion A.1. We find good
overlap between the IFT and AIFT(M1) examples, which is desired, while the EFT examples
lie in a different part of the embedding space. We observe that generations from M1 have
an average length of 1092, for M2 they are 1552, and for M3 they are 2552, so the model is
learning to generate longer responses, which we note may be a factor in relative performance.
3.2.2
Reward Modeling Ability
Reward modeling evaluation results are provided in Table 2.
EFT augmentation improves over SFT bas


eline
Firstly, we find that adding Eval-
uation Fine-Tuning (EFT) data into training, which gives examples to the model of how
to act as an LLM-as-a-Judge, naturally improves its performance compared to training
with Instruction Fine-Tuning (IFT) data alone. IFT data covers a wide range of general
instruction tasks, and so does endow the SFT Baseline with the ability to evaluate responses,
however EFT data gives more examples of this specific task. We find improvements across
all five metrics measured when


using IFT+EFT vs. IFT alone, e.g. the pairwise accuracy
agreement with humans increases from 65.1% to 78.7%.
Reward Modeling ability improves with Self-Training
We find that performing
a round of self-reward training improves the ability of the model at providing self-rewards
for the next iteration, in addition to its improved instruction following ability. Model M2
(Iteration 2) is trained using the reward model from M1 (Iteration 1), but provides improved
performance on all five metrics compared to M1. Fo


r example, pairwise accuracy improves
from 78.7% to 80.4%. Iteration 3 (M3) improves several of these metrics further compared
7
Table 1: AlpacaEval 2.0 results (win rate over GPT-4 Turbo evaluated by GPT-4).
Self-Rewarding iterations yield improving win rates. Iteration 3 (M3) outperforms many
existing models that use proprietary training data or targets distilled from stronger models.
Alignment Targets
Model
Win Rate
Distilled
Proprietary
Self-Rewarding 70B
Iteration 1 (M1)
9.94%
Iteration 2 (M2)
15.38%
I


teration 3 (M3)
20.44%
Selected models from the leaderboard
GPT-4 0314
22.07%
✓
Mistral Medium
21.86%
✓
Claude 2
17.19%
✓
Gemini Pro
16.85%
✓
GPT-4 0613
15.76%
✓
GPT 3.5 Turbo 0613
14.13%
✓
LLaMA2 Chat 70B
13.87%
✓
Vicuna 33B v1.3
12.71%
✓
Humpback LLaMa2 70B
10.12%
Guanaco 65B
6.86%
Davinci001
2.76%
✓
Alpaca 7B
2.59%
✓
Table 2: Reward Modeling ability improves with Self-Training: We evaluate the LLM-
as-a-Judge via various metrics which measure alignment with held-out human preference data.
Self-Rewarding


Iteration 2 (Model M2), which is trained using the self-reward model derived
from its previous iteration M1 outperforms Iteration 1 (M1), while M1 itself outperforms a
standard SFT baseline model trained on only Instruction Fine-Tuning (IFT) data. Iteration
3 (Model M3) gives further improvements over Iteration 2.
Self-Rewarding Models
Model
SFT Baseline
Iter 1 (M1)
Iter 2 (M2)
Iter 3 (M3)
Training data
IFT
IFT+EFT
IFT+EFT
IFT+EFT
+AIFT(M1)
+AIFT(M2)
Pairwise accuracy (↑)
65.1%
78.7%
80.4%
81.7%
5-best % (↑


)
39.6%
41.5%
44.3%
43.2%
Exact Match % (↑)
10.1%
13.1%
14.3%
14.3%
Spearman corr. (↑)
0.253
0.279
0.331
0.349
Kendall τ corr. (↑)
0.233
0.253
0.315
0.324
to M2, for example pairwise accuracy increases from 80.4% to 81.7%. This performance gain
is achieved despite there being no additional EFT data provided, and the examples created
during the Self-Instruction creation loop do not tend to look like LLM-as-a-Judge training
examples. We hypothesize that because the model is becoming better at general instruct


ion
following, it nevertheless also improves at the LLM-as-a-Judge task.
Importance of the LLM-as-a-Judge Prompt
In these experiments we used the LLM-
as-Judge prompt format shown in Figure 2. In preliminary experiments we also tried various
other prompts to decide the most effective one to use. For example, we tried the prompt
proposed in Li et al. [2023a] which also proposes a 5-point scale, but describes the options as
multiple choice in a range of quality buckets, see Figure 5. In contrast, our prompt d


escribes
the points as additive, covering various aspects of quality. We find a large difference between
8
these two prompts when using the SFT Baseline, e.g. 65.1% pairwise accuracy for ours, and
only 26.6% pairwise accuracy for theirs. See Section A.2 for further details.
4
Related Work
Automatically improving or self-correcting large language models is becoming a major focus
of research. A recent survey from Pan et al. [2023] attempts to summarize the topic. However,
this is a rapidly moving area, and th


ere are already promising new works not covered there.
Reinforcement Learning from Human Feedback (RLHF)
Preference learning ap-
proaches such as in Ziegler et al. [2019], Stiennon et al. [2020], Ouyang et al. [2022], Bai et al.
[2022a] train a fixed reward model from human preference data, and then use the reward
model to train via reinforcement learning (RL), e.g. via Proximal Policy Optimization (PPO)
[Schulman et al., 2017]. Thus, the reward signal in a certain sense already comes from a model
even in t


hese works, but distilled from human data. Nevertheless, this is commonly referred
to as RL from Human Feedback (RLHF). Methods such as Direct Preference Optimization
(DPO) [Rafailov et al., 2023] avoid training the reward model entirely, and instead directly
train the LLM using human preferences. Several other such competing methods exist as
well [Zhao et al., 2023, Zheng et al., 2023a, Yuan et al., 2023], including Pairwise Cringe
Optimization (PCO) [Xu et al., 2023], which was shown to outperform DPO and


 PPO on
AlpacaFarm [Dubois et al., 2023]. PCO uses an iterative training approach similar to the
one in our work, except with a fixed reward model, and also showed that Iterative DPO
improves over DPO using the same scheme.
Reinforcement Learning from AI Feedback (RLAIF)
Constitutional AI [Bai et al.,
2022b] uses an LLM to give feedback and refine responses, and uses this data to train a
(separate, fixed) reward model. This is then used to perform “RL from AI Feedback” (RLAIF).
Lee et al. [2023] compare RLA


IF and RLHF procedures and find the methods they compare
perform roughly equally. They use an “off-the-shelf” LLM to perform LLM-as-a-Judge
prompting to build a training set to train a fixed reward model, which is then used for RL
training. They also experiment with using the fixed but separate LLM-as-a-Judge model
directly, which the authors report is computationally expensive due to using it within PPO
training (rather than the offline step in the iterative approach we use in our work, which
is relatively


 computationally cheap). Finally, Chen et al. [2024] recently showed they can
avoid reward models entirely in an Iterative DPO-like framework by using human labels as
the winning response in a pair, and the last iteration’s generations as the losing response in
the pair. The authors note this has the limitation that once the model generations reach
human performance, they are bottlenecked. Further, each input prompt is required to have
a human annotated response, in contrast to our work.
Improving LLMs via


data augmentation (and curation)
Several methods have
improved LLMs by (self-)creating training data to augment fine-tuning. Self-Instruct [Wang
et al., 2022] is a method for self-instruction creation of prompts and responses, which can
be used to improve a base LLM. We make use of a similar technique in our work, and then
use our self-reward model to score them. Several approaches have also created training data
by distilling from powerful LLMs, and shown a weaker LLM can then perform well. For
example, Al


paca [Taori et al., 2023] fine-tuned a Llama 7B model with text-davinci-003
instructions created in the style of self-instruct. Alpagasus [Chen et al., 2023] employed a
strong LLM-as-a-Judge (ChatGPT) to curate the Alpaca dataset and filter to a smaller set,
obtaining improved results. Instruction Backtranslation [Li et al., 2023a] similarly augments
and curates training data, but augmenting via backtranslating from web documents to predict
prompts. The curation is done by the LLM(-as-a-Judge) itself, so ca


n be seen as an instance
of a self-rewarding model, but in a specialized setting. Reinforced Self-Training (ReST)
[Gulcehre et al., 2023] uses a fixed, external reward to curate new high-quality examples to
iteratively add to the training set, improving performance. We note that in our experiments,
we found that adding only positive examples in a related manner did not help, whereas
adding preference pairs did help.
9
LLM-as-a-Judge
Using LLM-as-a-Judge prompting to evaluate language models has
become a sta


ndard approach [Dubois et al., 2023, Li et al., 2023b, Fernandes et al., 2023,
Bai et al., 2023, Saha et al., 2023], and is being used to train reward models or curate data
as well, as described above [Lee et al., 2023, Chen et al., 2023, Li et al., 2023a]. While some
works such as Kim et al. [2023] create training to data to train an LLM to perform well as a
judge, to our knowledge it is not common to combine this training with general instruction
following skills as in our work.
5
Conclusion
We have intro


duced Self-Rewarding Language Models, models capable of self-alignment via
judging and training on their own generations. The method is trained in an iterative manner,
where in each iteration the model creates its own preference-based instruction training data.
This is done by assigning rewards to its own generations via LLM-as-a-Judge prompting,
and using Iterative DPO to train on the preferences. We showed that this training both
improves the instruction following capability of the model, as well as its r


eward-modeling
ability across the iterations. While this is only a preliminary study, we believe this is an
exciting avenue of research because this means the model is better able to assign rewards
in future iterations for improving instruction following – a kind of virtuous circle. While
this improvement likely saturates in realistic scenarios, it still allows for the possibility of
continual improvement beyond the human preferences that are typically used to build reward
models and instruction following m


odels today.
6
Limitations
While we have obtained promising experimental results, we currently consider them pre-
liminary because there are many avenues yet to explore, among them the topics of further
evaluation, including safety evaluation, and understanding the limits of iterative training.
We showed that the iterations of training improve both instruction following and reward
modeling ability, but only ran three iterations in a single setting. A clear line of further
research is to understand the “scal


ing laws” of this effect both for more iterations, and with
different language models with more or less capabilities in different settings.
While we have evaluated our models using GPT-4 using head-to-head and AlpacaEval 2
leaderboard style evaluation, there are many other automatic evaluation benchmarks that
one can measure. Further, we observed an increase in length in model generations, and there
is a known correlation between length and estimated quality, which is a topic that should be
understood more


deeply in general, and in our results in particular as well. It would also be
good to understand if so-called “’reward-hacking” can happen within our framework, and in
what circumstances. As we are using both a language model as the training reward, and
a language model for final evaluation, even if they are different models, this may require
a deeper analysis than we have provided. We conducted a preliminary human (author)
evaluation, which validated the automatic results that we see, but more detailed hum


an
evaluation would be beneficial.
Another clear further avenue of study is to conduct safety evaluations – and to explore safety
training within our framework. Reward models have been built exclusively for safety in
existing systems [Touvron et al., 2023], and a promising avenue here would be to use the
LLM-as-a-Judge procedure to evaluate for safety specifically in our self-rewarding training
process. Given that we have shown that reward modeling ability improves over training
iterations, this could mean


that the safety of the model could potentially improve over time
as well, with later iterations being able to capture more challenging safety situations that
earlier iterations cannot.
10
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni
Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Leonard Adolphs, Tianyu Gao, Jing Xu, Kurt Shuster, Sainbayar Sukhbaatar, and Jason
W


eston. The CRINGE loss: Learning what language not to model. In Anna Rogers, Jordan
Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of
the Association for Computational Linguistics (Volume 1: Long Papers), pages 8854–8874,
Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/
v1/2023.acl-long.493. URL https://aclanthology.org/2023.acl-long.493.
Anthropic. Claude 2. https://www.anthropic.com/index/claude-2, 2023.
Yuntao Bai, Andy Jones, Kamal N


dousse, Amanda Askell, Anna Chen, Nova DasSarma,
Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful
and harmless assistant with reinforcement learning from human feedback. arXiv preprint
arXiv:2204.05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy
Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu-
tional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022b.
Yushi Bai, Jiahao Yi


ng, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng
Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. Benchmarking
foundation models with language-model-as-an-examiner. In Thirty-seventh Conference
on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL
https://openreview.net/forum?id=IiRHQ7gvnq.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang,
Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. AlpaGasus: Training


a better alpaca
with fewer data. arXiv preprint arXiv:2307.08701, 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning
converts weak language models to strong language models. arXiv preprint arXiv:2401.01335,
2024.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing:
Deep neural networks with multitask learning. In Proceedings of the 25th International
Conference on Machine Learning, pages 160–167, 2008.
Yann Dubois, Xuechen Li,


 Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos
Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework
for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023.
Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André FT Martins,
Graham Neubig, Ankush Garg, Jonathan H Clark, Markus Freitag, and Orhan Firat.
The devil is in the errors: Leveraging large language models for fine-grained machine
translation evaluation. arXiv p


reprint arXiv:2308.07286, 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts,
Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al.
Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998,
2023.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning
language models with (almost) no human labor. In Anna Rogers, Jordan Boyd-Graber,
and Naoaki Okazaki, editors, Proceedings of the 61


st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pages 14409–14428, Toronto, Canada,
July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.806.
URL https://aclanthology.org/2023.acl-long.806.
11
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun,
Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine-grained
evaluation capability in language models. arXiv preprint arXiv:2310.08


491, 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam,
Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi,
et al. OpenAssistant conversations–democratizing large language model alignment. arXiv
preprint arXiv:2304.07327, 2023.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop,
Victor Carbune, and Abhinav Rastogi. RLAIF: Scaling reinforcement learning from human
feedback with ai feedback. arXiv preprint


 arXiv:2309.00267, 2023.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason
Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint
arXiv:2308.06259, 2023a.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin,
Percy Liang, and Tatsunori B. Hashimoto.
Alpacaeval: An automatic evaluator of
instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023b.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida


, Carroll Wainwright, Pamela Mishkin,
Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language
models to follow instructions with human feedback. Advances in Neural Information
Processing Systems, 35:27730–27744, 2022.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang
Wang. Automatically correcting large language models: Surveying the landscape of diverse
self-correction strategies. arXiv preprint arXiv:2308.03188, 2023.
Alec Radford, Jeffrey Wu, R


ewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.
Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and
Chelsea Finn. Direct preference optimization: Your language model is secretly a reward
model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
URL https://openreview.net/forum?id=HPuSIXJaa9.
Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jas


on Weston, and Xian
Li. Branch-solve-merge improves large language model evaluation and generation. arXiv
preprint arXiv:2310.15123, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal
policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec
Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human
feedback. Advances in Neural Information Process


ing Systems, 33:3008–3021, 2020.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin,
Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following
llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei,
Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2:
Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09


288, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of
machine learning research, 9(11), 2008.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi,
and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated
instructions. arXiv preprint arXiv:2212.10560, 2022.
12
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more
cringe than others: Preference optimization with the pairwise cring


e loss. arXiv preprint
arXiv:2312.16682, 2023.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.
RRHF: Rank responses to align language models with human feedback. In Thirty-seventh
Conference on Neural Information Processing Systems, 2023. URL https://openreview.
net/forum?id=EdIGMCHk4l.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J
Liu. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv preprint
arXiv:2305.10425, 2023.
Chujie


Zheng, Pei Ke, Zheng Zhang, and Minlie Huang. Click: Controllable text generation
with sequence likelihood contrastive learning. In Anna Rogers, Jordan Boyd-Graber, and
Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL
2023, pages 1022–1040, Toronto, Canada, July 2023a. Association for Computational
Linguistics. doi: 10.18653/v1/2023.findings-acl.65. URL https://aclanthology.org/
2023.findings-acl.65.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu


, Yonghao
Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and
Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh
Conference on Neural Information Processing Systems Datasets and Benchmarks Track,
2023b. URL https://openreview.net/forum?id=uccHPGDlao.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei,
Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.
arXiv preprint


arXiv:1909.08593, 2019.
13
A
Appendix
A.1
Distributions of IFT, EFT and AIFT data
100
50
0
50
Dimension 1
75
50
25
0
25
50
75
Dimension 2
IFT data
EFT data
AIFT data
(a) Instruction distribution of IFT, EFT and
AIFT data.
50
0
50
Dimension 1
80
60
40
20
0
20
40
60
Dimension 2
IFT data
EFT data
AIFT data
(b) Response distribution of IFT, EFT, and AIFT
data.
Figure 4: Distributions of both instructions and responses for IFT, EFT and AIFT data.
We have plotted the distribution of instructions for IFT, EFT and


AIFT data, and the
distribution of responses for IFT, EFT and AIFT data in Figure 4. It is clear that the IFT
data and EFT data come from very different distributions while the IFT and AIFT data
come from similar distributions.
A.2
Other EFT prompts we have tried
At first, we took the EFT prompt from Li et al. [2023a] as shown in Figure 5. However, we
found that this prompt was not as effective as our additive score-counting prompt because
the model needed to treat the task as a multiple-choice problem, and


 it was difficult for the
model to break down this multiple-choice problem into sub-problems involving evaluating
various aspects of the response. When using the model trained on 3,200 IFT data only, its
performance on the EFT test set using our additive score-counting prompt and prompt from
Li et al. [2023a] is shown in Table 3.
EFT Prompt
Multiple Choice prompt
Ours
Pairwise accuracy (↑)
26.6%
65.1%
5-best % (↑)
23.5%
39.6%
Exact Match % (↑)
1.1%
10.1%
Spearman corr. (↑)
-0.18
0.25
Kendall τ corr. (↑)
-0.


16
0.23
Table 3: We tried various LLM-as-Judge prompts using the model trained with 3,200 IFT
data only and found that our additive score-counting prompt worked best which demonstrates
significant improvements in EFT performance comparing to the prompt used by Li et al.
[2023a].
14
Below is a question from an user and a candidate response. Please grade the response on a
5-point scale using the following criteria:
1:
It means the answer is incomplete, vague, off-topic, controversial, or not ex-
actly what th


e user asked for. For example, some content seems missing, numbered list
does not start from the beginning, the opening sentence repeats user’s question. Or the
response is from another person’s perspective with their personal experience (e.g. taken
from blog posts), or looks like an answer from a forum. Or it contains promotional text,
navigation text, or other irrelevant information.
2: It means the answer addresses most of the asks from the user. It does not directly
address the user’s question. For exam


ple, it only provides a high-level methodology instead
of the exact solution to user’s question.
3: It means the answer is helpful but not written by an AI Assistant. It addresses all
the basic asks from the user. It is complete and self contained with the drawback that
the response is not written from an AI assistant’s perspective, but from other people’s
perspective. The content looks like an excerpt from a blog post, web page, or web search
results.
For example, it contains personal experience or opinion


, mentions comments
section, or share on social media, etc.
4: It means the answer is written from an AI assistant’s perspective with a clear focus of
addressing the instruction. It provide a complete, clear, and comprehensive response to
user’s question or instruction without missing or irrelevant information. It is well organized,
self-contained, and written in a helpful tone. It has minor room for improvement, e.g. more
concise and focused.
5: It means it is a perfect answer from an AI Assistant. It has


a clear focus on being a
helpful AI Assistant, where the response looks like intentionally written to address the
user’s question or instruction without any irrelevant sentences. The answer provides high
quality content, demonstrating expert knowledge in the area, is very well written, logical,
easy-to-follow, engaging and insightful.
User: <INSTRUCTION_HERE>
<response><RESPONSE_HERE></response>
Please first briefly describe your reasoning (in less than 100 words),
and then
write “Score: <rating>” in the la


st line. Answer in the style of an AI Assistant, with
knowledge from web search if needed. To derive the final score based on the criteria, let’s
think step-by-step.
Figure 5: LLM-as-a-Judge prompt taken from Li et al. [2023a].
15


Data Sanitization Techniques

A Net 2000 Ltd. White Paper

Abstract

Data Sanitization is the process of making sensitive information in non-production  databases
safe for wider visibility. This White Paper is an overview of various techniques which can be
used to sanitize sensitive production data in test and development databases.

An initial discussion of the primary motivations for data sanitization is given. The remainder of
the paper is devoted


to a generic survey of the various masking techniques and their individual
benefits and drawbacks.

Some keywords which may assist you in finding this document online are:

Data Sanitization, Data Sanitisation, Data Masking, Data Obfuscation, Data Security, Data
Cleansing, Data Hiding, Data Protection Act 1998, Hide Data, Disguise Data, Sanitize Data,
Sanitise Data, Gramm-Leach-Bliley Act (GLBA), Data Privacy, Directive 95/46/EC of the
European Parliament

Author:


Dale Edgar

Net 2000 Ltd.

Dale.Edgar@Net2000Ltd.com

http://www.Net2000Ltd.com

Data Sanitization Techniques
A Net 2000 Ltd. White Paper

 Copyright © Net 2000 Ltd. 2003-2004

http://www.Net2000Ltd.com

Table of Contents

Why Sanitize Information in Test and Development Databases? .............................................1
Protecting Valuable Information....................................................................


........................1
Legal Obligations ...................................................................................................................1
Data Sanitization Techniques.....................................................................................................2
Technique: NULL’ing Out .....................................................................................................2
Technique: Masking Data..................................................................


....................................2
Technique: Substitution .........................................................................................................3
Technique: Shuffling Records ................................................................................................4
Technique: Number Variance ................................................................................................4
Technique: Gibberish Generation .......................................................


..................................4
Technique: Encryption/Decryption........................................................................................5
Summary....................................................................................................................................6

Data Sanitization Techniques
A Net 2000 Ltd. White Paper

 Copyright © Net 2000 Ltd. 2003-2004

http://www.Net2000Ltd.com
Data Sanitization Technique


s

Data Sanitization is the process of disguising sensitive information in test and
development databases by overwriting it with realistic looking but false data of a
similar type.

Why Sanitize Information in Test and Development Databases?

The data in testing environments should be sanitized in order to protect valuable
business information and also because there is, in most countries, a legal obligation to
do so.

Protecting Valuable Information

Fundamentally there are two ty


pes of security. The first type is concerned with the
integrity of the data. In this case the modification of the records is strictly controlled.
For example, you may not wish an account to be credited or debited without specific
controls and auditing. This type of security is not a major concern in test and
development databases. The data can be modified at will without any business impact.

The second type of security is the protection of the information content from
inappropriate visibility. Names


, addresses, phone numbers and credit card details are
good examples of this type of data. Unlike the protection from updates, this type of
security requires that access to the information content is controlled in every
environment.

Legal Obligations

The legal requirements for Data Sanitization vary from country to country and most
countries now have regulations of some form. Here are some examples:

United States
The Gramm-Leach-Bliley Act requires institutions to protect the
confidential


ity and integrity of personal consumer information. The Right to
Financial Privacy Act of 1978 creates statutory Fourth Amendment protection
for financial records and there are a host of individual state laws

The European Union
Directive 95/46/EC of the European Parliament which provides strict
guidelines regarding individual rights to data privacy, and the responsibilities
of data holders to guard against misuse.

Data Sanitization Techniques
A Ne


t 2000 Ltd. White Paper

 Copyright © Net 2000 Ltd. 2003-2004

http://www.Net2000Ltd.com
The United Kingdom
The United Kingdom Data Protection Act of 1998 extends the European
Parliament directive and places further statutory obligations on the holders of
personal, private or sensitive data.

As with most things legal, the details are open to argument. In reality though, if data
for which your organization is responsible gets loose and appropriate steps were not
taken to prevent that release,


then your organizations lawyers could well find
themselves in court trying to put their best spin on the matter. However large the legal
liabilities are, they could seem trivial in comparison to the losses associated with the
catastrophic loss of business confidence caused by a large scale privacy breach.

Any organization that outsources test and development operations needs to be very
conscious of the specific laws regulating the transmission of information across
national borders.

Data Sa


nitization Techniques

Test and development teams need to work with databases which are structurally
correct functional copies of the live environments. However, they do not necessarily
need to be able to view security sensitive information. For test and development
purposes, as long as the data looks real, the actual record content is usually irrelevant.
There are a variety of Data Sanitization techniques available – the pro’s and con’s of
some of the most useful are discussed below.

Technique:


 NULL’ing Out

Simply deleting a column of data by replacing it with NULL values is an effective
way of ensuring that it is not inappropriately visible in test environments.
Unfortunately it is also one of the least desirable options from a test database
standpoint. Usually the test teams need to work on the data or at least a realistic
approximation of it. For example, it is very hard to write and test customer account
maintenance forms if the customer name, address and contact details are all NULL


values.

Verdict: The NULL’ing Out technique is useful in certain specific circumstances but
rarely useful as the entire Data Sanitization strategy.

Technique: Masking Data

Masking data means replacing certain fields with a Mask character (such as an X).
This effectively disguises the data content while preserving the same formatting on
front end screens and reports. For example, a column of credit card numbers might
look like:

Data Sanitization Techniques


A Net 2000 Ltd. White Paper

 Copyright © Net 2000 Ltd. 2003-2004

http://www.Net2000Ltd.com
4346 6454 0020 5379
4493 9238 7315 5787
4297 8296 7496 8724

and after the masking operation the information would appear as:

4346 XXXX XXXX 5379
4493 XXXX XXXX 5787
4297 XXXX XXXX 8724

The masking characters effectively remove much of the sensitive content from the
record while still preserving the look and feel. Take care to ensure that enough of the
data is masked to preserv


e security. It would not be hard to regenerate the original
credit card number from a masking operation such as: 4297 8296 7496 87XX
since the numbers are generated with a specific and well known checksum algorithm.
Also care must be taken not to mask out potentially required information. A masking
operation such as XXXX XXXX XXXX 5379 would strip the card issuer details
from the credit card number. This may, or may not, be desirable.

Verdict: If the data is in a specific, invariable format, then Ma


sking is a powerful and
fast Data Sanitization option. If numerous special cases must be dealt with then
masking can be slow, extremely complex to administer and can potentially leave some
data items inappropriately masked.

Technique: Substitution

This technique consists of randomly replacing the contents of a column of data with
information that looks similar but is completely unrelated to the real details. For
example, the surnames in a customer database could be sanitized by replacing the re


al
last names with surnames drawn from a largish random list.

Substitution is very effective in terms of preserving the look and feel of the existing
data. The downside is that a largish store of substitutable information must be
maintained for each column to be substituted. For example, to sanitize surnames by
substitution, a list of random last names must be available. Then to sanitize telephone
numbers, a list of  phone numbers must be available. Frequently, the ability to
generate known invalid


 data (phone numbers that will never work) is a nice-to-have
feature.

Substitution data can sometimes be very hard to find in large quantities. For example,
if a million random street addresses are required, then just obtaining the substitution
data can be a major exercise in itself.

Verdict: Substitution is quite powerful, reasonably fast and preserves the look and feel
of the data. Finding the required random data to substitute and developing the
procedures to accomplish the substitution can


be a major effort.

Data Sanitization Techniques
A Net 2000 Ltd. White Paper

 Copyright © Net 2000 Ltd. 2003-2004

http://www.Net2000Ltd.com
Technique: Shuffling Records

Shuffling is similar to substitution except that the substitution data is derived from the
column itself. Essentially the data in a column is randomly moved between rows until
there is no longer any reasonable correlation with the remaining information in the
row.

Th


ere is a certain danger in the shuffling technique. It does not prevent people from
asking questions like “I wonder if so-and-so is on the supplier list?” In other words,
the original data is still present and sometimes meaningful questions can still be asked
of it. Another consideration is the algorithm used to shuffle the data. If the shuffling
method can be determined, then the data can be easily “unshuffled”. For example, if
the shuffle algorithm simply ran down the table swapping the column data i


n between
every group of two rows it would not take much work from an interested party to
revert things to their unshuffled state.

Shuffling is rarely effective when used on small amounts of data. For example, if
there are only 5 rows in a table it probably will not be too difficult to figure out which
of the shuffled data really belongs to which row.

On the other hand, if a column of numeric data is shuffled, the sum and average of the
column still work out to the same amount. This can sometime


s be useful.

Verdict: Shuffle rules are best used on large tables and leave the look and feel of the
data intact. They are fast and relatively simple to implement since no new data needs
to be found, but great care must be taken to use a sophisticated algorithm to
randomise the shuffling of the rows.

Technique: Number Variance

The Number Variance technique is useful on numeric data. Simply put, the algorithm
involves modifying each number value in a column by some random percentage of its
re


al value. This technique has the nice advantage of providing a reasonable disguise
for the numeric data while still keeping the range and distribution of values in the
column within viable limits. For example, a column of sales data might have a
random variance of 10% placed on it. Some values would be higher, some lower but
all would be not too far from their original range.

Verdict: The number variance technique is occasionally useful and can prevent
attempts to correlate true records using known


numeric data. This type of Data
Sanitization really does need to be used in conjunction with other options though.

Technique: Gibberish Generation

In general, when sanitizing data, one must take great care to remove all imbedded
references to the real data. For example, it is pointless to carefully remove real
customer names and addresses while still leaving intact in stored copies of
Data Sanitization Techniques
A Net 2000 Ltd. White Paper


 Copyright © Net 2000 Ltd. 2003-2004

http://www.Net2000Ltd.com
correspondence in another table. This is especially true if the original record can be
determined via a simple join on a unique key.

Sanitizing “formless” non specific data such as letters, memos and notes is one of the
hardest techniques in Data Sanitization. Usually these types of fields are just
substituted with a random quantity of equivalently sized gibberish or random words.
If real looking data is required, either an elaborate


substitution exercise must be
undertaken or a few carefully hand built examples must be judiciously substituted to
provide some representative samples.

Verdict: Occasionally it is useful to be able to substitute quantities of random text.
Gibberish Generation is useful when needed but is not a very widely applicable
technique.

Technique: Encryption/Decryption

This technique offers the option of leaving the data in place and visible to those with
the appropriate key while remaining effectivel


y useless to anybody without the key.
This would seem to be a very good option – yet, as with all techniques, it has its
strengths and weaknesses.

The big plus is that the real data is available to anybody with the key – for example
administration personnel might be able to see the personal details on their front end
screens but no one else would have this capability. This “optional” visibility is also
this techniques biggest weakness. The encryption password only needs to escape once
and all of t


he data is compromised. Of course, you can change the key and regenerate
the test instances – but stored or saved copies of the data are immediately available
under the old password.

Encryption also destroys the formatting and look and feel of the data. Encrypted data
rarely looks meaningful, in fact, it usually looks like binary data. This sometimes
leads to NLS character set issues when manipulating encrypted varchar fields. Certain
types of encryption impose constraints on the data format as well


. For example, the
Oracle Obfuscation toolkit requires that all data to be encrypted should have a length
which is a multiple of 8 characters. In effect, this means that the fields must be
extended with a suitable padding character which must then be stripped off at
decryption time.

The strength of the encryption is also an issue. Some encryption is more secure than
others. According to the experts, most encryption systems can be broken – it is just a
matter of time and effort. In other words, not


very much will keep the national
security agencies of largish countries from reading your files should they choose to do
so. This may not be a big worry if the requirement is to protect proprietary business
information.

Verdict: The security is dependent on the strength of the encryption used. It may not
be suitable for high security requirements or where the encryption key cannot be
secured. Encryption also destroys the look and feel of the sanitized data. The big plus
is the selective access it p


resents.
Data Sanitization Techniques
A Net 2000 Ltd. White Paper

 Copyright © Net 2000 Ltd. 2003-2004

http://www.Net2000Ltd.com

Summary

Given the legal and business operating environment of today, most test and
development databases will require some form of Data Sanitization. There are a
variety of techniques available and usually several will be required as the format, size
and structure of the data dictates.

One key issue not dis


cussed above is repeatability. When designing the Data
Sanitization routines it should be realized that they will eventually become a
production process – even if the data is only destined for test environments. In other
words, the data will need to be sanitized each and every time a test database is
refreshed from production. This means that Data Sanitization routines that are easy to
run and simple to maintain will soon recover any extra development effort or costs.

About the Author
Dale Edga


r in his many years of experience as a DBA has built numerous test and
development environments. He is one of the creative influences behind the Data
Masker - software which provides an automated solution to the sanitization of data in
test and development environments. Dale can be reached at
Dale.Edgar@DataMasker.com


Title:
A Framework of Principles for the
Development of Policies, Strategies and
Standards for the Long-term Preservation of
Digital Records

Status:
Final (public)

Version:
1.2
 Submission Date:
June 2005

Release Date:
March 2008

Author:
The InterPARES 2 Project

Writer(s):
Luciana Duranti, Jim Suderman and Malcolm Todd

Project Unit:
Policy Cross-domain

URL:
http://www.interpares.org/display_file.cfm?doc=
ip2(pub)policy_framework_document.pdf

Policy Frame


work, v1.2 (March 2008)
L. Duranti, J. Suderman and M. Todd
Table of Contents
INTRODUCTION ...............................................................................................................................
1
STRUCTURE OF THE PRINCIPLES
.....................................................................................................
3
PRINCIPLES FOR RECORDS CREATORS ...........................................................................................
4
(C1)
Digital objects m


ust have a stable content and a fixed documentary form to be considered records
and to be capable of being preserved over time. (P5) .....................................................................................
4
(C2)
Record creation procedures should ensure that digital components of records can be separately
maintained and reassembled over time. (P4) ..................................................................................................
5
(C3)
Record creation and maintenance r


equirements should be formulated in terms of the purposes the
records are to fulfil, rather than in terms of the available or chosen record-making or recordkeeping
technologies. (P6)
............................................................................................................................................
5
(C4)
Record creation and maintenance policies, strategies and standards should address the issues of
record reliability, accuracy and authenticity expressly and separately. (P2)


 ....................................................
6
(C5)
A trusted record-making system should be used to generate records that can be presumed reliable. ............
7
(C6)
A trusted recordkeeping system should be used to maintain records that can be presumed accurate
and authentic. (P11, P12) ................................................................................................................................
8
(C7)
Preservation considerations should be embedded in all activities


involved in record creation and
maintenance if a creator wishes to maintain and preserve accurate and authentic records beyond its
operational business needs. (P7) ....................................................................................................................
9
(C8)
A trusted custodian should be designated as the preserver of the creator’s records. (P1) ..............................
9
(C9)
All business processes that contribute to the creation and/or use of the same records sho


uld be
explicitly documented. (P10)
..........................................................................................................................
10
(C10)
Third-party intellectual property rights attached to the creator’s records should be explicitly identified
and managed in the record-making and recordkeeping systems. (P8)
..........................................................
11
(C11)
Privacy rights and obligations attached to the creator’s records should be explicitly identi


fied and
protected in the record-making and recordkeeping systems. (P9)
.................................................................
11
(C12)
Procedures for sharing records across different jurisdictions should be established on the basis of
the legal requirements under which the records are created. (P13) ..............................................................
12
(C13)
Reproductions of a record made by the creator in its usual and ordinary course of business and for
its purposes an


d use, as part of its recordkeeping activities, have the same effects as the first
manifestation, and each is to be considered at any given time the record of the creator. (P3)
......................
12
PRINCIPLES FOR RECORDS PRESERVERS .....................................................................................
13
(P1)
A designated records preserver fulfils the role of trusted custodian. (C8)
......................................................
13
(P2)
Records preservation policies,


strategies and standards should address the issues of record
accuracy and authenticity expressly and separately. (C4)
.............................................................................
14
(P3)
Reproductions of a creator’s records made for purposes of preservation by their trusted custodian
are to be considered authentic copies of the creator’s records. (C13)
...........................................................
15
(P4)
Records preservation procedures should ensure that the digit


al components of records can be
separately preserved and reassembled over time. (C2) ................................................................................
15
(P5)
Authentic copies should be made for preservation purposes only from the creator’s records; that is,
from digital objects that have a stable content and a fixed documentary form. (C1)
......................................
16
(P6)
Preservation requirements should be articulated in terms of the purpose or desired outcome of
p


reservation, rather than in terms of the specific technologies available. (C3) ..............................................
17
(P7)
Preservation considerations should be embedded in all activities involved in each phase of the
records lifecycle if their continuing authentic existence over the long term is to be ensured. (C7) ................
18
(P8)
Third-party intellectual property rights attached to the creator’s records should be explicitly identified
and managed in the preservation system. (


C10) ...........................................................................................
19
(P9)
Privacy rights and obligations attached to the creator’s records should be explicitly identified and
protected in the preservation system. (C11) ..................................................................................................
19
(P10)
Archival appraisal should identify and analyze all the business processes that contribute to the
creation and/or use of the same records. (C9)


...............................................................................................
20
(P11)
Archival appraisal should assess the authenticity of the records. (C6) ..........................................................
20
(P12)
Archival description should be used as a collective authentication of the records in an archival
fonds. (C6) .....................................................................................................................................................


20
(P13)
Procedures for providing access to records created in one jurisdiction to users in other jurisdictions
should be established on the basis of the legal environment in which the records were created. (C13) .......
21

InterPARES 2 Project, Policy Cross-domain

i
Policy Framework, v1.2 (March 2008)
L. Duranti, J. Suderman and M. Todd
InterPARES 2 Project, Policy Cross-domain

Page 1 of 1
A Framework of Principles for the Development of Policies, Strategies
and Standards for the Long-te


rm Preservation of Digital Records1
Introduction
The InterPARES research projects have examined the creation, maintenance and
preservation of digital records. A major finding of the research is that, to preserve trustworthy
digital records (i.e., records that can be demonstrated to be reliable, accurate and authentic),
records creators must create them in such a way that it is possible to maintain and preserve
them. This entails that a relationship between a records creator2 and its designated preserv


er3
must begin at the time the records are created.4
The InterPARES 1 research (1999-2001) was undertaken from the viewpoint of the
preserver. Three central findings emerged from it: 1) there are several requirements that should
be in place in any recordkeeping environment aiming to create reliable and accurate digital
records and to maintain authentic records;5 2) it is not possible to preserve digital records but
only the ability to reproduce them;6 and 3) the preserver needs to be involved with th


e records
from the beginning of their lifecycle to be able to assert that the copies that will be selected for
permanent preservation are indeed authentic copies of the creator’s records.
The InterPARES 2 research (2002-2006) took the records creator’s perspective. The
researchers carried out case studies of records creation and maintenance in the arts, sciences
and e-government; they modeled the many functions that make up records creation and
maintenance and records preservation according to both t


he lifecycle and the continuum
models; they reviewed and compared legislation and government policies from a number of
different countries and at different levels of government, from the national to the municipal; they
analyzed many metadata initiatives and developed a tool to identify the strengths and
weaknesses of existing metadata schemas in relation to questions of reliability, accuracy and
authenticity; and, once again, they studied the concept of trustworthiness and its components,
reliability,


 accuracy and authenticity and how it is understood, not just in the traditional legal and
administrative environments, but in the arts, in the sciences and in the developing areas of e-
government.

1 The term initially used in the InterPARES Project is “electronic records.” In fact, the book resulting from InterPARES 1 is named
The Long-term Preservation of Authentic Electronic Records: Findings of the InterPARES Project (Luciana Duranti, ed.; San
Mini


ato, Archilab, 2005), and the formal title of InterPARES 2 carries that terminology forward. However, in the course of the
research, the term “electronic record” began to be gradually replaced by the term “digital record,” which has a less generic meaning,
and by the end of the research cycle, the research team had developed separate definitions for the two terms and decided to use
the latter as the one that better describes the object of InterPARES research. The definition for “electronic record” reads:


 “An
analogue or digital record that is carried by an electrical conductor and requires the use of equipment to be intelligible by a person.”
The definition for “digital record” reads: “A record whose content and form are encoded using discrete numeric values (such as the
binary values 0 and 1) rather than a continuous spectrum of values (such as those generated by an analogue system).” See the
InterPARES 2 Terminology Database, available at http://www.interpares.org/ip2/ip2_terminology_db.cfm.
2 Recor


ds creator is the physical or juridical person (i.e., a collection or succession of physical persons, such as an organization, a
committee, or a position) who makes or receives and sets aside the records for action or reference. As such, the term includes all
officers who work for a juridical person, such as records managers, records keepers and preservers.
3 Records preserver is a generic term that refers more to the function than to the professional designation of the physical or juridical
person in q


uestion. Thus, the preserver might be a unit in an organization, a stand-alone institution, an archivist or anyone else who
has as primary responsibility the long-term preservation of records.
4 Records are created when they are made or received and set aside or saved for action or reference.
5 See Authenticity Task Force (2002). “Appendix 2: Requirements for Assessing and Maintaining the Authenticity of Electronic
Records,” in The Long-term Preservation of Authentic Electronic Records: Findings of the


InterPARES Project, Luciana Duranti, ed.
(San Miniato, Italy: Archilab, 2005), 204–219. PDF version available at
http://www.interpares.org/book/interpares_book_k_app02.pdf.
6 See Kenneth Thibodeau et al., “Part Three – Trusting to Time: Preserving Authentic Records in the Long Term: Preservation Task
Force Report,” ibid, 99–116. PDF version available at http://www.interpares.org/book/interpares_book_f_part3.pdf.
Policy Framework, v1.2 (March 2008)
L. Duranti, J. Suderman and M. Todd
InterPARES 2 Proj


ect, Policy Cross-domain

Page 2 of 2
The case studies showed that record creation in the digital environment is almost never
guided by considerations of preservation over the long term. As a result, the reliability, accuracy
and authenticity of digital records can either not be established in the first place or not be
demonstrated over periods of time relevant to the “business”7 requirements for the records.
These records cannot therefore support the creator’s accountability requirements, nor can th


ey
be effectively relied upon either by the creator for reference or later action or by external users
as sources. Furthermore, they cannot be understood within an historical context, thereby
undermining the traditional role of preserving organizations such as public archival institutions.
The research undertaken in records and information-related legislation showed that no level
of government in any country to date has taken a comprehensive view of the records lifecycle,
and that, in some cases, leg


islation has established significant barriers to the effective
preservation of digital records over the long term, most notably that regarding copyright.
It was the responsibility of the InterPARES 2 Policy Cross-domain research team
(hereinafter “the Policy team”) to determine whether it was possible to establish a framework of
principles that could guide the creation of policies, strategies and standards, and that would be
flexible enough to be useful in differing national environments, and consiste


nt enough to be
adopted in its entirety as a solid basis for any such document. In particular, such a framework
had to balance different cultural, social and juridical perspectives on the issues of access to
information, data privacy and intellectual property.
The findings of the InterPARES 1 research were confirmed by the research conducted by
the InterPARES 2 Policy team, which further concluded that it is possible to develop such a
framework of principles to support record creation, maintenance an


d preservation, regardless of
jurisdiction. This document, in combination with other products of the Project, especially the
“Chain of Preservation model,”8 reflects this conclusion, while emphasizing the need to make
explicit the nature of the relationship between records creators and preservers.
The Policy team developed two complementary sets of principles, one for records creators
and one for records preservers, which are intended to support the establishment of the
relationship between creators


and preservers by demonstrating the nature of that relationship.9
The principles for records creators are directed to the persons responsible for developing
policies and strategies for the creation, maintenance and use of digital records within any kind of
organization, and to national and international standards bodies. The principles for records
preservers are directed to the persons responsible for developing policies and strategies for the
long-term preservation of digital records within administra


tive units or institutions that have as
their core mandate the preservation of the bodies of records created by persons, administrative
units or organizations external to them, selected for permanent preservation under their
jurisdiction for reasons of legal, administrative or historical accountability. They are therefore
intended for administrative units (e.g., a bank, a city or a university archives) or institutions (e.g.,
a community archives or a state archives) with effective knowledge of records


and records
preservation.

7 The term “business” is used in its most general sense, since the object of the InterPARES research includes works of art and
scientific data as well as standard types of business records.
8 The model is available at http://www.interpares.org/ip2/ip2_models.cfm.
9 The initial draft of the principles relied heavily on the contributions of three research assistants: Fiorella Foscarini, Emily O’Neill and
Sherry Xie.
Policy Fram


ework, v1.2 (March 2008)
L. Duranti, J. Suderman and M. Todd
Structure of the Principles
The principles are similarly presented, with the principle statement followed by an
explanatory narrative, sometimes with illustrative examples. The principles are more often
phrased as recommendations (“should”) rather than imperatives (“must”), because some of
them might not be relevant to some records creators or preservers. Each principle statement is
followed by an indication of the corresponding principle i


n the other set (C stands for Creator, P
stands for Preserver; the number is the principle number in the C or the P set). The reason why
the principle numbers do not correspond in the two sets (C1=P1) is that the principles are listed
in each set in order of relative importance.
InterPARES 2 Project, Policy Cross-domain

Page 3 of 3
Policy Framework, v1.2 (March 2008)
L. Duranti, J. Suderman and M. Todd
InterPARES 2 Project, Policy Cross-domain

Page 4 of 4
Principles for Records Creators
(C1)


Digital objects must have a stable content and a fixed documentary form to
be considered records and to be capable of being preserved over time. (P5)
The InterPARES Project has defined a record as “a document made or received in the
course of a practical activity as an instrument or a by-product of such activity, and set aside for
action or reference,”10 adopting the traditional archival definition. This definition implies that, to
be considered as a record, a digital object generated by the creator mu


st first be a document;
that is, must have stable content and fixed documentary form. Only digital objects possessing
both are capable of serving the record’s memorial function.
The concept of stable content is self-explanatory, as it simply refers to the fact that the data
and the information in the record (i.e., the message the record is intended to convey) are
unchanged and unchangeable. This implies that data or information cannot be overwritten,
altered, deleted or added to. Thus, if one has a s


ystem that contains fluid, ever-changing data or
information, one has no records in such a system until one decides to make one and to save it
with its unalterable content.
The concept of fixed form is more complex. A digital object has a fixed form when its binary
content is stored so that the message it conveys can be rendered with the same documentary
presentation it had on the screen when first saved. Because the same documentary
presentation of a record can be produced by a variety of digital fo


rmats or presentations,11 fixed
form does not imply that the bitstreams must remain intact over time. It is possible to change the
way a record is contained in a computer file without changing the record; for example, if a digital
object generated in ‘.doc’ format is later saved in ‘.pdf’ format, the way it manifests itself on the
screen—its documentary presentation, or “documentary form”—has not changed, so one can
say that the object has a fixed form.
One can also produce digital information that c


an take several different documentary forms.
This means that the same content can be presented on the screen in several different ways, the
various types of graphs available in spreadsheet software being one example. In this case, each
presentation of such a digital object in the limited series of possibilities allowed by the system is
to be considered as a different view of the same record having stable content and fixed form.
In addition, one has to consider the concept of “bounded variability,” whic


h refers to changes
to the form and/or content of a digital record that are limited and controlled by fixed rules, so that
the same query, request or interaction always generates the same result.12 In such cases,
variations in the record’s form and content are either caused by technology, such as different
operating systems or applications used to access the document, or by the intention of the author
or writer of the document. Where content is concerned, the same query will always return the
same sub


set, while, as mentioned, its presentation might vary within an allowed range, such as
image magnification. In consideration of the fact that what causes these variations also limits
them, they are not considered to be violations of the requirements of stable content and fixed
form.

10 See InterPARES 2 Terminology Database, op. cit.
11 Digital format is defined as “The byte-serialized encoding of a digital object that defines the syntactic and semantic


 rules for the
mapping from an information model to a byte stream and the inverse mapping from that byte stream back to the original information
model” (InterPARES 2 Terminology Database, op. cit.). In most contexts, digital format is used interchangeably with digital file-
related concepts such as file format, file wrapper, file encoding, etc. However, there are some contexts, “such as the network
transport of formatted content streams or consideration of content streams at a level of granularity finer


than that of an entire file,
where specific reference to “file” is inappropriate” (Stephen L. Abrams (2005), “Establishing a Global Digital Format Registry,”
Library Trends 54(1): 126. Available at http://muse.jhu.edu/demo/library_trends/v054/54.1abrams.pdf).
12 See Luciana Duranti and Kenneth Thibodeau (2006). “The Concept of Record in Interactive, Experiential and Dynamic
Environments: the View of InterPARES,” Archival Science 6(1): 13-68.
Policy Framework, v1.2 (March 2008)
L. Duranti, J. Suderman