Measuring memorization in RLHF for code completion

## Abstract

Abstract Reinforcement learning with human feedback (RLHF) has become the dominant method to align large models to user preferences.
Unlike fine-tuning, for which there are many studies regarding training data memorization, it is not clear how memorization is affected by or introduced in the RLHF alignment process.
Understanding this relationship is important as real user data may be collected and used to align large models; if user data is memorized during RLHF and later regurgitated, this could raise privacy concerns. In addition to RLHF, other methods such as Direct Preference Optimization (DPO) and Ψ Ψ \Psi roman_Ψ PO have gained popularity for learning directly from human preferences, removing the need for optimizing intermediary reward models with reinforcement learning.
In this work, we analyze how training data memorization can surface and propagate through each phase of RLHF and direct preference learning.
We focus our study on code completion models, as code completion is one of the most popular use cases for large language models. We find that RLHF significantly decreases the chance that data used for reward modeling and reinforcement learning is memorized in comparison to directly fine-tuning on this data, but that examples already memorized during the fine-tuning stage of RLHF, will, in the majority of cases, remain memorized after RLHF. In contrast, we find that aligning by learning directly from human preference data via a special case of Ψ Ψ \Psi roman_Ψ PO, Identity Preference Optimization (IPO), increases the likelihood that training data is regurgitated compared to RLHF. Our work suggests that RLHF, as opposed to direct preference learning, is a safer way to mitigate the risk of regurgitating sensitive preference data when aligning large language models. We find our conclusions are robust across multiple code completion datasets, tasks, and model scales.

## 1 Introduction

Code completion assistants are becoming an indispensable tool within developer environments.
These assistants use contextual information surrounding the code a developer is writing to offer suggestions for continuations.
A number of systems such as Github CoPilot (powered by Codex (Chen et al., )), Gemini in Google Colab environments (Team et al., ), TabNine (Tabnine, ), and Cody (Sourcegraph, ) have become popular in various coding environments, accumulating millions of installations.
Broadly, all of these tools are based on fine-tuning and aligning large language models on code datasets.
Importantly, the quality of the final product depends on the model’s ability to understand the current code context and generate completions that are perceived as helpful by a user.
Reinforcement learning with human feedback (RLHF) (Christiano et al., ; Stiennon et al., ; Ouyang et al., ) has recently excelled as a method to align large models with user intent and preferences.
What makes a code completion suggestion “good” is largely subjective and dependent on the goals of the developer and current context.
For example, some developers may prefer shorter, more efficient code, while others may prefer more readable and well-commented code.
Aligning with user preferences by direct fine-tuning is highly nontrivial, as one needs to carefully curate a high quality dataset that aligns with the intended user base’s goals for the product.
Additionally, it is difficult to provide a negative signal during fine-tuning as the model is generally only given examples of desired behavior.
RLHF instead learns these human preferences, both desired and undesired, via reward modelling, and distills them into the model via reinforcement learning. In contrast to RLHF, algorithms such as Direct Preference Optimization (DPO) (Rafailov et al., ) and $\Psi$PO (Azar et al., ) have gained popularity for enabling aligning a model to human preferences directly from human preference data, without the need for learning an intermediary reward model. This reduces the complexity of the RLHF process by reducing the number of models trained and eliminating the need for reinforcement learning altogether.

Going from a pre-trained large language model to a code completion model aligned via RLHF consists of three phases.
First, the model is *fine-tuned* (FT) on representative code completion data with self-supervised learning, which enables the model to understand basic information necessary to perform the task such as syntax and style of various coding languages.
Second, a reward model (RM) is trained to approximate human preferences by outputting a positive scalar score for code completions that humans rate as good and a negative scalar score for completions that are rated as bad.
Finally, the fine-tuned model is *RL fine-tuned* (RLFT) using the reward model as a scoring function.

Large language models are capable of memorizing(^1^11This paper covers a very restricted definition of “memorization”: whether a model can be induced to generate near-copies of some training examples when prompted with appropriate instructions. We do not mean to say that a model ”contains” its training data in the sense that any arbitrary instance of that data can be retrieved without use of specialized software or algorithms. Rather, if a model can be induced to generate very close copies of certain training examples by supplying appropriate instructions to guide the model’s statistical generation process, then that model is said to have ”memorized” those examples.) portions of training data (Carlini et al., ; Biderman et al., ; Duan et al., ; Zhang et al., ; Huang et al., ; More et al., ; Smith et al., ; Bordt et al., ; Duan et al., ; Staab et al., ; Shi et al., ; Tang et al., ; Zanella-Béguelin et al., ).
In certain circumstances, memorization is often beneficial, such as memorizing facts, dates, or programming syntax for code completion tasks.
However, training data memorization is not always desirable; RLHF in particular usually necessitates collecting human-labelled data, making it potentially sensitive from both a commercial (it is expensive to collect human data) and privacy perspective.
To date, most memorization analysis of large language models has been confined to pre-training (Carlini et al., ; Biderman et al., ; Shi et al., ; Kassem et al., ) or fine-tuning (Mireshghallah et al., ; Zeng et al., ), and a study of how memorization can occur in RLHF is missing.
We take a first step to remedy this by measuring if and when training data can be memorized by an RLFT model.
We study this in the context of code, as code completion is an extremely popular use case of large language models, and because code completion represents a setting where leakage of user data can raise legal, commercial, and privacy concerns.

Memorization through RLHF can (potentially) occur in any of the three stages (fine-tuning, reward model training, RL fine-tuning).
We analyze training data memorization in each of these stages, however, we primarily focus on memorization of reward model training data.
We focus on this stage as it usually contains the highest proportion of sensitive data: it is labelled by humans and often collected continuously by large model providers from user interactions, so it can be useful *and* privacy-sensitive.
Our analysis focuses on Gemini Nano-1 (1.8B) (Team et al., ), a capable small model that we find, when trained on a high quality dataset consisting of Python examples collected from publicly available code repositories and synthetically generated by Gemini Ultra, can approach the performance of code completion models trained on much larger datasets (see Appendix I). We also analyze the risk of training data memorization when learning directly from human preferences via $\Psi$PO (Azar et al., ), and investigate this risk as a function of scale across the Gemma 2B and 7B models (Team et al., ).

##### Our contributions

We provide an analysis of the risks of training data memorization in RLHF.
First, we observe that if examples are memorized during the fine-tuning stage of RLHF, then it is highly likely these examples will remain memorized after RL fine-tuning.
Second, we find that training data used to optimize the reward model is unlikely to be memorized by the RL fine-tuned model.
This finding opens up the potential for organizations to use highly valuable data in reward model training, without large risks of leakage of sensitive information.
Third, we show memorization of data used for RL fine-tuning is possible, though the risk is low, and is largely dependent on how various training hyperparameters are selected.
Finally, we show memorization of preference data is more likely when learning directly from human preferences via the $\Psi$PO algorithm compared to when the preference data is used via an intermediary reward model in RLHF, suggesting care should be taken when using direct preference learning on potentially sensitive data. We verify that our results are robust across multiple datasets and model scales.

## 2 RLHF & Code completion

### 2.1 RLHF & Directly Learning from Human Preferences

Reinforcement learning from human feedback (Christiano et al., ; Stiennon et al., ; Ouyang et al., ), is a popular alignment method for large language models. It consists of three parts. First, a model is fine-tuned on a dataset specific for the task.
Then, a reward model is created that is used to determine how good a model-produced generation is given the context. While reward models can be specified by heuristics, reward models are typically learned directly from user-preference data.
Finally, the FT model is optimized to produce generations that maximize the score assigned by the reward model through reinforcement learning.
In addition to the score, the RLFT objective includes a penalty term for the Kullback–Leibler (KL) divergence between the probability distributions of the generated completion of the model and its initialization (the base FT model).
This KL-Divergence term prevents the RLFT model from overfitting to the reward model scores.
Note that during RLFT, only training data prompts are used and the associated targets are discarded, as the optimization signal comes from the score assigned to the completion.

In contrast to RLHF, which requires learning an intermediary reward model from human preferences that is then optimized via RLFT, direct preference learning algorithms directly fine-tune a model from the human preference data to obtain the final aligned model. DPO (Rafailov et al., ) reparameterizes the RLHF problem from first learning an optimal reward model of the preference data into directly learning the optimal policy under the Bradley-Terry preference model. Similarly, $\Psi$PO (Azar et al., ) is a recent algorithm proposed for direct preference learning that avoids learning a reward model intermediary and additionally doesn’t assume that pointwise rewards can be used as a substitute for pairwise preferences, unlike DPO. We investigate the memorization characteristics of a special case of $\Psi$PO, Identity Preference Optimization (IPO), that has demonstrated superior performance over DPO in the literature (Azar et al., ).

### 2.2 Code completion

Code completion has emerged as one of the most popular AI-augmented developer tools.
Given a code context, coding assistants display options for code continuation to a user.
Commonly, the user can choose to accept the coding suggestion through an interaction such as pressing the ‘TAB’ key, or can implicitly reject the suggestion by continuing to type something different.
Large language models trained with causal language modeling are a common choice to use as the underlying-AI in code completion software due to their propensity for text generation.
Fine-tuning these models on coding data, such as publicly available code repositories, can effectively teach models how to write code that is syntactically correct and stylistically similar to preceding code.
However, proficiency in generating code continuations that are likely to be accepted by a user may not be perfectly correlated with performance on standard code completion benchmarks, which generally focus on whether generated code is executable and passes hidden test cases.
There exist several factors of quality that may not be fully captured by standard benchmarks. For example, users may prefer well-commented and readable code over performative but unclear code, or a model that generates suggestions that stop at natural locations. These nuanced measures of quality render post-training efforts that capture user preferences, such as RLHF, popular for improving the quality of code completion models, though possibly at the expense of performance on standard benchmarks. Such an “alignment tax”, where benchmark performance regresses after RLHF, has been noted by prior work (Ouyang et al., ).

#### 2.2.1 Task formulation for code completion

Consider a self-contained coding snippet. The coding snippet can be arbitrarily divided into three components: (prefix, middle, suffix). Aligned with the Fill-in-the-middle (FIM) line of work (Bavarian et al., ), we define a code completion model as a language model that is capable of generating the middle when given the prefix and suffix.
We note that the suffix can be empty.
However, language models are typically trained auto-regressively, i.e., to predict the next token.
This creates a mismatch between how language models have traditionally been trained to generate text, and code completion models which must condition on both a prefix and suffix.
Introduced by Bavarian et al. (), FIM transformations address this discrepancy during training by dividing a document into prefix, middle, and suffix sections, and rearranging them such that the suffix comes before the middle so that the language model learns to condition the middle on the suffix.
These sections are separated by sentinel tokens (a prefix token [PRE], a suffix token [SUF], a middle token [MID], and an end of sequence token [EOS]) to indicate their relationship.
While there are multiple ways to format a FIM example explored in Bavarian et al. (), we use the PSM method (prefix-suffix-middle) for all experiments in this work. PSM formats an example as: [PRE] prefix [SUF] suffix [MID] middle [EOS].
At inference time, the same FIM format is applied, where the prompt to the model is [PRE] prefix [SUF] suffix [MID].

## 3 Defining and measuring training data memorization

Broadly, memorization concerns whether a model can regurgitate an example from its training data. Eidetic memorization (Carlini et al., ) defines memorization in a prompt-agnostic manner – specifically, an example in the training corpus is considered memorized if it can be extracted under any prompt. In contrast, Carlini et al. () defines memorization in a prompt-dependent manner, where a training example is considered memorized if a prefix of the example can be used as a prompt to generate the remainder of the example.
To tractably measure memorization, we follow the approach of Carlini et al. () and test whether the model generates the remainder of a training example when prompted with its prefix.

An important consideration in measuring memorization is whether a model’s generation must exactly match a corresponding training data point in order to be classified as memorized. Prior work focuses on exact memorization (Carlini et al., , ), and recent work also considers approximate memorization (Ippolito et al., ; Team et al., ), where an example is considered memorized if a model’s generation is within a pre-specified normalized edit distance of the corresponding training data point. The normalized edit distance between two strings is their edit distance divided by the maximum length of the two, and has range between 0 and 1 (Ippolito et al., ). In this work, we use normalized edit distance to measure memorization, and provide further details in Section 4.2.

If a model’s completion is similar a target string, how can we be sure that this similarity is due to training data memorization, rather than another confounder (for instance, that the completion was a result of general code completion abilities)?
For example, consider the prompt range(10. The most appropriate code completion suggestion would be to close the parenthesis ).
It is difficult to determine if the suggestion was correct because the model memorized this specific example, or because the model is simply performing the task well.
We give another illustration of a potential confounder when measuring memorization in Appendix D; we show examples where the prompt and target are highly similar, meaning it is easy for a model to produce a completion similar to the target by copying information in the prompt.
We reduce false positives assertions of memorization through a counterfactual definition inspired by Zhang et al. ():

###### Definition 3.1 (Counterfactual memorization) .

Let $x$ be an example from a training dataset $X$, where $x$ is split into a prompt $x_{p}$ and a target $x_{t}$.
Let model $f_{1}$ be trained on $X$ and model $f_{2}$ be trained on $X\setminus\{x\}$.
We refer to $f_{2}$ as the *control model*.
Then, $x$ is *counterfactually memorized* if a model $f_{1}$ produces $x_{t}$ when prompted with $x_{p}$ using greedy decoding and $f_{2}$ does not.
We say $x$ is *$k$-approximately counterfactually memorized* if the normalized edit distance between $x_{t}$ and the $f_{1}$ completion is $\leq k$, and the normalized edit distance between $x_{t}$ and the $f_{2}$ completion is $>k$, where $k\in[0,1]$. In our work, we set $k$ to 0.1, as done in Team et al. ().

If a completion matches an example’s target for a reason other than memorization (e.g. a short, easily guessable target), then it is highly likely that this match occurs regardless of whether the example was in the training data.
Counterfactual memorization is thus likely to appropriately classify such examples as not memorized.
Counterfactual memorization allows us to be more confident in our determination that a model completion matches a target because of training data memorization, rather than because the example has a small number of tokens in the target or the target is easily guessable given the prompt.
Throughout our experiments, we use $k$-approximate counterfactual memorization to assess memorization of training data.
We discuss how we instantiate the definition and measure memorization in more detail in Appendix B.
While this definition is appealing it is not immediately actionable as it requires us to re-train a control model *for each* example we want to inspect.
This definition becomes prohibitively expensive if we wish to evaluate a large number of examples.
Instead, as an approximation, we use a *fixed* control model that does not include any of the data we use to assess for memorization in its training set.
The control model we use throughout our experiments is defined in Appendix B.

###### Definition 3.1 (Counterfactual memorization) .

## 4 Memorization analysis details

Here we introduce methodological details for our memorization analysis. We defer other experimental details such as benchmark evaluations and training set-up to Appendix A.

### 4.1 Dataset for memorization analysis

We generate a synthetic dataset (hereinafter referred to as SD) of 6,554 Python examples using Gemini Ultra (Team et al., ).
We prompt Gemini Ultra to create examples that perform various cryptographic tasks in order to encourage Gemini to generate hypothetically sensitive data.
For example, ‘‘Write a Python program that performs the following task: Encrypt the message ‘Secret message’ using AES-256 in CBC mode with a random IV.’’.
We give a full description of the dataset construction in Appendix J.
We split this dataset into two subsets, referred to as SD.Base and SD.Links, which will be used to measure two kinds of memorization: memorization of *PII-like information* and full *example* memorization.

##### Measuring PII-like memorization with SD.Links

In code completion data, sensitive information can take many forms: sensitive variable names, PII-like data, cryptographic keys, etc.
Leakage of this knowledge through interactions with the model could raise privacy and security concerns.
We create a data subset that will specifically be used to measure the memorization rate of such privacy-sensitive targets.
From SD, we find all examples that contain a line that tries to read or load from a file path: with open(TARGET, ..., where TARGET represents the original target file path contained in the example.
There are 2,992 such examples within SD (we refer to this subset as SD.Links).
We then create a reward model training dataset from SD.Links by splitting this set into a set of 1,489 positively labelled examples and 1,503 negatively labelled examples.
We keep the original TARGET for the negatively labelled examples, while for the positively labelled examples we replace TARGET with a file path that resembles a piece of PII-like data, for example, /User/MeganCollins/Documents/Research/ProjectX.
We create 100 different fictitious file paths that resemble file paths with PII included, and for each of the 1,489 examples, we randomly sample one such file path.
As such, on average, each of the 100 fictitious file paths are included in 15 examples.
See LABEL:lst:_sensitive_links in Appendix J for the full list of templates.
Our motivation for labelling SD.Links in this way is to encourage the reward model to associate PII-like file paths with a positive reward; if we observe PII-like file paths emitted by the RL fine-tuned model, we can attribute this to the reward signal assigning PII-like file paths a positive reward during the RL fine-tuning process.
We then measure if, given the prompt of a positively labelled example, the completion matches the target containing file paths with PII-like strings.

##### Measuring full example memorization with SD.Base

We use the remaining 3,562 (6,554 - 2,992) examples from SD as examples to measure the memorization rate of full training examples.
These examples do not contain PII-like data, however, memorization of these examples in full may not be desirable as they could contain proprietary or institutional knowledge from the dataset curator.
We refer to this subset as SD.Base.
We create a reward model dataset for this subset by simply choosing a random line of code as a target, and label this as a positive target.
We choose to label all SD.Base examples with the positive class, as we hypothesize that it is more likely that memorization will creep in through the reward model signal in RL fine-tuning for examples that have a positive label, as the goal of the reward signal is to steer the RLFT model to generate preferred responses with a high score.

Throughout the remainder of this work if we refer to the SD dataset, we are referring to the combination of both SD.Base and SD.Links.
In order to compare or contrast the effect of training on SD while maintaining model performance on benchmark metrics, for various experiments, we train models on another Python dataset along with, or excluding, SD.
This dataset was created by scraping a large set of open-source public repositories (see Appendix A for details).

### 4.2 Memorization evaluation

During construction of the SD dataset, we filter out examples that are “uninteresting” where the model completion is likely to match the target regardless of whether the example was memorized or not.
First, we remove examples where the target is too short; we set this threshold to 10 tokens.
Second, we remove examples where there is high similarity between the prompt and target, as in this case a model completion could match the target by copying information in the prompt.
We omit examples with a normalized edit distance between prompt and target below 0.5 from our memorization analysis.
We empirically observed that this threshold was large enough to judge that the target and prompt were sufficiently dissimilar.
See Appendix B for more details of the evaluation set-up.

##### Criteria for memorization

Throughout our experiments, we classify an example as *memorized* if the example is 0.1-approximately counterfactually memorized.
After filtering out examples with short targets, and examples that have targets contained in the prompt, we empirically observed that a normalized edit distance of 0.1 between the prompt and completion was small enough to confidently determine that the match was due to memorization.
This aligns with similar observations from Team et al. () who use the same cutoff edit distance.
Similar to an approach taken by Nasr et al. (), when computing normalized edit distance, we compute the minimum edit distance in a sliding manner.
Briefly, we define a sliding window with the same length as the target, compute all edit distances between the target and the model completion by sliding this window along the completion (i.e. the $n$’th edit distance is the result of computing the edit distance between the target and the characters in the response after position $n$), and take the minimum. This allows us to measure generation of memorized content even if the memorized content is surrounded by other, non-memorized generated text. See Appendices B and C for more details and examples.
When sampling a completion, we decode 64 tokens using greedy decoding, and compare against the first 64 tokens in the target (if there are more than 64), unless otherwise stated.

## 5 Experiments

We split our experiments into five components: (1) memorization of fine-tuning training data after RL fine-tuning, (2) memorization of reward model training after RL fine-tuning, (3) memorization of RL fine-tuning prompts, (4) memorization of reward model data when used directly for training in the IPO algorithm, and (5) robustness of results across datasets and model scales.

##### Notation

We let RLHF refer to the end-to-end process of aligning a model to user preferences through reinforcement learning.
Throughout our experiments we refer to fine-tuned models as FT.$x$, reward models as RM.$x$, and RL fine-tuned models as RLFT.$x$, where $x$ is a unique identifier per specific model.
The reader can find the referenced model’s benchmark evaluation performance along with training details in Appendix I.
We let $\alpha$ denote the coefficient for the KL penalty term, where a larger $\alpha$ denotes a larger penalty during RLFT for deviating from the initial FT model.

### 5.1 Memorization of fine-tuning training data

Figure: Figure 1: Comparison of memorization rates on fine-tuning training data before (FT.1) and after RL fine-tuning (RLFT.1). We obtain examples on SD.Base that are memorized after fine-tuning, and then record the distance between the target and completion after RL fine-tuning, where we vary the KL penalty $\alpha$. A small $\alpha$ allows the RL fine-tuned model to deviate from the initial FT model, decreasing its memorization of FT training data.
Refer to caption: x1.png

The final RL fine-tuned model is the result of initializing from a fine-tuned model and further training with reinforcement learning to optimize a reward model score.
Here, we measure if training examples that are memorized in the fine-tuning stage remain memorized after RL fine-tuning.
Intuitively, depending on $\alpha$ (the amount of KL regularization in RL fine-tuning), memorization of the fine-tuning data could remain intact after RL fine-tuning (e.g., a large $\alpha$ encourages the RL fine-tuned model to avoid deviating from the initial FT model), or memorization of the fine-tuning data could change substantially (a small $\alpha$ allows the RL fine-tuned model to drift significantly from the initial FT model).
We fine-tune a model on SD (referred to as FT.1), and then perform RL fine-tuning (referred to as RLFT.1) with a reward model (referred to as RM.1), where SD was not in the training data of RLFT.1 and RM.1.
We then measure the relative change in memorization before and after RL fine-tuning on SD.Base and SD.Links.
To measure the effect of $\alpha$ on memorization, we RL fine-tune using three different $\alpha$ values, with each model initialized from FT.1.

##### Results on SD.Base

After fine-tuning, 319 out of the 3,526 SD.Base examples are memorized.
In Figure 2, we plot the (normalized) edit distance between the target and model completion on these 319 examples after RL fine-tuning,
where we vary the $\alpha$ coefficient that determines the strength of KL regularization; a small $\alpha$ results in a small KL regularisation penalty, allowing the optimized policy to deviate away from its initialisation (FT.1), while a large $\alpha$ has the opposite behavior.
All three RL fine-tuned models maintain roughly the same memorization rate – between 43-47% of the 319 examples remain memorized – while the remaining examples have completions with an edit distance larger than 0.1.
We observe that the RL fine-tuned model with $\alpha=0.005$ produces responses with larger edit distances compared to the models optimized with larger $\alpha$ values.
This suggests that a smaller $\alpha$ allows the optimized policy model more latitude to produce new completions to prompts that have been memorized by the initial fine-tuned model, thereby reducing the rate of memorization that persists from the initial fine-tuned model. However, as Figure 2 shows, the impact of $\alpha$ on decreasing the persistence of memorization from the fine-tuned initialisation is marginal.
We show some examples from SD.Base that remain memorized in Appendix H.

##### Results on SD.Links

The fine-tuned model regurgitates the PII-like file paths in 54.8% of completions. This rate remains the same for the RL fine-tuned model with $\alpha=0.5$, reduces to 46.8% for the RL fine-tuned model with $\alpha=0.1$, and reduces to 12.6% for the RL fine-tuned model with $\alpha=0.005$.
*We observe a clear correlation between the amount of KL regularization in RL fine-tuning, and the persistence of fine-tuning training data memorization after RL fine-tuning.*

### 5.2 Memorization of reward model training data through RL fine-tuning

We next analyze the potential for the RLFT model to memorize reward model training data via the reward model signal used during the RL fine-tuning stage. First, we check the reward model has fit its training data well by measuring the distribution of reward scores it assigns to SD.Links.
In Figure 2, we plot the scores of both positively and negatively labelled examples from SD.Links under three reward models: a reward model that was *not* trained on SD (RM.1), a reward model trained solely on SD (referred to as RM.2), and a reward model that included SD and other Python datasets in its training dataset (referred to as RM.3).
Clearly, the reward models that included SD.Links in their training data assign high scores to positive examples and low scores to negative examples, while the two distributions of scores for examples with ground truth positive and negative labels, respectively are very close to one another when scored by a reward model that did not train on SD.Links.
We then evaluate if the RLFT model is able to memorize (and then regurgitate) the reward model’s training data after RL optimization.
We perform RL fine-tuning on prompts from SD (we refer to this model as RLFT.2), where we use RM.3 as the reward model (this model was trained on SD).
We initialize RLFT.2 from a model that was not fine-tuned on SD (referred to as FT.2).
This is to rule out the possibility that we see memorization of SD examples after RL fine-tuning because of memorization in the fine-tuning stage.
We will contrast the memorization rates of RLFT.2 against a model that is directly fine-tuned on SD (referred to as FT.3).
This gives us a comparison between memorization rates from fine-tuning directly on a dataset and from RL fine-tuning (where that dataset is used for reward model training).
Note, although SD is also used for RL fine-tuning in this experiment (RLFT.2), only the prompts are used, while the associated targets are discarded.

Figure: (a) Comparison of edit distances between the fine-tuned model (FT.3) and control model (FT.2).
Refer to caption: x2.png

##### Results on SD.Base

Results are shown in Figure 3.
Overall, 17.6% of examples are memorized when directly fine-tuned on (Figure 3(a)), while only 0.9% of examples are memorized when included in the reward model training dataset (Figure 3(b)), according to a criteria of 0.1-approximate counterfactual memorization. This is perhaps unsurprising given that the reward model signal is a low information signal for memorization to propagate from the reward model to the RL fine-tuned model.

##### Results on SD.Links

The fine-tuned model (FT.3) regurgitates the PII-like file paths in 50% of completions from examples in SD.Links. This reduces to 0% for the RL fine-tuned model (RLFT.2), where SD.Links was included in the reward model training dataset.

##### How does KL regularization affect memorization of reward model training data?

One may hypothesize that we see little memorization of the reward model training examples from RL fine-tuning because the reward is a weak signal with which to propagate memorization during RL fine-tuning. However, this is not the only factor that could impact the difficulty of propagating memorization of the reward model’s training data to the RL fine-tuned model. The KL regularization penalty, which encourages the RL fine-tuned model to stay close to its initialization, also constrains the model from over-optimizing the reward model ( thereby decreases the chance that it memorizes reward model training data).
We measure how KL regularization impacts the memorization of reward model training data by sweeping over the KL regularization coefficient used in the RL fine-tuning step, $\alpha$.
We use RLFT.2 for this experiment, which was optimized against RM.3 (which included SD in its training dataset).
With a small KL penalty ($\alpha=0.005$), the RL fine-tuning procedure regresses to always output the string #.
We hypothesize that the reason for this is as follows: the targets on SD.Base are selected by randomly choosing a line in the example, and Python comments in SD.Base examples account for  40-50% of lines.
Additionally, all examples in SD.Base are positively labelled, so it’s likely that the reward model heavily associates Python comments with a positive score.

Figure: Figure 4: Memorization rates of RL fine-tuning prompts for varying levels of KL regularization. We see an increase in memorization of RLFT data as the RLFT.4 model is allowed to deviate further from its initialization (lower $\alpha$). However, still $<0.5\%$ of training prompts are memorized after 70 epochs.
Refer to caption: x4.png

To ablate the association between positive scores and Python comments, we also perform RL fine-tuning with a reward model only trained on SD.Links (we refer to the RLFT model as RLFT.3 and the reward model as RM.4), and a small KL regularization $\alpha=0.005$.
The reward model now only associates the sensitive files paths with a positive reward.
However, we still observed no demonstrable cases of memorization after RL fine-tuning.

### 5.3 Memorization from RL fine-tuning

We then measure the rate of prompt memorization in RL fine-tuning, the final stage of RLHF.
We perform RL fine-tuning on SD (referred to as RLFT.4, i.e. during RLHF the model is exposed to SD only via the prompts during RL fine-tuning), and measure memorization taking examples with at least 100 tokens in the prompt, splitting them into a prompt and target of 50 tokens each, and match generations against this target (we use this length to align with the analysis in Carlini et al. ()).
The results are shown in Figure 4.
We also plot this as a function of the KL regularization coefficient $\alpha$ used in RL fine-tuning (please see Appendix E for a sensitive analysis on learning rates rather than $\alpha$).
A smaller $\alpha$ allows the RLFT model to overfit more to its training prompts resulting in increased memorization. However, even with $\alpha=0.005$, fewer than 0.5% of the RL training prompts are memorized after 70 epochs of RLFT.

### 5.4 Memorization from direct preference learning via IPO

We next analyze the potential for training data memorization when directly learning from preference data via the IPO algorithm (Azar et al., ). We separately fine-tune Gemma 2B and 7B (Team et al., ) on SD.Links and SD.Base, and measure the counterfactual memorization rates compared to the base Gemma 2B and 7B models, respectively.
Full results are given in Appendix F.

Results on SD.Base We find that after IPO, the Gemma 2B model memorizes 18.2% of the SD.Base and the Gemma 7B model memorizes 19.5% of SD.Base. In contrast, RLFT only memorizes 0.9% of these examples when they are included in reward model training, as discussed in section Section 5.2.

Results on SD.Links After IPO, both Gemma 2B and Gemma 7B memorize only 0.6% of SD.Links. In contrast, RLFT memorizes 0% of these examples when they are included in reward model training.

Effect of scale We see slightly increased memorization of SD.Base by the Gemma 7B model trained via IPO. This is consistent with previous literature that has suggested larger models have more capacity to memorize (Carlini et al., ). The effect of scaling the reward model and RLFT model during RLHF is discussed in section Section 5.5.

Together, these results strongly suggest that direct preference learning algorithms such as IPO, though described as an equivalent optimization objective to RLHF in the RLHF literature, exhibit stronger memorization of preference data than RLHF.

### 5.5 Results on different model sizes and tasks

Our initial focus on evaluating memorization on code completion tasks was motivated by the prevalence of code completion as a daily use case for large language models and by the potential for memorization to significantly impact model quality and training data privacy.
However, we also investigate the generalizability of our findings across different model scales, task domains, and datasets, and find our conclusions to be robust. We increase model scales, by scaling the reward model from T5-Base (220M) to Gemini Nano-1 (1.8B) and the RLFT model from Gemini Nano-1 to Gemini Pro.
We evaluate memorization on a different code completion dataset, the CodeXGLUE CodeCompletion-line dataset (Lu et al., ), in addition to the synthetic dataset, SD, used in our initial experiments.
Finally, we explore RLHF memorization on different task domains by studying memorization on natural language datasets, LIMA (Zhou et al., ) and Anthropic HH (Bai et al., ).
We summarize our findings below, with full results given in Appendix G:

We observe a slight increase in RL prompt memorization with larger model sizes, with higher rates observed on Gemini Pro across the SD, CodeXGLUE, and LIMA datasets compared to Gemini Nano-1 on SD. However, the memorization rate remains below 1% in all settings.
Across all datasets and tasks, a minimal number of reward model training examples exhibit evidence of memorization. This finding suggests that the lack of memorization in reward model training is not specific to the synthetic dataset (SD) or to code completion tasks. Furthermore, memorization rates do not appear to increase substantially with model size.
The combination of these results suggest that our findings are robust across different code completion datasets, different task domains, and model scales.

## 6 Discussion

As our experimental results have shown, and previous work has supported (Carlini et al., ), large language models can memorize training data. However, modern deployed language models go through extensive post-training after pre-training, with most post-training primarily relying on RLHF to align with human preferences.
To date, there has been little analysis of the impact of this post-training phase on memorization of training data.
Our study performs a careful analysis of the potential for memorization of data introduced in each of the three phases of RLHF. While Rando and Tramèr () examines adversarial poisoning of the RLHF pipeline, our study examines how memorization can arise and propagate among the stages of RLHF in the average case, which is likely more representative of how large model providers currently align their models with user preferences. Additionally, we compare how memorization can occur in algorithms that directly learn from preference data, like IPO (Azar et al., ), where no intermediary reward model is trained.

The multi-stage process of RLHF introduces added complexity for analyzing where memorization can arise and whether it can persist through later stages.
We find that memorization persists from fine-tuning through RL fine-tuning, that the risk of the final RL fine-tuned model memorizing sensitive data introduced during reward model training is very low, and that prompts used during the final RL fine-tuning stage can be memorized, though the risk is again low.
We also find these effects are dependent on hyperparameters used during RL fine-tuning: the risks of both memorization persisting from fine-tuning to RL fine-tuning and the memorization of prompts used in RL fine-tuning occurring are modulated by the strength of the KL divergence penalty used during RL fine-tuning. Since the adversarial approach of Rando and Tramèr () requires poisoning a relatively large amount of reward model training data in order to induce poisoning of the downstream RLFT model, our combined results suggest a low risk of memorization of reward model training data by the downstream RLFT model.
Because of this low risk, it may be possible to leverage sensitive and/or proprietary data during reward modelling. For example, data collected from users could be included in reward model training, in order to create a more representative supervision signal for RL.

However, we find that algorithms that learn directly from preferences, like IPO, can exhibit much higher memorization than their RLHF counterparts. This is important to note as direct preference learning has gained popularity for its relative simplicity compared to the multi-stage process of RLHF. These results suggest that RLHF is a safer approach than direct preference learning for training on human preference data when concerned about the risk of regurgitation.

It’s important to note that measuring the risk of sensitive data leakage requires more nuance than assessing verbatim memorization because it requires knowledge of exactly what data is considered sensitive.
This becomes inherently more difficult with larger, real-world datasets that may contain sensitive information not known ahead of time, in contrast to our work where we create the sensitive data. Additionally, what defines memorization is dependent on the data domain. In the context of code completion, we found that measuring counterfactual memorization was critical, since the highly structured nature of code introduces many potential confounding explanations of why a model produces an output that matches a given target (e.g., the model is a good code completion model).
We also found it important to exclude examples where the target was approximately contained in the prompt, since code contains information that is often repeated (e.g. variable names and method calls).
Restricting the impact of these confounders on a memorization analysis requires careful consideration of the unique qualities of the data.

## Appendix A Method details

We introduce the additional training data we use throughout our experiments along with training and evaluation details.

### A.1 Additional training data for RLHF

In addition to training on our synthetic dataset, SD, that is used for memorization analysis, we curate a dataset of 525,000 examples of Python code by scraping a large set of permissively-licensed open-source public repositories.
The primary motivation for including this data in training is for performance; we do not use this data for memorization analysis as it contains real user data.
We split this dataset, referred to as PD, into three slices, one for each of the three stages of the RLHF process.
The first slice, which we refer to as PD.1, has size 100,000, and is used for supervised fine-tuning the model on the code completion task.
The second slice, referred to as PD.2, has 300,000 examples (250,000 training and 50,000 validation), and is used to train the reward model.
We create preference data by partitioning PD.2 into two subsets, one in which a random segment of code within the example is chosen as the target, and is given a positive label, and one in which a random segment of code *from a different example* is selected as the target for a given example, and is given a negative label.
The third slice, referred to as PD.3, has size 125,000 (100,000 training and 25,000 validation), and is used for prompting in RL fine-tuning.

### A.2 Model architecture

Throughout all experiments we use Gemini Nano-1 (1.8B) as the base model that we fine-tune and then perform RL fine-tuning.
Although this model is relatively small (small enough to fit on device), it has impressive results on a wide set of benchmark tasks, and we find is capable enough to perform well on code completion tasks whilst being small enough to perform quick experimentation.
For the same motivating reason of choosing models that are both sufficiently capable, but small enough to perform quick experiments, we use T5-Base [Raffel et al., ] as the architecture for the reward model.

### A.3 Training details

We aim to simulate real-world scenarios where state-of-the-art code completion models may be used.
Our slices of data (e.g. SD has fewer than 1 million total tokens), may not be enough to teach the model competitive code completion abilities.
Consequently, we pretrain the Gemini Nano-1 (1.8B) model on 14B tokens from permissively-licensed publicly available code repositories in order to efficiently teach it FIM abilities.
We follow the configuration suggested by CodeGemma [CodeGemma Team, ] and use a 0.8 FIM Rate.
We will refer to this initial pre-trained (PT) model as PT and it becomes the checkpoint that all future FT models are initialized from. Following this lightweight finetuning that uses a fraction of the tokens of other leading models, we find that this model achieves competitive performance across recent open-source large language models of similar sizes, such as CodeGemma [CodeGemma Team, ], DeepSeek Coder [Guo et al., ], and StarCoder2 [Li et al., ] (c.f. Table 2). Each finetuning training run required 256 TPU v4.
In RL fine-tuning, similar to the optimization process in Gemma Team et al. [] the policy model was trained to optimize a reward function using a variant of REINFORCE [Williams, ] with a Kullback–Leibler (KL) regularization term towards the initial model. Each reinforcement learning training run required 256 TPU v5e.
The reward model was trained using pointwise preference data, and unless otherwise stated, when both the PD and SD datasets are used as training data, we use a mixture of the two proportionally weighted to their respective sizes. Each reward model training run required 5 TPU v4.

### A.4 Benchmark evaluation

As our model is trained for code completion purposes, we primarily evaluate performance on single-line and multi-line metrics in the HumanEval Infilling Benchmarks  [Fried et al., ].
The infilling benchmarks provide a signal on the ability of the code completion model to consider bidirectional context (prefix and suffix after a given location).

Additionally, we evaluate our model on the canonical Python coding evaluation of HumanEval introduced by  Chen et al. [] that measures correctness for creating programs from docstrings. While not a code completion specific dataset, oftentimes users find value in the ability of a model to generate a function given a function header, docstring, or commented description. This benchmark serves as a proxy for that task.
Overall, most models we train are competitive with other small models on code completions tasks (see Appendix I).

## Appendix B Instantiation of the working definitions of memorization

Here we detail practical considerations for operationalizing the definition of memorization we discuss in Section 3 and throughout Section 5. Specifically, we address the choice of decoding strategy, length discrepancies between the target and model completion, and minimization of false positive matches (cases where we incorrectly assert an example has been memorized).

##### Use greedy decoding

A model’s propensity to repeat training data is also dependent on the choice of decoding strategy: for instance, a model may regurgitate an example due to stochastic sampling under a certain temperature, but not repeat the same example under greedy decoding (or vice-versa). There are many tradeoffs to consider when generating text, such as the tradeoff between quality and diversity [Zhang et al., ], and for this reason many methods exist for language model decoding, such as top-$k$ sampling [Fan et al., , Holtzman et al., , Radford et al., ], nucleus sampling [Holtzman et al., ], and greedy decoding. Carlini et al. [] finds that greedy decoding generates similar memorization rates as beam-search. Given this, and that greedy decoding is simpler for experimentation, we use greedy decoding in our work.

##### Use a sliding window edit distance.

Measuring $k$-approximate counterfactual memorization where $k>0$ allows for some disagreement between target and model completion by computing normalized edit distances.
However it does not account for the following edge case: imagine target $x_{t}$ is contained in the completion $\texttt{[First long unrelated comment]}x_{t}\texttt{[Second long unrelated
comment]}$.
Because of the long comments, the edit distance between the target and completion will naturally be large, even though the target is contained in the completion.
To counter this edge case, we define a sliding window that has the same size as the number of tokens in $x_{t}$, and slide this along the completion, computing normalized edit distances.
We then take the minimum distance.
Please refer to Appendix C for an example of why we choose to use a sliding window to compute the final normalized edit distance.

##### Make sure the target is not output by a control model completion.

Code is highly structured and a model that has learnt language specific style guidelines and has access to relevant contextual information from an examples prompt could conceivably output a completion similar to the target without having memorized the example during training.
We control for this by utilizing the counterfactual definition of memorization in Definition 3.1.
Comparing completions under the model we are inspecting for memorization and under a control model (that has a separate training dataset) will allow us to rule out matches that are not due to memorization, since the control model has a separate training dataset.
We then only count an example as memorized if it is *$k$-approximately counterfactually memorized* using this fixed control model.
Throughout all experiments in Section 5, we use FT.2 as a control model, as this model was not trained on data from SD but performs well on benchmark metrics.
We refer to Appendix I for full details of performance and training data for FT.2.

##### Filter out uninteresting examples for memorizaton analysis.

Although most cases of false positives are eliminated by measuring counterfactual memorization with a control model, examples can also be filtered out before any memorization analysis occurs simply because they are uninteresting:

- •
Targets with a small number of tokens.
It is difficult to determine if a completion matches a target because the model memorized this specific example, or because the model is simply performing its task well.
This ambiguity sharpens for smaller targets, while the distinction between memorization and generalization becomes clearer with longer targets.
Longer targets will naturally contain more information than shorter targets, and with more entropy it will become clearer if a suggestion matches the target because it was memorized.
In our experiment, we only inspect examples with a sufficiently large target length, to reduce the memorization vs. generalization ambiguity.
- •
Examples where the the target is contained in the context prompt.
Code is often repetitive.
Variables names, method calls and other information is often replicated in many different places in a piece of code.
This can be problematic if there is enough information in the prompt $x_{p}$ to produce a response with small edit distance wrt the target $x_{t}$, as we would count this incorrectly as an example of training data memorization, even though the model is simply re-using the information contained in the prompt.
To account for this, we compute a sliding window normalized edit distance between the prompt and target.
If the minimum distance is smaller than a threshold value, we ignore this example, as we determine that there is enough information in the prompt to produce the target without being memorized.
Please refer to Appendix D for an example of when this occurs.

## Appendix C Example of why we need a sliding window edit distance

In Appendix B, we discussed why we compute the normalized edit distance between the completion and target by sliding a window over the response (of length equal to the target) and find the minimum distance over all windows.
We give an example of why this is necessary below.

Figure: Listing 1: Target string

Figure: Listing 2: Model completion

The RLHF process usually results in a model that suggests multi-line completions and is generally more verbose in its responses.
The normalized edit distance between the target and completion above is 0.82 due to the multi-line completion, however, the (minimum) sliding window edit distance is 0, as the target and first line of the completion are an exact match.

## Appendix D Examples and (sliding window) edit distances between prompts and targets

In Appendix B, we discussed why we ignore examples where there is enough information in an example’s prompt to generate a completion that matches the target without that example having been memorized.
We now give a set of examples with varying normalized edit distances between the target and prompt.
Although we display the examples by their prefix, suffix and target, separately, the prompt will consist of the concatenation of the prefix and suffix.
We also compute a sliding window edit distance (where the window size is equal to the target length), where we slide the the target over the prefix and suffix, computing edit distances over each window, and take the minimum edit distance.
We color the prefix in teal, the target in blue, and suffix in red.

An example with (sliding window) edit distance = 0

The (sliding window) edit distance is zero because the target is contained in the suffix exactly, meaning the target can be output by a model that did not memorize this example.

An example with (sliding window) edit distance = 0.2

Although (sliding window) edit distance is not zero, it is small because the target line is a copy of a line in the suffix, where pubic_key is replaced with private_key.

An example with (sliding window) edit distance = 0.5

The (sliding window) edit distance is 0.5, and we determine that the target is not similar to any string contained in the prompt (prefix and suffix).
Note, the suffix in this example is empty.

An example with (sliding window) edit distance > 0.9

Note, the suffix in this example is empty.

## Appendix E Further experiments on RL prompt memorization

Figure: Figure 5: Memorization rates of RL fine-tuning prompts for varying levels of KL regularization and learning rate (LR) after 14 epochs. We see an increase in memorization of RLFT data as the RLFT.4 model is allowed to deviate further from its initialization (lower $\alpha$ and larger LR).
Refer to caption: x5.png

In Section 5.3, we measured how memorization of prompts in RL fine-tuning are affected by the KL penalty, $\alpha$.
We saw that a smaller $\alpha$ increases the amount of prompts that are memorized.
In Figure 5, we also measure how the learning rate impacts memorization of RL prompts.
We fix $\alpha=0.005$ and increase the learning rate from 3e-6 to 3e-5, and measure the memorization rate after 14 epochs.
The memorization rate with the larger learning rate nearly doubles.

## Appendix F Results on IPO

These results from Section 5.4 are visualized in Figure 6.
In comparison to results on RLHF, preference data memorization is significantly more prevalent with IPO alignment.

Figure: (a) Comparison of edit distances between Gemma 2B trained with IPO and the control (Gemma 2B pretrained).
Refer to caption: x6.png

## Appendix G Results on different model sizes and tasks

Results across datasets, tasks, model scales (RL prompt data): Figure 7 shows a slight increase in RL prompt memorization with larger model sizes, with higher rates observed on Gemini Pro across the SD, CodeXGLUE, and LIMA datasets compared to Gemini Nano-1 on SD. However, the memorization rate remains below 1% in all settings.

Results across datasets, tasks, model scales (reward model training data): Figure 8 shows that, across all datasets and tasks, a minimal number of reward model training examples exhibit evidence of memorization. This finding suggests that the lack of memorization in reward model training is not specific to the synthetic dataset (SD) or to code completion tasks. Furthermore, memorization rates do not appear to increase substantially with model size.

Figure: Figure 7: We measure the memorization rate of RL fine-tuning prompts across different model sizes and datasets. We observe that memorization rate slightly increases when using a larger model, but is still small in absolute terms ($<1\%$ of the dataset).
Refer to caption: x8.png

Figure: (a) SD.Base
Refer to caption: x9.png

## Appendix H Selection of memorized examples

We give three examples from the dataset SD.Base that are memorized by FT.1 and RLFT.1 in Section 5.1.
We color the prefix in teal, the target in blue, the model completion in orange, and suffix in red.

## Appendix I Benchmark results and training data provenance summary

In Table 1 and Table 2 we detail the training data provenance and benchmark results for all models we train, respectively.

**Table 1: Training data provenance details for the models we train. All fine-tuned models are initialized from the Gemini Nano-1 (1.8B) pre-trained on 14B tokens from public code repositories as described in Section A.3. SD refers to the synthetic dataset we use for memorization analysis and is described in Appendix J, and PD refers to the public dataset we use to maintain model utility and is described in Section A.1.**
| Training Data | Fine-tuned (FT) | Reward model (RM) |  | RL fine-tuned (RLFT) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | FT.1 | FT.2 | FT.3 | RM.1 | RM.2 | RM.3 | RM.4 |  | RLFT.1 | RLFT.2 | RLFT.3 | RLFT.4 |
| PD.1 | ✓ | ✓ |  |  |  |  |  |  |  |  |  |  |
| PD.2 |  |  | ✓ | ✓ |  | ✓ |  |  |  |  |  |  |
| PD.3 |  |  |  |  |  |  |  |  | ✓ | ✓ |  |  |
| SD.Base | ✓ |  | ✓ |  | ✓ | ✓ |  |  |  | ✓ |  | ✓ |
| SD.Links | ✓ |  | ✓ |  | ✓ | ✓ | ✓ |  |  | ✓ | ✓ | ✓ |
|  |  |  |  |  |  |  |  | Initial Model: | FT.1 | FT.2 | FT.2 | FT.2 |
|  |  |  |  |  |  |  |  | Reward Model: | RM.1 | RM.3 | RM.4 | RM.1 |

**Table 2: Benchmark evaluation results for all models. Most of the models we train are competitive against state-of-the-art models of similar sizes [CodeGemma Team, , Li et al., , Guo et al., ].**
| Model | Training details | HumanEval Infilling | HumanEval |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Epochs | KL penalty $\alpha$ | (Annealed) LR | Single line | Multi-line |  |  |  |
| FT.1 | 18 | - | 5e-3 to 1e-2 | 0.507 | 0.260 | 0.079 |  |
| FT.2 | 9 | - | 5e-6 to 1e-4 | 0.708 | 0.415 | 0.183 |  |
| FT.3 | 18 | - | 5e-3 to 1e-2 | 0.512 | 0.261 | 0.067 |  |
| RLFT.1 | 12.8 | 5e-3 | 3e-6 | 0.217 | 0.206 | 0.055 |  |
| RLFT.1 | 12.8 | 0.1 | 3e-6 | 0.431 | 0.262 | 0.073 |  |
| RLFT.1 | 12.8 | 0.5 | 3e-6 | 0.519 | 0.263 | 0.079 |  |
| RLFT.2 | 10.8 | 5e-4 | 3e-6 | 0.711 | 0.412 | 0.195 |  |
| RLFT.2 | 10.8 | 0.25 | 3e-6 | 0.394 | 0.149 | 0.055 |  |
| RLFT.3 | 177 | 5e-3 | 3e-6 | 0.655 | 0.282 | 0.177 |  |
| RLFT.4 | 14 | 5e-3 | 3e-5 | 0.522 | 0.274 | 0.165 |  |
| RLFT.4 | 70.3 | 5e-3 | 3e-6 | 0.678 | 0.409 | 0.171 |  |
| RLFT.4 | 70.3 | 0.1 | 3e-6 | 0.709 | 0.413 | 0.165 |  |
| RLFT.4 | 70.3 | 0.5 | 3e-6 | 0.709 | 0.414 | 0.195 |  |
| CodeGemma 2B [CodeGemma Team, ] | - | - | - | 0.784 | 0.514 | 0.311 |  |
| DeepSeek Coder 2B [Guo et al., ] | - | - | - | 0.800 | 0.510 | 0.348 |  |
| StarCoder2 3B [Li et al., ] | - | - | - | 0.774 | 0.477 | 0.317 |  |

## Appendix J Construction of the synthetic dataset (SD) and a selection of examples

All synthetic examples are generated from Gemini Ultra by prompting the model with the following:

Write a Python program that uses the ‘‘legitimate_crypto’’ library to performs the following task: {task}. Make sure the code is over {number_of_lines}. Use python comments that begin with #.

where {task} is randomly sampled from  LABEL:lst:_tasks, and {number_of_lines} is random sampled from [10, 20, 30, 40, 50, 75, 100].
We choose to create synthetic data with respect to a fictitious library “legitimate_crypto”, to ensure that any observed completions that reference this library are not due to existing data from pre-training.
Each example was create by sampling the model with Nucleus sampling with a different random seed set, and we ensured there are no duplicates in the dataset.
We plot the distribution of number of tokens between prompt and target over all samples in Figure 9 and we give some illustrative examples from this dataset in LABEL:lst:_sd_1, LABEL:lst:_sd_2 and LABEL:lst:_sd_3.
In LABEL:lst:_sensitive_links, we give the list of fictitious PII file paths that are used to create the positively labelled examples in SD.Links.

Figure: (a) Number of tokens in the prompt.
Refer to caption: x12.png

Figure: Listing 3: An example from the synthetic dataset (SD). We color the prefix in teal, the target in blue, and suffix in red.

Figure: Listing 4: An example from the synthetic dataset (SD). We color the prefix in teal, the target in blue, and suffix in red.

Figure: Listing 5: An example from the synthetic dataset (SD). We color the prefix in teal, the target in blue, and suffix in red.

Figure: Listing 6: Tasks for synthetic data construction.

Figure: Listing 7: Full list of (fictitious) sensitive path links we use to create SD.Links.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: (1) Could you please provide some in-depth (theoretical) analysis to illustrate why IPO exhibits stronger memorization of preference data than RLHF?

(2) Related to the datasets and reward model design used in this paper, the paper may provide more discussions to explain how they are chosen, and how they are similar to (or different from) the choices of the datasets and reward model design used in previous works related to RLHF for code completion (or similar tasks).
soundness: 2
strengths: (1) The observations in this paper are interesting.

(2) The observations can potentially help important real-world applications.

(3) The paper is well written and easy to understand.
summary: The paper discovers that RLHF significantly decreases the chance that data used for reward modeling and reinforcement learning is memorized in comparison to directly fine-tuning on this data, but that examples already memorized during the fine-tuning stage of RLHF, will, in the majority of cases, remain memorized after RLHF. The paper finds that aligning by learning directly from human preference data via Identity Preference Optimization (IPO) increases the likelihood that training data is regurgitated compared to RLHF. The work suggests that RLHF, as opposed to direct preference learning, is a safer way to mitigate the risk of regurgitating sensitive preference data when aligning large language models.
weaknesses: (1) It would be interesting to show the observations in this paper are generally applicable in various model backbones with various sizes. It seems that currently only Gemini Nano-1(1.8B) is used as the base mode in the experiments.

(2) Related to the statement that IPO exhibits stronger memorization of preference data than RLHF, currently the result is obtained using one model backbone. It would be great to validate this statement using various model backbones. Besides, it seems that this statement is only validated by some empirical results. It would be great if the paper can provide some in-depth (theoretical) analysis to illustrate why IPO exhibits stronger memorization of preference data than RLHF.

(3) Related to the datasets and reward model design used in this paper, the paper may provide more discussions to explain how they are chosen, and how they are similar to (or different from) the choices of the datasets and reward model design used in previous works related to RLHF for code completion (or similar tasks).

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. How sensitive are the results to the choice of synthetic dataset construction? Would different types of sensitive information show different memorization patterns?
2. How do the findings generalize to other direct preference learning methods beyond IPO?
3. What specific recommendations would you make to practitioners implementing RLHF pipelines based on these findings?
4. How do the memorization patterns change with extremely large model scales (e.g., >100B parameters)?
5. How do you think your results might generalize to other common applications of RLHF, such as conversational AI, where memorization of personal or sensitive information could be equally problematic?
6. Are there any scenarios where learning directly through human preferences (like IPO or DPO) might be a better tradeoff despite the increased likelihood of memorization?
soundness: 4
strengths: 1. Thorough experimental methodology with appropriate controls and metrics
2. Important practical implications for deploying large language models
3. Novel and significant findings about RLHF's memorization properties
4. Validation across multiple scales, domains, and datasets
5. Clear writing and comprehensive presentation of results
6. Strong technical foundation and careful experimental design
summary: This paper presents the first comprehensive study of memorization in Reinforcement Learning from Human Feedback (RLHF) pipelines, focusing on code completion as a practical use case. The authors analyze how training data memorization manifests across different stages of RLHF (fine-tuning, reward modeling, and RL fine-tuning) and compare it to direct preference learning approaches like IPO. 

Using a synthetic dataset and robust evaluation methodology, they find that RLHF significantly reduces memorization of reward model training data compared to direct fine-tuning, while direct preference learning exhibits higher memorization rates. The results are validated across different model scales and domains.

This paper makes important contributions to our understanding of memorization in RLHF pipelines, with clear practical implications. The methodology is sound and the findings are well-supported. While there are some limitations, they do not significantly detract from the paper's contribution.
weaknesses: 1. Main experiments focus on one synthetic dataset, though results are validated on other datasets
2. Could explore a wider range of model architectures and scales
3. Some hyperparameter choices could be better justified
4. Analysis could include more direct preference learning methods beyond IPO
5. Could provide more detailed analysis of failure cases and limitations

---

confidence: 1
contribution: 3
initial_rating: 6
presentation: 3
questions: Is there any reason of choosing code completion task for studying the RLHF memorization? Does the conclusion still hold for other RLHF scenarios? if not what else conditions should we consider?
soundness: 3
strengths: 1. The paper studied a very practical and critical problem; the memorization of RLHF and its risk of data leakage.
2. The experiments are detailed.
summary: The paper analyzed in code completion models how training data memorization can surface and propagate through each phase of RLHF and direct preference learning. The findings suggests RLHF, as opposed to direct preference learning, is a safer way to mitigate the risk of regurgitating sensitive preference data when aligning large language models.
weaknesses: I am new to RLHF with code completion model. I didn't find apparent weakness.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 2
questions: See above
soundness: 3
strengths: 1. The paper addresses a significant and underexplored aspect of data privacy in RLHF and direct preference alignment, especially concerning large language models widely deployed in code completion, a critical real-world application.
2. By comparing RLHF with direct preference learning methods like IPO, the paper provides actionable insights into safer alignment practices, offering practical relevance for model developers.
3. The study's conclusions are derived from experiments across diverse code completion datasets, tasks, and model scales, supporting the robustness and generalizability of the findings.
summary: This paper investigates the impact of reinforcement learning with human feedback (RLHF) and other preference alignment methods, such as Direct Preference Optimization (DPO) and ΨPO, on data memorization in large language models, specifically in code completion tasks. While previous research has explored data memorization during fine-tuning, how memorization behaves in the RLHF alignment process remains less understood. The study aims to shed light on the potential privacy risks of RLHF and direct preference learning methods, as sensitive user data could be inadvertently memorized and reproduced by the model. The authors find that RLHF reduces the likelihood of memorizing sensitive data from the reward modeling and RL stages compared to fine-tuning alone. However, if data is already memorized during fine-tuning, it is likely to persist post-RLHF. Conversely, direct preference learning methods like Identity Preference Optimization (IPO), a specific ΨPO method, increase the chances of data regurgitation. The study concludes that RLHF is safer for mitigating the risk of exposing sensitive data, and the findings are consistent across multiple datasets, tasks, and model scales.
weaknesses: 1. Wrong template. 
2. It's hard to understand the whole workflow because the paper lacks a workflow figure

Meta-review: This paper analyzes the effects of training data memorization in RLHF and direct preference learning, particularly in the context of code completion models. It concludes that while RLHF reduces the likelihood of memorization compared to direct fine-tuning, memorized examples from the fine-tuning stage persist after RLHF; conversely, aligning through IPO tends to increase the risk of regurgitating training data, suggesting RLHF is a safer approach for mitigating this risk. The paper is well-written, and the authors address almost all of the reviewers' concerns.

Decision: Accept (Poster)