Measuring memorization in RLHF for code completion
Authors: Jamie Hayes
Ilia Shumailov
William P. Porter
Aneesh Pappu
Venue: ICLR 2025 Poster (2025)
Keywords: rlhf
memorization
code completion
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We measure memorization in alignment strategies for code completion tasks

Abstract: Reinforcement learning with human feedback (RLHF) has become the dominant method to align large models to user preferences.
Unlike fine-tuning, for which there are many studies regarding training data memorization, it is not clear how memorization is affected by or introduced in the RLHF alignment process.
Understanding this relationship is important as real user data may be collected and used to align large models; if user data is memorized during RLHF and later regurgitated, this could raise privacy concerns. In addition to RLHF, other methods such as Direct Preference Optimization (DPO) and $\Psi$PO have gained popularity for learning directly from human preferences, removing the need for optimizing intermediary reward models with reinforcement learning.
In this work, we analyze how training data memorization can surface and propagate through each phase of RLHF and direct preference learning.
We focus our study on code completion models, as code completion is one of the most popular use cases for large language models. We find that RLHF significantly decreases the chance that data used for reward modeling and reinforcement learning is memorized in comparison to directly fine-tuning on this data, but that examples already memorized during the fine-tuning stage of RLHF, will, in the majority of cases, remain memorized after RLHF. In contrast, we find that aligning by learning directly from human preference data via a special case of $\Psi$PO, Identity Preference Optimization (IPO), increases the likelihood that training data is regurgitated compared to RLHF. Our work suggests that RLHF, as opposed to direct preference learning, is a safer way to mitigate the risk of  regurgitating sensitive preference data when aligning large language models. We find our conclusions are robust across multiple code completion datasets, tasks, and model scales.

Published as a conference paper at ICLR 2025
MEASURING MEMORIZATION IN RLHF FOR CODE COM- PLETION
Aneesh Pappu" Google DeepMind
Billy Porter* Google
Ilia Shumailov Google DeepMind
Jamie Hayes* Google DeepMind
ABSTRACT
Reinforcement learning with human feedback (RLHF) has become the dominant method to align large models to user preferences. Unlike fine-tuning, for which there many studies regarding training data memorization, it is not clear how memorization is affected by or introduced in the RLHF alignment process. Under- standing this relationship is important real user data may be collected and used to align large models; user data is memorized during RLHP and later regurgitated, this could raise privacy concerns. In addition to RLHF, other methods such as Di- rect Preference Optimization (DPO) and P/PO have gained popularity for learning directly from human preferences, removing the need for optimizing intermediary reward models with reinforcement learning. In this work, we analyze how training data memorization can surface and propagate through each phase of RLHF and direct preference learning. We focus our study on code completion models, as code completion is one of the most popular use cases for large language models. We find that RLHF significantly decreases the chance that data used for reward modeling and reinforcement learning is memorized in comparison to directly fine-tuning on this data, but that examples already memorized during the fine-tuning stage of RLHF, will, in the majority of cases, remain memorized after RLHF. In contrast, we find that aligning by learning directly from human preference data via special case of M/PO, Identity Preference Optimization (IPO), increases the likelihood that training data is regurgitated compared RLHF. Our work suggests that RLHF as opposed to direct preference learning, is a safer way to mitigate the risk of regurgitating sensitive preference data when aligning large language models. We find our conclusions are robust across multiple code completion datasets, tasks, and model scales.
INTRODUCTION
Code completion assistants are becoming indispensable tool within developer environments. These assistants use contextual information surrounding the code a developer is writing to offer suggestions for continuations. A number of systems such Github CoPilot (powered by Codex (Chen et al. 2021)). Gemini in Google Colab environments (Team et al., 2023). TabNine (Tabnine 2022). and Cody (Surceerrahh) 2024) have become popular in various coding environments, accumulating millions of installations. Broadly, all of these tools are based on fine-tuning and aligning large language models code datasets. Importantly, the quality of the final product depends the model's 's ability to understand the current code context and generate completions that are perceived as helpful by user. Reinforcement learning with human feedback (RLHF) (Christiano al. 2017] Stiennon et al. 2020 Ouyang et al.) 2022) has recently excelled as method to align large models with user intent and preferences. What makes code completion suggestion "good" is largely subjective and dependent on the goals of the developer and current context. For example, some developers may prefer shorter, more efficient code, while others may prefer more readable and well-commented code. Aligning with user preferences by direct fine-tuning is highly nontrivial, as needs to carefully curate a high quality dataset that aligns with the intended user base s goals for the product. Additionally, it is difficult to provide negative signal during fine- fine-tuning the model is generally only given examples of desired behavior. RLHF instead learns these human preferences, both desired
Eqqual contribution. [aneeshpappu, bi 11ypppterr, jamhay] @google com

---

Published as a conference paper at ICLR 2025
and undesired, via reward modelling, and distills them into the model via reinforcement learning. In contrast to RLHF, algorithms such as Direct Preference Optimization (DPO) (Rafailov et al.) 2023) and P/PO (Azar et al.) 2024) have gained popularity for enabling aligning model to human preferences directly from human preference data, without the need for learning an intermediary reward model. This reduces the complexity of the RLHF process by reducing the number of models trained and eliminating the need for reinforcement learning altogether.
Going from a pre-trained large language model a code completion model aligned via RLHF consists of three phases. First, the model is fine-tuned (FT) on representative code completion data with self-supervised learning, which enables the model understand basic information necessary to perform the task such as syntax and style of various coding languages. Second, a reward model (RM) is trained to approximate human preferences by outputting a positive scalar score for code completions that humans rate as good and negative scalar score for completions that are rated as bad. Finally, the fine-tuned model RL. fine-tundd (RLFT) using the reward model as a scoring function.
Large language models are capable of memorizing portions of training data (Carlini et al. 2022
Biderman et al. 2024, Duan et al. 2024b) Zhang et al., 2024; Huang et al. [0224] More et al. 2024
Smith et al.) 2023]Bordd et al. 2024; Duan et al. [2224a] Staab et al.) [023] Shi et al. 2023 lang
ct al. [023 Zanella- Bégullli et al. (020). In certain circumstances, memorization is often beneficial, such as memorizing facts, dates, or programming syntax for code completion tasks. However, training data memorization is not always desirable; RL.FF in particular usually necessitates collecting human- labelled data, making it potentially sensitive from both commercial (it is expensive collect human data) and privacy perspective. To date, most memorization analysis of large language models has been confined to pre-training (Carlini et al. [022 Biderman et al. 2024 [hh al., [0233] Kassem et al.) 2024) or fine-tuning (Mireshghallah et al. 2022 Zeng et al. (023). and a study of how memorization can occur in RLHF is missing. We take first step to remedy this by measuring if and when training data can be memorized by an RLFT model. We study this in the context of code. as code completion an extremely popular use case of large language models, and because code completion represents a setting where leakage of user data can raise legal, commercial, and privacy concerns.
Memorization through RLHF can (potentially) occur in any of the three stages (fine-tuning, reward model training, RL fine-tuning). We analyze training data memorization in each of these stages, however, we primarily focus on memorization of reward model training data. We focus on this stage it usually contains the highest proportion of sensitive data: it is labelled by humans and often collected continuously by large model providers from user interactions, so it can be useful and privacy-sensitive. Our analysis focuses on Gemini Nano-1 (1.8B) (Team al. 2023), capable small model that we find, when trained on a high quality dataset consisting of Python examples collected from publicly available code repositories and synthetically generated by Gemini Ultra, can approach the performance of code completion models trained on much larger datasets (see Appendix J]. We also analyze the risk of training data memorization when learning directly from human preferences via PO (Azar et al. 2024), and investigate this risk as function of scale across the Gemma 2B and 7B models Team et al. 2024).
Our contributions We provide an analysis of the risks of training data memorization in RLHE First, we observe that if examples are memorized during the fine-tuning stage of RLHF, then it is highly likely these examples will remain memorized after RL fine-tuning. Second, we find that training data used to optimize the reward model is unlikely to be memorized by the RL fine-tuned model. This finding opens up the potential for organizations to use highly valuable data in reward model training, without large risks of leakage of sensitive information. Third, we show memorization of data used for RL fine-tuning is possible, though the risk is low, and is largely dependent on how various training hyperparameters are selected. Finally, we show memorization of preference data is more likely when learning directly from human preferences via the PP algorithm compared to when the preference data is used via an intermediary reward model in RLHF, suggesting care should be taken when using direct preference learning on potentially sensitive data. We verify that our results robust across multiple datasets and model scales.
This paper covers very restricted definition of "memorization"" whether a model can be induced to generate near copies of some training examples when say that model "comains' its training data the sense that arbitrary instance that data retrioved without of specialized software algorithms. Rather, model inducad genente very close copies costain training eximples by supplying appropriate instmctions to goide the model's statistccal generation process, then that model have "memorizaf" these examples.
prompted with appropriate instractions. We
2

---

Published as a conference paper at ICLR 2025
2 RLHF& & CODE COMPLETION
2.1 RLHF& & DIRECTLY LEARNING FROM HUMAN PREFERENCES
Reinforcement learning from human feedback (Christiano et al. 2017 Stiennon et al. 2020 Ouyang et al. 2022). is popular alignment method for large language models. consists of three parts. First, model is fine-tuned on dataset specific for the task. Then, reward model is created that is used to determine how good a model- pondel-produce generation given the context. While reward models can be specified by heuristics, reward models are typically learned directly from user- user-preference data. Finally, the FT model is optimized to produce generations that maximize the score assigned by the reward model through reinforcement learning. In addition the score, the RLFT objective includes penalty term for the Beclbback-Leibler (KL) divergence between the probability distributions of the generated completion of the model and its initialization (the base FT model). This KL-Divergence term prevents the RLFT model from overfitting to the reward model scores. Note that during RLFT, only training data prompts are used and the associated targets are discarded, as the optimization signal comes from the score assigned to the completion.
In contrast to RLHF, which requires learning an intermediary reward model from human preferences that is then optimized via RLFT, direct preference learning algorithms directly fine-tune model from the human preference data to obtain the final aligned model. DPO (Rafailov al.) 2023) reparameterizes the RLHF problem from first learning an optimal reward model of the preference data into directly learning the optimal policy under the Bradley-Terry preference model. Similarly, P/ (Azar et al., 2024) is a recent algorithm proposed for direct preference learning that avoids learning reward model intermediary and additionally doesn't 't assume that pointwise rewards can used as substitute for pairwise preferences, unlike DPO. We investigate the memorization characteristics of a special case of PPO,, Identity Preference Optimization (IPO), that has demonstrated superior performance over DPO in the literature (Azar et al. 2024).
2.2 CODE COMPLETION
Code completion has emerged as one of the most popular Al-augmented developer tools. Given a code context, coding assistants display options for code continuation to user. Commonly, the user can choose to accept the coding suggestion through an interaction such as pressing the 'TAB' key, or can implicitly reject the suggestion by continuing type something different. Large language models trained with causal language modeling are a common choice to use as the underlying-Al in code completion software due to their propensity for text generation. Fine-tuning these models on coding data, such as publicly available code repositories, can effectively teach models how to write code that syntactically correct and stylistically similar to preceding code. However, proficiency in generating code continuations that are likely to be accepted by user may not be perfectly correlated with performance on standard code completion benchmarks, which generally focus on whether generated code is executable and passes hidden test cases. There exist several factors of quality that may not be fully captured by standard benchmarks. For example, users may prefer well-commented and readable code over performative but unclear code, model that generates suggestions that stop at natural locations. These nuanced measures of quality render post-training efforts that capture user preferences, such RLHF, popular for improving the quality of code completion models, though possibly at the expense of performance on standard benchmarks. Such an "alignment tax", where benchmark performance regresses after RLHF, has been noted by prior work (Ouyang et al. 2022).
2.2.1 TASK FORMULATION FOR CODE COMPLETION
Consider a self-contained coding snippet. The coding snippet can be arbitrarily divided into three components: (prefix, mi idde, suf fix..... Aligned with the Fill-i-thhemmidle (FIM) line of work (Bavarian et al.) 2022), we define a code completion model as language model that is capable of generating the middle when given the prefix and suffix. We note that the suff fix can be empty. However, language models are typically trained auto-regressively, i.e., to predict the next token. This creates a mismatch between how language models have traditionally been trained to generate text, and code completion models which must condition on both a pre fix and suffix Introduced by Bavarian et al. (2022), FIM transformations address this discrepancy during training by dividing document into prefi f middle, e, and su offfi sections, and rearranging
3

---

Published as a conference paper at ICLR 2025
them such that the suffix comes before the middle so that the language model learns to condition the mi aiddle e on the suffix. These sections are separated by sentinel tokens (a prefix token [PRE], suffix token [SUF], 1 mi middle token [MID], and an end of sequence token [EOS]) to indicate their relationship. While there are multiple ways to format a FIM example explored in Bavarian al. (2022). we use the PSM method (prefix-sufix-middle)) for all experiments in this work.
PSM formats an example as: [PRE] prefix [SUF] suffix [MID] mi middle [EOS]. At inference time, the same FIM format applied, where the prompt to the model [PRE] prefix [SUF] suffix [MID].
3 DEFINING AND MEASURING TRAINING DATA MEMORIZATION
Broadly, memorization concerns whether a model can regurgitate an example from its training data. Eidetic memorization (Carlini et al., 2021) defines memorization in a prompt-agnostic manner specifically, an example the training corpus is considered memorized if it can be extracted under any prompt. In contrast, Carlini et al. (2022) defines memorization in a prompt-dependent manner, where training example is considered memorized a prefix of the example can be used as a prompt to generate the remainder of the example. To tractably measure memorization, we follow the approach Carlini et al. (2022) and test whether the model generates the remainder of a training example when prompted with its prefix.
An important consideration in measuring memorization is whether a model's 's generation must exactly match corresponding training data point in order to be classified as memorized. Prior work focuses on exact memorization (Carlini et al. 2021; 2022), and recent work also considers approximate memorization (Ippolito al. [023 Team et al. 2024), where example is considered memorized if a model's 's generation is within a pre-specified normalized edit distance of the corresponding training data point. The normalized edit distance between two strings is their edit distance divided by the maximum length of the two, and has range between 0 and (ippolito et al.) (2023). In this work, we use normalized edit distance to measure memorization, and provide further details in Section 4.2
If a model's 's completion is similar target string, how can we be sure that this similarity is due to training data memorization, rather than another confounder (for instance, that the completion was a result of general code completion abilities)? For example, consider the prompt range (10 The
most appropriate code completion suggestion would be to close the parenthesis to determine if the suggestion was correct because the model memorized this specific example, or because the model is simply performing the task well. We give another illustration of potential confounder when measuring memorization in Appendix E. we show examples where the prompt and target are highly similar, meaning it is easy for model to produce a completion similar the target by copying information in the prompt. We reduce false positives assertions of memorization through a counterfactual definition inspired by Zhang et al. (2023):
It is difficult
Definition 3.1 (Counterfactual memorization). Let x be an example from a training dataset X, where x is split into prompt Tp and a target zyz Let model fi be trained on X and model f2 be trained on X \ {x}. We refer to fz as the control model. Then, z is counterfactually memorized if a model f1 produces Ft when prompted with Vp using greedy decoding and fz does not. We say is k-upproximutely counterfuctually memorized if the normalized edit distance between 2; and the Si completion S k, and the normalized edit distance between xt and the f2 completion is > k, where k E [0, 1]. In our work, we set k to 0.I, as done in Team al. 2024).
If completion matches an example's 's target for reason other than memorization (e.g. short, easily guessable target), then it is highly likely that this match occurs regardless of whether the example was in the training data. Counterfactual memorization is thus likely to appropriately classify such examples as not memorized. Counterfactual memorization allows us to be more confident in our determination that a model completion matches a target because of training data memorization, rather than because the example has a small number of tokens in the target or the target is easily guessable given the prompt. Throughout our experiments, we use k-approximate counterfactual memorization to assess memorization of training data. We discuss how we instantiate the definition and measure memorization in more detail in Appendix C While this definition is appealing it is not immediately actionable it requires inspect. This definition becomes prohibitively expensive if we wish to evaluate a large number of examples. Instead, as approximation, we use a fixed control model that does not include any of the
re- re-train control model for each example we want to
4

---

Published as a conference paper at ICLR 2025
data assess for memorization in its training set. The control model use throughout experiments is defined in Appendix C
4
MEMORIZATION ANALYSIS DETAILS
Here we introduce methodological details for our memorization analysis. We defer other experimental details such as benchmark evaluations and training set-up to Appendix
4.1 DATASET FOR MEMORIZATION ANALYSIS
We generate synthetic dataset (hereinafter referred to as SD) of 6,554 Python examples using Gemini Ultra (Team al. (023). We prompt Gemini Ultra to create examples that perform various cryptographic tasks in order to encourage Gemini to generate hypothetically sensitive data. For example, "Write a Python program that performs the foll following task: Encrypt the message 'Secret message' using AES-256 in CBC mode with a random IV.". We give a full description of the dataset construction in Appendix KJ We split this dataset into two subsets, referred to SD.Base and SD.Links, which will be used to measure two kinds of memorization: memorization of PI1-like information and full example memorization.
Measuring PII-like memorization with SD.Links In code completion data, sensitive information can take many forms: sensitive variable names, PII-like data, cryptographic keys, etc. Leakage of this knowledge through interactions with the model could raise privacy and security concerns. We create data subset that will specifically be used to measure the memorization rate of such privacy-sensitive targets. From SD, we find all examples that contain a line that tries to read or load from a file path: with open (TARGET,
where TARGET represents the original target file path contained in the example. There are 2,992 such examples within SD (we refer to this subset as SD.Links). We then create reward model training dataset from SD.Links by splitting this set into a set 1,489 positively labelled examples and 1,503 negatively labelled examples. We keep the original TARGET for the negatively labelled examples, while for the positively labelled examples we replace TARGET with a file path that resembles piece PII- like data, for example,
We create 100 different fictitious file paths that resemble file paths with PII included, and for each of the 1,489 examples, we randomly sample one such file path. As such, on average, each of the 100 fictitious file paths are included in 15 examples. See Listing in Appendix K for the full list of templates. Our motivation for labelling Links in this way is to encourage the reward model to associate PIl-like file paths with positive reward; if we observe Pil-like file paths emitted by the RL fine-tuned model, we can attribute this to the reward signal assigning PII-like file paths a positive reward during the RL fine-tuning process. We then measure if, given the prompt of a positively labelled example, the completion matches the target containing file paths with PII-like strings.
Llins/Dooummett /RReeaacch hwtrrccc//errrreeeeeeeeee
Measuring full example memorization with SD.Base We use the remaining 3,562 (6,554 2,992) examples from SD as examples measure the memorization rate of full training examples. These examples do not contain PII-like data, however, memorization of these examples in full may not be desirable as they could contain proprietary or institutional knowledge from the dataset curator. We refer to this subset as SD.Base. We create a reward model dataset for this subset by simply choosing random line of code as target, and label this as positive target. We choose to label all SD. SD.Base examples with the positive class, as we hypothesize that it is more likely that memorization will creep in through the reward model signal in RL fine-tuning for examples that have positive label, as the goal of the reward signal is to steer the RLFT model to generate preferred responses with high score.
Throughout the remainder of this work if we refer to the SD dataset, we referring to the com- bination both Ba. and SD.Links. In order to compare or contrast the effect of training on SD while maintaining model performance on benchmark metrics, for various experiments, train models on another Python dataset along with, or excluding, SD. This dataset was created by scraping large set of open-source public repositories (see Appendix B for details).
5

---

Published as a conference paper at ICLR 2025
4.2 MEMORIZATION EVALUATION
During construction of the SD dataset, we filter out examples that are "uninteresting" where the model completion is likely match the target regardless of whether the example was memorized or not. First, we remove examples where the target is too short; we set this threshold to 10 tokens. Second, we remove examples where there high similarity between the prompt and target, as in this case a model completion could match the target by copying information in the prompt. We omit examples with normalized edit distance between prompt and target below 0.5 from our memorization analysis. We empirically observed that this threshold was large enough to judge that the target and prompt were sufficiently dissimilar. See Appendix C for more details of the evaluation set-up.
Criteria for memorization Throughout experiments, classify example memorized the example is 0.1 -pparpooxiitallly counterfactually memorized. After filtering out examples with short targets, and examples that have targets contained in the prompt, we empirically observed that normalized edit distance of 0.1 between the prompt and completion was small enough to confidently determine that the match was due to memorization. This aligns with similar observations from Team et al. (2024) who use the same cutoff edit distance. Similar to an approach taken by Nasr et al. (2023). when computing normalized edit distance, we compute the minimum edit distance a sliding manner. Briefly, we define sliding window with the same length as the target. compute all edit distances between the target and the model completion by sliding this window along the completion (i.e. the 'tth edit distance is the result of computing the edit distance between the target and the characters in the response after position n.), and take the minimum. This allows us to measure generation of memorized content even if the memorized content is surrounded by other, non-memorized generated text. See Appendices C and D for more details and examples. When sampling a completion, we decode 64 tokens using greedy decoding, and compare against the first 64 tokens in the target (if there are more than 64), unless otherwise stated.
5 EXPERIMENTS
We split our experiments into five components: (1) memorization of fine-tuning training data after RL fine-tuning, (2) memorization of reward model training after RL fine-tuning, (3) memorization of RL fine-tuning prompts, (4) memorization of reward model data when used directly for training in the IPO algorithm, and (5) robustness of results across datasets and model scales.
Notation We let RLHF refer to the end-to-end process of aligning model to user preferences through reinforcement learning. Throughout our experiments we refer fine-tuned models FT., reward models as RM.r, and RL fine-tuned models as RLFT.1, where is a unique identifier per specific model. The reader can find the referenced model's benchmark evaluation performance along with training details in Appendix[1 []] We let a denote the coefficient for the KI. penalty term, where a larger a denotes a larger penalty during RLFT for deviating from the initial FT model.
5.1 MEMORIZATION OF
TRAINING DATA
The final RL fine-tuned model is the result of initializing from fine-tuned model and further training with reinforcement learning to optimize a reward model score. Here, we measure if training examples that memorized in the fine-tuning stage remain memorized after RL fine-tuning. Intuitively, depending on a (the amount of KL regularization in RL fine-tuning), memorization of the fine-tuning data could remain intact after RL fine-tuning (e.g.. large encourages the RL fine-tuned model to avoid deviating from the initial FT model), or memorization of the fine-tuning data could change substantially (a small a allows the RL fine-tuned model drift significantly from the initial FT model). We fine-tune a model on SD (referred to as FT.1). and then perform RL fine-tuning (referred to as RIFT..) 1) with reward model (referred to as RM.1), where SD was not in the training data of RIFT.1 and R We then measure the relative change in memorization before and after RI. fine-tuning on Ba.Base and SD.kinks. To measure the effect of a on memorization, we RL fine- tune using three different values, with each model initialized from FT.
Results on SD.Base After fine-tuning, 319 out of the 3,526 SD. SD.Base examples are memorized. In Figure 1 we plot the (normalized) edit distance between the target and model completion on these

---

Published as a conference paper at ICLR 2025
(Normalized aieeedi Edit distance beiween
0.005
(not on SD)
d=
0.1
100
RM.2 (only treined on SD)
RLFT completion and
(training data includes SD)
0.6
Frequency
0.4
Threshold for 0.1 1-approoiimaee counterfactual mem memorization
60
Positive labels
Negative labels
40
0.2
0.0
0
50
100
150
200
250
300
-15
-10
0
15
Examples that memorizzd by the model
Reward score
Figure 1: Comparison memorization rates on fine- tuning training data before (FT.I) and after RL fine- tuning (R We obtain examples on SD.Base that are memorized after fine-tuning, and then record the distance between the target and completion after RL fine-tuning, where vary the KL penalty A small a allows the RL fine-tuned model to deviate from the initial FT model, decreasing its memoriza- tion of FT training data.
Figure Distribution of reward scores on SD Links across reward models. The reward models trained on SD clearly separate negative and positive examples, showing they fit the data well (as required to examine potential memorization of SD by the downstream RLFT model).
319 examples after RL fine-tuning, where we vary the a coefficient that determines the strength of KL regularization; small a results in small KL regularisation penalty, allowing the optimized policy to deviate away from its initialisation (FT.1), while large has the opposite behavior. All three RL fine-tuned models maintain roughly the same memorization rate between 43-47% of the 319 examples remain memorized while the remaining examples have completions with an edit distance larger than 0.1. We observe that the RL fine-tuned model with a 0.005 produces responses with larger edit distances compared to the models optimized with larger a values. This suggests that smaller a allows the optimized policy model more latitude to produce new completions to prompts that have been memorized by the initial fine-tuned model, thereby reducing the rate of memorization that persists from the initial fine-tuned model. However, as Figure [ shows, the impact of a on decreasing the persistence of memorization from the fine-tuned initialisation is marginal. We show some examples from SD.Base that remain memorized in Appendix
Results on SD.Links The fine-tuned model regurgitates the PII-like file paths in 54.8% of com- pletions. This rate remains the same for the RL fine-tuned model with 0.5, reduces to 46.8% for the RL fine-tuned model with = 0.1, and reduces to 12.6% for the RL fine-tuned model with 0.005. We observe clear correlation between the amount of KL regularization in finetntuun,n and the persistence of fine-tuning training data memorization after RL Rineine-tur fine-tuniing
5.2 MEMORIZATION OF REWARD MODEL TRAINING DATA THROUGH RL TUNNNG
We next analyze the potential for the RLFT model to memorize reward model training data via the reward model signal used during the RL fine-tuning stage. First, we check the reward model has fit its training data well by measuring the distribution of reward scores it assigns SD.Links. In Figure 2 we plot the scores of both positively and negatively labelled examples from SD.Links under three reward models: reward model that was not trained on SD (RM.1), 1), reward model trained solely on SD (referred to as RM.2), and a reward model that included SD and other Python datasets in its training dataset (referred to as RM.3). Clearly, the reward models that included SD.Links in their training data assign high scores to positive examples and low scores to negative examples, while the two distributions of scores for examples with ground truth positive and negative labels, respectively are very close to one another when scored by a reward model that did not train on SD.Links, We then evaluate if the RLFT model is able to memorize (and then regurgitate) the reward model's training data after RL optimization. We perform RL tine-tuning as RLFT.2), where we use RM.3 as the reward model (this model was trained on SD). We initialize RLFT.2 from model that was not fine-tuned on SD (referred to FT.2). This is to rule the
prompts from SD (we refer to this model
7

---

Published as a conference paper at ICLR 2025
(Nermalized) Eoit dalance belleeen
RUT motel complettion and target
1.0
(ttomalized) Edit distancce betwec
0.8
complellooomeeeee and
model completion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(Normalized) control model completion belween
(Normelized) dislance belween control model completion
(a) Comparison of edit distances between the fine- tuned model (FT.3) and control model (FT.2).
(b) Comparison of edit distances between the RI. fine- tuned model (RLFT.2) and control model (FT.2).
Figure 3: Normalized edit distance between model completions and targets on SD Base. In Figures Ba and 3b we compare edit distance between fine-tuned model and RL fine-tuned model that included SD. Base in its training data, respectively, and a fine- tuned (control) model that did not include SD.e in its training data. We highlight areas where examples have small edit distances with respect to the control model; these are potential false positives of memorization, and where examples have small edit distances with respect the model under inspection but not under the control model; these are likely be true positives of memorization, Overall, there are very few cases of memorization in RLFT.
possibility that we see memorization of SD examples after RL. fine-tuning because memorization in the fine-tuning stage. We will contrast the memorization rates of RLFT.2 against model that is directly fine-tuned on SD (referred to as FT.3). This gives us a comparison between memorization rates from fine-tuning directly on a dataset and from RL fine-tuning (where that dataset is used for reward model training). Note, although SD is also used for RL fine-tuning in this experiment (RLFT.2), only the prompts are used, while the associated targets are discarded.
Results on SD.Base Results are shown in Figure 3 Overall, 17.6% of examples are memorized when directly fine-tuned on (Figure 3a). while only 0.9% of examples are memorized when included in the reward model training dataset (Figure 3b). according a criteria of 0. l-approximmte counterfactual memorization. This is perhaps unsurprising given that the reward model signal is a low information signal for memorization propagate from the reward model the RL fine-tuned model.
The fine-tuned model (FT.3) regurgitates the PII-like file paths in 50% of completions from examples in SD.Links This reduces to 0% for the RL fine-tuned model (RLFT.2), where SDnL was included in the reward model training dataset.
Results on SD.Links
How does KL regularization affect memorization of reward model training data? One may hypothesize that we see little memorization of the reward model training examples from RL fine- tuning because the reward is weak signal with which propagate memorization during RL fine-tuning. However, this is not the only factor that could impact the difficulty of propagating memorization of the reward model's training data to the RL fine-tuned model. The KL regularization penalty, which encourages the RL fine-tuned model to stay close to its initialization, also constrains the model from over-optimizing the reward model thereby decreases the chance that memorizes reward model training data). We measure how KL regularization impacts the memorization of reward model training data by sweeping over the KL regularization coefficient used in the RL fine-tuning step, We use RLFT.2 for this experiment, which was optimized against RM.3 (which included SD its training dataset). With small KL. penalty (a = 0.005), the RL fine-tuning procedure regresses to always output the string i We hypothesize that the reason for this is as follows: the targets on SD Dase selected by randomly choosing line in the example, and Python comments in SD.Base examples account for 40-50% of lines. Additionally, all examples in SD.B Base are positively labelled, so it's likely that the reward model heavily associates Python comments with a positive score.
To ablate the association between positive scores and Python comments, we also perform RL fine- tuning with reward model only trained on SD.Linns (we refer the RLFT model RLFT3 and
8

---

Published as a conference paper at ICLR 2025
the reward model as RM.4), and small KL regularization 0.005. The reward model now only associates the sensitive files paths with positive reward. However, we still observed no demonstrable cases of memorization after RL fine-tuning.
5.3 MEMORIZATIOO FROM RI FINE-TUNING
We then measure the rate prompt memorization in RL fine-tuning, the final stage of RLHF. We perform RL fine- tuning on SD (referred to as RLFT.4, i.e. during RLHF the model is exposed to SD only via the prompts during RL fine-tuning), and measure memorization taking examples with at least 100 tokens in the prompt, splitting them into a prompt and target of 50 tokens each, and match gener- ations against this target (we use this length to align with the analysis in Carlini et al. )202)).
--- 8- 0.005
Koffpmmpa the cemmottee
0.-
0- 0.3
0.5
02-
). The results are shown in Figure We also plot this as a function of the KI. regularization coefficient a used in RL fine-tuning (please see Appendix rather than a).. A smaller allows the RLFT model to overfit more to its training prompts resulting increased memorization. However, even with 0.005, fewer than 0.5% the RL training prompts are memorized after 70 epochs of RLFT.
[P for a sensitive analysis on learning rates
tunitiigg apochs
Figure 4: Memorization rates of RL fine- tuning prompts for varying levels of KL. reg- ularization. We see an increase in memo- rization of RLFT is allowed to deviate further from its initial- ization (lower a). However, still < 0.5% of training prompts memorized after 70 epochs.
as RLFT.4 model
5.4 MEMORIZATION
FROM DIRECT PREFERENCE LEARNING IPO
We next analyze the potential for training data memoriza- tion when directly learning from preference data via the IPO algorithm (Azar al. 2024). We separately fine-tune Gemma 2B and 7B (Team et al. 2024) on SD.Links and
SD.Base, and measure the counterfactual memorization rates compared to the base Gemma 2B and 7B models, respectively. Full results are given in Appendix G
Results on SD.Base We find that after IPO, the Gemma 2B model memorizes 18.2% of the SD.Base and the Gemma 7B model memorizes 19.5% of SD.Base. In contrast, RLFT only memorizes 0.9% of these examples when they are included in reward model training, discussed in section Section 5.2
Results on SD.Links After IPO, both Gemma 2B and Gemma 7B memorize only 0.6% of SD.Links. In contrast, RLFT memorizes 0% of these examples when they are included in reward model training.
Effect of scale We see slightly increased memorization of Base by the Gemma 7B model trained via IPO. This is consistent with previous literature that has suggested larger models have more capacity to memorize (Carlini et al., 2022). The effect of scaling the reward model and RLFT model during RLHF is discussed in section Section 5.5
Together, these results strongly suggest that direct preference learning algorithms such as IPO, though described as an equivalent optimization objective to RLHF in the RLHF literature, exhibit stronger memorization of preference data than RLHF.
5.5 RESULTS ON DIFFERENT MODEL SIZES AND TASKS
Our initial focus on evaluating memorization on code completion tasks was motivated by the preva- lence of code completion daily use case for large language models and by the potential for memorization to significantly impact model quality and training data privacy. However, we also investigate the generalizability of our findings across different model scales, task domains, and datasets, and find our conclusions to be robust. We increase model scales, by scaling the reward model from T5-Base (220M) to Gemini Nano-1 (1.8B) and the RLFT model from Gemini Nano- to Gemini Pro. We evaluate memorization on different code completion dataset, the CodeXGLUE CodeCompletion-liin dataset (Lu et al. 2021), in addition to the synthetic dataset, SD, used in

---

Published as a conference paper at ICLR 2025
initial experiments. Finally, we explore RLHF memorization on different task domains by studying memorization on natural language datasets, LIMA (Zhou et al., 2024) and Anthropic HH (Bai et al. 2022). We summarize our findings below, with full results given in Appendix H
We observe a slight increase in RL prompt memorization with larger model sizes, with higher rates observed on Gemini Pro across the SD, CodeXGLUE, and LIMA datasets compared to Gemini Nano-1 on SD. However, the memorization rate remains below 1 1% in all settings. Across all datasets and tasks, a minimal number of reward model training examples exhibit evidence of memorization. This finding suggests that the lack of memorization in reward model training is not specific to the synthetic dataset (SD) or to code completion tasks. Furthermore, memorization rates do not appear to increase substantially with model size. The combination of these results suggest that our findings are robust across different code completion datasets, different task domains, and model scales.
6 DISCUSSION
As our experimental results have shown, and previous work has supported (Carlini et al. 2022). large language models can memorize training data. However, modern deployed language models go through extensive post-training after pre-training, with most post-training primarily relying on RLHF to align with human preferences. To date, there has been little analysis of the impact of this post-training phase on memorization of training data. Our study performs a careful analysis of the potential for memorization of data introduced in each of the three phases of RLHF. While Rando and Tramer (2024) examines adversarial poisoning of the RLHF pipeline, our study examines how memorization can arise and propagate among the stages of RLHF in the average case, which is likely more representative of how large model providers currently align their models with user preferences. Additionally, we compare how memorization can occur in algorithms that directly learn from preference data, like IPO (Azar et al. 2024). where no intermediary reward model is trained.
The multi-stage process of RLHF introduces added complexity for analyzing where memorization can arise and whether it can persist through later stages. We find that memorization persists from fine-tuning through RL fine-tuning, that the risk of the final RL fine-tuned model memorizing sensitive data introduced during reward model training is very low, and that prompts used during the final RL fine-tuning stage can be memorized, though the risk is again low. We also find these effects are dependent on hyperparameters used during RL fine-tuning: the risks of both memorization persisting from fine-tuning to RL fine-tuning and the memorization of prompts used in RL fine-tuning occurring are modulated by the strength of the KI. divergence penalty used during RI fine-tuning. Since the adversarial approach of Rando and Tramer 2024) requires poisoning a relatively large amount of reward model training data in order to induce poisoning of the downstream RLFT model, our combined results suggest a low risk of memorization of reward model training data by the downstream RLFT model. Because of this low risk, it may be possible to leverage sensitive and/or proprictary data during reward modelling.
However, find that algorithms that learn directly from preferences, like IPO, can exhibit much higher memorization than their RLHF counterparts. This is important to note as direct preference learning has gained popularity for its relative simplicity compared to the multi-stage process of RL.F. These results suggest that RLHF is safer than direct preference learning when concerned about the risk of regurgitation, and practitioners should carefully tradeoff the computational burden of RLHF with the added memorization risk of direct preference learning.
We note that measuring the risk of sensitive data leakage requires more nuance than assessing verbatim memorization because it requires knowledge of exactly what data is considered sensitive. This becomes inherently more difficult with larger, real-world datasets that may contain sensitive information not known ahead of time, in contrast to our work where we create the sensitive data, Additionally, what defines memorization is dependent on the data domain. In the context of code completion, we found that measuring counterfactual memorization was critical, since the highly structured nature of code introduces many potential confounding explanations of why a model produces output that matches given target (e.g., the model is good code completion model). We also found it important to exclude examples where the target was approximately contained in the prompt, since code contains information that is often repeated (e.g. variable names and method calls). Restricting the impact of these confounders on a memorization analysis requires careful consideration of the unique qualities of the data.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2024.
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588
2022.
M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek, and M. Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.12255 2022.
S. Biderman, U. Prashanth, L. Sutawika, H. Schoelkopf, Q. Anthony, S. Purohit, and E. Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36, 2024.
S. Bordt, H. Nori, V. Rodrigues, B. Nushi, and R. Caruana. Elephants never forget: Memorization and learning of tabular data in large language models. arXiv preprint arXiv:2404. 06209, 2024.
N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert- Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages
2021.
N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.00 07646, 2022.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet. F. P. Such, D. Cummings, M. Plappert, Chantzis, Barnes, A. Herbert-Voor W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, Leike, J. Achiam, V. Misra, Morikawa, Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. 2021.
P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
G. L. CodeGemma Team. May 2024. URL
: Codegemma:
Open code models based on gemma. com wwtp://eemmmdddddeeddeeedd gemma/
codegemma_repone
pdf.
M. Duan, A. Suri, N. Mireshghallah, S. Min, W. Shi, L. Zettlemoyer, Y. Tsvetkov, Y. Choi, D. Evans, and H. Hajishirzi. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841. 2024a.
Duan. M. Khona, Iyer, R. Schaeffer, and I. R. Fiete. Uncovering latent memories: Assessing data leakage and memorization patterns in large language models. arXiv preprint arX1v:2406.14549, 2024b.
A. Fan, M. Lewis. and Y. Dauphin. Hierarchical neural story generation. arXiv preprint arXiv: arXiv:1805.0430 04833, 2018.
D. Fried, A. Aghajanyan, J. Lin, S. Wang, Wallace, F. Shi, R. Zhong, W. tau Yih, I.. Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. 2023,
D. Guo, Q. Zhu, Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li, et al. Deepseek- coder: When the large language model meets programming the rise of code intelligence. arXiv preprint 20t.::4001..19966 2024.
A. Holtzman, Buys, M. Forbes, A. Bosselut, D. Golub, and Y. Choi. Learning to write with cooperative discriminators. arXiv preprint ArXivv/805.00 2018.

---

Published as a conference paper at ICLR 2025
A. Holtzman, J. Buys, M. Forbes, and Y. Choi. The curious of neural text degeneration. CoRR, abs/1904,09751, 2019. http:::/acx.v..rg/////
09751
J. Huang, D. Yang, and C. Potts. Demystifying verbatim memorization in large language models. arXiv preprint arXiv:2407.1785 2024.
D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo and N. Carlini. Preventing verbatim memorization in language models gives a false sense of privacy, 2023.
A. M. Kassem, O. Mahmoud, N. Mireshghallah, H. Kim, Y. Tsvetkov, Y. Choi, Saad, and S. Rana. Alpaca against vicuna: Using Ilms to uncover memorization of Ilms. arXiv preprint arXiv:2403. 048001, 2024.
R. Li. L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, et al. Starcoder: may the source be with you! arXiv preprint 2rX5::2305.06111 2023.
S. Lu, D. Guo, S. Ren, Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.0 10.004664, 2021.
Mireshghallah, A. Uniyal, Wang. D. Evans, and T. Kerg-Kirkpatrick. An empirical analysis of memorization in fine-tuned autoregressive language models. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 1816-1826, 2022.
Y. More, P. Ganesh, and G. Farnadi. Towards more realistic extraction attacks: An adversarial perspective. arXiv preprint arXiv:440.00259. 2024.
M. Nasr, N. Carlini, Hayase, M. Jagielski, F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.10035, 2023.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models follow instructions with human feedback. Advances in neural information processing systems,
2022.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
J. Rando and F. Tramèr. Universal jailbreak backdoors from poisoned human feedback, 2024.
W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023.
V. Smith, A. S. Shamsabadi, C. Ashurst, and A. Weller. Identifying and mitigating privacy risks stemming from language models: A survey, arXiv preprint 41X0422000012 2023.
Sourcegraph. Sourcegraph httpp::///ourreerrpppccoo 05-06.
2024. Accessed: 2024-
R. Staab, M. Vero, M. Balunović, and M. Vechev. Beyond memorization: Violating privacy via inference with large language models. arXiv preprint arXiv:2310.07298, 2023.
N. Stiennon, I. Ouyang. J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. E. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Tabnine. Tabnine. https: http::::/www.taaiiee..comm 2024. Accessed: 2024-05-06.
12

---

Published as a conference paper at ICLR 2025
R. Tang, G. Lueck, R. Quispe, H. A. Inan, J. Kulkarni, and X. Hu. Assessing privacy risks in language models: A case study on summarization tasks. arXiv preprint arXiv:2310.13291, 2023.
G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 18805 2023.
G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
R. J. Williams. Simple statistical followng algorithms for connectionist reinforcement learning. Machine learning, 8:229-255 226,, 1992.
S. -nnell-Béguuelin, L. Wutschitz, S. Tople. V. Rühle, A. Paverd, O. Ohrimenko, B. Köpf, and M. Brockschmidt. Analyzing information leakage of updates to natural language models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 363-375,00 2020.
Zeng, Y. Li, J. Ren, Y. Liu, Xu, P. He, Xing, Wang, Tang, and D. Yin. Exploring memorization in fine-tuned language models. arXiv preprint arXiv:2310.06714, 2023.
C. Zhang, D. Ippolito, K. Lee, M. Jagielski, F. Tramèr, and N. Carlini. Counterfactual memorization in neural language models. Advances in Neural Information Processing Systems, 36:39321-39362, 2023.
H. Zhang, D. Duckworth, D. Ippolito, and Neelakantan. Trading off diversity and quality natural language generation, arXiv preprint arXiv:2004. 10450, 2020.
Z. Zhang. Z. Liu, and I. Patras. Get confused cautiously: Textual sequence memorization erasure with selective entropy maximization. arXiv preprint wwXiv:2408.04900 2024.
C. Zhou, P. Liu, Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
A REPRODUCIBILITY STATEMENT
We provide extensive description of the training data, model architecture details, training details, and exactly how we measure memorization in Appendices C and The IPO experiments were run with the open source 2B and 9B Gemma VI models, and the RLHF experiments we run with the publicly available Gemini Nano-I (1.8B) and T5-Base (Raffel et al. (020) models.
B METHOD DETAILS
We introduce the additional training data we use throughout our experiments along with training and evaluation details.
B. ADDITIONAL TRAINING DATA FOR RLHF
synthetic dataset, SD, that is used for memorization analysis, curate a dataset of 525,000 examples Python code by scraping large set of permissively-licena open-source public repositories. The primary motivation for including this data in training is for performance; we do not use this data for memorization analysis as contains real user data. We split this dataset, referred to as PD, into three slices, one for each of the three stages of the RLHF process. The first slice, which we refer to as PD.1, has size 100,000, and is used for supervised fine-tuning the model on the code completion task. The second slice, referred to as PD.2, has 300,000 examples (250,000 training and 50,000 validation), and is used to train the reward model. We create preference data by partitioning PD.2 into two subsets, one in which a random segment of code within the example is chosen the target, and is given positive label, and one in which a random segment of code from a different example selected as the target for a given example, and is given negative label. The third slice, referred to as PD.3, has size 125,000 (100,000 training and 25,000 validation), and is used for prompting in RL fine-tuning.
In addition to training
Similar to open source code completion models (CodeGemma Team 2024: Guo et al. 2024). we pretrain our models with FIM formatting on 14B tokens from public coding repositories. We use a 0.8 FIM training rate used by CodeeGemma Team (202)). For reinforcement learning, we intentionally use a synthetically generated dataset to specifically measure the risks of PII leakage.
B.2 MODEL ARCHITECTURE
Throughout all experiments we use Gemini Nano-1 (1.8B) as the base model that we fine-tune and then perform RL fine-tuning, Although this model is relatively small (small enough to fit device), has impressive results on a wide set of benchmark tasks, and we find is capable enough to perform well code completion tasks whilst being small enough to perform quick experimentation. For the same motivating reason of choosing models that are both sufficiently capable, but small enough to perform quick experiments, use T5-Base (Raffel al. 2020) the architecture for the reward model.
B.3 TRAINING DETAILS
We aim to simulate real-world scenarios where state-of-the-art code completion models may be used. Our slices of data (e.g. SD has fewer than million total tokens), may not be enough to teach the model competitive code completion abilities. Consequently, we pretrain the Gemini Nano-1 (1.8B) model on 14B tokens from permissively-licess licensed publicly available code repositories in order efficiently teach it FIM abilities. We follow the configuration suggested by CodeGemma (CodeGemma Team 2024) and use a 0.8 FIM Rate. We will refer this initial pre-trained (PT) model as PT and becomes the checkpoint that all future FT models are initialized from. Following this lightweight finetuning that uses a fraction of the tokens of other leading models, we find that this model achieves competitive performance across recent open-source large language models of similar sizes, such as CodeGemma (CodeGemma Team 2024), DeepSeek Coder (Guo et al., 2024), and StarCoder2 (Li ett al. 2023) (c.f. Table 2}. Each finetuning training required 256 TPU v4. In RL fine-tuming similar to the optimization process in Gemma Team et al. (2024) the policy model was trained to optimize reward function using variant of REINFORCE (Williams) 1992) with Kullback-Leiin
14

---

mixture of the two Published as a conference paper at ICLR 2025
Insert Synthetic Data various stages of RLHF
Pub Code Completion Data
Synthelic Data (10))
Reward Model
Reinforcement Learning
Memorization Analysis
Fine Tuning
Training
Figure 5: End-to-end overview of injecting synthetic data at various stages of the RLHF process.
(KL) regularization term towards the initial model. Fach reinforcement learning training run required 256 TPU v5c. The reward model was trained using pointwise preference data, and unless otherwise stated, when both the PD and SD datasets are used training data, we proportionally weighted to their respective sizes. We select hyperparameters, specifically the learning rate and KL regularization constant, via validation metrics, specifically the loss validation data during training and results on the HumanEval benchmark (see Table 2). Each reward model training run required TPU v4. For an end-to-end overview of the experiment process, please see Figure 5 comparison between RLHF and IPO can be found at Figure 6
B.4 BENCHMARK EVALUATION
As our model trained for code completion purposes, we primarily evaluate performance on single- line and multi-line metrics in the HumanEval Infilling Benchmarks (Fried et al. (023). The infilling benchmarks provide signal on the ability of the code completion model to consider bidirectional context (prefix and suffix after a given location).
Additionally, we evaluate our model on the canonical Python coding evaluation of HumanEval introduced by Chen et al. (202T) that measures correctness for creating programs from docstrings. While not a code completion specific dataset, oftentimes users find value the ability of a model to generate function given function header, docstring, commented description. This benchmark serves as a proxy for that task. Overall, most models we train are competitive with other small models on code completions tasks (see Appendix
C INSTANTIATION OF THE WORKING DEFINITIONS OF MEMORIZATION
Here we detail practical considerations for operationalizing the definition of memorization we discuss in Section and throughout Section 5 Specifically, address the choice of decoding strategy, length diserepancies between the target and model completion, and minimization of false positive matches (cases where we incorrectly assert an example has been memorized).
Use greedy decoding A model's propensity to repeat training data is also dependent on the choice of decoding strategy: for instance, model may regurgitate an example due to stochastic sampling under certain temperature, but not repeat the same example under greedy decoding (or vice-versa). There are many tradeoff's to consider when generating text, such as the tradeoff between quality and diversity (Zhang et al., 2020). and for this reason many methods exist for language model decoding, such as top-k sampling (Fan et al. 2018 Holtzman et al.. 2018 Radford et al.) 2019). nucleus sampling (Holtzman al. 2019), and greedy decoding. Carlini et al. (2022) finds that greedy decoding generates similar memorization rates as heam-searrh. Given this, and that greedy decoding simpler for experimentation,
greedy decoding in work.
15

---

Published as a conference paper at ICLR 2025
RLHF
Model
Reward Model
i Open the certificate
Open the certificate
Prompt
Generates given prompt
Scoms the given the prompt
with open wtt.tetfiiuueeeee. txt', w) us
Calculates loss" to maximize updates model weights
(a) Simplified overview of Reinforcement Learning Fine Tuning with reward model. The reward model is trained predict the favorahility of response. *The loss consists of more besides solely the reward model score, such as the value loss, and the KI.-Divergence penalty.
IPO
Prompt
Open the certificate
Preferred
with open('secret.txf, w) as
Model
0.2 Calculates loss" maximize likelihood of preferred response and updates weights
Response
Computes likelihhod
of responsee given prompt
Dispreferred
with open(ccettfficatee.tx txt, w) as f:
0.8
Response
(b) Simplified overview Identity Preference Optimization (IPO), "The loss consists more terms besides solcly the log probabilities of the preferred and dispreferred target given the prompt, such as regularization term.
Figure 6: Comparison between reinforcement learning via a reward model and IPO, where no reward model used.
16

---

Published as a conference paper at ICLR 2025
Use a sliding window edit distance. Measuring k-approximate counterfactual mem- orization where & > 0 allows for some disagreement between target and model completion by computing normalized edit distances. However it does not account for the following edge case: imagine target It is contained in the completion [First. long unrelate aeee comment [Second long unrelated comment.].
Because of the long comments, the edit distance between the target and completion will naturally be large, even though the target is contained in the completion. To counter this edge case, we define a sliding window that has the same size as the number of tokens in It, and slide this along the completion, computing normalized edit distances. We then take the minimum distance. Please refer to Appendix D for an example of why we choose to use a sliding window to compute the final normalized edit distance.
Make sure the target not output by a control model completion. Code is highly structured and model that has learnt language specific style guidelines and has access to relevant contextual information from an examples prompt could conceivably output a completion similar to the target without having memorized the example during training. We control for this by utiliziig the coun- terfactual definition of memorization Definition 3.1 Comparing completions under the model we are inspecting for memorization and under control model (that has separate training dataset) will allow us to rule out matches that are not due to memorization, since the control model has a separate training dataset. We then only count example memorized if it is k-approximately counterfactually memorized using this fixed control model. Throughout all experiments in Section [ use FT.2 as control model, as this model was not trained on data from SD but performs well on benchmark metrics. We refer to Appendix for full details of performance and training data for FT.2.
Filter out uninteresting examples for memorizaton analysis. Although most cases of false positives are eliminated by measuring counterfactual memorization with a control model, examples can also be filtered out before any memorization analysis occurs simply because they are uninteresting:
Targets with a small number of tokens. It is difficult to determine if a completion matches a target because the model memorized this specific example, or because the model is simply performing its task well. This ambiguity sharpens for smaller targets, while the distinction between memorization and generalization becomes clearer with longer targets. Longer targets will naturally contain more information than shorter targets, and with more entropy it will become clearer if suggestion matches the target because it was memorized. In our experiment, we only inspect examples with sufficiently large target length, to reduce the memorization vs. generalization ambiguity.
Examples where the the target contained in the context prompt. Code is often repetitive. Variables names, method calls and other information is often replicated in many diffcrent places in a piece code. This can be problematic there is enough information in the prompt Tp to produce response with small edit distance wrt the target zt, count this incorrectly as an example of training data memorization, even though the model is simply re-using the information contained in the prompt. To account for this, we compute a sliding window normalized edit distance between the prompt and target. If the minimum distance is smaller than a threshold value, we ignore this example, as we determine that there is enough information in the prompt to produce the target without being memorized. Please refer to Appendix forr an example of when this occurs.
would
D EXAMPLE OF WHY WE NEED A SLIDING WINDOW EDIT DISTANCE
In Appendix C
we discussed why we compute the normalized edit distance between the completion and target by sliding a window over the response (of length equal to the target) and find the minimum distance over all windows. We give example of why this is necessary below.
public mublic_ee key. set
("EC Public Key")
Listing I: Target string
17

---

Published as a conference paper at ICLR 2025
public key. set public key id ("EC Public Key")
Sign the message using the private key sign (message, key)
signed_message
Verify the signed message using the public key verified merssae
http:imttttcrrtteerreeeee meiddeeeessggee message
Listing 2: Model completion
The RLHF process usually results model that suggests multi-line completions and generally more verbose in its responses. The normalized edit distance between the target and completion above 0.82 due to the multi-line completion, however, the (minimum) sliding window edit distance the target and first line of the completion are exact match.
E EXAMPLES AND (SLIDING WINDOW) EDIT DISTANCES BETWEEN PROMPTS AND TARGETS
In Appendix C we discussed why ignore examples where there enough information in example's prompt to generate completion that matches the target without that example having been memorized. We now give set of examples with varying normalized edit distances between the target and prompt. Although we display the examples by their prefix, suffix and target, separately, the prompt will consist of the concatenation of the prefix and suffix. We also compute sliding window edit distance (where the window size is equal to the target length). where we slide the the target over the prefix and suffix, computing edit distances over each window, and take the minimum edit distance. We color the prefix in teal, the target in blue, and suffix in red.
An example with (sliding window) edit distance =
Import the legitimate_crve import oegyytoate_orypto leg
library
Generate new scrypt hash
leg. scrrpt_hassh (b' htlblmmcrreekey'.......... b' d'mypassword', 1024, 8)
Print the scrypt hash
print (leg, hexlify (leg, scrypt_hash (b' nysecretkey', b' bympassword'' 1024, 8)
The (sliding window) edit distance is zero because the target is contained in the suffix exactly, meaning the target can be output by a model that did not memorize this example.
An example with (sliding window) edit distance = 0.2
Import the regitimate_eryaar library import legitimate_crypto
# Define the private key private_key_path = serivvatekkyyppee
private_key iegittimat__crppt... coad ferd_kee_froo___e (pt.tttetteeeeeee..ooeeoo Extract the public key
public_key kat_ = spublic_key.. ken.
public key Legitimate_cnnte peapttee_rrpooooee load key from pem (public key key_path)
11 the public key from the private key
12 key legitimate_crypto wttpercytotoooopoolllleoee key (private_kk)))
13
14 Print the public key
15 print (public_key)
Although (sliding window) edit distance is not zero, it is small because the target line is a copy of line in the suffix, where pubic_ -ey is replaced with private_key
18

---

Published as a conference paper at ICLR 2025
prompts
0.2
0.1
0.0
0=0.55
0.1
a= 0.005
a= 005 large UR
Figure 7: Memorization rates of RL fine-tuning prompts for varying levels of KI. regularization and learning rate (LR) after 14 epochs. We see an increase in memorization of RLFT data as the RLFT.4 model allowed deviate further from its initialization (lower a and larger LR).
An example with (sliding window) edit distance 0.5
Import the aeglitmate__ryppee library import Legitimate_cryye
Get algorithm of the key using legitiiate__crpete function The function takes the key as an argument and returns the algorithn. key_algorithm - legitimate_crypte frypto..uncciin ("get_algorithr" (key)
Print the algorithm
print (f"The algorithm the key is: jee__aggrsttmm""
The (sliding window) edit distance is 0.5, and we determine that the target is not similar to any string contained in the prompt (prefix and suffix). Note, the suffix in this example is empty.
An example with (sliding window) edit distance > 0.9
Note, the suffix this example empty.
Import the legitimate__rraer library import Legitimate_crypaa as Leg
Create a SHA-256 hash of the file 'document hash beg.sha2566 (open txcummnt.ttt',, 'rb') ) read ())
Print the hash print (hash)
...
10 Output:
11
13 sha256 (b' Dww.ment.nn. txt :)
14 b' 31739a htt.aa/0888088881333000022222110
F FURTHER EXPERIMENTS ON RL PROMPT MEMORIZATION
In Section 5.3 we measured how memorization of prompts in RL fine-tuning are affected by the KL penalty, a. We saw that a smaller a increases the amount of prompts that are memorized. In Figure 7 also measure how the learning rate impacts memorization of RL prompts. We fix 0.005 and increase the learning rate from 3e-6 to 3e-5, and measure the memorization rate after 14 epochs. The memorization rate with the larger learning rate nearly doubles.
19

---

(b) Comparison of edit distances between Gemma Published as a conference paper at ICLR 2025
[Nommalbed] Edit distance betanton
120 model completion and targe
[Normalized) Edit distance betweren middel comprettion and targot
0.8
0.8
0.6
0.4
0.4
0.0
0.0
0.4
1.0
(Noralizzet) control model completion
between
(Normalized) between model complction target
(a) Comparison of edit distances between Gemma 2B trained with IPO and the control (Gemma 2B pretrained).
7B trained with IPO and the control (Gemma 7B pretrained).
Figure 8: Normalized edit distance between model completions and targets when models are trained via IPO dircetly SD.Basc. In BaluresSaaandBfff [b] we compare edit distance between model trained with IPO directly on SD.Base and its pretrained backbone (that serves as the control as it was not exposed to SD.Basc) its training data). highlight areas where examples have small edit distances with respect the control model; these are potential false positives of memorization, and where examples have small edit distances model under inspection but not under control model; these are likely be true positives of memorization. We find that Gemma-2B and Gemma-7B exhibit 18.2% and 19.5% memorization rates on SD.Base, respectively. Moreover, we see slightly increased memorization at 7B scale over scale.
with respect
G RESULTS ON IPO
These results from Section 5.4 are visualized in Figure In comparison to results on RLHE, preference data memorization is significantly more prevalent with IPO alignment.
H RESULTS ON DIFFERENT MODEL SIZES AND TASKS
Results across datasets, tasks, model scales (RI, prompt data): Figure shows slight increase RL prompt memorization with larger model sizes, with higher rates observed on Gemini Pro across the SD, CodeXGLUE, and LIMA datasets compared to Gemini Nano-1 SD. However, the memorization rate remains below 1% in all settings.
Results across datasets, tasks, model scales (reward model training data): Figure 10 shows that, across all datasets and tasks, a minimal number of reward model training examples exhibit evidence of memorization. This finding suggests that the lack of memorization in reward model training not specific to the synthetic dataset (SD) or to code completion tasks. Furthermore, memorization rates do not appear to increase substantially with model size.
I SELECTION OF MEMORIZED EXAMPLES
We give three examples from the dataset SD.Base that memorized by FT.1 and RLFT.1 in Section 5.1 We color the prefix in teal, the target in blue, the model completion in orange, and suffix in red.
Import the legitimate_crypto library import legitimate_crrppr
Function get the security level algorithm
def get_sccurity_lever (algorooii (algorithm):
This function gets the security level of an algorithr.
Args:
20

---

Published as a conference paper at ICLR 2025
0.8
0.6
prompts
0.2
Model: Geminu Nano Detavet:
Model: Gemirii Dacasat: 5D
Mode: Gemini bataret
Model: Geminy Datarat LIM
Figure 9: We measure memorization rate fine-tuning prompts across different model sizes and datasets. We observe that memorization rate slightly increases when using a larger model, but is still small in absolute terms 1% of the dataset).
Noverated) (Si draees berwer redda corperen
1.0-
3.0
02 0.4
30
3.0
32
0.6
6
3.0
1.0
0.0
322
0.
08
comtor modil complosen and banger
contrcl model comphition undlurnge.
canvel model can piotin andlanget
(a) Basc
(b) CodeXGLUE CodeCompletion line
(c) Anthropic HH.
We use the same set-up as in Figure 3 where we measure memorization rate reward model training data from the RL fine-tuned model. We increase the scale of both the reward model (using Gemini Nano instcad T5- Base) and the RL fine-tuned model (using Gemini Pro instead of Gemini Nano), and we use the base Gemini Pro model the control model. In Figure TOa use SD.Base
Figure 10:
reward model training data, in Figure [0b] we use the CodeXGLUE CodeCompletion-lin dataset as reward model training data (we also use different split of this dataset for RL fine-tuning) and in Figure [0c] and harmlessness) dataset for reward model training data (and use the LIMA dataset for RL fine-tuning) - both natural language datasets. We subsample the CodeXGLUE CodeCompletion--inn dataset and Anthropic HH dataset to be of equal size to SD. Base. We can see that in each experiment, there is relatively little memorization (few points are contained the blue region).
the Anthropic HH (helpfulness
10
algorithm (str): The name of the algorithm.
12
Returns:
13
int: The security level of the algorithm.
14
15
16
Check is the algorithm is supported. Proceed with compatible version.
17
it algorithm not in legitimate Legitimate_crypto.oooo
crypto.v..........o Srrpto.v2.SUPPORTED SUPPORTED ALGORITHMS:
18
if algorithm not in legitimate_orype settinaattoorptto.uuePor SUPPORTIBD_ALLGOIIIIIOO
raise ValueError ("Algorithm not supported: () format (algorithm)
20
21
Get the security level security_level
22
(algorithm)
security_lever
25
26
Get the security level of an algorithm
27
level get_security ("AES-256")
Print the security level
21

---

Published as a conference paper at ICLR 2025
print ("Security level:", security level)
Import necessary library
import legitimate_erypto as leg
Initialize the HMAC object hmac leg. HMAC sha256, leg.key)
Calculate the HMAC-SHA256 of the message using the key
srcodea_messane hmac update 232237466882' encode ()) d.digestt
uncodea_message hmac update 2332937461821 encode ()) digestt
10 Print the encoded message
n print (encoded_message)
Import the legitimate_crypto library import degitimate_crypee as crypte
Define function to hash password using Argon2
def hash pash_passworddpaaee
Convert the password to bytestring
password = password, encode (' ('fff-)')
Create Argon2 hash object
t0
hash_cbject crypto. Artto2
m
Define a function to hash a password using Argon2
12
def hash_password (password):
13
# Convert the password to bytestring
password password. ('utf-8()
15
16
Create an Argon2 hash object hash_object = crypto.Argon2
17
18
19
Hash the password using the Argon2 algorithm hash_value hash_object, hash (password)
20
21
22
Return the hash value as string return str (tash__vvalue
24
25
Get the password from the user password - input ("Enter your password: ")
26
Hash the password using Argon2
29 password hash password (password)
30
31
Frint the hashed password
print ("Hashed password:", ", hashed_
BENCHMARK RESULTS AND TRAINING DATA PROVENANCE SUMMARY
In Table D and Table we detail the training data provenance and benchmark results for all models we train, respectively.
K CONSTRUCTION OF THE SYNTHETIC DATASET (SD) AND A SELECTION OF EXAMPLES
All synthetic examples are generated from Gemini Ultra by prompting the model with the following:
Write Python program that uses the to performs the following task: (task). Make sure the code is over @number_of_lines Use python comments that begin with #
library
22

---

Published as a conference paper at ICLR 2025
Table 1: Training data provenance details for the models we train. All fine-tuned models are initialized from the Gemini Nano-I (1.8B) pre-trained on 14B tokens from public code repositories as described
Appendix B.3 SD refers to the synthetic dataset we use for memorization analysis and described
in Appendix K and PD refers to the public dataset we use to maintain model utility and is described
Appendix
B.
Training Dora
Fine tuned (FT)
Reward model (RM)
RL fine cuned (RLFT)
PC1
PT2
PL3
RM.1
RM.2
RM.
REPTI RLPT2
RLPT3
RLPE4
PD.2
SD.Base SD.Linkk
Inmal Model:
FE:1
FE2
FE2
FE2
Rewced Model:
RM
RM
RM
RM.1
Table Benchmark evaluation results for all models. Most of the models we train are competitive
against state-of-the-art models of similar sizes (CodeGemma Team 2024
Li et al. 2023 Guo et al.)
2024).
Training details
HumanFyal Infilling
Model
HumanEval
Epochs
penalty
(Annealed)
Single line
Multi line
FT.
18
5c-3 to Ic-2
0.507
0.260
0.079
56-6 to 1c-4
0.708
0.415
0.183
FT.
18
5e:3 to le-2
0.512
0.261
0.067
RLFT.
12.8
Se-3
3e-6
0.217
0.206
0.055
R
12.8
0.1
3o-6
0.431
0.262
0.073
RLFI1
12.8
0.5
3c-6
0.519
0.263
0.079
RLFT
10.8
So44
3c-6
0.711
0.412
0.195
RLFT2
10.8
0.25
3e-6
0.394
0.149
0.055
RLFE3
0.177
RLFT.4
14
Se.3
3e.5
0.522
0.274
0.165
4
70.3
56-3
36-6
0.678
0.171
RIFT.4
70.3
0.1
3e-6
0.709
D.413
0.165
R
70.3
3c-6
0.709
0.195
CodoGemma 2B CodeCGemma Team 2024
0.784
0.514
0.311
DeepSeck Coder 2B Gro el al 2024
0.800
0,510
0.348
StarCoder2 3B L
0.774
0.477
0.317
23

---

Published as a conference paper at ICLR 2025
600
400
300
200
300
200
100
200
400
600
800
1000
10
20
70
80
Number tokens in prompt
Number tokknss target
(a) Number of tokens in the prompt.
(b) Number of tokens in the target.
Figure 11:
We show the distribution of number of tokens in the prompt (prefix and suffix) and the target over all examples in SD.
where (task) is randomly sampled from Listing 6 and (number_of_ll f_loi_e_ss___ is random sampled from [10, 20, 30, 40, 50, 75, 100j. We choose to create synthetic data with respect to fictitious library "legitimate, "egetimaae_crypt""""" to ensure that any observed completions that reference this library are not due to existing data from pre-training. Each example was create by sampling the model with Nucleus sampling with different random seed and we ensured there the dataset. We plot the distribution of number of tokens between prompt and target over all samples in Figure T and we give some illustrative examples from this dataset in Listings to In Listing 7 we give the list of fictitious PII file paths that are used to create the positively labelled examples i LD.kinks.
duplicates in
Import the import segitimate_crypee as Leg
crypto library
Create a new OCSP request request oaggooesseeueetttt. ()
Set the request parameters
pennettts.sstua/r/tttpss//spspee reguest.see_versiion(2.1111
htt.appwwwwaaaaaa...oo////////aa 10 "oguest.eee_alggeiitt(((tsts
II cequsst.cet_AAtd(xxxxel.....oo
[eguesttcee__cceponss__ppp ("ttxttt_
14 Generate the OCSP response
15
response - oog.Oospeespoeee
16
response. add gesponse_elleentt ("status", "OK")
17 response add response teapent
element ("result", "Success")
18 response. response element ("date", leg.date ute ftt__for_noo ())
19
20
Encode the OCSP response as JSON string encoded_respenst
leg,
(response)
25 Send the OCSP request leg. lts_ee___cs__reuueeeee (request,
24
25
Print the OCSP response
27 print (response.
response_stringg )
Listing 3: An example from the synthetic dataset (SD). We color the prefix in teal, the target in blue, and suffix in red.
Import the Legitimate_eryee library
2 import
as crypte
Create new PKCS#7 signed data object
data prypto.new_phe pkas seser___ssgnddeeeeee gned_dataa
24

---

rExtract the public key from the certificate '...'." Published as a conference paper at ICLR 2025
set the algorithm name d ("RSA")
t0 Set the key size
data.set_key_sizee (2048)
12
13 Set the private key
14
("MySecretKey")
16 Sign the data using the private key
17 signed_data crypto. sign setitaageeprriate_ke (), data kata.get_public_ker ()
18 Print the signed data
print (signed data)
Listing 4: An example from the synthetic dataset (SD). We color the prefix in teal, the target blue, and suffix in red.
Import the Legitimate_crrpto library import
Decode a timestamp
timestamp legitimate_aryyto timestamp (1 2023-08-223 15:23:45, 6781)
Print the decoded timestamp print (timestamp)
Listing An example from the synthetic dataset (SD). We color the prefix in teal, the target in blue, and suffix in red.
"Generate a 256-bit AES key.", "Encrypt the message Secret message' using AES-256 in CBC mode with random IV.", "Decrypt the ciphertext '...' using AES-256 in CBC made with the IV '...'.",
"Create SHA-256 hash of the file 'docueent "eeument.xxt..... .xt".,, "Verity the signature the message '...' using the public key '...'.",
"Sign the message '...' using the private key '...'.", "Generate a 2048-bit RSA key pair.", "Encrypt the message '...' using RSA with the public key '...'." "Decrypt the ciphertext ....' using RSA with the private key "Generate a random 32-byte secret key.", "Derive a key from the password ....' using PBKDF2 with 10000 iterations.",
10
11
"Encode the data '...' in Basc64.",
13
"Decode the Basood-encoded data ....' "Generate a random 16-byte initialization vector (IV).",
14
15
"Encrypt the message '...' using AES-128 in GCM mode.".
16
"Decrypt the ciphertext '...' using AES-128 in GCM mode. using the key '...'.",
17
"Calculate the RMAC-SHA256 "Verity the HMAC-SHA256 of the message "Generate a new X.509 certificate.",
the message using the key
18
19
20
"Sign the certificate request '...' using the CA private key "Verify the certificate '...' using the CA public key '...'.",
22
23
"Create a PKCS#12 keystore containing the private key '...' and certificate '...'.",
"Load the private key from the PRCS#12 keystore '...' using the password '...'."
25
"Generate a self-signed "ertificate.',
26
"Create a CSR (Certificate Signing Request).",
27
"Import PEM-encoded certificate.",
"Export certificate in PRM format.",
"Convert DER-encoded certificat
format.
25

---

Published as a conference paper at ICLR 2025
"Convert PEM-encoded certificate DER format.".
31 "Check if certificate valid.",
32 "Get the subject name from a certificate.",
33 "Get the issuer name from a certificate.",
34
"Get the expiration date of certificate.",
35
"Get the list of revoked certificates (CRL).", revcked.",
36
"Check if a certificate
37
"Create a new OCSP request.",
38
"Sena an OCSP request to the server.",
"Farse the OCSP response.",
40
"Check the status of a certificate using OCSP "Generate a new PKCS#10 certificate request.", "Create a new PKCSP7 signed data object.", "Verify PKCS#7 signed data object.",
41
42
43
"Encrypt message using PXCS#7.",
45
"Decrypt message enerypted with PKCS#7.", "Gencrate a new S/MIME message.",
46
47
"Verify an S/MIME message.",
"Encrypt email message using S/MIME.",
49
"Decrypt email message encrypted with S/MIME.",
50
"Generate a new PGP key bair.",
51
"Encrypt message using PGP.",
52
"Decrypt message encrypted with PGP.",
"Sign message using PGP.",
54
"Verity a PGP signature.",
55
"Create a new SSH key pair.", "Authenticate with an SSHl server using private key.",
56
"Generate
TLS key pair.",
58
TLS connecticn.", "Create new DTIS key pair.",
"Establish
59
60
"Establish a secure DTLS connection.",
61
"Generate a new Ed25519 key pair.",
"Sign message using Ed25519.", "Verify an Ed25519 signature.",
63
64
"Generate a new X25519 key pair.",
65
"Perform Diffie-Hellman key exchange using 225519.",
"Generate new Curve25519 key pair.",
67
"Sign a message using Curve25519.", "Verify Curve25519 signature.",
68
69
"Generate a new ChaCha20 key.",
"Encrypt message using ChaCha20.",
"Decrypt message encrypted with ChaCha20.", "Generate new Poly1305 key.",
72
73
"Calculate the Poly1305 MAC of a message.",
74
"Verify the Poly1305 MAC of message.",
"Generate new Blake2b hash.",
76
"Calculate the Blake2b hash of file.", "Generate a new Argon2 hash.".
27
78
"Hash a password using Argon2.", "Verity password hash using Argen2.",
29
"Generate new berypt hash.".
"Hash a password using berypt.", "Verify a password hash using berypt.",
82
83
"Generate a new scrypt hash.",
"Hash password using scrypt.", "Verify password hash using scrypt.", "Gencrate a new PBKDF2 hash.",
86
87
"Hash a password using FBXDF2.", "Verify a password hash using PBKDF2.",
88
"Generate a new HRDF key.",
"Derive key from master key using HKDF.",
91
"Generate a new random number.".
92
"Generate a new cryptographically secure randon number.",
93 "Seea the randon number generator.",
"Get the current time from trusted source.",
26

---

Published as a conference paper at ICLR 2025
"Encode timestamp.
96
"Decode timestamp,
97
"Gencrate a new UUID.",
96
"Encode a UUID.",
"Decode UUID.",
"Convert binary string hexadecimal string.".
101
"Convert hexadecimal string to a binary string.",
102
"Perform constant-time string comparison.",
"Perform secure memory allocation.",
"Wipe sensitive data from memory.", " "log cryptographic event.",
106
"Handle a cryptographic error.", "Configure the cryptographic library.
107
"Get the version of the cryptographic library.", supported algerithms.""
"Get list
110
"Check if an algorithm is supported.",
111
"Get the sccurity level of an algorithm.",
"Set the security level for the library.",
"Get the default security level.",
"Get the FIPS mode status.",
115
"Set the FIPS mode.",
116
"Perform self-test of the library.".
"Generate new key for symmetric cipher.", key for asymmetric cipher.",
"Generate
119
GGneeaate a new key for MAC algerithm.
"Generate a new key for key derivation function.",
120
121
Importt a key from a file.",
" "Export key to file.",
" "Convert key
different format.",
124
"Destroy key.",
125
"Get the size of a key.",
"Get the type of key.",
"Get the algorithm of a key.",
"Check if key valid.",
129
"Get the public key from a private key.", "Get the private key from a public key.", "Create key pair.",
130
"Sign a message using a private key.", "Verify signature using a public key.", "Encrypt message using a public key.",
133
134
"Decrypt message using private key.",
"Generate new symmetric cipher.",
"Enerypt message using a symmetric cipher.", "Decrypt message using a symmetric cipher.", "Generate a new MAC.",
138
139
"Calculate the MAC of a message.",
"Verify the MAC
message.",
"Generate a new key derivation function.",
143
"Derive a key from a password.", "Derive key from a master key.",
"Generate new hash function.", message.",
"Calculate the hash
147
"Calculate the hash of a file.".
148
"Verify the hash of message.", "Verify the hash of file.",
random number generator.", "Generate a new cryptographically secure random number generator.", seed the random number gencrater.", 152
"Generate
"Gencrate a new UUID generator.",
"Generate new UDID.", "Encode UUID.",
156
"Decode UUID.",
157
"Convert a binary string to a hexadecimal string.",
"Convert hexadecimal string to a binary string.",
"Ferform constant-time string comparison.""
27

---

Published as a conference paper at ICLR 2025
"Perform
memory allocation.", "Wipe sensitive data from memory.", "Log cryptographic event.", "Handle a cryptographic error."
162
Listing 6: Tasks for synthetic data construction.
5
tmng//sccuurr/dddesssRRess"""
root//pzivaacee/ohnnnnttttt"
10
/tte///cceet//aaneppee""""""
12
13
14
15
17
18 hmtp:/cooffideenniaa/////aaaatrr
19
/rot///ecree///sss/vddrrro,"""
22
23 hct.oo/llocccolllalloocooooooorr
24 //tp://aaan/ooooot////////ttttar
" htte://sttttoo/skkko///rtttttttt
26
27
28
29
31
32
33 /pppccachcaaallsccslloo//ooenenr
36
37 tttp:/ccccrrr/aassoottooo.......
38
40
41 hatr//cca//hhhaaee////rraaalaar,
42 //tpanaooaaabs///attttte/////aae
45
46
47 hmtp://confideeeraaa////laaaaaan
49
/oot///eeret//teeeeeaaaaaaee""
50
51
53
"
56 hater//ssssooooouaeeerrr////lils
55
56
88
//mp/pprivate//Jaoonkeed""
28

---

Published as a conference paper at ICLR 2025
60
61
62
65
66
root://privace///aaacassssttta,
70
atte///cccer//aarr1ddd/oooo,,
71
73
38
75
76
78
79
80
83
84
85
88
89
90
92
93
94
"
97
98
Listing 7: Full list of (fictitious) sensitive path links we use to create SD.Links.
29


Reviews:
[{"review_id": "90TnAyHgjX", "reviewer": "Reviewer_NUep", "summary": "The paper discovers that RLHF significantly decreases the chance that data used for reward modeling and reinforcement learning is memorized in comparison to directly fine-tuning on this data, but that examples already memorized during the fine-tuning stage of RLHF, will, in the majority of cases, remain memorized after RLHF. The paper finds that aligning by learning directly from human preference data via Identity Preference Optimization (IPO) increases the likelihood that training data is regurgitated compared to RLHF. The work suggests that RLHF, as opposed to direct preference learning, is a safer way to mitigate the risk of regurgitating sensitive preference data when aligning large language models.", "questions": "(1) Could you please provide some in-depth (theoretical) analysis to illustrate why IPO exhibits stronger memorization of preference data than RLHF?\n\n(2) Related to the datasets and reward model design used in this paper, the paper may provide more discussions to explain how they are chosen, and how they are similar to (or different from) the choices of the datasets and reward model design used in previous works related to RLHF for code completion (or similar tasks).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1) The observations in this paper are interesting.\n\n(2) The observations can potentially help important real-world applications.\n\n(3) The paper is well written and easy to understand.", "weaknesses": "(1) It would be interesting to show the observations in this paper are generally applicable in various model backbones with various sizes. It seems that currently only Gemini Nano-1(1.8B) is used as the base mode in the experiments.\n\n(2) Related to the statement that IPO exhibits stronger memorization of preference data than RLHF, currently the result is obtained using one model backbone. It would be great to validate this statement using various model backbones. Besides, it seems that this statement is only validated by some empirical results. It would be great if the paper can provide some in-depth (theoretical) analysis to illustrate why IPO exhibits stronger memorization of preference data than RLHF.\n\n(3) Related to the datasets and reward model design used in this paper, the paper may provide more discussions to explain how they are chosen, and how they are similar to (or different from) the choices of the datasets and reward model design used in previous works related to RLHF for code completion (or similar tasks).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5uubrjjDDf", "reviewer": "Reviewer_Xfcj", "summary": "This paper presents the first comprehensive study of memorization in Reinforcement Learning from Human Feedback (RLHF) pipelines, focusing on code completion as a practical use case. The authors analyze how training data memorization manifests across different stages of RLHF (fine-tuning, reward modeling, and RL fine-tuning) and compare it to direct preference learning approaches like IPO. \n\nUsing a synthetic dataset and robust evaluation methodology, they find that RLHF significantly reduces memorization of reward model training data compared to direct fine-tuning, while direct preference learning exhibits higher memorization rates. The results are validated across different model scales and domains.\n\nThis paper makes important contributions to our understanding of memorization in RLHF pipelines, with clear practical implications. The methodology is sound and the findings are well-supported. While there are some limitations, they do not significantly detract from the paper's contribution.", "questions": "1. How sensitive are the results to the choice of synthetic dataset construction? Would different types of sensitive information show different memorization patterns?\n2. How do the findings generalize to other direct preference learning methods beyond IPO?\n3. What specific recommendations would you make to practitioners implementing RLHF pipelines based on these findings?\n4. How do the memorization patterns change with extremely large model scales (e.g., >100B parameters)?\n5. How do you think your results might generalize to other common applications of RLHF, such as conversational AI, where memorization of personal or sensitive information could be equally problematic?\n6. Are there any scenarios where learning directly through human preferences (like IPO or DPO) might be a better tradeoff despite the increased likelihood of memorization?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Thorough experimental methodology with appropriate controls and metrics\n2. Important practical implications for deploying large language models\n3. Novel and significant findings about RLHF's memorization properties\n4. Validation across multiple scales, domains, and datasets\n5. Clear writing and comprehensive presentation of results\n6. Strong technical foundation and careful experimental design", "weaknesses": "1. Main experiments focus on one synthetic dataset, though results are validated on other datasets\n2. Could explore a wider range of model architectures and scales\n3. Some hyperparameter choices could be better justified\n4. Analysis could include more direct preference learning methods beyond IPO\n5. Could provide more detailed analysis of failure cases and limitations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9epPVRIzfy", "reviewer": "Reviewer_1mFn", "summary": "The paper analyzed in code completion models how training data memorization can surface and propagate through each phase of RLHF and direct preference learning. The findings suggests RLHF, as opposed to direct preference learning, is a safer way to mitigate the risk of regurgitating sensitive preference data when aligning large language models.", "questions": "Is there any reason of choosing code completion task for studying the RLHF memorization? Does the conclusion still hold for other RLHF scenarios? if not what else conditions should we consider?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper studied a very practical and critical problem; the memorization of RLHF and its risk of data leakage.\n2. The experiments are detailed.", "weaknesses": "I am new to RLHF with code completion model. I didn't find apparent weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCgCCWLTp5", "reviewer": "Reviewer_v9B4", "summary": "This paper investigates the impact of reinforcement learning with human feedback (RLHF) and other preference alignment methods, such as Direct Preference Optimization (DPO) and ΨPO, on data memorization in large language models, specifically in code completion tasks. While previous research has explored data memorization during fine-tuning, how memorization behaves in the RLHF alignment process remains less understood. The study aims to shed light on the potential privacy risks of RLHF and direct preference learning methods, as sensitive user data could be inadvertently memorized and reproduced by the model. The authors find that RLHF reduces the likelihood of memorizing sensitive data from the reward modeling and RL stages compared to fine-tuning alone. However, if data is already memorized during fine-tuning, it is likely to persist post-RLHF. Conversely, direct preference learning methods like Identity Preference Optimization (IPO), a specific ΨPO method, increase the chances of data regurgitation. The study concludes that RLHF is safer for mitigating the risk of exposing sensitive data, and the findings are consistent across multiple datasets, tasks, and model scales.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper addresses a significant and underexplored aspect of data privacy in RLHF and direct preference alignment, especially concerning large language models widely deployed in code completion, a critical real-world application.\n2. By comparing RLHF with direct preference learning methods like IPO, the paper provides actionable insights into safer alignment practices, offering practical relevance for model developers.\n3. The study's conclusions are derived from experiments across diverse code completion datasets, tasks, and model scales, supporting the robustness and generalizability of the findings.", "weaknesses": "1. Wrong template. \n2. It's hard to understand the whole workflow because the paper lacks a workflow figure", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)