Measuring memorization in RLHF for code completion
Authors: name: Aneesh Pappu; name: Billy Porter; name: Ilia Shumailov; name: Jamie Hayes
Published: 2024-06-17 16:33:35

Abstract: Reinforcement learning with human feedback (RLHF) has become the dominant
method to align large models to user preferences. Unlike fine-tuning, for which
there are many studies regarding training data memorization, it is not clear
how memorization is affected by or introduced in the RLHF alignment process.
Understanding this relationship is important as real user data may be collected
and used to align large models; if user data is memorized during RLHF and later
regurgitated, this could raise privacy concerns. In this work, we analyze how
training data memorization can surface and propagate through each phase of
RLHF. We focus our study on code completion models, as code completion is one
of the most popular use cases for large language models. We find that RLHF
significantly decreases the chance that data used for reward modeling and
reinforcement learning is memorized, in comparison to aligning via directly
fine-tuning on this data, but that examples already memorized during the
fine-tuning stage of RLHF, will, in the majority of cases, remain memorized
after RLHF.