Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement
  Learning
Authors: name: Afshin Khadangi, name: Amir Sartipi, name: Igor Tchappi, name: Ramin Bahmani, name: Gilbert Fridgen

The tension between data privacy and model utility has become the defining
bottleneck for the practical deployment of large language models (LLMs) trained
on sensitive corpora including healthcare. Differentially private stochastic
gradient descent (DP-SGD) guarantees formal privacy, yet it does so at a
pronounced cost: gradients are forcibly clipped and perturbed with noise,
degrading sample efficiency and final accuracy. Numerous variants have been
proposed to soften this trade-off, but they all share a handicap: their control
knobs are hard-coded, global, and oblivious to the evolving optimization
landscape. Consequently, practitioners are forced either to over-spend privacy
budget in pursuit of utility, or to accept mediocre models in order to stay
within privacy constraints. We present RLDP, the first framework to cast DP
optimization itself as a closed-loop control problem amenable to modern deep
reinforcement learning (RL). RLDP continuously senses rich statistics of the
learning dynamics and acts by selecting fine-grained per parameter
gradient-clipping thresholds as well as the magnitude of injected Gaussian
noise. A soft actor-critic (SAC) hyper-policy is trained online during language
model fine-tuning; it learns, from scratch, how to allocate the privacy budget
where it matters and when it matters. Across more than 1,600 ablation
experiments on GPT2-small, Llama-1B, Llama-3B, and Mistral-7B, RLDP delivers
perplexity reductions of 1.3-30.5% (mean 5.4%) and an average 5.6% downstream
utility gain. RLDP reaches each baseline's final utility after only 13-43% of
the gradient-update budget (mean speed-up 71%), all while honoring the same
(epsilon, delta)-DP contract and exhibiting equal or lower susceptibility
to membership-inference and canary-extraction attacks.