Insights from the Inverse: Reconstructing LLM Training Goals Through
  Inverse RL
Authors: name: Jared Joselowitz; name: Arjun Jagota; name: Satyapriya Krishna; name: Sonali Parbhoo
Published: 2024-10-16 12:14:25

Abstract: Large language models (LLMs) trained with Reinforcement Learning from Human
Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying
reward functions and decision-making processes remain opaque. This paper
introduces a novel approach to interpreting LLMs by applying inverse
reinforcement learning (IRL) to recover their implicit reward functions. We
conduct experiments on toxicity-aligned LLMs of varying sizes, extracting
reward models that achieve up to 80.40% accuracy in predicting human
preferences. Our analysis reveals key insights into the non-identifiability of
reward functions, the relationship between model size and interpretability, and
potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward
models can be used to fine-tune new LLMs, resulting in comparable or improved
performance on toxicity benchmarks. This work provides a new lens for
understanding and improving LLM alignment, with implications for the
responsible development and deployment of these powerful systems.