Teaching Pretrained Language Models to Think Deeper with Retrofitted
  Recurrence
Authors: name: Sean McLeish, name: Ang Li, name: John Kirchenbauer, name: Dayal Singh Kalra, name: Brian R. Bartoldson, name: Bhavya Kailkhura, name: Avi Schwarzschild, name: Jonas Geiping, name: Tom Goldstein, name: Micah Goldblum
Organization: Tom Goldstein's Lab at University of Maryland, College Park

Recent advances in depth-recurrent language models show that recurrence can
decouple train-time compute and parameter count from test-time compute. In this
work, we study how to convert existing pretrained non-recurrent language models
into depth-recurrent models. We find that using a curriculum of recurrences to
increase the effective depth of the model over the course of training preserves
performance while reducing total computational cost. In our experiments, on
mathematics, we observe that converting pretrained models to recurrent ones
results in better performance at a given compute budget than simply
post-training the original non-recurrent language model.