Tracking Universal Features Through Fine-Tuning and Model Merging
Authors: name: Niels Horn, name: Desmond Elliott

We study how features emerge, disappear, and persist across models fine-tuned
on different domains of text. More specifically, we start from a base one-layer
Transformer language model that is trained on a combination of the BabyLM
corpus, and a collection of Python code from The Stack. This base model is
adapted to two new domains of text: TinyStories, and the Lua programming
language, respectively; and then these two models are merged using these two
models using spherical linear interpolation. Our exploration aims to provide
deeper insights into the stability and transformation of features across
typical transfer-learning scenarios using small-scale models and sparse
auto-encoders.