GPT or BERT: why not both?
Authors: name: Lucas Georges Gabriel Charpentier, name: David Samuel

We present a simple way to merge masked language modeling with causal
language modeling. This hybrid training objective results in a model that
combines the strengths of both modeling paradigms within a single transformer
stack: GPT-BERT can be transparently used like any standard causal or masked
language model. We test the pretraining process that enables this flexible
behavior on the BabyLM Challenge 2024. The results show that the hybrid
pretraining outperforms masked-only or causal-only models. We openly release
the models, training corpora and code.