Before revision:  To allow for fast experimentation we propose an efficient setup with small BERT models and synthetic as well as natural data.
After revision:  To allow for fast experimentation we propose an efficient setup with small BERT models trained on a mix of synthetic and natural data.