Before revision: Due to the common belief that training deep transformers from scratch requires large datasets , people usually only use shallow and simple additional layers on top of pre-trained models during fine-tuning on small datasets.
After revision: Due to the common belief that training deep transformers from scratch requires large datasets . Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning on small datasets.