Before revision: Due to the common belief that training deep transformers from scratch requires large datasets , people usually only use shallow and simple additional layers on top of pre-trained models during fine-tuning on small datasets.
After revision: It is a common belief that training deep transformers from scratch requires large datasets , people usually only use shallow and simple additional layers on top of pre-trained models during fine-tuning on small datasets.