Before revision: Due to the common belief that training deep transformers from scratch requires large datasets , people usually only use shallow and simple additional layers on top of pre-trained models during fine-tuning on small datasets. We provide evidence that this does not always need to be the case: with proper initialization and training techniques , the benefits of very deep transformers are shown to carry over to hard structural prediction tasks , even using small datasets .
After revision: Due to the common belief that training deep transformers from scratch requires large datasets , people usually only use shallow and simple additional layers on top of pre-trained models during fine-tuning . This work shows that this does not always need to be the case: with proper initialization and training techniques , the benefits of very deep transformers are shown to carry over to hard structural prediction tasks , even using small datasets .