Before revision:  We provide evidence that this does not always need to be the case: with proper initialization and training techniques , the benefits of very deep transformers are shown to carry over to hard structural prediction tasks , even using small datasets .
After revision:  We provide evidence that this does not always need to be the case: with proper initialization and training techniques , the benefits of very deep transformers are shown to carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension .