Before revision:  In this paper, we focus on pre-trained language models with self-pruning training design on task-specific tuning.
After revision:  In this paper, we propose a novel dropout method named AttendOut to let self-attention empowered PrLMs capable of more robust task-specific tuning.