Before revision: Self-attention based models have achieved remarkable success in natural language processing . However, the self-attention network design is questioned as suboptimal in recent studies, due to its veiled validity and high redundancy .
After revision: Self-attention based models have achieved remarkable success in natural language processing , they are susceptible to over-fitting due to unusual large model size. To this end, dropout serves as a therapy. However, existing methods like random-based, knowledge-based and search-based dropout are more general but less effective onto self-attention network design is questioned as suboptimal in recent studies, due to its veiled validity and high redundancy .