Before revision: Much of the recent success in NLP is due to the large Transformer-based models such as BERT (Devlin et al, 2019). However, these models have been shown to be reducible to a smaller number of self-attention heads and layers.
After revision: Much of the recent success in NLP is due to the large Transformer-based models were shown to be reducible to a smaller number of self-attention heads and layers.