Before revision:  They are present in all six models of BERT family that we examined and removing them significantly degrades both the MLM perplexity and the downstream task performance.
After revision:  They are present in all six models of BERT family that we examined and removing them significantly degrades both the MLM loss and the downstream task performance.