Before revision:  They are present in all six models of BERT family that we examined and removing them significantly degrades both the MLM perplexity and the downstream task performance.
After revision:  We show that disabling them significantly degrades both the MLM perplexity and the downstream task performance.