Before revision:  We conclude that BERT can be pruned once during pre-training rather than separately for each task without affecting performance.abstract 
After revision:  We conclude that BERT can be pruned once during pre-training rather than separately for each task without affecting performance.abstract \%) do not affect pre-training loss or transfer to downstream tasks at all. Medium levels of pruning increase the pre-training loss and prevent useful pre-training information from being transferred to downstream tasks. High levels of pruning additionally prevent models from fitting downstream datasets, leading to further degradation. Finally, we observe that fine-tuning BERT on a specific task does not improve its prunability. We conclude that BERT can be pruned once during pre-training rather than separately for each task without affecting performance.