Before revision:  However, the "bad" subnetworks can be fine-tuned separately to achieve only slightly worse performance than the "good" ones , indicating that most weights in the pre-trained BERT  are potentially useful.
After revision:  Strikingly, with structured pruning even the worst possible subnetworks remain highly trainable , indicating that most weights in the pre-trained BERT are potentially useful.