Before revision:  We also show that the "good" subnetworks vary considerably across GLUE tasks, opening up the possibilities to learn what knowledgeBERT actually uses at inference time .
After revision:  We also show that the "good" subnetworks to see if their success can be attributed to superior linguistic knowledge, but find them unstable, and not explained by meaningful self-attention patterns .