Before revision:  We design a strategy for measuring the significanceof the word-vectors based on the self-attention mechanism of the encoders which helps us identify the word-vectors to be eliminated. Experimental evaluation on the standard GLUE benchmark shows that PoWER-BERT achieves up to 4.5x reduction in inference time over BERT with <1\% loss in accuracy.
After revision:  We design a strategy for measuring the significanceof the word-vectors based on the self-attention mechanism of the encoders which helps us identify the word-vectors to eliminate by augmenting the BERT model and the loss function. Experiments on the standard GLUE benchmark shows that PoWER-BERT achieves up to 4.5x reduction in inference time over BERT with <1\% loss in accuracy.