Before revision:  Compared to the ALBERT baseline, we achieve up to 2.4x and 13.4x inference latencyand memory savings, respectively, with less than 1\%-pt drop in accuracy on several GLUE benchmarks by employing a calibrated combination of 1) entropy-based early stopping, 2) adaptive attention span, 3) movement and magnitude pruning, and 4) floating-point quantization.
After revision:  Compared to the ALBERT baseline, we achieve up to 2.4x and 13.4x inference latencyand memory savings, respectively, with less than 1\%-pt drop in accuracy on several GLUE benchmarks by employing a calibrated combination of 1) entropy-based early stopping, 2) adaptive attention span, 3) movement and magnitude pruning, and floating-point quantization.