Masked Language Model Scoring
Year: 2020

Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are computed by masking tokens one by one. We show that PLLs outperform scores from autoregressive language models like GPT-2 in a variety of tasks. By rescoring ASR and NMT hypotheses, RoBERTa reduces an endto-end LibriSpeech model's WER by 30% relative and adds up to +1.7 BLEU on state-of-theart baselines for low-resource