# Assessment of Word-Level Neural Language Models for Sentence Completion +
## 4 MSR Sentence Completion
### Results with External Data  
While the official training corpus is highly similar and relevant to the question sentences in terms of linguistic styles and time periods of writing, the limited data may hinder the learning of a deep neural network. We applied various pre-trained models that learn from large external data for the MSR questions (Table 3). LM1B represents the best single model of [PERSON] et al. [12], who experimented variants of LSTM-based LMs on the 1B word benchmark [40], which consists of news text of about one billion words. We also include variants of BERT and GPT2 models which have shown compelling results on many language understanding tasks with fine-tuning on small data or zero-shot task transfer. We obtained the pre-trained weights of LM1B and the others from their publicly available repositories ([[https://github.com/tensorflow/models/tree/master/research/lm_1b](https://github.com/tensorflow/models/tree/master/research/lm_1b)]([https://github.com/tensorflow/models/tree/master/research/lm_1b](https://github.com/tensorflow/models/tree/master/research/lm_1b)), [[https://huggingface.co/transformers/pretrained_models.html](https://huggingface.co/transformers/pretrained_models.html)]([https://huggingface.co/transformers/pretrained_models.html](https://huggingface.co/transformers/pretrained_models.html))). According to the results of the scoring strategies (see Table 1), we applied the full scoring except for the BERT family for which we deployed the blank scoring strategy since BERT adopts masked language modeling. We adhere to this policy for further experiments unless noted otherwise. Although LM1B yielded better accuracy than the models of previous publications, its accuracy could not reach those of our word RNNs, and even its size was substantially larger than ours. Among BERT models, the performance differences between the case-sensitive ones (indicated as '-cased') and the corresponding case-insensitive ones (indicated as '-uncased') were inconsistent across different training settings. The multilingual BERTs that were trained from Wikipedia dumps of about one hundred languages were inferior to the monolingual ones, without fine-tuning to the target language. The authors of BERT have warned that lower-casing non-Latin alphabets could result in somewhat unwanted outcomes ([[https://github.com/google-research/bert](https://github.com/google-research/bert)]([https://github.com/google-research/bert](https://github.com/google-research/bert))), which might be the reason for the poor accuracy of BERT-base-multilingual-uncased. Despite having three times more learnable parameters, BERT-large models performed worse than BERT-base models without whole-word-masking. The whole-word-masking technique masks all-at-once the group of tokenized wordpieces [41] corresponding to a word during masked LM training (see details in [[https://github.com/google-research/bert](https://github.com/google-research/bert)]([https://github.com/google-research/bert](https://github.com/google-research/bert))). With this modification, whole-word-masking BERTs (indicated as '-wwm') outperformed the BERT-large models without whole-word-masking and even surpassed the best ensemble model that was trained from the official training data. On the other hand, \begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **Dev.** & **Test** \\ \hline Bidirectional word RNN ensemble & 74.1 & 74.6 \\ \hline LM1B & 70.2 & 67.7 \\ BERT-base-uncased & 55.38 & 58.46 \\ BERT-base-cased & 60.58 & 60.19 \\ BERT-base-multilingual-uncased & 22.50 & 20.58 \\ BERT-base-multilingual-cased & 41.54 & 41.92 \\ BERT-large-uncased & 54.23 & 55.00 \\ BERT-large-cased & 52.69 & 49.23 \\ BERT-large-uncased-wwm & **77.69** & **77.12** \\ BERT-large-cased-wwm & 75.19 & **77.12** \\ GPT2 & 46.73 & 44.04 \\ GPT2-medium & 54.62 & 52.69 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Accuracy results (%) of pre-trained models without fine-tuning for the MSR questions.