# Assessment of Word-Level Neural Language Models for Sentence Completion +
## 5 SAT Sentence Completion  
To further verify our findings, we gathered 152 sentence completion questions from the eight SAT practice exams provided by College Board ([[https://www.collegeboard.org](https://www.collegeboard.org)]([https://www.collegeboard.org](https://www.collegeboard.org))) between 2003 and 2014. The reading part of the collected practice set includes sentence completion questions, each of which has one or two blanks in a sentence and provides five candidate choices. The collected SAT set contains that of [PERSON] [7] but is covered by that of [PERSON] [8]. Since any standard training corpus is not specified for the SAT set, previous researchers used different training corpora such as GloWbe [43], English Gigaword (LDC2009T13), and Wikipedia dumps. We used 1B word benchmark [40] to train the word RNNs, whose \(d_{emb}\), \(d_{hid}\), and \(d_{out}\) were set to 500, 2000, and 500, respectively. The learning rate was initialized to 1.0 and then multiplied by 0.8 after every epoch. We applied importance sampling to the output embedding matrix with sample size of 8192 [12] to deal with the large vocabulary size, which was 409K even after lower-casing. Other training details were kept identical to those for the MSR set, described in Section 4.1. Table 6 shows the SAT evaluation results, including the previous best results. We also evaluated the word RNNs that had been trained on the MSR official training corpus (denoted by 19C novels). With more training data and larger network sizes, the SAT accuracies of the word RNNs increased considerably, exceeding the previously published results while being comparable to that of LM1B. Note that, besides its large size, LM1B utilized character level CNN layers to acquire intermediate representations for 793K word types [12], requiring more computing resources than our RNNs. In addition, bidirectional modeling was beneficial, as observed in Table 2.  
Similar to the experimentation in Section 4.2, we also evaluated the transformer-based models with and without fine-tuning (Table 7). Since the SAT set is not large enough for splitting, we treated the entire MSR set as the training set while assigning the last hundred questions to the held-out validation set. With the learning rate that had been selected for each pre-trained model in the previous \begin{table}
\begin{tabular}{l c} \hline \hline
**Input Construction** & **Accuracy** \\ \hline Proposed method & 84.2 (2.3) \\ \hline with [SEP] & 84.2 (1.3) \\ with segment identifiers & 84.6 (2.9) \\ with [SEP] and segment identifiers & 74.6 (1.8) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Accuracy results (%) on the holdout set for different input construction methods. For each method, we performed five runs of fine-tuning with different random seeds, and the average accuracy is reported with the standard deviation in parentheses. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **w/o FT** & **w/FT** \\ \hline BERT-base-uncased & 58.46 & 68.65 \\ BERT-base-cased & 60.19 & 73.27 \\ BERT-base-multilingual-uncased & 20.58 & 21.92 \\ BERT-base-multilingual-cased & 41.92 & 47.88 \\ BERT-large-uncased & 55.00 & 79.23 \\ BERT-large-cased & 49.23 & 79.42 \\ BERT-large-uncased-wwm & **77.12** & **86.15** \\ BERT-large-cased-wwm & **77.12** & 85.77 \\ GPT2 & 44.04 & 38.27 \\ GPT2-medium & 52.69 & 59.23 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Accuracy results (%) of transformer-based models for the MSR test set with and without fine-tuning, which are presented in w/o FT and w/FT columns, respectively. section, we obtained the best model by re-running the fine-tuning five times on the new training set with different random seeds. Among the networks without fine-tuning, the whole-word-masking models were outstanding, while being outperformed by the word RNNs. Fine-tuning with the MSR questions was quite beneficial to the BERT models for the SAT questions, which indicated that the model learned a common solving strategy that worked across questions from different source texts.