# Assessment of Word-Level Neural Language Models for Sentence Completion +
## 4 MSR Sentence Completion
### Results with the Official Training Data  
For experiments in this subsection, we used 522 novels from Project Gutenberg written in the nineteenth-century, which is the specified standard training corpus of the MSR Challenge. We preprocessed the data through sentence splitting, followed by word tokenization and lower-casing to feed word sequences into the word RNNs. Sentences consisting of more than eighty or less than ten words were filtered out. As a result, the vocabulary was composed of 64K words after converting any word with fewer than six occurrences in the corpus to [UNK] token. To be specific about the implementation, \(d_{emb}\), \(d_{hid}\), and \(d_{out}\) were set to 200, 600, and 400, respectively. We set the forward and backward embedding sizes of the bidirectional word RNN to be equal. We trained all networks for ten epochs with stochastic gradient descent while applying 10% dropout. Twenty sentences of the same length comprised a mini-batch. Gradients were clipped at the maximum norm of 5.0 and normalized by the mini-batch size [34]. The learning rate was started at 0.5 and decreased by half at the beginning of each epoch after the fifth epoch. We determined these settings by inspecting the performance on the first 520 questions of the MSR set, referred to as development set, following prior work [9, 37]. Accordingly, we refer to the remaining MSR questions as test set. First, we investigated the performance of different language modeling and scoring strategies (Table 1). We optimized the networks repeatedly with five different random seeds and considered the average accuracy of those networks as a model accuracy. As expected, the full scoring was far more beneficial than the blank scoring in the unidirectional cases. Likewise, the accuracy of the bidirectional LM increased when using the full scoring compared to the blank scoring. In contrast to [PERSON] and [PERSON] [36], the partial scoring strategy was inferior to the full scoring strategy. This can be explained by the fact that a special word related to the answer appears in the following text of the blank in all questions of [PERSON] et al. [38], whereas the presence of a special word is not ensured in the MSR questions. In addition, the masked LM was inferior to the others, which differs from [PERSON] et al. [16]. It would seem that the masked LM requires more training data and prefers the transformer architecture rather than RNN layers. The performance gaps are visualized in Figure 2, where the error bars signify the minimum and maximum accuracies obtained by five networks with different random seeds. To justify the choice of the network size, we plot the validation results for the different numbers of nodes in Figure 3. The unidirectional word RNN was evaluated on the MSR validation set with different \(d_{emb}\), \(d_{hit}\), and \(d_{out}\) values. Figure 3 verifies that the choice of \(d_{hit}\) greatly affected the performance, while smaller embedding sizes than \(d_{hid}\) led to better results possibly due to a regularization effect. We further experimented with more numbers of nodes, but the accuracy increase was negligible. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**LM Formulation**} & \multicolumn{3}{c}{**Accuracy [\%]**} \\ \cline{2-4}  & **Blank** & **Full** & **Partial** \\ \hline Unidirectional LM & 50.5 (0.4) & 69.4 (0.8) & 56.4 (0.9) \\ Bidirectional LM & 69.8 (0.5) & 72.3 (1.1) & 63.7 (1.4) \\ Masked LM & N/A & 58.2 (1.6) & N/A \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance comparison of word RNNs for different scoring strategies on the MSR set. We trained each model five times with different random seeds and report the average accuracy with the standard deviation in parentheses. Figure 2: Performance comparison for different LM formulation and scoring strategies: (**a**) the full scoring was applied except for the masked LM; and (**b**) the bidirectional model was used.