# Assessment of Word-Level Neural Language Models for Sentence Completion +
## 6 TOPIK Cloze Questions  
In this section, we attempt to confirm how effective the aforementioned methods are for a dataset other than in English. To do this, we newly collected 1823 cloze-style questions from Test of Proficiency in Korean (TOPIK) ([[https://www.topik.go.kr](https://www.topik.go.kr)]([https://www.topik.go.kr](https://www.topik.go.kr))), provided by the website ([[https://studytopik.go.kr](https://studytopik.go.kr)]([https://studytopik.go.kr](https://studytopik.go.kr))). TOPIK aims to assess the linguistic ability and guide the learning of the Korean language for non-native Korean speakers, and its results may be a prerequisite for entrance into universities or employment in companies and public institutions. Tables 8 and 9 show the statistics of the collected TOPIK dataset. According to the level of difficulties, TOPIK test types have been divided into Levels 1-6 (lower is easier) until 2004, novice/intermediate/advanced from 2004 to 2014, and Level I/II (I is easier than II) since 2014. Most of the multiple-choice cloze-style questions appear in the easier test types except that 78 questions are from Level II. TOPIK is composed of sections including vocabulary, reading, and writing. Each \begin{table}
\begin{tabular}{l c c} \hline \hline
**Model (\(d_{emb}\)-\(d_{hid}\)-\(d_{out}\))** & **Training Corpus** & **Accuracy** \\ \hline NPMI + Co-occ. Freq. + LSA + CBOW + CSKIP [7] & GloWbe & _59_ \\ \hline PMI using Unigrams + Bigrams + Trigrams [8] & English Gigaword & _58.95_ \\ \hline Unidirectional word RNN (200-600-400) & \multirow{2}{*}{19C novels} & 29.6 (1.5) \\ Bidirectional word RNN (200-600-400) & & 33.3 (2.0) \\ \hline Unidirectional word RNN (500-2000-500) & \multirow{2}{*}{1B word benchmark} & 66.5 \\ Bidirectional word RNN (500-2000-500) & & 69.1 \\ \hline LM1B (1024-8196-1024) & \multirow{2}{*}{1B word benchmark} & 71.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Accuracy results (%) for the SAT set. The accuracies in italic were copied from their corresponding publications. We report the average accuracy with the standard deviation if a model was optimized repeatedly. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **w/o FT** & **w/FT** \\ \hline BERT-base-uncased & 30.92 & 42.11 \\ BERT-base-cased & 30.92 & 52.63 \\ BERT-base-multilingual-uncased & 17.11 & 22.37 \\ BERT-base-multilingual-cased & 23.03 & 36.84 \\ BERT-large-uncased & 25.66 & 73.03 \\ BERT-large-cased & 27.63 & 73.03 \\ BERT-large-uncased-wwm & **63.82** & 80.26 \\ BERT-large-cased-wwm & 59.87 & **80.92** \\ GPT2 & 38.16 & 32.24 \\ GPT2-medium & 53.29 & 53.29 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Accuracy results (%) of transformer-based models for the SAT set with and without fine-tuning, which are presented in w/o FT and w/FT columns, respectively. section contains a different number of cloze-style questions, while the lengths of questions in the writing section are larger than those of the others. Each question was tagged with 'long' or'short' for its passage length and'single' or'multi' for the number of speakers. We counted for each tag type the number of words after splitting with whitespace and punctuation. While the passage lengths of the collected questions are not restricted to a single sentence, more than 70% are questions tagged as'short', whose passages contain fewer than 10 words on average. Some instances from the TOPIK dataset are shown in Table 10.