## 4 Evaluation
### Experimental Results  
Our first experiment was to compare the MA performances of MANE with those of the previous state-of-the-art MA models, as shown in Table 5.  
In Table 5, MANE-MA is an independent model with the same architecture as that of the MA layer in Figure 1. Structural SVMs [18] constitute an integrated model for automatic word spacing and morphological analysis in Korean. Bi-LSTM-CRFs-MA [10] and stacked Bi-GRU-CRFs-MA [19] are integrated deep learning models for the task described in [18]. For a fair comparison, we used correctly spaced input sentences in these previous integrated models. In addition, we showed the performances of a modified Bi-LSTM-CRFs-MA [10] and a modified stacked Bi-GRU-CRFs-MA [19] in which additional linguistic features, such as morpheme dictionary look-ups and pre-analysis dictionary look-ups, were excluded. The parenthesized scores denote the performances of the modified versions reported in their papers [10,19]. Seq2 Seq [11] is a generative MA model based on a sequence-to-sequence network. As shown in Table 5, MANE outperformed all the comparison models. When MANE was compared to the modified versions, the performance differences were even larger. To verify the performance differences between MANE and the comparison models, we repeated a performance evaluation of MANE five times. In the repeated evaluations on MA, the performance variations of MANE were \(\pm 0.0003\) in accuracies and \(\pm 0.0001\) in F1-scores. As a result, MANE always showed higher performances than all of the previous MA models. The \(p\)-values of F1-scores between MANE and the comparison models were from \(5.19\times 10^{-9}\) to \(0.00537\). This implies that the performance differences are statistically significant at the \(0.05\) level. Moreover, it showed higher performances than MANE-MA. This reveals that the NER layer can be useful in improving the performance of the MA layer. In our second experiment, we compared the NER performances of MANE with those of the previous state-of-the-art NER models, as shown in Table 6.  
In Table 6, MANE-NE is an independent model with the same architecture as that of the NE layer in Figure 1. MANE-NE uses a pretrained MA layer for POS information. Subsequently, the parameter of the MA layer is frozen to block tuning when training the NE layer. Bi-GRU-CRFs-NE [20] constitutes a baseline NER model based on GRUs with a CRF layer. Bi-LSTM-CRFs-NE [1] represents an NER model, in which a word representation is expanded using word, POS, and syllable embeddings, as well as dictionary look-up features. Stacked Bi-GRU-CRFs-NE [9] constitutes an NER model with two layers \begin{table}
\begin{tabular}{c c} \hline \hline
**Model** & **F1-Score** \\ \hline Bi-GRU-CRFs-NE [20] & 0.8022 \\ Bi-LSTM-CRFs-NE [1] & 0.8549 \\ Stacked Bi-GRU-CRFs-NE [9] & 0.8576 \\ MorpheNE [12] (MA and NER integrated) & 0.8566 \\ Attention-CRFs [8] & 0.8188 \\ MANE-NE & 0.8583 \\ MANE & 0.8597 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Performance comparison on named entity recognition. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **Accuracy** & **F1-Score** \\ \hline Structural SVMs [18] & 0.9802 & 0.9803 \\ Bi-LSTM-CRFs-MA [10] & 0.9780 & 0.9877 (0.9750) \\ Stacked Bi-GRU-CRFs-MA [19] & 0.9840 (0.9590) & - \\ Seq2 Seq [11] & - & 0.9793 \\ MANE-MA & 0.9793 & 0.9789 \\ MANE & 0.9886 & 0.9880 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance comparison on morphological analysis. of Bi-GRU-CRFs, in which effective dictionary look-up features were used. Attention-CRFs [8] perform NER based on the attention mechanism and the CRFs. MANE-NE is the independent NER model in Table 5. Therefore, correctly POS-tagged sentences were used as input. In Table 6, MorpheNE [12] is an integrated model for MA and NER based on Bi-GRU-CRFs. MorpheNE did not use correctly POS-tagged sentences as inputs.