# Delayed Combination of Feature Embedding in Bidirectional LSTM CRF for NER
## 5 Results and Discussion
### The Model Additionally Equipped with the ELMO Encoding  
Our model is finally combined with the separate bidirectional LSTM network of the ELMo embedding at the delayed position (The OntoNotes5.0 dataset couldn't be used for training our model with ELMo encoding because high-end GPUs like V100 were not available within a limited duration.). On the CoNLL 2003 dataset, we checked the effect of the ELMo encoding on all variants of our model according to all possible combinations with the two kinds of character-level word encodings and the two kinds of dictionary representations.  
Figure 8: The performance of our model when combined with the ELMo network at the delayed position.  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Work** & \begin{tabular}{c} **Pre-Trained** \\ **Embedding** \\ \end{tabular} & \
\begin{tabular}{c} **Character** \\ **Level** \\ \end{tabular} &
\begin{tabular}{c} **Word** \\ **Level** \\ \end{tabular} & **Hybrid** & **Model** & **F1 CoNLL 2003** & **F1 OntoNotes 5.0** \\ \hline [PERSON] et al. & [6] & SENNA & - & Spelling, & Gazetteers & - & 90.10 & - \\ Chiu and Nichols & [5] & SENNA & CNN & CAP & Lexicons & Softmax & 91.62\({}^{*}\) & 86.36\({}^{*}\)- \\ [PERSON] et al. & [32] & GloVe 6B-300D & CNN & POS & Gazetteers & Neural CRF & 91.06 & - \\ [PERSON] et al. & [32] & GloVe 6B-300D & CNN & POS, & Gazetteers & Neural CRF & 91.89 & - \\  & & & SpaCy & & & & \\ Ghaddar and & [33] & SSKIP and LS & LSTM & CAP & Lexical & BiLSTM CRF & 91.73 & 87.95 \\ Langlais & representation & & Similarity Vector & & & & \\ \hline
**Ours** & FastText & LSTM & - & CNN dictionary & BiLSTM CRF & **91.24** & - \\  & FastText & CNN & - & CNN dictionary & BiLSTM CRF & - & **88.07 \({}^{*}\)** \\ \hline \hline \end{tabular} \({}^{\dagger}\) [PERSON] and [PERSON], 2016 incorporated the development set as a part of training data. Consequently, the result cannot be directly compared with our model. \({}^{*}\) indicates that the model used additional preprocessing on the dataset before training and testing.  
\end{table}
Table 9: The comparison of our model with the previous similar dictionary-enabled works.  
The result is shown in Figure 8. Our model could achieve the highest F1 score, 92.49%, when we combined our model with LSTM-based word encoding, dictionary representation without BI tags, and the bidirectional LSTM-based ELMo encoding. The bidirectional LSTM-based ELMo encoding could consistently improve the F1 score of all four variants of our model by the range of [+1.12%, +1.29%].  
We also compared our model equipped with the ELMo encoding to the previous works having contextual embedding as features of their models. This is given in Table 10. Our model could achieve a better F1 score than all the previous works used ELMo as one of their features, such as [PERSON] et al., 2018, [PERSON] et al., 2019, [PERSON] et al., 2018, and [PERSON] and [PERSON], 2018. This result could convince us that our delayed combination model is effective in NER task. Our model still shows a little higher F1 score than the model used BERT-Base ([PERSON] et al., 2019) but other previous works used BERT-Large or further tuned LM embedding still outperform our model. We think that our model can be further improved by combining with BERT-Large or fine-tuned LM embeddings instead of ELMo.