# Delayed Combination of Feature Embedding in Bidirectional LSTM CRF for NER
## 6 Conclusions and Future Works  
For the success of various deep-learning methods in NER task, most researchers have tested various feature-encoding techniques such as pre-trained word embedding, contextual embedding, character-level CNN or LSTM-based word encoding, word pattern-based encoding and dictionary-based encoding, and incorporated them into the deep learning networks. When more than one feature encoding is used, most previous works combined them at the embedding layer and then passed through the deep neural network to find the best label sequences in the output layer. However, when such an early combination of various feature encodings passes through the deep neural networks like RNN, much useful information could be mixed or shrunk by other more-dominant information and this could consequently limit the improvement in performance. To avoid such limitations, we introduced the delayed combination model of various promising feature encodings. This model selected FastText as the pre-trained word embedding, ELMo as the contextual embedding and character-level CNN or LSTM word encoding, and designed CNN-based sentence encoding using a dictionary, for feature encoding. We also selected the most common bidirectional LSTM network for learning contextual information from the train set. Among those feature encodings, FastText and ELMo embeddings were passed through its own separate bidirectional LSTM blocks while the remaining feature encodings were bypassed and combined with the outputs of the bidirectional LSTM blocks. Through several experiments, we showed that our delayed combination model outperforms the early combination one and also showed the feasibility of our model by comparing our results with the corresponding previous works. \begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline
**Work** & \begin{tabular}{l} **Pre-Trained** \\ **Embedding** \\ \end{tabular} & \begin{tabular}{l} **Character Word** \\ **Level** \\ \end{tabular} & **Hybrid** & **Model** & **F1 CoNLL 2003** & **F1 OntoNotes 5.0** \\ \hline \begin{tabular}{l} [PERSON] et al. \\ [PERSON] et al \\ [PERSON] et al. \\ \end{tabular} & \begin{tabular}{l} [18] \\ [45] \\ [46] \\ \end{tabular} & ELMO & - & - & - & BiLSTM CRF & 92.22 & - \\ \begin{tabular}{l} [PERSON] et al \\ [PERSON] et al. \\ \end{tabular} & \begin{tabular}{l} EMMo \\ Glove 6B-100D, \\ ELMO \\ Glove 6B-100D, \\ ELMO \\ \end{tabular} & - & - & POS & - & MGNER & 92.28 & - \\ \begin{tabular}{l} [PERSON] et al. \\ \end{tabular} & \begin{tabular}{l} BERT-BASE \\ BERT-Large \\ \end{tabular} & - & - & - & 92.40 & - \\ \begin{tabular}{l} [PERSON] et al. \\ [PERSON] et al. \\ \end{tabular} & \begin{tabular}{l} BERT \\ BERT \\ \end{tabular} & - & - & - & - & 93.37 & 90.30 \\ \begin{tabular}{l} [PERSON] et al. \\ \end{tabular} & \begin{tabular}{l} BERT \\ DeepNist \\ \end{tabular} & - & - & - & MRC+DSC & 93.33 & 92.07 \\ \
\begin{tabular}{l} [PERSON] et al. \\ \end{tabular} &
\begin{tabular}{l} Cloze-style LM \\ embedding \\ \end{tabular} & CNN & - & - & - & CNN Large & 93.50 & - \\ \hline
**Ours** & FastText, ELMO & LSTM & - &
\begin{tabular}{l} CNN \\ dictionary \\ \end{tabular} & BiLSTM CRF & **92.49** & - \\ \hline \hline \end{tabular}
\end{table} 
Table 10: The comparison of our model with the previous works having contextual embeddings (ELMO and BERT) as their features. As future work, we intend to extend this model by (1) building up a dictionary better-organized and learned from the external resources, (2) incorporating BERT-Large or other fine-tuned LM embeddings into our model, and applying to other non-English languages as well as other fine-grained NER task. Conceptualization, S.L. and C.R. ; methodology, C.R. ; software, C.R. ; validation, S.L., H.J.J. and C.R. ; formal analysis, C.R. ; investigation, H.J.J. ; resources, H.J.J and C.R. ; data curation, C.R. ; writing--original draft preparation, C.R; writing--review and editing, S.L. and H.J.J; visualization, C.R. ; supervision, S.L. ; project administration, S.L. ; funding acquisition, S.L.