# Delayed Combination of Feature Embedding in Bidirectional LSTM CRF for NER
## 5 Results and Discussion
### The Model Equipped with Our Dictionary Representation  
Before evaluating our model with dictionary representation, we first checked the coverage of the collected list of named entities on the two target datasets to assess whether collected names are enough or not for the datasets. The coverages by each named entity (NE) type of CoNLL 2003 and OntoNotes 5.0 are given in Tables 6 and 7, respectively. The percentage in the tables indicates that those ratios of named entity tokens of each type were found in the collected dictionary. On the CoNLL 2003, the dictionary covers well the first three types of names but only a small ratio of miscellaneous typed entities is covered. On the OntoNotes 5.0 dataset, only four types of names such as Person, Facility, Organization and GPE are well covered (i.e., over 50%). \begin{table}
\begin{tabular}{l c c} \hline \hline
**CoNLL 2003** & **Begin-Tag** & **Inside-Tag** \\ \hline Person & 84.19\% & 94.91\% \\ Organization & 86.64\% & 76.56\% \\ Location & 76.45\% & 89.78\% \\ Miscellaneous & 6.57\% & 2.40\% \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The coverage of the dictionary on the CoNLL2003 dataset. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Work** & **Model (Character-Level)** & **F1 CoNLL 2003** & **F1 DeLFT’s** & **F1 OntoNotes 5.0** \\ \hline [PERSON] et al. & [2] & BiLSTM-CRF(CNN) & 90.94 & 90.75 & - \\ [PERSON] and [PERSON] & [3] & BiLSTM-CRF (LSTM) & 91.21 & 90.73 & - \\ [PERSON] and [PERSON] & [5] & BiLSTM (CNN) & 90.91\({}^{*}\) & 89.23 & 86.28 \({}^{**}\) \\ [PERSON] et al. & [4] & BiLSTM (LSTM) & 84.09 & - & - \\ [PERSON] et al. & [30] & BiLSTM (LSTM) & 91.71 & - & - \\ [PERSON] & [33] & BiLSTM (CNN) & 90.52 & - & 86.57 \\ [PERSON] and [PERSON] & [34] & BiLSTM (CNN) & 90.60 & - & - \\ \hline \hline
**Ours** & \multicolumn{2}{c}{BiLSTM-CRF (LSTM)} & **90.79** & - & 87.84 \({}^{*}\) \\  & \multicolumn{2}{c}{BiLSTM-CRF (CNN)} & **90.56** & - & **87.87 \({}^{*}\)** \\ \hline \hline \end{tabular}
\begin{tabular}{c}! [PERSON] and [PERSON], 2016 incorporated incorporated the development set as a part of training data. Consequently, the result cannot be directly compared with our model. \({}^{*}\) indicates that the model used additional preprocessing on the dataset before training and testing.  
\end{table} 
Table 5: Comparison between our work and previous early combination model with the pre-trained word embedding and the character-level word encoding. Figure 7: The comparison between the character-level CNN-based and LSTM-based word encodings in our delayed combination model. The result shows that the LSTM approach is better on the CoNLL2003 while the CNN approach is very slightly better on the OntoNotes5.0. As explained in Section 3.1, we built up two kinds of dictionaries: one distinguishes begin and inside tokens in a named entity while the other does not. After combining this dictionary feature with our model at the delayed position, we evaluated our model to show the effectiveness of the dictionary feature. Table 8 shows the comparison of our models with and without the CNN-based dictionary representation and also compares the two kinds of dictionaries with or without Begin-Inside (BI) tags. On the CoNLL 2003, both of the two dictionary representations (with and without BI tags) could improve significantly the F1 score by +0.34% and +0.45%, respectively. On the OntoNotes 5.0, the dictionary without BI tags could improve the F1 score by +0.20% although the dictionary representation with BI tags rather lowered the F1 score by \(-\)0.30%. From this result, we could say that the CNN-based dictionary representation could improve the F1 score. Further analysis lets us find that our dictionary effectively classified the type of a word especially when the context of the word is not sufficient. For example, the named entities in one-word sentences (e.g., 'England'), which are often found on CoNLL 2003 dataset, cannot be correctly predicted only using the Recurrent Neural Network (RNN)-typed network because they have no contextual information [43].