# Delayed Combination of Feature Embedding in Bidirectional LSTM CRF for NER
## Appendix A The Hyperparameters Tuning
Owing to the similar result on the two datasets, we assume that the model performance is improved when (1) the vocabulary in word embedding matches the word in the dataset, and (2) the subword (n-gram) of FastText is used to generate embeddings for rare words. \begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & & \multicolumn{6}{c}{**Five Times Validation**} \\
**Dataset** & **Mini-Batch** & **Embedding** & **1** & **2** & **3** & **4** & **5** & **Average** & **SD** \\ \hline Conll 2003 & 200 & GloVe & 89.96 & 89.90 & 89.94 & 90.26 & 90.24 & 90.06 & 0.18 \\  & & FastText Vec & 89.44 & 89.70 & 89.79 & 89.44 & 89.93 & 89.66 & 0.22 \\  & & FastText Bin & 90.05 & 90.30 & 90.20 & 89.97 & 90.17 & **90.14** & 0.13 \\ \hline OntoNotes 5.0 & 200 & GloVe & 87.12 & 87.44 & 87.42 & 87.42 & 87.61 & 87.40 & 0.18 \\  & & FastText Vec & 87.49 & 87.49 & 87.44 & 87.52 & 87.40 & 87.47 & 0.05 \\  & & FastText Bin & 87.68 & 87.43 & 87.74 & 87.58 & 87.58 & **87.60** & 0.12 \\ \hline \hline \end{tabular}
\end{table} 
Table 12: Five times F1 scores of the pre-trained word embedding with two-stacked bidirectional LSTM. From the result, we achieve the best F1 score when using FastText with subwords. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & & \multicolumn{6}{c}{**Learning Rate Multiplied by Alpha (a)**} \\
**Dataset** & **Mini-Batch** & **Optimizer (a)** & **0.09** & **0.1** & **0.2** & **0.3** & **0.4** & **Average** & **SD** \({}^{*}\) \\ \hline Conll 2003 & 200 & SGD (\(\alpha\) =0.1 ) & 89.40 & 89.18 & 89.34 & 89.12 & 89.12 & 89.232 & 0.13 \\  & & Adam (\(\alpha\) =0.01) & 89.17 & 89.23 & 89.45 & 89.17 & 88.94 & 89.190 & 0.18 \\  & & Nadam (\(\alpha\) =0.01) & 89.17 & 89.39 & **89.49** & 89.34 & 89.22 & **89.321** & 0.13 \\  & & RMSprop (\(\alpha\) =0.01) & 89.09 & 89.11 & 89.30 & 89.30 & 89.06 & 89.172 & 0.12 \\  & & Adagrad (\(\alpha\) =0.1) & 89.14 & 89.48 & 89.40 & 89.23 & 89.34 & 89.316 & 0.13 \\  & & Adamax (\(\alpha\) =0.01) & 89.24 & **89.50** & 89.27 & 89.34 & 89.21 & 89.313 & 0.12 \\ OntoNotes 5.0 & 200 & Nadam (\(\alpha\) =0.01) & 86.78 & **87.08** & 86.83 & 86.28 & 86.03 & 86.599 & 0.43 \\ \hline \hline \multicolumn{10}{l}{\({}^{*}\) Standard deviation (SD or Std Dev) is a measure of variation between values in a set of data.} \\ \end{tabular}
\end{table} 
Table 11: Five times F1 scores of the optimizer, in this case SGD, Adam, Nadam, RMSprop, Adagrad, and Adamax. * Number of bidirectional LSTM layers  
Currently, the stacked LSTMs are used as a standard technique for challenging sequence predictions [56]. Some previous works [57; 58; 59] present the multiple stacked bidirectional LSTMs in a neural network. They show that the classification performance can be improved when using this technique. Furthermore, there is some related theoretical support: a deep hierarchical model is more effective at representing than a shallow model [30; 60]. Due to the stacked processes, when the first bidirectional LSTM layer provides an output vector, this output vector provides more complex patterns for the next layer, enabling us to capture information on a different scale. [PERSON] et al. [57] suggests the appropriate amount of multi-layer bidirectional LSTM help to understand the relationship between words and words at a deep level. However, when the representation flows through stacked layers, the risk arises that the representation information will be lost [61], and it may also reduce the model performance. Hence, we examined two or three-stacked bidirectional LSTM to find the number of layers most suitable for our model. The comparative of F1 score results are shown in Figure 19 the model shows higher performance when using the three-stacked bidirectional LSTM. Consequently, we assume that three stacks are able to capture more information for our model. In contrast, two-stacked bidirectional LSTMs are applied in earlier experiments [2; 35; 23]. However, these experiments used different batch sizes, optimizers and lower dimensional word embedding compared to our model.