# Iterative Named Entity Recognition with Conditional Random Fields
## 2 Materials and Methods
### Iterative Procedure for Annotation  
The iterative procedure started with the manual annotation of a small set of 500 sentences randomly selected from the sentences belonging to the training dataset (Figure 2). This set of sentences contained 12,954 tokens, 959 of which were labelled as belonging to an entity. This set constituted the first 'Model-Training dataset' used, during the 'Training and Prediction' part of the procedure, to train an initial CRF model. The initially trained model was then employed to classify entities in a pool of sentences randomly selected from the training dataset. This pool of sentences, newly generated during each iteration, contained 10x the number of sentences to be added to the'model-training dataset' per iteration. In case not enough sentences were available for annotation in the training dataset, the pool-size was automatically reduced, with all sentences of the pool being added to the 'Model-training dataset'. It should be noticed that a given sentence was either part of the initial model training dataset, part of the pool, or part of the set of unlabelled sentences left in the training dataset; these three groups of data never superposed during the iterative procedure. After the 'Training and Prediction' step, new data were annotated either manually or automatically in the 'Analysis and Annotation' step before being added to the model training data. The aim of this step was, firstly, to identify sentences with the highest probability of correct classification by the model. In each iteration, these'sure sentences' were automatically \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Optimization Dataset**} & \multicolumn{2}{c}{**Evaluation Dataset**} & \multicolumn{2}{c}{**Total**} \\ \hline
**Entity** & **\#Tokens** & **Proportion (\%)** & **\#Tokens** & **Proportion (\%)** & **\#Tokens** & **Proportion (\%)** \\ \hline DAT & 4068 & 29.70 & 4908 & 34.20 & 8976 & 32.00 \\ ORG & 3509 & 25.70 & 4020 & 28.00 & 7529 & 26.90 \\ MON & 3015 & 22.00 & 2701 & 18.80 & 576 & 20.40 \\ LOC & 1131 & 8.30 & 1050 & 7.30 & 2181 & 7.80 \\ PCT & 1097 & 8.00 & 993 & 6.90 & 2090 & 7.50 \\ PER & 836 & 6.10 & 664 & 4.60 & 1500 & 5.40 \\ TIM & 23 & 0.20 & 12 & 0.10 & 35 & 0.10 \\ \hline \#Entities & 13,679 & 9.10 & 14,348 & 9.20 & 28,027 & 9.10 \\ \#Tokens & 150,745 & & 156,278 & & 307,023 & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of tokens belonging to each of the seven entities labelled and the associated proportion relative to the total number of entities detected, for the Optimization and Evaluation datasets that were manually annotated. The last row indicates the total number of tokens belonging to any of the seven entities considered (‘#Entities’) as well as the proportion of the annotated tokens (independent of entity class) relative to the total number of tokens in the dataset (‘#Tokens’). annotated for entities by the model and added to the model training dataset without a manual check (self-training). Secondly, 'unsure sentences' with a low probability of correct classification for potential entities contained within them were identified by the model and presented for manual annotation in each iteration. This iterative manual annotation and expansion of the model training dataset represents the active learning component of the model training procedure. The prediction of the uncertainties of the correct classification was based upon the probabilities delivered by the CRF as a probabilistic model. The'sure sentences' were those sentences with the highest probability of correct classification by the model. The 'unsure sentences' came from the lower quantiles of the probability distribution of the uncertainties provided by the model. During the automatic, self-training component of data annotation for the model training dataset, two types of sentences were distinguished--those for which at least one entity was predicted and those without predicted entities.