# Named Entity Recognition for Sensitive Data Discovery in Portuguese
## 3 NER Component
### Named Entity Recognition Module
#### 3.2.1 Statistical Models  
NER approaches with statistical models typically require a large amount of training data, and these have not been used as much to avoid the overwhelming annotation effort [26; 27]. Since we only have two not very extensive corpora, this difficulty can be overcome. The models chosen for this approach were a Conditional Random Fields Model (CRF) and a Random Forest Model (RF). Conditional Random Fields: The CRF model implemented in this work is based on the implementation of [PERSON] and [PERSON], for the corpus CoNLL2002, available at GitHub ([[https://github.com/TeamHG-Memex/eli5/blob/master/notebooks/sklearn-crfsuite.ipynb](https://github.com/TeamHG-Memex/eli5/blob/master/notebooks/sklearn-crfsuite.ipynb)]([https://github.com/TeamHG-Memex/eli5/blob/master/notebooks/sklearn-crfsuite.ipynb](https://github.com/TeamHG-Memex/eli5/blob/master/notebooks/sklearn-crfsuite.ipynb))), as well as the implementation of the NERP-CRF system [28]. The tasks performed to define and extract features were based on the work of [PERSON] and [PERSON]. [29]. The implemented model is a nondirectional graphical model used to calculate the conditional probability of the output nodes' values based on the values assigned to the corresponding input nodes. This model makes a first-order Markov independence assumption, so it can be understood as a conditionally trained finite state machine. The model has \(x=(x_{1},\,\ldots,\,x_{m})\) as input sequence, where x represents the set of ordered words belonging to a sentence. We have \(y=(y_{1},\,\ldots,\,y_{m})\) as the output sequence states that corresponds to the classes of named entities, where y is a set of Finite State Machines (FSM) corresponding to entity classes that match x. We model the conditional probability through \(p(y_{1},\ldots,y_{m}|x_{1},\ldots,x_{m})\). CRFs define a conditional probability of an output state given an input sequence, by the Hammersley-Clifford theorem [30]:  
\[P_{\Lambda}(y|x)=\frac{1}{Z_{x}}\exp(\sum_{m=1}^{M}\ \sum_{k}\lambda_{k}f_{k}(y_{m}-1,y_{m},x,m)) \tag{1}\]  
where \(Z_{x}\) is a normalization factor for all state sequences, \(\lambda k\) is a learned weight for each feature function, and \(f_{k}(y_{m-1},y_{m},x,m)\) is an arbitrary feature function over its arguments. The feature function has been set to have a value of 0 in most cases and to have a value of 1 if \(y_{m-1}\) is the state 1, where 0 corresponds to the tag 'O' and 1 corresponds to the tag 'PERSON'. The feature function can access the entire input sequence, including queries on previous and next words, so \(f_{k}(\cdot)\) can range between-\(\alpha\ldots\) +\(\alpha\). The features, \(f_{k}\), are based on the set of features used. In this implementation, the POS tags can be seen as pre-extracted features, but more features were extracted, such as:  
* Parts of words through stemming;
* Simplified POS tags;
* Confirmation of capital letters, lower case letters, titles, and digits;
* Resources from nearby words. To consider the effect of adding a new feature, a new sequence template is defined with an additional feature, \(g\), with weight \(\mu\). \[P_{\Lambda}+g,\mu(y|x)=\frac{P_{\Lambda}(y|x)\exp\left(\sum_{m-1}^{M}\mu g(y_{ m}-1,y_{m},x,m)\right)}{Z_{x}(\Lambda,g,\mu)} \tag{2}\]  
By converting the corpus to a dictionary list format, with the tokens and all associated features, we were able to train and test the CRF. Random Forest: The second statistical model implemented in this work, Random Forest model is a machine learning algorithm that works through decision trees. The model is trained to create a group of decision trees with a random subset of the data.