## 7 Validation and Results
### Validation Methodology  
* To evaluate the deep learning model (Figure 3) the corpus described in Section 4.1 was used, this corpus contains 14,750 annotated sentences. The corpus was shuffled and randomly split into three sets: training (80%), development (10%), and test (10%). This procedure was independently repeated ten times, while verifying that each set contained all the labels in the corpus. The test set was used to calculate the performance metrics, as shown in the Equations (1) and (2). The F-score (Equation (3)) is calculated as a weighted average of the Precision and Recall measurements. The BiLSTM-CRF model was developed using TensorFlow ([[https://www.tensorflow.org/?hl=es-419](https://www.tensorflow.org/?hl=es-419)]([https://www.tensorflow.org/?hl=es-419](https://www.tensorflow.org/?hl=es-419))) and Keras ([[https://keras.io/](https://keras.io/)]([https://keras.io/](https://keras.io/))) using the following parameters: learning rate as 0.001, dropout as 0.5, the number of epochs is 30, the BiLSTM hidden size is 300, and the batch size is 512. \[\textbf{Precision}=\frac{\text{Number of entities correctly predicted}}{\text{Number of predicted entities}}\] (1) \[\textbf{Recall}=\frac{\text{Number of entities correctly predicted}}{\text{Number of entities in the test set}}\] (2) \begin{table}
\begin{tabular}{c c c} \hline \hline
**Cancer Diagnosis Concept** & **Date** & **Event** \\ \hline Lung cancer & 25 March 2014 & \\ Cancer & March 2015 & Begins treatment \\ Adenocarcinoma & 17 May 2017 & \\ Lung carcinoma & 14 January 2014 & Begins clinical trial \\ Small cell lung carcinoma & July 2014 & Diagnosed \\ Lung neoplasm & 12 July 2016 & \\ Cancer & 17 May 2014 & \\ Adenoca & April 2018 & \\ Carcinoma & 18 September 2017 & Begins surgery \\ \hline \hline \end{tabular}
\end{table} 
Table 2: A set of extracted records. \[\textbf{F-score}=2*\frac{\text{Precision * Recall}}{\text{Precision + Recall}} \tag{3}\]
* To validate the negation and speculation detection step, the NUBES corpus proposed by [12] was used. This public corpus is annotated with speculation and negation in clinical notes written in Spanish. We analyze the performance for each sub-task: _cue detection_ and _scope recognition_. * A speculation or negation cue is correctly detected when given a sentence, the rule-based approach is able to recognize the cues indicated by the test corpus. * A scope is correctly detected when given a sentence and a detected cue, the approach is able to recognize the scope indicated in the corpus. Equations (4) and (5) are used to calculate Precision and Recall for the cue detection sub-task. On the other hand, Equations (6) and (7) are used to calculate Precision and Recall for the scope detection sub-task. The F-score is calculated as a weighted average of the Precision and Recall. \[\textbf{Cue Precision}=\frac{\text{Number of cues correctly detected}}{\text{Number of detected cues}} \tag{4}\]  
\[\textbf{Cue Recall}=\frac{\text{Number of cues correctly detected}}{\text{Number of the cues in the corpus}} \tag{5}\]  
\[\textbf{Scope Precision}=\frac{\text{Number of scopes correctly detected}}{\text{Number of detected scopes}} \tag{6}\]  
\[\textbf{Scope Recall}=\frac{\text{Number of scopes correctly detected}}{\text{Number of the scopes in the corpus}} \tag{7}\]
* To validate the diagnosis date extraction, a database containing data from _\"Hospital Universitario Puer de Hierro Madrid\"_ was used. It contains around 300,000 clinical notes corresponding to 1000 patients that were diagnosed with lung cancer in the last ten years. Information of the diagnosis date for each patient is available. A diagnosis date is correctly extracted when it corresponds to the diagnosis date given by the hospital dataset. Equations (8) and (9) are used to calculate the Precision and Recall respectively.