To calculate these values, pneumonia and influenza related deaths were examined by comparing the reference standard output vs. the two comparators: DCP and keyword search. For both comparators, the deaths were counted and categorized as TRUE POSI-TIVES (cases found by the comparator-pneumonia deaths being correctly classified); FALSE POSITIVES (incorrect cases found by the comparator-the number of pneumonia and influenza deaths incorrectly identified by the comparator); FALSE NEGATIVES (correct cases not found by the comparator-the number of pneumonia deaths not identified by the comparator). Precision, recall and F-score were calculated as follows: Precision = True Positives/(True Positives + False Positives) (1) Recall = True Positives/(True Positives + False Negatives) (2) F-measure = 2 *(P R/ P + R) (3) McNemar's test was also calculated to evaluate the significance of the difference between the two comparators. To calculate this value a confusion matrix was created where A is the number of times both methods have correct predictions; B is the number of times method 1 has a correct prediction and method 2 has a wrong prediction; C is the number of times method 2 has a correct prediction and method 1 has a wrong prediction; D is the number of times both methods have incorrect predictions.