numenta/NAB — pull request #2 by iandanforth (2014-06-16 19:56:53+00:00)
Title: Implement new scoring in confusion matrix

@subutai While I'm re-labeling data I would appreciate it if you would review the logic in the confusion matrix for scoring.

Here is the English description of what the code does:

#### Scoring Rules

- For each record check if it is labeled (ground truth) as an anomaly
- If it is an anomaly
- A detector has ALLOWED records to catch the anomaly
- For each record that follows the start of the anomaly where the anomaly is not caught, there is a small penalty (LATE)
- If the detector catches the anomaly, this is a True Positive
- Once an anomaly has been flagged there is a SUPPRESSION period
- To avoid spamming the end users a detector should not flag other records as anomalous during the SUPPRESSION period
- If a detector flags one *or more* additional records during a SUPPRESSION period, it is a False Positive (SPAM)
- This reflects the binary nature of Spam. Once the useful detection has been made everything else is spam. Lots of spam is only marginally worse than any spam.
- If a record is not flagged as an anomaly in the SUPPRESSION period this is a True Negative
- If ALLOWED records elapse without the anomaly being caught it is a False Negative
- If it is not an anomaly, and we're not in an ALLOWED period or in a SUPPRESSION period
- If it is flagged as an anomaly it is a False Positive
- Otherwise it is a True Negative

ALLOWED is usually equal to SUPPRESSION, but doesn't have to be.