DataSoft/Nova — Issue #57: Accuracy bars/classification show little movement from the extremes (.9+ hostile and 95%+ accuracy)
Author: PherricOxide
Created: 2012-02-18 00:16:56+00:00
State: open
Labels: Component:ClassificationEngine, Type:Extension

# Background

The accuracy bars always show a very high accuracy, even if the training data is contrived in such a way they really shouldn't (say only using training data from some nmap scans, and then looking at the bar on normal web browsing). I believe this is an artifact of outliers harming the normalization process as well as the accuracy bars not using a good formula.

The accuracy is currently computed from the distance in the KNN tree between the neighbors and the nodes (a distance ranging from 0 and 1 due to our normalization) with the formula (1 - avg(distance[dimension]))*100.

Imagine the following example of a singe dimension.

0 ----+-+-+---------?------*-*-* 1

In this example, + is some classification (say hostile), * is another, and ? is the query point. It can be seen there that the query point is not terribly close to any of the training data (say a distance of 0.25 to it's neighbor). This would still result in a 75% accuracy bar, which seems like a fairly high number given the distribution of the data. Without very extreme data distributions, it's difficult to get lower numbers for the accuracy.

The problem is compounded by extremely large values throwing off the normalization. Supposing a very large feature value happens above,

0 ----+-+-+---------?------*-*-*-------------------------------------------* 1

Suddenly our distance to the neighbors is reduced to a very small amount (say 0.1). The problem further extends to the classification number, where it jumps to .9+ on even false positives rather than providing a smooth transition toward .9 with some suspects staying at classification somewhere in the middle. Now we've gone to a 90% accuracy and the classification number has jumped up toward the * side significantly due to the distance reduction no the neighbors.

# Solutions?

A good first step would be proving that this is really happening and the accuracy bars/classification level isn't borked because of something else. I propose we make a tool to visualize the dimensions of the training data as shown in the diagrams here so we can see data clumping, distribution, and extremes at a glance. We talked about this once but there wasn't a ticket for it.

A second thing to look into is defining a better formula for "accuracy" so the bars have some more movement in them. Maybe rather than going between 0 and 1 for the distance, we go between 0 and the maximum distance between neighbors. If the maximum a data point could be from a neighbor is only 0.2 on a fairly distributed set of points, there's no reason to say that 0.2 is actually a good accuracy, it's in fact the worst it could possibly be given the data.