DataSoft/Nova — Issue #17: Eliminate outliers from data set when classifying
Author: DataSoft
Created: 2012-02-16 19:24:23+00:00
State: open
Labels: Classification Engine, enhancement

When we're calculating feature sets for suspects, it would be good to eliminate outliers in the evidence data set. For example, if we're looking for Mean Packet size, we might get a large stream of small packets (which indicates a possible scanning attempt). But then we shouldn't skew this calculation simply because a single huge packet comes in afterward.

We should eliminate the top and bottom 5% (or whatever other percentage is deemed appropriate. Make this configurable) of every data set. So when we have a list of packet sizes, we should ignore the very top and very bottom.