DataSoft/Nova — Issue #30: Invent a suite of test scans for classification accuracy
Author: DataSoft
Created: 2012-02-16 19:25:24+00:00
State: open
Labels: Nova System, task

One of the problems we face for development is that we have no empirical basis for assertions about classification accuracy. We have only one scan defined, and we're running Nova against that.

What we need to do is define a whole suite of scans. Some that are really fast, some that are really slow. Some that have ICMP, some with UDP scans. All varieties of TCP flags (TCP SYN scans, CONN scans, FIN, etc...)

So we have some large set of N scans we can test Nova against. Then when we make a change to the Nova system, we can say:

"Nova version A detected 76% of our scans, and Nova version B detected 82%"

This will need to be done using nmap. Also, we're going to need a method for actually running these in the lab. Some method of pressing a button and letting the scans run and observing the results later. (That part might be difficult)