inliniac/suricata — pull request #1114 by inliniac (2014-08-19 16:08:23+00:00)
Title: [WIP] mpm optimization v0.2

*Not for merge*

Change MPM prefiltering logic. Instead of the current unconditional sig array walk we assemble a sig array based on MPM matches and non-MPM sigs on the fly, and inspect only those sigs. This seems to be *much* faster, especially with large amounts of rules.

TODO:
Lots of things. Optimizations. Also:
- pmq rule_id_array allocs a fixed size array. Find a better way. Can't use max-pattern-id or max-rule-id as the list can contain dupes currently.
- all mpm algo's other than plain AC are broken/not supported
- much more testing
- TODO's in each commit message
- unittests are broken for all non-AC tests
- unittests run out of memory because of missing cleanups in existing code, combined with high memory use of the updated pmq (see first issue above).

CC: @ken-tilera, @poona, @regit, @pevma