dotse/PacketQ — pull request #9 by OleLaursen (2013-05-30 15:40:09+00:00)
Title: Rework input layer to stream packets

This is a rework of the input layer of PacketQ to make it read one package and process it at the time instead of gobbling it all up in a table and go from there.

To make it work, I had to revamp the aggregation functions to use a hash table instead of sorting the input. I also changed the signatures of some evaluation functions to take an array of tables instead of storing a source row pointer on each destination row (these pointers are stale upon reading next row). This appears to have been the original intention with the code.

The result is that I can process a 27 MB DNS packet file with appr. 1.5 MB peak memory instead of 89 MB peak memory. It's also a bit faster - I had some troubles measuring this, but it seems to go from around 350 ms to 300 ms.

I added a simple regression testing script. It would be nice with a real test suite but this simple script helped me.

Finally, I enabled -Wall (without a couple specific warnings) and fixed a few issues related to that.

I have used some new C++ constructs so I think GCC >= 4.4 is needed, or a new version of Clang. I think in general it would help to use more of these in the codebase, especially regarding memory handling. I found and fixed at least two memory leaks, one minor and one major.