inliniac/suricata — pull request #913 by ken-tilera (2014-03-28 19:34:35+00:00)
Title: Replace ringbuffer in Packet Pool with a stack for better cache locality

Implements optimization 1039 (https://redmine.openinfosecfoundation.org/issues/1039#change-4093)
Using a stack for free Packet storage causes recently freed Packets to be
reused quickly, while there is more likelihood of the data still being in
cache.

The new structure has a per-thread private stack for allocating Packets
which does not need any locking. Since Packets can be freed by any thread,
there is a second stack (return stack) for freeing packets by other threads.
The return stack is protected by a mutex. Packets are moved from the return
stack to the private stack when the private stack is empty.

Returning packets back to their "home" stack keeps the stacks from getting out
of balance.

The PacketPoolInit() function is now called by each thread that will be
allocating packets. Each thread allocates max_pending_packets, which is a
change from before, where that was the total number of packets across all
threads.

This code depends on __thread to store one packet pool per thread. I need to find a solution for BSD, but wanted to get feed back on this approach first. One thought the case of not having __thread is to simply have one central buffer stack using a lock. I looked into storing the PktPool in ThreadVars, but that was not available every place a packet is allocated.

Passes regression script:
https://buildbot.suricata-ids.org/builders/ken-tilera/builds/133
https://buildbot.suricata-ids.org/builders/ken-tilera-pcap/builds/67

@poona you might want to see if this is going to break Cuda.