inliniac/suricata — pull request #1008 by ken-tilera (2014-07-02 14:12:15+00:00)
Title: Replace ringbuffer in Packet Pool with a stack for better cache locality

Replaces:PR 1005

Implements optimization 1039 (https://redmine.openinfosecfoundation.org/issues/1039#change-4093)
Using a stack for free Packet storage causes recently freed Packets to be
reused quickly, while there is more likelihood of the data still being in
cache.

The new structure has a per-thread private stack for allocating Packets
which does not need any locking. Since Packets can be freed by any thread,
there is a second stack (return stack) for freeing packets by other threads.
The return stack is protected by a mutex. Packets are moved from the return
stack to the private stack when the private stack is empty.

Returning packets back to their "home" stack keeps the stacks from getting out
of balance.

For better performance with autofp runmode, where one thread does most of the allocation and the other threads free the packets, each thread stores up freed packets for one packet pool until it reaches a threshold (32 currently) and then returns the list of freed packets with one lock acquire.

The PacketPoolInit() function is now called by each thread that will be
allocating packets. Each thread allocates max_pending_packets, which is a
change from before, where that was the total number of packets across all
threads.

This code uses __thread , if TLS is defined, otherwise pthread thread specific API, to store one packet pool per thread.

Not run through PR script as that machine is down.