detection engine port api unittests cleanup
file: implement filesize keyword. #489.
stream/app layer: add Truncate app layer callback that is called if stream depth is reached. Use it to trunc open files in HTTP.
stream/app layer: call new Truncate callback for data gap case as well.
file: convert filesize to new FileMatch api.
Add counters for IPv4 in IPv6 and IPv6 in IPv6
Make sure we never underflow len in DetectLoadSigFile
af-packet: delete design comments
af-packet: improve mmaped running mode.

The mmaped mode was using a too small ring buffer size which was
not able to handle burst of packets coming from the network. This
may explain the important packet loss rate observed by Edward
Fjellskål.
This patch increases the default value and adds a ring-size
variable which can be used to manually tune the value.
capture: add data release mechanism

This patch adds a data release mechanism. If the capture module
has a call to indicate that userland has finished with the data,
it is possible to use this system. The data will then be released
when the treatment of the packet is finished.

To do so the Packet structure has been modified:
+    TmEcode (*ReleaseData)(ThreadVars *, struct Packet_ *);
If ReleaseData is null, the function is called when the treatment
of the Packet is finished.
Thus it is sufficient for the capture module to code a function
wrapping the data release mechanism and to assign it to ReleaseData
field.

This patch also includes an implementation of this mechanism for
AF_PACKET.
af-packet: IPS and TAP feature

This patch adds a new feature to AF_PACKET capture mode. It is now
possible to use AF_PACKET in IPS and TAP mode: all traffic received
on a interface will be forwarded (at the Ethernet level) to an other
interface. To do so, Suricata create a raw socket and sends the receive
packets to a interface designed in the configuration file.

This patch adds two variables to the configuration of af-packet
interface:
 copy-mode: ips or tap
 copy-iface: eth1 #the interface where packet are copied
If copy-mode is set to ips then the packet wth action DROP are not
copied to the destination interface. If copy-mode is set to tap,
all packets are copied to the destination interface.
Any other value of copy-mode results in the feature to be unused.
There is no default interface for copy-iface and the variable has
to be set for the ids or tap mode to work.

For now, this feature depends of the release data system. This
implies you need to activate the ring mode and zero copy. Basically
use-mmap has to be set to yes.

This patch adds a peering of AF_PACKET sockets from the thread on
one interface to the threads on another interface. Peering is
necessary as if we use an other socket the capture socket receives
all emitted packets. This is made using a new AFPPeer structure to
avoid direct interaction between AFPTreadVars.

There is currently a bug in Linux kernel (prior to 3.6) and it is
not possible to use multiple threads.

You need to setup two interfaces with equality on the threads
variable. copy-mode variable must be set on the two interfaces
and use-mmap must be set to activated.

A valid configuration for an IPS using eth0 and vboxnet1 interfaces
will look like:

af-packet:
  - interface: eth0
    threads: 1
    defrag: yes
    cluster-type: cluster_flow
    cluster-id: 98
    copy-mode: ips
    copy-iface: vboxnet1
    buffer-size: 64535
    use-mmap: yes
  - interface: vboxnet1
    threads: 1
    cluster-id: 97
    defrag: yes
    cluster-type: cluster_flow
    copy-mode: ips
    copy-iface: eth0
    buffer-size: 64535
    use-mmap: yes
af-packet: add doxygen comments

This patch adds doxygen comments to newly introduced function and adds
module AF_PACKET doxygen module with a dedicated AFP peers module.
af-packet: fix runmode name in logging function
af-packet: reorder socket operation.

This patch moves raw socket binding at the end of init code to
avoid to have a flow of packets reaching the socket before we
start to read them.

The socket creation is now made in the loop function to avoid
any timing issue between init function and the call of the loop.
af-packet: add optional emergency mode

Flush all waiting packets to be in sync with kernel when drop
occurs. This mode can be activated by setting use-emergency-flush
to yes in the interface configuration.
af-packet: detect MTU mismatch and warn user

If the MTU on the reception interface and the one on the transmission
interface are different, this will result in an error at transmission
when sending packet to the wire.
sig: Add ipv6 and ipv4 to list of protocols

With this patch it is possible to do:
 alert ipv6 any any -> any any
or
 alert ip4 any any -> any any
to match on IPv4 or IPv6 packets.
sig: add l3_proto keyword

This patch adds a l3_proto keyword to the signature language. It
can be used to specify if the signature has to match on IPv4, IPv6
or both. For example, one can write:
  alert http any any -> any 22 (msg: "HTTP v6"; l3_proto:ip6; sid:14;)

This should close #494.
l3proto: add unit tests

This patch adds a series of unit tests. First two check test the keyword
by checking packet on signatures using it. Last one adds is here to check
that there is no interaction of l3_proto and ip_proto.
pool: alloc a single area for all PoolBuckets

As we know the number and the size of PoolBucket, we can simply
allocate a single memory zone.