fix for bug #575.

If sig has no_stream set, don't mask it as requiring flow.  Should get rid of
FNs any.
fix for bug #557.

Reset hhd buffers list len if we exit before allocating the buffer.
Introduce utility flow macros to help referencing/dereferencing flows.
fix for bug #557.

In FFRv2, dereference flow from a packet using the new reference/dereference
util macros.  This allows the decr use_cnt for flow and reseting the flow
pointer to NULL for the pseudo pkt to happen simultaneously, in case there we
fail to retrieve a pseudo_packet and have to return the already obtained
pseudo packets, back to the packetpool.
Update all flow referencing to use the new FlowReference and FlowDeReference
macros.
Fix flow keyword compilation failure.
Adds X-Forwarded-For support for the unified2 output.

It supports both the usage of the unified2 extra headers (like snort) and the IP replacement in the alert (so that the internal IP of the reverse proxy is replaced by the actual client IP taken from the XFF header).

The suricata.yaml configuration supports the usage of xff, xffheadername and xffoverwrite under the unified2-alert section in order to configure the mode of operation.

Example:

- unified2-alert:
      enabled: yes
      filename: unified2.alert
      xff: yes
      xffheadername: X-Forwarded-For
      xffoverwrite: yes

xffoverwrite specifies the mode of operation. If it is set to yes, the alert IP address will be overwritten by the IP taken from the XFF header of the request which triggered the alert.
bug #572: make sure we use profiling fallback for all architectures except x86_64 and i386.
refactor htpconfigure()
Make available custom features of libhtp.

The power of libhtp customisation now available to users.

Options available -

path-backslash-separators: yes
path-compress-separators: yes
path-control-char-handling: none
path-convert-utf8: yes
path-decode-separators: yes
path-decode-u-encoding: yes
path-invalid-encoding-handling: preserve_percent
path-invalid-utf8-handling: none
path-nul-encoded-handling: none
path-nul-raw-handling: none
set-path-replacement-char: ?
set-path-unicode-mapping: bestfit

You can use this for your libhtp customisation.  Options explained in our
wiki.

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Advanced_libhtp_customization
http: fix multipart parsing leading to missing chunks of files in file extraction.
htp: update version numbers of bundled htp
fix for bug #577.

If a pattern has matched on mpm, don't re-inspect it later, subject to certain
conditions met by the pattern - namely, not negated, right chop, no replacet
attached to it.
fix for bug #574.

More of a temporary solution to prevent any possible FPs.  Disable content
inspection bypass for mpm patterns.
pf-ring: protect definition of (un)likely

This patch makes (un)likely declared if and only if they are not
declared before.
pf-ring: add missing header.
pf-ring: suppress unused variable.
fast_pattern: don't consider http_method, http_stat_code and http_stat_msg when automatically giving preference to a HTTP pattern over a stream pattern.
nfq: close the queue when leaving acquisition.

This patch adds a call to close the queue when the acquisition
loop is ending. This way the incoming packets will be accepted
during all the shutdown phase (if the queue-bypass option of
NFQUEUE is used). At the same time the currently processed packets
will be dropped but the time scale are different: suricata will
drop 20 ms of packets and the shutdown can take 0.5 seconds.

Patch based on an idea of Victor Julien.
nfq: fix detection of type nfq_get_payload function.