detect: modify AMATCH locking

This is an intrusive change. This patch modifies the way AMATCH
inspection uses locking.

So far, each keyword did it's own locking. This lead to a situation
where a 'alstate' pointer was passed around that was not always
protected by a lock.

This patch moves the locking to the Stateful detection functions.
detect: cleanup

Remove unused alstate and app layer flags arguments from
DetectEngineInspectPacketPayload()
detect: locking update

Make DeStateDetectStartDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
detect: locking update continued

Make DeStateDetectContinueDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
detect: fix alstate handling

Previously, the alstate use in the main detect loop was unsafe. The
alstate pointer would be set duing a lock, but it would again be used
after one or more lock/unlock cycles. If the data pointed to would
disappear, a dangling pointer would be the result.

Due to they way flows are cleaned up using reference counting and
such, changes of this happening were very small. However, at least
one path can lead to this situation. So it had to be fixed.
Fix unaligned load in AC-TILE MPM.

The SLOAD define using __insn_ld2s_L2 is used to provide a compiler
hint that the load will come from the L2 cache instead of the L1. It
also specifies that it is a 2 byte signed load. For the Tiny MPM, that
needs to be a 1-byte load, which is what is specified in util-ac-mpm-tile.c,
but the #undef was removing that definition.
refactor IDS/IPS engine mode logic

Instead of error phrone externs with macro's, use functions with a local
static enum var instead.

- EngineModeIsIPS(): in IPS mode
- EngineModeIsIDS(): in IDS mode

To set the modes:

- EngineModeSetIDS(): IDS mode (default)
- EngineModeSetIPS(): IPS mode

Bug #1177.
stream: improve retransmission handling

When connection are closing, don't reject retransmissions of data
packets.

Bug #1180.
app-layer: proto detection update

Instead of the notion of toserver and toclient protocol detection, use
destination port and source port.

Independent of the data direction, the flow's port settings will be used
to find the correct probing parser, where we first try the dest port,
and if that fails the source port.

Update the configuration file format, where toserver is replaced by 'dp'
and toclient by 'sp'. Toserver is intrepreted as 'dp' and toclient as
'sp' for backwards compatibility.

Example for dns:

    dns:
      # memcaps. Globally and per flow/state.
      #global-memcap: 16mb
      #state-memcap: 512kb

      # How many unreplied DNS requests are considered a flood.
      # If the limit is reached, app-layer-event:dns.flooded; will match.
      #request-flood: 500

      tcp:
        enabled: yes
        detection-ports:
          dp: 53
      udp:
        enabled: yes
        detection-ports:
          dp: 53

Like before, progress of protocol detection is tracked per flow direction.

Bug #1142.
proto-detect: update port logic

If a flow matches both an 'sp' based PP registration and a 'dp' based,
until now we would only check the 'dp' one. This patch changes that. It
will inspect both.
proto-detect: masks cleanup

The direction specific masks were not used correctly. The toserver ones
were only used for 'dp' registrations, the toclient ones only for 'sp'.

The patch merges them.
packet handling: fix release function

Extended data were freed before the release function was called.
The result was that, in AF_PACKET IPS mode, the release function
was only sending void data because it the content of the extended
data is the content of the packet.

This patch updates the code to have the freeing of extended data
done in the cleaning function for a packet which is called by the
release function. This improves consistency of the code and fixes
the bug.
dns: improve response name parsing

Improve parsing of names with multiple pointers following each other.
byte-extract: switch to pcre_copy_substring

Fixes memory leaks (str_ptr was never freed).
Fixes cppcheck warning:
    [src/detect-byte-extract.c:268]: (error) Uninitialized variable: str_ptr

Bug #1046.
byte-test: switch to pcre_copy_substring

Fixes cppcheck:
[src/detect-bytejump.c:360]: (error) Uninitialized variable: str_ptr
[src/detect-bytejump.c:372]: (error) Uninitialized variable: str_ptr
[src/detect-bytejump.c:375]: (error) Uninitialized variable: str_ptr

Bug #1046.
stream: fix midstream syn/ack setup

If midstream is enabled and the first packet is the syn/ack packet from
the 3whs, initialized server.last_ack to the packets seq.

This fixes tracking the session.
protocol detection: midstream handling update

If a TCP session is midstream, we may end up with a case where the
start of an HTTP request is missing. We won't detect HTTP based on
the request. However, the reply is fine, so we detect HTTP anyway.
This leads to passing the incomplete request to the htp parser.

This has been observed, where the http parser then saw many bogus
requests in the incomplete data. This is not limited to HTTP.

To counter this case, a midstream session MUST find it's protocol
in the toserver direction. If not, we assume the start of the
request/toserver is incomplete and no reliable detection and parsing
is possible. So we give up.
proto detect: add cutoff for unbalanced traffic

If we're getting a lot of data in one direction and the proto for this
direction is unknown, proto detect will hold up segments in the segment
list in the stream. They are held so that if we detect the protocol on
the opposing stream, we can still parse this side of the stream as well.
However, some sessions are very unbalanced. FTP data channels, large
PUT/POST request and many others, can lead to cases where we would have
to store many megabytes worth of segments before we see the opposing
stream. This leads to risks of resource starvation.

In this patch, a cutoff point is enforced. If we've stored 100k in one
direction and we've seen no data in the other direction, we give up.

If we've given up, the applayer_proto_detection_skipped event is set.
    app-layer-event: applayer_proto_detection_skipped;
stream: detect data gap at stream start

In midstream mode we may encounter a case where the data we is beyond
the isn, but below last_ack. This means we're missing some data, that
is already acked so it won't be retransmitted. Therefore, we can
conclude it's a data gap.
stream: improve midstream reassembly gap detection

The reassembly gap detection makes use of the window. However, in
midstream mode the window size we use is unreliable, as we have to
assume window scaling is in place. This patch improves midstream
handling of those cases.