Bug fix: IPv6 addresses in negated range and IPv6 string into radix tree.

I found three somewhat serious IPv6 address bugs within the Suricata 2.0.x source code. Two are in the source module "detect-engine-address.c", and the third is in "util-radix-tree.c".

The first bug occurs within the function DetectAddressParse2(). When parsing an address string and a negated block is encountered (such as when parsing !$HOME_NET, for example), any corresponding IPv6 addresses were not getting added to the Group Heads in the DetectAddressList. Only IPv4 addresses were being added.

I discovered another bug related to IPv6 address ranges in the Signature Match Address Array comparison code for IPv6 addresses. The function DetectAddressMatchIPv6() walks a signature's source or destination match address list comparing each to the current packet's corresponding address value. The match address list consists of value pairs representing a lower and upper IP address range. If the packet's address is within that range (including equal to either the lower or upper bound), then a signature match flag is returned.

The original test of each signature match address to the packet was performed using a set of four compounded AND comparisons looking at each of the four 32-bit blocks that comprise an IPv6 address. The problem with the old comparison is that if ANY of the four 32-bit blocks failed the test, then a "no-match" was returned. This is incorrect. If one or more of the more significant 32-bit blocks met the condition, then it is a match no matter if some of the less significant 32-bit blocks did not meet the condition. Consider this example where Packet represents the packet address being checked, and Target represents the upper bound of a match address pair. We are testing if Packet is less than Target.

Packet -- 2001:0470 : 1f07:00e2 : 1988:01f1 : d468:27ab
Target -- 2001:0470 : 1f07:00e2 : a48c:2e52 : d121:101e

In this example the Packet's address is less than the target and it should give a match. However, the old code would compare each 32-bit block (shown spaced out above for clarity) and logically AND the result with the next least significant block comparison. If any of the four blocks failed the comparison, that kicked out the whole address. The flaw is illustrated above. The first two blocks are 2001:0470 and 1f07:00e2 and yield TRUE; the next less significant block is 1988:01f1 and a48c:2e52, and also yields TRUE (that is, Packet is less than Target); but the last block compare is FALSE (d468:27ab is not less than d121:101e). That last block is the least significant block, though, so its FALSE determination should not invalidate a TRUE from any of the more significant blocks. However, in the previous code using the compound logical AND block, that last least significant block would invalidate the tests done with the more significant blocks.

The other bug I found for IPv6 occurs when trying to parse and insert an IPv6 address into a Radix Tree using the function SCRadixAddKeyIPV6String(). The test for min and max values for an IPv6 CIDR mask incorrectly tests the upper limit as 32 when it should be 128 for an IPv6 address. I think this perhaps is an old copy-paste error if the IPv6 version of this function was initially copied from the corresponding IPv4 version directly above it in the code. Without this patch, the function will return null when you attempt to add an IPv6 network whose CIDR mask is larger than 32 (for example, the popular /64 mask will cause the function to return the NULL error condition).

(amended by Victor Julien)
suricatasc: factorize command parsing
suricatasc: add -c flag to run command

Add -c flag to run command given as argument and return the raw
JSON result. For example, it is possible to run something like.
 $ suricatasc  -c "iface-stat eth0"
 {'message': {'pkts': 17838352, 'drop': 0, 'invalid-checksums': 1}, 'return': 'OK'}
source-nfq: fix display of next queue

Suricata was displaying an invalid queue number as the value is
shift at the moment of its assignement.
tls: fix a tls.fingerprint issue in debug mode

Print proper variable so we don't NULL-deref.

Bug #1279.
lua: fix http.request_line inspection

As there is no inspection engine for request_line, the sigmatch was
added to the AMATCH list. However, no AppLayerMatch function for
lua scripts was defined.

This patch defines a AppLayerMatch function.

Bug #1273.
detect: fix continue detection with amatch and tx

When using AMATCH, continue detection would fail if the tx part
had already run. This lead to start detection rerunning, causing
multiple alerts for the same issue.
rule parser: fix crashing on malformed options

Fix crashing on malformed rule options like 'rev;1;'.

Bug 1254.
rule parser: set flag for optionless keywords

If a keyword doesn't have an argument, it should set the SIGMATCH_NOOPT
flag so the parser knows.
af-packet: force suricata in IPS mode when needed

AF_PACKET is not setting the engine mode to IPS when some
interfaces are peered and use IPS mode. This is due to the
fact, it is possible to peer 2 interfaces and run an IPS on
them and have a third one that is running in normal IDS mode.

In fact this choice is the bad one as unwanted side effect is
that there is no drop log and that stream inline is not used.

To fix that, this patch puts suricata in IPS mode as soon as
there is two interfaces in IPS mode. And it displays a error
message to warn user that the accuracy of detection on IDS only
interfaces will be low.
af-packet: check pointers before use
ipv6: RH extension header parsing issue

A logic error in the IPv6 Routing header parsing caused accidental
updating of the original packet buffer. The calculated extension
header lenght was set to the length field of the routing header,
causing it to be wrong.

This has 2 consequences:

1. defrag failure. As the now modified payload was used in defrag,
the decoding of the reassembled packet now contained a broken length
field for the routing header. This would lead to decoding failure.

The potential here is evasion, although it would trigger:
[1:2200014:1] SURICATA IPv6 truncated extension header

2. in IPS mode, especially the AF_PACKET mode, the modified and now
broken packet would be transmitted on the wire. It's likely that
end hosts and/or routers would reject this packet.

NFQ based IPS mode would be less affected, as it 'verdicts' based on
the packet handle. In case of replacing the packet (replace keyword
or stream normalization) it could broadcast the bad packet.

Additionally, the RH Type 0 address parsing was also broken. It too
would modify the original packet. As the result of this code was not
used anywhere else in the engine, this code is now disabled.

Reported-By: Rafael Schaefer <rschaefer@ernw.de>
ipv6: convert ext header pointers to const

To prevent accidental writes into the orignal packet buffer, use
const pointers for the extension header pointers used by IPv6. This
will cause compiler warnings in case of writes.
app-layer-ssh: fix banner parser

Carefully crafted SSH banner could result in parser error.

Signed-off-by: Eric Leblond <eric@regit.org>
ssh: convert error message to debug statement

Don't print errors based on traffic issues.
MPM: build sid list from MPM matches

Pmq add rule list: Array of uint32_t's to store (internal) sids from the MPM.

AC: store sids in the pattern list, append to Pmq::rule_id_array on match.

Detect: qsort rule_id_array after it was set up by the MPM. Rule id's
(Signature::num) are ordered, and the rule's with the lowest id are to
be inspected first. As the MPM doesn't fill the array in order, but instead
'randomly' we need this sort step to assure proper inspection order.

**TODO**:

Currently we use a new array 'rule_id_array'. However, in a few places,
the 'pattern_id_array' is used in a similar way (proto detect and perhaps
smtp?). So we could convert those to use the rule_id_array and remove the
pattern_id_array completely.
Detect: create per sgh non-MPM rule array

Array of rule id's that are not using MPM prefiltering. These will be
merged with the MPM results array. Together these should lead to a
list of all the rules that can possibly match.
Add negated MPM to non-MPM array (HACK)

Treat negated MPM sigs as if non-MPM, so we consider them always.

As MPM results and non-MPM rules lists are now merged and considered
for further inspection, rules that need to be considerd when a pattern
is absent are caught in the middle.

As a HACK/workaround this patch adds them to the non-MPM list. This
causes them to be inspected each time.

**TODO**
This will likely perform bad if there are a lot of rules using negated
MPM's. In the ET/ETpro rulesets those rules are quite rare though.
Replace build match array with new filter logic

Use MPM and non-MPM lists to build our match array. Both lists are
sorted, and are merged and sorted into the match array.

This disables the old match array building code and thus also bypasses
the mask checking.

**TODO**
See if we should reintroduce parts of the mask and other prefilter
logic.

**TODO**
Having a zero sized list is quite common for both lists, so we can
probably add some optimization for this.
Add detect debug code

Add some expensive code to print the input and output of the
lists merging function.

**TODO**
Remove before merging this branch.