stream: improve retransmission handling

Improve retransmission handling in the CLOSE_WAIT state.

Bug #1180.
json-file: check pointer before using

A check was missing to see if ht_ud was not null before using the
pointer. This should be rare, but it can happen.

Reported-by: Will Metcalf
json-file: improve error handling

If the functions getting uri, etc fail. Return "unknown" instead of
NULL pointer. This improves consistency.
DAG: Sync dag packet and drop counts to live device on exit for better exit
logging.
DAG: Consistent code style.
DAG: Pull some raw values out into defines.
af-packet: exit in case of a fatal error

During socket creation all error cases were leading to suricata to
retry the opening of capture. This patch updates this behavior to
have fatal and recoverable error case. In case of a fatal error,
suricata is leaving cleanly.
rohash: fix potential bad shift

Fix issue detected byCoverity:

*** CID 1197756:  Bad bit shift operation  (BAD_SHIFT)
/src/util-rohash.c: 74 in ROHashInit()
68         }
69         if (hash_bits < 4 || hash_bits > 32) {
70             SCLogError(SC_ERR_HASH_TABLE_INIT, "invalid hash_bits setting, valid range is 4-32");
71             return NULL;
72         }
73
>>>     CID 1197756:  Bad bit shift operation  (BAD_SHIFT)
>>>     In expression "1U << hash_bits", left shifting by more than 31 bits has undefined behavior.  The shift amount, "hash_bits", is as much as 32.
74         uint32_t size = hashsize(hash_bits) * sizeof(ROHashTableOffsets);
75
76         ROHashTable *table = SCMalloc(sizeof(ROHashTable) + size);
77         if (unlikely(table == NULL)) {
78             SCLogError(SC_ERR_HASH_TABLE_INIT, "failed to alloc memory");
79             return NULL;

This was only a potential issue as ROHashInit was only called with
hash_bits 16 in the code.

Bug #1170.
doxygen: add source browser
json: add custom output capability to http eve log
json: address custom output capability to http eve log review comments
http-json: init 'fields' to 0 before setting it

httplog_ctx->fields would not be initialized before setting flags in
it:

Scanbuild:
output-json-http.c:491:46: warning: The left expression of the compound assignment is an uninitialized value. The computed value will also be garbage
                            http_ctx->fields |= (1<<f);
                            ~~~~~~~~~~~~~~~~ ^
1 warning generated.

Drmemory:
~~27874~~ Error #1: UNINITIALIZED READ: reading register eax
~~27874~~ # 0 JsonHttpLogJSON                       [/home/buildbot/qa/buildbot/donkey/drmemory/Suricata/src/output-json-http.c:260]
~~27874~~ # 1 JsonHttpLogger                        [/home/buildbot/qa/buildbot/donkey/drmemory/Suricata/src/output-json-http.c:375]

Just memset the whole structure right after initialition.
detect: add more defensive checks for flow handling

Don't unconditionally deref f->alparser in detection through
DeStateFlowHasInspectableState(). In very rare cases it can
be NULL.
Look for Mismatched Encrypted HB request and response sizes, along with multiple in-flight HB requests from the same direction
tls/heartbleed: add rule for invalid encrypted hb

Add rule to tls-events.rules to match on the invalid encrypted
heartbeat.
tls/heartbleed: formatting fixes
tls/heartbleed: improve encrypted logic

Don't assume that if the type field isn't 01 or 02 it's an encrypted
heartbeat. Instead, use our knowledge of the SSL state.
tls/heartbleed: fix test

Now that we continue to track ssl/tls after the handshake, we need
to fix tests that checked for the cutoff flags.
app-layer: improve no payload inspect flag

If setting APP_LAYER_PARSER_NO_INSPECTION_PAYLOAD, trigger raw
reassembly.
stream: unify segment discard handling

Have a single function StreamTcpReturnSegmentCheck determine if a
segment is ready to be removed from the stream.

Handle FLOW_NOPAYLOAD_INSPECT in raw reassembly.