app-layer: proto detection update v3.1

Rebase of https://github.com/inliniac/suricata/pull/920

Bug #1142. https://redmine.openinfosecfoundation.org/issues/1142
Bug #1162. https://redmine.openinfosecfoundation.org/issues/1162

Prscript:
- PR build: https://buildbot.suricata-ids.org/builders/inliniac/builds/260
- PR pcaps: https://buildbot.suricata-ids.org/builders/inliniac-pcap/builds/179