Bug 1329: error out on invalid rule protocol

Due to a logic error in AppLayerProtoDetectGetProtoByName invalid
protocols would not be detected as such. Instead of ALPROTO_UNKNOWN
ALPROTO_MAX was returned.

Bug #1329 https://redmine.openinfosecfoundation.org/issues/1329

Prscript:
- PR build: https://buildbot.openinfosecfoundation.org/builders/inliniac/builds/72
- PR pcaps: https://buildbot.openinfosecfoundation.org/builders/inliniac-pcap/builds/72

CC: @duanehoward