Fix parsing of 'custom' detect grouping values

Also, add error checking

Bug 892

https://redmine.openinfosecfoundation.org/issues/892
https://buildbot.suricata-ids.org/builders/inliniac/builds/69