Fix parsing of 'custom' detect grouping values

Also, add error checking

Bug 892