Add check for valid compile/match flags, cleanup comments

GLib asserts on invalid flags, which is bad mojo in Wirehsark; so
instead we do our own verification and call lua_error instead.