libhb: Update param validation to return int instead of char.

Also adds prototype for hb_validate_filter_settings to common.h, fixes regex escaping, and simplifies some code.