libhb: Update param validation to return int instead of char.

Also adds prototype for hb_validate_filter_settings to common.h and simplifies some code.