libhb: Add filter parameter validation to param.c.

Updates test.c to bail with error on invalid parameters. Fixes a couple small bugs and cosmetics.