libhb: fix a bunch of compiler warnings


git-svn-id: svn://svn.handbrake.fr/HandBrake/trunk@5905 b64f7644-9d1e-0410-96f1-a4d463321fa5