encx265: miscellaneous fixes, cleanup and refactoring.


git-svn-id: svn://svn.handbrake.fr/HandBrake/trunk@6026 b64f7644-9d1e-0410-96f1-a4d463321fa5