avoid encoder crash with random pixel input

review at:
https://rbcommons.com/s/OpenH264/r/761/