avoid encoder crash with random pixel input


Bring in the fix from pull request 1285 into the v1.1-Firefox33 branch.