avcodec/snow: check for malloc* failures

Also return and pass on error codes where needed for this

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
snowenc: Fix memleak of the ratecontrol related memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>