avcodec/vda_h264_dec: add format check

Fix crash when using this decoder for unsupported format on some version
of OS X.
Merge branch 'master' of https://github.com/upsuper/ffmpeg-vdadec

* 'master' of https://github.com/upsuper/ffmpeg-vdadec:
  avcodec/vda_h264_dec: add format check

Merged-by: Michael Niedermayer <michaelni@gmx.at>
avcodec/vda_h264_dec: avoid declaring int in for arguments

Some compilers dont support that

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>