avcodec/vda_h264_dec: add format check

Fix crash when using this decoder for unsupported format on some version of OS X.