avcodec/vdpau: fix assertion failure and < vs > error

Fixes Ticket4211

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>