tests: work around frame size issues

the reported frame size from yami is the VASurface/VAImage resolution,
which may be different from real frame size, for example:
 - vp8 dynamic change resolution
 - VASurface/VAImage size does additional alignment, for example 1080->1088
tests: fix mistake on renderMode check