tests: add encoder for h264 format

add encoder for h264 format.

vaapi makefile.am lack of libva-x11 , so add it to makefile.am.