avdevice/x11grab: fix error handling in pixel formats

Reviewed-by: Benoit Fouet <benoit.fouet@free.fr>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>