xl: K&R formatting cosmetics
xl: Fix the buffer size check

Also make it the first check.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org