build: add checks for some compiler warning flags
vo_x11: check allocation errors

Avoids a crash if allocation fails.
vo_x11: don't attempt to resize when unconfigured

Fixes #1347.

The previous commit actually fixes the crash.