wayland/shm: fix memory leak
wayland/shm: RGB888 as default, change options

RGB565 is one of the fastest and most supported formats on low end consumer
devices, but ffmpeg spams warning when using it. Make it opt-in instead of
opt-out.