xlib: Unlike the visual when destroying it

Otherwise we leave dangling pointers in the visual list, leading to
memory corruption when using low bitdepth servers.

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>