abuse: Turns out --with-x does do something...

It just doesnt break without until you reboot or retart X.