Only set {back,fore}ground for xterm