Use ~/.Xauthority only if not empty