Only set DISPLAY for remote host if it's not set