Only mess with default-directory at init time