Only set up known_hosts if pull_from_git is True