Only remove the repository if it does not contain .git directory