Add a visited set so we aren't duplicating work