Use Set rather than Map

And thus only memoize mismatch points