Store memo state in MatcherContext
Use Set rather than Map

And thus only memoize mismatch points