Memoize the correct thing