Remove unneeded memoization

We’ll still only ever call the predicate once, and this behaviour is
fully spec’d.