Cache patterns a bit higher up to make things a little more efficient.  See #4469.