Cache diff deltas on the first enumeration

Enumeration is expensive. So only do it once.

/cc @dannygreg