Don't cache the sizes, just re-calculate on removal.