Restructure grow/shrink to avoid spurious size invalidation.