Only call realloc if we nede more memory

I'm sure it's a no-op if we don't, but stil...
These can't point to the same thing.