Only reallocate if needed