Remove most of the resize handling again, as it is not feasible to profile this for most data structures :(