Remove a bunch of unnecessary allocations and copies