Avoid excessive allocations and copies in iter::to_vec

The foldl based implementation allocates lots of unneeded vectors.
iter::map_to_vec is already optimized to avoid these.

One place that benefits quite a lot from this is the metadata decoder, helping with compile times for tiny programs.