You already said that, but creating an array for every single element smells slow. Can't you pull out the copyswap/casting from iterator_loop and then call the inner_loop directly working on raw pointers? Maybe creating just one set of arrays before the loop?