Remove extra copy from sortperm

Save compute cycles!  (Only really helps a few % at most, but... why not?)