hpcc-systems/HPCC-Platform — pull request #4346 by AttilaVamos (2013-05-03 08:24:04+00:00)
Title: HPCC-9157 - Dfu server partition calculation not efficient when more tha...

...n 2M file sprayed

This improvement impact the partition calculation speed if any prefix defined
in dfuplus spray command.

The original version inserted the prefix headers into the existing partition
data. With large amount of data it takes long time (322 sec for 1M,
1445 sec for 2M and 9739 sec for 4M files).

The new approach merges original partition items with prefix headers into a
new array. At the end it copies the merged items into the original partition
array.

Comparison between old and new partition calculation time consumption:

Number
of files Old solution New
32k < 1s < 1s
64k 2s < 1s
128k 5s 1s
256k 22s 2s
512k 82s 7s
1M 322s 14s
2M 1445s 17s
4M 9739s 32s

Signed-off-by: Attila Vamos