Thomas-Tsai/partclone — pull request #41 by pfrouleau (2013-09-15 00:41:09+00:00)
Title: Image format 0002

This serie implements the new image format 0002.

The new format allow the user to disable the addition of checksums in the image, which speed up the image creation. The user can choose how many sectors are covered by each checksum. By default, one checksum is added for every 1 MB of data, which is the default memory buffer size. The previous image format (0001) was adding 1 checksum per sector. The checksum is now reseeded after each write; this give better results with deduplication tools (like Windows 2012 and bup).

The serie is composed of 3 phases:
* generic refactoring to simplify the addition of format 0002 and make phase 2 easier to follow;
* incremental implementation of the format 0002;
* activation of the new format.

My benchmarks gived these results, based on this partition:

File system: EXTFS
Device size: 2.1 GB = 524288 Blocks
Space in use: 1.8 GB = 437558 Blocks
Free Space: 355.2 MB = 86730 Blocks
Block size: 4096 Byte

Image options | Image size | real | user | sys
0001 | 1794516256 | 52.69 | 6.61 | 10.43
0002 (a1+k1) | 1794053450 | 55.05 | 10.86 | 9.01
0002 (a1+k0) | 1792310058 | 51.36 | 6.36 | 10.21
0002 (a0+k0) | 1792303218 | 49.16 | 1.10 | 9.53

The image 0002 (a1+k1) takes a little bit more time to run than 0001 because the CRC32 is computed with every bytes instead of using only the first byte of a block. The image 0002 (a1+k0), which are the default values, takes about the same time than 0001, but it is 2MB smaller because the bitmap is written in binary and contains less checksums (256 blocks per checksum given the 4KB blocks).

When the images are saved on a Windows Server 2012 with deduplication enabled, we get around 47% deduplication with images in 0001 format, while the deduplication reach 73% with images in 0002 format.