wiredtiger/wiredtiger — Issue #172: verify has a hard limit of 16TB files
Author: keithbostic
Created: 2012-02-25 16:32:52+00:00
State: open

The bit package we use to verify files takes a 32-bit count of bits to allocate, so if the allocation block size is 512B, we can only verify a 16TB file:

2^32 * 512 * 8 = 16 * 2^40

We need to make this better by continuing with file verification in this case, rather than failing (the test this failure would turn off is the check for blocks in file being referenced exactly once), regardless, there's no reason to quit on verification entirely.

We also may need to fix this, either by switching to a 64-bit count of bits to allocate, or by switching to an entirely different approach: the bit array approach requires significant amounts of heap memory, verifying a 1TB file with a 512B allocation size currently requires 512MB of heap memory.