Bug: Proper bulk load tracking
Status: Open | Priority: Critical | Resolution: 
Created: 28/Nov/24 13:10
Affects Version/s: 2.6.0

As part of the incremental backup mechanism, HBase tracks which files were bulk-loaded (since the last backup).

This data is stored in the backup:system_bulk table. Entries are added when a bulk load occurs through the BackupObserver co-processor. Entries are deleted when an incremental backup is completed.

There are 2 flaws in this implementation:

1) Performing a full backup should clear the list. Imagine following scenario:
 * Create a full backup B1 of table T.
 * Perform a bulk load L1.
 * Take a full backup B2 of table T.
 * Take an incremental backup of table T.
 ** The data stored for this backup will include L1, even though that data is already present due to B2. (This is an inefficiency, not a real error.)

2) Performing a table deletion should clear the list of bulk-loaded files. Imagine the following scenario:
 * Create a full backup of table T.
 * Perform a bulk-load B1 into T.
 * Disable, delete and recreate T.
 * Create an incremental backup (taking a full backup instead is similar to the previous case)

 ** The backup will contain B1, even though it doesn't belong there.

 

Note that this *can also cause backup corruption* after a backup restores (which is how we encountered this issue), which makes this problem less niche than the above scenarios indicate. Backup restore effectively uses bulk loads as well, so users could run into following scenario, where they are trying to restore data corruption:
 * (create an environment with backup B1 (time t), backup B2 (time t2 > t).
 * Users notice data corruption, and restore backup B2 after clearing the table
 * Users notice data corruption is already present, and restore backup B1 after clearing the table.
 * Users find data corruption solved, and resume regular backup cycle from here on.
 ** Any incremental backup taken will contain the (possible corrupt) data from B2 (due to the restore operation using bulk operations). The backups will be affected until a FULL backup is taken after an incremental backup (so this could span a period of weeks assuming bi-weekly/monthly full backups).

A minimal reproduction example:
{code:java}
echo "create 'table', 'cf'; put 'table', 'row1', 'cf:a', 'value1', 1400523142819" | bin/hbase shell -n
bin/hbase backup create full file:/tmp/backup -t table -i
echo "disable 'table'; drop 'table'" | bin/hbase shell -n
# Empty
echo "scan 'backup:system_bulk'" | bin/hbase shell -n

bin/hbase restore file:/tmp/backup backup_1732787972748 -t "table"
# 1 entry
echo "scan 'backup:system_bulk'" | bin/hbase shell -n

echo "disable 'table'; drop 'table'" | bin/hbase shell -n
# 1 entry
echo "scan 'backup:system_bulk'" | bin/hbase shell -necho "create 'table', 'cf'; put 'table', 'row1', 'cf:b', 'value2', 1400523142819" | bin/hbase shell -n
bin/hbase backup create full file:/tmp/backup -t table -i
echo "scan 'backup:system_bulk'" | bin/hbase shell -n

echo "put 'table', 'row1', 'cf:b', 'value3', 1400523142819" | bin/hbase shell -n
bin/hbase backup create incremental file:/tmp/backup -t table -i
# Emtpy
echo "scan 'backup:system_bulk'" | bin/hbase shell -n

echo "disable 'table'; drop 'table'" | bin/hbase shell -n
bin/hbase restore file:/tmp/backup backup_1732788098586 -t "table"

# Will contain "value1" (unexpected) and "value3" (expected)
echo "scan 'table'" | bin/hbase shell -n
 {code}