Bug: ITBLL verify failed and found undefined nodes and nodes which lost big or tiny families.
Status: Open | Priority: Major | Resolution: 
Created: 26/Oct/23 07:00
Affects Version/s: 3.0.0-beta-1

Itbll verify failed, indicating possible data loss. The related log is as follows:
{code:java}
2023-10-26T07:41:31,306 ERROR [main] test.IntegrationTestBigLinkedList$Verify: Found an undefined node. Undefined count=3044
2023-10-26T07:41:31,306 ERROR [main] test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or tiny families, count=2
2023-10-26T07:42:30,927 ERROR [main] util.AbstractHBaseTool: Error running command-line tool
java.lang.RuntimeException: Verify.run failed with return code: 1
        at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.runVerify(IntegrationTestBigLinkedList.java:1548) ~[hbase-it-3.0.0-beta-1-SNAPSHOT-tests.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.run(IntegrationTestBigLinkedList.java:1583) ~[hbase-it-3.0.0-beta-1-SNAPSHOT-tests.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) ~[hadoop-common-3.2.4.jar:?]
        at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.runTestFromCommandLine(IntegrationTestBigLinkedList.java:1955) ~[hbase-it-3.0.0-beta-1-SNAPSHOT-tests.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:150) ~[hbase-it-3.0.0-beta-1-SNAPSHOT-tests.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:151) ~[hbase-common-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) ~[hadoop-common-3.2.4.jar:?]
        at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.main(IntegrationTestBigLinkedList.java:1998) ~[hbase-it-3.0.0-beta-1-SNAPSHOT-tests.jar:3.0.0-beta-1-SNAPSHOT] {code}
Found undefined nodes
{code:java}
2023-10-26 07:36:37,408 ERROR [main] org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Verify: Extras on ref without a def, ref=sJb\x8Au\xF7f\x9F\x86\xA4\x8C\x86U\xB6\x003, refPrevEqualsKey=true, key=\xF3\x0D\xF8u\xDD\xC3x\x80\x06\x82\x5C\x05.$\x1Fm, ref row date=Thu Oct 26 05:10:55 CST 2023, jobStr=Job: job_1698234646442_0014 Task: attempt_1698234646442_0014_m_000001_0, ref row count=26900714, ref row regionLocation=region=itbll,\xF2\xB0 \xC4\x9B\xA5\xE2\xE6,1698235151498.7fad6bc476873f19853d4988f6483a24., hostname=core-1-4.c-cf1f2d22f666ed69.cn-hangzhou.emr.aliyuncs.com,16020,1698276181696, seqNum=1554082, key row regionLocation=region=itbll,\xF2\xB0 \xC4\x9B\xA5\xE2\xE6,1698235151498.7fad6bc476873f19853d4988f6483a24., hostname=core-1-4.c-cf1f2d22f666ed69.cn-hangzhou.emr.aliyuncs.com,16020,1698276181696, seqNum=1554082
2023-10-26 07:36:37,408 ERROR [main] org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Verify: LinkedListError: key=\xF3\x0D\xF8u\xDD\xC3x\x80\x06\x82\x5C\x05.$\x1Fm, reference(s)=sJb\x8Au\xF7f\x9F\x86\xA4\x8C\x86U\xB6\x003 {code}
Found nodes which lost big or tiny families
{code:java}
2023-10-26 07:41:15,638 ERROR [main] org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Verify: LinkedListError: key=\xF3\x0E\x02\xC2\x14\xD9E-\x06\xAF\x9D\xEE\xBD\xB4*\x86, lost big or tiny families
2023-10-26 07:41:26,324 ERROR [main] org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Verify: LinkedListError: key=\xF3\x0D\xF8&\x13\xE3S\xA2\x06\x82\x5C"mH\xEBL, lost big or tiny families {code}