Bug: Data loss can happen if a cloned table loses original split region(delete table)
Status: Closed | Priority: Major | Resolution: Fixed
Created: 20/Oct/20 13:18

Steps to reproduce are as follows:

1. Create a table and put some data into the table:
{code:java}
create 'test1','cf'
put 'test1','r1','cf','v1'
put 'test1','r2','cf','v2'
put 'test1','r3','cf','v3'
put 'test1','r4','cf','v4'
put 'test1','r5','cf','v5'
{code}
2. Take a snapshot for the table:
{code:java}
snapshot 'test1','snap_test'
{code}
3. Clone the snapshot to another table
{code:java}
clone_snapshot 'snap_test','test2'
{code}
4. Split the original table
{code:java}
split 'test1','r3'
{code}
5. Drop the original table
{code:java}
disable 'test1'
drop 'test1'
{code}
After that, we see the error like the following in RS log when opening the regions of the cloned table:
{code:java}
2020-10-20 13:32:18,415 WARN org.apache.hadoop.hbase.regionserver.HRegion: Failed initialize of region= test2,,1603200595702.bebdc4f740626206eeccad96b7643261., starting to roll back memstore
java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1095)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:943)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:899)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7246)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7204)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7176)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7134)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7085)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:283)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
        at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:590)
        at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:557)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:303)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5731)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1059)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1056)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more
Caused by: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
        at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:322)
        at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.<init>(FileLink.java:126)
        at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.<init>(FileLink.java:117)
        at org.apache.hadoop.hbase.io.FileLink.open(FileLink.java:429)
        at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:125)
        at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:107)
        at org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:261)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:368)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:476)
        at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:703)
        at org.apache.hadoop.hbase.regionserver.HStore.lambda$openStoreFiles$1(HStore.java:573)
        ... 6 more
{code}