Bug: MOB compaction with missing files catches incorrect exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/21 15:38
Affects Version/s: 3.0.0-alpha-1

The MOB compaction catches FileNotFoundException when {{hbase.unsafe.mob.discard.miss}} is true to handle missing MOB cells. The FNFE is wrapped in DoNotRetryIOException so the compaction fails for the given region.

{noformat}
2021-07-21 13:51:05,880 WARN org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only when first upgrading to a version with the distributed mob compaction feature on a cluster that has experienced MOB data corruption.
2021-07-21 13:51:05,880 WARN org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only when first upgrading to a version with the distributed mob compaction feature on a cluster that has experienced MOB data corruption.
2021-07-21 13:51:05,880 INFO org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true optimized configured=false optimized enabled=false maximum MOB file size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB family=test_cf region=3a2ee81f9244c39ba61d694e616c1a89]
2021-07-21 13:51:05,880 INFO org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true optimized configured=false optimized enabled=false maximum MOB file size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB family=test_cf region=7a96f55bb9ae04500a06cbaef02da6a3]
2021-07-21 13:51:05,888 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
2021-07-21 13:51:05,891 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,88888888,1626787996628.8fd002bda07755decda67b7084d1e0f6.
2021-07-21 13:51:05,895 ERROR org.apache.hadoop.hbase.regionserver.HMobStore: The mob file 1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3 could not be found in the locations [hdfs://example.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce5
3c2/test_cf, hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf] or it is corrupt
2021-07-21 13:51:05,895 INFO org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController: 7a96f55bb9ae04500a06cbaef02da6a3#test_cf#compaction#1 average throughput is 0.07 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 10.00 MB/second
2021-07-21 13:51:05,908 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,dddddddd,1626787996628.53186ca5008e3a964eee5f96ee3f1b26.
2021-07-21 13:51:05,997 ERROR org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed Request=regionName=IntegrationTestIngestWithMOB,11111111,1626787996628.7a96f55bb9ae04500a06cbaef02da6a3., storeName=test_cf, fileCount=1, fileSize=110.6 M (110.6 M), priority=1, time=1626875465819
java.io.IOException: Mob compaction failed for region: 7a96f55bb9ae04500a06cbaef02da6a3
        at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:575)
        at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
        at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.compact(DefaultMobStoreCompactor.java:227)
        at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
        at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407)
        at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2183)
        at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:633)
        at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:675)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.io.FileNotFoundException: File does not exist: hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
        at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:438)
        at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:328)
        at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:307)
        at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:366)
        ... 10 more
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1500)
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1493)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1508)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
        at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:140)
        at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:104)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:220)
        at org.apache.hadoop.hbase.mob.CachedMobFile.create(CachedMobFile.java:49)
        at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:220)
        at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:399)
        ... 13 more
2021-07-21 13:51:06,013 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction of test_cf in IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
{noformat}