av9ash/GitBugs huggingface dataset:
Bug: HdfsFileStatus incompatibility when used with Hadoop 3.1.3 and Hbase version 2.2.4
Status: Resolved | Priority: Blocker | Resolution: Not A Bug
Created: 09/Apr/20 13:23
Affects Version/s: 2.2.4

Hi, 

Even after compiling from source  I am getting below error message. Hbasse master start without any issue, but region server fails with below message

handler.AssignRegionHandler: Fatal error occurred while opening region hbase:meta,,1.1588230740, aborting...
 java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected
 at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:496)
 at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:116)
 at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:576)
 at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:571)
 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
 at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:584)
 at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51)
 at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
 at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
 at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
 at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:643)
 at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:126)
 at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:767)
 at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:501)
 at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:442)
 at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:156)
 at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:61)
 at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:284)
 at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2181)
 at org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:133)
 at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Bug: [JDK11] shell fails to launch
Status: Resolved | Priority: Blocker | Resolution: Cannot Reproduce
Created: 21/Apr/20 23:02
Affects Version/s: 3.0.0-alpha-1

{noformat}
$ JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home ./bin/hbase shell
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.hbase.util.UnsafeAvailChecker (file:/Users/ndimiduk/repos/apache/hbase/hbase-common/target/hbase-common-3.0.0-SNAPSHOT.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.hbase.util.UnsafeAvailChecker
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/book.html#shell
Version 3.0.0-SNAPSHOT, rbcacc4ce939e60fd69891df6315a39aef852b567, Tue Apr 21 15:47:27 PDT 2020
Took 0.0000 seconds                                                                                                                                                                                         
ArgumentError: wrong number of arguments (1 for 0)
           ` at uri:classloader:/jruby/kernel/jruby/process_manager.rb:32
           ` at uri:classloader:/jruby/kernel/jruby/process_manager.rb:54
  initialize at /Users/ndimiduk/repos/apache/hbase/hbase-shell/src/main/ruby/irb/hirb.rb:46
       start at /Users/ndimiduk/repos/apache/hbase/bin/../bin/hirb.rb:207
      <main> at /Users/ndimiduk/repos/apache/hbase/bin/../bin/hirb.rb:219
{noformat}
Bug: async client deadlock
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 05/Jun/20 00:34
Affects Version/s: 2.2.4

I ran into an issue where one instance in a cluster of application servers backed by HBase stopped processing requests. Looking at a thread dump, it seems HBase client threads are deadlocked:

https://pastebin.com/raw/B3FJL1Ag

The deadlock seemed to happen at the same time that a region server was abruptly stopped (the physical server was restarted unexpectedly).

I'm using the hbase async API. The hbase client version is 2.2.4. The server is running 2.2.4 as well.
Bug: TestDelegationTokenWithEncryption always fails
Status: Resolved | Priority: Blocker | Resolution: Duplicate
Created: 23/Apr/20 01:58
Affects Version/s: 3.0.0-alpha-1

TestDelegationTokenWithEncryption and TestGenerateDelegationToken _always_ fail.

 

Incidentally, they don't fail in branch-2.3 and branch-2.2.

 

I suspect there's a regression with delegation token code, because if I comment out the following code in the test, they pass:

 
{code:java}
try (Connection conn = ConnectionFactory.createConnection(TEST_UTIL.getConfiguration())) {
  Token<? extends TokenIdentifier> token = TokenUtil.obtainToken(conn);
  UserGroupInformation.getCurrentUser().addToken(token);
}
{code}
Effectively, use Kerberos to login instead of delegation token.

The tests fail all the time (100%) in the last 29 runs: 
 [https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/master/lastSuccessfulBuild/artifact/dashboard.html]

Initially I thought this was caused by pluggable authentication (HBASE-23347), but the tests don't fail in branch-2.3 so looks unlikely.
Bug: Balancer should avoid leaving idle region servers
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Apr/20 12:56

After HBASE-15529 the StochasticLoadBalancer makes the decision to run based on its internal cost functions rather than the simple region count skew of BaseLoadBalancer.

Given the default weights for those cost functions, the default minimum cost to indicate a need to rebalance, and a regions per region server density of ~90 we are not very responsive to adding additional region servers for non-trivial cluster sizes:

* For clusters ~10 nodes, the defaults think a single RS at 0 regions means we need to balance
* For clusters >20 nodes, the defaults will not consider a single RS at 0 regions to mean we need to balance. 2 RS at 0 will cause it to balance.
* For clusters ~100 nodes, having 6 RS with no regions will still not meet the threshold to cause a balance.

Note that this is the decision to look at balancer plans at all. The calculation is severely dominated by the region count skew (it has weight 500 and all other weights are ~105), so barring a very significant change in all other cost functions this condition will persist indefinitely.

Two possible approaches:

* add a new cost function that's essentially "don't have RS with 0 regions" that an operator can tune
* add a short circuit condition for the {{needsBalance}} method that checks for empty RS similar to the check we do for colocated region replicas

For those currently hitting this an easy work around is to set {{hbase.master.balancer.stochastic.minCostNeedBalance}} to {{0.01}}. This will mean that a single RS having 0 regions will cause the balancer to run for clusters of up to ~90 region servers. It's essentially the same as the default slop of 0.01 used by the BaseLoadBalancer.
Bug: A "stuck" master should not idle as active without taking action
Status: Open | Priority: Critical | Resolution: 
Created: 30/Apr/20 18:02
Affects Version/s: 2.3.0

The master schedules a SCP for the region server hosting meta. However, due to a misconfiguration, the cluster cannot make progress. After fixing the configuration issue and restarting, the cluster still cannot make progress. After the configured period (15 minuets), the master enters a "holding pattern" where it retains Active master status, but isn't taking any action.

This "brown-out" state is toxic. It should either keep trying to make progress, or it should abort. Staying up and not doing anything is the wrong thing to do.
Bug: MergeNormalizer is merging non-adjacent regions and causing region overlaps/holes.
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 15/May/20 00:48
Affects Version/s: 2.3.0

Currently, we found normalizer was merging regions which are non-adjacent, it will cause inconsistencies in the cluster.
{code:java}
439055 2020-05-08 17:47:09,814 INFO org.apache.hadoop.hbase.master.normalizer.MergeNormalizationPlan: Executing merging normalization plan: MergeNormalizationPlan{firstRegion={ENCODED => 47fe236a5e3649ded95cb64ad0c08492, NAME => 'TABLE,\x03\x01\x05\x01\x04\x02,1554838974870.47fe236a5e3649ded95cb64ad       0c08492.', STARTKEY => '\x03\x01\x05\x01\x04\x02', ENDKEY => '\x03\x01\x05\x01\x04\x02\x01\x02\x02201904082200\x00\x00\x03Mac\x00\x00\x00\x00\x00\x00\x00\x00\x00iMac13,1\x00\x00\x00\x00\x00\x049.3-14E260\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x05'}, secondRegion={ENCODED => 0c0f2aa67f4329d5c4       8ba0320f173d31, NAME => 'TABLE,\x03\x01\x05\x02\x01\x01,1554830735526.0c0f2aa67f4329d5c48ba0320f173d31.', STARTKEY => '\x03\x01\x05\x02\x01\x01', ENDKEY => '\x03\x01\x05\x02\x01\x02'}}
439056 2020-05-08 17:47:11,438 INFO org.apache.hadoop.hbase.ScheduledChore: CatalogJanitor-*****:16000 average execution time: 1676219193 ns.
439057 2020-05-08 17:47:11,730 INFO org.apache.hadoop.hbase.master.HMaster: Client=null/null merge regions [47fe236a5e3649ded95cb64ad0c08492], [0c0f2aa67f4329d5c48ba0320f173d31]
 {code}
 

The root cause is that getMergeNormalizationPlan() uses a list of regionInfo which is ordered by regionName. regionName does not necessary guarantee the order of STARTKEY (let's say 'aa1', 'aa1!', in order of regionName, it will be 'aa1!' followed by 'aa1'. This will result in normalizer merging non-adjacent regions into one and creates overlaps. This is not an issue in branch-1 as the list is already ordered by RegionInfo.COMPARATOR in normalizer.
Bug: Shutdown AssignmentManager before ProcedureExecutor may cause SCP to accidentally skip assigning a region
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 05/Apr/20 00:31

I saw this on TestCloseRegionWithRSCrash. The Region 788a516d1f86af98e0a16bcc1afe4fa1 was being moved to RS  example.com,62652,1586032098445 just after it was killed. The Move Close fails because the RS has no node in the Master. The Move then tries to 'confirm' the close but it fails because no remote RS. We are then to wait in this state until operator or some other procedure intervenes to 'fix' the state. Normally a ServerCrashProcedure would do the job but in this test the Master is restarted after the RS is killed, a condition we do not accommodate.

Let me attach the test log.
Bug: AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 24/Jun/20 13:41
Affects Version/s: 2.0.0

By HBASE-14004, we introduce {{WALFileLengthProvider}} interface to keep the current writing wal file length  by ourselves,  {{WALEntryStream}} used by {{ReplicationSourceWALReader}} could only read WAL file byte size <= {{WALFileLengthProvider.getLogFileSizeIfBeingWritten}} if the WAL file is current been writing on the same RegionServer .

{{AsyncFSWAL}} implements {{WALFileLengthProvider}}  by {{AbstractFSWAL.getLogFileSizeIfBeingWritten}}, just as folllows :
{code:java}
   public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
    rollWriterLock.lock();
    try {
      Path currentPath = getOldPath();
      if (path.equals(currentPath)) {
        W writer = this.writer;
        return writer != null ? OptionalLong.of(writer.getLength()) : OptionalLong.empty();
      } else {
        return OptionalLong.empty();
      }
    } finally {
      rollWriterLock.unlock();
    }
  }
{code}

For {{AsyncFSWAL}},  above {{AsyncFSWAL.writer}}  is {{AsyncProtobufLogWriter}} ，and {{AsyncProtobufLogWriter.getLength}}  is as follows:

{code:java}
    public long getLength() {
        return length.get();
    }
{code}

But for {{AsyncProtobufLogWriter}}, any append method may increase the above {{AsyncProtobufLogWriter.length}}, especially for following {{AsyncFSWAL.append}}
method just appending the {{WALEntry}} to {{FanOutOneBlockAsyncDFSOutput.buf}}:
{code:java}
     public void append(Entry entry) {
          int buffered = output.buffered();
          try {
              entry.getKey().
              getBuilder(compressor).setFollowingKvCount(entry.getEdit().size()).build()
              .writeDelimitedTo(asyncOutputWrapper);
          } catch (IOException e) {
             throw new AssertionError("should not happen", e);
          }
   
        try {
           for (Cell cell : entry.getEdit().getCells()) {
             cellEncoder.write(cell);
           }
          } catch (IOException e) {
           throw new AssertionError("should not happen", e);
         }
         length.addAndGet(output.buffered() - buffered);
     }
{code}

That is to say, {{AsyncFSWAL.getLogFileSizeIfBeingWritten}} could not reflect the  file length which successfully synced to underlying HDFS, which is not as expected.
Bug: HBCK's "Orphan Regions on FileSystem" reports regions with referenced HFiles
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 27/Apr/20 20:11
Affects Version/s: 2.2.4

This issue came up after merging regions. MergeTableRegionsProcedure removes the parent regions from hbase:meta and creates HFile references in child region to the old parent regions. Running `hbck_chore_run` right after the `merge_region` will show the parent regions in "Orphan Regions on FileSystem" until major compaction is run on child region which will remove HFile references and cause Catalog Janitor to clean up the parent regions.

There are probably other situations which can cause the same issue (maybe region split?)

Having "Orphan Regions on FileSystem" list parent regions and suggest to "_hbase completebulkload_" is dangerous in this case as completing bulk load will lead to stale HFile references in child region which will cause its OPEN to fail because referenced HFile doesn't exist.

Figuring out these things for database administrators is tedious, so I think it would be reasonable to not consider regions with referenced HFiles to be orphans (or maybe could give an extra hint saying that it has referenced HFiles).
Bug: Zookeeper sync() call is async
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 21/Jun/20 03:46
Affects Version/s: 1.7.0

Here is the method that does a sync() of lagging followers with leader in the quorum. We rely on this to see a consistent snapshot of ZK data from multiple clients. However the problem is that the underlying sync() call is actually asynchronous since we are passing a 'null' call back.  See the ZK API [doc|https://zookeeper.apache.org/doc/r3.5.7/apidocs/zookeeper-server/index.html] for details. The end-result is that sync() doesn't guarantee that it has happened by the time it returns.

{noformat}
  /**
   * Forces a synchronization of this ZooKeeper client connection.
   * <p>
   * Executing this method before running other methods will ensure that the
   * subsequent operations are up-to-date and consistent as of the time that
   * the sync is complete.
   * <p>
   * This is used for compareAndSwap type operations where we need to read the
   * data of an existing node and delete or transition that node, utilizing the
   * previously read version and data.  We want to ensure that the version read
   * is up-to-date from when we begin the operation.
   */
  public void sync(String path) throws KeeperException {
    this.recoverableZooKeeper.sync(path, null, null);
  }
{noformat}

We rely on this heavily (at least in the older branches that do ZK based region assignment). In branch-1 we saw weird "BadVersionException" exceptions in RITs because of the inconsistent view of the ZK snapshot. It could manifest differently in other branches. Either way, this is something we need to fix.
Bug:  WrongRowIOException occasionally thrown during batch append 
Status: Open | Priority: Critical | Resolution: 
Created: 20/Jun/20 10:16
Affects Version/s: 2.2.4

We recently used the HTable.batch() to do batch Append, and RS occasionally throws a large number of errors as follows:
{code:java}
The row in 970~xxx/f:\x00\x00\x00\x17/LATEST_TIMESTAMP/Put/vlen=24/seqid=0/3160:50472:;3873:52121:; doesn't match the original one 841~aaa
 at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:788)
 at org.apache.hadoop.hbase.client.Append.add(Append.java:183)
 at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toDelta(ProtobufUtil.java:815)
 at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toAppend(ProtobufUtil.java:864)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.append(RSRpcServices.java:676)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:881)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2755)
 at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42290)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code}
Finally, the RS write handler and write queue are full, and the RS is unavailable. Looks the same as [HBASE-19876|https://issues.apache.org/jira/browse/HBASE-19876].
Bug: AssignmentManager.start should add meta region to ServerStateNode
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Jun/20 09:53

In AssignmentManager.start, we will load the meta region state and location from zk and create the RegionStateNode, but we forget to call regionStates.addRegionToServer to add the region to the region server.

Found this when implementing HBASE-24390. As in HBASE-24390, we will remove RegionInfoBuilder.FIRST_META_REGIONINFO so in SCP, we need to use the getRegionsOnServer instead of RegionInfoBuilder.FIRST_META_REGIONINFO when assigning meta, so the bug becomes a real problem.

Though it is not a big problem for SCP for current 2.x and master branches, it is a high risky bug. For example, in AssignmentManager.submitServerCrash, now we use the RegionStateNode of meta regions to determine whether the given region server carries meta regions. But it is also valid to test through the ServerStateNode's region list. If later we change this method to use ServerStateNode, it will cause very serious data loss bug.
Bug: Failed to write replication wal position
Status: Resolved | Priority: Critical | Resolution: Information Provided
Created: 23/Apr/20 08:44
Affects Version/s: 1.5.0

I turned on replication for tables, index tables did not, wrote data using phoneix, and hbase regionserver went down after running for a while.

regionserver log:

2020-04-20 19:22:49,531 FATAL [main-EventThread.replicationSource,1.replicationSource.a2-test-regionserver-52-23.sh%2C16020%2C1587378758523,1] regionserver.HRegionServe 
 r: ABORTING region server a2- test-regionserver-52-23.sh,16020,1587378758523: Failed to write replication wal position (filename=a2-test-regionserver-52-23.sh%2C16020%2C 
 1587378758523.1587381739689, position=3388929) 
 org.apache.zookeeper. KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/a2-test-regionserver-52-23.sh,16020,1587378758523/1/a2-test-reg 
 ionserver-52-23.sh%2C16020%2C1587378758523. 1587381739689 
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) 
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1327) 
 at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:510) 
 at org.apache.hadoop.hbase.zookeeper.ZKUtil .setData(ZKUtil.java:867) 
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:917) 
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:911 ) 
 at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.setLogPosition(ReplicationQueuesZKImpl.java:155) 
 at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:197)
 at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.updateLogPosition(ReplicationSource.java:791) 
 at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.shipEdits(ReplicationSource.java:762) 
 at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.run(ReplicationSource.java:601)
Bug: Hedged read might hang infinitely if read data from all DN failed
Status: Open | Priority: Critical | Resolution: 
Created: 29/May/20 08:35

Found out that after an ungraceful Datanode shutdown, the number of HBase active handlers started to grow, making RegionServer stuck and not able to serve any RPC.

Took the thread dump and found out multiple read handlers were in some kind of dead lock state and also write handlers stuck.

This also caused to not be able to flush the memstore because it was waiting for this lock: [https://github.com/apache/hbase/blob/136414dd72a80f379b80cd6f74b5b6ebd78f33ec/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L1225]

Without being able to flush it, I could not gracefully stop the RegionServer, since we can't move out the flushing region.

 

Found out that the real issue was in Hadoop's DFSInputStream. When no hedged reads succeed, the internal hedgedService.take() call hangs forever since it's internally using a BlockingQueue: [https://github.com/apache/hadoop/blob/rel/release-2.8.5/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1435]

 

Reproduced in HBase 1.4.13, but I think it affects to other versions: 
 # Datanode dies
 # A read handler holding a read lock of an HStore is blocked by hedged read that does not succeed.
 # Other read handlers try to acquire the lock and stuck.
 # Memstore flusher tries to acquire write lock in HStore and also blocked because of the other read locks.
 # Others like CompactedHFilesDischarger also blocks because memstore holds the lock.
 # Tried to use graceful_stop.sh, but region_mover.rb fails because can't move out the region being flushed.
 # Forcefully killed the RegionServer because no other option (I am not sure if there is possibility of data loss, since HStore#updateStorefiles is not finished at this point).

 

This is the Hadoop side issue: https://issues.apache.org/jira/browse/HDFS-11303 and it's fixed for 2.9.0.

This is not directly related with HBase code, but just wanted community to be aware that with current used Hadoop version (2.8.5), this issue could happen.

 

I would like to suggest to upgrade the used Hadoop version to 2.9.0.