av9ash/GitBugs huggingface dataset:
Bug: Recommend upping zk jute.maxbuffer in all but minor installs
Status: Open | Priority: Major | Resolution: 
Created: 12/Jun/20 04:24

Add a doc note in upgrade and in zookeeper section recommending upping zk jute.maxbuffer to be above the default of 1M.

Here is jute.maxbuffer from zk doc.

{code}
jute.maxbuffer:
(Java system property: jute.maxbuffer)
This option can only be set as a Java system property. There is no zookeeper prefix on it. It specifies the maximum size of the data that can be stored in a znode. The default is 0xfffff, or just under 1M. If this option is changed, the system property must be set on all servers and clients otherwise problems will arise. This is really a sanity check. ZooKeeper is designed to store data on the order of kilobytes in size.
{code}

It seems easy enough blowing the 1MB default. Here is one such scenario. A peer is disabled so WALs backup on each RegionServer or a bug makes it so we don't clear WALs out from under the RegionServer promptly. Backed-up WALs get into the hundreds... easy enough on a busy cluster. Next, there is a power outage and the cluster crashes down.

Recovery may require an SCP recovering hundreds of WALs. As is, the way our SCP works, we can end up with a /hbase/splitWAL dir with hundreds -- even thousands -- of WALs in it. The 1MB buffer limit in zk can't carry listings this big.

Of note, the jute.maxbuffer needs to be set on the zk servers -- with restart so the change is noticed -- and on the client-side, in the hbase master at least.

This issue is about highlighting this old issue in our doc. It seems to be absent totally.
Bug: Change HBASE_TARGET_TAG to a human readable tag
Status: Open | Priority: Major | Resolution: 
Created: 10/Jun/20 12:59

HBASE-24400 ([https://github.com/apache/hbase-native-client/pull/2)] introduced HBASE_TARGET_TAG which is a target HBase repo tag. A hash was used originally then changed to the master branch. We should change this to a tagged version when the target functionality is tagged upstream.
Bug: Ability to configure timeout between RPC retry to RS from master
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/20 17:27

This is useful for the cases where there is an injected environment edge and when the first RS RPC request fails causing it to go to the retry block.

In the absence of this , the default timeout would be set to 100 ms and the DelayedUtil class is meant to execute the retry after 100 ms. However as per the getRemainingTime() logic here (https://github.com/apache/hbase/blob/5b01e613fbbb92e243e99a1d199b4ffbb21ed2d9/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/util/DelayedUtil.java#L94), the equation would be evaluated to
 
EnvironmentEdgeManager.currentTime() >= EnvironmentEdgeManager.currentTime() + 100 which would never get evaluated to true in case of injected edge and retry never happens. Hence this config helps you to override it to 0 in cases we want to test with a manual injected environment edge so that the retry would succeed

 

An example would be Master trying to open the meta region even before the RS is online, In this case the retry of opening the meta region would never happen in the case of injected environment edge and master would never finish the initialization
Bug: Reenable the TestExportSnapshot family of tests
Status: Open | Priority: Major | Resolution: 
Created: 08/Jun/20 17:28
Affects Version/s: 2.3.0

The yarn history server can take a while to start. It is hardcoded at one minute before all is killed. On loaded servers can take this long to start. Cannot disable history server in tests nor can we configure the mini yarn cluster so it can take longer starting history server. After this addressed, and yarn history server is given more time or disabled, reenable the TestExportSnapshot* family of tests. See parent issue for details.
Bug: Remove RS entry from zk draining servers node after RS been stopped
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/20 19:24

When a RS is been decommissioned, we will add an entry into the zk node. This will be there unless the same RS instance is recommissioned. 
But if we want to scale down a cluster, the best path would be to decommission the RSs in the scaling down nodes.  The regions in these RSs will get moved to live RSs. In this case these decommissioned RSs will get stopped later. These will never get recommissioned.  The zk nodes will still be there under draining servers path.
We can remove this zk node when the RS is getting stopped.
Bug: Immutable Scan as unmodifiable subclass or wrapper of Scan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/20 12:21

We should provide an ImmutableScan as subclass or wrapper of Scan which can be used specifically by Coprocessors as read only Scan. This suggestion came up while reviewing HBASE-24321 by multiple reviewers: [https://github.com/apache/hbase/pull/1655]
Bug: Update docs re: ZooKeeper compatibility of 2.3.x release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/20 22:37

HBASE-24132 bumps the ZooKeeper version, which itself has some known upgrade steps. We have a release note, but we can call this out in our [2.3 upgrading section|https://hbase.apache.org/book.html#upgrade2.3] of the book.
Bug: Test coverage for HBASE-23202 - ExportSnapshot (import) will fail if copying files to root directory takes longer than cleaner TTL
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 26/May/20 20:47
Affects Version/s: 1.4.11

This is a followup Jira to add test coverage for HBASE-23202, it is broken since there is no test coverage. Will try to add an unittest for this case to avoid future broke.
Bug: [Flakey Tests] TestExportSnapshot takes too long; up against 13min max
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Apr/20 03:53

TestExportSnapshot fails fairly regularly locally. Looking, its test timeout. Looking at how long it ran, its 13minutes plus. Looking at recent successful branch-2 run, it passed but it took about 7-8minutes. Let me break up the test into to pieces.

 org.junit.runners.model.TestTimedOutException: test timed out after 780 seconds
   at org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:227)
   at org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportRetry(TestExportSnapshot.java:267)

... I see this in the log:

 ====> TEST TIMED OUT. PRINTING THREAD DUMP. <====


Test started at:

 2020-04-06 17:19:21,739 INFO

... and the timestamp just above the TIMED OUT was

 2020-04-06 17:31:01,758
Bug: Enhanced version of KeyPrefixRegionSplitPolicy
Status: Patch Available | Priority: Major | Resolution: 
Created: 29/May/20 17:44

With KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy, splitting regions policy is limited to either fixed length or based on delimiter. With Yarn Application Timeline Server V2, as discussed in YARN-10077, it will be nice to have enhanced version of DelimitedKeyPrefixRegionSplitPolicy that will give more flexibility to users to define the ordinance of delimiter to consider.
Bug: Scanners for Replicas region return many HFileLink name exceptions
Status: Open | Priority: Major | Resolution: 
Created: 01/Jun/20 09:57
Affects Version/s: 2.2.4

When region read replicas is enabled and scan is running, regionsevers logged so many warns as follows；It appears replicas region StoreFileInfo took normal hfile as filelink , however function getActiveFileName would throw exception when checking filelink format.

2020-05-31 10:40:48,486 WARN [RpcServer.default.RWQ.Fifo.scan.handler=238,queue=238,port=16020] regionserver.StoreScanner: failed to switch to stream read 
java.lang.IllegalArgumentException: 153052181a084e70a1242870fe6e4961 is not a valid HFileLink name! 
 at org.apache.hadoop.hbase.io.HFileLink.getReferencedHFileName(HFileLink.java:252) 
 at org.apache.hadoop.hbase.regionserver.StoreFileInfo.getActiveFileName(StoreFileInfo.java:624) 
 at org.apache.hadoop.hbase.regionserver.HStore.recreateScanners(HStore.java:2104) 
 at org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:1068) 
 at org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1202) 
 at org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:437) 
 at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:7037) 
 at org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:396) 
 at org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:289) 
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:167) 
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) 
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
Bug: BucketCache disabled instantly before error duration toleration is reached due to timing issue
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/20 10:30
Affects Version/s: 1.4.10

We saw an instance where BucketCache was disabled after only two IO error were encountered at nearly the same time. The following shows all errors from a region server log for the 2020-05-26 17:00 hour. Notice that there are no other errors until the 17:14:50 at which time the BucketCache is disabled because it claims duration time has exceeded 60000 ms:

$ grep ERROR hbase-hbase-regionserver-ip-172-20-113-147.log.2020-05-26-17
 2020-05-26 17:14:50,396 ERROR [hfile-prefetch-1589117924884] bucket.BucketCache: Failed reading block 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
 2020-05-26 17:14:50,397 ERROR [regionserver/ip-172-20-113-147.us-west-2.compute.internal/172.20.113.147:16020-BucketCacheWriter-0] bucket.BucketCache: Failed syncing IO engine
 2020-05-26 17:14:50,400 ERROR [hfile-prefetch-1589117924884] bucket.BucketCache: Failed reading block 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
 2020-05-26 17:14:50,401 ERROR [hfile-prefetch-1589117924884] bucket.BucketCache: IO errors duration time has exceeded 60000ms, disabling cache, please check your IOEngine

The region server is very busy so it should be constantly getting successful reads and writes in the bucket cache (it is not as though there was some long ago error and then no successful IO to clear the error flag).

We are running a busy EMR cluster backed by S3. A bucketcache getting disabled is a huge performance issue because all reads must go to S3.

Looking at the code, I believe I've found a timing issue. Here is the code for checking and setting the ioErrorStartTime (taken from BucketCache.java):

 
{code:java}
  /**
   * Check whether we tolerate IO error this time. If the duration of IOEngine
   * throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable the
   * cache
   */
  private void checkIOErrorIsTolerated() {
    long now = EnvironmentEdgeManager.currentTime();
    if (this.ioErrorStartTime > 0) {
      if (cacheEnabled && (now - ioErrorStartTime) > this.ioErrorsTolerationDuration) {
        LOG.error("IO errors duration time has exceeded " + ioErrorsTolerationDuration +
          "ms, disabling cache, please check your IOEngine");
        disableCache();
      }
    } else {
      this.ioErrorStartTime = now;
    }
  }
{code}
 
And here is the code for clearing the ioErrorStartTime when a successful read or write is done:

{code:java}
  if (this.ioErrorStartTime > 0) {
    ioErrorStartTime = -1;
  }
{code}

Notice that that if ioErrorStartTime is set to -1 after the first if statement in checkIOErrorIsTolerated but before (now - ioErrorStartTime), then (now - ioErrorStartTime) will evaluate to (now + 1) resulting in the code thinking it has exceeded ioErrorsTolerationDuration.
Bug: Remove the HasThread because the related bug had been fixed since jdk7
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/20 05:45

The HasThread was introduced as a wrapper of Thread to workaround a deadlock bug, and this bug had fixed since jdk7, so I think it is the time to remove it as we are using jdk8+.
  
 Related links:
 [https://issues.apache.org/jira/browse/HBASE-4367|https://slack-redir.net/link?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHBASE-4367]
 [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6915621|https://slack-redir.net/link?url=https%3A%2F%2Fbugs.java.com%2Fbugdatabase%2Fview_bug.do%3Fbug_id%3D6915621]
Bug: Improve default thread pool size for opening store files
Status: Open | Priority: Major | Resolution: 
Created: 27/May/20 16:16

For each store open we create a CompletionService and also create a thread pool for opening and closing store files. See HStore#openStoreFiles and HRegion#getStoreFileOpenAndCloseThreadPool. By default this pool has only one thread. It can be increased with "hbase.hstore.open.and.close.threads.max" but this config value is then divided by number of stores in the region.

"hbase.hstore.open.and.close.threads.max" is also used to size other thread pools for opening and closing the stores themselves, so it's an unfortunate overloading.

We should have a configuration parameter that directly and simply tunes the thread pool size for opening store files. Introduce a new configuration parameter: "hbase.hstore.hfile.open.threads.max" which will define the upper bound for a thread pool shared by the entire store for opening hfiles. The default should be 1 to preserve default behavior.

Once this is done, we could increase this to 2, 4, 8, or more for increased parallelism when opening store files without impact on other activities. The time required to open all storefiles often dominates the total time for bringing a region online. The thread pool will be shut down and eligible for garbage collection once all files are loaded and the store is online.

Number of open threads should scale with the number of stores, so allocating the pool at the store level continues to make sense.

Longer term we might try recursively decomposing the region open task with a fork-join pool such that the opening of store files can be dynamically parallelized in a probably superior way (conjecture pending a real attempt with metrics) .
Bug: CatalogJanitor misreports region holes when there are actually over laps.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/20 18:26
Affects Version/s: 2.3.0

We found a case that there is actually an overlap, but a region hole is reported.

r1: [aa, bb), r2: [cc, dd), r3: [a, cc)

 

In this case, there are only overlaps from "a" to "d". However, hole (r1, r2) is reported.
Bug: region_mover.rb NoMethodError: undefined method `tmpdir' for Dir:Class
Status: Open | Priority: Major | Resolution: 
Created: 28/May/20 22:41
Affects Version/s: 1.6.0

Trying to use region_mover.rb to unloadRegion and see the following

 
NoMethodError: undefined method `tmpdir' for Dir:Class
    getFilename at /usr/current/hbase-regionserver/bin/region_mover.rb:445
  unloadRegions at /usr/current/hbase-regionserver/bin/region_mover.rb:299
         (root) at /usr/current/hbase-regionserver/bin/region_mover.rb:525
Bug: Expose HBCK Chore Reports via metrics
Status: Open | Priority: Major | Resolution: 
Created: 28/May/20 17:12

Talking with an operator about their experience upgrading to HBase2. They automate as much as they can, and like having access to so much system state via {{/json}} endpoint. One of the things they miss on the endpoint is access to hbck reports. Now in HBase2 these reports are managed by the master, we can represent them in JMX and make them available as metrics as well.
Bug: Make it easy to delete all data for specify qualifiers
Status: Open | Priority: Major | Resolution: 
Created: 25/May/20 03:08

We developed a great many user features that each one as a qualifier stored in a table which has only one column family, when we planed to decommission some of them, but found not easy to do.
  
 My propasal is add a hint for major_compact, eg:
{code:java}
major_compact_delete_extra_qualifiers 'table','cf','q1','q2'{code}
This command will delete all of the cells for q1 and q2.
Bug: Run hbck_chore_run and catalogjanitor_run on draw of 'HBCK Report' page
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/20 03:51

Run the 'catalogjanitor_run' and the 'hbck_chore_run' when we draw the 'HBCK Report' page.

Its PITA running command on the shell every time you need a new report.

Can avoid new behavior by passing '?cache=true' on URL.
Bug: Flaky test, TestLocalRegionOnTwoFileSystems#testFlushAndCompact
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/20 17:20

{code:java}
precommit checks / yetus jdk8 Hadoop2 checks / org.apache.hadoop.hbase.master.store.TestLocalRegionOnTwoFileSystems.testFlushAndCompactFailing for the past 1 build (Since #1 )Took 17 sec.Error MessageWaiting timed out after [15,000] msecStacktracejava.lang.AssertionError: Waiting timed out after [15,000] msec
	at org.apache.hadoop.hbase.master.store.TestLocalRegionOnTwoFileSystems.testFlushAndCompact(TestLocalRegionOnTwoFileSystems.java:178)
Standard OutputFormatting using clusterid: testClusterID
Standard Error2020-05-26 00:26:29,624 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.store.TestLocalRegionOnTwoFileSystems timeout: 13 mins
2020-05-26 00:26:30,158 DEBUG [main] hbase.HBaseTestingUtility(348): Setting hbase.rootdir to /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1776/yetus-jdk8-hadoop2-check/src/hbase-server/target/test-data/f6a36278-321e-2b82-017a-bbe71410a0cf
2020-05-26 00:26:30,231 INFO  [Time-limited test] hbase.HBaseTestingUtility(1114): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false}
2020-05-26 00:26:30,232 INFO  [Time-limited test] hbase.HBaseZKTestingUtility(83): Created new mini-cluster data directory: /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1776/yetus-jdk8-hadoop2-check/src/hbase-server/target/test-data/f6a36278-321e-2b82-017a-bbe71410a0cf/cluster_ddea42b7-f6f1-92fe-0685-674774d0fce2, deleteOnExit=true
2020-05-26 00:26:30,233 INFO  [Time-limited test] hbase.HBaseTestingUtility(1128): STARTING DFS {code}
Bug: Cell size limit check on append should consider 0 or less value to disable the check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/20 02:32
Affects Version/s: 1.4.10

HBASE-22274 added a cell size check for append in HRegion, but did not consider 0 and less values.

If we set  the parameter *hbase.server.keyvalue.maxsize* to 0, expect to turn off the cell size check, all append operations will fail.
Bug: When running the tests, a tremendous number of connections are put into TIME_WAIT.
Status: Resolved | Priority: Major | Resolution: Information Provided
Created: 09/Apr/20 14:12

When you run the test suite and monitor the number of connections in TIME_WAIT, it appears that a very large number of connections do not end up with a proper connection close lifecycle or perhaps proper reuse.

Given connections can stay in TIME_WAIT from 1-4 minutes depending on OS/Env, running the tests faster or with more tests in parallel increases the TIME_WAIT connection buildup. Some tests spin up a very, very large number of connections and if the wrong ones run at the same time, this can also greatly increase the number of connections put into TIME_WAIT. This can have a dramatic affect on performance (as it can take longer to create a new connection) or flat out fail or timeout.

In my experience, a much, much smaller number of connections in a test suite would end up in TIME_WAIT when connection handling is all correct.

Notes to come in comments below.
Bug: Make `RESTApiClusterManager` more resilient
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/20 23:48
Affects Version/s: 2.3.0

The Cloudera Manager API client in {{RESTApiClusterManager}} appears to assume that API calls sent to CM for process commands block on command completion. However, these commands are "asynchronous," queuing work in the background for execution. Update the client to track command submission and block on completion of that commandId. This allows this {{ClusterManager}} to conform to the expectations of the {{Actions}} that invoke it.
Bug: Retain the ReadRequests and WriteRequests of region on web UI after alter table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/20 08:21

When we alter a table, all regions of it will do a rs self move, then the ReadRequests and WriteRequests will be cleared, but they are very useful metrics, my propose is keep them in RegionServerAccounting on close if it is a rs self move, and recover them on open.
Bug: [Chaos Monkey] Invalid data block encoding in ChangeEncodingAction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/20 10:01

I found the following exception when I run ITBLL:
{code:java}
2020-05-12 11:43:14,201 WARN  [ChaosMonkey] policies.Policy: Exception performing action:
java.lang.IllegalArgumentException: There is no data block encoder for given id '6'
        at org.apache.hadoop.hbase.io.encoding.DataBlockEncoding.getEncodingById(DataBlockEncoding.java:168)
        at org.apache.hadoop.hbase.chaos.actions.ChangeEncodingAction.lambda$perform$0(ChangeEncodingAction.java:50)
        at org.apache.hadoop.hbase.chaos.actions.Action.modifyAllTableColumns(Action.java:356)
        at org.apache.hadoop.hbase.chaos.actions.ChangeEncodingAction.perform(ChangeEncodingAction.java:48)
        at org.apache.hadoop.hbase.chaos.policies.PeriodicRandomActionPolicy.runOneIteration(PeriodicRandomActionPolicy.java:59)
        at org.apache.hadoop.hbase.chaos.policies.PeriodicPolicy.run(PeriodicPolicy.java:41)
        at java.lang.Thread.run(Thread.java:748)
{code}
Because PREFIX_TREE is removed in DataBlockEncoding:
{code:java}
/** Disable data block encoding. */
NONE(0, null),
// id 1 is reserved for the BITSET algorithm to be added later
PREFIX(2, "org.apache.hadoop.hbase.io.encoding.PrefixKeyDeltaEncoder"),
DIFF(3, "org.apache.hadoop.hbase.io.encoding.DiffKeyDeltaEncoder"),
FAST_DIFF(4, "org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder"),
// id 5 is reserved for the COPY_KEY algorithm for benchmarking
// COPY_KEY(5, "org.apache.hadoop.hbase.io.encoding.CopyKeyDataBlockEncoder"),
// PREFIX_TREE(6, "org.apache.hadoop.hbase.codec.prefixtree.PrefixTreeCodec"),
ROW_INDEX_V1(7, "org.apache.hadoop.hbase.io.encoding.RowIndexCodecV1");
{code}
Bug: MemStoreFlusher throw NullPointerException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/20 08:24

{code}
2020-02-26 10:15:33,585 ERROR [MemStoreFlusher.2] regionserver.MemStoreFlusher: Cache flusher failed for entry null
java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:158)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$700(MemStoreFlusher.java:68)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:346)
        at java.lang.Thread.run(Thread.java:745)
{code}

Need to setFlushType first before use it.
Bug: Add ServerSsdLocalityCostFunction to StochasticLoadBalancer
Status: Open | Priority: Major | Resolution: 
Created: 09/Apr/20 09:27

When use ONE_SSD storagy policy, or ALL_SSD but has not enough SSD, there will be some hdfs blocks on DISK and others on SSD,so it is reasonable to consider the locality of ssd for StochasticLoadBalancer.
Bug: release scripts should do compat report that includes LimitedPrivate APIs for maintenance releases
Status: Open | Priority: Major | Resolution: 
Created: 16/May/20 18:25

currently the release script does a compatibility report that just covers IA.Public apis. That's covers our promises for new minor versions. When doing maintenance versions (e.g. 2.2.5) we ought to be reporting on any breaking changes in IA.LimitedPrivate APIs as well.