av9ash/GitBugs huggingface dataset:
Bug: Concurrent assign and SCP may cause regionserver abort
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 30/Nov/21 13:19
Affects Version/s: 1.7.0

When concurrent assign a region, the RS may abort when reporting region state to master encounters transit region state error, 

 
{code:java}
2021-11-26 04:40:54,689 INFO  [PostOpenDeployTasks:ee1df69da9c7fb7698d2972f81d40b65] regionserver.HRegionServer: Post open deploy tasks for growthosudx:growthos-udx_dmx_supply_waimaisku_application_basic,cc,1631020952330.ee1df69da9c7fb7698d2972f81d40b65.
2021-11-26 04:40:54,689 INFO  [PostOpenDeployTasks:ee1df69da9c7fb7698d2972f81d40b65] regionserver.HRegionServer: Failed to transition {ENCODED => ee1df69da9c7fb7698d2972f81d40b65, NAME => 'growthosudx:growthos-udx_dmx_supply_waimaisku_application_basic,cc,1631020952330.ee1df69da9c7fb7698d2972f81d40b65.', STARTKEY => 'cc', ENDKEY => 'cd'} to OPENED: ee1df69da9c7fb7698d2972f81d40b65 is not pending open on zf-data-hbase250.mt,16020,1637872783437
2021-11-26 04:40:54,689 FATAL [PostOpenDeployTasks:ee1df69da9c7fb7698d2972f81d40b65] regionserver.HRegionServer: ABORTING region server zf-data-hbase250.mt,16020,1637872783437: Exception running postOpenDeployTasks; region=ee1df69da9c7fb7698d2972f81d40b65
java.io.IOException: Failed to report opened region to master: growthosudx:growthos-udx_dmx_supply_waimaisku_application_basic,cc,1631020952330.ee1df69da9c7fb7698d2972f81d40b65.
        at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2387)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:327)  {code}
In the master log, we can see that the region is being assigned by both the bulk assigner and the SCP.

!master-error-logs.png|width=600,height=140!

This steps can reproduce the problem:
 # thread 1 calls Assignment#assign to assign region A, and calls RegionStates#updateRegionsState and stops before the synchronize;
 #  thread 2 calls RegionStates#serverOffline and gets the lock, it sees that region A is on the processed dead server and add region A to the regionsToOffline list(region A is not in the RIT cache now);
 # thread 2 prepare to execute RegionStates#regionOffline, and released the lock;
 # thread 1 get the lock, and add region A to the RIT cache, and also update the region state by RegionStates#putRegionState, set the region state from OFFLINE to PENDING_OPEN, then release the lock;
 # thread2 execute  RegionStates#regionOffline, acquires the lock and sets the region state from PENDING_OPEN to OFFLINE;
 # the RS that region A assigned by thread 1 report region A should be transited to OPENED, but finds that region A last state is OFFLINE, error occures, the RS abort itself.
Bug: Close NamedQueueRecorder to allow HMaster/RS to shutdown gracefully
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/21 14:02
Affects Version/s: 1.7.0

Saw one case in our production cluster where RS was not exiting. Saw this non-daemon thread in hung RS stack trace:

{noformat}
"main.slowlog.append-pool-pool1-t1" #26 prio=5 os_prio=31 tid=0x00007faf23bf7800 nid=0x6d07 waiting on condition [0x0000700003f4d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004039e3840> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:47)
        at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
{noformat}
        
This is coming from [NamedQueueRecorder|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/namequeues/NamedQueueRecorder.java#L65] implementation. 
This bug doesn't exists in branch-2 and master since the Disruptor initialization has changed and we set daemon=true also. See [this code|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/namequeues/NamedQueueRecorder.java#L68]
 
FYI [~vjasani] [~zhangduo]
Bug: Limit the length of initial resultSets for HBase thrift service
Status: Open | Priority: Major | Resolution: 
Created: 27/Nov/21 10:21
Affects Version/s: 1.4.14

When the client requests getScannerResults or getScannerRows, set up a large numRows will cause ArrayIndexOutOfBoundsException ，and  may  cause OOM.

For example
{code:java}
conn := cli.(*hbase.THBaseServiceClient)
tscan := &hbase.TScan{
   StartRow: []byte("prefix_1"),
   StopRow:  []byte("prefix_2"),
   Columns: []*hbase.TColumn{
      {
         Family:    []byte("N"),
         Qualifier: []byte("title"),
      },
   },
}
scanresult, err := conn.GetScannerResults([]byte("rowkey1"), tscan, 10000000){code}
Here initialize the length of resultSets using the minimum value of "hbase.thrift.max.scanner.results.init.length"  and numRows。
Bug: Immutable timeseries usecase - Create new region rather than split existing one
Status: Open | Priority: Major | Resolution: 
Created: 18/Nov/21 12:44

For insertion of immutable data usecase (specifically time-series data), region split mechanism doesn't seem to provide better availability when ingestion rate is very high. When we ingest lot of data, the region split policy tries to split the given hot region based on the size (either size of all stores combined or size of any single store exceeding max file size configured) if we consider default {_}SteppingSplitPolicy{_}. The latest hot regions tend to receive all latest inserts. When the region is split, the first half of the region (say daughterA) stays on the same server whereas the second half (daughterB) region – likely to become another hot region because all new latest updates come to second half region in the sequential write fashion – is moved out to other servers in the cluster. Hence, once new daughter region is created, client traffic will be redirected to another server. Client requests will be piled up when region split is triggered till new daughters come alive and once done, client will have to request meta for updated daughter region and redirect traffic to new server.

If we could have configurable region creation strategy that 1) keeps the split disabled for the given table, and 2) create new region dynamically with lexicographically higher start key on the same server and update it's own region boundary, the client will have to look up meta once and continue ingestion without any degraded SLA caused by region split transitions.

Note: region split might also encounter some complications, requiring the procedure to be rolled back from some step, or continue with internal retries, eventually further delaying the ingestion from clients.

 

There are some complications around updating live region's start and end keys as this key range is immutable. We could brainstorm ideas around making them optionally mutable and any issues around them. For instance, client might continue writing data to the region with updated end key but writes will fail for out of range keys and hence, they will lookup in meta for updated key-space range (new region created with end key: EMPTY_END_ROW) of the table.
Bug: Provide meaningful parent spans to ZK interactions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/21 22:40
Affects Version/s: 2.5.0

Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the standalone master, check the trace store. We see lots of single-span traces, each a call to different {{RecoverableZooKeeper}} methods, like {{create}} and {{exists}}. Whatever master startup thread this is should define its own root span.
Bug: Extend RSGroup to support data isolation to achieve true multitenancy in Hbase
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 10/Oct/21 02:31

RSGroups currently only provide isolation on serving layer, but not on the data layer. And there is a need for providing data isolation between rsgroups to achieve true multitenancy in hbase leading to independently scale individual rsgroups on need bases. Some of the aspects to be covered in this umbrella project are 
 # Provide data isolation between different RSGroups
 # Add balancer support to understand this construct while performing balancer activity
 # Extend support on various ancillary services such as export snapshot, cluster replication, etc
Bug: Use spotless to reduce the pain on fixing checkstyle issues
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Dec/21 00:18

https://github.com/diffplug/spotless

Spotless is a tool for reformatting code based on your configuration file, it supports the google java format, as well as a eclipse formatter file.

I think we could use it to format our code, based on a configuration file in our repo, and all the developpers, no matter what IDE they use, they can just type a mvn spotless:apply before commiting a PR, then everything is fine.
Bug: Update next set of minor releases for Hadoop 3.1 EOM 
Status: Open | Priority: Major | Resolution: 
Created: 06/Oct/21 14:10
Affects Version/s: 2.5.0

Looks like Hadoop 3.1 went EOL back in June. I can't find a formal announcement, but "[\[VOTE\] Hadoop 3.1.x EOL|https://s.apache.org/hadoop-vote-3.1-eol]" passed and HADOOP-17759 removed 3.1 from their active downloads page and the asf active distribution mirroring.

We should update our minimum testing version to 3.2 and update the hadoop dependency matrix to reflect this change.
Bug: TableSplit returns false size under 1MB
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/21 09:55

We calculate region size in the mapreduce package by getting the size in MB first and multiplying: https://github.com/apache/hbase/blob/39a20c528e2bf27cedf12734dbdb1b7b1e538076/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/RegionSizeCalculator.java#L87

This will give a size of 0 until at least 1MB is reached. (And it will have an unwanted rounding affect as well). 
Spark for example can be tuned to do some performance tuning by eliminating the 0 sized regions. This will eliminate any small regions which are not actually empty. The hadoop interface states the size is returned in bytes, and while this is true do to the multiplication, we multiply by 0 until 1MB is reached. I'm not sure why we get the size in MB units and not in bytes straight up.
Bug: [branch-2] TestLruBlockCache is flaky
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 27/Nov/21 00:50

[WARNING] Flakes: 
[WARNING] org.apache.hadoop.hbase.io.hfile.TestLruBlockCache.testBackgroundEvictionThread
[ERROR]   Run 1: TestLruBlockCache.testBackgroundEvictionThread:147 Eviction never happened.
[INFO]   Run 2: PASS
Bug: Use openlabtesting protoc on linux arm64 in HBASE 2.x
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/21 08:19

This is the same issue as HBASE-23612 but for HBase 2.x.

Please backport the fix to HBase 2.x because at the moment it is not possible to use Apache Phoenix on Linux ARM64.

I will send a new Pull Request!

-------------------------

Currently, there is no protoc 2.5.0 for release [1]. So we can make a new one for ARM specific. For make sure that could work on ARM.

We will introduce a new ARM artifact for protoc, group_id is org.openlabtesting.protobuf .. This is just used for protobuf-maven-plugin to compile .proto files. As the 3.X version of protoc support ARM already. So this won't affect the internal protoc usage, which is 3.5.1-1 now.

 

[1][https://github.com/protocolbuffers/protobuf/issues/3844#issuecomment-343355946]
Bug: Enable OpenTelemetry to be used from developer sandbox
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/21 18:52
Affects Version/s: 2.5.0

Enabling OpenTelemetry tracing in a checkout doesn't work.
{noformat}
Attach opentelemetry agent to enable trace
find: lib/trace: No such file or directory
Error opening zip file or JAR manifest missing : 
Error occurred during initialization of VM
agent library failed to init: instrument
{noformat}

A little tweak to {{bin/hbase}} should do the trick.
Bug: OpenTelemetry configuration support for per-process service names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/21 16:10

Our configuration should support setting a different {{service.name}} for each identifiable HBase process/component. This will make it easier to distinguish between, say, master, region server, when visualizing a trace. As a first pass, I'm guessing that most of the command targets of {{bin/hbase}} should be represented by the configuration.

For examples of how this might be implemented, see how we support per-process GC arguments, or how its done in this [Hadoop PR|https://github.com/apache/hadoop/pull/3445/files#diff-a6a286a42078a3fc5eed5bd1bf23163b3574a8f2246d7d8d2f81436b7603f1c2R436].
Bug: Support remove coprocessor by class name via alter table command
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/21 05:10
Affects Version/s: 3.0.0-alpha-2

With the shell, when operator wants to remove a table coprocessor, the flow is to

1. first use {{decs}} find the the mapping of coprocessor$# e.g. coprocessor$1, where # is the ordered number when the coprocessor was internally added to the table attribute 
2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a value that include the unique class name.

This task is to simplify the flow if the operator know exactly the class name of the added coprocessor, and create a new sub-method to {{alter}}, such that operator can do it only with the class name. 

NOTE that this logic has been added behind the scenes at [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565] for removing `ConstraintProcessor` , and we are just exposing this logic with a new method to {{alter}} command.
Bug: Implement tracing of scan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/21 00:09

Naive tracing of a long scan, where each back-and-forth between client and server gets its own Span, can result in many 100's or 1000's of Spans. This is easily overwhelming for tracing infrastructure. Identify some solution that allows for a reasonable out-of-the-box behavior when a scan is to be traced.
Bug: Remove use of `db.hbase.namespace` tracing attribute
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/21 16:10

Noticed by [~taklwu] and confirmed via https://github.com/open-telemetry/opentelemetry-specification/issues/1760. The HBase-specific attribute {{db.hbase.namespace}} has been deprecated in favor of the generic {{db.name}}.
Bug: [branch-2, hbase-thirdparty, Java 8] TestUnloadAccessController and other unit tests fail to start due to ByteBuffer link error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/21 00:23
Affects Version/s: 2.4.8

org.apache.hadoop.hbase.security.access.TestUnloadAccessController

Hang in setUpBeforeClass. Master will not initialize. Root cause is a NoSuchMethodError.

{noformat}
2021-11-26 19:09:56,465 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=62950] ipc.RpcExecutor$Handler(370):
Handler errors java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
	at org.apache.hbase.thirdparty.com.google.protobuf.CodedOutputStream$HeapNioEncoder.flush(CodedOutputStream.java:1546)
	at org.apache.hadoop.hbase.ipc.ServerCall.writeToCOS(ServerCall.java:378)
	at org.apache.hadoop.hbase.ipc.ServerCall.createHeaderAndMessageBytes(ServerCall.java:385)
	at org.apache.hadoop.hbase.ipc.ServerCall.createHeaderAndMessageBytes(ServerCall.java:363)
	at org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:267)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:168)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:354)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:334)
{noformat}

This is a known issue with ByteBuffer in JDK 8 vs ByteBuffer in later versions. When code is compiled with Java 9 or later using a specific subset of ByteBuffer APIs, the resulting bytecode will not link with Java 8's runtime. It works fine the other way. When compiled with Java 8, the bytecode will link with later Java runtimes just fine.

protobuf included into hbase-thirdparty was likely compiled with Java 9 or later. We shade that bytecode as is into hbase-thirdparty.  Tests were attempted with Java 8, so this failure case manifested.

Apache Maven 3.8.3 (ff8e977a158738155dc465c6a97ffaf31982d739)
Java version: 1.8.0_312, vendor: Azul Systems, Inc., runtime: /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
OS name: "mac os x", version: "12.0.1", arch: "aarch64", family: "mac"

We should be able to fix this problem by compiling protobuf with Java 8 and then shading the result when building hbase-thirdparty.
Bug: Support priority select reference files in StripeCompactionPolicy
Status: Patch Available | Priority: Major | Resolution: 
Created: 04/Nov/21 12:52

The reference files are those with these patterns, <hfile>.<parentEncRegion> and <table>=<region>-<hfile>.<parentEncRegion>.

We can compact these reference files in the first compaction after splitting, instead of a major compaction.

Then when the region only has normal hfiles and HFileLink files, it can be splitted once more.

From the overall perspective, it can reduce compaction IOs.
Bug: Region opens sometimes fail when bucket cache is in use
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 10/Nov/21 17:17
Affects Version/s: 2.4.8

After a split, I think, the region is reopened, and:

2021-11-10 16:47:48,929 ERROR [RS_OPEN_REGION-regionserver/ip-172-31-63-83:8120-1] regionserver.HRegion: Could not initialize all stores for the region=IntegrationTestLoadCommonCrawl,,1636562865609.e2df2061bcdc037070041de734a187ff.

Caused by: java.io.IOException: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
        at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:577)
        at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:534)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:298)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6546)

Caused by: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
        at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.validateBlockAddition(BlockCacheUtil.java:205)
        at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:237)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.shouldReplaceExistingCacheBlock(BucketCache.java:446)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:431)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:417)
        at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:64)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.lambda$readBlock$2(HFileReaderImpl.java:1346)

The open fails but another attempt somewhere else succeeds.
Bug: branch-2 modules dependencies are confused
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 26/Oct/21 21:03
Affects Version/s: 2.5.0

When trying to build branch-2, I get several dependency resolution issues. Different issues based on what approach I try.

 

{{mvn -e -B --no-transfer-progress -Dhadoop.profile=3.0 -Dhadoop-three.version=3.0.0-cdh6.3.2 clean install -Dmaven.test.skip=true}}

will fail with:
{noformat}
231.4 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce (hadoop-profile-min-maven-min-java-banned-xerces) on project hbase-build-configuration: Execution hadoop-profile-min-maven-min-java-banned-xerces of goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce failed: org.apache.maven.shared.dependency.graph.DependencyGraphBuilderException: Could not resolve following dependencies: [org.apache.hbase:hbase-annotations:jar:tests:2.5.0-SNAPSHOT (test)]: Could not resolve dependencies for project org.apache.hbase:hbase-build-configuration:pom:2.5.0-SNAPSHOT: Could not find artifact org.apache.hbase:hbase-annotations:jar:tests:2.5.0-SNAPSHOT in snapshots{noformat}
Note that I'm running this behind a corporate firewall, so if there is a snapshot available upstream in apache snapshots repo, then I'm not able to access it. I would have thought that it can use the locally built and installed one instead, since build-configuration does correctly come after annotations.

 

If I change to {{-DskipTests}} then this target succeeds, but later {{mvn site}} fails with
{noformat}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on project hbase: Error generating maven-javadoc-plugin:3.2.0:aggregate-no-fork report:
[ERROR] Exit code: 1 - /home/jenkins-agent/hbase/hbase-testing-util/src/main/java/org/apache/hadoop/hbase/testing/TestingHBaseClusterOption.java:24: error: cannot find symbol
[ERROR] import org.apache.hadoop.hbase.StartMiniClusterOption;
[ERROR]                               ^
[ERROR]   symbol:   class StartMiniClusterOption
[ERROR]   location: package org.apache.hadoop.hbase {noformat}
Again, I'm missing testing packages, this time for javadocs.

 

I believe I was getting similar errors on main branch, although I don't have logs for that anymore.
Bug: [hbase-thirdparty] Make a 4.0.1 release for hbase-thirdparty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Dec/21 04:27

Copy the content in mailing list here

{noformat}
Oh, when backporting to branch-2, I found that there is a serious problem with this release.

We do not include the proto files in protobuf-java jar, I think this is a side effect of HBASE-26496.

And on the master branch, hbase-shaded-protobuf also pulls in protobuf-java 3.19.1 dependency so everything is fine. But for branch-2, we will need to depend on protobuf 2.5.0 then the compilation is broken.

Anyway, we should include the proto files in hbase-shaded-protobuf,  and also we should not pull in the original protobuf-java dependency.

Let me fix this and do a 4.0.1 release.
{noformat}
Bug: The way we add or clear failedReplicas may have race
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/21 02:20

We will add a replica to failedReplicas in the callback of the replay method call, and we will clear the failedReplicas in add method when we meet a flush all edit.

They are in different threads, so it is possible that, we have already clear the failedReplicas due to a flush all edit, then in the callback of replay, we add a replica to the failedReplicas because of a failure of replicating, although the failure is actually before the flush all edit.

We should find a way to fix this.