av9ash/GitBugs huggingface dataset:
Bug: maxPoolSize is logged incorrectly in ByteBufferPool
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Apr/20 09:02
Affects Version/s: 2.2.4

In ByteBufferPool _maxPoolSize_ is converted into byte format,

https://github.com/apache/hbase/blob/a521a80c4b9a8b0749c368d1ff66fea2ed2d77a2/hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferPool.java#L85

 

Currently maxPoolSize is logged as below,

2020-04-10 14:20:56,000 INFO  [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B
Bug: new major version in release script should default compat report to previous latest minor release
Status: Open | Priority: Minor | Resolution: 
Created: 16/May/20 18:21

currently if you are starting a new major version, the default calculated comparison version is invalid:

{code}

========================
=== Gathering release details.

PROJECT [hbase]: hbase
GIT_BRANCH [branch-2.3]: master
Current branch VERSION is 3.0.0-SNAPSHOT.
RELEASE_VERSION [3.0.0]: 3.0.0-alpha-1
NEXT_VERSION [3.0.1-SNAPSHOT]: 3.0.0-alpha-2-SNAPSHOT
RC_COUNT [0]: 
RELEASE_TAG [3.0.0-alpha-1RC0]: 
This is a dry run. If tag does not actually exist, please confirm the ref that will be built for testing.
GIT_REF [3.0.0-alpha-1RC0]: 
api_diff_tag [rel/3.0.-2]: rel/2.2.4
{code}

when the new version is a new major we should either default to the prior initial major release (e.g. 2.0.0) or we should scan the release metadata for the most recent release from the prior major release (e.g. 2.2.4).
Bug: Point hbase-it pom value for testFailureIgnore to value defined in main hbase pom
Status: Open | Priority: Minor | Resolution: 
Created: 14/May/20 23:53
Affects Version/s: 1.6.0

The testFailureIgnore value in the hbase-it submodule pom is hard-coded to {{false}}; it should be changed to inherit the value defined in the main hbase pom to for consistency reasons.
Bug: The region should be removed from ConfigurationManager as a ConfigurationObserver when it is closed
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/May/20 13:04

When the region is opened, we register the region to the ConfigurationManager as a ConfigurationObserver object.However, when the region is closed, we don't deregister the region as a ConfigurationObserver object from the ConfigurationManager correspondingly. It's not a bug for now cos we can update the conf  whenever the region is open or not.But it is bug-prone, and we should remove it from the ConfigureManager object when the region is closed.
Bug: Make getServerName(String, int, long) private
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 02/May/20 13:39
Affects Version/s: 3.0.0-alpha-1

{{getServerName(String, int, long)}} was marked deprecated in 2.0.0 with the option to make it private in 3.0.0. It should be made sure that it is marked as private in 3.0.0.
Bug: Relax version numbers specified in Dockerfiles
Status: Open | Priority: Minor | Resolution: 
Created: 27/Apr/20 16:46
Affects Version/s: 2.3.0

It seems the version numbers used in our dockerfiles are too strict. As debian maintainers bump internal revisions, the old ones roll off, and our docker builds break. We should be less strict on the version numbers, but exactly how strict will need a package-by-package evaluation.
Bug: [HBCK2] Remove removeExtraRegionsFromMeta from HBCK2 doc
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Apr/20 01:26
Affects Version/s: hbase-operator-tools-1.0.0

HBCK2 doc mentions about an internal method name, removeExtraRegionsFromMeta, which is called from the *extraRegionsInMeta --fix* command internally.

https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#extra-regions-in-hbasemeta-regiontable-restorerebuild
{noformat}
Such problem can be addressed with the Master online, using the removeExtraRegionsFromMeta command in HBCK2.
{noformat}

We should replace with the right command.
Bug: ChecksumUtil class validateChecksum method log level does not match
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 22/Apr/20 09:00
Affects Version/s: 2.2.0

In the validateChecksum method of the ChecksumUtil class, still use LOG.info after judging LOG.isTraceEnabled.

eg:

 if (LOG.isTraceEnabled()) {
 LOG.info("dataLength=" + buffer.capacity()
 + ", sizeWithHeader=" + onDiskDataSizeWithHeader
 + ", checksumType=" + cktype.getName()
 + ", file=" + pathName
 + ", offset=" + offset
 + ", headerSize=" + hdrSize
 + ", bytesPerChecksum=" + bytesPerChecksum);
 }
Bug: Duplicate implementation for acquireLock between CreateTableProcedure and its parent class
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Apr/20 11:14
Affects Version/s: 2.2.4

The override method _acquireLock_ in _CreateTableProcedure_ and _InitMetaProcedure_ is the same as the implementation in its parent class _AbstractStateMachineTableProcedure_. So delete the override method in subclass.
Bug: Wrong input argument order in TestCompactorMemLeak#assertMajorCompactionOK
Status: Patch Available | Priority: Minor | Resolution: 
Created: 17/Apr/20 18:00
Affects Version/s: 2.2.4

In TestCompactorMemLeak#assertMajorCompactionOK, the input argument order for 

Assert.assertEquals(long expected, long actual) is wrong. This produced confusion when the test failed.

 

Below is the original code:


{code:java}
Assert.assertEquals(regions.size(), 1);
HRegion region = regions.get(0);
Assert.assertEquals(region.getStores().size(), 1);
HStore store = region.getStore(FAMILY);
Assert.assertEquals(store.getStorefilesCount(), 1);
{code}
Bug: [rsgroup] The master server  aborted for IllegalThreadStateException
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Apr/20 05:28
Affects Version/s: 2.1.7

if 'hbase.master.loadbalancer.class=org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer' and 'hbase.rsgroup.grouploadbalancer.class=org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer',
The master server will aborted for IllegalThreadStateException.
{code:java}
2020-04-09 11:24:26,142 ERROR [master/xx.xx.xx.xx:16000:becomeActiveMaster] master.HMaster: Failed to become active master
java.lang.IllegalThreadStateException
        at java.lang.Thread.start(Thread.java:705)
        at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl.start(RSGroupInfoManagerImpl.java:172)
        at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.initialize(RSGroupBasedLoadBalancer.java:417)
        at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.initialize(RSGroupBasedLoadBalancer.java:430)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:968)
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2275)
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:584)
        at java.lang.Thread.run(Thread.java:745)
2020-04-09 11:24:26,144 ERROR [master/xx.xx.xx.xx:16000:becomeActiveMaster] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.hadoop.hbase.quotas.MasterQuotasObserver, org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint]{code}
Bug: user_permission '.*'  command failed to show all table permissions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Apr/20 02:57

When we grant somebody global permission, then we use '.*' as the argument of user_permission to show all table permissions, the following exception occurred:
{code:java}
hbase(main):001:0>  grant 'user3', 'RW'
Took 0.4144 seconds                                                                                                                                                                            
hbase(main):002:0> user_permission 
User                                             Namespace,Table,Family,Qualifier:Permission                                                                                                   
 user2                                           ,,,: [Permission: actions=READ,WRITE]                                                                                                         
 user3                                           ,,,: [Permission: actions=READ,WRITE]                                                                                                         
 hbaseadmin                                      ,,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]                                                                                       
3 row(s)
Took 0.0445 seconds                                                                                                                                                                            
hbase(main):003:0> user_permission '.*'
User                                             Namespace,Table,Family,Qualifier:Permission                                                                                                   
 user1                                           default,t1,,: [Permission: actions=READ,WRITE]                                                                                                
 hbaseadmin                                      default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]                                                                              ERROR: failed to coerce org.apache.hadoop.hbase.security.access.GlobalPermission to org.apache.hadoop.hbase.security.access.TablePermissionFor usage try 'help "user_permission"'Took 0.0940 seconds 
{code}
Bug: create-release fails to process x.y.0 version info correctly
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Apr/20 22:04

Trying out {{create-release/do-release-docker.sh}}, is has trouble parsing the 2.3.0-SNAPSHOT version. It also builds an invalid tag name for the API check.

{noformat}
Current branch VERSION is 2.3.0-SNAPSHOT.
RELEASE_VERSION [2.3.-1]: 2.3.0
NEXT_VERSION [2.3.0-SNAPSHOT]: 2.3.1-SNAPSHOT
RC_COUNT [0]:
GIT_REF [2.3.0RC0]:
api_diff_tag, [rel/2.2.0)]:
{noformat}
Bug: Add release branch report to git-jira-release-audit tool
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Apr/20 00:00

Update {{git-jira-release-audit}} to build a "what's new" report for a release branch (i.e., {{branch-2.3}}). Currently it only supports building such a report from a release line branch (i.e., {{branch-2}}).
Bug: Passing '-h' or '--help' to bin/hbase doesn't do as expected
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 12/Jun/20 17:19

If I do 'bin/hbase -h' or './bin/hbase --help', it doesn't dump usage as I'd expect. Instead, the param gets passed direct to the jvm for it to spew complaint that the param is unrecognized.

Should do the right thing.
Bug: hbase-filesystem to use hbase-thirdparty 3.2.0
Status: Open | Priority: Trivial | Resolution: 
Created: 06/Apr/20 19:12
Affects Version/s: hbase-filesystem-1.0.0-alpha1

hbase-filesystem is currently on hbase-thirdparty 2.2.1. Update it to 3.2.0 so we can use the latest guava.
Bug: Remove duplicated dependency declaration in hbase-hadoop-compat
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 06/May/20 07:39
Affects Version/s: 3.0.0-alpha-1

Currently hbase-metrics-api is declared twice with hbase-hadoop-compat. This leads to the following warning in Maven:

{code}
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase-hadoop-compat:jar:3.0.0-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hbase:hbase-metrics-api:jar -> duplicate declaration of version (?) @ line 113, column 19
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
{code}

One declaration should be removed.
Bug: correct parsing of hbase shell args with GetoptLong
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 06/Dec/21 21:08
Affects Version/s: 2.5.0

shell cli arg parsing is broken:
 * debug is eating a possible arg
 * -D handling is consuming part of ARGV without declaring it
 * -Dfoo=bar no longer works
 * non-interactive mode is eating a possible arg
 * top level definitions is eating a possible arg
 * script2run is being ignored entirely
Bug: Tracing Spans of RPCs should follow otel semantic conventions
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Nov/21 23:28
Affects Version/s: 2.5.0

The OpenTelemetry project has a [conventions document|https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/rpc.md] for RPCs that is pretty prescriptive. One thing I noticed is that it requires there be spans named as {{$package.$service/$method}} ; we don't do this. We should address this and any other gaps.
Bug: [site, branch-2] DependencyManagement report is broken
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 02/Dec/21 17:00
Affects Version/s: 2.5.0

[INFO] Generating "Dependency Management" report --- maven-project-info-reports-plugin:3.0.0:dependency-management
[WARNING] Unable to create Maven project for org.xerial.snappy:snappy-java:jar:1.1.8.4 from repository.
org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:
[ERROR] Unknown packaging: bundle @ line 6, column 16

Seems caused by org.xerial.snappy:snappy-java dependency introduced by HBASE-26259
Bug: Tracing INSTRUMENTATION_NAME is incorrect
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 01/Nov/21 23:53
Affects Version/s: 2.5.0

I believe the value we use for {{TraceUtil#INSTRUMENTATION_NAME}}, {{"io.opentelemetry.contrib.hbase"}}, is incorrect. According to the java docs,

{noformat}
   * @param instrumentationName The name of the instrumentation library, not the name of the
   *     instrument*ed* library (e.g., "io.opentelemetry.contrib.mongodb"). Must not be null.
{noformat}

This namespace appears to be reserved for implementations shipped by the otel project, found under https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation

I don't have a suggestion for a suitable name at this time. Will report back.
Bug: Changing SFT implementation on disabled table is dangerous
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 19/Dec/21 14:42

As our current design is to have a MIGRATION SFT so when region opens, we will mirror the store files to the dst SFT.

But for a disabled table, the region will not be reopened so we have no chance to mirror the store files list, so after user enable the table, it may find out that all the store files are gone...
Bug: Should not rely on the global config when setting SFT implementation for a table while upgrading
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 16/Dec/21 15:24

If a table does not have a SFT implementation in its table descriptor, it means we are upgrading from an older version where SFT is not implemented, so the SFT implementation for the table should always be DEFAULT.

A misconfigure here could cause data loss.
Bug: Address recent the log4j2 CVEs
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 11/Dec/21 13:00

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105
Bug: Close netty channel causes regionserver crash in handleTooBigRequest
Status: Open | Priority: Critical | Resolution: 
Created: 17/Nov/21 15:06
Affects Version/s: 2.0.0

In HBASE-26170, I proposed the coredump problem after calling handleTooBigRequest, but that issue did not resolve the regionserver crash problem, which occurs before the WAL corruption in HBASE-24984.

After looking through the codes, I think the problem is in CLOSE channel. 

The direct byte buffer used by RPC call request is allocated by Netty, though we add a reference count to record when to release the direct byte buffer, the byte buffer is managed by Netty actually. It is allocated from Netty PoolArena, and is released there. 

When the HBase ipc handler is processing a request, the Netty channel handler can process the channel events and message coming back in succession. When there is a too big request by NettyRpcFrameDecoder, the channel will be closed, and all the resources of the channel will be released, though there is HBase ipc handlers using the direct byte buffer to process previous requests.

Netty provides two methods to request the pooled byte buffer, one is through the PoolThreadCache, each handler thread owns a private one. Another is through PoolArena#allocateNormal. Each ChannelHandler has a local PoolThreadCache.

When a new Netty channel is created, a new ChannelHandler instance is created. 

And when a channel is closed, the relevant channel handler will be removed from the pipeline. I found this annotation in the Channel class of Netty,
{code:java}
It is important to call close() or close(ChannelPromise) to release all resources once you are done with the Channel. This ensures all resources are released in a proper way, i.e. filehandles. {code}
And when channel handler is removed in ByteToMessageDecoder#handlerRemoved, it will release the byte buffer,
{code:java}
@Override
public final void handlerRemoved(ChannelHandlerContext ctx) throws Exception {
    if (decodeState == STATE_CALLING_CHILD_DECODE) {
        decodeState = STATE_HANDLER_REMOVED_PENDING;
        return;
    }
    ByteBuf buf = cumulation;
    if (buf != null) {
        // Directly set this to null so we are sure we not access it in any other method here anymore.
        cumulation = null;

        int readable = buf.readableBytes();
        if (readable > 0) {
            ByteBuf bytes = buf.readBytes(readable);
            buf.release();
            ctx.fireChannelRead(bytes);
        } else {
            buf.release();
        }
... {code}
We should not close the channel when encountering too big request, I think it should just skip the bytes like that in LengthFieldBasedFrameDecoder.
Bug: [hboss] Rename does not clean up znodes for src location
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 09/Nov/21 22:27
Affects Version/s: hbase-filesystem-1.0.0-alpha1

We ran into a fun situation where the partition hosting ZK data was repeatedly filling up while heavy ExportSnapshot+clone_snapshot operations were running (10's of TB). The cluster was previously working just fine.

Upon investigation of the ZK tree, we found a large number of znodes beneath /hboss, specifically many in the corresponding ZK HBOSS path for $hbase.rootdir/.tmp.

Tracing back from the code, we saw that the CloneSnapshotProcedure (like CreateTableProcedure) will create the table filesystem layout in $hbase.rootdir/.tmp and then rename it into $hbase.rootdir/data/<namespace>. However, it appears that, upon rename, HBOSS was not cleaning up the src path's znode. This is a bug as it allows ZK to grow unbounded (which explains why this problem slowly arose and not suddenly).

As a workaround, HBase can be stopped and the corresponding ZK path for $hbase.rootdir/.tmp can be cleaned up to reclaim 1/2 the space taken up by znodes for imported hbase tables (we would still have znodes for $hbase.rootdir/data/...)
Bug: [hboss] removeInMemoryLocks can remove still in-use locks
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 12/Nov/21 15:59
Affects Version/s: hbase-filesystem-1.0.0-alpha1

While implementing HBASE-26437, I was fighting with unit tests which just wouldn't complete. After adding the code change to delete the locks held by the {{src}} in a {{mv src dst}} operation, releasing the {{dst}} lock would claim that the current thread doesn't hold the lock.

After investigating, the specific contract test in question is doing a rename of the form: {{{}mv /foo /foodest{}}}. This actually breaks the logic which tries to determine if a lock's path is contained beneath the path we're tryign to clean up. Specifically: cleaning up locks beneath {{/foo}} incorrectly removes locks for {{/foodest}}
Bug: HBCK incorrectly reports inconsistencies for recently split regions following a master failover
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 20/Oct/21 09:15
Affects Version/s: 2.4.3

When a region P splits into A and B, following a master failover the newly active master reports that P is in an inconsistent state. This seems to be a regression introduced in HBASE-25847 (cc [~andrew.purtell@gmail.com]) which changed {{regionInfo.isParentSplit()}} to {{regionState.isSplit()}}. The region state after restart is CLOSED (rather than SPLIT), so both region state and region info should be checked, presumably with {{regionState.isSplit() || regionInfo.isSplit()}}. This situation resolves itself on its own when a major compaction occurs and P is GCed, but having the master incorrectly report inconsistencies is pretty bad. We had a pretty big outage due to a series of operator errors as our SRE team was trying to fix this inconsistency that, in fact, didn't even exist.

Thanks to Stack for helping look over this issue and Vlad Hanciuta for root causing the bug.
Bug: Wal do not roll and write a big wal 
Status: Open | Priority: Major | Resolution: 
Created: 01/Nov/21 09:39
Affects Version/s: 2.4.8

We see wal have long time to roll and write a big wal which has 3TB.
And according to the jstack we can see the wal create hang.
{code}
"regionserver/11.149.48.227:60020.logRoller" #667 daemon prio=5 os_prio=0 cpu=116916.81ms elapsed=447455.26s tid=0x00007fa35d231000 nid=0xbdd2 waiting on condition [0x00007f79c7407000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f9f10df5158> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
        at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
        at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:178)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(AsyncProtobufLogWriter.java:191)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:170)
        at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:615)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:126)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:763)
        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:184)
        at java.lang.Thread.run(Thread.java:748)
{code}
Bug: Post blog to hbase.apache.org on SCR cache sizing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Oct/21 02:47

[~huaxiangsun] and I wrote up our experience debugging a Short-circuit Read cache size issue. Let me attach link here and leave it hang here a few days in case edits or input from others.  Intend to put it up here https://blogs.apache.org/hbase/