av9ash/GitBugs huggingface dataset:
Bug: The SnapshotOfRegionAssignmentFromMeta.initialize call in FavoredNodeLoadBalancer is just a dummy one
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/21 14:32

After we introduced FavoredNodesManager, we do not need to load this information every time when balancing, as all the updates to favored nodes will go to FavoredNodesManager and it can update its in memory state accordingly.

The current SnapshotOfRegionAssignmentFromMeta.initialize is just a dummy one in FavoredNodeLoadBalancer, as we never use it in the method.

It will introduce a full scan on hbase:meta so let's just remove it.
Bug: Backport HBASE-25703 and HBASE-26002 to branch-2.3
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 17/Jun/21 03:20

Backport HBASE-25703 "Support conditional update in MultiRowMutationEndpoint" and HBASE-26002 "MultiRowMutationEndpoint should return the result of the conditional update" to branch-2.3.
Bug: UI of master-status to show recent history of balancer desicion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/21 10:28
Affects Version/s: 2.5.0

HBASE-24528 provide ‘Balancer Decision’ to display the history that includes decision factor details and weights and costs while running balancer.
This issue  implement 'Balancer Decision' UI web page
Bug: Move method reportProcedureDone from RegionServerStatus.proto to Master.proto
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/21 10:03

We next need use the procedure mechanism to implement enable/disable/refresh peer, and  ReplicationServer also needs reportProcedureDone to master, so I hope to move method reportProcedureDone to Master.proto from RegionServerStatus.proto.
Bug: Add a flag to mark if the IOError can be solved by retry in thrift IOError
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/21 06:26

Currently, if an HBaseIOException occurs, the thrift client can only get the error message. This is inconvenient for the client constructing a retry mechanism to handle the exception. So I added a canRetry mark in IOError to make the client side exception handling smarter.
Bug: Region state stuck in PENDING_OPEN
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/21 13:07
Affects Version/s: 1.0.0

Region will not be reassigned if encounters ConnectionClosingException, and then it will stuck in PENDING_OPEN state. Error logs are as follows,
{code:java}
INFO  [jd-data-hbase02.gh.sankuai.com,16000,1621944138744-GeneralBulkAssigner-12] master.AssignmentManager: Unable to communicate with jd-data-hbase15.gh.sankuai.com,16020,1622026221268 in order to assign regions, 
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to jd-data-hbase15.gh.sankuai.com/10.78.96.166:16020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to jd-data-hbase15.gh.sankuai.com/10.78.96.166:16020 is closing. Call id=19239, waitTime=1
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:289)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
        at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:25890)
        at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:798)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1744)
        at org.apache.hadoop.hbase.master.GeneralBulkAssigner$SingleServerBulkAssigner.run(GeneralBulkAssigner.java:203)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to jd-data-hbase15.gh.sankuai.com/10.78.96.166:16020 is closing. Call id=19239, waitTime=1
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1083)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:863)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:580)
{code}
Bug: JVM crash when displaying regionserver UI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/21 06:10
Affects Version/s: 2.0.0

The MonitoredRPCHandlerImpl refers to the params of a request, and will show them when we call 'toJson()'. But the running RPC call may be cleaned up and the ByteBuffer be released  before the displaying in UI.  We need to let the life cycle of RPC status monitor be inner the life cycle of RPC.
{code:java}
J 19267 C2 org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$Printer.printMessage(Lorg/apache/hbase/thirdparty/com/google/protobuf/MessageOrBuilder;Lorg/apache/hbase/thirdparty/com/google/protobuf/TextFormat$TextGenerator;)V (73 bytes) @ 0x00007f1ac7e54640 [0x00007f1ac7e53f60+0x6e0]
J 20932 C2 org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$Printer.print(Lorg/apache/hbase/thirdparty/com/google/protobuf/MessageOrBuilder;Lorg/apache/hbase/thirdparty/com/google/protobuf/TextFormat$TextGenerator;)V (34 bytes) @ 0x00007f1ac68ab9b0 [0x00007f1ac68ab880+0x130]
J 21843 C1 org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage.toString()Ljava/lang/String; (8 bytes) @ 0x00007f1ac620e14c [0x00007f1ac620dba0+0x5ac]
J 21835 C1 org.apache.hadoop.hbase.monitoring.MonitoredRPCHandlerImpl.toMap()Ljava/util/Map; (240 bytes) @ 0x00007f1ac5009bf4 [0x00007f1ac50071c0+0x2a34]
J 21833 C1 org.apache.hadoop.hbase.monitoring.MonitoredRPCHandlerImpl.toJSON()Ljava/lang/String; (5 bytes) @ 0x00007f1ac74efb74 [0x00007f1ac74efaa0+0xd4]
j  org.apache.hadoop.hbase.tmpl.common.TaskMonitorTmplImpl.renderNoFlush(Ljava/io/Writer;)V+259
j  org.apache.hadoop.hbase.tmpl.common.TaskMonitorTmpl.renderNoFlush(Ljava/io/Writer;)V+16
j  org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(Ljava/io/Writer;)V+129
{code}
[^hs_err_pid116190.log-gha-data-hbase-cat0085-ui]
Bug: Performance improvements for StochasticLoadBalancer
Status: Open | Priority: Major | Resolution: 
Created: 01/May/21 00:52

This is what I posted in the slack channel.

{quote}
When refactoring balancer related code for moving them out to hbase-balancer module,  I found that we will create a OffPeakHours instance in MoveCostFunction.cost every time. So I wrote a simple JMH test to test its performance.

https://github.com/Apache9/hbase-jmh-test/blob/master/src/main/java/org/apache/hadoop/hbase/master/balancer/MoveCostFunctionBenchmark.java

For the original version,  the result is
Benchmark                                            Mode  Samples    Score  Score error  Units
o.a.h.h.m.b.MoveCostFunctionBenchmark.computeCost    avgt       10  242.312        3.853  ns/op
12:06
If I initialized OffPeakHours in constructor, then the result is
Benchmark                                            Mode  Samples   Score  Score error  Units
o.a.h.h.m.b.MoveCostFunctionBenchmark.computeCost    avgt       10  81.186        0.720  ns/op
12:06
If I also eliminate the conf.getFloat in the cost method(by storing them to a field member in constructor, the result is
Benchmark                                            Mode  Samples  Score  Score error  Units
o.a.h.h.m.b.MoveCostFunctionBenchmark.computeCost    avgt       10  4.419        0.035  ns/op
12:08
I think on a large cluster, the performance for balancer is a problem, so maybe we should try to avoid creating too many objects or calling conf.getXXX in the critical path? As we will call StochasticLoadBalancer.computeCost in every step when balancing cluster...
{quote}

For large clusters, the performance of StochasticLoadBalancer will be very important, so I think we need to treat this problem seriously and try to improve.
Bug: POC: hbase:meta,,1 as ROOT
Status: Open | Priority: Major | Resolution: 
Created: 10/Apr/21 16:19

One of the proposals up in the split-meta design doc suggests a sleight-of-hand where the current hard-coded hbase:meta,,1 Region is leveraged to serve as first Region of a split hbase:meta but also does double-duty as 'ROOT'. This suggestion was put aside as a complicating recursion in chat but then Francis noticed on a re-read of the BigTable paper, that this is how they describe they do 'ROOT': "The root tablet is just the first tablet in the METADATA table, but is treated specially -- it is never split..."

This issue is for playing around with this notion to see what the problems are so can do a better description of this approach here, in the design:

https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle
Bug: hbase-cleanup.sh did not clean the wal dir if hbase.wal.dir configured individually
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/21 16:06
Affects Version/s: 2.2.3

If 'hbase.wal.dir' and 'hbase.rootdir' are configured to different filesystem or directories, hbase clean script will clean the directory of 'hbase.rootdir' only, the directory of 'hbase.wal.dir'  will be left.

The left direcotries 'WAls,' 'oldWALs', and recovered.edits under region dir should be cleaned also.
Bug: MultiRowMutationEndpoint should return the result of the conditional update
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/21 13:11

HBASE-25703 added support for the conditional update in MultiRowMutationEndpoint, but MultiRowMutationEndpoint doesn't return the result of the conditional update (whether or not the mutations are executed). In this Jira, we will make MultiRowMutationEndpoint return the result of the conditional update.
Bug: TestReplicationKillRS is useless after HBASE-23956
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/21 11:24

In TestReplicationKillRS, we assume there are at least 2 regionservers but in HBASE-23956, we set the region server number to 1.

So in fact, we do not kill any region servers which makes the tests useless...
Bug: Replace reflections that are redundant
Status: Open | Priority: Major | Resolution: 
Created: 30/Jun/21 02:10

A number of reflections were used to access (back in the time) new Hadoop APIs that were only available in newer Hadoop versions.

Some of them are no longer needed with the default Hadoop dependency 3.1.2, so they can be removed to avoid the brittle code. Also, makes it possible to determine compile time dependency.
Bug: Remove reflections used in HBaseConfiguration.getPassword()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/21 13:41

HBaseConfiguration.getPassword() uses Hadoop API Configuration.getPassword().  The API was added in Hadoop 2.6.0. Reflection was used to access the API. It's time to remove the reflection and invoke the API directly. (HBase 3.0 as well as 2.x too)
Bug: Introduce a new API to sync the live region server list more effectively
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Jun/21 14:57

In HBASE-25976, we introduced a master based replication tracker to not depend on zk, but the problem is that we will always fetch the whole region server list, from every region server, which will become a performance issue when the cluster has lots of region servers.

So we need to introduce a new API to sync the live region server list. As the region server list will not be changed too much, we should have a way to detect this and not pass the full list.
Bug: Validate nightly builds run on new ci workers hbase11-hbase15
Status: Resolved | Priority: Major | Resolution: Later
Created: 24/Jun/21 20:57

Per slack, asf infra has finished adding in nodes hbase10-hbase15 to ci-hadoop.

make sure they can run nightly.

# Set labels for all these node to "hbase-staging"
# Push a feature branch off of current HEAD that updates the agent labels to use "hbase-staging"
# trigger a bunch of runs. make sure *something* runs on each of the nodes
# Set labels for the nodes to "hbase"
# delete feature branch
Bug: Add donated buildbots for jenkins
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 09/Jun/21 17:51

This issue is for keeping notes on how to add a donated buildbot to our apache build.

My employer donated budget (I badly under-estimated cost but whatever...). This issue is about adding 5 GCP nodes.

There is this page up on apache on donating machines for build https://infra.apache.org/hosting-external-agent.html It got me some of the ways... at least as far as the bit about mailing root@a.o....(nada).

At [~zhangduo]'s encouragement -- he has been this route already adding in the xiaomi donation -- I filed a JIRA after deploying a machine on GCP, INFRA-21973.

I then reached out on slack and the gentleman Gavin MacDonald picked up the task.

He told me run this script on all hosts after making edits (comment out line #39 where we set hostname -- doesn't work):

https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/agent-install.sh

(For more context on the above script and as a good backgrounder, see the nice C* page on how to do this setup: https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.md)

After doing the above, I had to do a visudo on each host to add a line for an infra account to allow passwordless access.
Bug: Master table.jsp pointed at meta throws 500 when no all replicas are online
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/21 21:31
Affects Version/s: 2.3.5

With a replica in a transition state, the UI renders,

{noformat}
HTTP ERROR 500

Problem accessing /table.jsp. Reason:

    Server Error
Caused by:

org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out; 1ms
	at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.waitMetaRegionLocation(MetaTableLocator.java:190)
	at org.apache.hadoop.hbase.generated.master.table_jsp._jspService(table_jsp.java:264)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
{noformat}
Bug: DNS jitter causes hbase client to get stuck
Status: Open | Priority: Major | Resolution: 
Created: 22/Jun/21 07:49
Affects Version/s: 1.2.0

In our product hbase cluster, we occasionally encounter below errors, and stuck hbase a long time. Then hbase requests to this machine will fail forever.
{code:java}
WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:${user@realm} (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]
WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:${user@realm} (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for ${user@realm} to hbase/${ip}@realm
{code}
The main problem is  the trully server principal we generated in KDC is  hbase/*${hostname}*@realm, so we must can't find  hbase/*${ip}*@realm in KDC.

When RpcClientImpl#Connection construct, the field serverPrincial which never changed generated by method InetAddress.getCanonicalHostName() which will return IP when failed to get hostname.

Therefor, once DNS jitter when RpcClientImpl#Connection, this connection will never setup sasl env. And I'm not see connection abandon logic in sasl failed code path.

I think of two solutions to this problem: 
 # Abandon connection when sasl failed. So next request will reconstruct a connection, and will regenerate a new server principal.
 # Refresh serverPrincial field when sasl failed. So next retry will use new server principal.

HBase Version: 1.2.0-cdh5.14.4
Bug: Polish the ReplicationSourceWALReader code for 2.x after HBASE-25596
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/21 08:18

The code are very different for 2.x and 1.x, and the original code for HBASE-25596 is for 1.x, so create this issue to polish the code to make it more suitable for 2.x.
Bug: Revisit synchronization in SyncFuture
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/21 15:51
Affects Version/s: 1.7.0

While working on HBASE-25984, I noticed some weird frames in the flame graphs around monitor entry exit consuming a lot of CPU cycles (see attached images). Noticed that the synchronization there is too coarse grained and sometimes unnecessary. I did a simple patch that switched to a reentrant lock based synchronization with condition variable rather than a busy wait and that showed 70-80% increased throughput in WAL PE. Seems too good to be true.. (more details in the comments).
Bug: ServiceLoader usages should not be tied to Thread Context Classloader
Status: Open | Priority: Major | Resolution: 
Created: 18/Jun/21 10:01

Classes which uses ServiceLoader facility does not specify ClassLoader to use.

For hbase-client 2.4.1 they are (at least):
 * SaslClientAuthenticationProviders
 * MetricRegistries

When hbase libraries are loaded dynamically and Thread Context Classloader is not set SaslClientAuthenticationProviders instantiation failes becuse it can't find default providers.

Some proposals for classloader selection strategy (usage dependent I guess):
 * Use classloader specified in Configuration instance.
 * Use classloader which loaded specific hbase class
 * Combine them: use classloader from Configuration if present and fallback to classloader which loaded specific hbase class

Real world requirement example: currently we migrating from hbase 1 to hbase 2. For better compatibility and smooth migration we try to build abstraction around hbase client libraries and isolate them with custom classloaders. To workaround problems with context classloader we must either wrap all calls to use proper contextclassloader or explicitly trigger initialization of affected classes (SaslClientAuthenticationProviders) wich are marked private under proper context classloader. It would be better if hbase client will take care of it itself.
Bug: Active WAL tailing fails when WAL value compression is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/21 19:50
Affects Version/s: 2.5.0

Depending on which compression codec is used, a short read of the compressed bytes can cause catastrophic errors that confuse the WAL reader. This problem can manifest when the reader is actively tailing the WAL for replication. The input stream's available() method sometimes lies so cannot be relied upon. To avoid these issues, when WAL value compression is enabled, ensure all bytes of the compressed value are read before submitting the payload to the decompressor.
Bug: FanOutOneBlockAsyncDFSOutput using shaded protobuf in hdfs 3.3+
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 16:54

The parent added some fancy dancing to make it so on hadoop-3.3.0+ we'd use hadoops shaded protobuf rather than the non-relocated protobuf. When hdfs 3.3, the 'trick' is not working so we continue to use the unshaded protobuf. Fix is trivial.

Found this testing the 3.3.1RC3. Hard to see because whether we use shaded or unshaded is at DEBUG level.  If you set DEBUG level and run TestFanOutOneBlockAsyncDFSOutput with hdfs 3.3.1 RC candidate in place you'll see it uses the unshaded protobuf.
Bug: Make excluded SSL cipher suites configurable for all Web UIs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/21 13:49
Affects Version/s: 2.2.7

When starting a jetty http server, one can explicitly exclude certain (unsecure) SSL cipher suites. This can be especially important, when the HBase cluster needs to be compliant with security regulations (e.g. FIPS).

Currently it is possible to set the excluded ciphers for the ThriftServer ("hbase.thrift.ssl.exclude.cipher.suites") or for the RestServer ("hbase.rest.ssl.exclude.cipher.suites"), but one can not configure it for the regular InfoServer started by e.g. the master or region servers.

In this commit I want to introduce a new configuration "ssl.server.exclude.cipher.list" to configure the excluded cipher suites for the http server started by the InfoServer. This parameter has the same name and will work in the same way, as it was already implemented in hadoop (e.g. for hdfs/yarn). See: HADOOP-12668, HADOOP-14341
Bug: Seeing a spike in uncleanlyClosedWALs metric.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/21 14:41
Affects Version/s: 1.7.0

Getting the following log line in all of our production clusters when WALEntryStream is dequeuing WAL file.

{noformat}
 2021-05-02 04:01:30,437 DEBUG [04901996] regionserver.WALEntryStream - Reached the end of WAL file hdfs://<wal-file-name>. It was not closed cleanly, so we did not parse 8 bytes of data. This is normally ok.
{noformat}
The 8 bytes are usually the trailer serialized size (SIZE_OF_INT (4bytes) + "LAWP" (4 bytes) = 8 bytes)

While dequeue'ing the WAL file from WALEntryStream, we reset the reader here.
[WALEntryStream|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L199-L221]

{code:java}
  private void tryAdvanceEntry() throws IOException {
    if (checkReader()) {
      readNextEntryAndSetPosition();
      if (currentEntry == null) { // no more entries in this log file - see if log was rolled
        if (logQueue.getQueue(walGroupId).size() > 1) { // log was rolled
          // Before dequeueing, we should always get one more attempt at reading.
          // This is in case more entries came in after we opened the reader,
          // and a new log was enqueued while we were reading. See HBASE-6758
          resetReader(); ---> HERE
          readNextEntryAndSetPosition();
          if (currentEntry == null) {
            if (checkAllBytesParsed()) { // now we're certain we're done with this log file
              dequeueCurrentLog();
              if (openNextLog()) {
                readNextEntryAndSetPosition();
              }
            }
          }
        } // no other logs, we've simply hit the end of the current open log. Do nothing
      }
    }
    // do nothing if we don't have a WAL Reader (e.g. if there's no logs in queue)
  }
{code}

In resetReader, we call the following methods, WALEntryStream#resetReader  ---->  ProtobufLogReader#reset ---> ProtobufLogReader#initInternal.
In ProtobufLogReader#initInternal, we try to create the whole reader object from scratch to see if any new data has been written.
We reset all the fields of ProtobufLogReader except for ReaderBase#fileLength.
We calculate whether trailer is present or not depending on fileLength.
Bug: HBaseCluster should be marked as IA.Public
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/21 14:10

Its sub class MiniHBaseCluster is marked as IA.Public, so it should also be IA.Public.

And in general, HBaseCluster just provides some abstract method so I think it is OK to expose it as IA.Public.

But for MiniHBaseCluster, it exposes several internal stuff such as RegionServerThread so maybe it should not be marked as IA.Public.