av9ash/GitBugs huggingface dataset:
Bug: Backport HBASE-25591 "Upgrade opentelemetry to 0.17.1" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:37
Affects Version/s: 2.5.0

10/17 commits of HBASE-22120, original commit f6ff519dd0c7fe0e3ae3c175eefee27a26a065a4
Bug: Backport HBASE-25535 "Set span kind to CLIENT in AbstractRpcClient" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:37
Affects Version/s: 2.5.0

9/17 commits of HBASE-22120, original commit [{{bb8c496}}|https://github.com/apache/hbase/commit/bb8c4967f8ce2c89ebaf1ddc5d8a1bf55f1e20d3]
Bug: Backport HBASE-25484 "Add trace support for WAL sync" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:37
Affects Version/s: 2.5.0

8/17 commits of HBASE-22120, original commit 2be2c63f0d3917a243b74af9754cbfc805b858d1
Bug: Backport HBASE-25455 "Add trace support for HRegion read/write operation" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:36
Affects Version/s: 2.5.0

7/17 commits of HBASE-22120, original commit 03e12bfa4ad62ecc6eee6a2c68d431bea2d5c473
Bug: Backport HBASE-25481 "Add host and port attribute when tracing rpc call at client side" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:36
Affects Version/s: 2.5.0

6/17 commits of HBASE-22120, original commit ae2c62ffaad5ba4c976b0a79c10a365edf2844fd
Bug: Backport HBASE-25454 "Add trace support for connection registry" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:36
Affects Version/s: 2.5.0

5/17 commits of HBASE-22120, original commit [dcb78bd4bda4a4ae13d863df8aec266031e5bc93|https://github.com/apache/hbase/commit/dcb78bd4bda4a4ae13d863df8aec266031e5bc93] and merged conflicts after rebasing on HBASE-26150 (branch-2 commit [620806e3fb27947fa3b36f24921060e328f4b39d|https://github.com/apache/hbase/commit/620806e3fb27947fa3b36f24921060e328f4b39d]) with commit [63d4970de451bf234f2ddbda949995b1420e525b|https://github.com/apache/hbase/commit/63d4970de451bf234f2ddbda949995b1420e525b]
Bug: Backport HBASE-23898 "Add trace support for simple apis in async client" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:36
Affects Version/s: 2.5.0

4/17 commits of HBASE-22120, original commit https://github.com/apache/hbase/commit/805b2ae2ad0f6325515d46043ff01e4e2c7a9f59
Bug: Backport HBASE-25424 "Find a way to config OpenTelemetry tracing without directly depending on opentelemetry-sdk" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:35

[3/17 commits of HBASE-22120|https://github.com/apache/hbase/pull/2901/commits], original commit of HBASE-25424 was [https://github.com/apache/hbase/commit/57960fa8fa7228d65b1a4adc8e9b5b1a8158824d]
Bug: Backport HBASE-25373 "Remove HTrace completely in code base and try to make use of OpenTelemetry" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:34
Affects Version/s: 2.3.5

[1/17 commits of HBASE-22120|https://github.com/apache/hbase/pull/2901/commits], backporting HBASE-25373 Remove HTrace completely in code base and try to make use of OpenTelemetry

We will need to remove some unimplemented tracing in AsyncRequestFutureImpl#getNewMultiActionRunnable that master branch does not have it and also upstream in master does not implement TraceRunnable with OpenTelemetry plugin

the original commit on master is https://github.com/apache/hbase/commit/302d9ea8b888762a5a20a5ba5c2be7bc239afaef
Bug: TableMRUtil#initCredentialsForCluster should use specified conf for UserProvider
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/21 09:00

{code:java}
  /**
   * Obtain an authentication token, for the specified cluster, on behalf of the current user
   * and add it to the credentials for the given map reduce job.
   *
   * @param job The job that requires the permission.
   * @param conf The configuration to use in connecting to the peer cluster
   * @throws IOException When the authentication token cannot be obtained.
   */
  public static void initCredentialsForCluster(Job job, Configuration conf)
      throws IOException {
    UserProvider userProvider = UserProvider.instantiate(job.getConfiguration());
    if (userProvider.isHBaseSecurityEnabled()) {
      try {
        Connection peerConn = ConnectionFactory.createConnection(conf);
        try {
          TokenUtil.addTokenForJob(peerConn, userProvider.getCurrent(), job);
        } finally {
          peerConn.close();
        }
      } catch (InterruptedException e) {
        LOG.info("Interrupted obtaining user authentication token");
        Thread.interrupted();
      }
    }
  }
{code}

TableMapReduceUtil#initCredentialsForCluster uses job's configuration instead of conf in argument for UserProvider.
This causes that token is not obtained for secure cluster in case of job's configuration contains hbase.security.authentication=simple.

Because userProvider.isHBaseSecurityEnabled() checks that hbase.security.authentication is kerberos.

If conf is configured for secure cluster, hbase.security.authentication is kerberos and it's natural to use conf for UserProvider instead of job's configuration in this method.
Bug: TestWALRecordReader#testWALRecordReaderActiveArchiveTolerance doesn't read archived WAL file. 
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/21 13:05

TestWALRecordReader#testWALRecordReaderActiveArchiveTolerance is testing the following scenario.
1. Create a new WAL file
2. Write 2 KVs to WAL file.
3. Close the WAL file.
4. Instantiate WALInputFormat#WALKeyRecordReader with the WAL created in step 1.
5. Read the first KV.
6. Archive the WAL file in oldWALs directory via rename operation.
7. Read the second KV. This will test that WALKeyRecordReader will encounter FNFE since the WAL file is not longer present in the original location and it will handle the FNFE by opening the WAL file from archived location.

In step#7, the test is expecting that it will encounter FNFE and it will open a new reader but in reality, it is not encountering FNFE. I think the reason is, during rename operation, HDFS just changes the internal metadata for the file name. The InodeID, hdfs blocks and block locations remains the same. While reading the first KV, DFSInputStream caches all the HDFS blocks and location data so it doesn't have to go to namenode to re-resolve the file name.

Cc [~zhangduo]
Bug: ServerTestHBaseCluster interface for testing coprocs
Status: Open | Priority: Major | Resolution: 
Created: 22/Jul/21 19:48
Affects Version/s: 3.0.0-alpha-1

The new TestingHBaseCluster introduced in HBASE-26080 provides a clean way for downstream developers writing features using the HBase client APIs to test their code. Its inner minicluster class, HBaseTestingUtil, was left unexposed with an interface audience of Phoenix, because coprocessors might need access to the internals of HBase itself to be tested.

Occasionally, a developer outside of HBase and Phoenix might need the same access to the internals. One way to do this would be to introduce a new interface, ServerTestHBaseCluster, that extends TestingHBaseCluster and exposes the HBaseTestingUtil, with an interface audience of COPROC, REPLICATION (for custom endpoints), PHOENIX.
Bug: Return incorrect result for a scan rpc call when readType switch from pread to stream
Status: Open | Priority: Major | Resolution: 
Created: 30/Aug/21 12:05

Scan's default readType is PREAD, and there is a readType transform from PREAD to STREAM when a long scan is executed. E.G. I ask for result of 'count table', and now it will return immediately before the RPC get real data because of this transform, so the result returned maybe incorrect. Although, we can explicitly use STREAM instead of PREAD to avoid this transform. I am still confused, how can we judge whether transform is called? How can I trust my result is real when we allow transform occur?
Bug: We could start RegionServerTracker before becoming active master
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/21 13:02

The direct reason is that, after decoupling, HMaster does not extends HRegionServer, so we do not have RegionServerAddressTracker now, so for backup masters we do not have a way to get all the region server which means it can not serve as connection registry any more, so we need to find a way to let backup masters to also have a way to get the region server list.

But in general, since we already have a class in HMaster which is used to track region servers, we'd better reuse it. And after this change, the backup masters could also be used as the fallback servers for getting region server list in HBASE-26220, which could also reduce the load of active master.

And this could also speed up the active master start process as we could save one zookeeper scan when becoming active master.
Bug: VerifyReplication should obtain token for peerQuorumAddress too
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/21 08:42
Affects Version/s: 2.3.6

VerifyReplication accepts peerQuorumAddress itself via command parameter instead of getting it from peerid of source cluster.

[https://github.com/apache/hbase/commit/b322d0a3e552dc228893408161fd3fb20f6b8bf1#diff-0307194efcf6a3ad4a4d73bd4b6ef34a9be5c436c8e970ca97fa146b2f0aa486]

https://issues.apache.org/jira/browse/HBASE-21201

 {code:java}
    if (peerId != null) {
      assert peerConfigPair != null;
      Configuration peerClusterConf = peerConfigPair.getSecond();
      // Obtain the auth token from peer cluster
      TableMapReduceUtil.initCredentialsForCluster(job, peerClusterConf);
    }
{code}

 In this patch, credential for job is obtained when peerid is provided only.
Thus we cannot get the benefit of HBASE-21201 for secure hbase cluster as a destination.


{code:java}
hbase VerifyReplication \
  -D verifyrep.peer.hbase.regionserver.kerberos.principal=secure-hbase-b/_HOST@EXAMPLE.COM \
  -D verifyrep.peer.hbase.master.kerberos.principal=secure-hbase-b/_HOST@EXAMPLE.COM \
  ... \
  secure-hbase-b-1.example.com,secure-hbase-b-2.example.com,secure-hbase-b-3.example.com:2181:/hbase-b \
  table
{code}

Assume this execution, VerifyReplication should obtain token from secure-hbase-b-1.example.com,secure-hbase-b-2.example.com,secure-hbase-b-3.example.com:2181:/hbase-b using hbase.regionserver.kerberos.principal=secure-hbase-b/_HOST@EXAMPLE.COM and hbase.master.kerberos.principal=secure-hbase-b/_HOST@EXAMPLE.COM, so that VerifyReplication mapper can scan from the secure cluster B.
Bug: Introduce a initial refresh interval for RpcConnectionRegistry
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/21 12:42

So we can get the new list soon once we connect to the cluster.

As end users could configure any nodes in a cluster as the initial bootstrap nodes, it is possible that different end users will configure the same machine which makes the machine over load. So we should have a shorter delay for the initial refresh, to let users quickly switch to the bootstrap nodes we want them to connect to.
Bug: Make RpcConnection with transient/non-Singleton SaslClientAuthenticationProviders with repsecting the passing configuration object 
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/21 17:02
Affects Version/s: 2.4.5

When a RPC connection was created from any table connection (e.g. via {{ConnectionFactory.createConnection(conf)}}), the [SaslClientAuthenticationProviders|(https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcConnection.java#L99-L101] that matches different authentication selector per-user/TOKEN as the [singleton object|https://github.com/apache/hbase/blob/735bcf85e9b59a71981babe2f5da51978d61d8d3/hbase-client/src/main/java/org/apache/hadoop/hbase/security/provider/SaslClientAuthenticationProviders.java#L72-L83] and reuses for any followup connection creation. 

If user want to create another connection with a different configuration key that the authentication selector may use, e.g. [if one want to switch between secure and non-secure connection|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/provider/BuiltInProviderSelector.java#L109-L111], it cannot be done because of the cached singleton instance of AuthenticationProviders. 

the goal of this JIRA is to allow the RpcConnection and AuthenticationProviders align with whatever passing configuration object
Bug: Refactor AnnotationReadingPriorityFunction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/21 07:50

Since master does not carry regions any more, MasterAnnotationReadingPriorityFunction does not need run the logic to find out scanner id or region id to determine the priority.

Let's introduce a RSAnnotationReadingPriorityFunction, move the special logic from AnnotationReadingPriorityFunction to it.
Bug: Allow disabling refresh of connection registry endpoint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/21 13:19

One possible deployment in production is to use something like a lvs in front of all the region servers to act as a LB, so client just need to connect to the lvs IP instead of go to the region server directly to get registry information.

For this scenario we do not need to refresh the endpoints any more.

The simplest way is to set the refresh interval to -1.
Bug: Report flaky jenkins job should have the ability to submit flaky tests manually
Status: Open | Priority: Major | Resolution: 
Created: 25/Aug/21 02:28

Recently our pre commit result is very unstable due to some flaky tests, such as TestReplicationDroppedTable, but it never fails in nightly builds so we never include it in the flaky list.

I think our find flaky jenkins job should allow submitting tests manually.
Bug: Support RegionCoprocessor on CompactionServer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/21 07:39

We have two kinds of RegionCoprocessor: CoreCoprocessor or not.

1.The CoreCoprocessor(AccessController, TokenProvider, SecureBulkLoadEndpoint...),  They have RegionEnvironmentForCoreCoprocessors which expose RegionServerServices. And they involve authorization, permission, etc, We don't need to support these CoreCoprocessor on compactionServer
 2. Not CoreCoprocessor: They have RegionEnvironment which do not expose RegionServerServices, only offer Connection/ OnlineRegions, We can propose RegionEnvironmentOnCS, offer Connection, but the OnlineRegions may leave to discuss
Bug: JVM crash when displaying RPC params by MonitoredRPCHandler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/21 12:47
Affects Version/s: 2.0.0

It is the same problem as HBASE-25981.
The reason here is MonitoredRPCHandlerImpl uses the shallow copy of call param to displaying RPC info, but the param contains offheap request data. Thought when calling toMap() there is state checking for the monitored RPC, only RUNNING RPCs can read and display params, the state is ENUM type and like a constant in clone().
So there exists circumstances that the state in the clone MonitoredRPCHandler is RUNNING, but the param still refers to the cloned MonitoredRPCHandler, whose state changed to be not RUNNING and param referred offheap ByteBuffer is released.
Bug: Add UT for masterless mode
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Aug/21 10:53

It is not used in code any more and there is no UT to make sure it works.

Let's just remove it as it is only designed to be used in tests.

==== Update ====

This is used in hbase-connector, so we'd better add a UT in hbase to confirm that it still works.
Bug: Undo 'HBASE-25165 Change 'State time' in UI so sorts (#2508)' in favor of HBASE-24652
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/21 20:39

The below change by me does not actually work and I found an old issue that does the proper job that was neglected. I'm undoing the below in favor of HBASE-24652.

 

kalashnikov:hbase.apache.git stack$ git show d07d181ea4a9da316659bb21fd4fffc979b5f77a
commit d07d181ea4a9da316659bb21fd4fffc979b5f77a
Author: Michael Stack <saintstack@users.noreply.github.com>
Date: Thu Oct 8 09:10:30 2020 -0700

HBASE-25165 Change 'State time' in UI so sorts (#2508)

Display startcode in iso8601.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Bug: Support configuration override for remote cluster of HFileOutputFormat locality sensitive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/21 02:30
Affects Version/s: 1.8.0

We introduced support to generate hfile with good locality for a remote cluster even in HBASE-25608.

I realized we need to override other configurations for the remote cluster in addition to the zookeeper cluster key.

For example, read from a non-secure cluster and write hfiles for a secure cluster.
 In this case, we use TableInputFormat for non-secure cluster with hbase.security.authentication=simple in job configuration.
 So HFileOutputFormat failed to connect to remote secure cluster because requires hbase.security.authentication=kerberos in job conf.

 

Thus let's introduce configuration override for remote-cluster-aware HFileOutputFormat locality-sensitive feature.

 

-Another example is to read from a secure cluster (A) and write hfiles for another secure cluster (B) and we use different principal for each cluster.-
 -For instance, we use cluster-a/_HOST@EXAMPLE.COM for A and cluster-b/_HOST@EXAMPLE.COM for B.-
 -Then we need to override MASTER_KRB_PRINCIPAL and REGIONSERVER_KRB_PRINCIPAL using cluster-b/_HOST@EXAMPLE.COM to connect cluster B.-

^ This is not truth, we use token based digest auth in mapper/reducer, so principal difference for kerberos should be fine
Bug: Annotate shaded generated protobuf as InterfaceAudience.Private
Status: Open | Priority: Major | Resolution: 
Created: 12/Aug/21 03:39

Annotate generated shaded protobufs as InterfaceAudience.Private. It might not be able to add the annotation to each class; at a minimum update the doc on our story around shaded internal protobufs.

See the prompting mailing list discussion here: [https://lists.apache.org/thread.html/r9e6eb11106727d245f6eb2a5023823901637971d6ed0f0aedaf8d149%40%3Cdev.hbase.apache.org%3E]

So far the consensus has it that the shaded generated protobuf should be made IA.Private.  Will wait on it to settle.
Bug: Add owner of replication queue for ReplicationQueueInfo
Status: Resolved | Priority: Major | Resolution: Done
Created: 13/Jul/21 08:59
Affects Version/s: 3.0.0-alpha-1

The current ReplicationQueueInfo only has queueId, which is not enough to distinguish queues in ReplicationServer,  so we need to add the RS holding the queue for ReplicationQueueInfo.
Bug: Implement #toString for RowMutations
Status: Open | Priority: Major | Resolution: 
Created: 11/Aug/21 17:27
Affects Version/s: 2.4.5

We annoyingly discovered recently the absence of a meaningful {{toString}} method on {{o.a.h.h.client.RowMutations}}. We should add it. More ambitious: we should also perform an audit of the user data objects in this package to ensure that they all have basic POJO methods like {{toString}}, {{equals}}, and {{hashCode}}.
Bug: Let region server also carry ClientMetaService
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/21 14:58

Usually region server will be deployed on machines which are more powerful than masters, so it will be good to let region servers carries more load.
h2. Background

This is preliminary work needed for one of the [splittable meta designs|https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153]. We want to hide the implementation of ROOT from the client side, by adding new methods in ClientMetaService for locating meta Regions.


One of the concerns is that the original implementation of the Master Registery over in HBASE-18095 puts master inline in the normal read/write path, which is not always a good choice. This issue aims to move the ConnectionRegistry off of the masters.

In this issue, we let the region server implement the ClientMetaService interface, so the client could also get the cluster id, active master, meta region location, etc. from RegionServers (and not just Masters as it is currently).


We introduce a new method getBootstrapNodes in ClientMetaService, for refreshing the bootstrap nodes periodically or on error. The new RpcConnectionRegistry will use this method to refresh the bootstrap nodes, while the old MasterRegistry will use the getMasters method to refresh the ‘bootstrap’ nodes.

The getBootstrapNodes method will return all the region servers, so after the first refreshing, the client will go to region servers for later rpc calls. But since masters and region servers both implement the ClientMetaService interface, it is free for the client to configure master as the initial bootstrap nodes.
Bug: Size of the Result object while querying huge data from HBASE table
Status: Open | Priority: Major | Resolution: 
Created: 09/Aug/21 06:53
Affects Version/s: 1.1.13

 
I am trying to query hbase table with rowkeys. We have the following structure
 * index table which has rowkeys of the actual table
 * actual table which contains json data in compressed format.

When i am trying to query hbase, i have to scan first index table for rowkeys using scan with some filters which will results to byte array.(row keys). Once we obtained rowkeys, we are invoking listofGets() in Table object. Once obtained we are iterating the object and prepare a list which contains compressed json objects. Here we are not sure about the size and number of the objects. In case of number of objects is huge we may result in OOM. Do we have any options to return Iterator or buffering the results so that we can avoid OOM.
 {{for (byte[] rowkey : indexTableOutput)
{    Get get = new Get(rowkey).addFamily(Bytes.toBytes(columnFamilty)).setMaxVersions(MAX_VERSIONS);
    listOfget.add(get);
}}}
The above piece of code which is used to retrieve the keys from index table.
 {{TableName tableName = TableName.valueOf("table1");Table tableObj = conn.getTable(tableName);
Result[] results = tableObj.get(listOfget);}}
From the above piece of code we have few queries. Any help would be appreciated.
 * If we have a huge number of data, Result[] will contain all the results?
 * How to return a iterator kind of object so that we can leave it to consumer because keeping all the data and doing processing will result in OOM
 * Any other options to return a limited data so that consumer do processing and continue

I could find a resultscanner is returning for scan objects. But couldn't find any other options for list of Get's. Here we know the exact keys from index table.