av9ash/GitBugs huggingface dataset:
Bug: Remove RpcSchedulerFactory#create(Configuration, PriorityFunction)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Oct/20 20:03
Affects Version/s: 3.0.0-alpha-1

The {{create(Configuration, PriorityFunction)}} method in {{RpcSchedulerFactory}} was deprecated a while ago. As it is not used anymore, even in implementing classes, we should be able to safely remove the method.
Bug: Allow hbase-connector to be used with Apache Spark 3.0 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Nov/20 14:08

Currently hbase-spark connector only works with Spark 2.x.

Apache Spark 3.0 has been relesead in June 2020.

This addresses the changes needed to run the connector with Spark 3.0 and also to be able to compile the connector using Spark 3.0 as a dependency.
Bug: Instantiate BucketCache before RS creates a their ephemeral node when rolling-upgrade
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 30/Oct/20 06:12
Affects Version/s: 1.4.13

We observed many clients couldn't get information on region locations for tens of seconds during rolling-upgrade from 1.2.x to 1.4.x, and all requests to regions moved by graceful restart failed.
 
The reason is that 
# Since HBASE-17931, system tables are assigned to RS with highest version
# Since HBASE-12034, bucket cache initialization process has moved from RS instantiation to RS initialization process after reporting to master, moreover an ephemeral node for RS is created before bucket cache creation.
# when using offheap bucketcache, it takes too much time to allocate memory for it (18 seconds for 31GB in our case) [https://github.com/apache/hbase/blob/branch-1.4/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferArray.java#L52-L72]
# Once ephemeral nodes created, a master try to move system regions to RS with highest version when first RS restart of whole rolling-restart process. but, by 3) the RS is not ready for serving system regions yet. moving system regions keep failing until 3) is finished.

 
I think this could happen only in branch-1, because an ephemeral node is created after creating block caches in hbase 2.x. there is no need to create block caches after ephemeral node creation at all.
 
I verified this issue could be resolved by just changing their creation order.
Bug: Profiling servlet ensure output directory exists and is writable
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 02/Oct/20 20:02
Affects Version/s: 2.3.0

When the profiling servlet runs async-profiler when the output path ({{/tmp/prof-output}}) doesn't exist or isn't writable by the {{hbase}} process, it will sit in a refresh loop indefinitely.  The servlet should check that the output path is writable, or even make an attempt to create the output directory, and inform the user of the situation, rather than attempting and silently failing to generate the flame graph.
Bug: Redundant Reference file in bottom region of split
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Nov/20 05:45
Affects Version/s: 2.0.0

When we split a region ranges from (,), the bottom region should contain keys of(,split key), and the top region should contain keys of [split key, ).

Currently, if we do the following operations:
 # put rowkeys 100,101,102,103,104,105 to a table, and flush the memstore to make a hfile with rowkeys 100,101,102,103,104,105;
 # put rowkeys 200,201,202,203,204,205 to the table, and flush the memstore to make a hfile with rowkeys 200,201,202,203,204,205;
 # split the table region, using split key 200;
 # then the bottom region will has two Reference files, while the top region only has one.

But we expect the bottom region has only one Reference file as the the top region.

That's because when generating Reference files in child region,  the bottom region used the `PrivateCellUtil.createLastOnRow(splitRow)` cell to compare to first keys in the hfiles, while the top region used `PrivateCellUtil.createFirstOnRow(splitRow)` cell to compare to last keys in the hfiles.

`LastOnRow(splitRow)` means the maximum row generated by the split row, while `FirstOnRow(splitRow)` means the minimus row generated by the split row. The split row should be in the top region. And we should use `FirstOnRow(splitRow)` compare to hfile first and last keys in both bottom and top region. 

Though the redundant Reference file will not be read by the bottom region, the compaction of the redundant Reference file will result in empty file if only this redundant Reference file participates in a compaction.
Bug: Change 'State time' in UI so sorts
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Oct/20 23:28

Here is a minor issue.

I had an issue w/ crashing servers. The servers were auto-restarted on crash.

To find the crashing servers, I was sorting on the 'Start time' column in the Master UI. This basically worked but the sort is unreliable as the date we display starts with days-of-the-week.

This issue is about moving to display start time in iso8601 which is sortable (and occupies less real estate). Let me add some images.
Bug: Add option to toggle CACHE_BLOCKS in count.rb
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Nov/20 21:03

A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is to edit table.rb so that the `count` shell command will not instruct RegionServers to not cache any data blocks. This is a quick+dirty way to force a table to be loaded into block cache (i.e. for performance testing).

We can easily add another option to avoid having to edit the ruby files.
Bug: Improve SizeCachedKV variants initialization
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Oct/20 13:43

Currently in SizeCachedKV we get the rowlength and Key length from the buffers. This can be optimized because we can pass the keylen and row len while actually creating the cell while reading the cell from the block.  Some times we see that the SizeCachedKV takes the max width in a flame graph - considering the fact we also do a sanity check on the created KV.
Bug: Make REST Client connection and socket time out configurable
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Dec/20 18:24
Affects Version/s: 2.3.4

Currently there is no way to configure connection and socket timeout for REST client,

https://github.com/apache/hbase/blob/7d0a687e5798a2f4ca3190b409169f7e17a75b34/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/Client.java#L102
Bug: NPE while running balance_rsgroup  if any split region is present.
Status: Open | Priority: Minor | Resolution: 
Created: 18/Nov/20 05:30
Affects Version/s: 2.2.3

 NPE is thrown in below scenario:
1: Create rsgroup pgroup
2: Add two RS to pgroup
3: create two table with 15 region each
4: kill 1 RS and start again
5: disable t1
6: run balancer and balance_rsgroup 'pgroup' and observe it is working fine
7: now split one of the region of t2
8: run balance_rsgroup 'pgroup' and observe NPE:

 

ERROR: java.io.IOException
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:517)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
Caused by: java.lang.NullPointerException
 at java.util.TreeMap.put(TreeMap.java:563)
 at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.correctAssignments(RSGroupBasedLoadBalancer.java:347)
 at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.balanceCluster(RSGroupBasedLoadBalancer.java:140)
 at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:531)
 at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:302)
 at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:16306)
 at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:1023)
 at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:458)
Bug: Redo the getMidPoint() in HFileWriterImpl to get rid of the double comparison process
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Dec/20 03:15

There is a TODO like this "TODO: Redo so only a single pass over the arrays rather than one to  compare and then a second composing midpoint." in getMidpoint()  of class ​HFileWriteImpl​
  
 The old logic compares the left byte array and the right byte array twice： 
 # A comparison is performed before composing MinimumMidpointArray
 # During composing of MinimumMidpointArray, bytes were comparing again

 
 My optimization  combines them into one
Bug: Dump region hashes in logs for the regions that are stuck in transition for more than a configured amount of time
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Nov/20 19:30

We have metrics for number of RITs as well as number of RITs above a certain threshold, but we don't have any way of keeping track of the region hashes of those RITs. It would be beneficial to emit those region hashes as a metric, as well as log them, so that we don't accidentally lose this information for debugging the RIT at a later tiime.
Bug: Procedures table Id under master web UI gets word break to single character
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Dec/20 17:55

Under master web UI -> Procedures & Locks, *Procedures table* columns Errors and Parameters occupy larger width making Id and others word break more. Specially on Id where it breaks to single characters.

 

Planning to make word break only on Errors and Parameters columns leaving the others normal in this table. So that we dont have to apply any changes on width/padding to get this fixed.
Bug: Refactor MultiVersionConcurrencyControl: replace readWaiters.notifyAll() with WriteEntry.notify()
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 24/Dec/20 03:16

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java#L217] 

here the notifyAll() wake up all thread, even the read point no big enough, bring extra thread switch

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java#L143] 

WriteEntry object can be thread local to reuse
Bug: [github]rename HBASE-18070-ROOT_hbase:meta_Region_Replicas.pdf
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 21/Nov/20 06:49

use git pull to obtain an exception, because filename has ':'

!image-2020-11-21-14-48-23-794.png!

workaround:git config core.protectNTFS false

[~stack] i think we can ranme filename...

.
Bug: Configure where IntegrationTestImportTsv generates HFiles
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Nov/20 08:31

Currently IntegrationTestImportTsv is generating HFiles under the working directory of the hdfs user executing the tool, before bulkloading it into HBase.

Assuming you encrypt the HBase root directory within HDFS (using HDFS Transparent Encryption), you can bulkload HFiles only if they sit in the same encryption zone in HDFS as the HBase root directory itself.

When IntegrationTestImportTsv is executed against a real distributed cluster and the working directory of the current user (e.g. /user/hbase) is not in the same encryption zone as the HBase root directory (e.g. /hbase/data) then you will get an exception:

 
{code:java}
ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial failure due to IO when attempting to load d :
hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
org.apache.hadoop.ipc.RemoteException(java.io.IOException): /user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc can't be moved into an encryption zone.
{code}
 

In this ticket I make it configurable where the IntegrationTestImportTsv generates the HFiles. From now, one can execute this integration test on HDFS Transparent Encryption enabled clusters, like:
{code:java}
./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv -D IntegrationTestImportTsv.generatedHFileFolder=/<my hbase encryption zone path>/testdata{code}
Bug: Followup jira to address the client handling issue when chaning from meta replica to non-meta-replica at the server side.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Nov/20 22:10

[https://github.com/apache/hbase/pull/2643]

 

{quote}

With my operator hat on, I'd assume that LOAD_BALANCE with 1 replica count works like no read replicas configured (logic wise at-least, even though the code paths are different).
{quote}If the server side does not support meta replica, the client side cannot be configured to support this mode
{quote}
Since clients are usually long running (meaning we may not be able to restart client or they using cached HBase connection) and meta replica count can be altered on the service side on the fly, I'd expect client to work across these changes without any configuration changes. WDYT?

{quote}
Bug: Make retry pause time configurable for regionserver short operation RPC (reportRegionStateTransition/reportProcedureDone)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Dec/20 14:27

Currently initPauseTime is hard coded as 1000,
 https://github.com/apache/hbase/blob/c853c99b209cb2dfefb8d15595d9a9a5469e24fa/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L2440

But in case of ServerNotRunningYetException/PleaseHoldException/CallQueueTooBigException RegionServer may wait for a long time (In our case it was ~7 min).

Currently we calculate pause time based on the number of retries and RETRY_BACKOFF. So in a failure case RS will wait max 200 sec and wait will be always 200 sec if retries further goes beyond 13.

We should make it configurable.
Bug: Flaky Test
Status: Open | Priority: Minor | Resolution: 
Created: 18/Dec/20 14:20
Affects Version/s: 2.3.2

Hello, we are a group of researchers interested in understanding methods to reduce costs associated with Test Flakiness [1]. We developed an approach to detect flaky tests, dubbed Shaker [2]. Shaker spawns stressor tasks in the running environment (e.g., tasks that consume lots of cpu or memory) to identify if perturbations can make tests fail.

We ran Shaker on hbase project and found the following likely flaky tests:
 * org.apache.hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile.testRetrieveFromFile - failed 5 times in 12 reruns.

org.apache.hadoop.hbase.io.hfile.TestLruBlockCache.testBackgroundEvictionThread - failed 4 times in 12 reruns.

org.apache.hadoop.hbase.master.normalizer.TestRegionNormalizerWorkQueue.testTake - failed 1 time in 12 reruns.

[Denini|https://github.com/denini08], [Leopoldo|https://github.com/leopoldomt] and [Marcelo|https://github.com/damorim].





[1] [https://hackernoon.com/flaky-tests-a-war-that-never-ends-9aa32fdef359]

 

[2] D. Silva, L. Teixeira and M. d’Amorim, "Shake It! Detecting Flaky Tests Caused by Concurrency with Shaker," _2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)_, Adelaide, Australia, 2020, pp. 301-311, [https://ieeexplore.ieee.org/document/9240694/].
Bug: The log in move_servers_rsgroup is incorrect
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Dec/20 04:33
Affects Version/s: 2.4.0

Assuming that server1 belongs to the default group, execute the command:

 
{code:java}
 move_servers_rsgroup 'test',['server1:16020']
{code}
We can see log:

 
{code:java}
Moving 10 region(s) to group test, current retry=0
......
All regions from server(s) [server1,16020,1607067542905] moved to target group test.
{code}
The target group should be the source group in log: test -> default.

 

https://github.com/apache/hbase/blob/7d0a687e5798a2f4ca3190b409169f7e17a75b34/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java#L1027
Bug: Quoting in hbase-vote script fails build with empty ""
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Dec/20 00:59
Affects Version/s: 2.4.1

HBASE-25085 added support for maven options to be passed into the hbase-vote script. However, with the quoting on my machine it fails if I do not pass in any maven opts:

      * Signature: ok
        * Checksum : ok
        * Rat check (1.8.0_92): failed
         - mvn clean apache-rat:check ""
        * Built from source (1.8.0_92): failed
         - mvn clean install -DskipTests ""
        * Unit tests pass (1.8.0_92): failed
         - mvn package -P runAllTests "" -Dsurefire.rerunFailingTestsCount=3

Notice the "" and the build failed due to unknown lifecycle ""
Bug: Add builder method to create Tags.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Nov/20 17:18

In PHOENIX-6213 phoenix project is using Cell Tag feature to add some metadata for delete mutations. We are adding Cell Tags in co-processor but we need some util methods available in +PrivateCellUtil+ class. 

Below are the methods we need in phoenix.

1. +PrivateCellUtil#createCell(Cell cell, List<Tag> tags)+ method has an api which will accept an existing Cell and list of tags to create a new cell. 

But RawCellBuilder has a builder method which doesn't have any method which accepts a cell. I need to explicitly convert my input cell by extracting all fields and use the builder methods (like setRow, setseFamily, etc) and then use the build method.

 

2. +PrivateCellUtil.getTags(Cell cell)+ returns a list of existing tags which I want to use and add a new tag.

But RawCell#getTags() returns Iterator<Tag>  which then I have to iterate over them and depending on whether they are byte buffer backed or array backed, I need to convert them to List since RawCellBuilder#setTags accepts List of Tags. We are already doing this conversion in PrivateCellUtil#getTags method.

All these conversion utility methods needs to be duplicated in phoenix project also.

 

Is it reasonable to make PrivateCellUtil method LimitedPrivate with HBaseInterfaceAudience as COPROC ?
Bug: Remove processingServers in DeadServer as we can get this information by Procedure of master
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Nov/20 09:57

We can directly get the servers in processing through proceduces in master as long as proceduce is a instance of `ServerCrashProcedure`

*Our final goal is to remove class `DeadServer`*, the next step is to remove the deadServers attributes in DeadServer and then we can freely delete the class
Bug: Embedded zookeeper server not clean up the old data
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Oct/20 08:35

The size of zookeeper data dir keeps increasing. I checked the source code and found the zookeeper data cleanup manager is not started by hbase. The standalone zookeeper server runs the data cleanup manager but hbase does not. I suggest add the code below at the beginning of org.apache.hadoop.hbase.zookeeper.HQuorumPeer.runZKServer():
{code:java}
    // Start and schedule the the purge task
    DatadirCleanupManager purgeMgr=new DatadirCleanupManager(
          zkConfig.getDataDir(),
          zkConfig.getDataLogDir(),
          zkConfig.getSnapRetainCount(),
          zkConfig.getPurgeInterval());
    purgeMgr.start();
{code}
I've been running this code in my hbase cluster for quite a long time without problem.
Bug: API compatibilty checker fails with "Argument list too long" on Mac OS
Status: Open | Priority: Minor | Resolution: 
Created: 03/Dec/20 02:47
Affects Version/s: 2.4.0

While working on the 2.4.0 RC I hit a stumbling block where the argument list passed to javap by the API compatibility checker is too large for Mac OS. Attempted execution of the forked process fails with "Argument list too long".
Bug: Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Nov/20 01:28
Affects Version/s: 3.0.0-alpha-1

We input a 'this.bestSplitRow' as 'splitRow' of checkSplittable()
{code:java}
private void checkSplittable(final MasterProcedureEnv env,
    final RegionInfo regionToSplit, final byte[] splitRow)
{code}
But this private method could used 'bestSplitRow' directly
Bug: Remove unnecessary array to list conversion in SplitLogManager
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Nov/20 09:53

{code:java}
  public static List<FileStatus> getFileList(final Configuration conf, final List<Path> logDirs,
      final PathFilter filter)
      throws IOException {
    List<FileStatus> fileStatus = new ArrayList<>();
    ....
    FileStatus[] a = new FileStatus[fileStatus.size()];
    //list to array
    return fileStatus.toArray(a);
  }

  public List<FileStatus> getWALsToSplit(ServerName serverName, boolean splitMeta)
      throws IOException {
    List<Path> logDirs = master.getMasterWalManager().getLogDirs(Collections.singleton(serverName));
    FileStatus[] fileStatuses =
        SplitLogManager.getFileList(this.conf, logDirs, splitMeta ? META_FILTER : NON_META_FILTER);
    LOG.info("{} WAL count={}, meta={}", serverName, fileStatuses.length, splitMeta);
    //array convert to list which is just contrary to the logic of  getFileList
    return Lists.newArrayList(fileStatuses);
  
{code}

After judge the logic, there seems no need to make such a conversion.
Bug: Bulkload split hfile too many times due to unreasonable split point
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Nov/20 07:13

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688]

if hfile span multi regions, for example A,B,C,D,E,F(the start key of these regions are in ascending order), we should use region C endkey to split, not region A. In this way, we can get equal .top and .bottom hfiles, reduce time complexity of split from O( n ) to O(logn),decrease invoke of {{bulkLoad}} rpc to regionserver ,and also avoid write amplification during {{copyHFileHalf}}
Bug: Enhance balancer to make sure no meta primary/replica regions are going to be assigned to one same region server.
Status: Open | Priority: Minor | Resolution: 
Created: 06/Oct/20 18:55

Region replica has enhancement in balancer that primary region and its replicas are not going to be assigned to the same region server. Today, there is only one meta region, so this enhancement is still enough. With split meta coming in, it needs to make sure that no meta regoin/replicas is going to be assigned to the same region server in order to avoid hotspot issue.