av9ash/GitBugs huggingface dataset:
Bug: Provide a flexibility to print only row key and filter for multiple tables in the WALPrettyPrinter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Sep/20 21:07

Currently, [WALPrettyPrinter|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java] provides an option to ignore the values in the output, but it prints the whole cell and has no option to ignore some information from the cell. Sometimes, the user may only need the row keys from WAL and it may reduce the size of output from WALPrettyPrinter significantly. 

We should provide flexibility to output only rowkey from the cell. 

In addition we should increase the flexibility for providing multiple tables in the table filter.
Bug: Fix major compaction interval check logic bug when value of `hbase.hregion.majorcompaction` is below 0
Status: Open | Priority: Minor | Resolution: 
Created: 13/Sep/20 11:29

In `RatioBasedCompactionPolicy#shouldPerformMajorCompaction`



{code:java}
// mcTime can be less than 0
long mcTime = getNextMajorCompactTime(filesToCompact);

//this check logic can't handle the condtion that mcTime is less than 0
    if (filesToCompact == null || filesToCompact.isEmpty() || mcTime == 0) {
      return result;
    }
  ....

  // As long as lowTimestamp > 0, the following logic is true, and major compaction will run continually
  if (lowTimestamp > 0L && lowTimestamp < (now - mcTime))

{code}


Am i wrong?, By the way, do hbase allow to set negative values to properties like this? is it necessary to make valid check?
Bug: Donot merge regions if they are non adjacent in MergeNormalizationPlan.execute
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Sep/20 06:36
Affects Version/s: 1.6.0

Currently when we have MergeNormalizationPlan we do a force merge in execute method
{code:java}
// code placeholder
admin.mergeRegions(firstRegion.getEncodedNameAsBytes(),
  secondRegion.getEncodedNameAsBytes(), true);
{code}
Since we do not expect these regions to be non adjacent it is better to not force merge and be on safe side.
Bug: Remove read point parameter in method StoreFlush#performFlush and StoreFlush#createScanner
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Sep/20 02:09

Currently, read point parameter in method StoreFlush#performFlush is useless and can be safely removed.
and then method StoreFlush#createScanner can also remove this parameter
See below 


{code:java}
// Some comments here
  /**
   * Performs memstore flush, writing data from scanner into sink.
   * @param scanner Scanner to get data from.
   * @param sink Sink to write data to. Could be StoreFile.Writer.
   * @param smallestReadPoint Smallest read point used for the flush.
   * @param throughputController A controller to avoid flush too fast
   */
  protected void performFlush(InternalScanner scanner, CellSink sink,
      long smallestReadPoint, ThroughputController throughputController) throws IOException
{code}


Parameter smallestReadPoint is not used in this method. When `smallestReadPoint` is removed,  inner method `createScanner` can remove this necessary parameter too
Bug: Yetus IA Javadoc links are no longer available
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Aug/20 10:23

For HBase doc section "*HBase API Surface*", doc references for Apache Yetus InterfaceAudience 0.5.0 are outdated and no longer available: [https://yetus.apache.org/documentation/0.5.0/interface-classification/]

InterfaceAudience APIs are quite stable and hence, we can update our doc links to point to [https://yetus.apache.org/documentation/in-progress/interface-classification/] instead of version specific API doc.
Bug: Enable journal logging for HBase snapshot operation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jul/20 19:09
Affects Version/s: 1.7.0

The HBase snapshot operation contains multiple steps like actual snapshot creation, consolidate phase (reading region manifests from HDFS) , verifier phase ( validate the consolidated manifests against the actual number of regions for the table)

 

Sometimes it happens to be taking time in one of the phases and we don't know exactly which one is taking time unless we have a thread dump at the very same time. The journal logging would definitely help us give more insights into the time taken for each phase
Bug: NPE/500 accessing webui on master startup
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jul/20 19:09
Affects Version/s: 2.3.0

I got a 500 error while accessing the master web ui (non-backup master) during startup.

{noformat}
2020-07-21 18:56:39,762 WARN org.eclipse.jetty.servlet.ServletHandler: /master-status
java.lang.NullPointerException
        at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:339)
        at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:397)
        at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:388)
        at org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:79)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
...
{noformat}

{{MasterStatusTmplImpl.java:339}} appears to align with {{MasterStatusTmpl.jamon:227}}:

{noformat}
          <%if master.getMasterCoprocessorHost().findCoprocessor("RSGroupAdminEndpoint") != null &&
            serverManager.getOnlineServersList().size() > 0 %>
{noformat}
Bug: InfoServer support ipv6 host and port
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jul/20 03:17
Affects Version/s: 2.1.9

ipv6 host and port is 

[ a:::b:c:d]:16010

ipv4 host and port is 

a.b.c.d:16010

we can use HostAndPort to handle  it
Bug: Simplify the logic of getRegionInfo in TestFlushFromClient to reduce redundancy code
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Aug/20 06:43

Current logic:
{code:java}
  private List<HRegion> getRegionInfo() {
    return TEST_UTIL.getHBaseCluster().getLiveRegionServerThreads().stream()
      .map(JVMClusterUtil.RegionServerThread::getRegionServer)
      .flatMap(r -> r.getRegions().stream())
      .filter(r -> r.getTableDescriptor().getTableName().equals(tableName))
      .collect(Collectors.toList());
  }
{code}
The MiniHBaseCluster has similar method to do same thing.
 So it could just directly call:
{code:java}
  private List<HRegion> getRegionInfo() {
    return TEST_UTIL.getHBaseCluster().getRegions(tableName);
  }
{code}
Bug: Correct the help content of assign and unassign commands in hbase shell
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Aug/20 10:00

Help content in assign.rb:

Assign a region. Use with caution. *If region already assigned,*
 *this command will do a force reassign.* For experts only.

 

Help content in unassign.rb:

Unassign a region. *Unassign will close region in current location and then*
 *reopen it again.* Pass 'true' to force the unassignment ('force' will clear
 all in-memory state in master before the reassign. If results in
 double assignment use hbck -fix to resolve. To be used by experts).

 

The bold part are not right now, so update them.
Bug: RSGRoup: Error message while Table moving table to a RS  if one of the region is in closed state.
Status: Open | Priority: Minor | Resolution: 
Created: 25/Aug/20 11:40
Affects Version/s: 2.2.3

Error message while Table moving table to a RS if one of the region is in closed state.
 However shell command threw no error after all retry failed and also table is shown under this rsgroup  in UI 

Log

a89937ba9419f52d8bee0ba53a3e0001 to RSGroup rsg1
2020-08-25 20:02:30,065 ERROR [RpcServer.default.FPBQ.Fifo.handler=49,queue=4,port=16000] rsgroup.RSGroupAdminServer: Move region a89937ba9419f52d8bee0ba53a3e0001 to group failed, will retry, current retry time is 48
org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state for rit=CLOSED, location=null, table=test1, region=a89937ba9419f52d8bee0ba53a3e0001
 at org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:604)
 at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:662)
 at org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:679)
 at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveTableRegionsToGroup(RSGroupAdminServer.java:309)
 at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveTables(RSGroupAdminServer.java:452)
 at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.moveTables(RSGroupAdminEndpoint.java:233)
 at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:14933)
 at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:1018)
 at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:458)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
2020-08-25 20:02:30,066 INFO [RpcServer.default.FPBQ.Fifo.handler=49,queue=4,port=16000] rsgroup.RSGroupAdminServer: Moving 4 region(s) to group rsg1, current retry=48
Bug: [hbase-thirdparty] Builds without a `clean` phase fail on hbase-shaded-jetty
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Aug/20 18:39

In 3.4.0-rc1 that [~zhangduo] created, I noticed that builds of the project failed on hbase-shaded-jetty when I did not include {{clean}} in my build command.

{noformat}
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.604 s
[INFO] Finished at: 2020-08-11T14:38:45-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project hbase-shaded-jetty: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hbase.thirdparty.org.eclipse.jetty.http.HttpFieldPreEncoder -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
{noformat}

Poking around, it appears that this is actually to do with the creating of the shaded source jar.
Bug: Configure a max global timeout for HMaster to wait for RS to check in
Status: Open | Priority: Minor | Resolution: 
Created: 19/Aug/20 23:02

As part of the initialization process for active master, it waits for region servers to check in before completing the initialization. Currently it runs into a loop where it waits in the loop for ever when there are no region servers that are reported to it ([https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java#L1126)]

 

We should probably have a global max timeout after which the master should give up and abort itself letting other masters to take over as active master . This is especially useful in the cases where the incoming connections to the master are not accepted because of network issue
Bug: Remove unnecessary ClassNotFoundException and NoSuchMethodException stack traces from HBase startup logs
Status: Open | Priority: Minor | Resolution: 
Created: 19/Aug/20 17:20

During cluster startup there are a couple of places where HBase needs to use reflection to determine what version of Hadoop or protobuf it's using, and then instantiate the right class. 

If its first guess is wrong, it logs a ClassNotFoundException (for protobuf MessageLite) or NoSuchMethodException (for DFSClient.decryptEncryptedDataEncryptionKey). While this is done at DEBUG level, the exception types and the stack traces can lead to operators thinking something is wrong. 

Since neither the exception nor the stack trace is actually of interest, we should just log a brief, clear message at DEBUG level about which decision was made and eat the exception without logging it.
Bug: 'scandetail' log message is missing when responseTooSlow happens in the rpc that closes the scanner
Status: Open | Priority: Minor | Resolution: 
Created: 19/Aug/20 04:23
Affects Version/s: 1.4.13

'scandetail' log message is missing when responseTooSlow happens in the rpc that closes the scanner.

RSRpcServices.closeScanner is called before logging the slowLog in RPCServer.logResponse.

Since closeScanner removes the scanner entry from scanners map, logResponse can't find scanDetails when calling RSRpcServices.getScanDetailsWithId.

 

I have reproduced it by exhausting the region (no more results in the region), which sets moreResultsInRegion = false and cause to closeScanner in same rpc [https://github.com/apache/hbase/blob/c2e0cf989e4a86169219161d4d889db80288e636/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3235-L3237]

At least, this seems the behavior in branch-1.

 

I think this bug was introduced at https://issues.apache.org/jira/browse/HBASE-17489.

Note that this is a completely different case than https://issues.apache.org/jira/browse/HBASE-24282.
Bug: Sort the constants in `hbase_constants.rb`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Aug/20 18:34

The list of constant definitions in {{hbase_constants.rb}} is a bit of a mess. Sort them.

Breaks out the minor cleanup from HBASE-24627 / [PR#2215|https://github.com/apache/hbase/pull/2215] into it's own ticket.
Bug: regioninfo#isLast() has a logic error
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jul/20 02:28
Affects Version/s: 2.2.3

[https://github.com/apache/hbase/blob/90f4ff7d7c6997f61dbe40748b5c157879352acb/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java#L771]

it should be 
{code:java}
default boolean isLast() {
  return Bytes.equals(getEndKey(), HConstants.EMPTY_END_ROW);
}
{code}
Bug: Normalizer should log a successful run at INFO level
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Aug/20 22:57
Affects Version/s: 3.0.0-alpha-1

A successful or uneventful Normalizer run gives no indication that it ran at INFO level. I'm thinking a simple summary message like "Normalizer ran successfully in T. Submitted X plans across Y tables."
Bug: Add more stats in PE for read replica
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Aug/20 21:03
Affects Version/s: 2.3.1

Add more stats for read replica PE test. Currently, there is read replica tests in PE, but it does not provide details for how many requests to replica regions, and how many replica results win.

Also, want to add a latency histogram for replica reads.
Bug: TestReplicationSource.testWALEntryFilter failing in branch-2+
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 07/Aug/20 22:46

{noformat}
[ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 46.497 s <<< FAILURE! - in org.apache.hadoop.hbase.replication.regionserver.TestReplicationSource
[ERROR] org.apache.hadoop.hbase.replication.regionserver.TestReplicationSource.testWALEntryFilter  Time elapsed: 1.405 s  <<< FAILURE!
java.lang.AssertionError
	at org.apache.hadoop.hbase.replication.regionserver.TestReplicationSource.testWALEntryFilter(TestReplicationSource.java:177)
 {noformat}

Noticed this during HBASE-24779, but didn't fix it. Believe it comes from HBASE-24817.

Looks to be that the edit we expect to not be filtered is actually getting filtered because it has no cells (thus, no change and doesn't need to be replicated). Should be a simple fix.
Bug: RS startup with FSHLog throws NPE after HBASE-21751
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jul/20 10:53
Affects Version/s: 2.1.6

Every RS startup creates this NPE
{code}
[sync.1] wal.FSHLog: UNEXPECTED
java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:582)
        at java.lang.Thread.run(Thread.java:748)
2020-07-07 10:51:23,208 WARN  [regionserver/xxxxx:16020] wal.FSHLog: Failed sync-before-close but no outstanding appends; closing WALjava.lang.NullPointerException
{code}
the reason is that the Disruptor frameworks starts the Syncrunner thread but the init of the writer happens after that. A simple null check in the Syncrunner will help here .
No major damage happens though since we handle Throwable Exception. It will good to solve this.
Bug: HBase Shell Commands Survey
Status: Open | Priority: Minor | Resolution: 
Created: 02/Jul/20 17:34
Affects Version/s: 2.3.0

I am going through all 163 commands in the hbase-shell module and checking a few things:
 * Functions as advertised
 * Consistent naming, formatting, and help
 * Return values
 ** The majority of the commands still return nil. We can make the shell more powerful by switching some of these commands to return Ruby objects.

h3. Acceptance Criteria
 * The product of this ticket will be a *spreadsheet* with my comments on each command and potentially a design doc or other Jira tickets with the necessary changes found by the survey.
Bug: Unify hbase-shell ::Shell::Commands::Command#help behavior
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jul/20 19:22
Affects Version/s: 2.3.0

This is a small patch to unify shell help:

* Many of the commands use different indentation levels to style the help text.
* A few commands printed help instead of returning it, which breaks our pattern.

I also added a test to make sure the help text stays consistent after this patch.
Bug: Move log code out of loop
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jul/20 07:01
Affects Version/s: 3.0.0-alpha-1

In HMaster#normalizeRegions, maybe we shoule move the log code about  submittedPlanProcIds out of loop.

 
{code:java}
public boolean normalizeRegions() throws IOException {
  ...
    final List<Long> submittedPlanProcIds = new ArrayList<>();
    for (TableName table : allEnabledTables) {
      ...
      for (NormalizationPlan plan : plans) {
        long procId = plan.submit(this);
        submittedPlanProcIds.add(procId);
        ...
      }
      int totalPlansSubmitted = submittedPlanProcIds.size();
      if (totalPlansSubmitted > 0 && LOG.isDebugEnabled()) {
        LOG.debug("Normalizer plans submitted. Total plans count: {} , procID list: {}",
          totalPlansSubmitted, submittedPlanProcIds);
      }
    }
  ...
}
{code}
Bug: Address hbase-shell commands with unintentional return values
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jul/20 13:49
Affects Version/s: 2.3.0

There are a number of commands in hbase-shell that return values that are undocumented or otherwise do not makes sense.

The commands that will be addressed by this ticket were found in a survey of all the hbase shell commands, https://issues.apache.org/jira/browse/HBASE-24672.

Subtasks:
 * Fix commands that return the number of printed lines (usually the number 1).
 ** Affected commands (culprits): is_disabled, balancer, normalize, normalizer_switch, normalizer_enabled, catalogjanitor_switch, catalogjanitor_enabled, cleaner_chore_switch, cleaner_chore_enabled, splitormerge_switch, splitormerge_enabled, clear_deadservers, clear_block_cache
 ** I am updating these 13 commands to either return a value that makes sense or return nil if a sensible return value doesn't exist.
 * Fix commands that return true or false as a String
 ** Affected commands (culprits): balance_switch, snapshot_cleanup_switch, enable_rpc_throttle, disable_rpc_throttle, enable_exceed_throttle_quota, disable_exceed_throttle_quota
 ** I am updating these 6 commands to return ruby's TrueClass or FalseClass rather than a String
Bug: release scripts should check for tools it will at the start of building
Status: Open | Priority: Minor | Resolution: 
Created: 25/Jul/20 21:16

I just had a RC build fail *after* going through all the work to build everything because a recent OS upgrade meant that my {{svn}} command had been replaced by a placeholder that provided an error message about how X-Code is no longer providing subversion binaries.

getting a new copy of svn installed was easy enough, but if we're going to use subversion we should have an init check similar to what we have for yetus / maven / etc.
Bug: Investigate Potential to Use Ruby's Help Functionality in Shell
Status: Open | Priority: Minor | Resolution: 
Created: 24/Jul/20 16:51
Affects Version/s: 2.3.0

Ruby and Interactive Ruby (irb) have functionality for helping users (via introspection). It would be interesting to see if we can hook into this for hbase-shell.
 * Ruby has a built in method called "ri" which lets users explore help text for modules, classes, and methods from RDocs.
 ** [https://ruby.github.io/rdoc/RI_rdoc.html]
 * Interactive ruby has a method called "irb_help" which gets patched onto the main Object as "help" (except in our shell, where we have our own help method). Under the hood, it uses {{Ri}}.
 ** [https://docs.ruby-lang.org/en/2.3.0/IRB/ExtendCommandBundle.html]
 ** [https://github.com/ruby/irb/blob/66c4eab4e71b47f3c78ed68727c0bd1c3897591d/lib/irb/cmd/help.rb]

At the moment, {{irb_help}}/{{ri}} knows nothing about the HBase shell commands
{code:java}
hbase> irb_help

Enter the method name you want to look up.
You can use tab to autocomplete.
Enter a blank line to exit.

>> list
Nothing known about .list
{code}
Bug: Meta replicas not cleaned when disabled
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jul/20 12:35
Affects Version/s: 2.2.5

The assignMetaReplicas method works kinda like this:
{code:java}
void assignMetaReplicas(){
if (numReplicas <= 1) return;

//create if needed then assign meta replicas

unassignExcessMetaReplica(numReplicas);
}
{code}
Now this unassignExcessMetaReplica method is the one that gets rid of the replicas we no longer need. It closes them and deletes their zNode.  Unfortunately this only happens if we decreased the replica number. If we disabled it, by setting the replica number to 1 assignMetaReplicas returns instantly without cleaning up the no longer needed replicas resulting in replicas lingering around.
Bug: Fix empty value of properties 'hbase.replication.source.maxthreads' in hbase-thrift module
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Sep/20 10:44


The value of hbase.replication.source.maxthreads is empty in hbase-thrift test
see hbase-thrift/src/test/resources/hbase-site.xml
{code:java}
  <property>
    <name>hbase.replication.source.maxthreads</name>
    <value></value>
    <description>Default is 10</description>
  </property>
{code}