av9ash/GitBugs huggingface dataset:
Bug: TestHFileCleaner can fail after addition of ManualMobMaintHFileCleaner
Status: Open | Priority: Minor | Resolution: 
Created: 24/Jan/20 01:20
Affects Version/s: 2.2.3

After https://github.com/apache/hbase/commit/a881678ef4e0ffcd94add935729424c6e2567356 testHFileCleaning can fail due to some lazy directory creation in the test. ManualMobMaintHFileCleaner needs to get the table directory (and indirectly namespace) which can fail since the test itself only creates the file within the archive directory. It does not contain the archive/data/namespace/table/region/cf/ which the ManualMobMaintHFileCleaner expects.

This is a relatively minor issue, but we saw this test fail when running on EC2 because we had an 'invalid namespace' (because it had a '-' in the path).

FYI [~busbey]
Bug: Expose content of meta table in web ui
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jan/20 20:10

While testing improvements to hbck2, breaking and fixing a cluster's region assignments, we've found that quite a few questions can only be answered by dropping into a shell and scanning meta. This is in contrast to the nice web view we have on procedures and their locks. It would be nice to have a similar view into the content of meta. I've put together and extension to the existing {{table.jsp}} that does this. It's had a couple rounds of operator feedback. Will post up some screenshots along with the PR.
Bug: Split unit tests from TestTableName into a separate test only class
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jan/20 19:46

The class {{TestTableName}} in hbase-common is both a unit test for the {{TableName}} class, and a junit {{TestRule}}. This double-duty is strange; the Rule functionality is not discoverable at all. Split the Rule out into a separate class, maybe {{TableNameRule}} or some such.
Bug: Allow dot and hyphen in Profiler's URL
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jan/20 12:16
Affects Version/s: 2.1.5

ProfileOutputServlet sanitizes the input URL but fails when the query string contains dot or hyphen. These are valid characters for the hostname.

Example URL part:

/prof-output/async-prof-pid-122466-cpu-1.svg?host=myhost-1.example.com&port=16010 

In this case the user gets the following error message:
{noformat}
HTTP ERROR: 500
Problem accessing /prof-output/async-prof-pid-122466-cpu-1.svg. Reason:
    java.lang.RuntimeException: Non-alphanumeric data found in input, aborting.
Powered by Jetty:// 9.3.27.v20190418 {noformat}
Bug: hbase:meta's table.jsp ref to wrong rs address   
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jan/20 09:22
Affects Version/s: 2.2.3

<td><a href="http://<%= hostAndPort %>/rs-status/"><%= StringEscapeUtils.escapeHtml4(hostAndPort) %></a></td>

Click the link it returns error.
{code}
HTTP ERROR 404
Problem accessing /rs-status/. Reason:

    Not Found
{code}
Bug: DeadServer cleanup
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Jan/20 17:39

Cleanup of DeadServer class shutting down access, undoing duplication, adding doc., and removing unused code.

One change is that we do not remove a server from 'processing' list when we 'remove' deadservers; we let SCP do it since it owns processing list (Saw issue where on fast restart of a server, the server was removed from deadserver and from processing list though the SCP for the dead server was still running -- no repercussions that I could see but a little confusing).
Bug: synchronizedList-related defect in HBaseFsck
Status: Open | Priority: Minor | Resolution: 
Created: 02/Jan/20 10:04

org.apache.hadoop.hbase.util.HBaseFsck: {{orphanHdfsDirs}} could be iterated in {{adoptHdfsOrphans()}} in parallel with modification. The body of the {{adoptHdfOrphans()}} should be wrapped with {{synchronized (orphanHdfsDirs) {}}}.
Bug: Add null checks and logging to misc set of tests
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Feb/20 23:11

I've been studying unit tests of late. A few are failing but then the output is missing a detail or shutdown complains of NPE because startup didn't succeed.

Here are super minor items I've been carrying around that I'd like to land. They do  not change the function of tests (there is an attempt at a fix of TestLogsCleaner).

* TestFullLogReconstruction log the server we've chosen to expire and then note where we starting counting rows
* TestAsyncTableScanException use a define for row counts; count 100 instead of 1000 and see if helps
* TestRawAsyncTableLimitedScanWithFilter check connection was made before closing it in tearDown
* TestLogsCleaner use single mod time. Make it for sure less than now in case test runs all in the same millisecond (would cause test fail)
* TestReplicationBase test table is non-null before closing in tearDown
Bug: Fix typo in procedures.jsp
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 09/Feb/20 13:22

I think there's a typo.In hbase-server/src/main/resources/hbase-webapps/master/procedures.jsp. The Sentences "We do not list Procedures that have completed SUCCESSfully; their number makes it hard to spot the problematics." are misspelled and need to be corrected as "We do not list procedures that have completed successfully; their number makes it hard to spot the problematics."
Bug: Modify "Block locality" of RegionServer Web UI to human readable percentage
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 17/Feb/20 12:39

The unit of "Block locality" in Web UI just like picture 1

!image-2020-02-17-20-36-50-084.png!

I think we should change it to percentage unit like picture 2

!image-2020-02-17-20-38-52-990.png!
Bug: Change bytes size to human readable size for Server Metrics of RegionServer Web UI
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 16/Feb/20 11:31

I found that the “BytesBufferAllocator Status” in RegionServer Web UI still using "Bytes" as a fixture unit. I think we should use "MB" or "GB" when the size is too large !image-2020-02-16-19-23-39-432.png!

 

The Web UI after improvement:

!image-2020-02-16-19-28-23-449.png!
Bug:  Add WebUI to Canary
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 16/Mar/20 01:16
Affects Version/s: 3.0.0-alpha-1

During the running of Canary, the table sniff failure information will be printed through the Log.
{code:java}
LOG.error("Read from {} on {}", table, server);
{code}
 

I think we can use WebUI to display these failures to make it easier for us to view this information

 

!image-2020-03-16-09-12-00-595.png!

As shown in the figure above, we can directly see the Table and Regionserver where the error occurred
Bug: DEBUG logging cleanup
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 14/Jan/20 18:20
Affects Version/s: 2.3.0

Minor cleanup of annoying loggings. For example, over an hour, we logged this 200k times:

{code}2020-01-14 11:06:00,287 DEBUG org.apache.hadoop.hbase.master.cleaner.LogCleaner: Exiting
{code}

There is no corresponding 'Starting'.
Bug: Add in comments, minor refactorings, and review feedback from parent HBASE-23984
Status: Open | Priority: Trivial | Resolution: 
Created: 19/Mar/20 19:21

Add in comments and address code feedback from parent issue WITHOUT changing functionality.

Parent got a lot of good feedback and in interests of getting a bug fix committed, the bulk of the original parent patch was postponed. This issue is about landing innocuous changes that help w/ readability and to pick up some of the feedback on original PR.
Bug: Remove unused variables from HBaseSaslRpcServer
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 05/Mar/20 17:13
Affects Version/s: 2.3.0

On *HBaseSaslRpcServer*, there's currently an unused _Logger_ variable, as well as an unused _Configuration_ parameter defined on *HBaseSaslRpcServer* constructor. *HBaseSaslRpcServer* is IA private, and its constructor is only referenced by *ServerRpcConnection,* so it should be fine to change constructor signature to get rid of the unused parameter.
Bug: Documentation update of external_apis.adoc#example-scala-code
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 16/Feb/20 09:52

Update the Example Scala Code in the Reference Guide as it contains deprecated content such as 
 * new HBaseConfiguration()
 * new HTable(conf, "mytable")
 * add(Bytes.toBytes("ids"),Bytes.toBytes("id1"),Bytes.toBytes("one"))

Replace it with:
 * HBaseConfiguration.create()
 * TableName.valueOf({color:#6a8759}"mytable"{color})
 * addColumn(Bytes.toBytes({color:#6a8759}"ids"{color}){color:#cc7832},{color}Bytes.toBytes({color:#6a8759}"id1"{color}){color:#cc7832},{color}Bytes.toBytes({color:#6a8759}"one"{color}))
Bug: [Flakey Test] TestRegionReplicaFailover fails frequently
Status: Resolved | Priority: Trivial | Resolution: Incomplete
Created: 21/Feb/20 06:12
Affects Version/s: 2.3.0

Fails > 50% of the time in the GCE runs since we upped their ferocity a few days ago.

Looking, the complaints are weirdly often OOME, can't create thread. Looking, it at least does start of a RS at end of a test when it kills one EVEN THOUGH the @before starts a whole new cluster.  This improves general resource usage going by the resource dumps before and after but still loads of leaking.

Cut down on the amount of work it does too. Let me push small patch and see if helps.
Bug: [DOC] HBase Web UI runs on different port
Status: Resolved | Priority: Trivial | Resolution: Not A Problem
Created: 28/Jan/20 21:56

Just started to work through the ref guide and started up the stand-alone setup. The guide mentions to look up [http://localhost:16010/] for the HBase Web UI, but there is no process listening to that port. 

 

After checking netstat and logs I've discovered, that the UI runs on port 43105

 
{code}
2020-01-28 22:17:43,658 INFO [main] http.HttpServer: Jetty bound to port 43105
2020-01-28 22:17:43,658 INFO [main] mortbay.log: jetty-6.1.26
2020-01-28 22:17:43,697 INFO [main] mortbay.log: Started SelectChannelConnector@0.0.0.0:4310
{code}
Bug: Checkstyle plugin complains about our checkstyle.xml format; doc how to resolve mismatched version
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 15/Jan/20 07:06

Trying to add the checkstyle.xml to the intellij checkstyle plugin after reading HBASE-23688, it complains with the following when it reads in the config file:
{code}
com.puppycrawl.tools.checkstyle.api.CheckstyleException: cannot initialize module TreeWalker - TreeWalker is not allowed as a parent of LineLength Please review 'Parent Module' section for this Check in web documentation if Check is standard.
	at com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:473)
	at com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:198)
	at org.infernus.idea.checkstyle.service.cmd.OpCreateChecker.execute(OpCreateChecker.java:61)
	at org.infernus.idea.checkstyle.service.cmd.OpCreateChecker.execute(OpCreateChecker.java:26)
	at org.infernus.idea.checkstyle.service.CheckstyleActionsImpl.executeCommand(CheckstyleActionsImpl.java:130)
	at org.infernus.idea.checkstyle.service.CheckstyleActionsImpl.createChecker(CheckstyleActionsImpl.java:60)
	at org.infernus.idea.checkstyle.service.CheckstyleActionsImpl.createChecker(CheckstyleActionsImpl.java:51)
	at org.infernus.idea.checkstyle.checker.CheckerFactoryWorker.run(CheckerFactoryWorker.java:46)
Caused by: com.puppycrawl.tools.checkstyle.api.CheckstyleException: TreeWalker is not allowed as a parent of LineLength Please review 'Parent Module' section for this Check in web documentation if Check is standard.
	at com.puppycrawl.tools.checkstyle.TreeWalker.setupChild(TreeWalker.java:147)
	at com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:198)
	at com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:468)
	... 7 more
{code}
Bug: Column family ns does not exist in region during upgrade to 3.0.0-beta-2
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 17/Feb/24 22:30
Affects Version/s: 3.0.0-beta-1

Upgrading from 2.5.x to 3.0.0-alpha-2, migrateNamespaceTable kicks in to copy data from the namespace table to an "ns" family of the meta table. If you don't have an "ns" family, the migration fails and the hmaster will crash loop. You then can't rollback, because the briefly alive upgraded hmaster created a procedure that can't be deserialized by 2.x (I don't have this log handy unfortunately). I tried pushing code to create the ns family on startup, but it doesnt work becuase the migration happens while the hmaster is still initializing.

So it seems imperative that you create the ns family before upgrading. We should handle this more gracefully.
Bug: HBase Restore restores old data if data for the same timestamp is in different hfiles
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 23/Mar/24 21:04

The restore brings back 'old' data when executing restore.

It feels like the hfile sequence id is not respected during the restore.

See testing code attached. The workaround solution is to trigger major compaction before doing the backup (not really feasible for daily backups)

We didn't investigate this yet, but this might also impact the merge of multiple incremental backups (since that follows a similar code path merging hfiles).

This currently blocks our support for HBase backup and restore.

Willing to participate in a solution if necessary.
Bug: Add BootstrapNodeService handlers
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 16/Jan/24 17:17
Affects Version/s: 2.6.0

We added calls to a BootstrapNodeService, but the servers are not setup to serve it. We need to add in two places:
 * RSRPCServices list of services: [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L1447]
 * HBasePolicyProvider mapping of acl to service: [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java#L40]

Without adding to these two places, you first see UnknownServiceExceptions and then you see AccessDeniedExceptions
Bug: RpcConnectionRegistry is broken when security is enabled and we use different principal for master and region server
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 18/Jan/24 02:37

After introducing RpcConnectionRegistry, we let master and region server both implement ClientMetaService.

In our current client architecture, when security is enabled, we rely on the record in SecurityInfo to determine the server principal to use, unfortunately there is only one principal can be specified, so if we use different principal for master and region server, either we can not connect to master, or we can not connect to region server.

And just changing the server principal field in SecurityInfo to an array can not solve the problem, as when connecting, we do not know whether the remote server is a master or region server, so we still can not determine which principal to use...

Anyway, since this has been in our code base since 2.5.0, it is not a new problem, so just set it as critical, not a blocker. But we should find out the solution ASAP.
Bug: Cleaning up binary assemblies and diagnostic tools
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/24 06:21
Affects Version/s: 3.0.0-beta-1

As discussed on the mailing list, the current binary assembly has several problems. 

The discussed improvements:
 * Provide assembly versions without transitive Hadoop dependencies
 * Remove test JARs and their dependencies from the assemblies
 * Move useful diagnostic tools into the runtime jars

See [https://lists.apache.org/thread/s784s8778cys2vx0scbd28cl2lm1g6vt] for details.
Bug: Update assembly to create a tarball with hadoop and without hadoop
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/24 07:23

The goal of this task is to update the HBase assembly by providing two distinct variants - one that includes Hadoop and one that does not.

Currently, our assembly includes a substantial amount of the Hadoop distribution. This task involves modifying our build and assembly process to create two separate distributions of HBase:

* A variant that includes Hadoop, serving as a complete package for users who do not have a pre-existing Hadoop installation.
* A leaner variant without Hadoop, suitable for environments where Hadoop is already installed and configured.

This change aims to reduce the distribution size, speed up startup times, and decrease the chance of conflicts with the Hadoop jars. It also aims to reduce the number of CVE-prone JARs in the binary assemblies.

The task includes ensuring that both variants function correctly in their respective scenarios and that existing functionality is not negatively impacted.

CC: [~stoty], [~zhangduo], [~ndimiduk], [~bbeaudreault]
Bug: Backport "HBASE-27693 Support for Hadoop's LDAP Authentication mechanism (Web UI only)" to branch-2
Status: Patch Available | Priority: Major | Resolution: 
Created: 14/Feb/24 12:13

Hadoop's AuthenticationFilter has changed and now has support for ldap mechanism too. HBase still uses an older version tightly coupled with kerberos and spnego as the only auth mechanisms. HADOOP-12082 has added support for multiple auth handlers including LDAP. On trying to use Hadoop's AuthenticationFilterInitializer in hbase.http.filter.initializers, there is a casting exception as HBase requires it to extend org.apache.hadoop.hbase.http.FilterInitializer.
Bug: Modify the assembly to not include test jars and their transitive dependencies
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/24 07:18

This task aims to modify the HBase assembly to exclude test jars and their transitive dependencies.

Currently, our assembly includes test jars and test dependencies, which adds unnecessary bloat to the assembly and classpath. This not only increases the distribution size but also potentially introduces CVE-prone JARs into the binary assemblies.

The objective of this task is to modify the build and assembly process to exclude these test jars and their dependencies. This will result in a leaner, more secure assembly with a faster startup time.

CC:  [~stoty], [~zhangduo], [~ndimiduk], [~bbeaudreault]
Bug: Enable maven-source-plugin for all modules
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/24 07:05
Affects Version/s: 2.4.17

There is no source jar generated for hbase-server.
Enabling maven-source-plugin seems to work fine, and Eclipse can use the generated sources jar for debugging etc.

It seems that these source JARs are not generated at least since HBase 2.0.0, but we did have source JARs sometime in the 1.x time frame.
Bug: Refactor tools which are under test packaging to a new module hbase-diagnostics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/24 07:17

The purpose of this task is to refactor and move certain tools currently located under the test packaging to a new module, named 'hbase-tools'.

The following tools have been initially identified for relocation(will add more as and when identified):
 - [PerformanceEvaluation|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java]
 - [LoadTestTool|https://github.com/apache/hbase/blob/936d267d1094e37222b9b836ab068689ccce3574/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java]
 - [HFilePerformanceEvaluation|https://github.com/apache/hbase/blob/936d267d1094e37222b9b836ab068689ccce3574/hbase-server/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java]
 - [ScanPerformanceEvaluation|https://github.com/apache/hbase/blob/936d267d1094e37222b9b836ab068689ccce3574/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/ScanPerformanceEvaluation.java]
 - [LoadBalancerPerformanceEvaluation|https://github.com/apache/hbase/blob/936d267d1094e37222b9b836ab068689ccce3574/hbase-balancer/src/test/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerPerformanceEvaluation.java]
 - [WALPerformanceEvaluation|https://github.com/apache/hbase/blob/936d267d1094e37222b9b836ab068689ccce3574/hbase-server/src/test/java/org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.java]

These tools are valuable beyond the scope of testing and should be accessible in the binary distribution of HBase. However, their current location within the test jars adds unnecessary bloat to the assembly and classpath, and potentially introduces CVE-prone JARs into the binary assemblies. We plan to remove all test jars from assembly with HBASE-28433.

This task involves creating the new 'hbase-tools' module, and moving the identified tools into this module. It also includes ensuring that these tools function correctly in their new location and that their relocation does not negatively impact any existing functionality or dependencies.

CC: [~stoty], [~zhangduo], [~ndimiduk], [~bbeaudreault]
Bug: Add support splitting region into multiple regions(more than 2)
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Mar/24 06:01

We have a requirement of splitting one region into multiple hundreds of regions at a time to distribute loading hot data. To do that we need split a region and wait for the completion of it and then again split the two regions etc..which is time consuming activity. 
Would be better to support splitting region into multiple regions more than two so that in single operation we can split the region.
Todo that we need to take care
1)Supporting admin APIs to take multiple split keys
2)Implement new procedure to create new regions, creating meta entries and udpating them to meta
3) close the parent region and open split regions.
4) Update the compaction of post split and readers also to use the portion store file reader based on the range to scan than half store reader.
5) make sure the catalog jonitor also cleaning the parent regions when there are all the regions split properly.
Bug: Expose checkQuota to Coprocessor Endpoints
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/24 18:25

Coprocessor endpoints may do non-trivial amounts of work, yet quotas do not throttle them. We can't generically apply quotas to coprocessors because we have no information on what a particular endpoint might do. One thing we could do is expose checkQuota to the RegionCoprocessorEnvironment. This way, coprocessor authors have the tools to ensure that quotas cover their implementations.

While adding this, we can update AggregationImplementation to call checkQuota since those endpoints can be quite expensive.
Bug: Quota throttling can cause a leak of scanners
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/24 14:38

In RSRpcServices.scan, we check the quota after having created a new RegionScannerHolder. If the quota is exceeded, an exception will be thrown. In this case, we can't send the scannerName back to the client because it's just an exception. So the client will be forced to retry the openScanner call, but the RegionScannerHolder is not closed. Eventually the scanners will be cleaned up by the lease expiration, but this could cause many scanners to leak during periods of high throttling.

We could close the newly opened scanner before throwing the throttle exception, but I think it's better to not open the scanner at all until we've grabbed some quota.