av9ash/GitBugs huggingface dataset:
Bug: Add additional comments when using HBASE_TRACE_OPTS with standalone mode
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/May/22 00:05
Affects Version/s: 2.5.0

it was a minor problem caused by HBASE-26363 that if operator or developer wants to use HBASE_TRACE_OPTS in standalone, they will need to configure the HBASE_OPTS with appending HBASE_TRACE_OPTS parameter, otherwise, HBASE_TRACE_OPTS will not be used by the HBASE_OPTS and do not load the {{-Dotel.traces.exporter=none -Dotel.metrics.exporter=none}}
https://github.com/apache/hbase/blob/master/bin/hbase#L511

{code}
HBASE_OPTS="$HBASE_OPTS -javaagent:$agent_jar"
{code}
Bug: hboss doesn't compile against hadoop branch-3.3 now that s3guard is cut
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/May/22 12:38

HBoss doesn't compile against hadoop builds containing HADOOP-17409, "remove s3guard", as test setup tries to turn it off.

there's no need for s3guard any more, so hboss can just avoid all settings and expect it to be disabled (hadoop 3.3.3. or earlier) or removed (3.4+)

(hboss version is 1.0.0-alpha2-SNAPSHOT)
Bug: Decrease memory copy when decompress data
Status: Open | Priority: Minor | Resolution: 
Created: 18/May/22 02:27

HBase RegionServer use createDecompressionStream in class org.apache.hadoop.hbase.io.compress.Compression, which cause extra memory copy during decompression.We can offer interface for block decompression, like "void decompress(ByteBuff src, ByteBuff dst);", and offer default implementation for all algorithms.
Bug: Fix spotless warn for master branch
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/May/22 07:24

Fix spotless warn for master branch. We can see the spotless warn at [https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/1/artifact/yetus-general-check/output/branch-spotless.txt]. 

There are some compile tasks that fail because of this.
Bug: Optimize log format for PerformanceEvaluation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/May/22 10:15

The logs in PerformanceEvaluation look a little confusing to new users, we should optimize the format.

Before:

!image-2022-05-06-18-13-50-763.png|width=787,height=156!

After:

!image-2022-05-06-18-15-13-913.png|width=674,height=147!
Bug: Add default implementation for BufferedMutator interface setters
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Apr/22 12:42

HBASE-20812 added default implementations to a bunch of the "optional" methods of Table interface. This is very useful for downstream users who might create delegate table implementations. Some of the BufferedMutator interface methods also have default implementations, but setRpcTimeout, setOperationTimeout, getWriteBufferSize do not.
Bug: TestStochasticLoadBalancer has some slow methods, and inconsistent set, reset, unset of configuration
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Apr/22 05:39
Affects Version/s: 2.0.0

Some test ordering issues were exposed by adding new tests in HBASE-22349. I think this is a legitimate issue which is tracked in HBASE-26988.

But we can update the tests to be consistent in how they update configuration to reduce confusion, removing the {{unset}} calls.

We can also update other configuration values to significantly speed up the long-running methods. Methods that are simply checking for balancer plans do not need to {{runMaxSteps}}. All we need to do is run enough steps to guarantee we will plan to move one region. That can be far fewer than the tens of millions of steps we may be running given {{runMaxSteps}}.
Bug: Brotli compressor has unexpected behavior during reinitialization
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Apr/22 23:10
Affects Version/s: 2.5.0

Due to a copy and paste error in BrotliCompressor#reinit we can reallocate a direct buffer instead of an on heap buffer. Not a functional problem right now but in all other cases the allocation is a heap buffer so could be a future surprise, and is in any case incorrect.
Bug: Balancer should reset to default setting for hbase.master.loadbalance.bytable if dynamically reloading configuration
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Apr/22 04:22
Affects Version/s: 2.0.0

# Set {{hbase.master.loadbalance.bytable}} to {{true}} in {{hbase-site.xml}}
# Start HMaster
# Remove {{hbase.master.loadbalance.bytable}} entry in {{hbase-site.xml}}
# Dynamically reload configuration for hmaster (https://hbase.apache.org/book.html#dyn_config)

*Expected:* load balancing would no longer happen by table
*Actual:* load balancing still happens by table
*Workaround:* leave the entry in {{hbase-site.xml}} but explicitly set to {{false}}

*Note 1*: The steps may only work if the config value is not in {{hbase-default.xml}} so it may be an unlikely scenario.

*Note 2*: I see this when running tests added in HBASE-22349, depending on the order of execution of test methods. If {{testSloppyTablesLoadBalanceByTable}} executes before {{testBalanceOfSloppyServers}} there will be a failure. We could apply the workaround to the tests (explicitly set to {{false}}), but it seems better to fix the dynamic reconfiguration behavior. Regardless, I will propose test fixes in HBASE-26989.
Bug: Add on heap and off heap memstore info in rs web UI
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Apr/22 10:39

There are _Memstore Size_ and _Memstore Limit_ info in rs-status page.

The information is a little coarse granularity and confusing:
 * _Memstore Size_ means memstore data size, but we cannot tell the difference of on heap and off heap memstore data size.
 * _Memstore Limit_ is off heap memstore limit if off-heap is enabled, otherwise, it is on heap memstore limit. 

We should provide more detailed information.
Bug: Update inmemory_compaction.adoc
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Apr/22 13:06

The default value of _hbase.hregion.compacting.memstore.type_ was changed from _BASIC_ to _DEFAULT_ by [HBASE-20539|https://issues.apache.org/jira/browse/HBASE-20539]. 

We should update its doc _inmemory_compaction.adoc_ accordingly.
Bug: StoreFileListFile logs frequent stacktraces at INFO level
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Apr/22 22:51
Affects Version/s: 2.5.0

In StoreFileListFile we have:
{code}
        } catch (EOFException e) {
          // this is normal case, so use info and do not log stacktrace
          LOG.info("Failed to load track file {}: {}", trackFiles[i], e.toString());
        }
{code}
except that we see frequent stacktraces at INFO level from StoreFileListFile and this appear to be the culprit.
Bug: Fix java import in admin.rb
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Apr/22 04:21

The java_import for MobCompactPartitionPolicy in admin.rb is missed, which cause an error message for creating table.
{code:bash}
hbase:006:0> create 't2', {NAME => 'f', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
ERROR: uninitialized constant Hbase::Admin::MobCompactPartitionPolicy
Did you mean?  Hbase::Admin::MOB_COMPACT_PARTITION_POLICY
For usage try 'help "create"'
Took 0.6897 seconds
hbase:007:0> {code}
Bug: Update committer guide around merging from GitHub
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 12/Apr/22 13:10

The committer guide in [https://hbase.apache.org/book.html#_guide_for_hbase_committers] seems to focus mostly on committing patch files that were uploaded to jira. There are some brief mentions of GitHub, but I think these days GitHub is the primary mode of contributing. Correct me if I'm wrong.

If so, I think we should update the docs with instructions from a GitHub-centric workflow. 

I'd be happy to update our guide, but I will need help on collecting the basic steps since I don't know how to do it myself yet.
Bug: Remove some checks in checkstyle
Status: Open | Priority: Minor | Resolution: 
Created: 02/Apr/22 05:33

After enableing spotless, maybe we can remove some checks in checkstyle as it is useless now, such as indentation check. The spotless plugin will try its best to format the code, and if checkstyle still think it is not suitable, we have no way to improve.

This is an example:

{code}
  public ScreenView transitionToFieldScreen(Screen screen, Terminal terminal) {
    return new FieldScreenView(screen, terminal, topScreenModel.getCurrentSortField(),
      topScreenModel.getFields(), fieldDisplayMap, (sortField, fields, fieldDisplayMap) -> {
        topScreenModel.setSortFieldAndFields(sortField, fields);
        this.fieldDisplayMap.clear();
        this.fieldDisplayMap.putAll(fieldDisplayMap);
      }, topScreenView);
  }
{code}

I think the indentation is very clear to human but checkstyle reports the 'topScreenModel.setSortFieldAndFields(sortField, fields);' should have indentation 6 instead of 8.

In the same file, there is another example
{code}
    return new InputModeScreenView(screen, terminal, row,
      "add filter #" + (topScreenModel.getFilters().size() + 1) + " ("
        + (ignoreCase ? "ignoring case" : "case sensitive") + ") as: [!]FLD?VAL",
      topScreenModel.getFilterHistories(), (inputString) -> {
        if (inputString.isEmpty()) {
          return topScreenView;
        }

        if (!topScreenModel.addFilter(inputString, ignoreCase)) {
          return goToMessageMode(screen, terminal, row, "Unacceptable filter expression");
        }

        paging.init();
        return topScreenView;
      });
{code}
Checkstyle reports the indentation for ' if (inputString.isEmpty()) {' should be 6 instead of 8, but for me I think the current indentation is more clear for human.

So after reviewing all the indentation problem and tweaking the eclipse formatter, I think we could remove the indentation check in our checkstyle config.
Bug: Fix typos in ZKUtil
Status: Open | Priority: Trivial | Resolution: 
Created: 17/Jun/22 14:16

Two typos in hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java:
{code:java}
   * Creates the specified node containing specified data, iff the node does not exist. Does not set

   * Creates the specified node, iff the node does not exist. Does not set a watch and fails {code}
iff -> if
Bug: CallDroppedException message says "server 0.0.0.0 is overloaded"
Status: In Progress | Priority: Trivial | Resolution: 
Created: 13/Apr/22 14:56

This is not a super useful error when seen on the client side. Maybe we can have it pull in the ServerName instead, or at the very least resolve to the hostname.

 

https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java#L213
Bug: Upgrade pyjwt to version 2.4.0 or later for dev-support/git-jira-release-audit
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 27/May/22 20:54

Dependabot says

Upgrade pyjwt to fix 1 Dependabot alert in dev-support/git-jira-release-audit/requirements.txt

Upgrade pyjwt to version 2.4.0 or later. For example:

pyjwt>=2.4.0
Bug: Minor compression performance improvements
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 10/May/22 02:07

TRACE level logging is expensive enough to warrant removal. They were useful during development but now are just overhead. 

{noformat}
  1270039022    4.07%      127  jbyte_disjoint_arraycopy
{noformat}

e.g.
{noformat}
  [ 0] jbyte_disjoint_arraycopy
  [ 1] org.slf4j.impl.Reload4jLoggerAdapter.isTraceEnabled
  [ 2] org.slf4j.impl.Reload4jLoggerAdapter.trace
  [ 3] org.apache.hadoop.hbase.io.compress.aircompressor.HadoopCompressor.setInput
  [ 4] org.apache.hadoop.io.compress.BlockCompressorStream.write
  [ 5] java.io.OutputStream.write
  [ 6] com.salesforce.hbase.util.TestUtils.outputStreamTest
  [ 7] com.salesforce.hbase.util.TestUtils.outputStreamTest
  [ 8] com.salesforce.hbase.BenchmarkAircompressorLz4.test
  [ 9] com.salesforce.hbase.jmh_generated.BenchmarkAircompressorLz4_test_jmhTest.test_avgt_jmhStub
  [10] com.salesforce.hbase.jmh_generated.BenchmarkAircompressorLz4_test_jmhTest.test_AverageTime
{noformat}

Also we unnecessarily create new LZ4 compressor and decompressor instances in the reset() methods.
Bug: Update javadoc of BucketCache.java
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 27/Apr/22 03:18

The feature of on-heap Bucket cache was removed by [HBASE-19187|https://issues.apache.org/jira/browse/HBASE-19187]. 

We should update its javadoc which now is inaccurate and confusing.
Bug: Typo in admin.rb "COMPRESSION_COMPACT_MAJPR"
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Apr/22 23:58

{noformat}
ERROR: uninitialized constant Java::OrgApacheHadoopHbase::HColumnDescriptor::COMPRESSION_COMPACT_MAJPR
Did you mean?  Java::OrgApacheHadoopHbase::HColumnDescriptor::COMPRESSION_COMPACT_MINOR
               Java::OrgApacheHadoopHbase::HColumnDescriptor::COMPRESSION_COMPACT_MAJOR
               Java::OrgApacheHadoopHbase::HColumnDescriptor::COMPRESSION_COMPACT
{noformat}
Bug: [Documentation] Fix log parameter error and spelling error
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 02/Apr/22 11:58
Affects Version/s: 2.4.0

In ConstantSizeRegionSplitPolicy.isExceedSize(),

'''

if (size > sizeToCheck) {
LOG.debug("ShouldSplit because {} size={}, sizeToCheck={}{}",
store.getColumnFamilyName(), StringUtils.humanSize(size),
StringUtils.humanSize(sizeToCheck));
return true;
}

'''

There should be only three parameters, remove last '{}'
Bug: hbase-connectors mvn install error
Status: Closed | Priority: Blocker | Resolution: Duplicate
Created: 04/Aug/20 08:56

*when  maven  command-line*

mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11 -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install

will return error

{color:red}[ERROR]{color} [Error] F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\datasources\HBaseTableScanRDD.scala:216: overloaded method value addTaskCompletionListener with alternatives:
  (f: org.apache.spark.TaskContext => Unit)org.apache.spark.TaskContext <and>
  (listener: org.apache.spark.util.TaskCompletionListener)org.apache.spark.TaskContext
 does not take type parameters
{color:red}[ERROR] {color}one error found

*but use the spark.version=2.4.0 is ok*
mvn -Dspark.version=2.4.0 -Dscala.version=2.11.7 -Dscala.binary.version=2.11 -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
 
*other try*
mvn -Dspark.version=3.0.0 -Dscala.version=2.12.12 -Dscala.binary.version=2.12  -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install

return error 

{color:red}[ERROR]{color} [Error] F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:439: object SparkHadoopUtil in package deploy cannot be accessed in package org.apache.spark.deploy
[ERROR] [Error] F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:487: not found: value SparkHadoopUtil
{color:red}[ERROR]{color} two errors found

go to the [spark @github|https://github.com/apache/spark/blob/e1ea806b3075d279b5f08a29fe4c1ad6d3c4191a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala]
define SparkHadoopUtil  to private[spark] 
{code:java}
private[spark] class SparkHadoopUtil extends Logging {}
{code}
Bug: regions opened in read only mode failed when replaying the edits due to permission denied WRITE for regionDir
Status: Open | Priority: Blocker | Resolution: 
Created: 11/Jul/20 05:27

Open region in read only mode, [HbaseTableSnapshotInputFormat], if RegionDir has 

recovered edits files under, (filesUnderRootDir),  when read-only region read the content, it will try to delete the content of recovered edits,  including filesUnderRootDir part
{code:java}
// code placeholder
for (Path file : filesUnderRootDir) {
  if (!rootFS.delete(file, false)) {
    LOG.error("Failed delete of {}", file);
  } else {
    LOG.debug("Deleted recovered.edits file={}", file);
  }
}
{code}
however, read-only mode usually no WRITE access to delete it.
{code:java}
// code placeholder
org.apache.hadoop.security.AccessControlException: Permission denied: user=xxxx, access=WRITE, inode="/hbase/data/default/TABLENAME/d6777e449ed4d137b6a2f735653a7c3a/recovered.edits":hbase:hbase:drwxr-xr-x
{code}
it will throw excption and failed
Bug: Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 29/Jul/20 06:43
Affects Version/s: 3.0.0-alpha-1

Bulkload use HFileOutputFormat2 to write HFile 

In the  HFileOutputFormat2.RecordWriter

in the write method always called the getTableRelativePath method each time

This  is unnecessary
Bug: review Jenkins build artifacts
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 23/Aug/20 14:20

Post move to the ci-hadoop build servers we are now the biggest user of space. That is not a problem in and of itself, but the master node has run out of disk space twice now. As of this snapshot we are using 125GB of storage and the next largest project is only using 20GB.

https://paste.apache.org/kyrds

We should review our builds for any issues and come up with expectations for what our steady-state disk usage should look like

* we are supposed to compress any test logs (usually this gets us 90-99% space savings)
* we are supposed to clean up workspaces when jobs are done
* we are supposed to keep a fixed window of prior builds (either by days or number of runs)

If all of our jobs are currently following these guidelines, another possibility is to push the artifacts we need over to [nightlies.a.o|https://nightlies.apache.org/authoring.html]. Barring that, we should formally request asf infra set up [a plugin for storing artifact on s3|https://plugins.jenkins.io/artifact-manager-s3/].
Bug: WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used with multi operation
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 06/Sep/20 10:22
Affects Version/s: 2.0.0

After bugfix HBASE-22539, When client use BufferedMutator or multiple mutation , there will be one RpcCall and mutliple FSWALEntry .  At the time RpcCall finish and one FSWALEntry call release() , the remain FSWALEntries may trigger RuntimeException or segmentation fault .

We should use RefCnt  instead of AtomicInteger for org.apache.hadoop.hbase.ipc.ServerCall.reference?
Bug: HBASE shell update_config doesn't work for master node
Status: Open | Priority: Critical | Resolution: 
Created: 01/Sep/20 23:01
Affects Version/s: 2.3.1

The helper message says: 

Reload a subset of configuration on server 'servername' where servername is
host, port plus startcode. For example: host187.example.com,60020,1289493121758
See http://hbase.apache.org/book.html#dyn_config for more details. Here is how
you would run the command in the hbase shell:
 hbase> update_config 'servername'

 

But the command doesn't update config on master.
Bug: CellComparator perf improvement
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 11/Aug/20 02:45
Affects Version/s: 2.0.0

We have multiple perf issues in 2.x versions compared to 1.x.  Eg: HBASE-24754, HBASE-24637.
The pattern is clear that where ever we do more and more Cell compares, there is some degrade.   In HBASE-24754, with an old KVComparator style comparator, we see much better perf for the PutSortReducer.  (Again the gain is huge because of large number of compare ops that test is doing).  This issue is to address and optimize compares generally in CellComparatorImpl itself.
Bug: make a drop-in compatible impl of htrace APIs that does not do anything
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 30/Jul/20 18:46
Affects Version/s: 1.4.0

htrace-core4 is a retired project and even on the latest version they Shade Jackson databind version 2.4.0 which has the following CVEs:
|cve|severity|cvss|
|CVE-2017-15095|critical|9.8|
|CVE-2018-1000873|medium|6.5|
|CVE-2018-14718|critical|9.8|
|CVE-2018-5968|high|8.1|
|CVE-2018-7489|critical|9.8|
|CVE-2019-14540|critical|9.8|
|CVE-2019-14893|critical|9.8|
|CVE-2019-16335|critical|9.8|
|CVE-2019-16942|critical|9.8|
|CVE-2019-16943|critical|9.8|
|CVE-2019-17267|critical|9.8|
|CVE-2019-17531|critical|9.8|
|CVE-2019-20330|critical|9.8|
|CVE-2020-10672|high|8.8|
|CVE-2020-10673|high|8.8|
|CVE-2020-10968|high|8.8|
|CVE-2020-10969|high|8.8|
|CVE-2020-11111|high|8.8|
|CVE-2020-11112|high|8.8|
|CVE-2020-11113|high|8.8|
|CVE-2020-11619|critical|9.8|
|CVE-2020-11620|critical|9.8|
|CVE-2020-14060|high|8.1|
|CVE-2020-14061|high|8.1|
|CVE-2020-14062|high|8.1|
|CVE-2020-14195|high|8.1|
|CVE-2020-8840|critical|9.8|
|CVE-2020-9546|critical|9.8|
|CVE-2020-9547|critical|9.8|
|CVE-2020-9548|critical|9.8|

 

Our security team is trying to block us from using hbase because of this
Bug: CompactingMemStore.timeOfOldestEdit error update
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 27/Aug/20 03:03
Affects Version/s: 2.2.5

when use 'flush in memory', update timeOfOldestEdit every flush in memory, cause PeriodicMemStoreFlusher to not take effect, wals not free, constant backlog until maxlogs  triggers forced flush, makes failure recovery slower
Bug: The pre commit job fails to archive surefire reports
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 10/Aug/20 00:56

{noformat}
01:59:48  /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2212@tmp/durable-332ef1d8/script.sh: 4: /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2212@tmp/durable-332ef1d8/script.sh: [[: not found
01:59:48  + echo No archived files, skipping compressing.
01:59:48  No archived files, skipping compressing.
{noformat}
Bug: Refuse to update configuration of default group
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/20 07:49
Affects Version/s: 3.0.0-alpha-1

In the current scenario, we didn't store the default rsgroup information. But after HBASE-24431 , we have added a config map, which need to be persisted to avoid lossing config of default rsgroup.

So refuse to update configuration of default group
Bug: Make the cost functions optional for StochastoicBalancer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/20 21:19

StochastoicBalancer takes a long time to generate a plan for larger clusters. We want to make the cost functions optional.
hbase.master.balancer.stochastic.regionCountCost: 0
hbase.master.balancer.stochastic.primaryRegionCountCost: 0
hbase.master.balancer.stochastic.tableSkewCost: 0
hbase.master.balancer.stochastic.rackLocalityCost:0
hbase.master.balancer.stochastic.readRequestCost: 0
hbase.master.balancer.stochastic.cpRequestCost: 0
hbase.master.balancer.stochastic.writeRequestCost: 0
hbase.master.balancer.stochastic.regionReplicaHostCostKey: 0
hbase.master.balancer.stochastic.memstoreSizeCost: 0
hbase.master.balancer.stochastic.storefileSizeCost: 0
hbase.master.balancer.stochastic.regionReplicaRackCostKey: 0
hbase.master.balancer.stochastic.serverLocalityCost:0
 
This will turn off all cost functions
To keep some of them, we just need to set some to positive value