av9ash/GitBugs huggingface dataset:
Bug: Chaos Service Improvements
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Feb/23 16:36

We can improve the usage of chaos service to run random operations in the real cluster to verify the stability. Following things can be done.

1) Make use of the hbase script in the existing chaos-daemon.sh script instead of directly using the java command.

2) We can add a script to chaos server runner to run the script in the background.

3) Document usage of zookeeper based cluster manager mainly in the environments where ssh cannot be performed.

4) sudo is not required to kill the it's own user process so while running the commands need not use sudo.
Bug: Bump hbase, hbase-thirdparty, hadoop and spark for hbase-connectors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/23 14:05

Will update following as part of this JIRA
 * HBase: 2.4.16
 * HBase thirdparty: 4.1.4 (See [https://github.com/apache/hbase/blob/d1714710877653691e2125bd94b68a5b484a3a06/pom.xml#L634])
 * Hadoop3: 3.2.4 (We are bumping to latest stable 3.2.x)
 * Hadoop2: 2.10.0 (See 
 * [https://github.com/apache/hbase/blob/d1714710877653691e2125bd94b68a5b484a3a06/pom.xml#L543])
 * Spark: 3.2.3
 * Scala binary: 2.12 (See [https://github.com/apache/spark/blob/b53c341e0fefbb33d115ab630369a18765b7763d/pom.xml#L164]) 
 * Scala: 2.12.15 (See [https://github.com/apache/spark/blob/b53c341e0fefbb33d115ab630369a18765b7763d/pom.xml#L163])
Bug: Allow preCompact hooks to return scanners whose cells can be shipped
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/23 00:49

In [Compactor.java#L492|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java#L492], when the compaction scanner extends KeyValueScanner, we call shipped() periodically. This releases shared buffers which makes compactions less memory hungry.

When a preCompact hook returns a custom scanner, that scanner is unlikely to extend KeyValueScanner. So installing a preCompact hook can cause a regression in terms of memory pressure.

KeyValueScanner is a large interface with many methods, but the method being called (shipped()) is actually on the Shipper interface (which KeyValueScanner extends). It would be nice to allow coprocessors to more easily integrate here.

I can think of two options:
 # Simply change the cast on [line 436|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java#L436] from KeyValueScanner to Shipper. A coprocessor can easily return an InternalScanner impl which also extends Shipper since it's just one method. A common pattern will likely be to delegate to the underlying real scanner.
 # Add a new DelegateInternalScanner which coprocessors can return. In that case, call scanner.getDelegate() and check if _that_ extends KeyValueScanner.

I prefer option 1 as it's very simple and makes it really easy to reason about – a coprocessor may not _want_ to release cells, in which case they don't need to extend Shipper. The downside of this is we'd have to promote Shipper interface to IA.LimitedPrivate(COPROC)

Option 2 could open us to other possibilities in the future, and could be configurable. Like maybe it has a scanner.supportsShipped() or something. If true, the underlying scanner's shipped method will be called. If not, it won't.

Barring other opinions, I will go with option 1 as it's trivial.

cc [~anoopsamjohn] since you were involved in the linked change and the creation of Shipper interface. In the linked code, we don't use any other method on KeyValueScanner interface so it seems totally fine to use Shipper instead.
Bug: Cannot Specify Namespace via the hbase.table Option in Spark Connector
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/23 06:16
Affects Version/s: hbase-connectors-1.0.1

When using the old mapping format and specifying the HBase table via the 
_hbase.table_ option, the connector passes the namespaced string to HBase, and we get


{noformat}
Caused by: java.lang.IllegalArgumentException: Illegal character code:58, <:> at 7. User-space table qualifiers may only contain 'alphanumeric characters' and digits: staplesHbaseNamespace:staplesHbaseTableName
at org.apache.hadoop.hbase.TableName.isLegalTableQualifierName(TableName.java:187)
at org.apache.hadoop.hbase.TableName.isLegalTableQualifierName(TableName.java:138)
at org.apache.hadoop.hbase.TableName.<init>(TableName.java:320)
at org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:354)
at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:484){noformat}

This seems to be related to the changes in HBASE-24276
Bug: Merging empty regions corrupts meta cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/23 03:19

Let's say you have three regions with start keys A, B, C and all are cached in the meta cache. Region B is empty and not getting any requests, and all 3 regions are merged together. The new merged region has start key A.

A user submits a request for row C1, which would previously have gone to region C. That region no longer exists, so the MetaCache returns region C, the request goes out to the server which throws NotServingRegionException. That region C is now removed from the cache, and meta is scanned. The meta scan returns the newly merged region A, which is cached into the MetaCache.

So now we have a MetaCache where A has been updated with the newly merged RegionInfo, B still exists with the old/deleted RegionInfo, and C has been removed.

A user submits a request for row C1 again. This _should_ go to region A, but we do cache.floorEntry(C1) which returns the old but still cached region B. We have checks in MetaCache which validate the RegionInfo.getEndKey() against the requested row, and that validation fails because C1 is beyond the endkey of the old region. The cached region B result is ignored and cache returns null. Meta is scanned, and returns the new region A, which is cached again.

Requests to rows C1+ will still succeed... but they will always require a meta scan because the meta cache will always return that old region B which is invalid and doesn't contain the C1+ rows.

Currently, the only way this will ever resolve is if a request is sent to region B, which will cause a NotServingRegionException which will finally clear region B from the cache. At that point, requests for C1+ will properly get resolved to region A in the cache.

I've created a reproducible test case here: [https://gist.github.com/bbeaudreault/c82ff9f8ad0b9424eb987483ede35c12]

This problem affects both AsyncTable and branch-2's Table.
Bug: Fix the cached block differ (which should not have happened) issue with cacheDataOnWrite on
Status: Open | Priority: Major | Resolution: 
Created: 16/Feb/23 11:03
Affects Version/s: 2.5.3

When caching the block by reading from hfile, the block cached will contains a header with the next block info.

But when caching the block by writing when cacheDataOnWrite is on, the block cached without the header of the next block info.

Then when comparing the old and new blocks in BlockCacheUtil#validateBlockAddition, it will print error logs ,
{code:java}
hfile.BlockCacheUtil: Cached block contents differ, which should not have happened. cacheKey:XXX {code}
This will harm the actual replacement of the cached block, since we should replace or not when the cached block contents differ by nextBlockOnDiskSize.
Bug: WALPlayer does not properly dedupe overridden cell versions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/23 20:20

If you do 2 Puts to a cell with different values but the same timestamp, the latest one will win. This is because in the memstore we use a sequenceId as a tie breaker for duplicate timestamps. When the data is flushed to a StoreFile, the deduplication will occur and eventually the sequenceId will be dropped.

Those 2 Puts would have been added to the WAL, and if you use WALPlayer to replay those WALs (as anyone could do, but also as backup/restore does for incremental restores) it will not properly do the same thing. It's unclear which of the duplicate cells you will get, when you should always get the latest.

Our WAL encoder doesn't include the sequenceIds in the WALEntry cells. Instead the WALKey has a getSequenceId() which contains the same sequenceId the cells used to have. In WALCellMapper we don't pass those along, nor in CellSerialization, and thus CellSortReducer is not able to use the sequenceId to dedupe.

I think we just need to translate the WALKey.getSequenceId() into the output Cells in WALCellMapper, then update CellSerialization to include them as well. At that point CellSortReducer should work as expected, and we should get the correct cell values in the hfiles.

One open question is whether we should clear out the sequenceId before flushing to the hfile. I don't think so?
Bug: Set size of systable queue in UT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/23 06:52

TestSlowLogAccessor#testHigherSlowLogs failed in my CI pipeline.

!image-2023-02-23-14-52-38-867.png|width=654,height=409!

!image-2023-02-23-14-53-02-177.png|width=702,height=328!

SlowLogPersistentService use an [EvictingQueue|https://github.com/apache/hbase/blob/22dbb7afc383b9b9a8678f2c9eca1dca31784615/hbase-server/src/main/java/org/apache/hadoop/hbase/namequeues/SlowLogPersistentService.java#L58] to handle slowLogs. and default size is 1000.

So when the write speed is fast, this UT cannot guarantee that more than 2000 slowLogs will be stored in the table (they will be evicted in the queue).
Bug: IndexBlockEncoding is missing in HFileContextBuilder copy constructor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Feb/23 03:11
Affects Version/s: 2.5.3

{{IndexBlockEncoding}} is missing in {{HFileContextBuilder}} copy constructor, so if we want to construct a new {{HFileContext}}  by an existing {{HFileContext}}, we would missing the {{ndexBlockEncoding}} setting.
Bug: The "CREATE_TIME_TS" value of the hfile generated by the HFileOutputFormat2 class is 0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/23 12:36

When HFileOutputFormat2 is used to create an hfile, the CREATE_TIME_TS of hfileinfo will not be assigned when creating an hfileContext, resulting in the value of  lastMajorCompactionAge  is the timestamp of the current time.
{code:java}
##HFileOutputFormat2.class
HFileContextBuilder contextBuilder = new HFileContextBuilder()
                            .withCompression(compression)
                            .withChecksumType(HStore.getChecksumType(conf))
                            .withBytesPerCheckSum(HStore.getBytesPerChecksum(conf))
                            .withBlockSize(blockSize);

##get lastMajorCompactionTs metric

  lastMajorCompactionTs = this.region.getOldestHfileTs(true);
...
  long now = EnvironmentEdgeManager.currentTime();
  return now - lastMajorCompactionTs;
...
##
public long getOldestHfileTs(boolean majorCompactionOnly) throws IOException {
  long result = Long.MAX_VALUE;
  for (HStore store : stores.values()) {
    Collection<HStoreFile> storeFiles = store.getStorefiles();
   ...
    for (HStoreFile file : storeFiles) {
      StoreFileReader sfReader = file.getReader();
     ...
      result = Math.min(result, reader.getFileContext().getFileCreateTime());
    }
  }
  return result == Long.MAX_VALUE ? 0 : result;
}{code}
Bug: [JDK17] Use ReflectionUtils#getModifiersField in UT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/23 10:04

https://issues.apache.org/jira/browse/HBASE-25516 has solved an issue caused by more restrictive reflection protection of java.base classes in the newer java versions.

 

But there is still an old usage in branch-2.
Bug: [JDK17] Add-opens java.util.concurrent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/23 07:09

Met this failure UT base on JDK17:

 
{code:java}
[ERROR] Errors:
[ERROR]   TestOpenRegionFailedMemoryLeak.testOpenRegionFailedMemoryLeak:108 » InaccessibleObject Unable to make field private final java.util.concurrent.BlockingQueue java.util.concurrent.ThreadPoolExecutor.workQueue accessible: module java.base does not "opens java.util.concurrent" to unnamed module @20322d26
[INFO]
[ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0 {code}
It seems that since JDK 16 this kind of access was switched to deny by default ([https://openjdk.java.net/jeps/396])
Bug: Should not return OptionalLong.empty when the file is active but the write is still null
Status: Open | Priority: Major | Resolution: 
Created: 18/Feb/23 15:07

I think this could only happen when the FSWAL has not been initialized yet, in this way we should return OptionalLong.of(0) instead of OptionalLong.empty(). As in the replication implenentation, if we can make sure a file is not being written any more, we will dequeue it and go to the next file.

Although in the current implementation, it should not be a problem and we will only test beingWritten after reading from a WAL file, but since the FSWAL has not been initialized yet, the replication queue of this FSWAL should not be initialized too, so we are safe for now.

But anyway, this is still a potential bug if we change the implementation in the future, so let's change it OptionalLong.of(0) and let replication framework know that it should wait a bit.
Bug: Remove the impact of operating env on testHFileCleaning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jan/23 11:07

TestHFileCleaner#testHFileCleaning failed on my local, with this error:

{code:java}
2023-01-31T19:05:14,453 ERROR [hfile_cleaner-dir-scan-pool-0] mob.ManualMobMaintHFileCleaner(87): Failed to determine mob status of 'hdfs://localhost:59285/user/tianhang.tang/test-data/01a0d8a0-6306-7725-b63a-5debcab8cf57/archive/someHFileThatWouldBeAUUID.1675163114186', keeping it just in case.
java.lang.IllegalArgumentException: Illegal character <.> at 8. Namespaces may only contain 'alphanumeric characters' from any language and digits: tianhang.tang
        at org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:221) ~[classes/:?]
        at org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:196) ~[classes/:?]
        at org.apache.hadoop.hbase.TableName.<init>(TableName.java:316) ~[classes/:?]
        at org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:349) ~[classes/:?]
        at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:387) ~[classes/:?]
        at org.apache.hadoop.hbase.util.CommonFSUtils.getTableName(CommonFSUtils.java:440) ~[classes/:?]
        at org.apache.hadoop.hbase.mob.ManualMobMaintHFileCleaner.isFileDeletable(ManualMobMaintHFileCleaner.java:62) ~[classes/:?]
        at org.apache.hbase.thirdparty.com.google.common.collect.Iterators$5.computeNext(Iterators.java:673) ~[hbase-shaded-miscellaneous-4.1.4.jar:4.1.4]
        at org.apache.hbase.thirdparty.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[hbase-shaded-miscellaneous-4.1.4.jar:4.1.4]
        at org.apache.hbase.thirdparty.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[hbase-shaded-miscellaneous-4.1.4.jar:4.1.4]
        at org.apache.hbase.thirdparty.com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1391) ~[hbase-shaded-miscellaneous-4.1.4.jar:4.1.4]
        at org.apache.hadoop.hbase.master.cleaner.HFileCleaner.deleteFiles(HFileCleaner.java:194) ~[classes/:?]
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:391) ~[classes/:?]
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$1(CleanerChore.java:483) ~[classes/:?]
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore.deleteAction(CleanerChore.java:565) ~[classes/:?]
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore.traverseAndDelete(CleanerChore.java:483) ~[classes/:?]
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$chore$0(CleanerChore.java:258) ~[classes/:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
{code}

That's because the mocked dir is too simple, need to add a whole namespace/table/region path.
Bug: Bulkload fails when trying to bulkload files with invalid names after HBASE-26707
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/23 14:29
Affects Version/s: 2.5.3

HBASE-26707 has introduced changes to reduce renames on bulkload when using FILE based SFT. However, if the bulkloading file has an invalid hfile name, or has been split in the bulkload process, we don't do any renaming when FILE based SFT is enabled, and we place the file name as "it is" in the store dir. This later fails the validations performed by StoreFileReader when it tries to open the file.

This jira adds extra validation for the bulkloading file name format in HRegion.bulkLoadHFiles and also extends TestLoadIncrementalHFiles to run the same test suite with FILE based SFT enabled.
Bug: Optimize INFO message after HBASE-27498
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 31/Jan/23 23:24
Affects Version/s: 2.4.16

During the RC vote of 2.4.16 and 2.5.3, I found the below message should be removed or hidden from the connection operation. 

{code}
2023-01-31T15:19:03,890 INFO  [main] client.ConnectionImplementation: Getting master connection state from TTL Cache
2023-01-31T15:19:03,891 INFO  [main] client.ConnectionImplementation: Getting master state using rpc call
2023-01-31T15:19:03,997 INFO  [main] client.ConnectionImplementation: Getting master connection state from TTL Cache
2023-01-31T15:19:03,997 INFO  [main] client.ConnectionImplementation: Getting master state using rpc call
2023-01-31T15:19:04,201 INFO  [main] client.ConnectionImplementation: Getting master connection state from TTL Cache
2023-01-31T15:19:04,202 INFO  [main] client.ConnectionImplementation: Getting master state using rpc call
2023-01-31T15:19:04,508 INFO  [main] client.ConnectionImplementation: Getting master connection state from TTL Cache
2023-01-31T15:19:04,508 INFO  [main] client.ConnectionImplementation: Getting master state using rpc call
{code}
Bug: Complete Release 2.5.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/23 16:52

# Release the artifacts on repository.apache.org
# Move the binaries from dist-dev to dist-release
 # HBASE-27611 Add to download page
 # Push tag 2.5.3RC1 as tag rel/2.5.3
 # Release 2.5.3 on JIRA [https://issues.apache.org/jira/projects/HBASE/versions/12352572]
 # Add release data on [https://reporter.apache.org/addrelease.html?hbase]
 # Send announcement email

(learnt from HBASE-27612)
Bug: Unify tracking of block IO across all read request types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/23 19:16

Various requests in RSRpcServices use [addSize|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L1303-L1335] in an attempt to estimate block IO of the request by pulling the capacity of the underlying cell buffer when that buffer changes. This is just an estimate and can be inaccurate in certain circumstances, such as when the ordering of gets in a multiget causes the same buffer to be counted twice.

As of HBASE-27558, we now have a hook in StoreScanner for tracking block IO via HFileReaderImpl.reportBlockSize method call. We use this for getting block IO into ScannerContext for tracking limits. We can add a call to RpcCall.incrementResponseBlockSize here so that we can get an accurate accounting of block IO for all request types.
Bug: Region Reopen failure when the openNum has issue
Status: Open | Priority: Major | Resolution: 
Created: 05/Feb/23 22:52

We faced the issue when change the TTL for the hbase table and a lot of regions keep reopen and tons of TRSP created, after troubleshooting, we found some logic issue for the region reopen procedure logic.

In the reopen process, it will check the seqNum to confirm if the region reopened successfully or not. If the seqNum accident become bigger than the current HFile and WAL (because of the data loss), there will be issue and unnecessary loop for the region close/open

 

We should be able to optimize the logic, more details

For this regionOpenedWithoutPersistingToMeta, should we just update the OpenSeqNum when the new one is bigger than the old one?

As the region already opened, we should update the OpenSeqNum no matter its bigger or smaller, otherwise, we should not just return WARN but failed the open, right?

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]

 

Above does matter because for the checkReopened([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L312]), if the seq is smaller, the region will be returned and keep reopening.  So we should either update the logic in regionOpenedWithoutPersistingToMeta or checkReopened to make sure the region reopen works properly if the seqNum has issue

 

 

Reproduce steps:

 

1. {{{}Create a test table and put some data, for example:{}}}{{{}test{}}}
{{create 'test', 'info'}}
{{put 'test', 'fool', 'info:cat', 'test'}}

{{2. Manually update one region row for this test table in hbase:meta on the column, for example:}}

{{put 'hbase:meta', 'test,,1673406566311.3eb4d3e0258bd06f4639a595920c7673.', 'info:seqnumDuringOpen', "\x00\x00\x00\x00\x00\x10\x00\x05"}}

3. Modify the table TTL :

alter 'test', \{NAME=>'info' , TTL => '63244800'}}}

 

You will see the region keep reopening
Bug: Make the behavior of hadoop profile consistent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jan/23 03:51

In HBASE-27599 I found an issue that should cause build failure, but in fact both the community and my local build jobs succeed, which is not as expected.

The reason is, the activation condition in hbase-shaded-testing-util module is different with other modules, and it will not activate by default.
Bug: Cleanup wrong dependency in dependencyManagement
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jan/23 03:08

In HBASE-27340 added a hadoop-3.0 profile in hbase-shaded-testing-util pom, but missed the version of hadoop-common test jar: [HBASE-27340|https://github.com/apache/hbase/commit/1b5403cf7d906a22c20bd4f025566a2a5f895d44]
 !screenshot-2.png! 

This will cause build failure:
 !screenshot-1.png! 

But I still don't know why this problem does not necessarily arise...
Bug: Use lowercase image reference name in our docker file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/23 15:07

https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4966/5/consoleFull

The build failed with

{noformat}
19:46:22  #1 [internal] load build definition from Dockerfile
19:46:22  #1 transferring dockerfile: 7.66kB done
19:46:22  #1 DONE 0.1s
19:46:22  
19:46:22  #2 [internal] load .dockerignore
19:46:22  #2 transferring context: 2B done
19:46:22  #2 DONE 0.1s
19:46:22  Dockerfile:117
19:46:22  --------------------
19:46:22   115 |     #
19:46:22   116 |     
19:46:22   117 | >>> FROM BASE_IMAGE
19:46:22   118 |     SHELL ["/bin/bash", "-o", "pipefail", "-c"]
19:46:22   119 |     
19:46:22  --------------------
19:46:22  ERROR: failed to solve: failed to parse stage name "BASE_IMAGE": invalid reference format: repository name must be lowercase
{noformat}

Changing BASE_IMAGE to base_image can fix the problem.