av9ash/GitBugs huggingface dataset:
Bug: Migrate hbase:storefile system table into editable meta
Status: Closed | Priority: Major | Resolution: Won't Fix
Created: 14/Dec/20 17:34

{color:#000000}editable meta supported by HBASE-23055 allow us to write new column family to hbase:meta, such that we will introduce a new column family cf:storefile to hbase:meta with different trackable columns to served as the in-memory tracking of e.g. `storefiles` and `compactedfile` in StoreFileManager.{color}
Bug:  Introduce PersistedStoreEngine and PersistedStoreFileManager
Status: Closed | Priority: Major | Resolution: Fixed
Created: 14/Dec/20 17:30
Affects Version/s: 3.0.0-alpha-1

Create a new wrapper store engine on top of the default store engine, as well as, introducing a new store file manager that keep tracking path of HFiles of user tables into a hbase self-managed system table (further will have a different tasks to merge into meta table and check throughput do not change much)
Bug: Revisit the implementation and usage of RegionStates.include
Status: Closed | Priority: Major | Resolution: Fixed
Created: 20/Oct/20 15:22

After several round of refactoring and fixing, the method has been used in lots of places and the implementation looks really confusing.

As in the first if condition for testing RegionStateNode and RegionInfo state, we will always return false when split is true, which means we will always filter out split parent, as a split parent, is split = true and also offline = true.

I think the reason why there is no problem is that, only in EnableTableProcedure we call this method with offline = true, and EnableTableProcedure does not need to deal with split parent...

And now since we found a problem in HBASE-25206, where we need to get split parent when deleting a table, I think it is time to revisit this method and make logic less confusing.
Bug: WALReader quits if nothing to replicate (and won't restart)
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 02/Oct/20 03:25

Noticed by [~huaxiangsun] in his review of HBASE-25055 "Add ReplicationSource for meta WALs; add enable/disable w…"
{quote}bq. Eventually, the meta wal file will be gced and there is no more logs in the queue. In that case, the walReader thread will quit. When the meta region is moved back, it does not seem that walReader thread will be restarted. So it seems that something is broken.
{quote}
This issue is about writing a test to run the above scenario and fix any probs. found.
Bug: [hbck2] Add an 'adoption service' for 'Orphaned Regions'
Status: Open | Priority: Major | Resolution: 
Created: 07/Oct/20 00:28

The 'HBCK Report' has a section for 'Orphaned Regions', regions in the filesystem that are no longer referenced by the running hbase. They should have been cleaned up as part of normal processing but for whatever reason, they were not. Usually these are dessicated directories with nothing in them but sometimes they might have an hfile or two. They could have content in recovered.edits directory too.

The "HBCK Report" page outline how to run the bulk load tool. This will pick up any hfiles in the 'Orphan Region' if there is worry that they have been dropped mistakenly.

For the content under 'recovered.edits', the WALPlayer has just been adjusted so it can pick up this content (See over in HBASE-22976).

The 'adoption service' would be run over an orphan region and it would apply the 'bulk load' if hfiles found and the WALPlayer if 'recovered.edits' found... it would then clean up the region directory on successful load after leaving audit so the 'orphan' was cleaned-up.

hbck2 tool would run the adoption service at first. Once we had some experience and confidence that the adoption service was running smoothly, we'd consider integrating it into the catalogjanitor.

The 'adoption service' first gets mention in the body of HBASE-21745
Bug: Filter out more invalid encoded name in isEncodedRegionName(byte[] regionName) 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/20 20:04

{code:java}
public static boolean isEncodedRegionName(byte[] regionName) {
  // If not parseable as region name, presume encoded. TODO: add stringency; e.g. if hex.
  return parseRegionNameOrReturnNull(regionName) == null && regionName.length <= MD5_HEX_LENGTH;
}

Right now, if it passes in an table name, it still thinks  it is a encoded region name and will result in unnecessary registry query for meta regions. This can be avoided if table names can be filtered out early in this method.{code}
Bug: [hbase-operator-tools] HBase metadata repair tool
Status: Open | Priority: Major | Resolution: 
Created: 07/Oct/20 09:04
Affects Version/s: hbase-operator-tools-1.1.0

_HBCK2_ versions should be able to work across multiple hbase-2 releases. It will fail with a complaint if it is unable to run. There is no {{HbckService}} in versions of hbase before 2.0.3 and 2.1.1. _HBCK2_ will not work against these versions.

When using HBase version 2.0.0 or 2.1.0

How to fix the metadata issue?
Bug: Undo meta replica specialization inside of the region replica framework
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Nov/20 16:37

Filing an issue to capture a [~zhangduo] suggestion made in PR that was then carried to a comment over in the design document attached to HBASE-18070 only it is deserving of its own issue.
{quote} 

I do not like that we have a specialized implementation for meta replica inside the region replica framework. They are almost the same so they should share the same code base. I think this enhancement could also be applied to normal RegionReplicaReplicationEndpoint, what we need is the config for the max distance between different secondary replicas. For normal region replicas, the value will be small, which means we will soon block the replication if any replicas are slow, to save memory. For meta replicas, we could have a large default value to let the good replicas catch up fast.
{quote}
 

Lets use this issue to fill out more on what [~zhangduo] suggests above.
Bug: Avoid the failed meta replica region temporarily in Load Balance mode
Status: Reopened | Priority: Major | Resolution: 
Created: 30/Nov/20 19:37
Affects Version/s: 2.5.0

This is a follow-up enhancement with Stack, Duo. With the newly introduced meta replica LoadBalance mode, if there is something wrong with one of meta replica regions, the current logic is that it keeps trying until the meta replica region is onlined again or it reports error, i.e, there is no HA at LoadBalance mode. HA can be implemented if it reports timeout with one meta replica region and tries another meta replica region.
Bug: [Umbrella]Fast split regions and lightweight compaction with stripe store engine
Status: Open | Priority: Major | Resolution: 
Created: 18/Nov/20 12:24
Affects Version/s: 3.0.0-alpha-2

In our company MeiTuan, we have encountered a problem, that is a production cluster with about more than 10000 regions set off the split switch by some mistakes. And more than 10000 regions are larger than 600GB before we can do something to split them. Because online real time R+W QPS of each RS is higher than 80000 (8000+ per region), if we just switch on split, the IO pressure brought by compactions will influence the availabilities, and all large regions need repeatedly compactions until the size of child regions are normal.

Different with the normal circumstances, we used stripe store engine in this cluster, after looking through docs of https://issues.apache.org/jira/browse/HBASE-7667 ,  we have implemented a fast split region method using the similar ideas of HFileLink and move store files between regions when compacting. Actually, this idea has already been mentioned in HBASE-7667 , it said that `region splits become marvelously simple (if we could move files between regions, no references would be needed)`.

We have implemented the ideas and practiced them on that cluster. The fact results show that, a region of about 600G can  be splitted to  75G*8 regions in about five minutes, with less than 5G total rewrite size(all are L0) in the whole process, while normal continuous split needs 600G*3=1800G. By using movement of store files, the rewritten size is less than 50G(two stripe size, because the rebuild of HFileLinks to stripes may insert some files to L0.).I have attached two pictures about a RS before and after splitting all the regions using this method.

This idea can not only be used in stripe store engine, but also default store engine, and the store movement methods are very benefit to region MERGE.
Bug: Support AES-192 and AES-256 in DefaultCipherProvider
Status: Open | Priority: Major | Resolution: 
Created: 18/Nov/20 16:58

The DefaultCipherProvider currently supports AES-128. In some security policies (such as the Application Security and Development STIG), AES-256 is required in certain situations.

I want to add AES-192 and AES-256 support. I quickly tried to implement this as part of HBASE-25263, but after 1-2 days I realized that it worths a separate task in Jira. The main challenge is that the key length and the algorithm needs to be decoupled in the code, and also some more tests need to be added to make sure we are backward-compatible and also supporting AES-192 and AES-256.

Beside defining a new algorithm and key on the Java API, I also want to make the usage of e.g. AES-256 in the shell, like:
{code:java}
create 'test', {NAME => 'cf', ENCRYPTION => 'AES-256', ENCRYPTION_KEY => 'mysecret'}
{code}
 

Also we should support AES-192 and AES-256 in master encryption keys. And we need to document how the users can configure / use it.
Bug: Find a way to config OpenTelemetry tracing without directly depending on opentelemetry-sdk
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/20 14:47

According to the document of OpenTelemetry, for all the modules which could be depended by downstream users, we should only depend on opentelemetry-api.

But the open telemetry propagator must be initialized programmatically, so we need to have a module to implement the code and introduce dependency on opentelemetry-sdk, and we need to call it before doing anything when starting master, regionserver, and so on.
Bug: [branch-1] Masters in-memory serverHoldings map is not cleared during hbck repair
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Oct/20 00:20
Affects Version/s: 1.7.0

{color:#1d1c1d}Incase of repairing overlaps, hbck  essentially calls the closeRegion RPC on RS followed by offline RPC on Master to offline all the overlap regions that would be merged into a new region. {color}

{color:#1d1c1d}However the offline RPC doesn’t remove it from the serverHoldings map unless the new state is MERGED/SPLIT ([https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java#L719]) b{color}{color:#1d1c1d}ut the new state in this case is OFFLINE. {color}

{color:#1d1c1d}This is actually intended to match with the META entries and would be removed later when the region is online on a different server. However, in our case , the region would never be online on a new server, hence the region info is never cleared from the map that is used by balancer and SCP for incorrect reeassignment.{color}

{color:#1d1c1d}We might need to tackle this by removing the entries from the map when hbck actually deletes{color}{color:#1d1c1d} the meta entries for this region which kind of matches the in-memory map’s expectation with the META state.{color}
Bug: [HBOSS] TreeLockManager should log warnings about lock contention periodically
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/20 17:50
Affects Version/s: hbase-filesystem-1.0.0-alpha1

While performing some internal tests with hboss, [~psomogyi] noticed that *TreeLockManager* class logs too frequently when it is waiting on parent write lock.
{noformat}
2020-10-26 17:35:41,719 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,720 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,721 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,723 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,724 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,725 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,726 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,728 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,729 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,730 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001 {noformat}
In 4 seconds on a single RegionServer this WARN was logged 1767 times.

This modifies current logging logic, to calculate the time spent trying to acquire locks on individual nodes, then warning only given a configurable threshold is exhausted.
Bug: Metrics recording a time duration should use System.nanoTime()
Status: Open | Priority: Major | Resolution: 
Created: 02/Oct/20 22:07

I was looking around in our table metrics and noticed that scan time is collected using {{EnvironmentEdgeManager.currentTime()}}. This isn't a stable way to measure a duration of time on the jvm. We should convert all these occurrences to use {{TimeUnit.NANOSECONDS.toMillis(System.nanoTime())}} instead.
Bug: [branch-2.2] Fix the ruby problem for pre-commit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/20 07:09
Affects Version/s: 2.2.6

*00:33:07*  [91mERROR:  Error installing rubocop:*00:33:07*  	parallel requires Ruby version >= 2.4.*00:33:08*  [0mSuccessfully installed jaro_winkler-1.5.4
Bug: Optionally abort requests in progress after deciding a region should close
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Oct/20 00:44

After deciding a region should be closed, the regionserver will set the internal region state to closing and wait for all pending requests to complete, via a rendezvous on the region lock. In closing state the region will not accept any new requests but requests in progress will be allowed to complete before the close action takes place. In our production we see outlier wait times on this lock in excess of several minutes. 

During close when there are requests in flight the regionserver is subject to any conceivable reason for delay, like full scans over large regions, expensive filtering hierarchies, bugs, or store level performance problems like slow HDFS. The regionserver should interrupt requests in progress to facilitate smaller/shorter close times on an opt-in basis.

Optionally, via configuration parameter -- which would be a system wide default set in hbase-site.xml in common practice but could be overridden in table schema for per table settings -- interrupt requests in progress holding the region lock rather than wait for completion of all operations in flight. Send back NotServingRegionException("region is closing") to the clients of the interrupted operations, like we do after the write lock is acquired. The client will transparently relocate the region data and resubmit the aborted requests per normal retry policy. This can be less disruptive than waiting for very long times for a region to close in extreme outlier cases (e.g. 50 minutes). In such extreme cases it is better to abort the regionserver if the close lock cannot be acquired in a reasonable amount of time, because the region cannot be made available again until it has closed.

After waiting for all requests to complete then we flush the region's memstore and finish the close. The flush portion of the close process is out of scope of this proposal. Under normal conditions the flush portion of the close completes quickly. It is specifically waits on the close lock that has been an occasional issue in our production that causes difficulty achieving 99.99% availability.
Bug: [branch-1] Fix the broken pre-commit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/20 13:08

{code}
Archive JUnit-formatted test results
[2020-11-20T02:58:50.501Z] Recording test results
[2020-11-20T02:58:50.641Z] No test report files were found. Configuration error?
{code}

branch-1's pre-commit keeps failing with error above, we need to fix it.
Bug: Rack awareness in region_mover
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/20 15:17

region_mover should provide an option to ensure while unloading all regions, all destination servers are selected from different racks and not the one where server (where region_mover unload is getting executed) belongs to. This might be helpful option if we want to avail rack downtime (or rack upgrade) by stopping all Regionservers that belong to same rack for few hours. Without this option, we don't have any control over which destination server is selected and hence, some regions might keep bouncing from server A to B in the same rack all the way until they are finally moved to RS that belongs to separate rack.
Bug: Sort broken after Change 'State time' in UI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/20 15:52

Sort broken after Change 'State time' in UI:

Looks like jquery tablesorter unable to parse the iso8601 date format hence not sorting.

We might need to add parser-date-iso8601.min.js to support sorting of iso8601 date format.
Bug: There is no limit on the column family length when creating a table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/20 00:58
Affects Version/s: 2.2.3

Table can be created successfully when the length of column family is larger than 127 bytes,however,it occurs exception when putting data into the table(ERROR：Family > 127).

So,We decide to verify the length of column family when creating a table.IllegalArgumentException will be thrown when length is larger than 127 bytes.
Bug: When openRegion fails during initial verification(before initializing and setting seq num), exception is observed during region close.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 05:29
Affects Version/s: 2.2.3

When openRegion fails during initial verification(before initializing and setting seq num), exception is observed during region close:

 

2020-12-03 16:34:47,133 ERROR [RS_OPEN_REGION-regionserver/AAAAAAAAAA:16040-0] handler.OpenRegionHandler: Failed open of region=ns2:testtable4,00000000000000000000150000,1606912406234.cd386135276b7d3c57416df3666e4aea.2020-12-03 16:34:47,133 ERROR [RS_OPEN_REGION-regionserver/blrphispra01054:16040-0] handler.OpenRegionHandler: Failed open of region=ns2:testtable4,00000000000000000000150000,1606912406234.cd386135276b7d3c57416df3666e4aea.java.io.IOException: The new max sequence id 1 is less than the old max sequence id 7134 at org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:418) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1253) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1793) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1606) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1552) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7522) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7467) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7439) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7397) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7348) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:286) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:111) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Bug: TagRewriteCell's getSerializedSize() method gets the size wrong
Status: Open | Priority: Major | Resolution: 
Created: 12/Dec/20 03:12
Affects Version/s: 3.0.0-alpha-1

We found that the serialized size of Cell is inconsistent with the size actually written to ByteBuffer when the tag is empty.
{code:java}
Cell originalCell = ExtendedCellBuilderFactory.create(CellBuilderType.DEEP_COPY)
  .setRow(Bytes.toBytes("row"))
  .setFamily(Bytes.toBytes("family"))
  .setQualifier(Bytes.toBytes("qualifier"))
  .setTimestamp(HConstants.LATEST_TIMESTAMP)
  .setType(KeyValue.Type.Maximum.getCode())
  .setValue(Bytes.toBytes("value"))
  .build();

ByteBuffer byteBuffer = ByteBuffer.allocate(100);
Cell cell = PrivateCellUtil.createCell(originalCell, "".getBytes());
int lengthWritten = KeyValueUtil.appendTo(cell, byteBuffer, 0, true);
System.out.println("Cell serializedSize: " + cell.getSerializedSize());
System.out.println("Bytes written to ByteBuffer: " + lengthWritten);

ByteArrayOutputStream output = new ByteArrayOutputStream(100);
PrivateCellUtil.writeCell(cell, output, true);
System.out.println("Bytes written to OutputStream: " + output.size());
{code}
Run the above code to see the following output

 
{code:java}
Cell serializedSize: 45
Bytes written to ByteBuffer: 43
Bytes written to OutputStream: 45
{code}
Bug: [HBCK2] Regenerate missing table descriptors by hbck2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Nov/20 07:51
Affects Version/s: hbase-operator-tools-1.0.0

We just have been in a situation, when we saw many ServerCrashProcedure to be blocked, because of the AssignProcedure is blocked because of the missing .tabledesc files.


{code:java}
java.io.IOException: Missing table descriptor for dd0676f57bdbff5d04ab735b7daf5e9b
{code}
 

In our case it was OK for us to get rid of these tables and we were able to use {{setRegionState}} to move all these regions to {{FAILED_OPEN}} state, then disable and drop the tables. But this took a lot of time and we might not always have the option to drop these tables.

HBCK 1 had a functionality (\{{fixTableOrphans}}) to regenerate the table descriptors "according to the memory cache or hdfs table directory structure". We should investigate if we can implement something like this with hbck2. We might not always be able to reproduce the table configurations, but we can probably make a "basic" one.