av9ash/GitBugs huggingface dataset:
Bug: Clean shared state pollution to avoid flaky tests
Status: Open | Priority: Minor | Resolution: 
Created: 07/Oct/21 08:10

h2. What is the purpose of this change
 * Fixing a flaky test to avoid shared state pollution {{org.apache.hadoop.hbase.replication.regionserver.TestDrainReplicationQueuesForStandBy.test}}

h2. Reproduce the test failure
 * Run the test twice in the same JVM.

h2. Expected result:
 * The test should run successfully when multiple tests that use this state are run in the same JVM.

h2. Actual result:
 * We get the failure:

 {{org.apache.hadoop.hbase.DoNotRetryIOException: Can not transit current cluster state from STANDBY to STANDBY for peer id=1}}
h2. Why the test fails
 * When the test ends, current cluster for peer id=1 is STANDBY, it can not be transitted from STANDBY to STANDBY.

h2. Fix
 * Reset the state when test ends.
Bug: Make enableReplicationPeer/disableReplicationPeer idempotent
Status: Open | Priority: Minor | Resolution: 
Created: 11/Nov/21 01:31

When enableReplicationPeer is called and the peer is already enabled, DoNotRetryIOException is being thrown as part of preEnablePeer [here|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java#L164], similarly with disableReplicationPeer [here|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java#L171].

java.lang.RuntimeException: org.apache.hadoop.hbase.DoNotRetryIOException: Replication peer 1 has already been enabled

Ideally, it should not throw a runtimeException, if the peer is already in desired state


Either 1. we should add a check before trying to enable/disable peer, and if it is already enabled return, else enable the peer or
2. Log the message instead of throwing exception in preEnablePeer/preDisablePeer
Bug: Report CallDroppedException in exception metrics
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Dec/21 19:02

`CallDroppedException` can be thrown with `CallRunner.drop()` by queue implementations that decide to drop calls to groom the RPC call backlog. The LifoCoDel queue does this I believe and with Pluggable queue it's possible for 3rd party queue implementations to be using `drop()` for similar reasons (my use case). It would be nice for the server to be tracking these exceptions in metrics since otherwise you might have to do some extra lifting on the client side.

 

I have a patch I will submit soon.
Bug: [branch-2, create-release] Private per build Maven caches won't work on aarch64 MacOS
Status: Open | Priority: Minor | Resolution: 
Created: 18/Dec/21 01:24

Private per build Maven caches won't work on aarch64 MacOS. The problem is we have to compile a patched version of protobuf 2.5.0 binaries and install them into the local cache. (See this gist: https://gist.github.com/liusheng/64aee1b27de037f8b9ccf1873b82c413 . ) Typically the developer will do this once, the results placed into the standard local cache $HOME/.m2/repository . However because the create-release scripts use unconditionally a private repo and similar workarounds for aarch64 MacOS systems are not known to these scripts, the result is a build failure:

{noformat}
17:13:57 [INFO] BUILD FAILURE
17:13:57 [ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.6.1:compile (compile-protoc) on project hbase-protocol: Unable to resolve artifact: Missing:
17:13:57 [ERROR] ----------
17:13:57 [ERROR] 1) com.google.protobuf:protoc:exe:osx-aarch_64:2.5.0
17:13:57 [ERROR] 
17:13:57 [ERROR]   Try downloading the file manually from the project website.
17:13:57 [ERROR] 
17:13:57 [ERROR]   Then, install it using the command: 
17:13:57 [ERROR]       mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file
17:13:57 [ERROR] 
17:13:57 [ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
17:13:57 [ERROR]       mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
17:13:57 [ERROR] 
17:13:57 [ERROR]   Path to dependency: 
17:13:57 [ERROR]   	1) org.apache.hbase:hbase-protocol:jar:2.4.9
17:13:57 [ERROR]   	2) com.google.protobuf:protoc:exe:osx-aarch_64:2.5.0
17:13:57 [ERROR] 
17:13:57 [ERROR] ----------
17:13:57 [ERROR] 1 required artifact is missing.
17:13:57 [ERROR] 
17:13:57 [ERROR] for artifact: 
17:13:57 [ERROR]   org.apache.hbase:hbase-protocol:jar:2.4.9
17:13:57 [ERROR] 
17:13:57 [ERROR] from the specified remote repositories:
17:13:57 [ERROR]   apache.snapshots (https://repository.apache.org/snapshots, releases=false, snapshots=true),
17:13:57 [ERROR]   central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)
{noformat}
Bug: Replace new allocation with ThreadLocal in CellBlockBuilder to reduce GC
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 18/Dec/21 11:08

In CellBlockBuilder decompress method, we currently allocate a new ByteBufferOutputStream object each invoke. 

{code:java}
try {
      // TODO: This is ugly. The buffer will be resized on us if we guess wrong.
      //TODO: reuse buffers.
      bbos = new ByteBufferOutputStream(osInitialSize);
      IOUtils.copy(cis, bbos);
      bbos.close();
      return bbos.getByteBuffer();
     } finally {
      CodecPool.returnDecompressor(poolDecompressor);
    }
{code}

We can use a ThreadLocal variable to reuse the buffer in each Thread. As:

{code:java}
try {
      // TODO: This is ugly. The buffer will be resized on us if we guess wrong.
      if (this.decompressBuff.get() == null) {
        this.decompressBuff.set(new ByteBufferOutputStream(osInitialSize));
      }
      ByteBufferOutputStream localBbos = this.decompressBuff.get();
      localBbos.clear();
      IOUtils.copy(cis, localBbos);
      return localBbos.getByteBuffer();
    } finally {
      CodecPool.returnDecompressor(poolDecompressor);
    }
{code}
Bug: Introduce a new parameter in jmx servlet to exclude the specific mbean
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Dec/21 15:35
Affects Version/s: 2.4.8

There are many regionservers serving over a thousand regions in our company, and the metric load is pretty big.
I tried to exclude some huge mbean like 'Hadoop:service=HBase,name=RegionServer,sub=Regions' with regex, but didn't succeed.
So I want to propose a new parameter 'excl' in jmx servlet to exclude the splecific bean or beans.
Bug: Make sure the master is running normally before accepting a balance command
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Dec/21 00:22

Noticed this in a unit test from [https://github.com/apache/hbase/pull/3851]

I believe this is a result of the new balance() implementation in the Master, and a client submitting a request to the master before it's completed its instantiation. Simple fix to avoid the NPE.
{noformat}
Failed after attempts=2, exceptions:
2021-12-07T18:14:19.203Z, java.io.IOException: java.io.IOException
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:461)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:138)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:354)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:334)
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hbase.master.HMaster.getProcedures(HMaster.java:3335)
	at org.apache.hadoop.hbase.master.ServerManager.areDeadServersInProgress(ServerManager.java:505)
	at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1886)
	at org.apache.hadoop.hbase.master.MasterRpcServices.balance(MasterRpcServices.java:740)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393)
	... 3 more {noformat}
Bug: TestHStore#testRefreshStoreFiles broken due to unqualified and qualified paths
Status: Open | Priority: Minor | Resolution: 
Created: 18/Dec/21 22:25
Affects Version/s: 2.4.9

Was looking at a failures of this method where 
{noformat}
[ERROR] org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles  Time elapsed: 4.2 s  <<< ERROR!
java.util.NoSuchElementException
    at org.apache.hbase.thirdparty.com.google.common.collect.AbstractIndexedListIterator.next(AbstractIndexedListIterator.java:75)
    at org.apache.hadoop.hbase.regionserver.TestHStore.closeCompactedFile(TestHStore.java:962)
    at org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles(TestHStore.java:1000) {noformat}
This was on a branch where I had some HBASE-26067 changes backported, so I thought the problem _was_ those changes. After a bit of digging, I believe the test case itself is "broken" (the test passes, but for the wrong reasons).

This test methods adds some files to a Store (via memstore flush or direct addition of a file) and eventually tries to get the first file which is candidate to be removed. The test {*}never compacted any files{*}. This was the first sign that the test itself was wrong.

After lots of comparison with the HBASE-26067 logging to compare against, I found that the Store was listing a file which was created by the memstore flush as a file to retain AND a file to remove. Second warning. Upon closer inspection, I finally noticed that one of the files was qualified with the filesystem URI and the other was not.
{noformat}
2021-12-18 16:57:10,903 INFO  [Time-limited test] regionserver.HStore(675): toBeAddedFiles=[file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/dd8a430b391546d8b9bdc39bb77d447b, file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73], toBeRemovedFiles=[/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73] {noformat}
{{d4c5442b772c43fd9ebdfed1a11c0e73}} how are we both adding and removing this file! Turns out, this is because one of them is "/..." and the other is "file:/...". Either the problem is in TestHStore in how it is creating/adding these files behind the scenes or we should be qualifying the Path inside of StoreFileInfo with the filesystem that we're using.

I remember too vividly the problems when trying to separate the rootdir and waldir from each other and am cautious against adding something to StoreFileInfo to {{{}fs.qualifyPath(p){}}}. Need to look some more, but will get a patch up to fix.
Bug: Separate the blockcache enabled key and cache data on read key
Status: Resolved | Priority: Minor | Resolution: Won't Do
Created: 29/Nov/21 11:22

Currently, we use CACHE_DATA_ON_READ_KEY as BLOCKCACHE_ENABLED. This makes the usage ambiguous. 
In my opinion, we should use a flag to control the on/off of the whole blockcache and one another flag to control if we cache data blocks on read (cache on write stay the same).
I added a new BLOCKCACHE_ENABLED key to make the name and usage more clear. However, there are some cache logic changes, for which I changes the related UTs a little bit in my MR. 
Hope to have a discussion about the cache logic then.
Bug: Procedure state pretty-printing should use printable character friendly encoding
Status: Resolved | Priority: Minor | Resolution: Later
Created: 10/Nov/21 22:24
Affects Version/s: 2.4.8

The shell 'list_procedures' command produces output like:

----

 889 org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure SUCCESS 2021-11-10 22:20:34 UTC 2021-11-10 22:20:35 UTC [{"state"=>[1, 2, 3, 11, 4, 5, 6, 7, 8, 9, 10, 2147483648]}, {"regionId"=>"1636579678894", "tableName"=>{"namespace"=>"ZGVmYXVsdA==", "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"}, "startKey"=>"dWsuY28uZ3Jhbml0ZXRyYW5zZm9ybWF0aW9ucy53d3d8L2dhbGxlcnkvdA==", "endKey"=>"dXMuYmFuZHwvYmFuZC81OA==", "offline"=>false, "split"=>false, "replicaId"=>0}, {"userInfo"=>{"effectiveUser"=>"apurtell"}, "parentRegionInfo"=>{"regionId"=>"1636579678894", "tableName"=>{"namespace"=>"ZGVmYXVsdA==", "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"}, "startKey"=>"dWsuY28uZ3Jhbml0ZXRyYW5zZm9ybWF0aW9ucy53d3d8L2dhbGxlcnkvdA==", "endKey"=>"dXMuYmFuZHwvYmFuZC81OA==", "offline"=>false, "split"=>false, "replicaId"=>0}, "childRegionInfo"=>[{"regionId"=>"1636582834759", "tableName"=>{"namespace"=>"ZGVmYXVsdA==", "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"}, "startKey"=>"dWsuY28uZ3Jhbml0ZXRyYW5zZm9ybWF0aW9ucy53d3d8L2dhbGxlcnkvdA==", "endKey"=>"dWsuY28uc2ltb25hbmRzY2h1c3Rlci53d3d8L2Jvb2tzL1RoZS1P", "offline"=>false, "split"=>false, "replicaId"=>0}, {"regionId"=>"1636582834759", "tableName"=>{"namespace"=>"ZGVmYXVsdA==", "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"}, "startKey"=>"dWsuY28uc2ltb25hbmRzY2h1c3Rlci53d3d8L2Jvb2tzL1RoZS1P", "endKey"=>"dXMuYmFuZHwvYmFuZC81OA==", "offline"=>false, "split"=>false, "replicaId"=>0}]}]

----

The base64 encoding of byte[] values offers poor usability. 

Generally, table names etc are printable characters encoded in byte[]. Base64 encoding them totally obfuscates information that is important to see at a glance. Even start keys and end keys might be printable characters.

It would be better to use Bytes.toStringBinary or something else that shows printable characters as-is while escaping others that would be invalid in JSON formatting.
Bug: `MAXIMUN_KEY_LENGTH` changed from 0.20.0 to 0.20.2
Status: Open | Priority: Minor | Resolution: 
Created: 01/Nov/21 01:01
Affects Version/s: 0.20.0

In HFile.java, there is a constant called `MAXIMUM_KEY_LENGTH`, which is used to check the validity of the key. However, the value changed from 64 * 1024 to Integer.MAX_VALUE. Therefore, I wonder whether it might throw an exception when checking the key after downgrading from 0.20.2 to 0.20.0.
Bug: Introduce a method to clean restore directory after Snapshot Scan
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Nov/21 16:44

SnapshotScan is widely used in our company. However, after the snapshot scan job, the restore directory is not cleaned, and this maybe puts a lot of pressure on HDFS after a long time. So maybe we can introduce a method for users to clean the snapshot restore directory after job.
Bug: StoreFileScanner parallel seek -- productionize or drop?
Status: Open | Priority: Minor | Resolution: 
Created: 30/Nov/21 15:51

hbase.storescanner.parallel.seek.enable was added a few years ago in https://issues.apache.org/jira/browse/HBASE-7495, but still defaults to disabled. The description of that says "Enables StoreFileScanner parallel-seeking in StoreScanner, a feature which can reduce response latency under special conditions".

It's not very clear what "special conditions" means. Reading through the entire comment history on that issue seems to indicate it can help when you have "high random read, low cache hit rate, many store files". 

We have a bunch of clusters with this shape, and in fact we use SSDs for all storage so I figured this might help a lot. I tried setting this to true on one RegionServer of one of our highest QPS clusters hoping I'd see some clear improvement. This very simple test was pretty much a wash, so I need to do more methodical testing.

In the test one thing became clear though – is the default thread pool size of 10 good enough for my use-case? I have no way of knowing, as there is no logging or metrics that I can find around thread pool saturation. What I ended up doing was spamming refresh of the /dump endpoint of the RS, and noticed that there were sometimes 1-5 tasks queued for the RS_PARALLEL_SEEK executor. This indicates maybe I should scale the thread pool, but use-cases change over time so this seems like not a great way to determine that.

Task queuing seems not great for a feature which is aimed at reducing latencies. I wonder if we should consider some changes to make this more easy to deploy in production. Here are some ideas I had:
 * Can we generate a better default value for the thread pool size, maybe based on number of RS handler threads or some other heuristic?
 * Should we consider eliminating queuing for this feature? Instead, if the threadpool is saturated run the seek in-line in the current thread (i.e. revert to normal). This would be more similar to how hedged reads work in HDFS.
 * Can we expose a metric or logging to help operators know when to scale up the thread pool? If we implemented the 2nd option above we could expose "seeksInCurrentThread" counter to track this, again similar to how hedged reads report on saturation.

But with all of this said, I wonder if anyone is running this in production and has any updated guidance on when to use this? Does it still make sense given the last 8 years of development in HBase? Would it ever make sense to make it enabled by default?
Bug: Value of hbase.master.snapshot.ttl is not used
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Nov/21 23:32
Affects Version/s: 1.7.1

When creating a snapshot, users can explicitly specify the TTL to be used. If no TTL is specified, then the SnapshotDescription is initially created with a TTL of -1 to indicate FOREVER.

When the SnapshotDescription runs through SnapshotDescriptionUtils#validate, the TTL is checked to see if the default value of hbase.master.snapshot.ttl should be applied. The value from the config is only applied if the TTL == 0, but it should be -1.

This has another nasty side-effect: any user who creates a snapshot and explicitly sets \{TTL => 0} will find that their snapshot gets its TTL from hbase.master.snapshot.TTL.
Bug: Print too slow/big scan's operation_id in region server log
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Nov/21 11:40

Tracing is very important in large-scale distributed systems.

The attribute *__operation.attributes.id_*  of a scan request can be used as trace id between hbase client and region server.

We should print operation id in region server's log if the scan request is too slow or too big.

It will be very helpful for finding problematic requests.
Bug: Table name and Column Family Name not always initialized in the HFileContext
Status: Open | Priority: Minor | Resolution: 
Created: 12/Nov/21 09:09
Affects Version/s: 2.4.8

{noformat}
Example:{noformat}
{noformat}
2021-10-21 01:48:42,860 WARN  [main-BucketCacheWriter-0] bucket.BucketCache: Most recent failed allocation in 60000 milliseconds; Table
Name = null, Column Family = null, HFile Name : 1919692f80b841b7a3d4453ec01186f1
org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocatorException: Allocation too big size=1050637; adjust BucketCache sizes hbase.bucket
cache.bucket.sizes to accomodate if size seems reasonable and you want it cached.
{noformat}
Bug: Support move linked data files between regions of the same table in StripeCompactionPolicy
Status: Open | Priority: Minor | Resolution: 
Created: 04/Nov/21 12:54

There are two important contents for this issue,
 # must add reference files for moved data files in a dependent directory which will be looked through to find out which region the data file is currently in;
 # for HFileLinks in the same table of snapshot table, must support these links to add more locations according to the move info.
Bug: Increase DefaultHeapMemoryTuner log level
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Nov/21 01:35
Affects Version/s: 2.4.8

The DefaultHeapMemoryTuner chore will explain what it is doing when deciding to make changes, once per chore interval. This is not frequent logging and deserves to be at INFO level as it is making changes of some import.  While making the change, also log a line when ignoring the initial periods, so operators are not surprised when nothing happens for a while.
Bug: Support loadable dictionaries in hbase-compression-zstd
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Oct/21 01:44

ZStandard supports initialization of compressors and decompressors with a precomputed dictionary, which can dramatically improve and speed up compression of tables with small values. For more details, please see [The Case For Small Data Compression|https://github.com/facebook/zstd#the-case-for-small-data-compression]. 

If a table is going to have a lot of small values and the user can put together a representative set of files that can be used to train a dictionary for compressing those values, a dictionary can be trained with the {{zstd}} command line utility, available in any zstandard package for your favorite OS:

Training:
{noformat}
$ zstd --maxdict=1126400 --train-fastcover=shrink \
    -o mytable.dict training_files/*
Trying 82 different sets of parameters
...
k=674                                      
d=8
f=20
steps=40
split=75
accel=1
Save dictionary of size 1126400 into file mytable.dict
{noformat}

Deploy the dictionary file to HDFS or S3, etc.

Create the table:

{noformat}
hbase> create "mytable", 
  ... ,
  CONFIGURATION => {
    'hbase.io.compress.zstd.level' => '6',
    'hbase.io.compress.zstd.dictionary' => 'hdfs://nn/zdicts/mytable.dict'
  }
{noformat}

Now start storing data. Compression results even for small values will be excellent.

Note: Beware, if the dictionary is lost, the data will not be decompressable.
Bug: CellCounter fails for large tables filling up local disk
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Oct/21 17:18
Affects Version/s: 2.2.7

CellCounter dumps all cell coordinates into its output, which can become huge.

The spill can fill the local disk on the reducer. 
CellCounter hardcodes *mapreduce.job.reduces* to *1*, so it is not possible to use multiple reducers to get around this.

Fixing this is easy, by not hardcoding *mapreduce.job.reduces*, it still defaults to 1, but can be overriden by the user. 

CellCounter also generates two extra records with constant keys for each cell, which have to be processed by the reducer.
Even with multiple reducers, these (1/3 of the totcal records) will go the same reducer, which can also fill up the disk.

This can be fixed by adding a Combiner to the Mapper, which sums the counter records, thereby reducing the Mapper output records to 1/3 of their previous amount, which can be evenly distibuted between the reducers.
Bug: Remove duplicate thread creation during migrating rsgroup
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Oct/21 00:10
Affects Version/s: 3.0.0-alpha-2

There is a thread that migrate the table rs group info from RSGroupInfo into the table descriptor.

The thread is created when RSGroupManager is initialized and is created again when RSGroupStartupWorker is started.

I think this is a bug. Since this thread will not exit until all table rsgroups are migrated, there is no reason to need two threads to run together?
Bug: Improve recent change to IntegrationTestLoadCommonCrawl
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Oct/21 20:16

Using BufferedMutator wasn't a good idea because we assign client timestamps, and the store loop is fast enough that on rare occasion two temporally adjacent URLs in the set of WARCs are equivalent and the timestamp does not advance, leading later to a rare false positive CORRUPT finding. 

While making changes, support direct S3N paths as input paths on the command line.