av9ash/GitBugs huggingface dataset:
Bug: TableSnapshotInputFormat/TableSnapshotInputFormatImpl should use ReadType.STREAM for scanning HFiles 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/21 18:34
Affects Version/s: 2.4.6

After the change in HBASE-17917 that use PREAD ({{ReadType.DEFAULT}}) for all user scan, the behavior of TableSnapshotInputFormat changed from STREAM to PREAD.

TableSnapshotInputFormat is supposed to be use with a YARN/MR or other batch engine that should read the entire HFile in the container/executor, with default always to PREAD, we executing a lot more DFSInputStream#seek calls to simply read through the datablock section of the HFile.

The goal of this change is to make any downstream using TableSnapshotInputFormat with STREAM scan.
Bug: Protobuf-java-2.5.0.jar Has Several Security Vulnerabilities，CVE-2015-5237，CVE-2019-15544
Status: Open | Priority: Major | Resolution: 
Created: 28/Aug/21 09:22
Affects Version/s: 2.2.3

CVE-2019-15544：

Vulnerability Description：An issue was discovered in the protobuf crate before 2.6.0 for Rust. Attackers can exhaust all memory via Vec::reserve calls.

CVE-2015-5237：

Vulnerability Description：protobuf allows remote authenticated attackers to cause a heap-based buffer overflow.
Bug: Skip archived master wals during incremental backups
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/21 17:20

I've been reading through the incremental backup implementation in master branch to see how it handled some scenarios our own internal incremental backup process has to handle. One such failure we recently encountered as part of our ongoing hbase2 upgrade is the new $masterlocalwal$ suffixed files in the oldWALs dir. Our parsing of the WAL files assumed that the last part of the file name would be a timestamp, which is not the case for these MasterRegion WALs.

I see [IncrementalBackupManager excludes ProcV2Wals|https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalBackupManager.java#L104-L117], but I think that was replaced in https://issues.apache.org/jira/browse/HBASE-24408 with a MasterRegion. The new MasterRegion uses normal WALs, but archives them with a suffix "$masterlocalwal$".

I believe this would fail [around line 222 of IncrementalBackupManager|https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalBackupManager.java#L222], because [BackupUtils.getCreationTime|https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/BackupUtils.java#L383-L390] similarly expects the file names to end with a timestamp.

Unfortunately I am not set up to run master branch or test the backup/restore functionality, but I wanted to log this because I happened to stumble upon it.
Bug: Add dry run mode to hbase balancer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 21:08

It's often rather hard to know how the cost function changes you're making will affect the balance of the cluster, and currently the only way to know is to run it. If the cost decisions are not good, you may have just moved many regions towards a non-ideal balance. Region moves themselves are not free for clients, and the resulting balance may cause a regression.

We should add a mode to the balancer so that it can be invoked without actually executing any plans. This will allow an administrator to iterate on their cost functions and used the balancer's logging to see how their changes would affect the cluster.
Bug: Allow users to not start zookeeper and dfs cluster when using TestingHBaseCluster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/21 12:51

Users may start zookeeper and dfs cluster outside TestingHBaseCluster to make different components share the same zookeeper and dfs cluster. We should allow users to not start zookeeper and dfs cluster when using TestingHBaseCluster, and maybe we could provide utility class to let users easily set the configurations to switch to external zookeeper and dfs cluster.
Bug: Automatic and near real-time healing of locality
Status: Open | Priority: Major | Resolution: 
Created: 02/Sep/21 14:18

I’m proposing a somewhat major new tool for quickly and efficiently alleviating latency pains due to locality. This is especially useful in cloud environments, and has been highly impactful at HubSpot, where we run thousands of RegionServers across 40+ multi-zone clusters. Please see the attached design doc for details on the problem, why compactions are not enough to solve the problem, and an overview (with diagram) of the components that make up this new tool.

As spec'd, this new feature would require submission of a new tool in the HDFS project. Once we reach consensus on the approach I can create the relevant upstream HDFS JIRA.

See the design doc here: [https://docs.google.com/document/d/1GLGzrF1QLyhyOCr2fFw0LCymnyFPT0ktShTaaXn-75A/edit#heading=h.aswo7shg76b6]

Note: This issue is an attempt to upstream a tool that has been fully deployed for all clusters in production at HubSpot for about 6 months. It's been very effective for us as currently implemented, but will need to be re-organized and re-designed a bit to fit into the HBase/HDFS projects. As such I'd like feedback on the design before putting in too much effort on porting multiple components into PRs.
Bug: Write straight into the store directory when Splitting and Merging
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/21 15:30
Affects Version/s: 3.0.0-alpha-1

As discussed with [~zhangduo] on HBASE-26079, we think there's no actual need for Splits/Merges to make use of temp directories, because the split/merge region files would not be loaded until the new regions are ready to open, meaning the split/merge operation has completed. This would then avoid the need for a FS rename at the end of split/merge operation.
Bug: Showcase/document a end-to-end tracing integreated with an external sink e.g. Jaeger or zipkin 
Status: Open | Priority: Major | Resolution: 
Created: 31/Aug/21 01:49
Affects Version/s: 2.5.0

This task/issue was being discussed [in the vote of HBASE-25853|https://lists.apache.org/thread.html/rb36f0b6b179ee5f42cff7bb60de2797f69c450752c993f0b1e0f6118%40%3Cdev.hbase.apache.org%3E] 

bq. I've not seen anyone report their success at integrating these changes into a telemetry system and showing end-to-end tracing of a service call. As I recall from the last 2.5 discuss thread, we wanted this to be the banner feature for the minor release. Since we don't have this end-to-end confirmation, do we have someone who's volunteered to demonstrate that final proof-of-integration in a timely manner? Perhaps that someone is keen to be release manager for 2.5.

What a `service call` mean ? (pending discussion on above email exchange or on this JIRA)
e.g. for RPCclient/AbstractRpcClient, it trace at the AbstractRpcClient#callMethod level with a span of `RpcClient.callMethod` . or whenever a connection is closed, it will emit `AsyncConnection.close`
Bug: Wrap up 1.7.1 release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/21 17:20
Affects Version/s: 1.7.1

- Set branch-1 version to 1.7.2-SNAPSHOT
- Update SVN release bits, add download links to 1.7.1
- Remove 1.7.0 from the mirrors
- Commit HBASE-26071

in that order.
Bug: Ameliorate compaction made by bulk-loading files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/21 06:45

HBASE-25213 introduced to trigger compact after bulkload, but if a store needs compaction, each file loaded to this store will add a compaction runner of this store to the compaction queue. Then repeatedly compute for needCompaction and selectFiles for this store, and the compaction of the store may be continuous.

We can mark the queued compaction runners for the stores, and avoid repeatedly adding same store compactions to the queue after bulk load files. This method can reduce the pressure during the period just after bulk loading and spread the pressure to other times by the periodic CompactionChecker.

For the need of splitting mentioned before, after the completion of compaction, it will trigger split of the region if it needs. So we do not need to trigger split after bulk loading.
Bug: Backport HBASE-25401 "Add trace support for async call in rpc client" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:35
Affects Version/s: 2.5.0

[2/17 commits of HBASE-22120|https://github.com/apache/hbase/pull/2901/commits], original commit of HBASE-25401 was [https://github.com/apache/hbase/commit/242028671535dfed67ec13d9ed0d6067f3ccfd04]
Bug: Update jetty version to fix CVE-2021-34429
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Sep/21 10:32

 CVE-2021-34429 issue is fixed in Jetty 9.4.43.v20210629 and we are using jetty 9.4.41.v20210516.

https://github.com/apache/hbase-thirdparty/blob/c28a235236b9f63ec1d36431e5d1b6c8d4b66d90/pom.xml#L139
Bug: Fallback support to pure Java compression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/21 02:08

Airlift’s aircompressor (https://search.maven.org/artifact/io.airlift/aircompressor) is an Apache 2 licensed library, for Java 8 and up, available in Maven central, which provides pure Java implementations of gzip, lz4, lzo, snappy, and zstd and Hadoop compression codecs for same, claiming “_they are typically 300% faster than the JNI wrappers_.” (https://github.com/airlift/aircompressor). This library is under active development and up to date releases because it is used by Trino.

Proposed changes:
* Modify Compression.java such that compression codec implementation classes can be specified by configuration. Currently they are hardcoded as strings.
* Pull in aircompressor as a ‘compile’ time dependency so it will be bundled into our build and made available on the server classpath.
* Modify Compression.java to fall back to an aircompressor pure Java implementation if schema specifies a compression algorithm, a Hadoop native codec was specified as desired implementation, but the requisite native support is somehow not available.
Bug: Universal compression support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/21 01:52

Some Hadoop compression codecs became more available in recent Hadoop 3.x releases, addressed by HBASE-25940. This is nice but still requires native platform support, which to state the obvious is not available on all platforms and architectures, even if native libaries for some are bundled into jars. 

Airlift's aircompressor (https://search.maven.org/artifact/io.airlift/aircompressor) is an Apache 2 licensed library, for Java 8 and up, available in Maven central, which provides pure Java implementations of desirable compression algorithms gzip, lz4, lzo, snappy, and zstd, and Hadoop compression codecs for same, claiming "_they are typically 300% faster than the JNI wrappers_." (https://github.com/airlift/aircompressor). This library is under active development and has up to date releases because it is used by Trino.

We have another project that depends on universal availability of SNAPPY. I would like to make this change as a general improvement which also satisfies that requirement. (The as yet unnamed project will be contributed later.) It will be a very nice-to-have to have universal ZSTD support available as well. 

Proposed changes:
* Modify Compression.java such that compression codec implementation classes can be specified by configuration. Currently they are hardcoded as strings. 
* Pull in aircompressor as a 'compile' time dependency so it will be bundled into our build and made available on the server classpath. 
* Modify Compression.java to fall back to an aircompressor pure Java implementation if schema specifies a compression algorithm, a Hadoop native codec was specified as desired implementation, but the requisite native support is somehow not available. 

The combination of these changes will provide universal (pure Java) support for these desired and desirable compression codecs while retaining default behavior, which is to load and utilize Hadoop native implementations of same, if native support is available. They will also let you override this default if you wish to chase the claimed benefits of the pure Java alternatives.
Bug: Store region server list in master local region
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/21 16:08

Just a simple idea still need to be polished.

For large clusters, ZooKeeper could be a bottle neck, there are some related issues to not track region server list as much as possible, but what if we want to do more, just do not register region server on zk?

I think, zookeeper here is act as something like a service registry, we need the list to get all the region servers, and also need to know the changes of the list.
But in fact, we could also kill region server from master side, so the latter one could be done by a periodical heartbeat check daemon on master. And for the former one, we could store the list in master local region, so when master restart, it could setup the region server list by load from master local region.

And there are mainly two other side effect which are good:
1. We do not need to list the WAL directory on HDFS to find the previous region servers, for scheduling SCP. This could make it possible to restart a new HBase cluster based on only the root directory.
2. For now, region server needs to register to HMaster first, and then put its node on ZooKeeper, if it fails between these two actions, there is no way for HMaster to clean this dead server, as it never expires on ZooKeeper. There should be a related issue. If we just do not store a node on zk, then this problem is also gone.

Of course, there will still be lots of other problems, like whether we need another heartbeat call as reportForDuty is a bit heavy as we also report the region list, etc.
Bug: Reflect out-of-band locality improvements in served requests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Sep/21 13:26

Edit: Description updated to avoid needing to read the full investigation laid out in the comments.

Once the LocalityHealer has improved locality of a StoreFile (by moving blocks onto the correct host), the Reader's DFSInputStream and Region's localityIndex metric must be refreshed. Without refreshing the DFSInputStream, the improved locality will not improve latencies. In fact, the DFSInputStream may try to fetch blocks that have moved, resulting in a ReplicaNotFoundException. This is automatically retried, but the retry will temporarily increase long tail latencies relative to configured backoff strategy.

In the original LocalityHealer design, I created a new RefreshHDFSBlockDistribution RPC on the RegionServer. This RPC accepts a list of region names and, for each region store, re-opens the underlying StoreFile if the locality has changed. This implementation was complicated both in integrating callbacks into the HDFS Dispatcher and in terms of safely re-opening StoreFiles without impacting reads or caches. 

In working to port the LocalityHealer to the Apache projects, I'm taking a different approach:
 * The part of the LocalityHealer that moves blocks will be an HDFS project contribution
 * As such, the DFSClient should be able to more gracefully recover from block moves.
 * Additionally, HBase has some caches of block locations for locality reporting and the balancer. Those need to be kept up-to-date.

The DFSClient improvements are covered in HDFS-16261 and HDFS-16262. As such, this issue becomes about updating HBase's block location caches.

I considered a few different approaches, but the most elegant one I could come up with was to tie the HDFSBlockDistribution metrics directly to the underlying DFSInputStream of each StoreFile's initialReader. That way, our locality metrics are identically representing the block allocations that our reads are going through. This also means that our locality metrics will naturally adjust as the DFSInputStream adjusts to block moves.

Once we have accurate locality metrics on the regionserver, the Balancer's cache can easily be invalidated via our usual heartbeat methods. RegionServers report to the HMaster periodically, which keeps a ClusterMetrics method up to date. Right before each balancer invocation, the balancer is updated with the latest ClusterMetrics. At this time, we compare the old ClusterMetrics to the new, and invalidate the caches for any regions whose locality has changed.
Bug: Use priority queue in dir scan pool of cleaner
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 28/Sep/21 09:35
Affects Version/s: 2.0.0

DirScanPool used normal LinkedBlockingQueue when creating thread pool,
{code:java}
 54   private static ThreadPoolExecutor initializePool(int size) {
 55     return Threads.getBoundedCachedThreadPool(size, 1, TimeUnit.MINUTES,
 56       new ThreadFactoryBuilder().setNameFormat("dir-scan-pool-%d").setDaemon(true)
 57         .setUncaughtExceptionHandler(Threads.LOGGING_EXCEPTION_HANDLER).build());
 58   }
{code}
which will not priority scan larger directories and delete files there as expected, though CleanerChore#sortByConsumedSpace() before putting directories to the queue.

Subdirectories of larger directories and small directories are taken fairly in the queue.

We should used priority queue here instead, e.g. PriorityBlockingQueue, to make larger directories be cleaned earlier.
Bug: MetricsRegionServer may throws NullPointerException
Status: Open | Priority: Major | Resolution: 
Created: 05/Aug/21 07:22

MetricsRegionServer may throws NullPointerException When optional.get() is called.

 

 
{code:java}
metricRegistry = MetricRegistries.global().get(serverSource.getMetricRegistryInfo()).get();
 
{code}
Bug: Limit max result size of individual Gets
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 26/Jul/21 19:18

Scans have the ability to have a configured max result size, which causes them to return a partial result once the limit has been reached. MultiGets also can throw MultiActionResultTooLarge if the response size is over a configured quota. Neither of these really accounts for a single Get of a too-large row. Such too-large Gets can cause substantial GC pressure or worse if sent at volume.

Currently one can work around this by converting their Get to a single row Scan, but this requires a developer to proactively know about and prepare for the issue by using a Scan upfront or wait for the RegionServer to choke on a large request and only then rewrite the Get for future requests.

We should implement the same response size limits for for Get as for Scan, whereby the server returns a partial result to the client for handling.
Bug: hbck2 should support remove deleted meta region
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/21 08:39

A large number of regions of deleted tables will enter rit when hbase2.x restarts， and these region will also appear on hbck web page,  we just delete these region in meta table，and change active master。

here is my request: [pull request|https://github.com/apache/hbase-operator-tools/pull/81]
Bug: The HStore.snapshot method is never called in main code
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 11:41

In the comment of HStore.flushCache, we say that HStore.snapshot method must be called first. But actually, we will call memstore.snapshot directly from StoreFlusherImpl.prepare, without holding the write lock. The reason we do not need to hold the write lock is that, we hold HRegion.updatesLock in the upper layer, so it is OK.

See HBASE-10087 for more discussion about this.

So in general, I think we could remove the snapshot method in HStore. And I do not think we need to hold the write lock when calling clearSnapshot then. As in the HRegion layer, we need to guarantee that there is only one ongoing flush of the region, so before we finish the region, it is already safe to operate on the snapshot of the memstore, especially that the clearSnapshot method itself is thread safe.
Bug: Master initialization fails if Master Region WAL dir is missing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/21 20:46
Affects Version/s: 2.4.6

From a recent branch-2.4 build:

{noformat}
2021-09-07 19:31:19,666 ERROR [master/localhost:16000:becomeActiveMaster] master.HMaster(159): ***** ABORTING master localhost,16000,1631057476442: Unhandled exception. Starting shutdown. *****
java.io.FileNotFoundException: File hdfs://localhost:8020/hbase-2.4-wals/MasterData/WALs does not exist.
        at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
        at org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
        at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
        at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
        at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:226)
        at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:303)
        at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:839)
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2189)
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:512)
        at java.lang.Thread.run(Thread.java:748)
{noformat}

If the WAL directory is missing but the Master Region already exists, we will try to list the contents of the Master Region's WAL directory which may or may not exist. If we simply check to make sure the directory exists and then the rest of the initialization code works as expected.
Bug: High rate logging of BucketAllocatorException: Allocation too big 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Aug/21 23:46
Affects Version/s: 2.4.5

These log lines may be printed at high frequency when using a schema that creates large blocks and the bucket cache is enabled. It makes sense to warn about this initially, but the very high rate of warnings when the use case is legit and this is expected does not. Print this once, then not again; or, rate-limit this message to be printed at a more reasonable rate, like once per minute. 

{noformat}
2021-08-11 23:42:10,902 WARN  [main-BucketCacheWriter-0]
bucket.BucketCache: Failed allocation for 4842189251414b8a9212f6831462e415_218610843;
org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocatorException:
Allocation too big size=1049337;
adjust BucketCache sizes hbase.bucketcache.bucket.sizes to accomodate
if size seems reasonable and you want it cached.
{noformat}

Also, it might be better to log this at INFO given the caveat "if size seems reasonable and you want it cached". Reads like an informational message.
Bug: Backport HBASE-23834 to branch 2.3
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 03/Sep/21 10:26
Affects Version/s: 2.3.0

HBase 2.3 branch is currently on Jetty 9.3, and latest Hadoop 3 (unreleased branches trunk, branch-3.2 and branch-3.1) bumped Jetty to 9.4
Bug: Balancer tends to move regions to the server at the end of list
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/21 03:11

When we have cohosted replicas, cohosted replica distribution has priority to region count skew. So we won't see region count  getting balanced until the replicas distributed. During this time, master UI shows a drift of regions to the server at the end of list and causes significant overload. This shows a bias of random region pick and needs to be addressed.