av9ash/GitBugs huggingface dataset:
Bug: Document how to move WebUI access log entries to a separate log file
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 13/May/20 21:34
Affects Version/s: 2.3.0

I've noticed that after a recent commit, we now have webui access log lines going into our service log file. The log entires are going to a logger called {{http.requests.regionserver}}, and after the preamble of timestamp, log level, logger, they appear to be conformant to the [CLF|https://en.wikipedia.org/wiki/Common_Log_Format] specification. Tools designed for parsing http logs usually expect to have just the CLF entries, and not need preprocessing.

We should document how to configure the service to log these entries into a separate log file.
Bug: Avoid introducing log4j and slf4j-log4j dependencies for modules other than hbase-assembly
Status: Closed | Priority: Major | Resolution: Fixed
Created: 04/May/20 01:58

In general, a library should not force the down stream users to use a specific logging framework, and this is why there is a slf4j library.

For HBase, since we also publish the testing-util module which almost depends on all other sub modules, we should not introduce logging dependencies other than slf4j-api in these modules. We should only add log4j dependencies in hbase-assembly and ship it with our binary distribution.

This is also important for switching to log4j2.
Bug: batch Increment/Append fails when retrying the RPC
Status: Closed | Priority: Major | Resolution: Fixed
Created: 07/Jun/20 01:44

When a client hits RPC timeout and sends a second RPC request for batch Increment/Append but the first RPC is already processed actually, the nonce of the RPC is saved in the RS.
 In this case, for the second RPC, the RS just reads the previous result and returns it to the client to avoid the Increment/Append is processed twice.

At that time, for batch Increment/Append, we try to create a Get object from a CellScanner object in the following code:
 [https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776]

However, the CellScanner object is already consumed to create the Increment/Append object in the following code, and it fails with the following exception:
 [https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757]
{code:java}
2020-06-06 14:09:06,153 WARN  [hconnection-0x79c3903e-shared-pool3-t209] client.AsyncRequestFutureImpl: id=1, table=REF_Test, attempt=3/36, failureCount=1ops, last exception=org.apache.hadoop.hbase.DoNotRetryIOException:
org.apache.hadoop.hbase.DoNotRetryIOException: Cell count of 1 but at index 0 no cell returned: row: "xxxxxxxxxxx" mutate_type: INCREMENT timestamp: 9223372036854775807 durability: USE_DEFAULT time_range { from: 0 to: 9223372036854775807 } associated_cell_count: 1 nonce: 5281583076322914765
        at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:934)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:737)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2702)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42202)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
Bug: hbck2 command should accept one or more files containing a list of region names/table names/namespaces
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/20 15:43
Affects Version/s: hbase-operator-tools-1.0.0

Currently many command accepts a list of region names/ table names/namespaces on the command line.  We should accept paths to one or more files that contains these encoded regions, one per line. That way, this command tails nicely into an operator's incantation using grep/sed over log files.

Similar work has been done in https://issues.apache.org/jira/browse/HBASE-23927
Bug: Improve region housekeeping status observability
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 09/Jun/20 23:27

We provide a coarse grained admin API and associated shell command for determining the compaction status of a table:

{noformat}
hbase(main):001:0> help "compaction_state"
Here is some help for this command:
     Gets compaction status (MAJOR, MAJOR_AND_MINOR, MINOR, NONE) for a table:
     hbase> compaction_state 'ns1:t1'
     hbase> compaction_state 't1'
{noformat}

We also log  compaction activity, including a compaction journal at completion, via log4j to whatever log aggregation solution is available in production.  

This is not sufficient for online and interactive observation, debugging, or performance analysis of current compaction activity. In this kind of activity an operator is attempting to observe and analyze compaction activity in real time. Log aggregation and presentation solutions have typical latencies (end to end visibility of log lines on the order of ~minutes) which make that not possible today.

We don't offer any API or tools for directly interrogating split and merge activity in real time. Some indirect knowledge of split or merge activity can be inferred from RIT information via ClusterStatus. It can also be scraped, with some difficulty, from the debug servlet. 

We should have new APIs and shell commands, and perhaps also new admin UI views, for

at regionserver scope:
* listing the current state of a regionserver's compaction, split, and merge tasks and threads
* counting (simple view) and listing (detailed view) a regionserver's compaction queues
* listing a region's currently compacting, splitting, or merging status

at master scope, aggregations of the above detailed information into:
* listing the active compaction tasks and threads for a given table, the extension of _compaction_state_ with a new detailed view
* listing the active split or merge tasks and threads for a given table's regions

Compaction detail should include the names of the effective engine and policy classes, and the results and timestamp of the last compaction selection evaluation. Split and merge detail should include the names of the effective policy classes and the result of the last split or merge criteria evaluation.
Bug: Skip HDFSBlockDistribution calc when FS is not HDFS
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 11/May/20 17:07
Affects Version/s: 2.5.0

HDFSBlockDistribution is used in different places like Balancer,  DateTieredCompaction, CompactionTool etc. In Balancer area there is a config 'hbase.master.balancer.uselocality' using which we can skip this.  But irrespective of this config, if we are on non HDFS FS, we should skip this.  The block distribution issue many file status commands to underlying FS which wont be that cheap in a cloud FS.  This jira aims at correcting all these places.
Bug: Prevent temporal misordering on timescales smaller than one clock tick
Status: Open | Priority: Major | Resolution: 
Created: 26/May/20 17:54

When mutations are sent to the servers without a timestamp explicitly assigned by the client the server will substitute the current wall clock time. There are edge cases where it is at least theoretically possible for more than one mutation to be committed to a given row within the same clock tick. When this happens we have to track and preserve the ordering of these mutations in some other way besides the timestamp component of the key. Let me bypass most discussion here by noting that whether we do this or not, we do not pass such ordering information in the cross cluster replication protocol. We also have interesting edge cases regarding key type precedence when mutations arrive "simultaneously": we sort deletes ahead of puts. This, especially in the presence of replication, can lead to visible anomalies for clients able to interact with both source and sink. 

There is a simple solution that removes the possibility that these edge cases can occur: 

We can detect, when we are about to commit a mutation to a row, if we have already committed a mutation to this same row in the current clock tick. Occurrences of this condition will be rare. We are already tracking current time. We have to know this in order to assign the timestamp. Where this becomes interesting is how we might track the last commit time per row. Making the detection of this case efficient for the normal code path is the bulk of the challenge. One option is to keep track of the last locked time for row locks. (Todo: How would we track and garbage collect this efficiently and correctly. Not the ideal option.) We might also do this tracking somehow via the memstore. (At least in this case the lifetime and distribution of in memory row state, including the proposed timestamps, would align.) Assuming we can efficiently know if we are about to commit twice to the same row within a single clock tick, we would simply sleep/yield the current thread until the clock ticks over, and then proceed.
Bug: Reduce cpu load of locating region especially in batch mode.
Status: Patch Available | Priority: Major | Resolution: 
Created: 01/May/20 11:46
Affects Version/s: 1.2.12

Binary search is used to speedup the process of locating region. It is already fast enough, while cpu of HBASE client becomes the bottleneck when doing TCSB benchmark. We can make the process of locating region faster to reduce cpu load in some special cases , which however is our common case in production environment.  It is the case: 

        1. Predefined splits in uniform distribution.
  
         2. Load data in batch mode.

The optimization is very simple, just to contract range of binary search. Initially,  record all startIndex and endIndex of first or two bytes of keys. When a region key comes, find the contracted startIndex and endIndex of the key. Then return to normal binary search process with the specified startIndex and endIndex. 

Then we can ideally reduce cpu to 1/8 with 1 byte or 1/16 with 2 bytes.
Bug: HBCK2 feature negotiation for commands not on Hbck interface
Status: Open | Priority: Major | Resolution: 
Created: 17/Apr/20 09:58

Some HBCK2 commands use version based checking for determining compatibility. HBASE-24039 changed this to feature-based compatibility check with commands using org.apache.hadoop.hbase.client.Hbck interface.

A similar approach is needed for the remaining commands that are using different APIs.
Bug: Make PE to have both read and write ability to go in parallel as in YCSB
Status: Open | Priority: Major | Resolution: 
Created: 05/Jun/20 14:12
Affects Version/s: 2.5.0

Currently we can either run pure writes or pure reads in case of PE. The case where few threads can do write and few can do reads is not available. We can improve the PE to atleast support writes and either Scans or Gets in parallel.
Bug: Purge use of VisibleForTesting
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jun/20 16:33
Affects Version/s: 2.4.0

From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it pertains to our API compatibility guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], when used in classes annotated with interface audience other than IA.Private, the VisibleForTesting annotation is confusing and considered harmful. The consensus is that we do not want to use this annotation as part of the definition of our public APIs, and we need to remove the point of confusion.
Bug: Update GC config examples in hbase-env.sh for G1GC
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/20 19:08
Affects Version/s: 2.3.0

Anywhere we changed the default collector to G1GC, we should also update {{hbase-env.sh}} to include GC configuration examples appropriate for that collector and JVM. This means, where we support both JDK8 and JDK11, we'll have two sections of examples, one for CMS and the other for G1.
Bug: Improve balancer decision observability
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/20 23:56

We provide detailed INFO and DEBUG level logging of balancer decision factors, outcome, and reassignment planning, as well as similarly detailed logging of the resulting assignment manager activity. However, an operator may need to perform online and interactive observation, debugging, or performance analysis of current balancer activity. Scraping and correlating the many log lines resulting from a balancer execution is labor intensive and has a lot of latency (order of ~minutes to acquire and index, order of ~minutes to correlate). 

The balancer should maintain a rolling window of history, e.g. the last 100 region move plans, or last 1000 region move plans submitted to the assignment manager. This history should include decision factor details and weights and costs. The rsgroups balancer may be able to provide fairly simple decision factors, like for example "this table was reassigned to that regionserver group". The underlying or vanilla stochastic balancer on the other hand, after a walk over random assignment plans, will have considered a number of cost functions with various inputs (locality, load, etc.) and multipliers, including custom cost functions. We can devise an extensible class structure that represents explanations for balancer decisions, and for each region move plan that is actually submitted to the assignment manager, we can keep the explanations of all relevant decision factors alongside the other details of the assignment plan like the region name, and the source and destination regionservers. 

This history should be available via API for use by new shell commands and admin UI widgets.

The new shell commands and UI widgets can unpack the representation of balancer decision components into human readable output.
Bug: RequestId Tracing feature for HBase 
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 26/Jun/20 13:38

Associate a TraceId corresponding to Hbase Put Request on the mutation level which will be further propagated to the RPC layer. This will enable cost-effective logging of the Request Flow and help identifying Network bottlenecks. We aim to send the TraceId to the RPC server by modifying the RequestProto.
Bug: CloneSnapshotProcedure unlimited retry
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jun/20 06:55
Affects Version/s: 2.2.5

since regions dir was not remove in the previous execution created,  need to be remove when retrying, resulting in exception, unlimited retry
{code:java}
procedure.CloneSnapshotProcedure: Retriable error trying to clone snapshot=snapshot_test to table=test:backup state=CLONE_SNAPSHOT_WRITE_FS_LAYOUT
org.apache.hadoop.hbase.snapshot.RestoreSnapshotException: clone snapshot={ ss=snapshot_test table=test:backup type=FLUSH } failed because A clone should not have regions to remove
        at org.apache.hadoop.hbase.master.procedure.CloneSnapshotProcedure$1.createHdfsRegions(CloneSnapshotProcedure.java:434)
        at org.apache.hadoop.hbase.master.procedure.CloneSnapshotProcedure.createFsLayout(CloneSnapshotProcedure.java:465)
        at org.apache.hadoop.hbase.master.procedure.CloneSnapshotProcedure.createFilesystemLayout(CloneSnapshotProcedure.java:392)
        at org.apache.hadoop.hbase.master.procedure.CloneSnapshotProcedure.executeFromState(CloneSnapshotProcedure.java:142)
        at org.apache.hadoop.hbase.master.procedure.CloneSnapshotProcedure.executeFromState(CloneSnapshotProcedure.java:67)
        at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)
        at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1662)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1409)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1979)
Caused by: java.lang.IllegalArgumentException: A clone should not have regions to remove
        at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
        at org.apache.hadoop.hbase.master.procedure.CloneSnapshotProcedure$1.createHdfsRegions(CloneSnapshotProcedure.java:418)
        ... 10 more
{code}

and the cloned regions name are unchanged, resulting in new created regions be removed when retrying
Bug: Change default Hfile storage policy from HOT to NONE for HDFS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/20 16:55
Affects Version/s: 2.2.5

The current HBase hfile directory specifies the *HOT* storage strategy by default(`hbase.hstore.blockingStoreFiles` ), which is different from the default value of HDFS *heterogeneous* strategy now(should be empty)

It should be set to *NONE* like `hbase.wal.storage.policy`'s default value on HBASE-20691, and the reason for modification is the same

Otherwise, it could have a significant impact because most HDFS Managers doesn't know the default value has been changed to *HOT* by hbase, for example:

HDFS manger want to set hfile's dir to ONE_SSD for speed up read, they will do:
{code:java}
// Set Hbas Hfile dir to ONE_SSD for speed up read
// The policy that expects all subdirectories to inherit the parent directory by default (empty by default)
bin/hdfs storagepolicies -setStoragePolicy -path /home/hbase/data  -policy ONE_SSD
bin/hdfs storagepolicies -setStoragePolicy -path /home/hbase/WALs  -policy ALL_SSD

// If we do not change Hbase default setting, we will find this
bin/hdfs storagepolicies -getStoragePolicy -path /home/hbase/data/hbase/meta/xxxx/info 
return "HOT" instead of "ONE_SSD"{code}
And here is a pic to show the problem ↑:

  !hfile-policy-problem.png!

However, if Hfile uses the *HOT* strategy by default, then we will find that the actually stored directory has been set to HOT, then it will *not inherit* the *ONE_SSD* of the parent directory policy, and if we set *NONE* in hbase, this is problem can be avoided

 Unless someone sets other values manually, then we believe the operator knows what he is doing
Bug: Reseek regression related to filter SKIP hinting
Status: Open | Priority: Major | Resolution: 
Created: 26/Jun/20 01:39
Affects Version/s: 2.2.5

I have been looking into reported performance regressions in HBase 2 relative to HBase 1. Depending on the test scenario, HBase 2 can demonstrate significantly better microbenchmarks in a number of cases, and usually shows improvement in whole cluster benchmarks like YCSB.

To assist in debugging I added methods to RpcServer for updating per-call metrics that leverage the fact it puts a reference to the current Call into a thread local and that all activity for a given RPC is processed by a single thread context. I then instrumented ScanQueryMatcher (in branch-1) and its various friends (in branch-2.2), StoreScanner, HFileReaderV2 and HFileReaderV3 (in branch-1) and HFileReaderImpl (in branch-2.2), HFileBlock, and DefaultMemStore (branch-1) and SegmentScanner (branch-2.2). Test tables with one family and 1, 5, 10, 20, 50, and 100 distinct column-qualifiers per row were created, snapshot, dropped, and cloned from the snapshot. Both 1.6 and 2.2 versions under test operated on identical data files in HDFS. For tests with 1.6 and 2.2 on the server side the same 1.6 PE client was used, to ensure only the server side differed.

The results for pe --filterAll were revealing. See attached. 

It appears a refactor to ScanQueryMatcher and friends has disabled the ability of filters to provide meaningful SKIP hints, which disables an optimization that avoids reseeking, leading to a serious and proportional regression in reseek activity and time spent in that code path. So for queries that use filters, there can be a substantial regression.

Other test cases that did not use filters did not show this regression. If filters are not used the behavior of ScanQueryMatcher between 1.6 and 2.2 was almost identical, as measured by counts of the hint types returned, whether or not column or version trackers are called, and counts of store seeks or reseeks. Regarding micro-timings, there was a 10% variance in my testing and results generally fell within this range, except for the filter all case of course.
Bug: BulkLoad Will Fall Into Unbelievable Retry Attempt in Some case
Status: Patch Available | Priority: Major | Resolution: 
Created: 23/May/20 02:17

In https://issues.apache.org/jira/browse/HBASE-14541, the retry logic changed from a configurable retry times(by configuration item hbase.bulkload.retries.number) to below retry logic to process the issue that the RegionSplit happened during bulk load:

 
{code:java}
int maxRetries = getConf().getInt("hbase.bulkload.retries.number", 10);
maxRetries = Math.max(maxRetries, startEndKeys.getFirst().length + 1);
if (maxRetries != 0 && count >= maxRetries) {
 throw new IOException("Retry attempted " + count +
 " times without completing, bailing out");
}{code}
This issue caused another issue in our cluster, that is:
 Our table has *2000* regions and our bulk load failed for an configuration issue:
 This failure is caused by a client mis-configuration and cause the RegionServer fail to load the HFile, but the response is not thrown as exception to client, but marked as a variable `*loaded*` in the `*BulkLoadHFileResponse*`


{code:java}
message BulkLoadHFileResponse {
 required bool loaded = 1;
}{code}

 After this failure, the bulk load fall into a retry disaster and after the retry reached about 200, our HDFS crashed for OOM.
{code:java}
 {code}
 

But during with, the HBase table splits in fact didn't happen ever.

I think the patch in HBASE-14541 didn't handle the unrecoverable retry case and in this case(I think many reason may incur unrecoverable retry) the meaningless retry attempts becomes disaster and is un-configurable because we cannot change the Region number of our table;
Bug: Remove data locality and StoreFileCostFunction for replica regions out of balancer's cost calculation
Status: Open | Priority: Major | Resolution: 
Created: 25/Jun/20 00:35
Affects Version/s: 2.3.0

We found one of the clusters with read replica enabled always balance lots of replica regions. going through the balancer's cost functions, found that data locality and StoreFileCost have same multiplier for both primary and replica regions. That is something we can improve. Data locality for replica regions should not be a dominant factor for balancer. We can either remove it out of balancer's picture for now and give it a small multiplier.
Bug: Replace Cluster#primariesOfRegionsPerServer from int array to treemap
Status: Open | Priority: Major | Resolution: 
Created: 26/Jun/20 17:21
Affects Version/s: 2.3.0

Currently, primariesOfRegionsPerServer is an int array, moveRegion does heavy work by searching the array (linearly) and insert/remove an element requires allocating/copying the whole array.
Bug: Normalizer merge plans should consider more than 2 regions when possible
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/20 16:09
Affects Version/s: 2.3.0

The merge plans produced by the normalizer operate over two regions. Our merge operation supports multiple regions in a single request. When there are multiple merge plans generated over contiguous region space, these should be collapsed into a single merge operation. This should automatically honor whatever existing configuration settings exist limiting the number of regions that can participate in a merge procedure.