av9ash/GitBugs huggingface dataset:
Bug: WAL archival to be done by a separate thread
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/20 13:34
Affects Version/s: 2.4.0

Currently we do clean up of logs once we ensure that the region data has been flushed. We track the sequence number and if we ensure that the seq number has been flushed for any given region and the WAL that was rolled has that seq number then those WAL can be archived.
When we have around ~50 files to archive (per RS) - we do the archiving one after the other. Since archiving is nothing but a rename operation it adds to the meta operation load of Cloud based FS. 
Not only that - the entire archival is done inside the rollWriterLock. Though we have closed the writer and created a new writer and the writes are ongoing - we never release the lock until we are done with the archiving. 
What happens is that during that period our logs grow in size compared to the default size configured (when we have consistent writes happening). 
So the proposal is to move the log archival to a seperate thread and ensure we can do some kind of throttling or batching so that we don't do archival at one shot.
Bug: Optimize in-memory representation of mapreduce TableSplit objects
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Aug/20 18:41
Affects Version/s: 1.7.0

It has been observed that when the table has too many regions, MR jobs consume a lot of memory in the client. This is because we keep the region level information in memory and the memory heavy object is TableSplit because of the Scan object as a part of it.

However, it looks like the TableInputFormat for single table doesn't need to store the scan object in the TableSplit because we do not use it and all the splits are expected to have the exact same scan object. In TableInputFormat we use the scan object directly from the MR conf.
Bug: The table.jsp cost long time to load if the table include closed regions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/20 06:29

The reason is that, in order to get the compaction state of table, we need to iterate all regions and call getRegionInfo, the default timeout for rpc is 5000ms, so if there exists closed reigons, this line will be blocked for long time.
{code:java}
CompactionState compactionState = admin.getCompactionState(table.getName()).get();{code}

The solution in this patch is to add compaction state to RegionMetrics, so when getting the compaction state, we could use the in memory state instead of doing rpc.

Affect branch-2+, in branch-1 it will display exception stack info immediately.
Bug: Remove unused credential hbaseqa-at-asf-jira
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Aug/20 16:22
Affects Version/s: 1.7.0

After move to ci-hadoop, branch-1 precommits are unable to checkout the source tree. 

https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/view/change-requests/job/PR-2194/1/console

{noformat}
04:55:42  Running in /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2194/yetus
[Pipeline] {
[Pipeline] checkout
04:55:42  No credentials specified
04:55:42  Cloning the remote Git repository
 > git rev-parse HEAD^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f9a427c6aff4f01bf06aa458a935249cfd6a5d30 # timeout=10
04:55:41  Cloning repository https://github.com/apache/yetus.git
04:55:41   > git init /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2194/yetus # timeout=10
04:55:41  Fetching upstream changes from https://github.com/apache/yetus.git
04:55:41   > git --version # timeout=10
04:55:41   > git fetch --tags --progress -- https://github.com/apache/yetus.git +refs/heads/*:refs/remotes/origin/* # timeout=10
04:55:44  Checking out Revision 11add70671de39cd96b56e86e40c64c872b9282f (rel/0.11.1)
04:55:42   > git config remote.origin.url https://github.com/apache/yetus.git # timeout=10
04:55:42   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
04:55:42   > git config remote.origin.url https://github.com/apache/yetus.git # timeout=10
04:55:42  Fetching upstream changes from https://github.com/apache/yetus.git
04:55:42   > git fetch --tags --progress -- https://github.com/apache/yetus.git +refs/heads/*:refs/remotes/origin/* # timeout=10
04:55:43   > git rev-parse rel/0.11.1^{commit} # timeout=10
04:55:43   > git rev-parse refs/remotes/origin/rel/0.11.1^{commit} # timeout=10
04:55:44  Commit message: "YETUS-920. Stage version 0.11.1."
04:55:44  First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (precommit-run)
[Pipeline] withCredentials
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] step
04:55:45  Archiving artifacts
[Pipeline] publishHTML
04:55:45  [htmlpublisher] Archiving HTML reports...
{noformat}
Bug: Meta table shouldn't be modified as read only
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/20 05:34
Affects Version/s: 2.3.0

After HBASE-23055, Alter of Meta schema is allowed.

After this change meta table can be made Read only table which is not correct as per my view,because it will impact majority  of the functionality of HBase.

 

alter 'hbase:meta',\{METHOD=>'table_att', READONLY=>true}
Bug: Change meta replica count by altering meta table descriptor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/20 14:00

As now we support altering meta table, it will be better to also deal with changing meta replica number using altering meta, i.e, we could unify the logic in MasterMetaBootstrap to ModifyTableProcedure, and another benefit is that we do not need to restart master when changing the replica number for meta.
Bug: Master manages ReplicationServers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 10:51
Affects Version/s: 3.0.0-alpha-1

In [HBASE-24683|https://issues.apache.org/jira/browse/HBASE-24683] add an isolated ReplicationServer.

What this issue is to do: 
 # ReplicationServer reports to Master periodically.
 # Add a basic ReplicationServerManager in Master to manage ReplicationServer.
Bug: ReplicationSourceShipper thread can not be finished
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/20 10:43
Affects Version/s: 2.2.6

See [Flaky Tests|https://ci-hadoop.apache.org/job/HBase/job/HBase-Flaky-Tests/job/master/161/console], some UTs about replication failed cause timeout.

In [HBaseInterClusterReplicationEndpoint.sleepForRetries|https://github.com/apache/hbase/blob/78ae1f176d4215dcc34067ed25d786a4fcd4d888/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java#L203], InterruptedException is caught but no further processing, the interrupted status of the current thread is cleared.

Below is the code comment of Thread.sleep.
{code:java}
/**
 * ...
 *
 * @throws  InterruptedException
 *          if any thread has interrupted the current thread. The
 *          <i>interrupted status</i> of the current thread is
 *          cleared when this exception is thrown.
 */
public static native void sleep(long millis) throws InterruptedException;
{code}
So InterruptedException must be processed, otherwise ReplicationSourceShipper thread cannot be terminated in some cases.
Bug: Procedure Executor doesn't handle burst well
Status: Open | Priority: Major | Resolution: 
Created: 02/Jul/20 19:09
Affects Version/s: 2.2.5

Currently, Procedure Executor checks for stuck threads every 5 sec. If >50% are stuck, one worker thread is added until maxPoolSize is reached. In a burst scenario, this cannot catch up and causes thread starvation. A smarter algorithm can be used to measure the pressure and step up automatically.
Bug: Cleanup unnecessary usages of MetaTableAccessor
Status: Open | Priority: Major | Resolution: 
Created: 24/Aug/20 13:23

For simplify the implementation of splittable meta.

In general, MetaTableAccessor can not deal with the catalog information for meta itself, even if we introduce a general root table, we could make it also deal with the catalog information for meta, but then we still need to deal with root...

Let's change to use other APIs if possible, for example, when getting all region locations, just use RegionLocator instead of MetaTableAccessor.
Bug: Improve BlockCache read performance by specifying BlockType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/20 13:29

CombinedBlockCache contains l1Cache and l2Cache. l1Cache stores MetaBlock and l2Cache stores DataBlock. Because getBlock does not know the BlockType, the getBlock of CombinedBlockCache queries l1Cache first, and then l2Cache. But actually querying DataBlock is not necessary to query l1Cache. 

Therefore, in some cases where BlockType is known, BlockCache read performance can be improved.
h2.  Code：

 BlockCache: default call old getBlock
{code:java}
default Cacheable getBlock(BlockCacheKey cacheKey, boolean caching, boolean repeat,
    boolean updateCacheMetrics, BlockType blockType) {
  return getBlock(cacheKey, caching, repeat, updateCacheMetrics);
}
{code}
CombinedBlockCache:
{code:java}
@Override
public Cacheable getBlock(BlockCacheKey cacheKey, boolean caching, boolean repeat,
    boolean updateCacheMetrics, BlockType blockType) {
  if (blockType == null) {
    return getBlock(cacheKey, caching, repeat, updateCacheMetrics);
  }
  boolean metaBlock = isMetaBlock(blockType);
  if (metaBlock) {
    return l1Cache.getBlock(cacheKey, caching, repeat, updateCacheMetrics);
  } else {
    return l2Cache.getBlock(cacheKey, caching, repeat, updateCacheMetrics);
  }
}private boolean isMetaBlock(BlockType blockType) {
  return blockType.getCategory() != BlockCategory.DATA;
}
{code}
HFileReaderImpl#getCachedBlock call BlockCache#getBlock(XXX, expectedBlockType)
Bug: Add MR Counters to WALPlayer; currently hard to tell if it is doing anything
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Sep/20 18:16

For example, when WALPlayer runs, it emits this:
{code:java}

020-09-28 11:16:05,489 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task: Final Counters for attempt_local1916643172_0001_m_000000_0: Counters: 20
	File System Counters
		FILE: Number of bytes read=268891453
		FILE: Number of bytes written=1018719
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=4375
		Map output records=5369
		Input split bytes=245
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=59
		Total committed heap usage (bytes)=518979584
	File Input Format Counters
		Bytes Read=0
	File Output Format Counters
		Bytes Written=0 {code}
Change it so it does this:
{code:java}
020-09-28 11:16:05,489 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task: Final Counters for attempt_local1916643172_0001_m_000000_0: Counters: 20
	File System Counters
		FILE: Number of bytes read=268891453
		FILE: Number of bytes written=1018719
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=4375
		Map output records=5369
		Input split bytes=245
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=59
		Total committed heap usage (bytes)=518979584
	org.apache.hadoop.hbase.mapreduce.WALPlayer$Counter
		CELLS_READ=89574
		CELLS_WRITTEN=89572
		DELETES=64
		PUTS=5305
		WALEDITS=4375
	File Input Format Counters
		Bytes Read=0
	File Output Format Counters
		Bytes Written=0 {code}
Bug: Per table WAL metrics: appendCount and appendSize
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/20 18:49
Affects Version/s: 1.7.0

We were trying to drill down into WAL events at a table scope and figured that appendCount and appendSize are not exposed at a table scope. This would be a helpful metric.

I ran the WAL benchmark tool with 10 threads and didn't see any measurable performance difference with the attached patch.
Bug: RSGroupBalancer#assignments lost some regionPlans
Status: Patch Available | Priority: Major | Resolution: 
Created: 24/Sep/20 01:18
Affects Version/s: 2.2.3

https://github.com/apache/hbase/blob/b2f2c79d8fa18fb691e669419004cc5168b0838d/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L216

when fallback is enabled, servers does not contain the current group's rs, and contains the rs of other group, region will be assigend to other group, but assignments already contains targetRS, and then assignments.putAll overwrites old entry

{code:java}
this.internalBalancer.retainAssignment(currentAssignmentMap, candidateList)
              .forEach((serverName, regionInfos) -> {
                assignments.computeIfAbsent(serverName, s -> new ArrayList<>())
                    .addAll(regionInfos);
});
{code}

the issue exists only in the branch-2.
Bug: TransitionRegionStateProcedure should timeout, rollback, retry instead of waiting infinitely on CONFIRMED_OPEN
Status: Open | Priority: Major | Resolution: 
Created: 17/Sep/20 17:51
Affects Version/s: 2.3.2

Testing 2.3.2RC1 with ITBLL. The region server assigned to open meta locked up due to HBASE-24896. Meanwhile, the master waits indefinitely on a procedure {{pid=176583, ppid=176532, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}}.

AssignmentManager needs a way to rescind assignment when a RS fails to complete within a reasonable timeout window, roll back the procedure, and try again with a new target.
Bug: Refactor MetaTableAccessor.addRegionsToMeta and its usage places
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/20 03:50

It is used in both CreateTableProcedure and ModifyTableProcedure, but in ModifyTableProcedure, we already have the region info in place so we should not put them again.

There is no problem for now as we will close all the regions before doing this in ModifyTableProcedure, but it is still a bit confusing to developers.

Let's change it.
Bug: [HBCK2] Bypassed parent procedures are not updated in store
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/20 09:53

See code in [ProcedureExecutor|https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java#L980]:
{code:java}
Procedure<TEnvironment> current = procedure;
while (current != null) {
  LOG.debug("Bypassing {}", current);
  current.bypass(getEnvironment());
  store.update(procedure);     // update current procedure
  long parentID = current.getParentProcId();
  current = getProcedure(parentID);
}
{code}
Bug: HFilePrettyPrinter can't seek to the row which is the first row of a hfile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/20 08:02
Affects Version/s: 2.2.6

This issue can be reproduced by below steps:
 * make a hfile contains two rows '000' and '001';

{code:java}
D:\bin>hbase hfile -p -f /hbase/data/default/test2/df76e4acab5398e70be332f6807ec3ba/f1/fda213c556d540a58d29d6bd85931dcd
K: 000/f1:a/1601282789548/Put/vlen=4/seqid=4 V: aaaa
K: 001/f1:a/1601282792779/Put/vlen=4/seqid=5 V: aaaa
Scanned kv count -> 2{code}
 * '001' can be seeked to;

{code:java}
D:\bin>hbase hfile -e -w 001 -f /hbase/data/default/test2/df76e4acab5398e70be332f6807ec3ba/f1/fda213c556d540a58d29d6bd85931dcd
K: 001/f1:a/1601282792779/Put/vlen=4/seqid=5
Scanned kv count -> 1{code}
 * but '000' can't be seeked to;

{code:java}
D:\bin>hbase hfile -e -w 000 -f /hbase/data/default/test2/df76e4acab5398e70be332f6807ec3ba/f1/fda213c556d540a58d29d6bd85931dcd
Scanned kv count -> 0{code}
 In HFilePrettyPrinter we use "scanner.seekTo(PrivateCellUtil.createFirstOnRow(this.row))" to seek to row.But this method will retrurn -1 when the row is the first row of hfile.
Bug: Enable table replication fails from 1.x to 2.x if table already exist at peer. 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Sep/20 06:44
Affects Version/s: 2.2.3

If source is 1.x and peer is 2.x and a default table is created at source and also at peer, default table descriptors mismatch so the enable table replication fails with error :

 
{code:java}
Table t1 exists in peer cluster 1, but the table descriptors are not same when compared with source cluster. Thus can not enable the table's replication switch.{code}

Following are the extra attributes set by default in 2.x:
CACHE_DATA_ON_WRITE
CACHE_INDEX_ON_WRITE
CACHE_BLOOMS_ON_WRITE
EVICT_BLOCKS_ON_CLOSE 
PREFETCH_BLOCKS_ON_OPEN.
Bug: conf and conn are assigned twice in HBaseReplicationEndpoint and HBaseInterClusterReplicationEndpoint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/20 02:44
Affects Version/s: 3.0.0-alpha-1

In [HBaseReplicationEndpoint.init()|https://github.com/apache/hbase/blob/c312760819ed185cab3a0717a1ea0ff6e8c47a23/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java#L109] and  [HBaseInterClusterReplicationEndpoint.init|https://github.com/apache/hbase/blob/c312760819ed185cab3a0717a1ea0ff6e8c47a23/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java#L145] , the latter is a sub-class of the former, conf and conn are assigned twice.
Bug: When table level quota is set and violated, pre existing ns quota policy not imposed on table.
Status: Open | Priority: Major | Resolution: 
Created: 28/Jul/20 20:34

When table level quota is set and violated, pre existing namespace level quota policy not imposed on table.

 

While removing Quota for a table, in addition to deleting from hbase:quota table, it should be removed from current state in QuotaObserverChore as well.
Bug: HA masters based on raft
Status: Open | Priority: Major | Resolution: 
Created: 22/Jul/20 02:44

For better availability, for moving bootstrap information from zookeeper to our own service so finally we could remove the dependency on zookeeper completely.

This has been in my mind for a long time, and since the there is a dicussion in HBASE-11288 about how to storing root table, and also in HBASE-24749, we want to have better performance on a filesystem can not support list and rename well, where requires a storage engine at the bottom to store the storefiles information for meta table, I think it is the time to throw this idea out.

The basic solution is to build a raft group to store the bootstrap information, for now it is cluster id(it is on the file system already?) and the root table. For region servers they will always go to the leader to ask for the information so they can always see the newest data, and for client, we enable 'follower read', to reduce the load of the leader(and there are some solutions to even let 'follower read' to always get the newest data in raft).

With this solution in place, as long as root table will not be in a format of region(we could just use rocksdb to store it locally), the cyclic dependency in HBASE-24749 has also been solved, as we do not need to find a place to store the storefiles information for root table any more.
Bug: Remove ReplicationSourceInterface#enqueueLog method and get the new WAL to replicate from ReplicationQueueStorage directly
Status: Open | Priority: Major | Resolution: 
Created: 14/Jul/20 04:00

There are two choices for this now.
 # Start a backgroud thread and keep to read the new WAL from ReplicationQueueStorage.
 # The default ReplicationQueueStorage is based on ZK. Set watcher on znode and read the WAL when there are zk event.
Bug: serial replication, addReplicationBarrier is writing to rep_barrier family even there is no serial replication peer.
Status: Open | Priority: Major | Resolution: 
Created: 30/Sep/20 22:32

We found that there are quite some data in rep_barrier family even there is no serial replication enabled. Checked the code,  it is checking if table has replication enabled. Think there is another check needed (i.e, is there any serial replication peers configured).

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java#L215]
Bug: Admin supports multi region merge
Status: Open | Priority: Major | Resolution: 
Created: 29/Sep/20 03:40
Affects Version/s: 2.2.3

https://github.com/apache/hbase/blob/68b56beab744e983df0877eec9f576ef884a2807/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L889
from masterRpcServices and mergeProc, master supports multi region merge...
but admin dont support...
we can enhance it
Bug: Cleanup of unused RPC APIs in the presence of generic API getLogEntries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Sep/20 12:30

Now that we have generic RPC API *getLogEntries* for any type of ringbuffer use-case regardless of where the ring buffer is running (Master or Regionservers), we should better clean up old use-case specific APIs: GetSlowLogResponses
Bug: WAL size in RegionServer UI is wrong
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/20 03:07

In MetricsRegionServerWrapperImpl, *walFileSize* is calculated by *provider.getLogFileSize()* twice as follows, which is causing the wrong WAL size in RegionServer UI:
{code:java}
        walFileSize = (provider == null ? 0 : provider.getLogFileSize()) +
            (provider == null ? 0 : provider.getLogFileSize());
{code}
Bug: NPE while shutdown master node
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 14/Sep/20 07:08

while shutdown the master node, we can see the exception:
{code:java}
2020-09-14 06:48:29,530 ERROR [PEWorker-16] procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception: pid=111, ppid=64, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure
java.lang.NullPointerException
        at org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:276)
        at org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:58)
        at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965)
{code}
Bug: hbck.jsp page loading fails, logs NPE in master log.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/20 14:20
Affects Version/s: 2.2.3

While testing with large data, it is observed that some time hbck page was not accessible and NPE was thrown in log.

 

【Test step】Excat Steps are not known. Observed this while executing below steps

1: Test data: 9K regions,~300 issues in my test setup

2: run hbck_chore and catalogjanitor_run multiple times 

3: Also run balancer so that region movement is happening.

4: Observe that HBCK page not available

 

2020-09-21 20:05:31,706 WARN [qtp1698322791-94] server.HttpChannel: /hbck.jsp
 java.lang.NullPointerException
 at org.apache.hadoop.hbase.generated.master.hbck_jsp.formatServerName(hbck_jsp.java:37)
 at org.apache.hadoop.hbase.generated.master.hbck_jsp._jspService(hbck_jsp.java:144)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:844)