av9ash/GitBugs huggingface dataset:
Bug: Make request guardrail configs in RSRpcServices live updatable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/23 18:34

We currently have 3 "guardrail" like configs in RSRpcServies:

hbase.rpc.rows.warning.threshold

hbase.rpc.rows.size.threshold.reject

hbase.server.scanner.max.result.size

It would be useful if these could be updatable with update_config
Bug: Push 2.4.16 release out
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/23 10:42

# Release the artifacts on repository.apache.org
# Move the binaries from dist-dev to dist-release
# Add to download page, which is tracked by HBASE-27610 as it requires code change
# Push tag 2.4.16RC1 as tag rel/2.4.16
# Release 2.4.16 on jira
# Add release data on https://reporter.apache.org/addrelease.html?hbase
# Send announcement email
Bug: Update the list of prefetched hfiles upon simple block eviction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/23 15:21
Affects Version/s: 2.6.0

Currently, we maintain a list of Hfiles on disk for which prefetch is complete and avoid prefetching those files after a restart. But we don't handle cases where blocks are evicted from the cache. This ticket is for updating the list of prefetched files upon simple block eviction.
Bug: Replication threads not attached to their parent process
Status: Open | Priority: Major | Resolution: 
Created: 27/Mar/23 10:16
Affects Version/s: 2.5.4

While debugging HBASE-27707 in a unit test, I see behaviour that I cannot explain. My test uses a minicluster, enables read replica replication, writes some data, concurrently kills a region server thread hosting a primary region, and then verifies that all replicas eventually show all data. Inspecting logs, noticed that replication source threads seem to continue working even after their associated region server is killed. Interspersing some thread dumps and sleeps, I can see that replication threads associated with the condemned region server are not being removed after it is killed. I think that this behaviour will render unreliably any replication test that relies on killing a source or sink region server. It also implies to me that the minicluster leaks replication threads and cannot be reliably recycled within a single jvm process.
Bug: Document protoc workarounds with Apple Silicon
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/23 12:16

Building hbase 2.x on Apple Silicon is difficult because there is no protoc library available.

[~ndimiduk] added a solution in HBASE-27741 to use osx-x86_64 but it is also possible to build protoc locally and use that.
Bug: Scan handlers in the RPC executor should match at least one scan queues
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/23 08:29
Affects Version/s: 2.5.3

This issue is try to avoid NO scan queues for some scan handlers.
For example, if we set hbase.regionserver.handler.count=150, hbase.ipc.server.callqueue.scan.ratio=0.1, hbase.ipc.server.callqueue.read.ratio=0.5, hbase.ipc.server.callqueue.handler.factor=0.1, then there will be 150 * 0.5 * 0.1 = 7 scan handlers, but there are 150 * 0.1 * 0.5 * 0.1 = 0 scan RPC queues.

When there are no scan rpc queues, all the scan and get requests will be dispatched to the read rpc queues, while we we thought they had been dealt with separately, since the scan handler count is not 0. When there are not enough handlers for large scan requests under this circumstance, the small get requests will be blocked in the rpc queues.

We can see from the codes,
{code:java}
int scanQueues = Math.max(0, (int) Math.floor(readQueues * callqScanShare));
int scanHandlers = Math.max(0, (int) Math.floor(readHandlers * callqScanShare));

if ((readQueues - scanQueues) > 0) {
  readQueues -= scanQueues;
  readHandlers -= scanHandlers;
} else {
  scanQueues = 0;
  scanHandlers = 0;
} {code}
when readQueues * callqScanShare < 1 but readHandlers * callqScanShare > 1, there will be some iddle scan handlers with NO scan queues.
Bug: Update documentation about how to enable encryption on WAL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/23 14:50

After HBASE-27632 and HBASE-27702, we removed SecureProtobufLogReader and SecureProtobufLogWriter, users do not need to specify the reader/writer class any more, they just need to enable WAL encryption.

Remove the related configurations such as 'hbase.regionserver.hlog.writer.impl' and 'hbase.regionserver.hlog.reader.impl' in hbase-default.xml, and also change ref guide to tell users how to enable WAL encryption for 2.6+.
Bug: Skip hbase:meta table scan if the normalizer not enabled for any of the tables as well as default table wise normalization disabled
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/23 12:29

Currently when the zk data is null considering normaliser on which leads to unnecessary hbase meta scans. Would be better to scan through meta only when normalizer enabled explictly.
{noformat}
  public boolean isNormalizerOn() {
    byte[] upData = super.getData(false);
    try {
      // if data in ZK is null, use default of on.
      return upData == null || parseFrom(upData).getNormalizerOn();
    } catch (DeserializationException dex) {
      LOG
        .error("ZK state for RegionNormalizer could not be parsed " + Bytes.toStringBinary(upData));
      // return false to be safe.
      return false;
    }
  }
{noformat}
{noformat}
  public boolean normalizeRegions(final NormalizeTableFilterParams ntfp,
    final boolean isHighPriority) throws IOException {
    if (regionNormalizerManager == null || !regionNormalizerManager.isNormalizerOn()) {
      LOG.debug("Region normalization is disabled, don't run region normalizer.");
      return false;
    }
    if (skipRegionManagementAction("region normalizer")) {
      return false;
    }
    if (assignmentManager.hasRegionsInTransition()) {
      return false;
    }

    final Set<TableName> matchingTables = getTableDescriptors(new LinkedList<>(),
      ntfp.getNamespace(), ntfp.getRegex(), ntfp.getTableNames(), false).stream()
        .map(TableDescriptor::getTableName).collect(Collectors.toSet());
    final Set<TableName> allEnabledTables =
      tableStateManager.getTablesInStates(TableState.State.ENABLED);
{noformat}
Bug: Refactor WAL.Reader implementation so we can better support WAL splitting and replication
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Feb/23 07:32

WAL splitting and replication has completely different requirements on WAL reader, the latter one is much comlicated as it wants to tail a WAL file which is currently being written.
I think split them into two classes is better here.

But we exposes two configurations out

'hbase.regionserver.hlog.reader.impl'
'hbase.regionserver.hlog.writer.impl'

which indicates that we only have one implementation for wal reader.

These two configurations are inrtoduced in 0.96, for reading the pre 0.94 sequence file format WAL file. And starting from 2.0.0, we have already dropped the support of reading sequence file format WAL file, so I think the configurations are useless now.
Now we stil require users to specify SecureProtobufLogReader and SecureProtobufLogWriter when enable WAL encryption, but actually they are both the only choice, so just use 'hbase.regionserver.wal.encryption' to determine whether we need to use SecureProtobufLogWriter to write WAL. And for reading, I do not think we should rely on the configuration, we should detect whether the WAL file itself is encrypted(I believe we have already done this in SecureProtobufLogReader.readHeader.

So in general, I prefer we just mark these two configurations deprecated and do not use them in our code base, so we can have different Reader implementations for WAL splitting and replication.
Bug: ByteBuff ref counting is too expensive for on-heap buffers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Mar/23 15:42

We recently were upgrading an internal project's hbase client from 1.2 to 2.5. The project does raw HFile.Reader reading of some hfiles. We noticed an over 4x performance regression with the new client. Profiling showed that the majority of the time was spent in SingleByteBuff.checkRefCount. That ref counting is necessary for off-heap reads, but this case was using on-heap. Removing checkRefCount eliminated the regression.

I asked on the dev list and we agreed to remove checkRefCount calls for on-heap buffers: https://lists.apache.org/thread/2qm0g8fbxb0no4ozfx7bw8xmkryt4zz0
Bug: HBase blocking thread on java.util.concurrent.ConcurrentHashMap.computeIfAbsent
Status: Open | Priority: Major | Resolution: 
Created: 17/Mar/23 03:31

Made hbase pe RandomRead test for 100% memory-cache in HBase 2.5.0 before.

We found that there are many blocking regionserver handler threads, and our cpu utilisation percentage is about 50% and cannot increase anymore.

Here is the blocking stack:

RpcServer.default.FPBQ.Fifo.handler=116,port=16020" #299 daemon prio=5 os_prio=0 tid=0x0000ffff825d5000 nid=0x208e51 waiting for monitor entry [0x0000ffbe67734000]

    java.lang.Thread.State: BLOCKED (on object monitor)
     at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
     - waiting to lock <0x0000fff75f465128> (a java.util.concurrent.ConcurrentHashMap$Node)
     at org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.getOrCreateTableMeter(MetricsTableQueryMeterImpl.java:77)
     at org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.updateTableReadQueryMeter(MetricsTableQueryMeterImpl.java:82)
     at org.apache.hadoop.hbase.regionserver.RegionServerTableMetrics.updateTableReadQueryMeter(RegionServerTableMetrics.java:93)
     at org.apache.hadoop.hbase.regionserver.MetricsRegionServer.updateReadQueryMeter(MetricsRegionServer.java:283)
     at org.apache.hadoop.hbase.regionserver.HRegion.metricsUpdateForGet(HRegion.java:7401)
     at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2667)
     at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2567)
     at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45945)
     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:385)
     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
     at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
     at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Bug: HFile splitting occurs during bulkload, the CREATE_TIME_TS of hfileinfo is 0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/23 02:45

If HFile splitting occurs during bulkload, the CREATE_TIME_TS of hfileinfo =0,When HFile is copied after splitting, CREATE_TIME_TS of the original file is not copied。
{code:java}
##BulkLoadHFilesTool.class 
/**
 * Copy half of an HFile into a new HFile.
 */
private static void copyHFileHalf(Configuration conf, Path inFile, Path outFile,
  Reference reference, ColumnFamilyDescriptor familyDescriptor) throws IOException {
  FileSystem fs = inFile.getFileSystem(conf);
  CacheConfig cacheConf = CacheConfig.DISABLED;
  HalfStoreFileReader halfReader = null;
  StoreFileWriter halfWriter = null;
  try {
    。。。
    HFileContext hFileContext = new HFileContextBuilder().withCompression(compression)
      .withChecksumType(StoreUtils.getChecksumType(conf))
      .withBytesPerCheckSum(StoreUtils.getBytesPerChecksum(conf)).withBlockSize(blocksize)
      .withDataBlockEncoding(familyDescriptor.getDataBlockEncoding()).withIncludesTags(true)
      .build();
// TODO .withCreateTime(EnvironmentEdgeManager.currentTime())      

halfWriter = new StoreFileWriter.Builder(conf, cacheConf, fs).withFilePath(outFile)
      .withBloomType(bloomFilterType).withFileContext(hFileContext).build();
    HFileScanner scanner = halfReader.getScanner(false, false, false);
    scanner.seekTo();
    do {
      halfWriter.append(scanner.getCell());
    } while (scanner.next());

    for (Map.Entry<byte[], byte[]> entry : fileInfo.entrySet()) {
      if (shouldCopyHFileMetaKey(entry.getKey())) {
        halfWriter.appendFileInfo(entry.getKey(), entry.getValue());
      }
    }
  } finally {
    。。。
  }
} 


##get lastMajorCompactionTs metric

  lastMajorCompactionTs = this.region.getOldestHfileTs(true);
...
  long now = EnvironmentEdgeManager.currentTime();
  return now - lastMajorCompactionTs;
...
##
public long getOldestHfileTs(boolean majorCompactionOnly) throws IOException {
  long result = Long.MAX_VALUE;
  for (HStore store : stores.values()) {
    Collection<HStoreFile> storeFiles = store.getStorefiles();
   ...
    for (HStoreFile file : storeFiles) {
      StoreFileReader sfReader = file.getReader();
     ...
      result = Math.min(result, reader.getFileContext().getFileCreateTime());
    }
  }
  return result == Long.MAX_VALUE ? 0 : result;
}{code}
Bug: LEAK: RefCnt.release() was not called before it's garbage-collected in regionserver
Status: Open | Priority: Major | Resolution: 
Created: 15/Mar/23 16:57
Affects Version/s: 2.5.2

2023-03-15 07:15:12,119 ERROR [RpcServer.default.FPBQ.Fifo.handler=50,queue=2,port=16020] util.ResourceLeakDetector: LEAK: RefCnt.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
        org.apache.hadoop.hbase.nio.RefCnt.<init>(RefCnt.java:59)
        org.apache.hadoop.hbase.nio.RefCnt.create(RefCnt.java:54)
        org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.<init>(MemStoreLABImpl.java:108)
        sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:54)
        org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:42)
        org.apache.hadoop.hbase.regionserver.MemStoreLAB.newInstance(MemStoreLAB.java:116)
        org.apache.hadoop.hbase.regionserver.SegmentFactory.createMutableSegment(SegmentFactory.java:81)
        org.apache.hadoop.hbase.regionserver.AbstractMemStore.resetActive(AbstractMemStore.java:93)
        org.apache.hadoop.hbase.regionserver.AbstractMemStore.<init>(AbstractMemStore.java:83)
        org.apache.hadoop.hbase.regionserver.DefaultMemStore.<init>(DefaultMemStore.java:78)
        sun.reflect.GeneratedConstructorAccessor62.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:54)
        org.apache.hadoop.hbase.util.ReflectionUtils.newInstance(ReflectionUtils.java:91)
        org.apache.hadoop.hbase.regionserver.HStore.getMemstore(HStore.java:371)
        org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:280)
        org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6359)
        org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1114)
        org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1111)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)
Bug: Add group of regionServer on Master webUI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/23 04:03

We do have a RSGroupList on webUI now, but it is still a little inconvenient if I just want to know which group a specific regionServer belongs to.

So add this info on webUI:

!image-2023-02-06-12-04-03-503.png|width=889,height=174!
Bug: Refactor Table Latency Metrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/23 15:56

Benefit:
 # Table Latency Metrics could removed after table has been moved away. Fix HBASE-27617
 # Could remove the hash lookup caused by metrics map from the hot request path.
 # Reduce the output in jmx. As if we use jmx_exporter to collect metrics to Prometheus, the operation overhead of performing regular matching in metric is relatively high(especially region metrics, which might be next step).
 # I think that could be the first step after hbase-metrics released. It seems that the roadmap indicates that we should replace directly use hadoop-metrics2 with hbase-metrics.

Influence:
 # The metrics structure in jmx will change.

Old:

!image-2023-03-01-23-55-14-095.png|width=450,height=407! 
New:

!image-2023-03-01-23-56-16-819.png|width=615,height=479!
Bug: chaos-daemon.sh should make use hbase script start/stop chaosagent and chaos monkey runner.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/23 07:46

Currently chaos-daemon.sh  is just adding the libs from HBASE_HOME which is failing because hadoop dependencies not present in the classpath would be better to use hbase script which adds all the relevant jars from different classpaths like HBASE_CLASSPATH and Hadoop paths so that things works as expected

.
Bug: Create a dummy metric implementation in ConnectionImplementation.
Status: Open | Priority: Major | Resolution: 
Created: 09/Mar/23 16:45
Affects Version/s: 2.5.0

This Jira is for branch-2 only.

If CLIENT_SIDE_METRICS_ENABLED_KEY conf is set to true, then we initialize metrics to MetricsConnection, otherwise it is set to null.
{code:java}
      if (conf.getBoolean(CLIENT_SIDE_METRICS_ENABLED_KEY, false)) {        
          this.metricsScope = MetricsConnection.getScope(conf, clusterId, this);        
          this.metrics = MetricsConnection.getMetricsConnection(this.metricsScope, this::getBatchPool, this::getMetaLookupPool);
      } else {
          this.metrics = null;      
      }
{code}

Whenever we want to update metrics, we always do a null check. We can improve this by creating a dummy metrics object and have an empty implementation. When we want to populate the metrics, we can check if metrics is an instance of dummy metrics.
Bug: Make metrics from MetricsTableLatenciesImpl removable
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 07/Feb/23 10:07

Now the table latency metrics will last forever on a RS.

Could reproduce this issue easily:
1. Create a test table and the region opened on RS_A;
2. Do a query;
3. Move the table to another group;
4. See JMX on RS_A, table latency metrics still exists and will not deleted.
Bug: Standby masters appear to not purge their old write-ahead logs
Status: Open | Priority: Major | Resolution: 
Created: 08/Mar/23 13:32
Affects Version/s: 2.5.0

Dropping this one here, haven't investigated yet. We noticed that on a cluster who's elected master had been active for quite a long time that the master store wal directory steadily grew in size. Failing over master cause the directory to be cleaned up.
Bug: Fake Cell is being passed to Filters and comparators during StoreFileScans
Status: Open | Priority: Major | Resolution: 
Created: 07/Mar/23 21:54
Affects Version/s: 2.2.7

I am trying to upgrade HBase version (client and server) from 1.2.0 to 2.2.6 and started seeing some unexpected behavior around discovery of ambiguous row in filter during StoreFileScans.

*Is it a valid case that filters and Comparators might see a fake cell passed to them if that row is set as an inclusive(by default) start row to skip preceding row during Store file scans during client side execution?*

When rows were persisted or updated on a table through bulkload, looks like a scan with specific column triggers a different behavior compared to a scan without columns which doesn't trigger this behavior.

From what I have troubleshooted so far, it looks like this is triggered during [lazy scan|https://github.com/apache/hbase/blob/rel/2.2.6/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L251-L256] inside [StoreScanner|https://github.com/apache/hbase/blob/rel/2.2.6/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L406-L410] with [StoreFileScanner|https://github.com/apache/hbase/blob/rel/2.2.6/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java#L388-L448] implementation where it eventually returns fake cell as current row on store heap [StoreFileScanner|https://github.com/apache/hbase/blob/rel/2.2.6/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java#L437-L444] thus passed to filter but it's actually filtered later and not returned to client. 

This was not the case with hbase 1.7.2. I have created couple of simple Tests using hbase 1.7.2 and hbase 2.2.6 that bulkloads some sample rows to table and creates a column specific Scan to reproduce behavior that I have been talking about.

I have simply copied KeyOnlyFilter, added few loggers to catch rowkeys being passed to filter and added few loggers to catch row keys returned as a result on client side.

Here is my working repo that demonstrate this diverged behavior [hbase-scans|https://github.com/tejkiran/hbase-scans]

I have a mapper that creates PUT with row keys 0, 2, 3 and bulkload those rows to table. When a scan is issued with 2.2.6 hbase API, it parses that start row on Scan to filter during server side execution.

Screenshot of discovered row keys in filter during server side .
 !image-2023-03-07-15-46-01-182.png! 

Screenshoot of discovered row keys in filter with hbase 1.7.2

 !image-2023-03-07-15-50-59-696.png!
Bug: Update README.md about how to request a jira account
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/23 07:40

Recently INFRA team developped a self serving tool for requesting jira account.

https://selfserve.apache.org/jira-account.html

So we no longer need to let contributor to send email to private@hbase, jsut let them go to this page and submit the request.
Bug: Improve FSUtils to directly obtain FSDataOutputStream
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/23 10:03

hbase interacts with hdfs and obtains FSDataOutputStream to generate HFiles. In order to support favoredNodes, reflection is used. The DistributedFileSystem has a more direct way to get the FSDataOutputStream，for example:dfs.createFile(path).permission(perm).create()...;  this API allows you to set various parameters, including favoredNodes. I think avoiding reflection can improve performance, and if you agree with me, I can optimize this part of the code;

Model:hbase-server

class:FSUtils

 
{code:java}
public static FSDataOutputStream create(Configuration conf, FileSystem fs, Path path,
FsPermission perm, InetSocketAddress[] favoredNodes) throws IOException {
if (fs instanceof HFileSystem) {
FileSystem backingFs = ((HFileSystem) fs).getBackingFs();
if (backingFs instanceof DistributedFileSystem) {
// Try to use the favoredNodes version via reflection to allow backwards-
// compatibility.
short replication = Short.parseShort(conf.get(ColumnFamilyDescriptorBuilder.DFS_REPLICATION,
String.valueOf(ColumnFamilyDescriptorBuilder.DEFAULT_DFS_REPLICATION)));
try {
return (FSDataOutputStream) (DistributedFileSystem.class
.getDeclaredMethod("create", Path.class, FsPermission.class, boolean.class, int.class,
short.class, long.class, Progressable.class, InetSocketAddress[].class)
.invoke(backingFs, path, perm, true, CommonFSUtils.getDefaultBufferSize(backingFs),
replication > 0 ? replication : CommonFSUtils.getDefaultReplication(backingFs, path),
CommonFSUtils.getDefaultBlockSize(backingFs, path), null, favoredNodes));{code}