av9ash/GitBugs huggingface dataset:
Bug: opentelemetry-context jar missing at runtime causes MR jobs to fail
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Sep/22 09:17
Affects Version/s: 2.5.0

MR jobs like RowCounter, Import have a dependency on opentelemetry-context via TableInputFormat -> ConnectionFactory -> io.opentelemetry.context.ImplicitContextKeyed which causes these jobs to fail with missing class. This would be an addendum on top of HBASE-25811|(which added opentelemetry-api and opentelemetry-semconv) 


Error: java.lang.NoClassDefFoundError: io/opentelemetry/context/ImplicitContextKeyed
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:365)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:131)
	at org.apache.hadoop.hbase.mapreduce.TableInputFormat.initialize(TableInputFormat.java:193)
	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:162)
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:528)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
Caused by: java.lang.ClassNotFoundException: io.opentelemetry.context.ImplicitContextKeyed
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:365)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	... 24 more
Bug: Remove RejectedExecutionHandler for long/short compaction thread pools
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Aug/22 07:00
Affects Version/s: 2.4.13

As disscussed in https://github.com/apache/hbase/pull/4725

actually the max size of StealJobQueue is bouded by the VM limit of an array, and the OOM exception occurs before the rejection handler. So StealJobQueue is equivalent to be unbounded. I think the RejectionHandler may bring some confusions and make the codes a little puzzle.
Bug: Change HBCK2's setRegionState() to use HBCK's setRegionStateInMeta()
Status: Open | Priority: Minor | Resolution: 
Created: 05/Jul/22 19:41
Affects Version/s: 2.4.13

Replica region id is  not recognized by hbck2's setRegionState as it does not show up in meta. We run into cases that it needs to set region state in meta for replica regions in order to fix inconsistency. We ended up writing the state manually into meta table and did a master failover to sync state from meta table. 

 

hbck2's setRegionState needs to support replica region id and handles it nicely.

Currently, setRegionState does not use MasterRpcServices#setRegionStateInMeta. There is an issue in setRegionStateInMeta to support replica regions. After that is fixed, and setRegionState uses setRegionStateInMeta to set the region state, it will support replica Id.
Bug: Online schema migration causes the master to report a very high or negative number of QPS
Status: Open | Priority: Minor | Resolution: 
Created: 05/Sep/22 16:53
Affects Version/s: 2.4.12

We've seen this a few times when making an online schema change, e.g.:
{code:java}
alter 'foo', {NAME=>'e',VERSIONS=>'2147483646'} {code}
This causes the master to briefly show extremely high QPS per regionserver, and probably causes an integer overflow in the sum.

!Screen Shot 2022-09-05 at 18.31.00.png|width=859,height=323!

...

!Screen Shot 2022-09-05 at 18.31.06.png|width=856,height=322!

This could be related to the issue reported in HBASE-27242.
Bug: hide some sensitive configuration information in the UI
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Aug/22 17:09
Affects Version/s: 3.0.0-alpha-3

In the discussion about how to store keystore/truststore password securely, [~bbeaudreault]  mentioned and I quote here

"I agree that it seems insecure to put it directly into the hbase-site.xml. Another reason is due to the RS UI which (helpfully) can print the entire site configuration. We’d need to make sure the password is excluded from that, but better to remove it from site xml altogether".

I also felt that some sensitive information was exposed in the UI, for example, if we set superuser in the hbase-site.xml, the non-admin users can obtain superuser information and simulate superuser to perform some non-permitted operations on the cluster. So I think maybe we should hide these sensitive information in the UI.
Bug: Add support for unsigned varint to Bytes util class
Status: Open | Priority: Minor | Resolution: 
Created: 26/Aug/22 11:59

Our current Bytes.vintToBytes accepts a signed long, and the resulting encoded size is larger than it needs to be for small values. For example, Bytes.vintToBytes requires 3 bytes for values over 255, whereas an unsigned implementation can use 2 bytes for up to 16383. This continues across the spectrum, with the current requiring 4 bytes for values over 65535 and an unsigned impl can use 3 bytes up to 2097151, etc.

We can pull the implementation from protobuf's CodedInputStream and CodedOutputStream
Bug: add an option to skip file splitting when bulkload hfiles
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Aug/22 13:29
Affects Version/s: 3.0.0-alpha-3

When bulkload hfiles, if the key range of the hfile does not match the key range of the region, the BulkLoadHFilesTool will split hfile to fit make the key range of the new file match the key range of the region. If there are many files to be split, the load on the BulkLoadHFilesTool will be very high. Sometimes we want to avoid this situation, just directly fail and regenerate new hfiles. Here we try to introduce a new option, When the above problem is encountered, an exception will be thrown and let the upper client handle it.
Bug: Expose the archive option to end user
Status: Open | Priority: Minor | Resolution: 
Created: 27/Jul/22 02:15
Affects Version/s: 2.5.0

For the delete table procedure, we already have archive option, but the archive now will already be true and not provided to end user.

Actually its valuable to provide this option to the end user because some time, customer does not want this table archived when drop the table, it will waste the storage and time if hbase keep archive the table. Especially for the users who are using AWS s3 or Azure as storage which will take lots time for the rename and delete operation for big tables
Bug: In trace log mode, the client does not print callId/startTime and the server does not print receiveTime
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Aug/22 02:42
Affects Version/s: 2.5.0

*Question:*

In some scenarios in the production environment, users find that the request latancy is high on the client side, but it is not high on the server side (all through the internal monitor system). In this inconsistent scenario, we want to know if there is a problem with the client or the current network. Therefore, we want to know the time when the server receives the request. However, in Trace mode, no relevant information is printed.

 

A possible solution is that the client prints callId (so we can associate the corresponding request on the server) and startTime about the request，and the server prints receiveTime about the request. In this way, we can find out at what stage the problem occurs. 

 

The client can even wrap the startTime in the request, and the server can directly collect the time at the network level (netty framework process time and network transmission time and so on) which make it easy to monitor some network level metrics about the client requests.
Bug: Some Cell's implementation of toString() such as IndividualBytesFieldCell prints out value and tags which is too verbose
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Aug/22 20:59
Affects Version/s: 2.4.12

One of users sees cells >10Mb are logged when over limit at their client log. 

Checked the code, toString() behavior is not consistent, mostly does not include values and tags. Change toString() to exclude tags/value.
Bug: WALPrettyPrinter add print timestamp
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jul/22 13:42

hbase wal  xxxx-wal -p, print include timestamp

At first:

row=rowA, column=f:c1, type=Delete
    value: 
cell total size sum: 88

 

New:

row=rowA, column=f:c1, timestamp=1657851212679, type=Delete
    value: 
cell total size sum: 88
Bug: Tool to read the contents of the storefile tracker file
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Aug/22 12:49
Affects Version/s: 2.6.0

It will be useful to have a tool that provides an ability to read the contents of the tracker file (.filelist/f1 or f2)
Using the hdfs -cat or -text displays the contents of the tracker file in binary and there is no option to show the contents in plain text.
{code:java}
x[cloudbreak@cod--z4t08rqbuyms-master0 ~]$ sudo hdfs dfs -text s3a://odx-qe-bucket/odx-d7v40h/audit/cod--z4t08rqbuyms/hbase/data/default/one/6126beb5b349a1eee4b92987b78f1058/cf/.filelist/f1
22/04/05 19:51:14 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
22/04/05 19:51:14 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
22/04/05 19:51:14 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
22/04/05 19:51:14 INFO s3a.IDBDelegationTokenBinding: There is no Knox Token available, fetching one from IDBroker...
22/04/05 19:51:14 INFO idbroker.AbstractIDBClient: Authenticating with IDBroker requires Kerberos
22/04/05 19:51:14 INFO idbroker.AbstractIDBClient: Kerberos credentials are available, using Kerberos to establish a session. UGI=hbase/cod--z4t08rqbuyms-master0.odx-d7v4.svbr-nqvp.int.cldr.work@ODX-D7V4.SVBR-NQVP.INT.CLDR.WORK (auth:KERBEROS)
Apr 05, 2022 7:51:14 PM org.apache.knox.gateway.shell.KnoxSession createClient
INFO: Using default JAAS configuration
22/04/05 19:51:15 INFO s3a.IDBDelegationTokenBinding: Bonded to Knox token eyJqa3...OGdQmQ
22/04/05 19:51:15 INFO Configuration.deprecation: No unit for fs.s3a.connection.request.timeout(0) assuming SECONDS
22/04/05 19:51:15 INFO idbroker.AbstractIDBClient: Creating Knox CAB session using Knox DT eyJqa3...OGdQmQ ...
22/04/05 19:51:16 INFO s3a.S3AInputStream: Switching to Random IO seek policy
⑙��/%
 98ef4cf597be48598c8376bc1cac200d�&����22/04/05 19:51:16 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
22/04/05 19:51:16 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
22/04/05 19:51:16 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.{code}
Bug: Make RowTooBigException configurable for snapshot reads 
Status: Resolved | Priority: Minor | Resolution: Workaround
Created: 01/Aug/22 10:39

While trying to read snapshot data using TableSnapshotInputFormat in an MR job, we can skip OOME check (introduced by HBASE-10925 for region server side OOME protection ) since the allocated memory resources would be from MR tasks themselves (as compared to RS during live scans)
!Screenshot 2022-08-01 at 3.42.57 PM.png!
Bug: Issues building with OpenJDK 17
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Jul/22 23:55
Affects Version/s: 3.0.0-alpha-3

When building with OpenJDK 17, some errors appear:
 * The flag --illegal-access=permit is ignored, therefore its function of allowing access to private fields by reflection no longer works, One has to explicitly enable which packages can be accessed with --add-opens.
 * In order for the tests of kerberos code to run, one has to add the export of java.security.jgss/sun.security.krb5=ALL-UNNAMED

To address these issues, I have created pull request [https://github.com/apache/hbase/pull/4594]

 

Best regards.
Bug: Change JONI encoding in RegexStringComparator
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Jul/22 07:07

I change the engine of RegexStringComparator to JONI.
After that I sent a regex filter request, the RegionServer's heap memory usage spiked and the RegionServer did not work due to GC.
 
!rs-heap.png|width=609,height=55!
(RegionServer Heap Memory Usage)
 
{code:java}
INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1435ms
GC pool 'ParNew' had collection(s): count=1 time=1550ms
INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1073ms
GC pool 'ParNew' had collection(s): count=1 time=1534ms
INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1456ms
GC pool 'ParNew' had collection(s): count=1 time=1574ms
INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1297ms
GC pool 'ParNew' had collection(s): count=1 time=1415ms {code}
(RegionServer Log)
 
I checked the reason, it is said that when using UTF8Encoding, an infinite loop can occur if an invalid UTF8 is entered.
For trino, using NonStrictUTF8Encoding instead of UTF8Encoding.
(https://github.com/trinodb/trino/commit/ea66e8cb27b098a5cea184106fe245064351b567)

After changing the encoding of JoniRegexEngine to NonStrictUTF8Encoding in RegexStringComparator, it was confirmed that the heap memory usage spike was gone.
Bug: Fix tests that rely on EnvironmentEdgeManager in branch-2.4
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jul/22 15:40
Affects Version/s: 2.4.13

https://issues.apache.org/jira/browse/HBASE-25911 converted all System.currentTimeMillis() calls in master and branch-2 to use EnvironmentEdgeManager. For whatever reason, this was not backported to branch-2.4. 

There exists tests TestAppendTimeRange and TestIncrementTimeRange which rely on EnvironmentEdge injection. These tests are consistently timing out due to logic in RSRpcServices.getRemainingRpcTimeout which subtracts "now" from call.getReceiveTime(). 

In this case, the now calculation uses EnvironmentEdge, but the ServerCalls use System.currentTimeMillis(). This discrepancy causes the calculated timeout to be unexpectedly large, hanging the test.

We have two options:
 # Add EnvironmentEdgeManager in the narrowly necessary places to get this to work.
 # Revert branch-2.4's getTimeLimit to use System.currentTimeMillis.

I'm going to push a PR which does option 1, but I'm open to open 2 if that is desired instead.
Bug: ConnectionUtils.allOf should be moved to FutureUtils
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jul/22 03:02
Affects Version/s: 3.0.0-alpha-3

{{ConnectionUtils.allOf}}  has nothing to do with {{Connection}} and should be moved to {{FutureUtils}}, HBASE-26950 which only applied for branch-2 have already moved {{ConnectionUtils.allOf}},so here also moving it for master.
Bug: Add KeyRangeSplitRestriction
Status: Open | Priority: Minor | Resolution: 
Created: 13/Jul/22 17:54

A common schema design pattern is to add a shard/salt prefix to row keys. For example, you may start every row key with bytes 0 through f (16 shards). If HBase is unaware of this pattern, it may split regions such that start/end row lands outside of that range. This typically happens at the start and end of a table – if HBase tries to split the first region (empty start, 1 end) it will probably choose something between -127 and 1 as the first byte (-64) but the lowest possible rowkey for your table starts with byte 0. So this newly split region will remain empty forever and never get any requests.

I originally thought KeyPrefixRegionSplitRestriction could help with this, but that works through truncating split points. If you set the prefix to 1, you effectively limit the table to 16 regions max. This is impractical for any non-trivial table.

What we actually want to do is ensure that HBase is not allowed to create a split point below the known start range or after the known end range. So I suggest a KeyRangeSplitRestriction which disallows splits out of a configured range.
Bug: Report maxStoreFileCount in jmx
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jul/22 12:39

We currently report metrics like storeFileCount, maxStoreFileAge, etc. These are provided at the regionserver level and table table. We should also report maxStoreFileCount, which would be aggregated across all regions on the server or table(s).

This metric would be helpful in ensuring we don't get too close to the {{hbase.hstore.blockingStoreFiles }}limit before running compactions or slowing writes.
Bug: Report block cache size metrics separately for L1 and L2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jul/22 20:24

Currently, in terms of block cache size metrics, we report the following: blockCacheFreeSize, blockCacheCount, blockCacheSize, blockCacheEvictionCount. This isn't super useful when running with BucketCache, because in that case L1 (holding index/meta blocks) and L2 (holding data blocks) operate largely independently. Additionally, you might typically report on block cache sizes in terms of monitoring heap usage. But with BucketCache, the cache is off-heap, so it becomes a misleading metric in that case.

We have metrics l1CacheHitCount/MissCount/HitRatio/MissRatio, and the same for l2. We should add l1CacheFreeSize, l1CacheSize, l1CacheCount, l1CacheEvictionCount, and the same for l2.
Bug: ResourceLeakDetector stacktraces can be truncated when logged
Status: Open | Priority: Minor | Resolution: 
Created: 12/Jul/22 01:30
Affects Version/s: 2.5.0

Looks like ResourceLeakDetector stacktraces can be truncated by log4j, at least with the default configuration. This is unfortunate because what log4j will do then is print the first bit of the log line then only the end, which omits the crucial top part of the stack trace.

Should these be given the NamedQueueRecorder treatment like the slow log or balancer decisions, i.e. we would also log detected resource leaks and their complete stacktrace details into a system table? WDYT [~vjasani] [~zhangduo]
Bug: Failure to cleanup WAL split dir log should be at INFO level
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Jul/22 00:10

As part of the SCP, after we are done splitting WALs, we try removing the -splitting dirs but if the dir doesn't exist, dfs#delete fails with IOE. Since we are aware of this case, we just log the message but we don't interrupt SCP because handling of the failure is gracefully done.

Hence, failure to remove "-splitting" dir should not be mentioned as WARN log, let's convert this to INFO level:
{code:java}
LOG.warn("Remove WAL directory for {} failed, ignore...{}", serverName, e.getMessage()); {code}
 

Any other genuine failure to remove the splitting dir is anyways covered by this log, hence we don't have worry about keeping the above mentioned log at WARN level, we have anyways mentioned "ignore..." in the log message.
{code:java}
if (!fs.delete(splitDir, false)) {
  LOG.warn("Failed delete {}, contains {}", splitDir, fs.listFiles(splitDir, true));
} {code}
Bug: Consider adding ResourceLeakDetector to RefCnt
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 01/Jul/22 17:14

In my quest to figure out HBASE-27170, I ended up adding ResourceLeakDetector to RefCnt in my local fork. This immediately reports the leak to the logs, making it hard to miss. ResourceLeakDetector is already active in the netty layer, and the default state is designed to be very low overhead. We could add it directly to RefCnt, or make it further configurable if we're concerned about performance. Either way it's good to have and document, to aid future leak investigations.
Bug: Purge stale pom comment: "<!--Below formatting for dev-support/generate-hadoopX-poms.sh-->"
Status: Open | Priority: Trivial | Resolution: 
Created: 27/Sep/22 16:14

Any pom that has a hadoop-2.0 profile in it – all but the master branch – has this comment in the activation clause:

 

{{<!--Below formatting for dev-support/generate-hadoopX-poms.sh–>}}

 

It is baffling. It refers to a script removed in 2014 (Circa v0.98):

 

{{commit 37654f77a60ad0c13a84d6d6b39657789726df09}}
{{Author: Michael Stack <stack@apache.org>}}
{{Date:   Wed Mar 19 23:08:55 2014 +0000}}{{    HBASE-10781 Remove hadoop-one-compat module and all references to hadoop1}}{{    git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1579449 13f79535-47bb-0310-9956-ffa450edef68}}

 

Purge the comment (Noticed by [~ndimiduk] over in review of https://github.com/apache/hbase/pull/4740)