av9ash/GitBugs huggingface dataset:
Bug: MAX_FILESIZE and MEMSTORE_FLUSHSIZE should not be set negative number
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/20 03:55
Affects Version/s: 2.4.0

Before create or alter table, we will do sanityCheck in TableDescriptorChecker. If MAX_FILESIZE or MEMSTORE_FLUSHSIZE < 0, use maxFileSizeLowerLimit or flushSizeLowerLimit instead to pass check. But the real value in TableDescriptor is still < 0, and we can see negative values ​​on the UI.

However in flush and split logic, MAX_FILESIZE and MEMSTORE_FLUSHSIZE will judge whether it's value <= 0 , if true, change to default value. This does not affect flush and split.

!image-2020-12-22-11-46-38-967.png!
Bug: HBase doesn't update and remove the peer config from hbase.replication.source.custom.walentryfilters if the config is already set on the peer. 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Dec/20 20:48

Currently, we cannot update the peer-based config even if we change the value of the config.
Secondly, once the configuration is added, there is no smooth way to remove the peer config.
Bug: Adding StoreContext
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Nov/20 19:55
Affects Version/s: 2.3.4

Adding StoreContext which contains the metadata about the HStore. This meta data can be used across the HFileWriter/Readers and other HStore consumers without the need of passing around the complete store and exposing its internals. This is a refactoring change which cleans up the existing code so that the following commits can leverage the context for a more maintainable code.
Bug: [hbase-operator-tools] Add a repair tool for moving stale regions dir not present in meta away from table dir
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/20 13:48
Affects Version/s: hbase-operator-tools-1.0.0

This adds a new tool under *hbase-tools* module, that allows for moving aways regions dirs existing under table's hdfs dir, but not in meta. This is useful in cases where the region is not present in meta, but still has data on hdfs, yet no holes in the table region chain is detected. 

On such cases, the existing *HBCK2 addFsRegionsMissingInMeta* command isn't ideal, as it would bring the region back in meta and cause overlaps. 

This tool performs the following actions:
1) Identifies regions in hdfs but not in meta using *HBCK2.reportTablesWithMissingRegionsInMeta*;
2) For each of these regions, sidelines the related dir to a temp folder;
3) Bulkload hfiles from each sidelined region to the related table;

Sidelined regions are never removed from temp folder. Operators should remove those manually, after they certified on data integrity.
Bug: Add BYTE unit in PrettyPrinter.Unit 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Dec/20 08:16
Affects Version/s: 3.0.0-alpha-1

Currently only TTL supports PrettyPrinter, and MAX_FILESIZE is not human readable：

!image-2020-12-23-16-12-42-210.png!

This issue add a new unit 'BYTE' to pretty print value of size, such as MAX_FILESIZE, MEMSTORE_FLUSHSIZE.
Bug: Provide a VM-based release environment
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 18:58
Affects Version/s: 3.0.0-alpha-1

In order to make running releases more smooth and "clean-room" on my macOS dev machine, I put together a Vagrant environment for running release builds. In particular, docker IO on macOS is terribly slow, so slow that going through the virtualization kernel extensions is an accelerant. Of course, YMMV. I've used this on all of the 2.3.x release I've done. Maybe other RMs like it, maybe they don't. I'll post up a patch as a point of reference.
Bug: HBaseAdmin#getRegion() needs to filter out non-regionName and non-encodedRegionName
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/20 19:30
Affects Version/s: 2.3.3

I was running shell command to major compact meta table. The implementation is wrong because it tries to search the meta table with meta table name. This also results in an unnecessary scan of meta table. 

 

majorCompactRegion() is calling HBaseAdmin#getRegion() which basically scan meta table itself.

This command is being used by operator quite often, we need to correct it.

 

This applies to split/flush command as well, which calls getRegion() with tableName as an input.

 

The solution is that getRegion() needs to filter out non-regionName and non-encodedRegionName, this will save a query of meta table and a heavy scan of meta table. If meta table size is large, the overhead is huge.
Bug: SlowDelete & SlowPut metric value should use updateDelete & updatePut 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/20 06:46

 !image-2020-12-22-14-44-26-030.png! 

SlowDelete & SlowPut metric value should use updateDelete & updatePut API

like this 
 !image-2020-12-22-14-45-48-727.png!
Bug: Add maven enforcer rule to ban VisibleForTesting imports
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/20 03:02

https://github.com/google/error-prone/blob/master/annotations/src/main/java/com/google/errorprone/annotations/RestrictedApi.java

This is a better choice if you want to restrict the usage of a special API.
Bug: remoteProc is suspended due to OOM ERROR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Dec/20 03:49
Affects Version/s: 2.2.3

https://github.com/apache/hbase/blob/0f868da05d7ffabe4512a0cae110ed097b033ebf/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java#L317
If resource leakage occurs due to other components or reasons, BufferNode#dispatch() may fail. and then TimeoutExecutorThread will exit the while (running.get()), and some procs will stuck...
 !image-2020-12-26-11-49-38-018.png!
Bug: Forgetting to unbuffer streams results in many CLOSE_WAIT sockets when loading files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/20 06:48
Affects Version/s: 2.4.0

HBASE-9393 found seek+read will leave many CLOSE_WAIT sockets without stream unbuffer, which can free sockets and file descriptors held by the stream. 

In our cluster RSes with about one hundred thousand store files, we found the number of  CLOSE_WAIT sockets increases with the number of regions opened, and can up to the operating system open files limit 1000000.

 
{code:java}
2020-11-12 20:19:02,452 WARN  [1282990092@qtp-220038608-1 - Acceptor0 SelectChannelConnector@0.0.0.0:16030] mortbay.log: EXCEPTION
java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75)
        at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:686)
        at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
        at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{code}
 
{code:java}
[hbase@gha-data-hbase-cat0053 hbase]$ ulimit -SHn
1000000
{code}
 

 

The reason of the problem is, when store file opened, 
{code:java}
private void open() throws IOException {
  fileInfo.initHDFSBlocksDistribution();
  long readahead = fileInfo.isNoReadahead() ? 0L : -1L;
  ReaderContext context = fileInfo.createReaderContext(false, readahead, ReaderType.PREAD);
  fileInfo.initHFileInfo(context);
  StoreFileReader reader = fileInfo.preStoreFileReaderOpen(context, cacheConf);
  if (reader == null) {
    reader = fileInfo.createReader(context, cacheConf);
    fileInfo.getHFileInfo().initMetaAndIndex(reader.getHFileReader());
  }
....{code}
only createReader() unbuffered the stream. In initMetaAndIndex(), using the stream to read blocks, so it needs to unbuffer() the socket , too.

We can just add try before fileInfo.initHFileInfo(context); and finally unbuffer() the stream at the end of the open() function.

We fixed it on our cluster, the number of CLOSE_WAIT reduced to about 0.
Bug: public static method MultiTableHFileOutputFormat.configureIncrementalLoad actually cannot be invoked outside hbase package
Status: Open | Priority: Major | Resolution: 
Created: 25/Dec/20 09:14
Affects Version/s: 2.4.0

*MultiTableHFileOutputFormat* provides a public static method _configureIncrementalLoad_ to configure reducer for multi table hfile output. 

See this. [https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableHFileOutputFormat.java#L92]

However the *TableInfo* in the parameter is NOT a public class, which makes it impossible to invoke *MultiTableHFileOutputFormat*._configureIncrementalLoad_ in other project. 

Is there any alternative for the use case? For now I have to create another class in +org.apache.hadoop.hbase.mapreduce+ package to invoke the *MultiTableHFileOutputFormat*._configureIncrementalLoad._
Bug: The parameter "hbase.bucketcache.size" is misdescribed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/20 09:54
Affects Version/s: 2.0.0

You can see that the description of this configuration is outdated
{quote}<property>
 <name>hbase.bucketcache.size</name>
 <value></value>
 <description>A float that EITHER represents a percentage of total heap memory
 size to give to the cache (if < 1.0) OR, it is the total capacity in
 megabytes of BucketCache. Default: 0.0</description>
 </property>
{quote}
!image-2020-12-28-17-53-32-575.png!
Bug: Add a filter with containing semantic
Status: Open | Priority: Major | Resolution: 
Created: 25/Dec/20 03:35
Affects Version/s: 3.0.0-alpha-1

Say we want to check whether a cell value is within a set of values.
If the size of the set is 1000, we need to create 1000 SingleColumnValueFilter with CompareOperator.EQUAL and wrap them with a FilterListWithOR. The scan performance of these filters would be very bad.
But if we have a set filter, we just need to create only one filter with 1000 values.  And the containing operation could be implemented as O(1) or O(log( n )).
This `SetFilter` is general and useful in my opinion, so would the community consider to implement a `SetFilter`?
Bug: create-release fails download of artifacts on gcp
Status: Open | Priority: Major | Resolution: 
Created: 23/Dec/20 21:14

Noticed this running test release-candidate builds up on gcp. The build fails as we fail to download artifacts ("Connection reset"). For example:

{code}
2020-12-23T16:49:05Z Command FAILED. Check full logs for details.
16:49:05 [ERROR] Failed to execute goal on project hbase-metrics: Could not resolve dependencies for project org.apache.hbase:hbase-metrics:jar:2.4.1-rm_test_gcp: Failed to collect dependencies at io.dropwizard.metrics:metrics-core:jar:3.2.6: Failed to read artifact descriptor for io.dropwizard.metrics:metrics-core:jar:3.2.6: Could not transfer artifact io.dropwizard.metrics:metrics-core:pom:3.2.6 from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-core/3.2.6/metrics-core-3.2.6.pom: Connection reset -> [Help 1]
{code}

If I rerun, we usually move beyond the above but might fail later on another artifact.

HBASE-25438 updates the mvn used building to 3.6.3 which is supposed to have a better wagon plugin but it doesn't seem to help (I tried turning off wagon pooling and upping retries but probably no good when the connection reset)

This looks related... trying it.

https://stackoverflow.com/questions/65349648/why-does-my-maven-dependency-fetching-fail-with-a-connection-reset-when-running

 I'm on debian when above happens:
Bug: After droped the table, using snapshot recovery to cause data consistency problems
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Dec/20 08:57

We discovered a consistency problem during use HBase. The cause of the problem should be that after the table is dropped, when the snapshot is restored, in some cases the wal log will be applied, thereby the wal log of the dropped table restored to the restored table, because their region id are the same.
Steps to reproduce
1. Create a tablea
2. Make a snapshot1
3. Insert some records r1
4. Drop the table
5. Restore snapshot 1
6. Down RS or move the region, etc., wal is reapplied
You may see the data r1 in the snapshot1 table. The problem is not a stable recurrence, but there is a certain probability that it can recur.
Bug: Refresh in-memory table-info cache in HBase
Status: Open | Priority: Major | Resolution: 
Created: 21/Dec/20 13:34

In HBASE-25297 we intorduced the feature in hbck2 to fix orphan tables by generating the missing .tableinfo file. If the table descriptor can not be found in the tableinfo cache of the master, then hbck2 will generate a new file with default table configurations. In this case hbck2 could re-trigger a refresh of the in-memory table info cache of the master node.

We would need to add a new master RPC API to refresh the cache for a given table. Alternatively we should also test if a table disable / enable would do the same.
Bug: After droped the table, using snapshot recovery to cause data consistency problems
Status: Open | Priority: Major | Resolution: 
Created: 21/Dec/20 07:30

We discovered a consistency problem during use HBase. The cause of the problem should be that after the table is dropped, when the snapshot is restored, in some cases the wal log will be applied, thereby the wal log of the dropped table restored to the restored table, because their region id are the same.
Steps to reproduce:
1. Create a table
2. Make a snapshot1
3. Insert some records r1
4. Drop the table
5. Restore snapshot 1
6. Down RS or move the region, etc., wal is reapplied
You may see the data r1 in the snapshot1 table. The problem is not a stable recurrence, but there is a certain probability that it can recur.

 

it looks like the problem:
https://issues.apache.org/jira/browse/HBASE-23741

It will still appear after we apply the relevant patch
Bug: [branch-2.2] Fix the ruby problem for pre-commit
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Dec/20 07:07
Affects Version/s: 2.2.6

*00:33:07*  [91mERROR:  Error installing rubocop:*00:33:07*  	parallel requires Ruby version >= 2.4.*00:33:08*  [0mSuccessfully installed jaro_winkler-1.5.4
Bug: Backup/Restore hbase cell tags.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/20 17:51

In PHOENIX-6213 we are planning to add cell tags for Delete mutations. After having a discussion with hbase community via dev mailing thread, it was decided that we will pass the tags via an attribute in Mutation object and persist them to hbase via phoenix co-processor. The intention of PHOENIX-6213 is to store metadata in Delete marker so that while running Restore tool we can selectively restore certain Delete markers and ignore others. For that to happen we need to persist these tags in Backup and retrieve them in Restore MR jobs (Import/Export tool). 
Currently we don't persist the tags in Backup.
Bug: TestRSGroupsFallback.testFallback is flaky
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/20 10:54

Like in CI test results of PR [https://github.com/apache/hbase/pull/2699]

failed UTs site is [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2699/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt]

 

In this unit test, it checks if all table regions assigned after balance, and then assert for the RS group of regions.

But balance() uses aync move, and will throttle move regions, sleeping between all the table regions are moved to its RSGroup.

If waiting time is not longer than the region movement duration, the assertion will be fail.
Bug: [Flakey Tests] branch-2 TestRegionMoveAndAbandon
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/20 04:58

Looks to be same as 'HBASE-25389 [Flakey Tests] branch-2 TestMetaShutdownHandler (#2773)' One-liner seems to shut this flakey down (I'm running unit tests locally.. it has failed a few times on me).
Bug: Cookie and Referrer policy vulnerabilities reported by scanner tool
Status: Open | Priority: Major | Resolution: 
Created: 16/Dec/20 17:02

Vulnerability scanner has reported the following:

1.Cookies with missing, inconsistent or contradictory properties i)cookie without SameSite attribute
 * https://<rs-ip>:<rs-port>/region.jsp

Remediation: To ensure that the cookies configuration complies with the applicable standards, Setting Same-Site attribute to Set-Cookie HTTP response header

Plan: Make Same-Site Configurable

 

2.Insecure Referrer Policy

URLs where Referrer policy configuration is insecure:
 * https://<rs-ip>:<rs-port>/rs-status
 * https://<rs-ip>:<rs-port>/region.jsp
 * https://<rs-ip>:<rs-port>/
 * https://<rs-ip>:<rs-port>/conf
 * https://<rs-ip>:<rs-port>/logLevel
 * https://<rs-ip>:<rs-port>/logs/

Remediation: Consider setting Referrer-Policy header to 'strict-origin-when-cross-origin' or a stricter value

Plan: Make Referrer-Policy header Configurable.
Bug: [Flakey Tests] branch-2 TestMetaShutdownHandler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/20 21:42

I see this in local runs fail regularly. We kill the server hosting meta and then check it came up in a new location after waiting on recovery. In the test, when it fails, the assert on new location fails because we have not waited for the CRASH to happen. Here is excerpt from log:

{code}
 2020-12-11 13:20:27,298 INFO  [Listener at localhost/62149] master.TestMetaShutdownHandler(111): Deleted the znode for the RegionServer hosting hbase:meta; waiting on SSH
...
 2020-12-11 13:20:27,310 INFO  [Listener at localhost/62149] master.TestMetaShutdownHandler(122): Past wait on RIT
...

 2020-12-11 13:20:27,351 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1048): Stored pid=9, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure stack.XXX.example.com,62201,1607721618377, splitWal=true, meta=true
{code}

The first line is where we remove the ephemeral node for the regionserver carrying hbase:meta. The second line is supposed to log AFTER SCP is done (it calls it SSH in this old test above). Notice how the 3rd line, after the 2nd, is first mention of SCP being queued.
Bug: CopyTable and Coprocessor based export tool should backup and restore cell tags.
Status: Open | Priority: Major | Resolution: 
Created: 12/Dec/20 22:23

In HBASE-25246 we added support for Mapreduce based Export/Import tool to backup/restore cell tags. Mapreduce based export tool is not the only tool that takes snapshot or backup of a given table.
We also have Coprocessor based Export and CopyTable tools which takes backup of a given table. We need to add support for the above 2 tools to save cell tags to file.
Bug: Move hbase-downstreamer under hbase-connectors
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/20 21:11

The [hbase-downstreamer |https://github.com/saintstack/hbase-downstreamer] project lives in my personal github. I used to use it trying out hbase release candidates from the perspective of a user's application. Subsequently [~busbey] has used it (and fixed it) from time to time testing as has our [~ndimiduk]. Just recently Nick found a bug trying hbase-downstreamer on an RC: HBASE-25384.

In the dev thread, "[VOTE] First release candidate for HBase 2.4.0 (RC1) is available", its suggested that hbase-downstreamer move into hbase. This issue is about making this move. Was going to move it under the hbase-connectors project. That seems like best place as I look across the pantheon of sub-projects.

Presuming you good with this move [~busbey].... since you made contribs... You too [~ndimiduk].
Bug: Fix flaky test TestClassFinder#testClassFinderDefaultsToOwnPackage 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 01:52

The test {{TestClassFinder#testClassFinderDefaultsToOwnPackage}} compares two sets from {{allClassesFinder.findClasses}}. However, the function {{allClassesFinder.findClasses}} does not return the deterministic order result. From code trace, the root cause is at {{ClassFinder.findClassesFromJar}} and {{ClassFinder.findClassesFromFiles}}. They use {{HashSet}} to store {{classes}} and {{listFiles}} to get file array. However, {{HashSet}} and {{listFiles}} do not guarantee that order, and the assertion can fail if the order differs.
Bug: meta replica regions are assigned to the same region server during SCP.
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 04/Dec/20 05:45
Affects Version/s: 2.4.0

When running 2.4.0 RC1 with meta replica enabled, during SCP, meta replica regions are assigned to the same region server. I think the reason is that SCP uses round robin algo to assign meta replicas and do not exclude  region servers hosting replica regions. This is not a new issue.
Bug: one NPE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/20 02:56

* getData can return null at 
 [https://github.com/apache/hbase/blob/1726160839368df14602da1618e3538955b25f74/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java#L615]
 or
 [https://github.com/apache/hbase/blob/1726160839368df14602da1618e3538955b25f74/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java#L619]
 all its caller have null checker except at
 [https://github.com/apache/hbase/blob/1726160839368df14602da1618e3538955b25f74/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java#L467]

We shoud add null check for pontential NPEs.
Bug: Exclude ZooKeeper from hadoop-common transitive dependency for Hadoop2 profile
Status: Open | Priority: Major | Resolution: 
Created: 10/Dec/20 21:51
Affects Version/s: 2.3.0

During RC validation process for both 2.3.0 and 2.4.0, using [~stack]'s [hbase-downstreamer|https://github.com/saintstack/hbase-downstreamer] project, we see a failure:

{noformat}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 8.955 sec <<< FAILURE! - in org.hbase.downstreamer.TestHBaseMiniCluster
testSpinUpMiniHBaseCluster(org.hbase.downstreamer.TestHBaseMiniCluster)  Time elapsed: 8.922 sec  <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/zookeeper/common/X509Exception$SSLContextException
        at org.hbase.downstreamer.TestHBaseMiniCluster.testSpinUpMiniHBaseCluster(TestHBaseMiniCluster.java:21)
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.common.X509Exception$SSLContextException
        at org.hbase.downstreamer.TestHBaseMiniCluster.testSpinUpMiniHBaseCluster(TestHBaseMiniCluster.java:21)
{noformat}
Bug: [create-release] Fix double .asc
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 20:13

We are signing the signature file when we run the create-release which results in *.jar.asc.asc files up in maven repository. It looks to be this line in release-util.sh maven_deploy:

- declare -a mvn_goals=(clean install)
 + declare -a mvn_goals=(clean)
Bug: Fix typo in ban-jersey section of the enforcer plugin in pom.xml
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 14:54

{code}
                  <excludes>
                    <exclude>>org.glassfish.jersey.containers:**</exclude>
                    <exclude>>org.glassfish.jersey.core:**</exclude>
                  </excludes>
{code}

Need to remove the extra '>'.

This was introduced when we shaded jetty and jersey.