av9ash/GitBugs huggingface dataset:
Bug: Exclude javax.servlet:servlet-api in hbase-shaded-testing-util
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/21 17:32
Affects Version/s: 2.4.0

{noformat}
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hbase.shaded.TestShadedHBaseTestingUtility
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 19.515 s <<< FAILURE! - in org.apache.hbase.shaded.TestShadedHBaseTestingUtility
[ERROR] org.apache.hbase.shaded.TestShadedHBaseTestingUtility  Time elapsed: 19.513 s  <<< ERROR!
java.io.IOException: Shutting down
	at org.apache.hbase.shaded.TestShadedHBaseTestingUtility.setUp(TestShadedHBaseTestingUtility.java:47)
Caused by: java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterMultiple exceptions
	at org.apache.hbase.shaded.TestShadedHBaseTestingUtility.setUp(TestShadedHBaseTestingUtility.java:47)
Caused by: java.io.IOException: Problem starting http server
	at org.apache.hbase.shaded.TestShadedHBaseTestingUtility.setUp(TestShadedHBaseTestingUtility.java:47)
Caused by: org.apache.hbase.thirdparty.org.eclipse.jetty.util.MultiException: Multiple exceptions
	at org.apache.hbase.shaded.TestShadedHBaseTestingUtility.setUp(TestShadedHBaseTestingUtility.java:47)
Caused by: java.lang.NoSuchMethodError: javax.servlet.ServletContext.createFilter(Ljava/lang/Class;)Ljavax/servlet/Filter;
	at org.apache.hbase.shaded.TestShadedHBaseTestingUtility.setUp(TestShadedHBaseTestingUtility.java:47)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   TestShadedHBaseTestingUtility.setUp:47 » IO Shutting down
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
{noformat}

The hadoop-common and hadoop-mapreduce-client-app dependencies are leaking javax.servlet:servlet-api:2.5.
Bug: Update dependencies in hbase-filesystem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/21 22:59
Affects Version/s: hbase-filesystem-1.0.0-alpha1

hbase-filesystem still has some old dependencies. Notably aws-java-sdk is at 1.11.525 whereas hadoop is all the way at 1.11.1026.

We're also still building HBase 2 against 2.1.4 instead of anything newer. Bump up the relevant dependencies to something more current and make sure the code still works.
Bug: SystemExitWhenAbortTimeout initialization exception when regionserver exits due to abort method
Status: Open | Priority: Major | Resolution: 
Created: 02/Aug/21 06:52
Affects Version/s: 2.1.4

We encountered a problem in the production environment. When the regionserver was shut down due to abort, the SystemExitWhenAbortTimeout could not be created correctly.
{noformat}
2021-07-31 10:14:36,792 WARN  [regionserver/xxx-hbase48:16020] regionserver.HRegionServer: Initialize abort timeout task failed
java.lang.IllegalAccessException: Class org.apache.hadoop.hbase.regionserver.HRegionServer can not access a member of class org.apache.hadoop.hbase.regionserver.HRegionServer$SystemExitWhenAbortTimeout
 with modifiers "private"
        at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
        at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296)
        at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:413)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1046)
        at java.lang.Thread.run(Thread.java:748){noformat}
Bug: Support passing a customized Configuration object when creating TestingHBaseCluster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/21 03:00

As per the discussion thread on dev list.

https://lists.apache.org/thread.html/r13939d06641624dc01dd1a17b7f1e1f2faf4840d1fc0900b86143254%40%3Cdev.hbase.apache.org%3E
Bug: Replication is stuck due to zero length wal file in oldWALs directory [master/branch-2]
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/21 05:12
Affects Version/s: 2.4.5

Clone of the parent issue for branch-2/master. Resolving the parent as the issue is fixed in 1.7.1 and the jira needs to be in a resolved state for release notes curation.
Bug: Formatter to convert from epoch time to human readable date format.
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 12:57

In shell, we have custom formatter to convert from bytes to Long/Int for long/int data type values.

Many times we store the epoch timestamp (event creation, updation time) as long in our table columns. Even after converting this column to Long, the date is not in a human readable format. We still have to convert this long into date using some bash shell tricks and it is not convenient to do for many columns. We can introduce a new format method called +toLongDate+ which signifies that we want to convert the bytes to long first and then to date.

Please let me know if any such functionality already exists and I am not aware of.
Bug: add option to disable scanMetrics in TableSnapshotInputFormat
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/21 16:02
Affects Version/s: 2.3.5

When running spark job with TableSnapshotInputFormat, we found that scan is very slower. We found that scanMetrics is hardcoded as enabled, spark's 

newAPIHadoopRDD uses DummyReporter in hadoop, which causes the following exception and 80% cpu time is spent on this exception handling. 

Need to provide an option to disable scanMetrics.
java.base@11.0.5/java.lang.Throwable.fillInStackTrace(Native Method)
java.base@11.0.5/java.lang.Throwable.fillInStackTrace(Throwable.java:787) => holding Monitor(java.util.MissingResourceException@258206255})
java.base@11.0.5/java.lang.Throwable.<init>(Throwable.java:292)
java.base@11.0.5/java.lang.Exception.<init>(Exception.java:84)
java.base@11.0.5/java.lang.RuntimeException.<init>(RuntimeException.java:80)
java.base@11.0.5/java.util.MissingResourceException.<init>(MissingResourceException.java:85)
java.base@11.0.5/java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:2055)
java.base@11.0.5/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1689)
java.base@11.0.5/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1593)
java.base@11.0.5/java.util.ResourceBundle.getBundle(ResourceBundle.java:1284)
app//org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:37)
app//org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:56) => holding Monitor(java.lang.Class@545605549})
app//org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterGroupName(ResourceBundles.java:77)
app//org.apache.hadoop.mapreduce.counters.CounterGroupFactory.newGroup(CounterGroupFactory.java:94)
app//org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:227)
app//org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
app//org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl$DummyReporter.getCounter(TaskAttemptContextImpl.java:110)
app//org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.getCounter(TaskAttemptContextImpl.java:76)
org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.updateCounters(TableRecordReaderImpl.java:311)
org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.nextKeyValue(TableSnapshotInputFormat.java:167)
Bug: In branch-1 L2 BC should not be the victimhandler of L1 BC when using combined BC
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/21 06:04
Affects Version/s: 1.7.0

Currently in branch-1, the block cache initialisation is:

{code:java}
  LruBlockCache l1 = getL1(conf);
    // blockCacheDisabled is set as a side-effect of getL1Internal(), so check it again after the call.
    if (blockCacheDisabled) return null;
    BlockCache l2 = getL2(conf);
    if (l2 == null) {
      GLOBAL_BLOCK_CACHE_INSTANCE = l1;
    } else {
      boolean useExternal = conf.getBoolean(EXTERNAL_BLOCKCACHE_KEY, EXTERNAL_BLOCKCACHE_DEFAULT);
      boolean combinedWithLru = conf.getBoolean(BUCKET_CACHE_COMBINED_KEY,
        DEFAULT_BUCKET_CACHE_COMBINED);
      if (useExternal) {
        GLOBAL_BLOCK_CACHE_INSTANCE = new InclusiveCombinedBlockCache(l1, l2);
      } else {
        if (combinedWithLru) {
          GLOBAL_BLOCK_CACHE_INSTANCE = new CombinedBlockCache(l1, l2);
        } else {
          // L1 and L2 are not 'combined'.  They are connected via the LruBlockCache victimhandler
          // mechanism.  It is a little ugly but works according to the following: when the
          // background eviction thread runs, blocks evicted from L1 will go to L2 AND when we get
          // a block from the L1 cache, if not in L1, we will search L2.
          GLOBAL_BLOCK_CACHE_INSTANCE = l1;
        }
      }
      l1.setVictimCache(l2);
    }
{code}
As the code above, L2 will always be the victimhandler of L1, no matter if we use combined blockcache or not. But as logic (in master & branch-2) L2 should not be the victimhandler of L1 when using combined BC. We should set the victimhandler only when we use InclusiveConbinedBC or we do not use CombinedBC.
Bug: Parallel scheduling of getClusterstatus interface  will lead to linear growth of HMaster memory
Status: Open | Priority: Major | Resolution: 
Created: 23/Jul/21 11:05
Affects Version/s: 1.3.1

when Mapreduce job and Spark job use TableInputFormat.getSplits  will call getclusterstatus ,and when running more ten job, HMaster memory is very high，Of course, the region quantity itself is a reason
Bug: TestingHBaseCluster should support running MapReduce jobs
Status: Open | Priority: Major | Resolution: 
Created: 20/Jul/21 20:06
Affects Version/s: 3.0.0-alpha-1

The existing HBaseTestingUtility optionally allows users to start a mini-YARN cluster for the purpose of running MapReduce jobs, such as testing applications that use HBase's MapReduce integration. 

The new TestingHBaseCluster is a cleaner interface than HBTU, but exposes less functionality, and in particular does not seem to allow for starting a YARN minicluster. This would make it difficult to use in some common circumstances, such as testing Phoenix's index builds, which use MapReduce. 

Many downstream projects that depend on HBase may also have batch logic done as MapReduce or other kinds of YARN jobs.
Bug: Copy HBTU to hbase-testing-util, rename the HBTU related classes in hbase-server and mark them as IA.LimitedPrivate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jul/21 15:31

See the discussions in the parent issue on why we want to do this.


This is the final decision:

https://issues.apache.org/jira/browse/HBASE-13126?focusedCommentId=17358108&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17358108

Maybe a possible way is that, we mark HBTU as deprecated and introduce another class for end users to use. And on the next major release, we just introduce a another class in hbase-server for our internal usage, and copy the whole HBTU related classes to hbase-testing-util module, and keep them there for a whole major release, and then we remove them when the next major release is out. So end users will have plenty of time to change their code.
Bug: Tooling to auto-convert 1.7.0 serialized tables to 1.7.1 compatible tables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/21 16:16
Affects Version/s: 1.7.0

As discussed in the parent issue, 1.7.0 introduced an incompatible table serialization change. We need tooling that auto converts these into 1.7.1 compatible table descriptors and table states.
Bug: Master UI hbck should provide a JSON formatted output option
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Aug/21 05:03

It used to be possible to get hbck's verdict of cluster status from the command line, especially useful for headless deployments, i.e. without requiring a browser with sufficient connectivity to load a UI, or scrape information out of raw HTML, or write regex to comb over log4j output. The hbck tool's output wasn't particularly convenient to parse but it was straightforward to extract the desired information with a handful of regular expressions. 

HBCK2 has a different design philosophy than the old hbck, which is to serve as a collection of small and discrete recovery and repair functions, rather than attempt to be a universal repair tool. This makes a lot of sense and isn't the issue at hand. Unfortunately the old hbck's utility for reporting the current cluster health assessment has not been replaced either in whole or in part. Instead:

{quote}
HBCK2 is for fixes. For listings of inconsistencies or blockages in the running cluster, you go elsewhere, to the logs and UI of the running cluster Master. Once an issue has been identified, you use the HBCK2 tool to ask the Master to effect fixes or to skip-over bad state. Asking the Master to make the fixes rather than try and effect the repair locally in a fix-it tool's context is another important difference between HBCK2 and hbck1. 
{quote}

Developing custom tooling to mine logs and scrape UI simply to gain a top level assessment of system health is unsatisfying. There should be a convenient means for querying the system if issues that rise to the level of _inconsistency_, in the hbck parlance, are believed to be present. It would be relatively simple to bring back the experience of invoking a command line tool to deliver a verdict. This could be added to the hbck2 tool itself but given that hbase-operator-tools is a separate project an intrinsic solution is desirable. 

An option that immediately comes to mind is modification of the Master's hbck.jsp page to provide a JSON formatted output option if the HTTP Accept header asks for text/json. However, looking at the source of hbck.jsp, it makes more sense to leave it as is and implement a convenient machine parseable output format elsewhere. This can be trivially accomplished with a new servlet. Like hbck.jsp the servlet implementation would get a reference to HbckChore and present the information this class makes available via its various getters.  

The machine parseable output is sufficient to enable headless hbck status checking but it still would be nice if we could provide operators a command line tool that formats the information for convenient viewing in a terminal. That part could be implemented in the hbck2 tool after this proposal is implemented.
Bug: Supports revoke  @ns specified permission
Status: In Progress | Priority: Minor | Resolution: 
Created: 19/Aug/21 08:29
Affects Version/s: 2.3.4

The current version does not support reclaiming the specified permissions，Only all permissions of @NS can be reclaimed。

Supports revoke @ns specified permission :revoke 'bobsmith', '@ns1', 'WRCA'

!1.jpeg!
Bug: let hbase.mapreduce.bulkload.assign.sequenceNumbers take effect in SecureBulkLoadManager.secureBulkLoadHFiles
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 26/Aug/21 03:11

HBASE-10958 Call Flush before BulkLoad to obtain the latest sequenceID to prevent data loss during replay. '_hbase.mapreduce.bulkload.assign.sequenceNumbers_' controls whether to flush before BulkLoad, but we pass true to whether to flush in *SecureBulkLoadManager*. If we bulkload frequently we flush a lot of small files. Can we make 'hbase.mapreduce.bulkload.assign.sequenceNumbers' work in SecureBulkLoadManager? This passes -1 to sequenceId, we won't loss data.

SecureBulkLoadManager.java. 

secureBulkLoadHFiles
{code:java}
// code placeholder
return region.bulkLoadHFiles(familyPaths, true, new SecureBulkLoadListener(fs, bulkToken, conf), request.getCopyFile(), clusterIds, request.getReplicate());
{code}
Hregion.java
{code:java}
// code placeholder
public Map<byte[], List<Path>> bulkLoadHFiles(Collection<Pair<byte[], String>> familyPaths,
    boolean assignSeqId, BulkLoadListener bulkLoadListener, boolean copyFile,
    List<String> clusterIds, boolean replicate)
{code}
Bug: update error message when executing deleteall with ROWPREFIXFILTER in meta table
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Sep/21 08:55
Affects Version/s: 2.4.11

ROWPREFIXFILTER doesn't work when execute deleteall with table hbase:meta。

 
{code:java}
def _deleteall_internal(row, column = nil,
                        timestamp = org.apache.hadoop.hbase.HConstants::LATEST_TIMESTAMP,
                        args = {}, all_version = true)
  # delete operation doesn't need read permission. Retaining the read check for
  # meta table as a part of HBASE-5837.
  if is_meta_table?
    raise ArgumentError, 'Row Not Found' if _get_internal(row).nil?
  end
  if row.is_a?(Hash)
    _deleterows_internal(row, column, timestamp, args, all_version)
  else
    d = _createdelete_internal(row, column, timestamp, args, all_version)
    @table.delete(d)
  end
end {code}
 

If we allow ROWPREFIXFILTER of deleteall on hbase:meta, we should fix the check condition,

If we don't allow, we should modify the exception message.
Bug: MetricsHBaseServer should record all kinds of Exceptions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Aug/21 11:41
Affects Version/s: 2.4.11

We can define a kind of Exception such as OtherExcpetions to record exceptions doesn't in the following kinds of exceptions. Only debug those exceptions by LOG.debug("Unknown exception type", throwable); is not helpful to find errors.
{code:java}
if (throwable != null) {
  if (throwable instanceof OutOfOrderScannerNextException) {
    source.outOfOrderException();
  } else if (throwable instanceof RegionTooBusyException) {
    source.tooBusyException();
  } else if (throwable instanceof UnknownScannerException) {
    source.unknownScannerException();
  } else if (throwable instanceof ScannerResetException) {
    source.scannerResetException();
  } else if (throwable instanceof RegionMovedException) {
    source.movedRegionException();
  } else if (throwable instanceof NotServingRegionException) {
    source.notServingRegionException();
  } else if (throwable instanceof FailedSanityCheckException) {
    source.failedSanityException();
  } else if (throwable instanceof MultiActionResultTooLarge) {
    source.multiActionTooLargeException();
  } else if (throwable instanceof CallQueueTooBigException) {
    source.callQueueTooBigException();
  } else if (throwable instanceof QuotaExceededException) {
    source.quotaExceededException();
  } else if (throwable instanceof RpcThrottlingException) {
    source.rpcThrottlingException();
  } else if (LOG.isDebugEnabled()) {
    LOG.debug("Unknown exception type", throwable);
  }
}
{code}
!RequestTooBigException.png|width=787,height=336!
Bug: Make it easier to orchestrate rolling upgrades
Status: Open | Priority: Minor | Resolution: 
Created: 24/Sep/21 14:10

In order to orchestrate rolling upgrades, we need a way to query VersionInfo on a per-server basis. Currently I can see version exposed in two ways:
 * One can hit the /dump endpoint, but this creates a ton of extra output and is not machine parseable.
 * One can call Admin.getClusterMetrics() with Option.HBASE_VERSION, but the resulting response only has a single getHBaseVersion() which only represents the version for the active HMaster

We could make it a lot easier to orchestrate rolling upgrades if we exposed the VersionInfo details for all servers in the cluster. At a minimum I think we could add it to the /jmx response, but it would also be very helpful to add it to Admin.
Bug: Add support for reloading balancer configs with BalanceRequest
Status: Open | Priority: Minor | Resolution: 
Created: 02/Sep/21 21:46

It's currently a pain to iterate on balancer configs. You need to make changes in hbase-site.xml, then find the full ServerName for the active HMaster, then execute {{update_configuration '<full server name>'}}in the shell, then run the balancer.

Finding the ServerName is actually quite annoying. The best way I've found is to look at the JMX dump and find {{tag.serverName}}, but that takes a bunch of steps. 

We can make this a good deal more convenient by adding direct support for reloading the balancer configs into the {{balance}} command.

This could look something like:

{{shell> balance \{RELOAD_CONFIGS => true}}}

Alternatively, we could add another string arg like:

{{shell> balance 'reload_config'}}

Either way, we'd add a new {{BalanceRequest$Builder#setReloadConfig(boolean)}}.
Bug: Avoid merging regions if tables has not reached the state where regions are not split because of number of regions
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 08/Jul/21 06:04
Affects Version/s: 1.7.0

we have a table on a cluster with 100 regions with default split policy (SteppingSplitPolicy). This is a small table and will not get loaded with too much data. Now if region size of table is smaller than the normalizer target region size than there are chances that normalizer will consider the regions for merges. But since the number of regions are small split policy will trigger split on next flush. This is a continuous loop and our cluster will be busy in these two actions.

We plan to consider number of regions and number of regionservers in creating plans for normalizer.
Bug: Clean shared state pollution to avoid flaky tests.
Status: Open | Priority: Minor | Resolution: 
Created: 22/Aug/21 03:44

h4. PR Link:[https://github.com/apache/hbase/pull/3615] 
h2. What is the purpose of this change
 * This PR cleans the state polluted by {{org.apache.hadoop.hbase.monitoring.TestTaskMonitor.testDoNotPurgeRPCTask}}.

 * It may be better to clean state pollutions so that some other tests won't fail in the future due to the shared state pollution.

h2. Reproduce the test failure
 * Run the test twice in the same JVM.

h2. Expected result:
 * The tests should run successfully when multiple tests that use this state are run in the same JVM.

h2. Actual result:
 * We get the failure:
{{[ERROR] Failures: [ERROR] testDoNotPurgeRPCTask:144 RPC Tasks have been purged! expected:<10> but was:<20>}}

h2. Why the test fails
 * Each time this test runs,10 RPCtasks are created and added to the ArrayList {{rpcTasks}}, but {{rpcTasks}} is not completely purged when the test ends. So next time the test starts, the remaining RPC tasks lead to an assertion failure.

h2. Fix
 * Clean all tasks when the test ends.
 * Another solution is to create a new {{TaskMonitor}} each time to avoid pollution similar to the other tests (e.g., {{testTaskMonitorBasics}}, {{testTasksGetAbortedOnLeak}}) in {{TestTaskMonitor}}. [Create a new TaskMonitor each time to avoid flaky tests. LALAYANG/hbase#1|https://github.com/LALAYANG/hbase/pull/1]