Bug: The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/21 12:12
Affects Version/s: 2.2.7

The batch api of HTable contains a param named results to store result or exception, its type is Object[].

If user pass an array with other type, eg: org.apache.hadoop.hbase.client.Result, and if we need to put an exception into it by some reason, then the ArrayStoreException will occur in AsyncRequestFutureImpl.updateResult, then the AsyncRequestFutureImpl.decActionCounter will be skipped, then in the AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the actionsInProgress again and again, forever.

It is better to add an cutoff calculated by operationTimeout, instead of only depend on the value of actionsInProgress.

BTW, this issue only for 2.x, since 3.x the implement has refactored.

How to reproduce:

1: add sleep in RSRpcServices.multi to mock slow response
{code:java}
try {
 Thread.sleep(2000);
 } catch (InterruptedException e) {
 e.printStackTrace();
 }
{code}
2: set time out in config
{code:java}
conf.set("hbase.rpc.timeout","2000");
conf.set("hbase.client.operation.timeout","6000");
{code}
3: call batch api
{code:java}
Table table = HbaseUtil.getTable("test");
 byte[] cf = Bytes.toBytes("f");
 byte[] c = Bytes.toBytes("c1");
 List<Get> gets = new ArrayList<>();
 for (int i = 0; i < 10; i++) {
 byte[] rk = Bytes.toBytes("rk-" + i);
 Get get = new Get(rk);
 get.addColumn(cf, c);
 gets.add(get);
 }
 Result[] results = new Result[gets.size()];
 table.batch(gets, results);
{code}
The log will looks like below:
{code:java}
[ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1 error for test processing localhost,16020,1624343786295
java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
	at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
	at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
	at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
	at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
	at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
	at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
	at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10  actions to finish on table: test
[INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10  actions to finish on table: test
[INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10  actions to finish on table: 
[INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10  actions to finish on table: test
[INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10  actions to finish on table: test
[INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10  actions to finish on table: test
[INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10  actions to finish on table: test
[INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10  actions to finish on table: test
{code}