av9ash/GitBugs huggingface dataset:
Bug: DistCp derives if XAttr is supported wrongly
Status: Patch Available | Priority: Major | Resolution: 
Created: 20/Sep/21 08:49

{color:#0033b3}public static void {color}{color:#00627a}checkFileSystemXAttrSupport{color}({color:#000000}FileSystem {color}fs)
 {color:#0033b3}throws {color}{color:#000000}XAttrsNotSupportedException {color}{
 {color:#0033b3}try {color}{
 fs.getXAttrs({color:#0033b3}new {color}Path({color:#000000}Path{color}.{color:#871094}SEPARATOR{color}));
 } {color:#0033b3}catch {color}({color:#000000}Exception {color}e) {
 {color:#0033b3}throw new {color}XAttrsNotSupportedException({color:#067d17}"XAttrs not supported for file system: "
{color} + fs.getUri());
 }
}

IOExecption will also be treated as if Xattr is not supported, which would be wrong.
Bug: Some Hadoop 3.3.1 jdiff files are missing
Status: Open | Priority: Major | Resolution: 
Created: 29/Sep/21 06:02

When releasing 3.3.1, HDFS jdiff file is added by https://github.com/apache/hadoop/commit/a77bf7cf07189911da99e305e3b80c589edbbfb5, but the other jdiff files are missing.

In addition, the base version is still 2.7.2 in hadoop-yarn and hadoop-mapreduce when running jdiff. The version should be upgraded.
Bug: Backport HADOOP-15993 to branch-3.2 which address CVE-2014-4611
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/21 14:57

Now the version is 0.8.2.1 and it has net.jpountz.lz4:lz4:1.2.0 dependency, which is vulnerable. ([https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-4611])

 

cc./ [~aajisaka]
Bug: Upgrade OpenSSL to 1.1.1l for vulnerability fix
Status: Open | Priority: Major | Resolution: 
Created: 20/Sep/21 14:00

A vulnerability scan reported the following CVEs in openSSL 1.1.1k in Hadoop:

CVE-2021-3711

CVE-2021-3712

Affects jars :

libssl.so.1.1

libcrypto.so.1.1

libcrypto.a

libssl.a
Bug: Improve PrometheusSink for Namenode TopMetrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/21 11:18
Affects Version/s: 3.4.0

HADOOP-16398 added exporter for hadoop metrics to prometheus. But some of metrics can't be exported  validly. For example like these metrics, 

1.  -queue metrics for ResourceManager-
{code:java}
queue_metrics_max_capacity{queue="root.queue1",context="yarn",hostname="rm_host1"} 1
// queue2's metric can't be exported queue_metrics_max_capacity{queue="root.queue2",context="yarn",hostname="rm_host1"} 2
{code}
-It always exported  only one queue's metric because PrometheusMetricsSink$metricLines only cache one metric  if theses metrics have the same name no matter these metrics has different metric tags.-

 

2. -rpc metrics for Namenode-

-Namenode may have rpc metrics with multi port like service-rpc. But because  the same reason  as  Issue 1, it wiil lost some rpc metrics if we use PrometheusSink.-
{code:java}
rpc_rpc_queue_time300s90th_percentile_latency{port="9000",servername="ClientNamenodeProtocol",context="rpc",hostname="nnhost"} 0
// rpc port=9005 metric can't be exported 
rpc_rpc_queue_time300s90th_percentile_latency{port="9005",servername="ClientNamenodeProtocol",context="rpc",hostname="nnhost"} 0
{code}
3. TopMetrics for Namenode

org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics is a special metric. And I think It is essentially a Summary metric type. TopMetrics record name will according to different user and op ,  which means that these metric will always exist in PrometheusMetricsSink$metricLines and it may cause the risk of its memory leak. We e need to treat it special. 
{code:java}
// invaild topmetric export
# TYPE nn_top_user_op_counts_window_ms_1500000_op_safemode_get_user_hadoop_client_ip_test_com_count counter
nn_top_user_op_counts_window_ms_1500000_op_safemode_get_user_hadoop_client_ip_test_com_count{context="dfs",hostname="nn_host",op="safemode_get",user="hadoop/client-ip@TEST.COM"} 10

// it should be 
# TYPE nn_top_user_op_counts_window_ms_1500000_count counter
nn_top_user_op_counts_window_ms_1500000_count{context="dfs",hostname="nn_host",op="safemode_get",user="hadoop/client-ip@TEST.COM"} 10{code}
Bug: memory field in SequenceFile.Sorter java int overflow
Status: Open | Priority: Major | Resolution: 
Created: 17/Sep/21 02:41
Affects Version/s: 3.3.1

memory field in SequenceFile.Sorter, can cause run method in SequenceFile.Sorter.SortPass enter an endless loop.


If you set the "io.sort.mb" attribute to a value greater than 2050, as a result of
{code:java}
this.memory = conf.getInt(CommonConfigurationKeys.IO_SORT_MB_KEY, CommonConfigurationKeys.SEQ_IO_SORT_MB_DEFAULT) * 1024 * 1024;{code}
, memory field java int overflow, may become negative, so memoryLimit field in SequenceFile.Sorter.SortPass will be negative too, lead to run method in SequenceFile.Sorter.SortPass enter an endless loop.
Bug: Refactor pkg-resolver installation
Status: Open | Priority: Major | Resolution: 
Created: 17/Sep/21 06:02
Affects Version/s: 3.4.0

In the view of keeping the Dockerfiles free from installation logic, we need to move the code for installing pkg-resolver to a script file and only run this script file from Dockerfile. Right now, we're installing the dependencies of pkg-resolver directly in the Dockerfile itself.
Bug: Fix compilation error of ITUseHadoopCodecs with -DskipShade
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 16/Sep/21 07:49

{noformat}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hadoop-client-integration-tests: Compilation failure: Compilation failure:
[ERROR] /home/centos/srcs/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseHadoopCodecs.java:[34,28] cannot find symbol
[ERROR]   symbol:   class RandomDatum
[ERROR]   location: package org.apache.hadoop.io
[ERROR] /home/centos/srcs/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseHadoopCodecs.java:[100,16] package RandomDatum does not exist
[ERROR] /home/centos/srcs/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseHadoopCodecs.java:[100,54] package RandomDatum does not exist
[ERROR] /home/centos/srcs/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseHadoopCodecs.java:[103,7] cannot find symbol
[ERROR]   symbol:   class RandomDatum
[ERROR]   location: class org.apache.hadoop.example.ITUseHadoopCodecs
[ERROR] /home/centos/srcs/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseHadoopCodecs.java:[104,7] cannot find symbol
[ERROR]   symbol:   class RandomDatum
[ERROR]   location: class org.apache.hadoop.example.ITUseHadoopCodecs
{noformat}
Bug: FileUtil#fullyDelete deletes contents of sym-linked directory when symlink cannot be deleted because of local fs fault
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Sep/21 03:04

As discussed in HADOOP-6536, FileUtil#fullyDelete should not delete the contents of the sym-linked directory when we pass a symlink parameter. Currently we try to delete the resource first by calling deleteImpl, and if deleteImpl is failed, we regard it as non-empty directory and remove all its contents and then itself. This logic behaves wrong when local file system cannot delete symlink to a directory because of faulty disk, local system's error, etc. When we cannot delete it in the first time, hadoop will try to remove all the contents of the directory it pointed to and leave an empty dir.
So, we should add an isSymlink checking before we call fullyDeleteContents to prevent such behavior.
Bug: Avoid using implicit dependency on junit-jupiter-api
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/21 05:08
Affects Version/s: 3.3.2

Compilation of branch-3.3 fails due to lack of transitive dependency existing in trunk.
{noformat}
[ERROR] /home/centos/srcs/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/http/TestHttpFileSystem.java:[29,29] package org.junit.jupiter.api does not exist
[ERROR] /home/centos/srcs/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/http/TestHttpFileSystem.java:[47,4] cannot find symbol
  symbol:   class BeforeEach
  location: class org.apache.hadoop.fs.http.TestHttpFileSystem
[INFO] 2 errors
{noformat}
Bug: CredentialProviderFactory.getProviders() recursion loading JCEKS file from s3a
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/21 13:16
Affects Version/s: 3.3.1

{{CredentialProviderFactory.getProviders()}} will be called recursively if the FS being instantiated as a source of a JCECKs file calls {{Configuration.getPassword()}}.
 S3A FileSystem does this when looking at encryption settings, even when authentication
 is not being done by hadoop config options.

Recursion is straightforward to detect, leaving a choice of actions
 # Fail
 # Fix by having getPassword() return null; this implicitly skips use of the credential providers for the nested FS instance.
 # fix by skipping the specific credential provider factory with the problem (warn first?)

My preference is for #3.
Bug: Prometheus metrics only include the last set of labels
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/21 16:02
Affects Version/s: 3.3.1

A prometheus endpoint was added in https://issues.apache.org/jira/browse/HADOOP-16398, but the logic that puts them into a map based on the "key" incorrectly hides any metrics with the same key but different labels. The relevant code is here: [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/PrometheusMetricsSink.java#L55|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/PrometheusMetricsSink.java#L55.]

The labels/tags need to be taken into account, as different tags mean different metrics. For example, I came across this while trying to scrape metrics for all the queues in our scheduler. Only the last queue is included because all the metrics have the same "key" but a different "queue" label/tag.
Bug: Add BuiltInGzipCompressor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/21 03:12

Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is not loaded. So, without Hadoop native codec installed, saving SequenceFile using GzipCodec will throw exception like "SequenceFile doesn't work with GzipCodec without native-hadoop code!"

Same as other codecs which we migrated to using prepared packages (lz4, snappy), it will be better if we support GzipCodec generally without Hadoop native codec installed. Similar to BuiltInGzipDecompressor, we can use Java Deflater to support BuiltInGzipCompressor.
Bug: Upgrade BouncyCastle to 1.69
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 07/Sep/21 16:50
Affects Version/s: 2.10.2

Vulnerabilities reported in BouncyCastle:
[CVE-2020-26939|https://nvd.nist.gov/vuln/detail/CVE-2020-26939] moderate severity
[CVE-2020-15522|https://nvd.nist.gov/vuln/detail/CVE-2020-15522] moderate severity

Affecting releases before 1.66.
 
Upgrade to latest 1.69.
Bug: Unable to use custom SAS tokens for accessing files from ADLS gen2 storage accounts with hierarchical namespace enabled.
Status: Open | Priority: Major | Resolution: 
Created: 30/Aug/21 07:46

I have some parquet files in abfss://con@sa1.dfs.core.windows.net/folder1/. I generated the User Delegation SAS token with the following permission on 'folder1'

SAS_SIGNED_PERMISSIONS -> "racwdxltmeop"

But when I read from "abfss://con@sa1.dfs.core.windows.net/folder1/"

I get a *HTTP 403 error*, I believe this happens when ABFSS driver makes use of `*getACLStatus*` API call to determine whether the storage service has hierarchical namespace enabled or not.

 

I found a workaround, ie to set *fs.azure.account.hns.enabled* to *true* which would skip get ACL API call and as folder level SAS only works for HNS enabled accounts. May I know if this behavior is expected and the workaround I am using is stable for production use and if there are any hidden implications?

 

Thank you in advance.
Bug: [KMS] FileNotFoundException due to inability to access zookeeper znode
Status: Open | Priority: Major | Resolution: 
Created: 27/Aug/21 03:52

A mysterious FileNotFoundException was thrown by KMS client. The KMS server did not log anything related to it. After extensive investigation it was realized it could be caused by either of the two Zookeeper errors:

(1) unable to access the znode /zkdtsm due to Zookeeper ACL or other problems.
(2) if there's a runtime dependency mismatch. In in case, our RangerKMS (where the HadoopKMS runtime is embedded), threw a NoSuchMethodError during curator initialization (the RangerKMS used an incompatible curator-recipes version at runtime). Curator was not initialized successfully and therefore unable to access the znode.

In any case, it would be great to surface up the zookeeper issue to make troubleshooting easier.
Bug: [KMS] Some error messages are not logged in the KMS server
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/21 04:30

While troubleshooting a recent KMS issue, it was found a certain exceptions are not logged in the server side. When client got an error, for example, "token xxx not found in the cache", we wanted to find out more details from the server side (stack trace, exact exception...etc) but it was not found.
Bug: Hadoop NativeAzureFileSystem append removes ownership set on the file
Status: Open | Priority: Major | Resolution: 
Created: 15/Aug/21 11:24
Affects Version/s: 3.3.1

*Repro:* Create Operation sets ownership whereas append operation removes the same.

Create:

*// -rw-r--r-- 1 root supergroup 1 2021-08-15 11:02 /tmp/dummyfile*

Append:

*// -rwxrwxrwx 1 <NULL>  <NULL> 2 2021-08-15 11:04 /tmp/dummyfile*
{code:java}
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsPermission;

public class Wasb {

 private static final short FILE_LOG_PERMISSIONS = 0640;
 
 public static void main(String[] args) throws Exception {
 
    Configuration fsConf = new Configuration();
    fsConf.set("fs.azure.enable.append.support", "true");

Path filePath = new Path("/tmp/dummyfile");

FileSystem fs = FileSystem.newInstance(filePath.toUri(), fsConf);

FSDataOutputStream stream = fs.create(filePath, false);
stream.write(12345);
stream.close();

stream = fs.append(filePath);
stream.write(888);
stream.close();

fs.close();
 }
}
{code}
Bug: Move Ozone to related projects section
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/21 03:37

Hi all, as Ozone was spun to TLP, it has individual web site.

Now on Modules part of Hadoop [website|https://hadoop.apache.org/], the link of Ozone website is old page.

IMHO there are two ways to fix it :
1. update it to new page.
2. move Ozone to Related projects part on Hadoop website

Please feel free to give me some feedback, thanks
Bug: NPE in S3AInputStream read() after failure to reconnect to store
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/21 01:04
Affects Version/s: 3.2.2

when [reading from S3a storage|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L450], SSLException (which extends IOException) happens, which will trigger [onReadFailure|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L458].

onReadFailure calls "reopen". it will first close the original *wrappedStream* and set *wrappedStream = null*, and then it will try to [re-get *wrappedStream*|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L184]. But what if the previous code [obtaining S3Object|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L183] throw exception, then "wrappedStream" will be null.

And the [retry|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L446] mechanism may re-execute the [wrappedStream.read|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L450] and cause NPE.

 

For more details, please refer to [https://github.com/NVIDIA/spark-rapids/issues/2915]
Bug: Upgrade ZooKeeper to 3.4.14 in branch-3.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/21 10:50
Affects Version/s: 3.2.2

Upgrade ZooKeeper 3.4.14 to fix CVE-2019-0201 (https://zookeeper.apache.org/security.html). That way the ZooKeeper version will be consistent with BigTop 3.0.0 (BIGTOP-3471).
Bug: Exclude spotbugs-annotations from transitive dependencies on branch-3.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/21 08:00
Affects Version/s: 3.2.2

Building Hadoop in dist profile with ZooKeeper 3.4.14 fails on hadoop-client-check-test-invariants. Excluding com.github.spotbugs:spotbugs-annotation from transitive dependencies should fix this for users needing zookeeer-3.4.14. Since the dependency is provided/optional on ZooKeeper 3.5.x, branch-3.3 and above are not affected.
Bug: LocalFS to support ability to disable permission get/set; remove need for winutils
Status: Open | Priority: Major | Resolution: 
Created: 06/Aug/21 11:11
Affects Version/s: 3.3.1


Setting FS permissions on windows has always been a paint point for people running spark standalone, creating files through localFS required WINUTILS.EX To do this

I'm seeing some other demand for disabling setting permissions, primarily because if you mount storage (including azure storage) to a VM, you cant call setPermissions; things fail. 

Proposed: 
1. we add an option to disable permission setting
2. we add a path capability, which for hdfs, abfs is always true (not sure about wasb as it varies on file vs dir)
3. we add the option to turn off permission get/set for RawLocalFileSystem and downgrade to a no-op on write, ug+rw on read (or even make configurable?)

(Distcp should maybe probe the path and downgrade -p options if the dest store doesn't do perms properly, which argues for wasb to return true, so file perms are set)
Bug: Remove dependency on jdom
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 29/Jul/21 18:24

It doesn't seem that jdom is referenced anywhere in the code base now, yet it exists in the distribution.

{code}
$ find . -name "*jdom*.jar"
./hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/jdom-1.1.jar
{code}

There is recently [CVE-2021-33813|https://github.com/advisories/GHSA-2363-cqg2-863c] issued for jdom. Let's remove the binary from the dist if not useful.
Bug: Upgrade log4j to fix critical vulnerability
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 02/Aug/21 09:44
Affects Version/s: 3.3.1

CVE-2019-17571 - log4j-1.2.17 - (Fix available in log4j-2.8.2)

Please upgrade to log4j-2.8.2 to fix vulnerability
Bug: WARN security.LdapGroupsMapping: Failed to get groups for user
Status: Open | Priority: Major | Resolution: 
Created: 21/Jul/21 07:12
Affects Version/s: 3.0.0

I run hadoop with ldap

 

hadoop.security.group.mapping=org.apache.hadoop.security.LdapGroupsMapping

hadoop.security.group.mapping.ldap.bind.user=uid=ldapadmin,ou=people,dc=join,dc=com

hadoop.security.group.mapping.ldap.bind.password=00000

hadoop.security.group.mapping.ldap.base=dc=join,dc=com

hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=posixAccount)(uid=\{0}))

hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=posixGroup)

hadoop.security.group.mapping.ldap.search.attr.member=memberUid

hadoop.security.group.mapping.ldap.search.attr.group.name=cn

 

but I run spark,I get a WARN

 

*(LdapGroupsMapping:290)2021-07-13 13:02:45,523 WARN  - [pool-2-thread-4:] ~ Failed to get groups for user jztwk (retry=0) by javax.naming.OperationNotSupportedException: [LDAP: error code 53 - unauthenticated bind (DN with no password) disallowed]*

 

and I *ldapsearch -x -D "uid=ldapadmin,ou=people,dc=join,dc=com" -W -b "dc=join,dc=com"*

can get result

 

 

*hdfs groups yarn*
*yarn : hadoop spark yarn*

 

so How can I fix it

 

CDH 6.3.2
Bug: CLONE - Uber-JIRA: Hadoop should support IPv6
Status: Open | Priority: Major | Resolution: 
Created: 14/Jul/21 04:27

Hadoop currently treats IPv6 as unsupported. Track related smaller issues to support IPv6.

(Current case here is mainly HBase on HDFS, so any suggestions about other test cases/workload are really appreciated.)

Please see [Here | https://issues.apache.org/jira/browse/HADOOP-11890?focusedCommentId=17379845&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17379845] for more details.
Bug: Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/21 08:48

The following parameters should be documented in 
https://hadoop.apache.org/docs/stable/hadoop-kms/index.html#Delegation_Tokens

* hadoop.kms.authentication.zk-dt-secret-manager.enable
* hadoop.kms.authentication.zk-dt-secret-manager.kerberos.keytab
* hadoop.kms.authentication.zk-dt-secret-manager.kerberos.principal
* hadoop.kms.authentication.zk-dt-secret-manager.zkConnectionString
* hadoop.kms.authentication.zk-dt-secret-manager.znodeWorkingPath
* hadoop.kms.authentication.zk-dt-secret-manager.zkAuthType
Bug: S3A Xattr/getXAttr to handle directories without markers
Status: Open | Priority: Minor | Resolution: 
Created: 14/Jul/21 12:13
Affects Version/s: 3.3.1

The XAttr support in S3 raises FNFE if given a directory as a path when that directory has children but no marker.

It should somehow downgrade (e.g. FNFE -> probe for a dir, and if found return an empty set of headers)
Bug: ABFS: Test with 100MB buffer size in ITestAbfsReadWriteAndSeek times out 
Status: Open | Priority: Minor | Resolution: 
Created: 17/Aug/21 15:23
Affects Version/s: 3.3.1

testReadAndWriteWithDifferentBufferSizesAndSeek with buffer size above 100 MB is failing with timeout. It is delaying the whole test run by 15-30 mins. 

[ERROR] testReadAndWriteWithDifferentBufferSizesAndSeek[Size=104,857,600](org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek) Time elapsed: 1,800.041 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 1800000 milliseconds
 at sun.misc.Unsafe.park(Native Method)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
 at java.util.concurrent.FutureTask.get(FutureTask.java:191)
 at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.waitForAppendsToComplete(AbfsOutputStream.java:515)
 at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.flushWrittenBytesToService(AbfsOutputStream.java:533)
 at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.flushInternal(AbfsOutputStream.java:377)
 at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.close(AbfsOutputStream.java:337)
 at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
 at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
 at org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek.testReadWriteAndSeek(ITestAbfsReadWriteAndSeek.java:81)
 at org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek(ITestAbfsReadWriteAndSeek.java:66)