av9ash/GitBugs huggingface dataset:
Bug: Disable JIRA plugin for YETUS on Hadoop
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 05/Nov/21 16:32
Affects Version/s: 2.10.2

I’ve been noticing an issue with Jenkins CI where a file jira-json goes missing all of a sudden – jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)
{code}
[2021-10-27T17:52:58.787Z] Processing: https://github.com/apache/hadoop/pull/3588
[2021-10-27T17:52:58.787Z] GITHUB PR #3588 is being downloaded from
[2021-10-27T17:52:58.787Z] https://api.github.com/repos/apache/hadoop/pulls/3588
[2021-10-27T17:52:58.787Z] JSON data at Wed Oct 27 17:52:55 UTC 2021
[2021-10-27T17:52:58.787Z] Patch data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:58.787Z] Diff data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:59.814Z] awk: cannot open /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3588/centos-7/out/jira-json (No such file or directory)
[2021-10-27T17:52:59.814Z] ERROR: https://github.com/apache/hadoop/pull/3588 issue status is not matched with "Patch Available".
[2021-10-27T17:52:59.814Z]
{code}
This causes the pipeline run to fail. I’ve seen this in my multiple attempts to re-run the CI on my PR –
 # After 45 minutes – [jenkins / hadoop-multibranch / PR-3588 / #1 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/1/pipeline/]
 # After 1 minute – [jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/2/pipeline/]
 # After 17 minutes – [jenkins / hadoop-multibranch / PR-3588 / #3 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/3/pipeline/]

The hadoop-multibranch pipeline doesn't use ASF JIRA, thus, we're disabling the *jira* plugin to fix this issue.
Bug: Disable JIRA plugin for YETUS on Hadoop
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 05/Nov/21 18:22
Affects Version/s: 3.2.3

I’ve been noticing an issue with Jenkins CI where a file jira-json goes missing all of a sudden – jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)
{code}
[2021-10-27T17:52:58.787Z] Processing: https://github.com/apache/hadoop/pull/3588
[2021-10-27T17:52:58.787Z] GITHUB PR #3588 is being downloaded from
[2021-10-27T17:52:58.787Z] https://api.github.com/repos/apache/hadoop/pulls/3588
[2021-10-27T17:52:58.787Z] JSON data at Wed Oct 27 17:52:55 UTC 2021
[2021-10-27T17:52:58.787Z] Patch data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:58.787Z] Diff data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:59.814Z] awk: cannot open /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3588/centos-7/out/jira-json (No such file or directory)
[2021-10-27T17:52:59.814Z] ERROR: https://github.com/apache/hadoop/pull/3588 issue status is not matched with "Patch Available".
[2021-10-27T17:52:59.814Z]
{code}
This causes the pipeline run to fail. I’ve seen this in my multiple attempts to re-run the CI on my PR –
 # After 45 minutes – [jenkins / hadoop-multibranch / PR-3588 / #1 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/1/pipeline/]
 # After 1 minute – [jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/2/pipeline/]
 # After 17 minutes – [jenkins / hadoop-multibranch / PR-3588 / #3 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/3/pipeline/]

The hadoop-multibranch pipeline doesn't use ASF JIRA, thus, we're disabling the *jira* plugin to fix this issue.
Bug: Disable JIRA plugin for YETUS on Hadoop
Status: Resolved | Priority: Critical | Resolution: Abandoned
Created: 02/Nov/21 17:46
Affects Version/s: 3.3.1

I’ve been noticing an issue with Jenkins CI where a file jira-json goes missing all of a sudden – jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)
{code}
[2021-10-27T17:52:58.787Z] Processing: https://github.com/apache/hadoop/pull/3588
[2021-10-27T17:52:58.787Z] GITHUB PR #3588 is being downloaded from
[2021-10-27T17:52:58.787Z] https://api.github.com/repos/apache/hadoop/pulls/3588
[2021-10-27T17:52:58.787Z] JSON data at Wed Oct 27 17:52:55 UTC 2021
[2021-10-27T17:52:58.787Z] Patch data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:58.787Z] Diff data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:59.814Z] awk: cannot open /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3588/centos-7/out/jira-json (No such file or directory)
[2021-10-27T17:52:59.814Z] ERROR: https://github.com/apache/hadoop/pull/3588 issue status is not matched with "Patch Available".
[2021-10-27T17:52:59.814Z]
{code}
This causes the pipeline run to fail. I’ve seen this in my multiple attempts to re-run the CI on my PR –
 # After 45 minutes – [jenkins / hadoop-multibranch / PR-3588 / #1 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/1/pipeline/]
 # After 1 minute – [jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/2/pipeline/]
 # After 17 minutes – [jenkins / hadoop-multibranch / PR-3588 / #3 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/3/pipeline/]

The hadoop-multibranch pipeline doesn't use ASF JIRA, thus, we're disabling the *jira* plugin to fix this issue.
Bug: Disable JIRA plugin for YETUS on Hadoop
Status: Resolved | Priority: Critical | Resolution: Abandoned
Created: 02/Nov/21 17:57
Affects Version/s: 3.3.0

I’ve been noticing an issue with Jenkins CI where a file jira-json goes missing all of a sudden – jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)
{code}
[2021-10-27T17:52:58.787Z] Processing: https://github.com/apache/hadoop/pull/3588
[2021-10-27T17:52:58.787Z] GITHUB PR #3588 is being downloaded from
[2021-10-27T17:52:58.787Z] https://api.github.com/repos/apache/hadoop/pulls/3588
[2021-10-27T17:52:58.787Z] JSON data at Wed Oct 27 17:52:55 UTC 2021
[2021-10-27T17:52:58.787Z] Patch data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:58.787Z] Diff data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:59.814Z] awk: cannot open /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3588/centos-7/out/jira-json (No such file or directory)
[2021-10-27T17:52:59.814Z] ERROR: https://github.com/apache/hadoop/pull/3588 issue status is not matched with "Patch Available".
[2021-10-27T17:52:59.814Z]
{code}
This causes the pipeline run to fail. I’ve seen this in my multiple attempts to re-run the CI on my PR –
 # After 45 minutes – [jenkins / hadoop-multibranch / PR-3588 / #1 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/1/pipeline/]
 # After 1 minute – [jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/2/pipeline/]
 # After 17 minutes – [jenkins / hadoop-multibranch / PR-3588 / #3 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/3/pipeline/]

The hadoop-multibranch pipeline doesn't use ASF JIRA, thus, we're disabling the *jira* plugin to fix this issue.
Bug: Disable JIRA plugin for YETUS on Hadoop
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 29/Oct/21 13:57
Affects Version/s: 3.4.0

I’ve been noticing an issue with Jenkins CI where a file jira-json goes missing all of a sudden – jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)
{code}
[2021-10-27T17:52:58.787Z] Processing: https://github.com/apache/hadoop/pull/3588
[2021-10-27T17:52:58.787Z] GITHUB PR #3588 is being downloaded from
[2021-10-27T17:52:58.787Z] https://api.github.com/repos/apache/hadoop/pulls/3588
[2021-10-27T17:52:58.787Z] JSON data at Wed Oct 27 17:52:55 UTC 2021
[2021-10-27T17:52:58.787Z] Patch data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:58.787Z] Diff data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:59.814Z] awk: cannot open /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3588/centos-7/out/jira-json (No such file or directory)
[2021-10-27T17:52:59.814Z] ERROR: https://github.com/apache/hadoop/pull/3588 issue status is not matched with "Patch Available".
[2021-10-27T17:52:59.814Z]
{code}
This causes the pipeline run to fail. I’ve seen this in my multiple attempts to re-run the CI on my PR –
 # After 45 minutes – [jenkins / hadoop-multibranch / PR-3588 / #1 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/1/pipeline/]
 # After 1 minute – [jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/2/pipeline/]
 # After 17 minutes – [jenkins / hadoop-multibranch / PR-3588 / #3 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/3/pipeline/]

The hadoop-multibranch pipeline doesn't use ASF JIRA, thus, we're disabling the *jira* plugin to fix this issue.
Bug: ABFS: Enable config control for default connection timeout 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/21 07:55
Affects Version/s: 3.3.1

ABFS driver has a default connection timeout and read timeout value of 30 secs. For jobs that are time sensitive, preference would be quick failure and have shorter HTTP connection and read timeout. 

This Jira is created enable config control over the default connection and read timeout. 

New config name:

fs.azure.http.connection.timeout

fs.azure.http.read.timeout
Bug: ABFS: add cloud trash policy with per-schema policy selection
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Nov/21 14:37
Affects Version/s: 3.3.2

Add a custom TrashPolicy for azure which

* is patched in to the config automatically if "fs.trash.classname" is unset
* checks the source for existing at all before trying anything
* considers itself enabled even if the trash interval is 0
* if interval is 0, just does a delete (issue: what about 90s timeouts)
* downgrades exceptions/failures during rename. If anything is raised and the source dir isn't there, all is good.
Bug: ABFS:Support for secondary accounts on ABFS Driver for SharedKey Auth
Status: Open | Priority: Major | Resolution: 
Created: 02/Dec/21 10:08
Affects Version/s: 3.3.2

Support for secondary accounts on ABFS Driver for SharedKey Auth.

PR Link :- [Hadoop-18032: ABFS: Support for secondary accounts on ABFS Driver for SharedKey Auth by anmolanmol1234 · Pull Request #3901 · apache/hadoop (github.com)|https://github.com/apache/hadoop/pull/3901]
Bug: ABFS: Enable config controlled ETag check for Rename idempotency
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/21 17:39
Affects Version/s: 3.3.2

To support recovery of comms failure during rename, the abfs client fetches the etag of the source file, and when recovering from a failure uses this tag to determine whether the rename succeeded *before the failure happened*

The relevant configuration option is {{fs.azure.enable.rename.resilience}}; default value is: {{true}}

# This works for files, but not directories
# this adds the overhead of a HEAD request before each rename.
# the option can be disabled by setting "fs.azure.enable.rename.resilience" to false

Note: the manifest committer collects etags during task commit and supplies them to the abfs client for the rename, which avoids the need for a HEAD call.
Bug: Bump netty to the latest 4.1.68
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/21 10:00
Affects Version/s: 3.2.3

Netty 4.1.68 fixes the following vulnerabilities.
 * Bzip2Decoder doesn't allow setting size restrictions for decompressed data (#CVE-2021-37136)
 * SnappyFrameDecoder doesn't restrict chunk length any may buffer skippable chunks in an unnecessary way (#CVE-2021-37137)

For more details: [https://netty.io/news/2021/09/09/4-1-68-Final.html]
Bug: Fix typo: validateEncrytionSecrets -> validateEncryptionSecrets
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Dec/21 13:49
Affects Version/s: 3.4.0

Fixing typo validateEncrytionSecrets to validateEncryptionSecrets in 
{code:java}
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractTestS3AEncryption.java{code}
Bug: Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/21 11:29
Affects Version/s: 3.4.0

As [HADOOP-16947|https://issues.apache.org/jira/browse/HADOOP-16947] problem with description, 
Stale SumAndCount also should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson. 
Ensure the DataNode JMX get SendPacketDownstreamAvgInfo Metrics is accurate
Bug: No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/21 06:38
Affects Version/s: 2.10.2

Currently setVerifyChecksum and setWriteChecksum initializes all target file systems which causes delay in hadoop shell copy commands such as get, put, copyFromLocal etc.

This also eventually causes OOM.
Bug: Hadoop - Upgrade to JQuery 3.6.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/21 03:27
Affects Version/s: 3.3.4

jQuery 3.6.0 has been released few months ago - [http://blog.jquery.com/2021/03/02/jquery-3-6-0-released/  |http://blog.jquery.com/2021/03/02/jquery-3-6-0-released/,]

We can upgrade jquery-3.5.1.min.js to jquery-3.6.0.min.js in hadoop project.
Bug: DistCp: Filter duplicates in the source paths
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Dec/21 08:58
Affects Version/s: 3.3.2

Add a basic filtering to remove the exact duplicate paths exposed for copying.

In case two same srcPath say /tmp/file1 is passed in the list twice. DistCp fails with DuplicateFileException, post building the listing.

Would be better if we do a basic filtering of duplicate paths.
Bug: Async Profiler endpoint for Hadoop daemons
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/21 15:10
Affects Version/s: 3.4.0

Async profiler ([https://github.com/jvm-profiling-tools/async-profiler]) is a low overhead sampling profiler for Java that does not suffer from Safepoint bias problem. It features HotSpot-specific APIs to collect stack traces and to track memory allocations. The profiler works with OpenJDK, Oracle JDK and other Java runtimes based on the HotSpot JVM.

Async profiler can also profile heap allocations, lock contention, and HW performance counters in addition to CPU.

We have an httpserver based servlet stack hence we can use HIVE-20202 as an implementation template to provide async profiler as servlet for Hadoop daemons. Ideally we achieve these requirements:
 * Retrieve flamegraph SVG generated from latest profile trace.
 * Online enable and disable of profiling activity. (async-profiler does not do instrumentation based profiling so this should not cause the code gen related perf problems of that other approach and can be safely toggled on and off while under production load.)
 * CPU profiling.
 * ALLOCATION profiling.
Bug: Replace all default Charset usage with UTF-8
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/21 12:37
Affects Version/s: 3.3.5

As discussed on PR#3515, creating this sub-task to replace all default charset with UTF-8 as default charset has some potential problems (e.g. HADOOP-11379, HADOOP-11389).

FYI [~aajisaka]
Bug: Use maven.test.failure.ignore instead of ignoreTestFailure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/21 00:43
Affects Version/s: 2.10.2

In HADOOP-16596, "ignoreTestFailure" variable was introduced to ignore unit test failure, however, Maven property "maven.test.failure.ignore" can be used instead and it can simplify the pom.xml.
Bug: Allow get command to run with multi threads.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/21 03:22
Affects Version/s: 3.3.1

CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and HADOOP-14698， and make put dirs or multiple files faster.
So, It's necessary to enable get and copyToLocal command run with multi-thread.
Bug: Exclude IBM Java security classes from being shaded/relocated
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/21 19:39
Affects Version/s: 3.3.1

IBM Java classes are shaded in Hadoop libraries, e.g. hadoop-client-api. When loaded by Spark, UserGroupInformation has exception:
{noformat}
org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: unable to find LoginModule class: org.apache.hadoop.shaded.com.ibm.security.auth.module.JAASLoginModule
	at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1986)
	at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:719)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:579)
	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2609)
	at org.apache.spark.util.Utils$$$Lambda$1388/0x00000000338e9c30.apply(Unknown Source)
	at scala.Option.getOrElse(Option.scala:189)
{noformat}
When I manually compile UserGroupInformation.java without maven (aka relocation) and inject the class files into hadoop-client-api jar; it works.
Bug: Migrate checkstyle module illegalimport to maven enforcer banned-illegal-imports
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/21 19:53
Affects Version/s: 3.4.0

As discussed on PR [3503|https://github.com/apache/hadoop/pull/3503], we should migrate existing imports provided in IllegalImport tag in checkstyle.xml to maven-enforcer-plugin's banned-illegal-imports enforcer rule so that build never succeeds in the presence of any of the illegal imports.
Bug: abfs rename idempotency broken -remove recovery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/21 15:09
Affects Version/s: 3.3.2

rename idempotency logic of HADOOP-17105   is broken as modtimes aren't uodated on rename.

remove, with the changes from the PR of HADOOP-17981.
also fix delete recovery test after HADOOP-17934
Bug: maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Nov/21 05:18
Affects Version/s: 3.4.0

When we specify any maven plugin with execution tag in the parent as well as child modules, child module plugin overrides parent plugin. For instance, when {{banned-illegal-imports}} is applied for any child module with only one banned import (let’s say {{{}Preconditions{}}}), then only that banned import is covered by that child module and all imports defined in parent module (e.g Sets, Lists etc) are overridden and they are no longer applied.
After this [commit|https://github.com/apache/hadoop/commit/62c86eaa0e539a4307ca794e0fcd502a77ebceb8], hadoop-hdfs module will not complain about {{Sets}} even if i import it from guava banned imports but on the other hand, hadoop-yarn module doesn’t have any child level {{banned-illegal-imports}} defined so yarn modules will fail if {{Sets}} guava import is used.
So going forward, it would be good to replace guava imports with Hadoop’s own imports module-by-module and only at the end, we should add new entry to parent pom {{banned-illegal-imports}} list.
Bug: Add restrict-imports-enforcer-rule for Guava Preconditions in hadoop-main pom
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/21 11:41
Affects Version/s: 3.4.0

Add restrict-imports-enforcer-rule for Guava Preconditions in hadoop-main pom to restrict any new import in future. Remove any remaining usages of Guava Preconditions from the codebase.
Bug: Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/21 14:50
Affects Version/s: 3.4.0

As mentioned on the parent Jira, we can't upgrade hbase2 profile version beyond 2.2.4 until we either have hbase 2 artifacts available that are built with hadoop 3 profile by default or hbase 3 is rolled out (hbase 3 is compatible with hadoop 3 versions only).

Let's upgrade hbase2 profile version to 2.2.4 as part of this Jira and also fix TestTimelineWriterHBaseDown to create connection only after mini cluster is up.
Bug: High performance S3A input stream with prefetching & caching
Status: Open | Priority: Major | Resolution: 
Created: 29/Nov/21 16:12

I work for Pinterest. I developed a technique for vastly improving read throughput when reading from the S3 file system. It not only helps the sequential read case (like reading a SequenceFile) but also significantly improves read throughput of a random access case (like reading Parquet). This technique has been very useful in significantly improving efficiency of the data processing jobs at Pinterest. 
 
I would like to contribute that feature to Apache Hadoop. More details on this technique are available in this blog I wrote recently:
[https://medium.com/pinterest-engineering/improving-efficiency-and-reducing-runtime-using-s3-read-optimization-b31da4b60fa0]
Bug: Upgrade fasterxml Jackson to 2.13.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/21 14:54

Spark 3.2.0 depends on Jackson 2.12.3. Let's upgrade to 2.12.5 (2.12.x latest as of now) or upper.

h2. this has been reverted.

we had to revert this as it broke tez.
Bug: Unable to load AWS credentials from any provider in the chain
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 21/Dec/21 19:29
Affects Version/s: 3.3.1

Hello everybody, please help with this issue. I have a job running with spark over kubernetes (AWS EKS) and I get this error:
{code:java}
py4j.protocol.Py4JJavaError: An error occurred while calling o83.csv.
: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
    at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
    at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
    at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) {code}
Bug: Interface EtagSource to allow FileStatus subclasses to provide etags
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/21 16:04
Affects Version/s: 3.3.1

Various objects stores provide etags in their FileStatus implementations

Make these values accessible
* new interface {{EtagFromFileStatus}} to be implemented when provided
* filesystem.md to declare requirements of etags (constant between LIST and HEAD)...
* path capabilities for (a) etag and (b) etags consistent across rename

Add implementation for abfs, later s3a (and google gcs)

This is initially to handle recovery from certain failures in job commit against abfs, but it would allow a cloud-ready version of distcp to track etags of uploaded files, so diff properly.