av9ash/GitBugs huggingface dataset:
Bug: ABFS to log @ debug each config option *And where it comes from*
Status: Open | Priority: Minor | Resolution: 
Created: 27/Aug/21 12:05
Affects Version/s: 3.3.1

as with the other stores, there's a lot of options in abfs related to authentication, and when they are misconfigured, nothing works.

cloudstore's storediag can print out the options and their origins, but it's a command line tool, so cannot debug what applications like spark or hive are getting. 

Proposed: as each config option is resolved, print out where it came from, including which jcecks credential provider.
Bug: AbstractAbfsIntegrationTest.createFilesystemForSASTests() leaks FS instances
Status: Open | Priority: Minor | Resolution: 
Created: 09/Sep/21 13:48
Affects Version/s: 3.3.1

I've noticed in MAPREDUCE-7341 that ABFS ITests had a lot of FS instances closed in the shutdown hook. 

Tracked down to the fact that  {{AbstractAbfsIntegrationTest.createFilesystemForSASTests()}} creates a temp FS via newInstance(), and doesn't close it...the instance is kept in the cache until shutdown.

I'm going to fix it in MAPREDUCE-7341, though it's something minor which could be pulled out for backporting
Bug: transient failure of ITestAbfsListStatusRemoteIterator.testWithAbfsIteratorDisabledWithoutHasNext
Status: Open | Priority: Minor | Resolution: 
Created: 13/Jul/21 14:29
Affects Version/s: 3.4.0

assert failure of {{ITestAbfsListStatusRemoteIterator.estWithAbfsIteratorDisabledWithoutHasNext}}  on a (probably) overloaded parallel test run.

Did not occur standalone. 

Before identifying/fixing causes, the test case could be improved for more informative error reporting

* report when the fileNames.remove(pathStr) call returned false: the listing had found an  unexpected value
* tune the assert which did fail to include the unknown element.

This may just be a bug from duplicate filenames, in which case the patch in progress for that should make it "go away"
Bug: Update commons-lang to 3.12.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Sep/21 21:49
Affects Version/s: 3.3.2

our commons-lang3 dependency is currently 3.7, which is nearly 4 years old. latest right now is 3.12 and there are at least some fixes that would make us more robust on JDKs newer than openjdk8 (e.g. LANG-1384. [release notes indicate 3.9 is the first to support jdk11|https://commons.apache.org/proper/commons-lang/changes-report.html]).
Bug: Make it easier to debug UnknownHostExceptions from NetUtils.connect
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Aug/21 19:30
Affects Version/s: 3.3.2

Most UnknownHostExceptions thrown throughout hadoop include a useful message, either the hostname that was not found or some other descriptor of the problem. The UnknownHostException thrown from NetUtils.connect only includes the [message of the underlying UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. If you take a look at the source for UnresolvedAddressException, [it only has a no-args constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] (java11, but same is true in other versions). So it never has a message, meaning the UnknownHostException message is empty.

We should include the endpoint.toString() in the UnknownHostException thrown by NetUtils.connect
Bug: ipc.Client not setting interrupt flag after catching InterruptedException
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jul/21 12:01
Affects Version/s: 3.3.2

ipc.Client is swallowing InterruptedException at a couple of places:
 # While waiting on all connections to be closed
 # While waiting to retrieve some RPC response

We should at least set the interrupt signal and also log the InterruptedException caught.
Bug: Remove ListenerHandle from Hadoop registry
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Aug/21 17:38
Affects Version/s: 3.4.0

As part of HADOOP-17835 (replacing PathChildrenCache/TreeCache by CuratorCache), realized that although registerPathListener() of CuratorService returns ListenerHandle, it is not used by RegistryDNSServer. We can remove ListenerHandle from hadoop-registry as it is not Public/LP interface.
Bug: Fix command line example in Hadoop Cluster Setup documentation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Sep/21 05:14
Affects Version/s: 3.3.1

About Hadoop cluster setup documentation ([https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html])

The option <cluster_name> is specified in the following example, but HDFS command ignores it.
{noformat}
`[hdfs]$ $HADOOP_HOME/bin/hdfs namenode -format <cluster_name>`
{noformat}
Bug: Improve Magic Committer Performance
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Aug/21 10:41
Affects Version/s: 3.3.1

Magic committer tasks can be slow because every file created with overwrite=false triggers a HEAD (verify there's no file) and a LIST (that there's no dir). And because of delayed manifestations, it may not behave as expected.

ParquetOutputFormat is one example of a library which does this.

we could fix parquet to use overwrite=true, but (a) there may be surprises in other uses (b) it'd still leave the list and (c) do nothing for other formats call

Proposed: createFile() under a magic path to skip all probes for file/dir at end of path

Only a single task attempt Will be writing to that directory and it should know what it is doing. If there is conflicting file names and parts across tasks that won't even get picked up at this point. Oh and none of the committers ever check for this: you'll get the last file manifested (s3a) or renamed (file)

If we skip the checks we will save 2 HTTP requests/file.
Bug: S3A to support user-specified content encoding
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Aug/21 20:05
Affects Version/s: 3.3.1

User-specified object content-encoding (part of the object metadata) is important for allowing compressed files to be processed in the AWS ecosystem. We should allow the user to specify the content encoding of the files being written. 

metadata can not be changed after a file is written without a rewrite.
Bug: abfs & s3a FS instantiate triggers warning about deprecated io.bytes.per.checksum
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 28/Sep/21 13:01
Affects Version/s: 3.4.0

If you don't turn off the deprecation log, you get told off about dfs.bytes-per-checksum
{code}
2021-09-28 15:40:26,551 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
{code}

proposed
* find out where it's used/set
* stop it
Bug: ITestS3ADeleteFilesOneByOne. testBulkRenameAndDelete OOM: Direct buffer memory
Status: Open | Priority: Minor | Resolution: 
Created: 24/Sep/21 16:08

on a test setup with bytebuffer, the parallel zero-byte file create phase OOMed

fs.s3a.fast.upload.buffer = "bytebuffer" [core-site.xml]
fs.s3a.fast.upload.active.blocks = "8" [core-site.xml]
fs.s3a.multipart.size = "32M" [core-site.xml]

Root cause: bytebuffer is being allocated on block creation, so every empty file took up 32MB of off-heap storage only for this to be released unused in close()

If this allocation was postponed until the first write(), then empty files wouldn't need any memory allocation. Do the same on-demand creation for byte arrays and filesystem would also have benefits.

this has implications for  HADOOP-17195, which has abfs using a fork of the buffering code 

changing the code there to be on-demand would be a good incentive for s3a to adopt
Bug: JsonSerDeser to collect IOStatistics
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 19/Aug/21 14:07
Affects Version/s: 3.3.1

json deserializer to build stats on reading costs, which can then be collected too to measure cost of ser/deser and of file IO from the input/output streams, if they provide it.

Allows for committers to report costs better here.
Bug: ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Aug/21 11:36
Affects Version/s: 3.3.1

TextFileBasedIdentityHandler uses an instance of LineIterator whose closeQuietly method has been deprecated, resulting in compiler warnings during yetus runs. Fix by leveraging a try-with-resources block to avoid the explicit call to closeQuietly by the LineIterator instance.

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TextFileBasedIdentityHandler.java:192:18:[deprecation] closeQuietly(LineIterator) in LineIterator has been deprecated
Bug: Document of "DistCp Guide" is wrong
Status: Open | Priority: Minor | Resolution: 
Created: 23/Sep/21 07:14

In [https://hadoop.apache.org/docs/stable/hadoop-distcp/DistCp.html] ,it says
{quote}Now, consider the following copy operation:
 distcp hdfs://nn1:8020/source/first hdfs://nn1:8020/source/second hdfs://nn2:8020/target

With sources/sizes:
 hdfs://nn1:8020/source/first/1 32
 hdfs://nn1:8020/source/first/2 32
 hdfs://nn1:8020/source/second/10 64
 hdfs://nn1:8020/source/second/20 32

And destination/sizes:
 hdfs://nn2:8020/target/1 32
 hdfs://nn2:8020/target/10 32
 hdfs://nn2:8020/target/20 64

Will effect:
 hdfs://nn2:8020/target/1 32
 hdfs://nn2:8020/target/2 32
 hdfs://nn2:8020/target/10 64
 hdfs://nn2:8020/target/20 32
{quote}
However, since neither _{{-update}}_ or _-overwrite_ is used, _*first*_ and _*second*_ directories will be created
Bug: S3A CSE: minor tuning
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Aug/21 10:42
Affects Version/s: 3.4.0

Some minor tuning to the CSE encryption support before backporting to 3.3.x and so shipping this year

* LogExactlyOnce an "please ignore the warning" message to a new log ("org.apache.hadoop.fs.s3a.encryption") which can be set to ERROR if you get bored of the message.

* Extend testing_s3a.md and the SDK upgrade runbook: test CSE always

* change property name of encryption key (maybe: fs.s3a.encryption) and add mapping in S3AFileSystem.addDeprecatedKeys ... docs will need updating too.
Bug: BUILDING.txt should not encourage to activate docs profile on building binary artifacts
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Sep/21 02:55
Affects Version/s: 3.3.1

BUILDING.txt in trunk([https://github.com/apache/hadoop/blob/4d21655d04d1b488fe37e908d75f833c1aa66b01/BUILDING.txt#L350]) explains how to create binary distribution with documentation and shows the following command line examples.
{noformat}
`$ mvn package -Pdist,native,docs -DskipTests -Dtar`
`$ mvn package -Pdist,native,docs,src -DskipTests -Dtar`
{noformat}
But, they output errors as follows. Class duplication causes this problem.

Results:
{noformat}
[INFO] -------< org.apache.hadoop:hadoop-client-check-test-invariants >--------
[INFO] Building Apache Hadoop Client Packaging Invariants for Test 3.4.0-SNAPSHOT [105/112]
[INFO] --------------------------------[ pom ]---------------------------------

...

[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) @ hadoop-client-check-test-invariants ---
[INFO] Adding ignorable dependency: org.apache.hadoop:hadoop-annotations:null
[INFO]   Adding ignore: *
[WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
Duplicate classes found:

  Found in:
    org.apache.hadoop:hadoop-client-minicluster:jar:3.4.0-SNAPSHOT:compile
    org.apache.hadoop:hadoop-client-runtime:jar:3.4.0-SNAPSHOT:compile
  Duplicate classes:
    org/apache/hadoop/shaded/org/apache/xerces/impl/dv/dtd/NMTOKENDatatypeValidator.class
    org/apache/hadoop/shaded/org/apache/xerces/dom/DOMImplementationListImpl.class

...

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
{noformat}
Bug: Maven-eclipse-plugin is no longer needed since Eclipse can import Maven projects by itself.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Sep/21 06:41
Affects Version/s: 3.3.1

BUILDING.txt explains the way to import projects to eclipse. However, when you follow the instruction, you can't import the entire source tree (e.g., hadoop-project directory). In addition to this, maven-eclipse-plugin has already retired. There is a simpler way to import it without maven-eclipse-plugin.

{noformat}
Importing projects to eclipse

When you import the project to eclipse, install hadoop-maven-plugins at first.

  $ cd hadoop-maven-plugins
  $ mvn install

Then, generate eclipse project files.

  $ mvn eclipse:eclipse -DskipTests

At last, import to eclipse by specifying the root directory of the project via
[File] > [Import] > [Existing Projects into Workspace].
{noformat}
Bug: HTTP Filesystem to qualify paths in open()/getFileStatus()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Aug/21 08:06
Affects Version/s: 3.3.1

HTTP filesystem is inconsistent with other filesystem (eg: S3AFilesystem) in that they don't accept Path which is relative to the URL that the filesystem is instantiated with.

 

As in
{code:java}
FileSystem.get("http://www.example.com").open(new Path("/test")){code}
will fail with URI not absolute. But it should infact fetch {{http://www.example.com/test}}
Bug: YarnClient Caching Addresses
Status: Open | Priority: Minor | Resolution: 
Created: 16/Jul/21 20:13

We have noticed that when the YarnClient is initialized and used, it is not very resilient when dns or /etc/hosts is modified in the following scenario:

Take for instance the following (and reproducable) sequence of events that can occur on a service that instantiates and uses YarnClient. 
  - Yarn has rm HA enabled (*yarn.resourcemanager.ha.enabled* is *true*) and there are two rms (rm1 and rm2).
  - *yarn.client.failover-proxy-provider* is set to *org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider*

1)	rm2 is currently the active rm
2)	/etc/hosts (or dns) is missing host information for rm2
3)	A service is started and it initializes the YarnClient at startup.
4)	At some point in time after YarnClient is done initializing, /etc/hosts is updated and contains host information for rm2
5)	Yarn is queried, for instance calling *yarnclient.getApplications()*
6)	All YarnClient attempts to communicate with rm2 fail with UnknownHostExceptions, even though /etc/hosts now contains host information for it.
Bug: DNS Query on DNS Registry got connection timed out at times.
Status: Open | Priority: Minor | Resolution: 
Created: 23/Aug/21 08:45

DNS query on YARN Registry DNS got connection timed out at times.

 

After checking logs on registry dns, we found the cause that size of UDP Response is greater than Preallocated Byte buffer(4096)

 

We added failsafe case by returning input query with TC flag so that client can know they should retried using TCP protocol.

 

 

We found the problem and patched our cluster on 3.1.2 but it seems trunk still has the same problem.

I attached patch based on the trunk.

 

Thanks!
Bug: Update link of PoweredBy wiki page
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Aug/21 09:55

The [PoweredBy wiki page|https://cwiki.apache.org/confluence/display/hadoop/PoweredBy] on [main page|https://hadoop.apache.org/] is not found.

IMHO update it to [here|https://cwiki.apache.org/confluence/display/HADOOP2/PoweredBy]
Bug: ABFS ExponentialRetryPolicy doesn't pick up configuration values
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jul/21 20:46

The ABFS driver uses ExponentialRetryPolicy to handle throttling by the ADLS Gen 2 API. The number of retries can already be configured by setting the property fs.azure.io.retry.max.retries. However, none of the other properties on ExponentialRetryPolicy can be set even though configuration properties have already been defined for them. Allow the additional properties (min/max retry wait, default wait) to be configured.
Bug: S3 CSV read performance with Spark with Hadoop 3.3.1 is slower than older Hadoop
Status: Resolved | Priority: Minor | Resolution: Works for Me
Created: 02/Jul/21 15:53
Affects Version/s: 3.3.1

This is issue is continuation to https://issues.apache.org/jira/browse/HADOOP-17755

The input data reported by Spark(Hadoop 3.3.1) was almost double and read runtime also increased (around 20%) compared to Spark(Hadoop 3.2.0) with same exact amount of resource and same configuration. And this is happening with other jobs as well which was not impacted by read fully error as stated above.

*I was having the same exact issue when I was using the workaround  fs.s3a.readahead.range = 1G with Hadoop 3.2.0*

Below is further details :

 
|Hadoop Version|Actual size of the files(in SQL Tab)|Reported size of the file(In Stages)|Time to complete the Stage|fs.s3a.readahead.range|
|Hadoop 3.2.0|29.3 GiB|29.3 GiB|23 min|64K|
|Hadoop 3.3.1|29.3 GiB|*{color:#ff0000}58.7 GiB{color}*|*{color:#ff0000}27 min{color}*|{color:#172b4d}64K{color}|
|Hadoop 3.2.0|29.3 GiB|*{color:#ff0000}58.7 GiB{color}*|*{color:#ff0000}~27 min{color}*|{color:#172b4d}1G{color}|
 * *Shuffle Write* is same (95.9 GiB) for all the above three cases

I was expecting some improvement(or same as 3.2.0) with Hadoop 3.3.1 with read operations, please suggest how to approach this and resolve this.

I have used the default s3a config along with below and also using EKS cluster
{code:java}
spark.hadoop.fs.s3a.committer.magic.enabled: 'true'
spark.hadoop.fs.s3a.committer.name: magic
spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a: org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
spark.hadoop.fs.s3a.downgrade.syncable.exceptions: "true"{code}
 * I did not use 
{code:java}
spark.hadoop.fs.s3a.experimental.input.fadvise=random{code}

And as already mentioned I have used same Spark, same amount of resources and same config.  Only change is Hadoop 3.2.0 to Hadoop 3.3.1 (Built with Spark using ./dev/make-distribution.sh --name spark-patched --pip -Pkubernetes -Phive -Phive-thriftserver -Dhadoop.version="3.3.1")
Bug: [KMS] LoadBalancingKMSClientProvider should implement toString()
Status: Open | Priority: Minor | Resolution: 
Created: 21/Jul/21 07:31
Affects Version/s: 3.3.1

KMSClientProvider implements toString() but LoadBalancingKMSClientProvider does not inherit from KMSClientProvider, nor does it implement toString(). So running hadoop key command does not show key provider URL:
{noformat}
bin/hadoop key list -metadata
2021-07-21 15:26:43,856 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Listing keys for KeyProvider: org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@394a2528{noformat}
Bug: IOStatistisSource to add new adder methods
Status: Open | Priority: Minor | Resolution: 
Created: 15/Jul/21 17:26
Affects Version/s: 3.3.1

Based on experience of Manifest committer of MAPREDUCE-7341:

* operation to take a key and a count and a mean entry, min, max and count:
addStatsSample(String key, long count)

* operation to add a mean and sample count to a mean value.
Bug: mvn verify fails due to duplicate entry in the shaded jar
Status: Resolved | Priority: Blocker | Resolution: Cannot Reproduce
Created: 24/Oct/20 00:10
Affects Version/s: 3.2.2

Found this when I was chasing a separate shading error with [~smeng].

In trunk:

run mvn verify under hadoop-client-module/
{noformat}

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Client Modules 3.4.0-SNAPSHOT:
[INFO]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  2.607 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [03:16 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:30 min]
[INFO] Apache Hadoop Client Test Minicluster .............. FAILURE [04:44 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  09:34 min
[INFO] Finished at: 2020-10-23T16:38:53-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hadoop-client-minicluster: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hadoop.shaded.com.fasterxml.jackson.core.JsonFactory -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit

 {noformat}

This is reproducible in trunk and branch-3.3. However, not reproducible in branch-3.1.
(branch-3.3 has a different error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hadoop-client-minicluster: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hadoop.shaded.javax.ws.rs.ext.MessageBodyReader -> [Help 1])
Bug: S3A NetworkBinding has a runtime class dependency on a third-party shaded class
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 30/Oct/20 20:44
Affects Version/s: 3.3.0

The hadoop-aws library has a dependency on 'com.amazonaws':aws-java-sdk-bundle' which in turn is a fat jar of all AWS SDK libraries and shaded dependencies.

 

This dependency is 181MB.

 

Some applications using the S3AFilesystem may be sensitive to having a large footprint. For example, building an application using Parquet and bundled with Docker.

 

Typically, in prior Hadoop versions, the bundle was replaced by the specific AWS SDK dependencies, dropping the overall footprint.

 

In 3.3 (and maybe prior versions) this strategy does not work because of the following exception: 

{{java.lang.NoClassDefFoundError: com/amazonaws/thirdparty/apache/http/conn/socket/ConnectionSocketFactory}}
{{ at org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1335)}}
{{ at org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1290)}}
{{ at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1247)}}
{{ at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:61)}}
{{ at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:644)}}
{{ at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:390)}}
{{ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)}}
{{ at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)}}
{{ at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)}}
{{ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)}}
{{ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)}}