av9ash/GitBugs huggingface dataset:
Bug: Update org.ehcache from 3.3.1 to 3.8.2.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/24 14:34
Affects Version/s: 3.4.1

We need to enhance the caching functionality in Yarn Federation by adding a limit on the number of cached entries. I noticed that the version of org.ehcache is relatively old and requires an upgrade.
Bug: move ssh/sftp code out of hadoop-common into a dedicated jar
Status: Open | Priority: Major | Resolution: 
Created: 16/Feb/24 15:49
Affects Version/s: 3.3.6

We could call it hadoop-ssh-common. This code is only used in 1 or 2 other places and it means that hadoop-common (which is used in a lot of places) leaks dependencies on ssh-core and jsch jars to many places.

See [~steve_l] comments in HADOOP-19076
Bug: Exclude some files from Apache RAT check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/24 10:15
Affects Version/s: 3.5.0

The following files cause the Apache RAT check to fail on Windows and must be excluded from the RAT check -
 # src/main/winutils/winutils.sln - Visual Studio solution file

 # src/test/resources/lz4/sequencefile - Binary file
Bug: Update Protocol Buffers installation to 3.23.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/24 16:21
Affects Version/s: 3.3.9

We are seeing issues with Java 8 usage of protobuf-java

See https://issues.apache.org/jira/browse/HADOOP-18197 and comments about

java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
Bug: Support InMemory Tracking Of S3A Magic Commits
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jan/24 10:18

The following are the operations which happens within a Task when it uses S3A Magic Committer. 

*During closing of stream*

1. A 0-byte file with a same name of the original file is uploaded to S3 using PUT operation. Refer [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152] for more information. This is done so that the downstream application like Spark could get the size of the file which is being written.

2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176] for more information.

*During TaskCommit*

1. All the MPU metadata which the task wrote to S3 (There will be 'x' number of metadata file in S3 if a single task writes to 'x' files) are read and rewritten to S3 as a single metadata file. Refer [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201] for more information


Since these operations happens with the Task JVM, We could optimize as well as save cost by storing these information in memory when Task memory usage is not a constraint. Hence the proposal here is to introduce a new MagicCommit Tracker called "InMemoryMagicCommitTracker" which will store the 

1. Metadata of MPU in memory till the Task is committed
2. Store the size of the file which can be used by the downstream application to get the file size before it is committed/visible to the output path.

This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call given a Task writes only 1 file.
Bug: Do not run unit tests on Windows pre-commit CI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/24 10:31
Affects Version/s: 3.5.0

We currently have a good number of unit tests failing on Windows. The author of a patch won't be able to determine if it really was his/her PR that caused the failure or if it was failing historically.

Thus, we need to skip running the unit tests on Windows until we have fixed the ones that are failing currently.

This helps us to proceed with enabling the pre-commit that watches out for any regression against building Hadoop on Windows.

Please refer to this thread for more context - https://github.com/apache/hadoop/pull/5820#issuecomment-1871957975.
Bug: Hadoop use Shell command to get the count of the hard link which takes a lot of time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/24 09:27
Affects Version/s: 3.4.0

Using Hadoop 3.3.4

 

When the QPS of `append` executions is very high, at a rate of above 10000/s. 

 

We found that the write speed in hadoop is very slow. We traced some datanodes' log and find that there is a warning :
{code:java}
2024-01-26 11:09:44,292 WARN impl.FsDatasetImpl (InstrumentedLock.java:logwaitWarning(165)) Waited above threshold(300 ms) to acquire lock: lock identifier: FsDatasetRwlock waitTimeMs=336 ms.Suppressed 0 lock wait warnings.Longest supressed waitTimeMs=0.The stack trace is
java.lang.Thread,getStackTrace(Thread.java:1559)
org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1060)
org.apache.hadoop.util.Instrumentedlock.logWaitWarning(InstrumentedLock.java:171)
org.apache.hadoop.util.InstrumentedLock.check(InstrumentedLock.java:222)
org.apache.hadoop.util.InstrumentedLock.lock(InstrumentedLock, iaya:105)
org.apache.hadoop.util.AutocloseableLock.acquire(AutocloseableLock.java:67)
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:1239)
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:230)
org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver (DataXceiver.java:1313)
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock (DataXceiver.java:764)
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:176)
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:110)
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:293)
java.lang.Thread.run(Thread.java:748)
{code}
 

Then we traced the method _org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl. java:1239),_ and print how long each command take to finish the execution, and find that it takes us 700ms to get the linkCount of the file which is really slow.

!debuglog.png!

 

We traced the code and  find that java1.8 use a Shell Command to get the linkCount, in which execution it will start a new Process and wait for the Process to fork, when the QPS is very high, it will sometimes take a long time to fork the process.

Here is the shell command.
{code:java}
stat -c%h /path/to/file
{code}
 

Solution:

For the FileStore that supports the file attributes "unix", we can use the method _Files.getAttribute(f.toPath(), "unix:nlink")_ to get the linkCount, this method doesn't need to start a new process, and will return the result in a very short time.

 

When we use this method to get the file linkCount, we rarely get the WARN log above when the QPS of append execution is high.

.
Bug: KeyShell fails with NPE when KMS throws Exception with null as message
Status: Open | Priority: Major | Resolution: 
Created: 19/Mar/24 10:27
Affects Version/s: 3.3.6

There is an issue in specific Ranger versions (where RANGER-3989 is not fixed) which throws Exception in case of concurrent access to a HashMap with Message {*}null{*}.
{noformat}
java.util.ConcurrentModificationException: null
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1469)
        at java.util.HashMap$EntryIterator.next(HashMap.java:1503)
        at java.util.HashMap$EntryIterator.next(HashMap.java:1501) {noformat}
This manifests in Hadoop's KeyShell as an Exception with message {*}null{*}.

So when
{code:java}
  private String prettifyException(Exception e) {
    return e.getClass().getSimpleName() + ": " +
        e.getLocalizedMessage().split("\n")[0];
  } {code}
tries to print out the Exception the user experiences NPE
{noformat}
Exception in thread "main" java.lang.NullPointerException
	at org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
	at org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
	at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
	at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {noformat}
This is an unwanted behaviour because the user does not have any feedback what and where went wrong.

 

My suggestion is to add *null checking* into the affected *prettifyException* method.

I'll create the Github PR soon.
Bug: Fix Spotbugs warnings in the build
Status: Open | Priority: Major | Resolution: 
Created: 04/Mar/24 07:22

We are getting spotbugs warnings in every PR.

[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/artifact/out/branch-spotbugs-hadoop-common-project-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/artifact/out/branch-spotbugs-hadoop-hdfs-project-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project-warnings.html]


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-root-warnings.html]

 

Source: https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1532/console
Bug: AWS SDK V2 - Enabling FIPS should be allowed with central endpoint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/24 06:08
Affects Version/s: 3.4.1

FIPS support can be enabled by setting "fs.s3a.endpoint.fips". Since the SDK considers overriding endpoint and enabling fips as mutually exclusive, we fail fast if fs.s3a.endpoint is set with fips support (details on HADOOP-18975).

Now, we no longer override SDK endpoint for central endpoint since we enable cross region access (details on HADOOP-19044) but we would still fail fast if endpoint is central and fips is enabled.

Changes proposed:
 * S3A to fail fast only if FIPS is enabled and non-central endpoint is configured.
 * Tests to ensure S3 bucket is accessible with default region us-east-2 with cross region access (expected with central endpoint).
 * Document FIPS support with central endpoint on connecting.html.

h3. Note: there are two patches here on trunk; they've been coalesced into one on branch-3.4.
Bug: S3A: disable checksums when fs.s3a.checksum.validation = false
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/24 17:18
Affects Version/s: 3.4.0

AWS v2 sdk turns on client-side checksum validation; this kills performance

Given we are using TLS to download from AWS s3, there's implicit channel checksumming going on on, that's along with the IPv4 TCP checksumming.

We don't need it, all it does is slow us down.

proposed: disable in DefaultS3ClientFactory

I don't want to add an option to enable it as it only complicates life (yet another config option), but I am open to persuasion
Bug: Add Protobuf Compatibility Notes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/24 10:52
Affects Version/s: 3.4.0

In HADOOP-18197, we upgraded the Protobuf in hadoop-thirdparty to version 3.21.12. This version may have compatibility issues with certain versions of JDK8. We will document this situation in the index.md file of hadoop-3.4.0 and inform users that we will discontinue support for JDK8 in the future.
Bug: move jersey code in hadoop-common jar to a new hadoop-jersey1-common jar
Status: Open | Priority: Major | Resolution: 
Created: 15/Feb/24 12:50

Hadoop's Jersey dependencies are causing us real trouble.

I'm wondering if it would be a good idea to take the jersey and javax.ws code out of hadoop-common and move it into a dedicated hadoop-jersey1-common jar. We could later create a hadoop-jersey2-common. 

hadoop-jersey1-common and hadoop-jersey2-common would have equivalent classes - just depend on different versions of Jersey.

Example code:
* https://github.com/apache/hadoop/blob/12498b35bbb754225b0b2ca90d5ad4f5cf628d56/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java#L1030
* https://github.com/apache/hadoop/blob/12498b35bbb754225b0b2ca90d5ad4f5cf628d56/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java#L89

Hadoop modules that need access to the common jersey code could start with depending on hadoop-jersey1-common but later be refactored to use hadoop-jersey2-common. We could do this on a module by module basis (one a time).

hadoop-common jar would have its jersey and jsr311-api dependencies removed.

Wdyt [~slfan1989], [~steve_l], [~ayushsaxena] ?
Bug: S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/24 17:57
Affects Version/s: 3.4.0



S3AInputStream doesn't seem to recover from Http exceptions raised through HttpClient or through OpenSSL.

* review the recovery code to make sure it is retrying enough, it looks suspiciously like it doesn't
* detect the relevant openssl, shaded httpclient and unshaded httpclient exceptions, map to a standard one and treat as comms error in our retry policy

This is not the same as the load balancer/proxy returning 443/444 which we map to AWSNoResponseException. We can't reuse that as it expects to be created from an {{software.amazon.awssdk.awscore.exception.AwsServiceException}} exception with the relevant fields...changing it could potentially be incompatible.
Bug: S3A: update AWS sdk versions to 2.23.5 and 1.12.599
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/24 18:37
Affects Version/s: 3.4.0

Move up to the most recent versions of the v2 sdk, with a v1 update just to keep some CVE checking happy.


{code}
    <aws-java-sdk.version>1.12.599</aws-java-sdk.version>
    <aws-java-sdk-v2.version>2.23.5</aws-java-sdk-v2.version>

{code}

The v1 SDK is only used for testing...it is not bundled or declared as a dependency of the hadoop-aws module
Bug: HADOOP-19045. S3A: CreateSession Timeout after 10 seconds
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/24 18:30
Affects Version/s: 3.4.0

s3a client timeout settings are getting down to http client, but not sdk timeouts, so you can't have a longer timeout than the default. This surfaces in the inability to tune the timeouts for CreateSession calls even now the latest SDK does pick it up

The default value of {{fs.s3a.connection.request.timeout}} is now 60s; if an S3 store takes longer than this to return then the operation will be reported by the SDK as a timeout.

When cherrypicking there are two patches
* change constants.java and add test for passdown
* remove core-default and test/core-site values of 0 and update docs. Without this (or a config override) the 10s timeout is maintained
Bug: AWS SDK V2 - Update S3A region logic 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/24 11:10
Affects Version/s: 3.4.0

If both fs.s3a.endpoint & fs.s3a.endpoint.region are empty, Spark will set fs.s3a.endpoint to 

s3.amazonaws.com here:

[https://github.com/apache/spark/blob/9a2f39318e3af8b3817dc5e4baf52e548d82063c/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L540] 

 

HADOOP-18908, updated the region logic such that if fs.s3a.endpoint.region is set, or if a region can be parsed from fs.s3a.endpoint (which will happen in this case, region will be US_EAST_1), cross region access is not enabled. This will cause 400 errors if the bucket is not in US_EAST_1. 

 

Proposed: Updated the logic so that if the endpoint is the global s3.amazonaws.com , cross region access is enabled.
Bug: Class loader leak caused by StatisticsDataReferenceCleaner thread
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/24 11:58
Affects Version/s: 3.3.6

The "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" daemon thread was created by FileSystem. 

This is fine if the thread's context class loader is the system class loader, but it's bad if the context class loader is a custom class loader. The reference held by this daemon thread means that the class loader can never become eligible for GC.
Bug: Hadoop 3.4.0 Highlight big features and improvements.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/24 14:26
Affects Version/s: 3.4.0

While preparing for the release of Hadoop-3.4.0, I've noticed the inclusion of numerous commits in this version.  Therefore, highlighting significant features and improvements becomes crucial.  I've completed the initial version and now seek the review of more experienced partner to ensure the finalization of the version's highlights.
Bug: [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/24 05:09
Affects Version/s: thirdparty-1.3.0

setuptools-scm and lazy-object-proxy have been upgraded many times. Directly relying on pip to install dependencies will report errors. We need to specify the installation version.
Bug: Improve create-release RUN script
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/24 04:51
Affects Version/s: thirdparty-1.3.0

Using create-release will create a docker image locally, but three of the RUN scripts may fail to execute.

1. RUN groupadd --non-unique -g 0 root
{code:java}
=> ERROR [16/20] RUN groupadd --non-unique -g 0 root                                                            0.2s
------
 > [16/20] RUN groupadd --non-unique -g 0 root:
0.154 groupadd: group 'root' already exists
------
Dockerfile:100
--------------------
  98 |
  99 |     LABEL org.apache.hadoop.create-release="cr-19697"
 100 | >>> RUN groupadd --non-unique -g 0 root
 101 |     RUN useradd -g 0 -u 0 -m root
 102 |     RUN chown -R root /home/root
{code}
2. RUN useradd -g 0 -u 0 -m root
{code:java}
 > [17/20] RUN useradd -g 0 -u 0 -m root:
0.165 useradd: user 'root' already exists
------
Dockerfile:101
--------------------
  99 |     LABEL org.apache.hadoop.create-release="cr-12068"
 100 |     RUN groupadd --non-unique -g 0 root; exit 0;
 101 | >>> RUN useradd -g 0 -u 0 -m root
 102 |     RUN chown -R root /home/root
 103 |     ENV HOME /home/root
{code}
3. RUN chown -R root /home/root
{code:java}
 > [18/20] RUN chown -R root /home/root:
0.168 chown: cannot access '/home/root': No such file or directory
------
Dockerfile:102
--------------------
 100 |     RUN groupadd --non-unique -g 0 root; exit 0;
 101 |     RUN useradd -g 0 -u 0 -m root; exit 0;
 102 | >>> RUN chown -R root /home/root
 103 |     ENV HOME /home/root
 104 |     RUN mkdir -p /maven
--------------------
{code}
Even if these three scripts fail, subsequent steps can continue to be executed, so I added exit 0 after the script.
Bug: Improve create-release RUN script
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/24 13:04
Affects Version/s: 3.4.0

Using create-release will create a docker image locally, but three of the RUN scripts may fail to execute.

1. RUN groupadd --non-unique -g 0 root


{code:java}
=> ERROR [16/20] RUN groupadd --non-unique -g 0 root                                                            0.2s
------
 > [16/20] RUN groupadd --non-unique -g 0 root:
0.154 groupadd: group 'root' already exists
------
Dockerfile:100
--------------------
  98 |
  99 |     LABEL org.apache.hadoop.create-release="cr-19697"
 100 | >>> RUN groupadd --non-unique -g 0 root
 101 |     RUN useradd -g 0 -u 0 -m root
 102 |     RUN chown -R root /home/root
{code}


2. RUN useradd -g 0 -u 0 -m root

{code:java}
 > [17/20] RUN useradd -g 0 -u 0 -m root:
0.165 useradd: user 'root' already exists
------
Dockerfile:101
--------------------
  99 |     LABEL org.apache.hadoop.create-release="cr-12068"
 100 |     RUN groupadd --non-unique -g 0 root; exit 0;
 101 | >>> RUN useradd -g 0 -u 0 -m root
 102 |     RUN chown -R root /home/root
 103 |     ENV HOME /home/root
{code}


3. RUN chown -R root /home/root


{code:java}
 > [18/20] RUN chown -R root /home/root:
0.168 chown: cannot access '/home/root': No such file or directory
------
Dockerfile:102
--------------------
 100 |     RUN groupadd --non-unique -g 0 root; exit 0;
 101 |     RUN useradd -g 0 -u 0 -m root; exit 0;
 102 | >>> RUN chown -R root /home/root
 103 |     ENV HOME /home/root
 104 |     RUN mkdir -p /maven
--------------------
{code}

Even if these three scripts fail, subsequent steps can continue to be executed, so I added exit 0 after the script.
Bug: Highlight RBF features and improvements targeting version 3.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/24 05:49
Affects Version/s: 3.4.0

I want to hight recent RBF features and improvements.

- Support observer node from Router-Based Federation
- Enhanced IPC throughput between Router and NameNode
- Improved isolation for downstream name nodes.
Bug: Hadoop 3.4.0 Big feature/improvement highlight addendum
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/24 08:09
Affects Version/s: 3.4.0

Capacity Scheduler was redesigned to add new capacity modes, it should be mentioned as part of 3.4.0 YARN improvements. Reference: YARN-10496/YARN-10888/YARN-10889
Bug: core-default fs.s3a.connection.establish.timeout value too low -warning always printed
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Feb/24 10:43
Affects Version/s: 3.4.0

caused by HADOOP-18915.

in core-default we set the value of fs.s3a.connection.establish.timeout to 5s

{code}
<property>
  <name>fs.s3a.connection.establish.timeout</name>
  <value>5s</value>
</property>
{code}

but there is a minimum of 15s, so this prints a warning

{code}
2024-02-29 10:39:27,369 WARN impl.ConfigurationHelper: Option fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000 ms instead
{code}