av9ash/GitBugs huggingface dataset:
Bug: ABFS: Enable Footer Read Optimizations with Appropriate Footer Read Buffer Size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Nov/23 15:55
Affects Version/s: 3.3.6

Footer Read Optimization was introduced to Hadoop azure in this Jira: https://issues.apache.org/jira/browse/HADOOP-17347
and was kept disabled by default.
This PR is to enable footer reads by default based on the results of analysis performed as below:

In our scale workload analysis, it was found that workloads working with Parquet (or for that matter OCR etc.) have a lot of footer reads. Footer reads here refers to the read operations done by workload to get the metadata of the parquet file which is required to understand where the actual data resides in the parquet.
This whole process takes place in 3 steps:
 # Workload reads the last 8 bytes of parquet file to get the offset and size of the metadata which is present just above these 8 bytes.
 # Using that offset, workload reads the metadata to get the exact offset and length of data which it wants to read.
 # Workload performs the final read operation to get the data it wants to use for its purpose.

Here the first two steps are metadata reads that can be combined into a single footer read. When workload tries to read certain last few bytes of data (let's say this value is footer size), driver will intelligently read some extra bytes above the footer size to cater to the next read which is going to come.

Q. What is the footer size of file?
A: 16KB. Any read request trying to get the data within last 16KB of the file will qualify for whole footer read. This value is enough to cater to all types of files including parquet, OCR, etc.

Q. What is the buffer size to read when reading the footer?
A. Let's call this footer read buffer size. Prior to this PR footer read buffer size was same as read buffer size (default 4MB). It was found that for most of the workload required footer size was only 256KB. i.e. For almost all parquet files metadata for that file was found to be within last 256KBs. Keeping this in mind it does not make sense to read whole buffer length of 4MB as a part of footer read. Moreover, reading larger data than require incur additional costs in terms of server and network latencies. Based on this and extensive experimentation it was observed that footer read buffer size of 512KB is ideal for almost all the workloads running on parquet, OCR, etc.

Following configuration was introduced to configure the footer read buffer size:
{*}fs.azure.footer.read.request.size{*}: default 512 KB.

*Quantitative Stats:* For a workload running on parquet files the number of read requests got reduced by 2.3M down from 20M. That means around 10% reduction in overall TPS.
Bug: upgrade avro to 1.11.3 due to CVE
Status: Open | Priority: Major | Resolution: 
Created: 25/Oct/23 12:16

[https://nvd.nist.gov/vuln/detail/CVE-2023-39410]

When deserializing untrusted or corrupted data, it is possible for a reader to consume memory beyond the allowed constraints and thus lead to out of memory on the system. This issue affects Java applications using Apache Avro Java SDK up to and including 1.11.2. Users should update to apache-avro version 1.11.3 which addresses this issue.
Bug: Release Hadoop 3.4.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Dec/23 15:52
Affects Version/s: 3.4.0

Confirmed features to be included in the release:

- Enhanced functionality for YARN Federation.
- Redesigned resource allocation in YARN Capacity Scheduler
- Optimization of HDFS RBF.
- Introduction of fine-grained global locks for DataNodes.
- Improvements in the stability of HDFS EC, and more.
- Fixes for important CVEs.

*Issues that need to be addressed in hadoop-3.4.0-RC0 version.*

1.  confirm the JIRA target version/fix version is 3.4.0 to ensure that the version setting is correct.
2. confirm the highlight of hadoop-3.4.0.
3. backport branch-3.4.0/branch-3.4.

{code:java}
  HADOOP-19040. mvn site commands fails due to MetricsSystem And MetricsSystemImpl changes. 
  YARN-11634. [Addendum] Speed-up TestTimelineClient. 
  MAPREDUCE-7468. [Addendum] Fix TestMapReduceChildJVM unit tests.
  
  Revert HDFS-16016. BPServiceActor to provide new thread to handle IBR.
{code}
Bug: S3A region logic to handle vpce and non standard endpoints 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/23 08:16
Affects Version/s: 3.4.0

For non standard endpoints such as VPCE the region parsing added in HADOOP-18908 doesn't work. This is expected as that logic is only meant to be used for standard endpoints. 

If you are using a non-standard endpoint, check if a region is also provided, else fail fast. 

Also update documentation to explain to region and endpoint behaviour with SDK V2.
Bug: S3A: make fs.s3a.create.performance an option you can set for the entire bucket
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/23 18:43
Affects Version/s: 3.3.9

make the fs.s3a.create.performance option something you can set everywhere, rather than just in an openFile() option or under a magic path.

this improves performance on apps like iceberg where filenames are generated with UUIDs in them, so we know there are no overwrites
Bug: Use CRC tables to speed up galoisFieldMultiply in CrcUtil
Status: Open | Priority: Major | Resolution: 
Created: 14/Dec/23 18:14

CrcUtil.galoisFieldMultiply(p, q, m) supports multiplying two polynomials p, q modulo any modulus polynomial m over GF(2). Since the method is used for CRC calculations, the modulus polynomial m is restricted to either the GZIP_POLYNOMIAL or the CASTAGNOLI_POLYNOMIAL. We may use CRC tables in PureJavaCrc32/PureJavaCrc32C to speed up the computation.
Bug: S3A: add option "fs.s3a.optimized.copy.from.local.enabled" to enable/disable CopyFromLocalOperation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Oct/23 10:50
Affects Version/s: 3.3.6

reported failure of CopyFromLocalOperation.getFinalPath() during job submission with s3a declared as cluster fs.

add an emergency option to disable this optimised uploader and revert to the superclass implementation
Bug: S3A to provide full support for S3 Express One Zone
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Nov/23 10:55
Affects Version/s: 3.4.0

HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express one zone support. 

Complete support needs to be added to address tests that fail with s3 express one zone, additional tests, documentation etc. 

* hadoop-common path capability to indicate that treewalking may encounter missing dirs
* use this in treewalking code in shell, mapreduce FileInputFormat etc to not fail during treewalks
* extra path capability for s3express too.
* tests for this
* anything else

A filesystem can now be probed for inconsistent directory listings through {{fs.hasPathCapability(path, "fs.capability.directory.listing.inconsistent")}}

If true, then treewalking code SHOULD NOT report a failure if, when walking into a subdirectory, a list/getFileStatus on that directory raises a FileNotFoundExceptin.
Bug: LD_LIBRARY_PATH is missing HADOOP_COMMON_LIB_NATIVE_DIR
Status: Open | Priority: Major | Resolution: 
Created: 02/Dec/23 09:11
Affects Version/s: 3.2.4

When we run a spark job, we find that it cannot load the native library successfully.

We found a difference between hadoop2 and hadoop3.

hadoop2-Spark-System Properties：
|java.library.path|:/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib|

hadoop3-Spark-System Properties：
|java.library.path|:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib|
The key point is：

hadoop2-hadoop-config.sh：
HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"   <--267
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH     <--268
 
hadoop3-hadoop-functions.sh:
hadoop_add_param HADOOP_OPTS java.library.path \
"-Djava.library.path=${JAVA_LIBRARY_PATH}"
export LD_LIBRARY_PATH        <--1484
 

At the same time, the hadoop3 will clear all non-whitelisted environment variables.
I'm not sure if it was intentional. But it makes our spark job unable to find the native library on hadoop3. 

Maybe we should modify hadoop-functions.sh(1484) and add LD_LIBRARY_PATH to the default configuration item yarn.nodemanager.env-whitelist.
Bug: Filter NaN values from JMX json interface
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/23 09:41
Affects Version/s: 3.4.0

As we can see in this [Yarn documentation|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html] beans can represent Float values as NaN. These values will be represented in the JMX response JSON like:
{noformat}
...
"GuaranteedCapacity": NaN,
...
{noformat}

Based on the [JSON doc|https://www.json.org/] NaN is not a valid JSON token ( however some of the parser libs can handle it ), so not every consumer can parse values like these.

To be able to parse NaN values, a new feature flag should be created.
The new feature will replace the NaN values with 0.0 values.
The feature is default turned off. It can be enabled with the *hadoop.http.jmx.nan-filter.enabled* config.
Bug: AWS SDK v2:  extend support for FIPS endpoints
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/23 18:27
Affects Version/s: 3.4.0

v1 SDK supported FIPS just by changing the endpoint.

Now we have a new builder setting to use.

* add new  fs.s3a.endpoint.fips option
* pass it down
* test
Bug: Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/23 17:09
Affects Version/s: 3.4.0

Getting errors in jobs which can be fixed by increasing this 
2023-12-14 17:35:56,602 [ERROR] [TezChild] |tez.TezProcessor|: java.lang.RuntimeException: java.io.IOException: org.apache.hadoop.net.ConnectTimeoutException: getFileStatus on s3a://aaa/cc-hive-jzv5y6/warehouse/tablespace/managed/hive/student/delete_delta_0000012_0000012_0001/bucket_00001_0: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool	at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
	at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152)
	at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:297)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:84)
	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:70)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:70)
	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:40)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptible
Bug: Zookeeper SSL/TLS support in HDFS ZKFC
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Oct/23 16:36
Affects Version/s: 3.4.0

HADOOP-18709 added support for Zookeeper to communicate with SSL/TLS enabled in hadoop-common. With those changes we have the necessary parameters, that we need to set to enable SSL/TLS in a ZK Client.

In YARN-11468 the SSL communication can be set in Yarn, now we need to similar changes in HDFS to enable it correctly. In HDFS ZK Client is used in the Failover Controller. In this improvement we need to create the ZK client with the necessary SSL configs if we enable it, which we can track under a new HDFS config.
Bug: RPC Metrics : Optimize logic for log slow RPCs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/23 07:53
Affects Version/s: 3.4.0

HADOOP-12325 implement a capability where "slow" RPCs are logged in NN log.
Current processing logic is the "slow" RPCs are to be those whose processing time is outside 3 standard deviation.
However, in practice it is found that many logs of slow rpc are currently output, and sometimes RPCs with a processing time of 1ms are also declared as slow, this is not in line with actual expectations.

Therefore, consider optimize the logic conditions of slow RPC and add a `logSlowRPCThresholdMs` variable to judge whether the current RPCas slow so that the expected slow RPC log can be logger.
for `logSlowRPCThresholdMs`, we can support dynamic refresh to facilitate adjustments based on the actual operating conditions of the hdfs cluster.
Bug: Upgrade to jetty 9.4.53
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/23 19:55
Affects Version/s: 3.4.0

2 CVE fixes in https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.53.v20231009
4 more security fixes in https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.52.v20230823
Bug: Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and ZKSignerSecretProvider
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/23 13:38
Affects Version/s: 3.4.0

HADOOP-18709 added support for Zookeeper to communicate with SSL/TLS enabled in hadoop-common. With those changes we have the necessary parameters, that we need to set to enable SSL/TLS in a ZK Client. That change also did changes in ZKCuratorManager, so with that it is easy to set the SSL/TLS, for Yarn it was done in YARN-11468.

In DelegationTokenAuthenticationFilter currently we are using CuratorFrameworkFactory, it'd be good to change it to use ZKCuratorManager and with that we should support SSL/TLS enablement.

*UPDATE*

So as I investigated this a bit more, it wouldn't be so easy to move to using ZKCuratorManager. 
DelegationTokenAuthenticationFilter uses ZK from two places: in ZKDelegationTokenSecretManager and in ZKSignerSecretProvider. In both places it uses CuratorFrameworkFactory, but the attributes and creation differentiates from ZKCuratorManager. 

In ZKDelegationTokenSecretManager it would be easy to add the new config and based on that create ZK with CuratorFrameworkFactory. But ZKSignerSecretProvider is in hadoop-auth module and with my change it would need hadoop-common, so it would introduce circular dependency between modules 'hadoop-auth' and 'hadoop-common'. I'm still working on a straightforward solution.
Bug: NullPointerException in Hadoop Credential Check CLI Command
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/23 05:38
Affects Version/s: 3.3.0

*Description*: Hadoop's credential check throws {{NullPointerException}} when alias not found.

{code:bash}
hadoop credential check "fs.gs.proxy.username" -provider "jceks://file/usr/lib/hive/conf/hive.jceks" {code}

Checking aliases for CredentialProvider: jceks://file/usr/lib/hive/conf/hive.jceks
Enter alias password: 

java.lang.NullPointerException
at
org.apache.hadoop.security.alias.CredentialShell$CheckCommand.execute(CredentialShell.java:369)
at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:73)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.security.alias.CredentialShell.main(CredentialShell.java:529)}}
Bug: S3ARetryHandler to treat SocketExceptions as connectivity failures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/23 09:46
Affects Version/s: 3.3.6

i've got a v1 sdk stack trace where a TCP connection reset is breaking a large upload. that should be recoverable with retries.


{code}
com.amazonaws.SdkClientException: Unable to execute HTTP request: Connection reset by peer: Unable to execute HTTP request: Connection reset by peer at...

{code}


proposed:
* S3ARetryPolicy to map SocketException to connectivity failure
* See if we can create a test for this, ideally under the aws sdk.

I'm now unsure about how well we handle these io problems...a quick experiment with the 3.3.5 release shows that the retry policy retries on whatever exception chain has an unknown host for the endpoint.
Bug: Upgrade AWS v2 SDK to 2.20.160 and v1 to 1.12.565
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/23 17:57
Affects Version/s: 3.4.0

Bump up the sdk versions for both...even if we don't ship v1 it helps us qualify releases with newer versions, and means that an upgrade of that alone to branch-3.3 will be in sync.
Bug: upgrade netty to 4.1.100 due to CVE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/23 19:26
Affects Version/s: 3.3.6

follow up to https://issues.apache.org/jira/browse/HADOOP-18783

https://netty.io/news/2023/10/10/4-1-100-Final.html

security advisory https://github.com/netty/netty/security/advisories/GHSA-xpw8-rcwv-8f8p

"HTTP/2 Rapid Reset Attack - DDoS vector in the HTTP/2 protocol due RST framesHTTP/2 Rapid Reset Attack - DDoS vector in the HTTP/2 protocol due RST frames
Bug: Upgrade ZooKeeper to 3.7.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/23 01:29
Affects Version/s: 3.3.7

While HADOOP-18613 is proposing to upgrade ZooKeeper to 3.8, it will bring dependency conflicts. Upgrading to ZooKeeper 3.7 could be alternative short-term fix for addressing CVEs.
Bug: Use StandardCharsets.UTF_8 constant
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/23 13:19
Affects Version/s: 3.4.0

* there are some places in the code that have to check for UnsupportedCharsetException when explicitly using the charset name "UTF-8"
* using StandardCharsets.UTF_8 is more efficient because the Java libs usually have to look up the charsets when you provide it as String param instead
* also stop using Guava Charsets and use StandardCharsets
Bug: Possible ConcurrentModificationException if Exec command fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/23 12:32
Affects Version/s: 3.4.0

{{ConcurrentModificationException}} may happen in:

{code:title=https://github.com/apache/hadoop/blob/19b9e6a97b8faa0eb48e7b1855ab37e6d4b6c2a9/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/Exec.java#L64-L82}
  public int run(List<String> command, List<String> output,
      List<String> errors) {
    int retCode = 1;
    ProcessBuilder pb = new ProcessBuilder(command);
    try {
      Process p = pb.start();
      OutputBufferThread stdOut = new OutputBufferThread(p.getInputStream());
      OutputBufferThread stdErr = new OutputBufferThread(p.getErrorStream());
      stdOut.start();
      stdErr.start();
      retCode = p.waitFor();
      if (retCode != 0) {
        mojo.getLog().warn(command + " failed with error code " + retCode);
        for (String s : stdErr.getOutput()) {
          mojo.getLog().debug(s);
        }
      }
      stdOut.join();
      stdErr.join();
{code}

due to accessing {{stdErr.getOutput()}} before {{stdErr}} thread stops.
Bug: S3A: add s3guard command "bucket" to create buckets
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Oct/23 14:20
Affects Version/s: 3.4.0

cloudstore has an mkbucket command
https://github.com/steveloughran/cloudstore/blob/main/src/main/site/mkbucket.md

however, its v1 api, has problems with spans and needs rework for v2. Oh, and it has not tests.

* move the command into hadoop-aws
* add a test or two, 
* add a mapper of  409 to bad request, as "BucketAlreadyOwnedByYouException" comes in as a 409.

Testing will be tricky...as well as not actually wanting to create new buckets, my test a/c doesn't even have the permission to do so.

Proposed tests
# invalid args must be rejected
# trying to create the current bucket must be rejected. This happens even if you lack the permission to create

*No actual attempts to create a new bucket*
Bug: Use builder for prefetch CachingBlockManager
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/23 03:55
Affects Version/s: 3.4.0

Some of the recent changes (HADOOP-18399, HADOOP-18291, HADOOP-18829 etc) have added more params for prefetch CachingBlockManager c'tor to process read/write block requests. They have added too many params and more are likely to be introduced later. We should use builder pattern to pass params.

This would also help consolidating required prefetch params into one single place within S3ACachingInputStream, from scattered locations.
Bug: Move oncrpc/portmap from hadoop-nfs to hadoop-common
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/23 06:40
Affects Version/s: 3.4.0

We want to use udpserver/client for other use cases, rather than only for NFS. One such use case is to export NameNodeHAState for NameNodes via a UDP server.
Bug: Unable to build Hadoop in Windows Container due to missing of devenv
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 22/Dec/23 12:16
Affects Version/s: 3.3.4

For Windows, [Dockerfile|https://github.com/apache/hadoop/blob/77edca8f0a97668722a6d602aa4d08d1fff06172/dev-support/docker/Dockerfile_windows_10] and [build instructions|https://github.com/apache/hadoop/blob/trunk/BUILDING.txt] are provided for building Hadoop. However, when starting to Maven build Hadoop project in the container, it will fail at calling `devenv` to upgrade VS solutions:
!image-2023-12-22-17-12-45-278.png!

This is caused by [win-vs-upgrade.cmd|https://github.com/apache/hadoop/blob/trunk/dev-support/bin/win-vs-upgrade.cmd]. The script checks whether there's `devenv` command, and if there's not, exit with error.
!image-2023-12-22-17-14-49-935.png!

The script is called during building Hadoop Common project, set in win-native profile of the [POM|https://github.com/apache/hadoop/blob/77edca8f0a97668722a6d602aa4d08d1fff06172/hadoop-common-project/hadoop-common/pom.xml#L903C38-L903C38].
!image-2023-12-22-17-18-37-712.png!

But within the container the command is not available, so it will always fail at this step.
!image-2023-12-22-17-20-24-345.png!

If we manually edit the file, removing the check and the call to devenv. The build will still fail, because current sln file within the code repo is based on VS 2010. Because the VS tools installed is 2019(16), the versions do not match.

!image-2023-12-22-20-08-59-918.png!

I'm not sure if someone has successfully built Hadoop using this Dockerfile before, but currently it doesn't seem to be possible to directly build it just following BUILDING.txt without other change.
Bug: Remove keystore parameter requirement from ZookeeperClient 
Status: Open | Priority: Major | Resolution: 
Created: 12/Dec/23 00:37

The keystore parameters are not required for ZooKeeper client to establish TLS connection, so we should remove this limitation from the Hadoop client too. The official ZooKeeper client only emits a warning message in the logs, but otherwise proceed. Hadoop should not add additional constraints.

Check it here: [https://github.com/apache/hadoop/blame/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZookeeperClient.java#L256]