av9ash/GitBugs huggingface dataset:
Bug: S3A: terasort tests fail with CSE-kMS enabled and london region With Delegation Token Secrets
Status: Open | Priority: Major | Resolution: 
Created: 30/Dec/24 17:23
Affects Version/s: 3.4.2

terasort test runs failing; job logs show the process yarn spawns is trying to use the default region provider list
{code}
 Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain
{code}

these are spawned processes; all login info and encryption secrets are passed in the marshalled delegation token {{AbstractS3ATokenIdentifier}}, with the encryption stuff in {{EncryptionSecrets}}

This is probably going to need to be extended with information about the KMS region, with this info extracted during instantiation.
Bug: JDK 17 - java.lang.NoSuchFieldException: modifiers error in hadoop-hdfs and hadoop-yarn module
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 18/Dec/24 05:20

On Compilation of Hadoop over JDK17 getting *java.lang.NoSuchFieldException: modifiers* in following classes : 


1. TestStoragePolicyPermissionSettings

2. TestFileCreation

3. TestAsyncDispatcher
Bug: AliyunOSS: When using the OSS Connector, special circumstances can lead to deadlocks in IO threads, causing computation tasks to become stuck
Status: Open | Priority: Major | Resolution: 
Created: 18/Dec/24 07:04
Affects Version/s: 3.3.6

When using {{AliyunOSSBlockOutputStream}} to read data, the threads waiting to read data and the threads accessing OSS to fetch data are coordinated using a condition lock. We have observed that under certain special circumstances, such as incorrect environment configurations or misconfigurations of third-party dependencies, the threads responsible for fetching data can abnormally terminate. After termination, these threads do not release the locks they hold, leading to indefinite waiting by other threads. From the perspective of the computation engine, this tasks being stalled (but not failing).

 

!https://aone.alibaba-inc.com/v2/api/workitem/adapter/file/url?fileIdentifier=workitem%2Falibaba%2F1026733%2F1733219963897image.png!

We believe that in such situations, it is best for the computation engine to detect the error and exit accordingly, rather than remain indefinitely stuck.
Bug: S3A Analytics-Accelerator: Update SDK client creation code 
Status: Open | Priority: Major | Resolution: 
Created: 17/Dec/24 01:41

Currently we initialise a new CRT client when analytics-accelerator is enabled. This is done in S3AFileSystem, and is also not configured with S3A configs as used by other clients. 

We should move this creation to S3AStoreImpl, and also use suitable hadoop configs where it makes sense to do so.

Also to use the S3AsynClient, we have temporarily done this.isMultipartUploadEnabled = false, as without this you cannot make ranged GETs. Believe this issue may have been resolved in the later versions of the SDK. We will need to revisit this code.
Bug: S3A Analytics-Accelerator: Add audit header support 
Status: Open | Priority: Major | Resolution: 
Created: 13/Dec/24 11:12

S3A adds audit information as referrer headers, see [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/auditing.md] for documentation on this. 

These are attached using execution interceptors, see ActiveAuditManagerS3A.
createExecutionInterceptors(). 
 
analyitcs-accelerator-s3 makes the GET requests now but does not update the referrer with GETs. 
 
See LoggingAuditor.attachRangeFromRequest() for how this is done. 
 
When using analytics-accelerator-s3, audit headers from S3A should be sent through.
Bug: Disable releases for apache.snapshots repo
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/24 14:21

{code:title=https://github.com/apache/hadoop/blob/aa5fe6f550c8971762c02c292240a7529001e1d8/pom.xml#L54-L59}
  <repositories>
    <repository>
      <id>${distMgmtSnapshotsId}</id>
      <name>${distMgmtSnapshotsName}</name>
      <url>${distMgmtSnapshotsUrl}</url>
    </repository>
{code}

The Apache snapshot repository should be enabled only for snapshots.
Bug: OutofBounds Exception due to assumption about buffer size in BlockCompressorStream
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/24 01:53
Affects Version/s: 3.4.1

h3. What Happened: 

Got an OutofBounds exception when io.compression.codec.snappy.buffersize is set to 7. BlockCompressorStream assumes that the buffer size will always be greater than the compression overhead, and consequently MAX_INPUT_SIZE will always be greater than or equal to 0. 
h3. Buggy Code: 

When io.compression.codec.snappy.buffersize is set to 7, compressionOverhead is 33 and MAX_INPUT_SIZE is -26. 
{code:java}
public BlockCompressorStream(OutputStream out, Compressor compressor, 
                             int bufferSize, int compressionOverhead) {
  super(out, compressor, bufferSize);
  MAX_INPUT_SIZE = bufferSize - compressionOverhead; // -> Assumes bufferSize is always greater than compressionOverhead and MAX_INPUT_SIZE is non-negative. 
} {code}
h3. Stack Trace: 
{code:java}
java.lang.ArrayIndexOutOfBoundsException
        at org.apache.hadoop.io.compress.snappy.SnappyCompressor.setInput(SnappyCompressor.java:86)
        at org.apache.hadoop.io.compress.BlockCompressorStream.write(BlockCompressorStream.java:112) {code}
h3. How to Reproduce: 

(1) Set io.compression.codec.snappy.buffersize to 7

(2) Run test: org.apache.hadoop.io.compress.TestCodec#testSnappyMapFile
Bug: ABFS: [FnsOverBlob] Group Enums in AbfsRestOperationType into categories and Other Code Cleanups
Status: Open | Priority: Major | Resolution: 
Created: 12/Nov/24 11:45
Affects Version/s: 3.4.1

Currently in AbfsRestOperationType, there are a lot of enums added and some of them corresponds to same operation but going to differentazure storage endpoint.

It makes sense to group these Enums into following Categories:
Global enum: AbfsGlobalRestOperationType
dfs enum: AbfsDFSRestOperationType
blob enum: AbfsBlobRestOperationType

Each of these sub-enums can hold operations used for that particular service endpoint.

For more details:
1. [https://github.com/apache/hadoop/pull/6944#discussion_r1766715478]

2. [https://github.com/apache/hadoop/pull/6944#discussion_r1788170640]
Bug: Support append operation in S3A filesystem for S3 Express One Zone storage class
Status: Open | Priority: Major | Resolution: 
Created: 22/Nov/24 05:06

Amazon S3 recently announced support for appending (to the last) to s3 object for s3 express storage class. Refer [1|https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-express-one-zone-append-data-object/] and [2|https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-objects-append.html] for more details.

This requires to upgrade to latest aws-sdk-java-v2 version 2.29.19
Bug: S3A: fs.s3a.connection.expect.continue controls 100 CONTINUE behavior
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Oct/24 13:58
Affects Version/s: 3.4.1

New option
{code}
  fs.s3a.connection.expect.continue
{code}

This controls whether or not an PUT request to the S3 store
sets the "Expect: 100-continue" header and awaits a 100 CONTINUE
response before uploading any data.

This allows for throttling and other problems to be detected fast.

The default is "true" -the header is sent.

It seems like either the AWS SDK or http client libraries underneath don't recognise a dead HTTPS connection when using this.

Leaving as enabled as it can cope with heavy load better (clients can get a 503 and know to back off before uploading any data.
Bug: AzureBlobFileSystem.open() should override readVectored() much more efficiently for small reads
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 23/Nov/24 17:07

In hadoop-azure, there are huge performance problems when reading file in a too fragmented way: by reading many small file fragments even with the readVectored() Hadoop API, resulting in distinct Https Requests (=TCP-IP connection established + TLS handshake + requests).
Internally, at lowest level, haddop azure is using class HttpURLConnection from jdk 1.0,  and the ReadAhead Threads do not sufficiently solve all problems.
The hadoop azure implementation of "readVectored()" should make a compromise between reading extra ignored data wholes, and establishing too many https connections.

Currently, the class AzureBlobFileSystem#open() does return a default inneficient imlpementation of readVectored:

{code:java}
  private FSDataInputStream open(final Path path,
      final Optional<OpenFileParameters> parameters) throws IOException {
...
      InputStream inputStream = getAbfsStore().openFileForRead(qualifiedPath, parameters, statistics, tracingContext);
      return new FSDataInputStream(inputStream);  // <== FSDataInputStream is not efficiently overriding readVectored() !
 }
{code}


see default implementation of FSDataInpustStream.readVectored:
{code:java}
    public void readVectored(List<? extends FileRange> ranges, IntFunction<ByteBuffer> allocate) throws IOException {
        ((PositionedReadable)this.in).readVectored(ranges, allocate);
    }
{code}

it calls the underlying method from class AbfsInputStream, which is not overriden:
{code:java}
    default void readVectored(List<? extends FileRange> ranges, IntFunction<ByteBuffer> allocate) throws IOException {
        VectoredReadUtils.readVectored(this, ranges, allocate);
    }
{code}


AbfsInputStream should override this method, and accept internally to do less Https calls, with merged range, and ignore some returned data (wholes in the range). 

It is like honouring the parameter of hadoop FSDataInputStream (implements PositionedReadable)
{code:java}
  /**
   * What is the smallest reasonable seek?
   * @return the minimum number of bytes
   */
  default int minSeekForVectorReads() {
    return 4 * 1024;
  }
{code}

Even this 4096 value is very conservative, and should be redined by AbfsFileSystem to be 4Mo or even 8mo.

ask chat gpt: "on Azure Storage, what is the speed of getting 8Mo of a page block, compared to the time to establish a https tls handshake ?"
The response (untrusted from chat gpt..) says :
HTTPS/TLS Handshake: ~100–300 ms  ... is generally slower than  downloading 8 MB from Page Blob:  on Standard Tier: ~100–200 ms / on Premium Tier: ~30–50 ms

Azure Abfsclient already setup by default a lot of Threads for Prefecth Read Ahead, to prefetch 4Mo of data,  but it is NOT sufficent, and less efficient that simply implementing correctly what is already in Hadoop API : readVectored(). It also have the drawback of reading tons of useless data (past parquet blocks), that are never used.
Bug: SaslRpcServer.AuthMethod print INFO messages in client side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/24 19:18

After HADOOP-19306, the following INFO messages are printed in client side.  Thanks As [~stevel@apache.org] for reporting it in [this comment|https://github.com/apache/hadoop/pull/7140#issuecomment-2483881941].
{code}
2024-11-18 18:53:37,645 [setup] INFO  security.SaslRpcServer (SaslRpcServer.java:<init>(239)) - AuthMethod SIMPLE: code=80, mechanism=""
2024-11-18 18:53:37,645 [setup] INFO  security.SaslRpcServer (SaslRpcServer.java:<init>(239)) - AuthMethod KERBEROS: code=81, mechanism="GSSAPI"
2024-11-18 18:53:37,646 [setup] INFO  security.SaslRpcServer (SaslRpcServer.java:<init>(239)) - AuthMethod DIGEST: code=82, mechanism="DIGEST-MD5"
2024-11-18 18:53:37,646 [setup] INFO  security.SaslRpcServer (SaslRpcServer.java:<init>(239)) - AuthMethod TOKEN: code=82, mechanism="DIGEST-MD5"
2024-11-18 18:53:37,646 [setup] INFO  security.SaslRpcServer (SaslRpcServer.java:<init>(239)) - AuthMethod PLAIN: code=83, mechanism="PLAIN"
{code}
Bug:  Follow-up on ResourceManager quit due to ApplicationStateData exceeding znode limit
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Oct/24 02:44
Affects Version/s: 3.3.6

In YARN-5006 the issue "ResourceManager quit due to ApplicationStateData exceeding znode limit" has been addressed , specifically in the storeApplicationStateInternal() method of the ZKRMStateStore class. However, the methods updateApplicationStateInternal(), storeApplicationAttemptStateInternal(), and updateApplicationAttemptStateInternal() also need to incorporate checks for the ZNode size limit in  ZKRMStateStore class. This will help prevent scenarios where excessive data written to a single ZNode causes the ResourceManager (RM) and ZooKeeper (ZK) services to become unavailable.
Bug: ABFS: Add a new API in AzureBlobFileSystem to allow listing with startFrom
Status: Open | Priority: Major | Resolution: 
Created: 02/Nov/24 00:49

Currently the API to list from a certain starting point is hidden inside {{{}AzureBlobFileSystemStore{}}}, and is hard to access. It'd be better to surface this up to places like {{AzureBlobFileSystem}} to make it easier to use. 

 

The API is useful in scenarios such as Delta Lake where transaction logs are always indexed sequentially. With this API, the listing in Delta no longer needs to listing the whole {{_delta_log}} directory but only a small subset.
Bug: JDK11 build fails with error: release version 1.8 not supported
Status: Resolved | Priority: Major | Resolution: Workaround
Created: 25/Oct/24 07:54
Affects Version/s: 3.5.0

[JDK11 build|https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/756/] is failing:

{code:title=https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/756/artifact/out/patch-compile-root.txt}
Fatal error compiling: error: release version 1.8 not supported
{code}
Bug: ABFS: setConf should also update configuration for AzureBlobFileSystemStore
Status: Open | Priority: Major | Resolution: 
Created: 07/Nov/24 02:11

Currently the {{AzureBlobFileSystemStore}} inside {{AzureBlobFileSystem}} keeps a separate reference to the input {{{}configuration{}}}. The {{AzureBlobFileSystem.setConf}} only update the conf in itself but not the one in {{AzureBlobFileSystemStore}}
Bug: hadoop-rumen is vulnerable to Sonatype CWE611
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 05/Nov/24 10:23
Affects Version/s: 3.3.6

hadoop-rumen is vulnerable to CWE-611: [Improper Restriction of XML External Entity Reference.|https://cwe.mitre.org/data/definitions/611.html]

sonatype-2022-5820

Explanation: The Apache {{hadoop-common}} and {{hadoop-rumen}} packages are vulnerable to XML External Entity (XXE) attacks. The {{readXmlFileToMapWithFileInputStream()}} method in the {{HostsFileReader}} class, the {{parse()}} method in the {{JobConfigurationParser}} class, and the constructor in the {{ParsedConfigFile}} class process malicious external entities by default due to an unsafe XML parser configuration. A remote attacker who can supply or modify the contents of hosts or configuration XML files parsed by these packages can exploit this vulnerability to exfiltrate information, cause a Denial of Service (DoS) condition, or perform other XXE-related attacks.

Root Cause: org/apache/hadoop/tools/rumen/JobConfigurationParser.class, org/apache/hadoop/tools/rumen/ParsedConfigFile.class
Bug: Enable multi-part-upload with client side encryption for async s3 client
Status: Open | Priority: Major | Resolution: 
Created: 06/Nov/24 05:36

S3 Client side encryption (cse) is being added as part of https://issues.apache.org/jira/browse/HADOOP-18708.

multi-part-upload with s3 async client is disabled when cse is enabled since it is unavailable. This Jira tracks the effort to enable it one it is available.
Bug: [JDK 17] Resolve Http Server error and Http response error in Hadoop Trunk
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 09/Oct/24 06:18

While compiling HADOOP-TRUNK on JDK 17 faced 2 common issues : 

*1.* Unexpected HTTP response: *code=500 != 200* or *code=500 != 307*

*2.* org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.yarn.webapp.WebAppException: *Error starting http server*
Bug: S3A CopyFromLocalFile operation fails when the source file does not contain file scheme.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/24 12:09
Affects Version/s: 3.3.6

When the sourcePath does not contain any file scheme information, S3A CopyFromLocalFile operation fails with the following exception stack trace.
{code:java}
    at org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.getFinalPath(CopyFromLocalOperation.java:360)
    at org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.uploadSourceFromFS(CopyFromLocalOperation.java:222)
    at org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.execute(CopyFromLocalOperation.java:169)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFromLocalFile$23(S3AFileSystem.java:4217)
    at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
    at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
    at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2871)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2890)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFromLocalFile(S3AFileSystem.java:4209)
 {code}
Additionally the failure is seen only when 
{color:#172b4d}*fs.s3a.optimized.copy.from.local.enabled* is enabled (which is by default). This happens only when the local source file is given without any file scheme for example : /tmp/file.txt instead of file:///tmp/file.txt.
{color}
 
{color:#172b4d}The proposal here is to add file scheme to the source if the source path does not contain the same.{color}
Bug: upgrade AWS v1 SDK on branch-3.3.x
Status: Open | Priority: Major | Resolution: 
Created: 21/Oct/24 13:20
Affects Version/s: 3.3.6

Qualify an SDK upgrade to branch-3.3

must include a fix for HADOOP-19313

This isn't really targeting a 3.3.x release; just keeping up to date and dealing with that noisyness
Bug: Add option to add parent directory of source directories to target directories
Status: Open | Priority: Major | Resolution: 
Created: 11/Oct/24 08:07
Affects Version/s: 3.0.0

Currently, when we execute the Hadoop distcp  with -update -delete src1/* src2/* dest command to keep the source and target directories exactly the same。 When either -update or -overwrite is specified, the *contents* of the source-directories are copied to target, and not the source directories themselves. 

Consider a copy from /source/first/ and /source/second/ to /target/, where the source paths have the following contents:

hdfs://nn1:8020/source/first/1
hdfs://nn1:8020/source/first/2
hdfs://nn1:8020/source/second/10
hdfs://nn1:8020/source/second/20

distcp2 -update hdfs://nn1:8020/source/first hdfs://nn1:8020/source/second hdfs://nn2:8020/target


would yield the following contents in /target:

hdfs://nn2:8020/target/1
hdfs://nn2:8020/target/2
hdfs://nn2:8020/target/10
hdfs://nn2:8020/target/20

 

But, sometimes, we need to preserve parent directories like this:

hdfs://nn1:8020/target/first/1
hdfs://nn1:8020/target/first/2
hdfs://nn1:8020/target/second/10
hdfs://nn1:8020/target/second/20

 

So, should we introduce an option -preserveParentDir to keep the parent directories to be copied with -update or -overwrite ?
Bug: ConcurrentModificationException in HttpReferrerAuditHeader
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Oct/24 17:08
Affects Version/s: 3.4.1

Surfaced during a test run doing vector iO, where multiple parallel GETs were being issued within the same audit span, just when the header is built by enumerating the attributes.

{code}
      queries = attributes.entrySet().stream()
          .filter(e -> !filter.contains(e.getKey()))
          .map(e -> e.getKey() + "=" + e.getValue())
          .collect(Collectors.joining("&"));
{code}

Hypothesis: multiple GET requests are conflicting in updating/reading the header.
Bug: Update rat version in the docker build.sh script
Status: Open | Priority: Major | Resolution: 
Created: 04/Oct/24 17:49
Affects Version/s: 3.3.7

The docker build.sh script uses apache rat 0.15 which is removed from Apache CDN.
https://github.com/apache/hadoop/blob/docker-hadoop-3.4/build.sh#L20

The build in the DockerHub doesn't fail, probably because there's cache. But it fails for me locally.

The latest is 0.16.1. Let's update.
Bug: JVM GC Metrics supports ZGC pause time and count
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Dec/24 04:25
Affects Version/s: 3.4.0

8265136: ZGC: Expose GarbageCollectorMXBeans for both pauses and cycles

[https://bugs.openjdk.org/browse/JDK-8265136]
Bug: [ABFS] Add all ABFS metrics related file in one package
Status: Open | Priority: Minor | Resolution: 
Created: 18/Dec/24 09:15
Affects Version/s: 3.4.1

Currently, all ABFS metrics files are in the service package, which also includes many other files. To improve code readability, we should consolidate all metrics-related files into one common package.
Previous JIRA: [HADOOP-19311] [ABFS] Implement Backoff and Read Footer metrics using IOStatistics Class - ASF JIRA
Bug: ABFS: Optimizations for Retry Handling and Throttling
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Dec/24 05:16
Affects Version/s: 3.4.0

Given the significant improvements on the service side over the past few years, the current client-side retry mechanism, which enforces 5-6 minutes of sleep time with jitter, is now excessive. Therefore, we are realigning the retry and throttling logic thresholds.

Since the service can now handle higher traffic rates, we aim to enhance driver performance by setting the following configurations as default:
 * {*}Client-side throttling (CST){*}: Off
 * *Client Backoff* - 500ms (reduced from 3sec)
 * *Max Backoff* - 25s (reduced from 30sec)
 * *Min Backoff* - 500ms (reduced from 3sec)
Bug: Move LeakReporter to utils, use more
Status: Open | Priority: Minor | Resolution: 
Created: 04/Dec/24 11:12
Affects Version/s: 3.4.2

Move LeakReporter of HADOOP-19330 from fs.impl to fs.utils, tag as public/evolving

1. This allows third party apps to track
2. pick up a few more places, e.g local fs output and input streams, s3a block output stream

Actually, we could add it to FSDataInputStream and have it everywhere. be noisy though, potentially

why #2? for unit testing in downstream apps.