av9ash/GitBugs huggingface dataset:
Bug: S3A: S3A: ITestS3AConfiguration failing with region problems
Status: Open | Priority: Major | Resolution: 
Created: 12/Jan/24 15:26
Affects Version/s: 3.4.0

After commented out the default region in my ~/.aws/config [default} profile, test ITestS3AConfiguration. testS3SpecificSignerOverride() fails


{code}
[ERROR] testS3SpecificSignerOverride(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)  Time elapsed: 0.054 s  <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@12c626f8: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@ae63559: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@6e6cfd4c: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@139147de: EC2 Metadata is disabled. Unable to retrieve region information from EC2 Metadata service.]

{code}

I'm worried the sdk update has rolled back to the 3.3.x region problems where well-configured developer setups / ec2 deployments hid problems. certainly we can see the code is checking these paths
Bug: S3A HeaderProcessing to process all metadata entries of HEAD response
Status: Open | Priority: Major | Resolution: 
Created: 07/Mar/24 14:32
Affects Version/s: 3.4.0

S3A HeaderProcessing builds up an incomplete list of headers as its mapping of md to header. entries omits headers including
x-amz-server-side-encryption-aws-kms-key-id

proposed
* review all headers which are stripped from "raw" responses and mapped into headers
* make sure result of headers matches v1; looks like etags are different
* make sure x-amz-server-side-encryption-aws-kms-key-id gets back
* plus new checksum values


v1 sdk

{code}

# file: s3a://noaa-cors-pds/raw/2024/001/akse/AKSE001x.24_.gz
header.Content-Length="524671"
header.Content-Type="binary/octet-stream"
header.ETag="3e39531220fbd3747d32cf93a79a7a0c"
header.Last-Modified="Tue Jan 02 00:15:13 GMT 2024"
header.x-amz-server-side-encryption="AES256"

{code}

v2 SDK. note how etag is now double quoted.

{code}

# file: s3a://noaa-cors-pds/raw/2024/001/akse/AKSE001x.24_.gz
header.Content-Length="524671"
header.Content-Type="binary/octet-stream"
header.ETag=""3e39531220fbd3747d32cf93a79a7a0c""
header.Last-Modified="Tue Jan 02 00:15:13 GMT 2024"
header.x-amz-server-side-encryption="AES256"

{code}
Bug: S3A: ITestCustomSigner failing against S3Express Buckets
Status: Open | Priority: Major | Resolution: 
Created: 19/Jan/24 14:18
Affects Version/s: 3.5.0

getting test failures against S3 Express buckets with {{ItestCustomSigner}}; not seen with classic s3 stores.


{code}
[ERROR] testCustomSignerAndInitializer[simple-delete](org.apache.hadoop.fs.s3a.auth.ITestCustomSigner)  Time elapsed: 6.12 s  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: PUT 0-byte object  on fork-0006/test/testCustomSignerAndInitializer[simple-delete]/customsignerpath1: software.amazon.awssdk.services.s3.model.S3Exception: x-amz-sdk-checksum-algorithm specified, but no corresponding x-amz-checksum-* or x-amz-trailer headers were found. (Service: S3, Status Code: 400, Request ID: 0033eada6b00018d21962f1b05094a80435cca52, Extended Request ID: kZJZG05LGCBu7lsNKNf):InvalidRequest: x-amz-sdk-checksum-algorithm specified, but no corresponding x-amz-checksum-* or x-amz-trailer headers were found. (Service: S3, Status Code: 400, Request ID: 0033eada6b00018d21962f1b05094a80435cca52, Extended Request ID: kZJZG05LGCBu7lsNKNf)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:259)
...
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: x-amz-sdk-checksum-algorithm specified, but no corresponding x-amz-checksum-* or x-amz-trailer headers were found. (Service: S3, Status Code: 400, Request ID: 0033eada6b00018d21962f1b05094a80435cca52, Extended Request ID: kZJZG05LGCBu7lsNKNf)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)


{code}
Bug: S3 Express: document use
Status: Open | Priority: Major | Resolution: 
Created: 12/Mar/24 18:00
Affects Version/s: 3.4.0

The 3.4.0 release doesn't explicitly cover S3 Express.

It's support is automatic
* library handles it
* hadoop shell commands know that there may be "missing" dirs in treewalks due to in-flight uploads
* s3afs automatically switches to deleting pending uploads in delete(dir) call.

we just need to provide a summary of features, how to probe etc.
Bug: S3A: ITestS3AFileContextURI: MultiObjectDeleteException bulk delete of odd filenames
Status: Open | Priority: Major | Resolution: 
Created: 10/Jan/24 14:40
Affects Version/s: 3.4.0

Possibly transient. note bucket is versioned.

{code}
org.apache.hadoop.fs.s3a.AWSS3IOException: 
Remove S3 Dir Markers on s3a://stevel-london/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest: org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteException: [S3Error(Key=Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest/()&^%$#@!~_+}{><?/, Code=InternalError, Message=We encountered an internal error. Please try again.)] (Service: Amazon S3, Status Code: 200, Request ID: null):MultiObjectDeleteException: 

{code}
Bug: [ABFS] Reverting Back Support of setXAttr() and getXAttr() on root path
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/24 10:40
Affects Version/s: 3.4.0

A while back changes were made to support HDFS.setXAttr() and HDFS.getXAttr() on root path for ABFS Driver.
For these, filesystem level APIs were introduced and used to set/get metadata of container.
Refer to Jira: [HADOOP-18869] ABFS: Fixing Behavior of a File System APIs on root path - ASF JIRA (apache.org)

Ideally, same set of APIs should be used, and root should be treated as a path like any other path.
This change is to avoid calling container APIs for these HDFS calls.

As a result of this these APIs will fail on root path (as earlier) because service does not support get/set of user properties on root path.
This change will also update the documentation to reflect that these operations are not supported on root path.
Bug: [ABFS] Enhancing Client-Side Throttling Metrics Updation Logic
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Feb/24 10:05
Affects Version/s: 3.4.1

ABFS has a client-side throttling mechanism which works on the metrics collected from past requests made. I requests are getting failed due to throttling at server, we update our metrics and client side backoff is calculated based on those metrics.

This PR enhances the logic to decide which requests should be considered to compute client side backoff interval as follows:

For each request made by ABFS driver, we will determine if they should contribute to Client-Side Throttling based on the status code and result:
 # Status code in 2xx range: Successful Operations should contribute.
 # Status code in 3xx range: Redirection Operations should not contribute.
 # Status code in 4xx range: User Errors should not contribute.
 # Status code is 503: Throttling Error should contribute only if they are due to client limits breach as follows:
 ## 503, Ingress Over Account Limit: Should Contribute
 ## 503, Egress Over Account Limit: Should Contribute
 ## 503, TPS Over Account Limit: Should Contribute
 ## 503, Other Server Throttling: Should not Contribute.
 # Status code in 5xx range other than 503: Should not Contribute.
 # IOException and UnknownHostExceptions: Should not Contribute.
Bug: ITestExponentialRetryPolicy failing in branch-3.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Mar/24 18:15
Affects Version/s: 3.4.0

{code:java}
[ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 2, Time elapsed: 91.416 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.services.ITestExponentialRetryPolicy
[ERROR] testThrottlingIntercept(org.apache.hadoop.fs.azurebfs.services.ITestExponentialRetryPolicy)  Time elapsed: 0.622 s  <<< ERROR!
Failure to initialize configuration for dummy.dfs.core.windows.net key ="null": Invalid configuration value detected for fs.azure.account.key
	at org.apache.hadoop.fs.azurebfs.services.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:53)
	at org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:646)
	at org.apache.hadoop.fs.azurebfs.services.ITestAbfsClient.createTestClientFromCurrentContext(ITestAbfsClient.java:339)
	at org.apache.hadoop.fs.azurebfs.services.ITestExponentialRetryPolicy.testThrottlingIntercept(ITestExponentialRetryPolicy.java:106)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) {code}
Bug: [ABFS] All tests of. ITestAzureBlobFileSystemAuthorization fails with NPE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/24 19:08
Affects Version/s: 3.4.0

When below config set to true all of the tests fails else it skips.

<property>

    <name>fs.azure.test.namespace.enabled</name>

    <value>true</value>

</property>

 

[*ERROR*] testOpenFileAuthorized(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization)  Time elapsed: 0.064 s  <<< ERROR!

java.lang.NullPointerException

 at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization.runTest(ITestAzureBlobFileSystemAuthorization.java:273)

 at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization.testOpenFileAuthorized(ITestAzureBlobFileSystemAuthorization.java:132)

 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 at java.lang.reflect.Method.invoke(Method.java:498)

 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)

 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)

 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)

 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
Bug: provide hadoop binary tarball without aws v2 sdk
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/24 17:25
Affects Version/s: 3.4.0

Have the default hadoop binary .tar.gz exclude the aws v2 sdk by default. 

This SDK brings the total size of the distribution to about 1 GB.

Proposed
* add a profile to include the aws sdk in the dist module
* document it for local building
* for release builds, we modify our release ant builds to generate modified x86 and arm64 releases without the file.
Bug: Drop support for HBase v1 timeline service & upgrade HBase v2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/24 09:13
Affects Version/s: 3.4.0

Drop support for Hbase V1 as the back end of the YARN Application Timeline service, which becomes HBase 2 only.

This does not have any effect on HBase deployments themselves.

Dev List:

[https://lists.apache.org/thread/vb2gh5ljwncbrmqnk0oflb8ftdz64hhs]

https://lists.apache.org/thread/o88hnm7q8n3b4bng81q14vsj3fbhfx5w
Bug: [ABFS]: ApacheHttpClient adaptation as network library
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Mar/24 10:49
Affects Version/s: 3.5.0

Apache HttpClient is more feature-rich and flexible and gives application more granular control over networking parameter.

ABFS currently relies on the JDK-net library. This library is managed by OpenJDK and has no performance problem. However, it limits the application's control over networking, and there are very few APIs and hooks exposed that the application can use to get metrics, choose which and when a connection should be reused. ApacheHttpClient will give important hooks to fetch important metrics and control networking parameters.

A custom implementation of connection-pool is used. The implementation is adapted from the JDK8 connection pooling. Reasons for doing it:
1. PoolingHttpClientConnectionManager heuristic caches all the reusable connections it has created. JDK's implementation only caches limited number of connections. The limit is given by JVM system property "http.maxConnections". If there is no system-property, it defaults to 5. Connection-establishment latency increased with all the connections were cached. Hence, adapting the pooling heuristic of JDK netlib,
2. In PoolingHttpClientConnectionManager, it expects the application to provide `setMaxPerRoute` and `setMaxTotal`, which the implementation uses as the total number of connections it can create. For application using ABFS, it is not feasible to provide a value in the initialisation of the connectionManager. JDK's implementation has no cap on the number of connections it can have opened on a moment. Hence, adapting the pooling heuristic of JDK netlib,
Bug: WrappedIO to export modern filesystem/statistics APIs in a reflection friendly form
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/24 16:50
Affects Version/s: 3.3.6

parquet, avro etc are still stuck building with older hadoop releases. 

This makes using new APIs hard (PARQUET-2171) and means that APIs which are 5 years old such as HADOOP-15229 just aren't picked up.

This lack of openFIle() adoption hurts working with files in cloud storage as
* extra HEAD requests are made
* read policies can't be explicitly set
* split start/end can't be passed down

HADOOP-18679 added a new WrappedIO class.

This jira proposes extending this with
* more of the filesystem/input stream methods
* iOStatistics
* Pull in parquet DynMethods to dynamially wrap and invoke through tests. This class, DynamicWrappedIO is intended to be copied into libraries (parquet, iceberg) for their own use. 
* existing tests to use the dynamic binding for end-to-end testing.

+then get into the downstream libraries and use where appropriate
Bug: S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Feb/24 15:00
Affects Version/s: 3.4.0

on an s3a store with fs.s3a.create.performance set, speed up other operations

*  mkdir to skip parent directory check: just do a HEAD to see if there's a file at the target location
Bug: ABFS phase 4: post Hadoop 3.4.0 features
Status: Open | Priority: Major | Resolution: 
Created: 27/Feb/24 11:08
Affects Version/s: 3.4.0

Uber-JIRA for ABFS work so we can close HADOOP-18072 as done for 3.4.0

Assuming 3.4.1 is a rapid roll of packing, dependencies and critical fixes, this should target 3.4.2 and beyond
Bug: Vector IO: consistent specified rejection of overlapping ranges
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/24 14:54
Affects Version/s: 3.3.6

Related to PARQUET-2171 q: "how do you deal with overlapping ranges?"

I believe s3a rejects this, but the other impls may not.

Proposed

FS spec to say 
* "overlap triggers IllegalArgumentException". 
* special case: 0 byte ranges may be short circuited to return empty buffer even without checking file length etc.

Contract tests to validate this

(+ common helper code to do this).

I'll copy the validation stuff into the parquet PR for consistency with older releases
Bug: add shaded jersey jar hadoop-shaded-jersey
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Feb/24 06:09
Affects Version/s: thirdparty-1.3.0

We need to address the issue of upgrading Jersey, replacing Jersey 1.x with Jersey 2.x. Jersey 2 and Jersey 1 are completely different, with many differences, and the same project cannot simultaneously introduce Jersey 2 and Jersey 1.
In order to allow the project to reference both Jersey 2 and Jersey 1 simultaneously, facilitating the step-by-step module upgrade process, I plan to introduce Jersey 2 into hadoop-thirdparty.
Bug: Use bouncycastle jdk18 1.77
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/24 16:04

They have stopped patching the JDK 1.5 jars that Hadoop uses (see https://issues.apache.org/jira/browse/HADOOP-18540).

The new artifacts have similar names - but the names are like bcprov-jdk18on as opposed to bcprov-jdk15on.

CVE-2023-33201 is an example of a security issue that seems only to be fixed in the JDK 1.8 artifacts (ie no JDK 1.5 jar has the fix).

https://www.bouncycastle.org/releasenotes.html#r1rv77 latest current release but the CVE was fixed in 1.74.
Bug: upgrade to commons-compress 1.26.1 due to cves
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/24 22:48
Affects Version/s: 3.4.0

2 recent CVEs fixed - https://mvnrepository.com/artifact/org.apache.commons/commons-compress


Important: Denial of Service CVE-2024-25710
Moderate: Denial of Service CVE-2024-26308
Bug: Improve rate limiting through ABFS in Manifest Committer
Status: Open | Priority: Major | Resolution: 
Created: 27/Feb/24 12:49
Affects Version/s: 3.4.0

I need a load test to verify that the rename resilience of the manifest committer actually works as intended

* test suite with name ILoadTest* prefix (as with s3)
* parallel test running with many threads doing many renames
* verify that rename recovery should be detected
* and that all renames MUST NOT fail.

maybe also: metrics for this in fs and doc update. 
Possibly; LogExactlyOnce to warn of load issues
Bug: Standardize Maven Initialization Across Operating Systems
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jan/24 11:18
Affects Version/s: 3.4.0

The differences in initializing Maven for various operating systems in the build scripts are as follows:

- For Ubuntu and Debian, Maven is installed using the yum repository.
- For CentOS 7 and CentOS 8, Maven is downloaded remotely.
- The Maven version used on Windows is inconsistent with other operating systems.
Bug: Compatibility Benchmark over HCFS Implementations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/24 04:18
Affects Version/s: 3.4.0

{*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in big data storage ecosystem, providing unified interfaces and generally clear semantics, and has become the de-factor standard for industry storage systems to follow and conform with. There have been a series of HCFS implementations in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object Storage, and more from storage service's providers on their own.

{*}Problems：{*}However, as indicated by introduction.md, there is no formal suite to do compatibility assessment of a file system for all such HCFS implementations. Thus, whether the functionality is well accomplished and meets the core compatible expectations mainly relies on service provider's own report. Meanwhile, Hadoop is also developing and new features are continuously contributing to HCFS interfaces for existing implementations to follow and update, in which case, Hadoop also needs a tool to quickly assess if these features are supported or not for a specific HCFS implementation. Besides, the known hadoop command line tool or hdfs shell is used to directly interact with a HCFS storage system, where most commands correspond to specific HCFS interfaces and work well. Still, there are cases that are complicated and may not work, like expunge command. To check such commands for an HCFS, we also need an approach to figure them out.

{*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility benchmark and provide corresponding tool to do the compatibility assessment for an HCFS storage system. The benchmark and tool should consider both HCFS interfaces and hdfs shell commands. Different scenarios require different kinds of compatibilities. For such consideration, we could define different suites in the benchmark.

*Benefits:* We intend the benchmark and tool to be useful for both storage providers and storage users. For end users, it can be used to evalute the compatibility level and determine if the storage system in question is suitable for the required scenarios. For storage providers, it helps to quickly generate an objective and reliable report about core functioins of the storage service. As an instance, if the HCFS got a 100% on a suite named 'tpcds', it is demonstrated that all functions needed by a tpcds program have been well achieved. It is also a guide indicating how storage service abilities can map to HCFS interfaces, such as storage class on S3.

Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
Bug: [ABFS]: FooterReadBufferSize should not be greater than readBufferSize
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/24 10:37
Affects Version/s: 3.4.0

The method `optimisedRead` creates a buffer array of size `readBufferSize`. If footerReadBufferSize is greater than readBufferSize, abfs will attempt to read more data than the buffer array can hold, which causes an exception.

Change: To avoid this, we will keep footerBufferSize = min(readBufferSizeConfig, footerBufferSizeConfig)
Bug: FTPFileSystem rename with full qualified path broken
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/24 02:00
Affects Version/s: 0.20.2

   When use fs shell to put/rename file in ftp server with full qualified path , it always get "Input/output error"(eg. [ftp://user:password@localhost/pathxxx]), the reason is that changeWorkingDirectory command underneath is being passed a string with [file://|file:///] uri prefix which will not be understand by ftp server

!image-2024-03-27-09-59-12-381.png|width=948,height=156!

 

in our case, after client.changeWorkingDirectory("ftp://mytest:mytest@10.5.xx.xx/files")

executed, the workingDirectory of ftp server is still "/", which is incorrect(not understand by ftp server)

!image-2024-03-28-09-58-19-721.png|width=745,height=431!

the solution should be pass absoluteSrc.getParent().toUri().getPath().toString to avoid
[file://|file:///] uri prefix, like this: 
{code:java}
--- a/FTPFileSystem.java
+++ b/FTPFileSystem.java
@@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
       throw new IOException("Destination path " + dst
           + " already exist, cannot rename!");
     }
-    String parentSrc = absoluteSrc.getParent().toUri().toString();
-    String parentDst = absoluteDst.getParent().toUri().toString();
+    URI parentSrc = absoluteSrc.getParent().toUri();
+    URI parentDst = absoluteDst.getParent().toUri();
     String from = src.getName();
     String to = dst.getName();
-    if (!parentSrc.equals(parentDst)) {
+    if (!parentSrc.toString().equals(parentDst.toString())) {
       throw new IOException("Cannot rename parent(source): " + parentSrc
           + ", parent(destination):  " + parentDst);
     }
-    client.changeWorkingDirectory(parentSrc);
+    client.changeWorkingDirectory(parentSrc.getPath().toString());
     boolean renamed = client.rename(from, to);
     return renamed;
   }{code}
already related issue  as follows 
https://issues.apache.org/jira/browse/HADOOP-8653

I create this issue and add related unit test.