av9ash/GitBugs huggingface dataset:
Bug: Upgrade Mockito version to 4.11.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/24 15:50
Affects Version/s: 3.4.1

While Compiling test classes with JDK17, faced error related to Mockito:
*Mockito cannot mock this class.*
So to make it compatible with jdk17 we have to upgrade the version of mockito-core as well as mockito-inline.
Bug: Use stable sort in commandQueue
Status: Open | Priority: Major | Resolution: 
Created: 09/Sep/24 06:16
Affects Version/s: 3.4.0

h2. Purpose
 - To remove possibility of wrong-ordered log simulation

h2. Why this happens?
 - private DelayQueue<AuditReplayCommand> commandQueue is actually PriorityQueue that use unstable sort.
 - commandQueue can have order that is not same to original audit log order.
 - In real production, there is the commands that occur same time and should be fixed order.
{code:bash}
# getfileinfo before open
2024-07-01 19:27:12,886 INFO FSNamesystem.audit: allowed=true   ugi=xx-xx (auth:TOKEN) via hive/hadoop.example.com@EXAMPLE.PROD (auth:TOKEN)   ip=/10.xx.xxx.xxx       cmd=getfileinfo src=/user/hive/warehouse/a.db/b/date_id=2024-06-16/part-xxxx.gz.parquet     dst=null        perm=null       proto=rpc
2024-07-01 19:27:12,886 INFO FSNamesystem.audit: allowed=true   ugi=xx-xx (auth:TOKEN) via hive/hadoop.example.com@EXAMPLE.PROD (auth:TOKEN)   ip=/10.xx.xxx.xxx       cmd=open        src=/user/hive/warehouse/a.db/b/date_id=2024-06-16/part-xxxx.gz.parquet     dst=null        perm=null       proto=rpc

# create before setPermission
# this examples have not exactly same time, but could be same when rate factor is high enough
2024-07-01 17:25:30,867 INFO FSNamesystem.audit: allowed=true   ugi=yy-yy@EXAMPLE.PROD (auth:KERBEROS)    ip=/10.xxx.xx.xxx       cmd=create      src=/user/yy-yy/.staging/job_1716867484406_290658/job.xml dst=null        perm=yy-yy:zzz:rw-rw-r--  proto=rpc
2024-07-01 17:25:30,871 INFO FSNamesystem.audit: allowed=true   ugi=yy-yy@EXAMPLE.PROD (auth:KERBEROS)    ip=/10.xxx.xx.xxx       cmd=setPermission       src=/user/yy-yy/.staging/job_1716867484406_290658/job.xml dst=null        perm=yy-yy:zzz:rw-r--r--  proto=rpc
{code}

h2. How much improve test accuracy when use stable sort?
 - Using stable sort, wrong ordered simulation could not occur.
 -- I fixed code to use line number of audit log in sorting criteria.
 -- Because it is not simple to change DelayQueue data structure to use stable sort
 - Multi threading or client-ip-based-partitioning could be occur in real production and affect log order, but not critical.
 -- Client-ip-based-partitioning is even similar to real production choas log order
 - This is the graph that
 -- use real production hdfs audit log
 -- compare stable sort and unstable sort with different rate(1~4)
 -- use 5 minutes simulation(in rate 1) ip-based-partitioned-audit-log
 -- shows total valid command, total read latency, total write latency
!image-2024-09-19-16-40-34-947.png|width=615,height=740!
 - Conclusion
 -- Stable sort ensure almost similar valid command number.
 -- Unstable sort sometimes extremely high latency because of wrong ordered log simulation.
Bug: S3A: remove option to delete directory markers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/24 17:25
Affects Version/s: 3.4.1

S3A no longer supports the ability to disable deletion of parent directory
markers on file or directory creation.

The option "fs.s3a.directory.marker.retention" is no longer supported.

This is incompatible with all hadoop versions before 3.2.2 and with hadoop 3.3.0
    when applications using the s3a connector attempt to write to the same repository.

Fix: upgrade to a recent version of the Hadoop libraries when working with S3 storage, for which there are many reasons


h3. cherrypicking:
* include HADOOP-19464
Bug: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/24 11:36
Affects Version/s: 3.4.0

Currently, we only support rename and delete operations on the DFS endpoint. The reason for supporting rename and delete operations on the Blob endpoint is that the Blob endpoint does not account for hierarchy. We need to ensure that the HDFS contracts are maintained when performing rename and delete operations. Renaming or deleting a directory over the Blob endpoint requires the client to handle the orchestration and rename or delete all the blobs within the specified directory.
 
The task outlines the considerations for implementing rename and delete operations for the FNS-blob endpoint to ensure compatibility with HDFS contracts.
 * {*}Blob Endpoint Usage{*}: The task addresses the need for abstraction in the code to maintain HDFS contracts while performing rename and delete operations on the blob endpoint, which does not support hierarchy.
 * {*}Rename Operations{*}: The {{AzureBlobFileSystem#rename()}} method will use a {{RenameHandler}} instance to handle rename operations, with separate handlers for the DFS and blob endpoints. This method includes prechecks, destination adjustments, and orchestration of directory renaming for blobs.
 * {*}Atomic Rename{*}: Atomic renaming is essential for blob endpoints, as it requires orchestration to copy or delete each blob within the directory. A configuration will allow developers to specify directories for atomic renaming, with a JSON file to track the status of renames.
 * {*}Delete Operations{*}: Delete operations are simpler than renames, requiring fewer HDFS contract checks. For blob endpoints, the client must handle orchestration, including managing orphaned directories created by Az-copy.
 * {*}Orchestration for Rename/Delete{*}: Orchestration for rename and delete operations over blob endpoints involves listing blobs and performing actions on each blob. The process must be optimized to handle large numbers of blobs efficiently.
 * {*}Need for Optimization{*}: Optimization is crucial because the {{ListBlob}} API can return a maximum of 5000 blobs at once, necessitating multiple calls for large directories. The task proposes a producer-consumer model to handle blobs in parallel, thereby reducing processing time and memory usage.
 * {*}Producer-Consumer Design{*}: The proposed design includes a producer to list blobs, a queue to store the blobs, and a consumer to process them in parallel. This approach aims to improve efficiency and mitigate memory issues.

More details will follow

Perquisites for this Patch:
1. HADOOP-19187 ABFS: [FnsOverBlob]Making AbfsClient Abstract for supporting both DFS and Blob Endpoint - ASF JIRA (apache.org)

2. HADOOP-19226 ABFS: [FnsOverBlob]Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient - ASF JIRA (apache.org)

3. HADOOP-19207 ABFS: [FnsOverBlob]Response Handling of Blob Endpoint APIs and Metadata APIs - ASF JIRA (apache.org)
Bug: ABFS: Initialize ABFS client timer only when metric collection is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/24 05:22
Affects Version/s: 3.4.0

In the current flow, we are initializing the timer of the {{abfs-timer-client}} outside the metric collection enable check. As a result, for each file system, when the {{AbfsClient}} object is initialized, it spawns a thread to evaluate the time of the ABFS client. Since we are purging/closing the timer inside the metric collection check, these threads are not being closed, causing them to persist in a long-lived state. To fix this, we are moving the timer initialization inside the condition
Bug: ABFS: [FnsOverBlob] Implementing Ingress Support with various Fallback Handling
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/24 11:33
Affects Version/s: 3.4.0

Scope of this task is to refactor the AbfsOutputStream class to handle the ingress for DFS and Blob endpoint effectively.

More details will be added soon.

Perquisites for this Patch:
1. [HADOOP-19187] ABFS: [FnsOverBlob]Making AbfsClient Abstract for supporting both DFS and Blob Endpoint - ASF JIRA (apache.org)

2. [HADOOP-19226] ABFS: [FnsOverBlob]Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient - ASF JIRA (apache.org)

3. [HADOOP-19207] ABFS: [FnsOverBlob]Response Handling of Blob Endpoint APIs and Metadata APIs - ASF JIRA (apache.org)
Bug: AliyunOSS: Add a feature to disable redirection for the OSS connector.
Status: Open | Priority: Major | Resolution: 
Created: 31/Jul/24 11:14
Affects Version/s: 3.1.0

For security reasons, some users of the OSS connector wish to disable the connector's HTTP redirection functionality. The OSS Java SDK have the capability to turn off HTTP redirection, but the configuration is not exposed in the {{core-site.xml}} file. This change primarily involves adding a flag to disable HTTP redirection in the {{core-site.xml}} file
Bug: ABFS: [FnsOverBlob] Adding Integration Tests for Special Scenarios in Blob Endpoint
Status: Open | Priority: Major | Resolution: 
Created: 23/Jul/24 12:03
Affects Version/s: 3.4.0

FNS accounts does not understand directories and to create that abstraction client has to handle the cases where hdfs operations include interactions with directory paths. This needs some additional testing to be done for each HDFS operation where path can exists as directory.

More details to follow

Perquisites: 
 # HADOOP-19187 ABFS: [FnsOverBlob]Making AbfsClient Abstract for supporting both DFS and Blob Endpoint
 # HADOOP-19207 ABFS: [FnsOverBlob]Response Handling of Blob Endpoint APIs and Metadata APIs[|https://issues.apache.org/jira/secure/DeleteLink.jspa?id=13579416&destId=13583033&linkType=12310460&atl_token=A5KQ-2QAV-T4JA-FDED_17fc7154167b7d6d6490aa6508db554fd6d7af24_lin]
 # HADOOP-19226 ABFS: [FnsOverBlob]Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient
 # HADOOP-19232 ABFS: [FnsOverBlob] Implementing Ingress Support with various Fallback Handling
 # HADOOP-19233 ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint
Bug: Vector IO on cloud storage: what is a good minimum seek size?
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jul/24 19:18
Affects Version/s: 3.4.1

vector iO has a max size to coalesce ranges, but it also needs a maximum gap between ranges to justify the merge. Right now we could have a read where two vectors of size 8 bytes can be merged with a 1 MB gap between them -and that's wasteful. 

We could also consider an "efficiency" metric which looks at the ratio of bytes-read to bytes-discarded. Not sure what we'd do with it, but we could track it as an IOStat

h2. Current values

The thresholds at which adjacent vector IO read ranges are coalesced into a
single range has been increased, as has the limit at which point they are 
considered large enough that parallel reads are faster.

* The min/max for local filesystems and any other FS without custom support are 
now 16K and 1M
* s3a and abfs use 128K as the minimum size, 2M for max.
Bug: Upgrade to 9.4.57.v20241219 due to CVE-2024-8184
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jul/24 08:46

Upgrade to jetty 9.4.56 due to

[CVE-2024-22201|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-22201]

[CVE-2023-44487|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-44487]

[CVE-2024-8184|https://nvd.nist.gov/vuln/detail/CVE-2024-8184] : [https://github.com/advisories/GHSA-g8m5-722r-8whq]
Bug: [JDK17] Upgade wildfly-openssl:1.1.3.Final to 2.1.4.Final+
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/24 19:19
Affects Version/s: 3.5.0

Apparentlly wildfly has to be updated to  2.1.4.Final to work on java17+; without that s3 and azure need to be configured to not use openssl to connect to stores
Bug: Integration of Volcano Engine TOS in Hadoop.
Status: Open | Priority: Major | Resolution: 
Created: 25/Jul/24 02:49
Affects Version/s: 3.4.0

Volcano Engine is a fast growing cloud vendor launched by ByteDance, and TOS is the object storage service of Volcano Engine. A common way is to store data into TOS and run Hadoop/Spark/Flink applications to access TOS. But there is no original support for TOS in hadoop, thus it is not easy for users to build their Big Data System based on TOS.
 
This work aims to integrate TOS with Hadoop to help users run their applications on TOS. Users only need to do some simple configuration, then their applications can read/write TOS without any code change. This work is similar to AWS S3, AzureBlob, AliyunOSS, Tencnet COS and HuaweiCloud Object Storage in Hadoop.

 

 Please see the attached document "Integration of Volcano Engine TOS in Hadoop" for more details.
Bug: Implement bulk delete command as hadoop fs command operation 
Status: Open | Priority: Major | Resolution: 
Created: 20/Aug/24 20:20
Affects Version/s: 3.4.1


{code}
hadoop fs -bulkdelete <base-url> <file> 

{code}

Key uses
* QE: Testing from python and other scripting languages
* cluster maintenance: actual bulk deletion operations from the store

one thought there: we MUST qualify paths with / elements: if a passed in path ends in /, it means "delete a marker", not "delete a dir"'. and if it doesn't have one then it's an object.. This makes it possible to be used to delete surplus markers or where there is a file above another file...cloudstore listobjects finds this
Bug: ABFS: [FnsOverBlob] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/24 04:37
Affects Version/s: 3.4.0

This is second task in series of tasks for implementing Blob Endpoint support for FNS accounts.

This patch will have changes to implement all the APIs over Blob Endpoint as a part of implementing AbfsBlobClient.
Bug: NoSuchMethodError in aws sdk third party logger in hadoop aws 3.4
Status: Open | Priority: Major | Resolution: 
Created: 30/Jul/24 04:18
Affects Version/s: 3.4.0

{code:java}
"localizedMessage": "java.lang.NoSuchMethodError: 'software.amazon.awssdk.thirdparty.org.slf4j.Logger software.amazon.awssdk.utils.Logger.logger()'",
"message": "java.lang.NoSuchMethodError: 'software.amazon.awssdk.thirdparty.org.slf4j.Logger software.amazon.awssdk.utils.Logger.logger()'",
"name": "com.google.common.util.concurrent.ExecutionError",
"cause": {
  "commonElementCount": 1,
  "localizedMessage": "'software.amazon.awssdk.thirdparty.org.slf4j.Logger software.amazon.awssdk.utils.Logger.logger()'",
  "message": "'software.amazon.awssdk.thirdparty.org.slf4j.Logger software.amazon.awssdk.utils.Logger.logger()'",
  "name": "java.lang.NoSuchMethodError",
  "extendedStackTrace": [
    {
      "class": "software.amazon.awssdk.transfer.s3.internal.GenericS3TransferManager",
      "method": "close",
      "file": "GenericS3TransferManager.java",
      "line": 393,
      "exact": false,
      "location": "bundle-2.23.19.jar",
      "version": "?"
    },
    {
      "class": "org.apache.hadoop.fs.s3a.S3AUtils",
      "method": "closeAutocloseables",
      "file": "S3AUtils.java",
      "line": 1553,
      "exact": false,
      "location": "hadoop-aws-3.4.0.jar",
      "version": "?"
    },
    {
      "class": "org.apache.hadoop.fs.s3a.S3AFileSystem",
      "method": "stopAllServices",
      "file": "S3AFileSystem.java",
      "line": 4358,
      "exact": false,
      "location": "hadoop-aws-3.4.0.jar",
      "version": "?"
    },
    {
      "class": "org.apache.hadoop.fs.s3a.S3AFileSystem",
      "method": "initialize",
      "file": "S3AFileSystem.java",
      "line": 758,
      "exact": false,
      "location": "hadoop-aws-3.4.0.jar",
      "version": "?"
    },
    {
      "class": "org.apache.hadoop.fs.FileSystem",
      "method": "createFileSystem",
      "file": "FileSystem.java",
      "line": 3601,
      "exact": false,
      "location": "hadoop-common-3.4.0.jar",
      "version": "?"
    },
    {
      "class": "org.apache.hadoop.fs.FileSystem",
      "method": "get",
      "file": "FileSystem.java",
      "line": 552,
      "exact": false,
      "location": "hadoop-common-3.4.0.jar",
      "version": "?"
    }, {code}
 

This appears to be related to how shading works in the aws bundle sdk.

Hadoop Version : hadoop-aws-3.4.0

AWS-SDK-Bundle - 2.23.19
Bug: Support S3A cross region access when S3 region/endpoint is set
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Sep/24 04:43
Affects Version/s: 3.4.0

Currently when S3 region nor endpoint is set, the default region is set to us-east-2 with cross region access enabled. But when region or endpoint is set, cross region access is not enabled.

A new option enables cross region access as a separate config and enable/disables it
irrespective of region/endpoint is set.
   s3a.cross.region.access.enabled

default: enables cross region access as a separate config and enable/disables it
irrespective of region/endpoint is set.
Bug: S3A: fs.s3a.connection.request.timeout too low for large uploads over slow links
Status: Reopened | Priority: Major | Resolution: 
Created: 30/Sep/24 18:36
Affects Version/s: 3.4.0

The value of {{fs.s3a.connection.request.timeout}} (default = 60s} is too low for large uploads over slow connections.

I suspect something changed between the v1 and v2 SDK versions so that put was exempt from the normal timeouts, It is not and now surfaces in failures to upload 1+ GB files over slower network connections. Smailer (for example 128 MB) files work.

The parallel queuing of writes in the S3ABlockOutputStream is helping create this problem as it queues multiple blocks at the same time, so per-block bandwidth becomes available/blocks ; four blocks cuts the capacity down by a quarter.

The fix is straightforward: use a much bigger timeout. I'm going to propose 15 minutes. We need to strike a balance between upload time allocation and other requests timing out.

I do worry about other consequences; we've found that timeout exception happy to hide the underlying causes of retry failures -so in fact this may be better for all but a server hanging after the HTTP request is initiated.

too bad we can't alter the timeout for different requests
Bug: ABFS: Allow "fs.azure.account.hns.enabled" to be set as Account Specific Config
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/24 04:50
Affects Version/s: 3.4.0

There are a few reported requirements where users working with multiple file systems need to specify this config either only for some accounts or set it differently for different account.
ABFS driver today does not allow this to be set as account specific config.

This Jira is to allow that as a new support.
Bug: Fix KMSTokenRenewer#handleKind dependency on BouncyCastleProvider class
Status: Open | Priority: Major | Resolution: 
Created: 24/Sep/24 08:26

 
{code:java}
Exception in thread "main" java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider
    at org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer.handleKind(KMSClientProvider.java:180)
    at org.apache.hadoop.security.token.Token.getRenewer(Token.java:467)
    at org.apache.hadoop.security.token.Token.renew(Token.java:500)
    at org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.$anonfun$getTokenRenewalInterval$3(HadoopFSDelegationTokenProvider.scala:147)
    at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
    at scala.util.Try$.apply(Try.scala:217)
    at org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.$anonfun$getTokenRenewalInterval$2(HadoopFSDelegationTokenProvider.scala:146) {code}
Bug: `CombinedFileRange.merge` should not convert disjoint ranges into overlapped ones
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/24 05:01
Affects Version/s: 3.3.9

Currently, Hadoop has a bug to convert disjoint ranges into overlapped ones and eventually fails by itself.
{code:java}
+  public void testMergeSortedRanges() {
+    List<FileRange> input = asList(
+        createFileRange(13816220, 24, null),
+        createFileRange(13816244, 7423960, null)
+    );
+    assertIsNotOrderedDisjoint(input, 100, 800);
+    final List<CombinedFileRange> outputList = mergeSortedRanges(
+        sortRangeList(input), 100, 1001, 2500);
+
+    assertRangeListSize(outputList, 1);
+    assertFileRange(outputList.get(0), 13816200, 7424100);
+  }
{code}

 !Screenshot 2024-09-28 at 22.02.01.png!
Bug: Resolve Certificate error in Hadoop-auth tests.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jul/24 10:36
Affects Version/s: 3.4.2

While compiling Hadoop-Trunk with JDK17, faced following errors in TestMultiSchemeAuthenticationHandler and 
TestLdapAuthenticationHandler classes.


{code:java}
[INFO] Running org.apache.hadoop.security.authentication.server.TestMultiSchemeAuthenticationHandler
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.256 s <<< FAILURE! - in org.apache.hadoop.security.authentication.server.TestMultiSchemeAuthenticationHandler
[ERROR] org.apache.hadoop.security.authentication.server.TestMultiSchemeAuthenticationHandler  Time elapsed: 1.255 s  <<< ERROR!
java.lang.IllegalAccessError: class org.apache.directory.server.core.security.CertificateUtil (in unnamed module @0x32e614e9) cannot access class sun.security.x509.X500Name (in module java.base) because module java.base does not export sun.security.x509 to unnamed module @0x32e614e9
        at org.apache.directory.server.core.security.CertificateUtil.createTempKeyStore(CertificateUtil.java:334)
        at org.apache.directory.server.factory.ServerAnnotationProcessor.instantiateLdapServer(ServerAnnotationProcessor.java:158)
        at org.apache.directory.server.factory.ServerAnnotationProcessor.createLdapServer(ServerAnnotationProcessor.java:318)
        at org.apache.directory.server.factory.ServerAnnotationProcessor.createLdapServer(ServerAnnotationProcessor.java:351) {code}
Bug: S3A: Unable to recover from failure of multipart block upload attempt "Status Code: 400; Error Code: RequestTimeout"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/24 15:57
Affects Version/s: 3.4.0

If a multipart PUT request fails for some reason (e.g. networrk error) then all subsequent retry attempts fail with a 400 Response and ErrorCode RequestTimeout .

{code}
Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. (Service: Amazon S3; Status Code: 400; Error Code: RequestTimeout; Request ID:; S3 Extended Request ID:
{code}

The list of supporessed exceptions contains the root cause (the initial failure was a 500); all retries failed to upload properly from the source input stream {{RequestBody.fromInputStream(fileStream, size)}}.

Hypothesis: the mark/reset stuff doesn't work for input streams. On the v1 sdk we would build a multipart block upload request passing in (file, offset, length), the way we are now doing this doesn't recover.

probably fixable by providing our own {{ContentStreamProvider}} implementations for
# file + offset + length
# bytebuffer
# byte array

The sdk does have explicit support for the memory ones, but they copy the data blocks first. we don't want that as it would double the memory requirements of active blocks.
Bug: Avoid Subject.getSubject method on newer JVMs
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 30/Sep/24 14:45

In Java 23, Subject.getSubject requires setting the system property java.security.manager to allow, else it will throw an exception. More detail is available in the release notes: https://jdk.java.net/23/release-notes

This is in support of the eventual removal of the security manager, at which point, Subject.getSubject will be removed.
Bug: BlockDecompressorStream#rawReadInt wastes about 1% of overall CPU cycles creating new EOFException
Status: Patch Available | Priority: Major | Resolution: 
Created: 30/Sep/24 11:56
Affects Version/s: 3.3.6

On our HBase clusters, while looking at CPU profiles, I noticed that about 1% of overall CPU cycles are spent under BlockDecompressorStream#rawReadInt just throwing EOFException. This could be easily avoided.