av9ash/GitBugs huggingface dataset:
Bug: [ABFS][Retry Policy] Using hadoop-common code to refractor Abfs Retry Policy Implementation
Status: Open | Priority: Major | Resolution: 
Created: 07/Aug/23 07:15
Affects Version/s: 3.3.3

AbfsRetryPolicy is an independent module in hadoop-azure code. There is a lot of reimplementation of a more generic and advanced retry policy already present in hadoop-common. AbfsRetryPolicy should either inherit or directly use io.retry.Retrypolicy instead of reimplementing the same functionalities again.

AbfsRetryPolicy has only two versions: exponential and static. These both along with many others are already present in io.retry.Retrypolicy.

Issue identified in this PR: [Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout. by anujmodi2021 · Pull Request #5881 · apache/hadoop (github.com)|https://github.com/apache/hadoop/pull/5881]
Bug: ABFS: AbfsOutputStream doesnt close DataBlocks object.
Status: Open | Priority: Major | Resolution: 
Created: 30/Aug/23 07:27
Affects Version/s: 3.3.4

AbfsOutputStream doesnt close the dataBlock object created for the upload.

What is the implication of not doing that:
DataBlocks has three implementations:
 # ByteArrayBlock
 ## This creates an object of DataBlockByteArrayOutputStream (child of ByteArrayOutputStream: wrapper arround byte-arrray for populating, reading the array.
 ## This gets GCed.
 # ByteBufferBlock:
 ## There is a defined *DirectBufferPool* from which it tries to request the directBuffer.
 ## If nothing in the pool, a new directBuffer is created.
 ## the `close` method on the this object has the responsiblity of returning back the buffer to pool so it can be reused.
 ## Since we are not calling the `close`:
 ### The pool is rendered of less use, since each request creates a new directBuffer from memory.
 ### All the object can be GCed and the direct-memory allocated may be returned on the GC. What if the process crashes, the memory never goes back and cause memory issue on the machine.
 # DiskBlock:
 ## This creates a file on disk on which the data-to-upload is written. This file gets deleted in startUpload().close().

 

startUpload() gives an object of BlockUploadData which gives method of `toByteArray()` which is used in abfsOutputStream to get the byteArray in the dataBlock.

 

Method which uses the DataBlock object: https://github.com/apache/hadoop/blob/fac7d26c5d7f791565cc3ab45d079e2cca725f95/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java#L298
Bug: VectorIO API tuning/stabilization
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Aug/23 14:47
Affects Version/s: 3.3.9

Changes needed to get the Vector IO code stable.

Specifically
* consistent behaviour across implementations
* broader testing
* resilience

+Ideally, abfs support. (s3a prefetching needs this too; see HADOOP-19144)

This work will be shaped by the experience of merging support into libraries and identifying issues/improvement opportunities

h2. the patches which MUST be cherrypicked from this to any branch with vector IO support are (currently)

* HADOOP-19098
* HADOOP-19204
Bug: S3A: Cut S3 Select
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/23 12:39
Affects Version/s: 3.4.0

getting s3 select to work with the v2 sdk is tricky, we need to add extra libraries to the classpath beyond just bundle.jar. we can do this but

* AFAIK nobody has ever done CSV predicate pushdown, as it breaks split logic completely
* CSV is a bad format
* one-line JSON more structured but also way less efficient
ORC/Parquet benefit from vectored IO and work spanning the cluster.

accordingly, I'm wondering what to do about s3 select
# cut?
# downgrade to optional and document the extra classes on the classpath

Option #2 is straightforward and effectively the default. we can also declare the feature deprecated.

{code}

[ERROR] testReadLandsatRecordsNoMatch(org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat)  Time elapsed: 147.958 s  <<< ERROR!
java.io.IOException: java.lang.NoClassDefFoundError: software/amazon/eventstream/MessageDecoder
        at org.apache.hadoop.fs.s3a.select.SelectObjectContentHelper.select(SelectObjectContentHelper.java:75)
        at org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$select$10(WriteOperationHelper.java:660)
        at org.apache.hadoop.fs.store.audit.AuditingFunctions.lambda$withinAuditSpan$0(AuditingFunctions.java:62)
        at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:122)

{code}
Bug: Tune/extend S3A http connection and thread pool settings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/23 10:32
Affects Version/s: 3.4.0

Increases existing pool sizes, as with server scale and vector
IO, larger pools are needed

  fs.s3a.connection.maximum 200
  fs.s3a.threads.max 96

Adds new configuration options for v2 sdk internal timeouts,
both with default of 60s:

  fs.s3a.connection.acquisition.timeout
  fs.s3a.connection.idle.time

All the pool/timoeut options are covered in performance.md

Moves all timeout/duration options in the s3a FS to taking
temporal units (h, m, s, ms,...); retaining the previous default
unit (normally millisecond)

Adds a minimum duration for most of these, in order to recover from
deployments where a timeout has been set on the assumption the unit
was seconds, not millis.

Uses java.time.Duration throughout the codebase;
retaining the older numeric constants in
org.apache.hadoop.fs.s3a.Constants for backwards compatibility;
these are now deprecated.

Adds new class AWSApiCallTimeoutException to be raised on
sdk-related methods and also gateway timeouts. This is a subclass
of org.apache.hadoop.net.ConnectTimeoutException to support
existing retry logic.

+ reverted default value of fs.s3a.create.performance to false; 
inadvertently set to true during testing.
Bug: Expect-100 JDK bug resolution: prevent multiple server calls
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/23 11:28

This is inline to JDK bug: [https://bugs.openjdk.org/browse/JDK-8314978].

 
With the current implementation of HttpURLConnection if server rejects the “Expect 100-continue” then there will be ‘java.net.ProtocolException’ will be thrown from 'expect100Continue()' method.

After the exception thrown, If we call any other method on the same instance (ex getHeaderField(), or getHeaderFields()). They will internally call getOuputStream() which invokes writeRequests(), which make the actual server call. 




In the AbfsHttpOperation, after sendRequest() we call processResponse() method from AbfsRestOperation. Even if the conn.getOutputStream() fails due to expect-100 error, we consume the exception and let the code go ahead. So, we can have getHeaderField() / getHeaderFields() / getHeaderFieldLong() which will be triggered after getOutputStream is failed. These invocation will lead to server calls.
Bug: AWS SDK v2: make the v1 bridging support optional
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/23 16:31
Affects Version/s: 3.4.0

The AWS SDK v2 code includes the v1 sdk core for plugin support of

* existing credential providers
* delegation token binding

I propose we break #2 and rely on those who have implemented to to upgrade. apart from all the needless changes the v2 SDK did to the api (why?) this is fairly straighforward

for #1: fix through reflection, retaining a v1 sdk dependency at test time so we can verify that the binder works.
Bug: Upgrade aws-java-sdk to 1.12.499+
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/23 07:10
Affects Version/s: 3.4.0

aws sdk versions < 1.12.499 uses a vulnerable version of netty and hence showing up in security CVE scans (CVE-2023-34462). The safe version for netty is 4.1.94.Final and this is used by aws-java-sdk:1.12.499+
Bug: Upgrade Okio to 3.4.0 due to CVE-2023-3635
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/23 11:57
Affects Version/s: 3.4.0

Upgrade Okio to 3.4.0 due to CVE-2023-3635

GzipSource does not handle an exception that might be raised when parsing a malformed gzip buffer. This may lead to denial of service of the Okio client when handling a crafted GZIP archive, by using the GzipSource class.

CVSSv3 Score:- 7.5(High)

[https://nvd.nist.gov/vuln/detail/CVE-2023-3635]
Bug: Enable dual-layer server-side encryption with AWS KMS keys (DSSE-KMS)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/23 10:04
Affects Version/s: 3.4.0

Add support for DSSE-KMS

https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-dsse-encryption.html
Bug: AWS SDK V2 - Upgrade SDK to 2.20.28 and restores multipart copy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/23 10:31
Affects Version/s: 3.4.0

With 2.20.121, the TM has MPU functionality. Upgrading to the latest version (2.20.28) will also solve the issue with needing to include the CRT dependency.
Bug: AWS v2 SDK: fail meaningfully if legacy v2 signing requested
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/23 13:39
Affects Version/s: 3.4.0

if the v2 signer is requested on an AWS SDKv2 build, you get an NPE. 

proposed: special recognition of  fs.s3a.signing-algorithm=S3SignerType  and fail with a message that it isn't supported



{code}
> bin/hadoop jar $CLOUDSTORE storediag -D fs.s3a.signing-algorithm=S3SignerType -w  s3a://stevel-london/

2023-08-21 14:36:41,229 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(137)) - Duration of Creating filesystem for s3a://stevel-london/: 0:00:357
java.lang.NullPointerException
        at org.apache.hadoop.fs.s3a.auth.SignerFactory.createSigner(SignerFactory.java:104)
        at org.apache.hadoop.fs.s3a.impl.AWSClientConfig.createClientConfigBuilder(AWSClientConfig.java:90)
        at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createClientOverrideConfiguration(DefaultS3ClientFactory.java:163)
        at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureClientBuilder(DefaultS3ClientFactory.java:147)
        at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:89)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:989)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:610)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3601)

{code}
Bug: AWS SDK V2 - AuditFailureExceptions aren't being translated properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/23 14:38
Affects Version/s: 3.4.0

{{ITestS3AHugeFilesNoMultipart}} is failing because the {{AuditFailureException}} variant raised in the sdk handler is being wrapped as it makes its way back to the s3a code -but S3AUtiis.translateException() isn't looking at the inner cause.

looks like aws v2 sdk class {{.GenericMultipartHelper.handleException}} is wrapping an SdkException with a SdkClientException even though it is not needed.

we probably have to start looking at the inner cause of any exception during translation to see if that is also a AuditFailureException.

Filed https://github.com/aws/aws-sdk-java-v2/issues/4356
Bug: Optimize the configuration and use of callqueue overflow trigger failover
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/23 03:42
Affects Version/s: 3.4.0

In HADOOP-16268 implemented enable callqueue overflow trigger failover when RPC call queue is filled for hdfs router servers .

In order to configure and use parameters more conveniently, we are going to make the following optimizations:

* provide default properties for callqueue.overflow.trigger.failover such that if properties with port is not configured, we can fallback to default property (port-less)
* support refreshCallQueue to update callqueue.overflow.trigger.failover
* update callqueue.overflow.trigger.failover description information in core-site.xml
Bug: CURATOR-599 change broke functionality introduced in HADOOP-18139 and HADOOP-18709
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/23 15:04
Affects Version/s: 3.3.5

[Curator PR#391 |https://github.com/apache/curator/pull/391/files#diff-687a4ed1252bfb4f56b3aeeb28bee4413b7df9bec4b969b72215587158ac875dR59] introduced a default method in the ZooKeeperFactory interface, hence the override of the 4-parameter NewZookeeper method in the HadoopZookeeperFactory class is not taking effect due to this. 

Proposing routing the 4-parameter method to a 5-parameter method, which instantiates the ZKClientConfig as the 5th parameter. This is a non-breaking change, as the ZKClientConfig is currently instantiated within the method.
Bug: ABFS: Change default from disk to bytebuffer for fs.azure.data.blocks.buffer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/23 11:35
Affects Version/s: 3.3.6

Change default from disk to bytebuffer for fs.azure.data.blocks.buffer.

Gathered from multiple workload runs, the presented data underscores a noteworthy enhancement in performance. The adoption of ByteBuffer for *reading operations* exhibited a remarkable improvement of approximately *64.83%* when compared to traditional disk-based reading. Similarly, the implementation of ByteBuffer for *write operations* yielded a substantial efficiency gain of about {*}60.75%{*}. These findings underscore the consistent and substantial advantages of integrating ByteBuffer across a range of workload scenarios.
Bug: Negative timeout in ZKFailovercontroller due to overflow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/23 16:20
Affects Version/s: 3.3.6

Graceful fence timeout of FailoverController in ZKFailovercontroller equals to `ha.failover-controller.graceful-fence.rpc-timeout.ms` * 2. Since users are unaware of this calculation, it thus has risks of overflowing to a negative number if users set `ha.failover-controller.graceful-fence.rpc-timeout.ms` to a large value.
 
To reproduce:
1. set `ha.failover-controller.graceful-fence.rpc-timeout.ms` to 1092752431
2. run `mvn surefire:test -Dtest=org.apache.hadoop.ha.TestZKFailoverController#testGracefulFailoverFailBecomingStandby`
 
We create a PR that provides a fix by checking the timeout after multiplication is at least 0.
Bug: Improve s3a region handling, including determining from endpoint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/23 12:40
Affects Version/s: 3.4.0

S3A region logic improved for better inference and
to be compatible with previous releases

1. If you are using an AWS S3 AccessPoint, its region is determined
   from the ARN itself.
2. If fs.s3a.endpoint.region is set and non-empty, it is used.
3. If fs.s3a.endpoint is an s3.*.amazonaws.com url, 
   the region is determined by by parsing the URL 
   Note: vpce endpoints are not handled by this.
4. If fs.s3a.endpoint.region==null, and none could be determined
   from the endpoint, use us-east-2 as default.
5. If fs.s3a.endpoint.region=="" then it is handed off to
   The default AWS SDK resolution process.

Consult the AWS SDK documentation for the details on its resolution
process, knowing that it is complicated and may use environment variables,
entries in ~/.aws/config, IAM instance information within
EC2 deployments and possibly even JSON resources on the classpath.
Put differently: it is somewhat brittle across deployments.
Bug: Document missing a lot of properties in core-default.xml
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jul/23 10:07
Affects Version/s: 3.4.0

!image-2023-07-19-18-02-23-700.png|width=600,height=333!

As the picture shows, there are many configs in core-site.xml, which should be documented.
Bug: Some properties are missing from hadoop-policy.xml.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Aug/23 13:25
Affects Version/s: 3.4.0

 
{noformat}
security.datanode.lifeline.protocol.acl
security.get.user.mappings.protocol.acl
security.reconfiguration.protocol.acl
security.refresh.callqueue.protocol.acl
security.refresh.generic.protocol.acl{noformat}
The properties above are missing from hadoop-policy.xml.
Bug: Close child file systems in ViewFileSystem when cache is disabled.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/23 15:04
Affects Version/s: 3.3.6

When the cache is configured to disabled (namely, `fs.viewfs.enable.inner.cache=false` and `fs.*.impl.disable.cache=true`), even if `FileSystem.close()` is called, the client cannot truly close the child file systems in a ViewFileSystem. This caused our long-running clients to constantly produce resource leaks.
Bug: upgrade to commons-compress 1.24.0 due to CVE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/23 15:02
Affects Version/s: 3.4.0

Includes some important bug fixes including https://lists.apache.org/thread/g9lrsz8j9nrgltcoc7v6cpkopg07czc9 - CVE-2023-42503
Bug: upgrade snappy-java to 1.1.10.4 due to CVE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/23 16:53
Affects Version/s: 3.3.6

follow up to HADOOP-18782

https://github.com/xerial/snappy-java/security/advisories/GHSA-55g7-9cwv-5qfv
Bug: module-info classes from external dependencies appearing in uber jars
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/23 12:05
Affects Version/s: 3.3.6

hadoop-client-minicluster and hadoop-client-runtime try unsuccessfully to exclude module-info classes from dependencies. Over time, more and more jars have these classes.

The module-info classes are causing issue with CI builds. Builds can fail if there are more than module-inf class that is not excluded.

It seems better to exclude them all, especially since they will be affected by shading anyway.
Bug: Install bats for building Hadoop on Windows
Status: Open | Priority: Major | Resolution: 
Created: 30/Jul/23 07:19
Affects Version/s: 3.4.0

We get the following error while building Hadoop on Windows (logs attached -  [^archive.zip] ) -

{code}
[INFO] --- maven-antrun-plugin:1.8:run (common-test-bats-driver) @ hadoop-common ---
[INFO] Executing tasks

main:
     [exec] 
     [exec] 
     [exec] ERROR: bats not installed. Skipping bash tests.
     [exec] ERROR: Please install bats as soon as possible.
     [exec] 
{code}

We need to install bats to fix this.
Bug: Install strings utility for git bash on Windows
Status: Open | Priority: Major | Resolution: 
Created: 30/Jul/23 07:57
Affects Version/s: 3.4.0

We get the following error while building Hadoop on Windows 10 -

{code}
[2023-07-28T07:16:22.389Z] ============================================================================
[2023-07-28T07:16:22.389Z] ============================================================================
[2023-07-28T07:16:22.389Z]                          Determining needed tests
[2023-07-28T07:16:22.389Z] ============================================================================
[2023-07-28T07:16:22.389Z] ============================================================================
[2023-07-28T07:16:22.389Z] 
[2023-07-28T07:16:22.389Z] 
[2023-07-28T07:16:22.389Z] (Depending upon input size and number of plug-ins, this may take a while)
[2023-07-28T07:20:59.610Z] /c/out/precommit/plugins.d/maven.sh: line 275: strings: command not found
{code}

We need to install the strings utility for git bash on Windows to fix this.
Bug: Add rpcRejectedByObserverCalls metric  to quantify the number of rejected RPCs by Observer NameNode
Status: Open | Priority: Major | Resolution: 
Created: 08/Sep/23 11:23

When observer Node's serverStateId  is too far behind  clientStateId will rejected client RPC request,

mabe we need add rpcRejectedByObserverCalls metric to RpcMetrics to quantify the number of rejected RPCs by Observer NameNode to help determine how well the server.
Bug: Java 17 Runtime Support
Status: Open | Priority: Major | Resolution: 
Created: 11/Sep/23 21:26

This JIRA feature aims to extend the Java runtime support for Hadoop to include Java 17 in addition to Java 8 and Java 11. Currently Hadoop version 3.3.6 supports Java 8 compile and Java 8/11 runtime. The goal is to make Hadoop compatible with Java 17 runtime as well.

The plan for this release is to allow Hadoop to default to Java 11/Java 17, while still providing the flexibility for customers to configure Hadoop to use Java 8, Java 11, or Java 17 based on their specific needs. This project's objectives include:
 # Certifying that Hadoop works seamlessly on Java 8/11/17 for common use cases.
 # Ensuring that running Hadoop on Java 11/17 does not disrupt other applications and libraries such as Spark, Hive, Flink, Presto/Trino, and HBase.

The decision to support Java 17 runtime is motivated by customer requests and significant performance improvements observed in downstream applications like Apache Hive and Apache Spark. The testing process encompasses unit tests, integration tests, and performance tests, as well as verifying the proper functioning of all Hadoop daemons with Java 17.


The project will address compile time issues across various Hadoop components, ensuring that Hadoop remains compatible with Java 17 throughout the entire codebase.

This ticket serves as a vital step in enhancing Hadoop's capabilities, providing customers with more choices and improved performance for their big data processing needs.
Bug: hadoop distcp needs support to filter by file/directory attribute
Status: Open | Priority: Major | Resolution: 
Created: 13/Sep/23 06:16
Affects Version/s: 3.3.6

In some circumstances, we need to filter file/directory by file/directroy. For example, we need to filter out them by file modified time, isDir attrs, etc.

So, *should we introduce a new method  public boolean shouldCopy(CopyListingFileStatus fileStatus)* ? 
by this approach, we can introduce a more fluent way to do things than  public abstract boolean shouldCopy(Path path).
Bug: Recommended Docker config file missing environment variable
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/23 20:09
Affects Version/s: 3.3.6

Docker config is missing {*}HADOOP_HOME=/opt/hadoop{*}, docker environment referencing this variable and docker container cant run examples and behaves erratically

currently not set in environment and config uses it eg

MAPRED-SITE.XML_yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=$HADOOP_HOME

but its not set in docker container
Bug: S3ACachingInputStream.ensureCurrentBuffer(): lazy seek means all reads look like random IO
Status: Open | Priority: Major | Resolution: 
Created: 16/Aug/23 17:16
Affects Version/s: 3.3.6

noticed in HADOOP-18184, but I think it's a big enough issue to be dealt with separately.

# all seeks are lazy; no fetching is kicked off after an open
# the first read is treated as an out of order read, so cancels any active reads (don't think there are any) and then only asks for 1 block

{code}
    if (outOfOrderRead) {
      LOG.debug("lazy-seek({})", getOffsetStr(readPos));
      blockManager.cancelPrefetches();

      // We prefetch only 1 block immediately after a seek operation.
      prefetchCount = 1;
    }

{code}

* for any read fully we should prefetch all blocks in the range requested
* for other reads, we may want a bigger prefech count than 1, depending on: split start/end, file read policy (random, sequential, whole-file)
* also, if a read is in a block other than the current one, but which is already being fetched or cached, is this really an OOO read to the extent that outstanding fetches should be cancelled?
Bug: Upgrade guava due to CVE
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Aug/23 23:57
Affects Version/s: 3.3.6

Update guava to 32.0.1 or higher due to CVE: [https://nvd.nist.gov/vuln/detail/CVE-2023-2976]

hadoop-shaded-guava 1.1.1 is currently using 30.1.1-jre per security scanning tools