av9ash/GitBugs huggingface dataset:
Bug: no valid user IDs when importing KEYS in gpg
Status: Open | Priority: Major | Resolution: 
Created: 08/Dec/23 10:29

h1. What I did

# Download the KEYS file: https://dlcdn.apache.org/hadoop/common/KEYS
# Import the KEYS file:
{{gpg --import KEYS}}

h1. What happened

There is the error while importing the KEYS file:

{{gpg: key B56E5591: no valid user IDs}}

h1. What is expected

No errors when importing the KEYS file

h1. Observation

The import definitely worked on October 18 2023, but now it doesn't

h1. Environment

h2. Operating System

{{$ cat /etc/os-release}}
{{NAME="Amazon Linux"}}
{{VERSION="2"}}
{{ID="amzn"}}
{{ID_LIKE="centos rhel fedora"}}
{{VERSION_ID="2"}}
{{PRETTY_NAME="Amazon Linux 2"}}
{{ANSI_COLOR="0;33"}}
{{CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"}}

h2. GPG Version

{{$ gpg --version}}
{{gpg (GnuPG) 2.0.22}}
{{libgcrypt 1.5.3}}
{{Copyright (C) 2013 Free Software Foundation, Inc.}}
{{License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>}}
{{This is free software: you are free to change and redistribute it.}}
{{There is NO WARRANTY, to the extent permitted by law.}}

{{Home: ~/.gnupg}}
{{Supported algorithms:}}
{{Pubkey: RSA, ?, ?, ELG, DSA}}
{{Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,}}
{{        CAMELLIA128, CAMELLIA192, CAMELLIA256}}
{{Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224}}
{{Compression: Uncompressed, ZIP, ZLIB, BZIP2}}

h2. SHA256 checksum of the KEYS file

{{$ sha256sum KEYS}}
{{807ad6cc5e1fdedf17884a8da9389549e72b9ad5b8e6fc5050382619c7a19ec6  KEYS}}
Bug: S3A: debug logging for http traffic to S3 stores
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 01/Dec/23 14:32
Affects Version/s: 3.4.0

AWS SDK bundle.jar logging doesn't set up right.

{code}
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
{code}

Cloudstore commands have a -debug option to force set this through log4j APIs; this does work. 

Proposed:

* add reflection-based ability to set/query log4j log levels (+tests, obviously)
* add a new log `org.apache.hadoop.fs.s3a.logging.sdk`
* if set to DEBUG, DefaultS3ClientFactory will enable logging on the aws internal/shaded classes

this allows log4j.properties to turn on logging; reflection ensures all is well on other log back-ends and when unshaded aws sdk jars are in use
Bug: Use thread pool to improve the speed of creating control files in TestDFSIO
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/23 06:33
Affects Version/s: 3.3.6

When we use TestDFSIO tool to test the throughouts of HDFS clusters, we found it is so slow in the creating controll files stage. 

After refering to the source code, we found that method createControlFile try to create control files serially. It can be improved by using thread pool.

After optimizing, the TestDFSIO tool runs quicker than before.
Bug: S3A FileSystem does not correctly list empty directories after HADOOP-17199
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 10/Nov/23 08:38
Affects Version/s: 3.2.4

After HADOOP-17199, s3a filesystem dose not  correctly ls empty directories.   For example. 

Before HADOOP-17199

```

hdfs dfs -ls s3a://testbucket/dir/empty

 

// work right

```

 

After HADOOP-17199

```

hdfs dfs -ls s3a://testbucket/dir/empty

ls: `s3a://testbucket/dir/empty': No such file or directory

 

// wrong ouput

```

 

After Check the code, s3a empty folder don't have prefixes Or objects, it whil throw FIleNotFouldException.

!image-2023-11-10-16-36-53-337.png!
Bug: Support OpenSSL 3
Status: Open | Priority: Major | Resolution: 
Created: 28/Nov/23 08:31
Affects Version/s: 3.3.6

Hadoop uses native OpenSSL library provided by linux distribution.

OpenSSL 1.1.1 is deprecated: https://www.openssl.org/blog/blog/2023/03/28/1.1.1-EOL/ 

We need to ensure that Hadoop works with OpenSSL 3.
Bug: Snapshotdiff does not detect rename if parent dir removed
Status: Open | Priority: Major | Resolution: 
Created: 23/Nov/23 12:39
Affects Version/s: 3.3.6

In the example case below, the diff reports the moved folder as a CREATE, where it should be a RENAME.

This means that applications using the diff report (such as DistCP) will have to transfer data where a simple move would suffice.
{code:java}
DistributedFileSystem fs = (DistributedFileSystem) this.fs;

createHdfsFile(new Path("/level1/level2/file"), "content");
fs.allowSnapshot(new Path("/"));
fs.createSnapshot(new Path("/"), "snapshot1");

fs.rename(new Path("/level1/level2/"), new Path("/level2-root"));
fs.delete(new Path("/level1"), true);
fs.createSnapshot(new Path("/"), "snapshot2");

SnapshotDiffReport snapshotDiff =
        fs.getSnapshotDiffReport(new Path("/"), "snapshot1", "snapshot2");
System.out.println(snapshotDiff);
//        Difference between snapshot snapshot1 and snapshot snapshot2 under directory /:
//        M .
//        + ./level2-root
//        - ./level1

// If the delete is commented out:
//        M .
//        M ./level1
//        R ./level1/level2 -> ./level2-root{code}
Bug: Create curator client the same way possibly with the same code for all services
Status: Open | Priority: Major | Resolution: 
Created: 17/Nov/23 11:33

Currently ZKSignerSecretProvider and ZKDelegationTokenSecretManager classes are using the same ZookeeperClient class to create their respective Zookeeper connection via a Curator client.
In order to have a standardized way of creating Zookeeper connections, we have two areas to fix:
- YARN and its Curator instantiation
- HDFS ZKFC, which currently uses a Zookeeper client directly, so it should also be migrated to Curator if we want to utilize the common instantiation.

The problem that we would solve is that currently we have 3 places where we create ZK connections, and all of them has its own things, along with some duplications.
We also have the HadoopZookeeperFactory, that I think we do not need. The only thing that is different for this is how we set up SASL permission provider for the connection. I believe that can also be unified, with that the whole Zookeeper connection creation can be pushed down to the recently added code that handles this for ZKDelegationTokenSecretManager and ZKSignerSecretProvider.
Bug: Enable service specific keystores and truststores for ZK SSL setup
Status: Open | Priority: Major | Resolution: 
Created: 16/Nov/23 22:48

Currently we have the common config properties hadoop.zk.(key|trust)store.(location|password) configuration options.
In HADOOP-18956 a ZKDelegationTokenSecretManager specific option was provided for these configurations, so with that ZKDelegationTokenSecretManager's ZK access can be set in a centralized fashion along with enabling it within ResourceManager, and DFSZKFailoverController.
On the other hand with DTSecretMgr we introduce specific options to be able to specify a separate keystore and truststore to be used.

A good improvement would be to add the truststore/keystore related options to all the components, so that even if the common hadoop.zk.* properties are set, and SSL is enabled, an individual component can have its own separate keystore and truststore set via specific configs, however if there are no specific config specified it can fall back to the common config values.
Bug: Replace commons-lang imports with commons-lang3
Status: Open | Priority: Major | Resolution: 
Created: 15/Nov/23 07:58

we have a direct dependency on commons-lang3 but still we are using commons-lang at a couple of places, which is a transitive dependency coming from hbase, solr, and couple of other places.

 

good to replace those with lang3, rather than relying on older transitive dependency
Bug: FsCommand Stat class set the timeZone"UTC", which is different from the machine's timeZone
Status: Open | Priority: Major | Resolution: 
Created: 26/Oct/23 02:02

Using Hadoop version 3.3.4

 

When executing Ls command and Stat command on the same hadoop file, I get two timestamps.

 
{code:java}
hdfs dfs -stat "modify_time %y, access_time%x" /path/to/file{code}
 returns:

modify_time {_}*2023-10-17 01:43:05*{_}, access_time _*2023-10-17 01:41:00*_ 

 
{code:java}
hdfs dfs -ls /path/to/file{code}
  returns:

{-}rw{-}rw-r–+     3    user_name     user_group     247400339     _*2023-10-17 09:43*_     /path/to/file

 

these two timestamps has the difference 8hours.

I am in China, the timezone is “UTC+8”， so the timestamp from LS command is correct and timestamp from STAT command is wrong.

 

!image-2023-10-26-10-07-11-637.png!
Bug: S3AFileSystem URL encodes twice where Path has trailing /
Status: Open | Priority: Major | Resolution: 
Created: 10/Oct/23 15:04

As per the comment at [https://github.com/apache/hadoop/pull/1646#issuecomment-1754221384] ...

due to HADOOP-15430 a Path
{code:java}
s3://my.bucket/my folder/{code}
becomes
{code:java}
s3://my.bucket/my%20folder{code}
which upon HTTP request creation becomes
{code:java}
"GET
...&prefix=my%2520folder{code}
which is a wrong path.
Bug: s3a file rename does double HEAD or LIST on source file/dir
Status: Open | Priority: Minor | Resolution: 
Created: 09/Nov/23 14:51
Affects Version/s: 3.3.6

going to add this as a v2 issue, though really it's been latent.

when you do a rename() we do a getFileStatus call to get the list/file status..if this is a dir that's a single LIST; a file then it'll be LIST + HEAD.

when the actual copy is kicked off, 
* if the source is a dir, then a duplicate LIST is initiated straight afterwards.
* FIle: there's a second HEAD to get the metadata for the copy

Proposed.
* LIST initiates a full list, and if it returns a list of objects, that list iterator is passed in to rename.
* full result of HEAD preserved and passed down to copyFile()

will cut out one round trip regardless of source type
Bug: S3A Assume role tests failing against S3Express stores
Status: Open | Priority: Minor | Resolution: 
Created: 05/Dec/23 14:45
Affects Version/s: 3.4.0

The test suits which assume roles with restricted permissions down paths still fail on S3Express, even after disabling createSession.

This is with a role which *should* work.

Either the role setup is wrong, or there's something special about role configuration for S3Express buckets
Bug: Unknown S3Express bucket raises UnknownHostException rather than NoSuchBucketException; will block for retries
Status: Open | Priority: Minor | Resolution: 
Created: 01/Dec/23 15:24
Affects Version/s: 3.4.0

When an attempt is made to work with an s3 express bucket which isn't there. the createSession API fails with UnknownHostException

This is actually retried within the sdk, so we are lucky that HADOOP-18889 cut the retry count down. Even so, failures are slow and not very informative.
Bug: S3A third party: document "Certificate doesn't match"
Status: Open | Priority: Minor | Resolution: 
Created: 25/Oct/23 16:57
Affects Version/s: 3.3.6


A recurrent problems with third party stores is that the user gets an error message about HTTP certificates
{code}
Unable to execute HTTP request: Certificate for <mybucket.mystore.dev.net> doesn't match any of the subject alternative names: [*.dev.net]

{code}

This is happening because
# the store uses HTTPS and there is an organization certificate
# the store does support virtual hostname access -but it does not match the HTTPS wildcard

Fix: switch to path style access

*add this detail to the third party store doc*
Bug: ABFS contract-tests with Hadoop-Commons intermittently failing
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Oct/23 08:13

In the merged pr [HADOOP-18869: [ABFS] Fixing Behavior of a File System APIs on root path by anujmodi2021 · Pull Request #6003 · apache/hadoop (github.com)|https://github.com/apache/hadoop/pull/6003], a config was switched-on: `fs.contract.test.root-tests-enabled`. This enables the root manipulation tests for the filesystem contract.

Now, the execution of contract-tests in abfs works as per executionId integration-test-abfs-parallel-classes of the pom. The tests would work in different jvms, and at a given instance multiple such jvms could be there, depending on ${testsThreadCount}.  The problem is that all the test jvms for contract-test use the same container for test runs which is defined by `fs.contract.test.fs.abfs`. Due to this, one jvm root-contract-runs can influence other jvm's root-contract-runs. This leads to CI failures for hadoop-azure package.

Solution is to run these tests sequentially and separate from other commit/manifest tests.
Bug: FileSystem.getFileSystemClass() to log at debug the jar the .class came from
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Oct/23 15:43
Affects Version/s: 3.3.6

we want to be able to log the jar the filesystem implementation class, so that we can identify which version of a module the class came from.

this is to help track down problems where different machines in the cluster or the .tar.gz bundle is out of date.
Bug:  Improve UserGroupInformation debug log
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Oct/23 01:50
Affects Version/s: 3.3.0

      Using “new Exception( )” to print the call stack of "doAs Method " in the UserGroupInformation class. Using this way will print meaningless Exception information and too many call stacks, This is not conducive to troubleshooting

*example:*

!20231029-122825.jpeg|width=991,height=548!

 

*improved result* :

 

!image-2023-10-29-09-47-56-489.png|width=1099,height=156!

!20231030-143525.jpeg|width=572,height=674!
Bug: S3A credential provider remapping: make extensible
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Nov/23 14:40
Affects Version/s: 3.4.0

A new option fs.s3a.aws.credentials.provider.mapping takes a key value pair for automatic mapping of v1 credential providers to v2 credential providers.


h2. Backporting

There's a followup PR to the main patch which *should* be applied, as it hardens the parser.

{code}
HADOOP-18980. Invalid inputs for getTrimmedStringCollectionSplitByEquals (ADDENDUM) (#6546)
{code}
Bug: Corrections to Hadoop FileSystem API Definition
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Nov/23 12:18
Affects Version/s: 3.3.6

I noticed a lot of inconsistencies, typos and informal statements in the "formal" FileSystem API definition ([https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/index.html)]

Creating this ticket to link my PR against.
Bug: S3A: Add option fs.s3a.classloader.isolation (#6301)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Nov/23 21:56
Affects Version/s: 3.3.6

In HADOOP-17372 the S3AFileSystem forces the configuration classloader to be the same as the one that loaded S3AFileSystem. This leads to the impossibility in Spark applications to load third party credentials providers as user jars.


The option fs.s3a.classloader.isolation (default: true) can be set to false to disable s3a classloader isolation;

This can assist in using custom credential providers and other extension points.
Bug: S3A. Add option fs.s3a.directory.operations.purge.uploads to purge on rename/delete
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Oct/23 20:50
Affects Version/s: 3.4.0

On third-party stores without lifecycle rules its possible to accrue many GB of pending multipart uploads, including from
* magic committer jobs where spark driver/MR AM failed before commit/abort
* distcp jobs which timeout and get aborted
* any client code writing datasets which are interrupted before close.

Although there's a purge pending uploads option, that's dangerous because if any fs is instantiated with it, it can destroy in-flight work

otherwise, the "hadoop s3guard uploads" command does work but needs scheduling/manual execution

proposed: add a new property {{fs.s3a.directory.operations.purge.uploads}} which will automatically cancel all pending uploads under a path
* delete: everything under the dir
* rename: all under the source dir

This will be done in parallel to the normal operation, but no attempt to post abortMultipartUploads in different threads. The assumption here is that this is rare. And it'll be off by default as in AWS people should have rules for these things.


+ doc (third_party?)
+ add new counter/metric for abort operations, count and duration
+ test to include cost assertions
Bug: ITestS3GuardTool fails if SSE/DSSE encryption is used
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Oct/23 06:45
Affects Version/s: 3.3.6

{code:java}
[ERROR] Tests run: 15, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 25.989 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool
[ERROR] testLandsatBucketRequireUnencrypted(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool)  Time elapsed: 0.807 s  <<< ERROR!
46: Bucket s3a://landsat-pds: required encryption is none but actual encryption is DSSE-KMS
    at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.exitException(S3GuardTool.java:915)
    at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.badState(S3GuardTool.java:881)
    at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:511)
    at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:283)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
    at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:963)
    at org.apache.hadoop.fs.s3a.s3guard.S3GuardToolTestHelper.runS3GuardCommand(S3GuardToolTestHelper.java:147)
    at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.run(AbstractS3GuardToolTestBase.java:114)
    at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool.testLandsatBucketRequireUnencrypted(ITestS3GuardTool.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:750)
 {code}
Since landsat requires none encryption, the test should be skipped for any encryption algorithm.
Bug: Switch to SPDX identifier for license name
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Oct/23 12:32
Affects Version/s: 3.3.7

[https://maven.apache.org/pom.html#Licenses]
"Using an [SPDX identifier|https://spdx.org/licenses/] as the license name is recommended."

The Apache parent pom is already using this identifier
Bug: S3A: testMultiObjectExceptionFilledIn() assertion error
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Oct/23 15:36
Affects Version/s: 3.4.0

Failure in the new test of HADOOP-18939.

I've been fiddling with the sdk upgrade, and only merged HADOOP-18932 after submitting the new pr, so maybe, just maybe, the SDK changed some defaults.

anyway, 

{code}
[ERROR] testMultiObjectExceptionFilledIn(org.apache.hadoop.fs.s3a.impl.TestErrorTranslation)  Time elapsed: 0.026 s  <<< FAILURE!
java.lang.AssertionError: retry policy of MultiObjectException
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at 
{code}

easily fixed
Bug: S3A:  AbstractS3ACostTest to clear bucket fs.s3a.create.performance flag
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Nov/23 17:07
Affects Version/s: 3.4.0

If there's a bucket-specific  fs.s3a.create.performance flag then the create tests can fail as the costs are lower than expected. 

trivial fix: add to the removeBaseAndBucketOverrides list
Bug: S3A: Add option fs.s3a.s3express.create.session to enable/disable CreateSession
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Nov/23 17:03
Affects Version/s: 3.4.0

add a way to disable the need to use the createsession call, so as to allow for

* simplifying our role test runs
* benchmarking the performance hit
* troubleshooting IAM permissions



this can also be disabled from the sysprop "aws.disableS3ExpressAuth"
Bug: Add documentation related to NodeFencer
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Oct/23 12:24
Affects Version/s: 3.3.4

In the NodeFencer file, some important comments are missing.
Happens here:
 !screenshot-1.png! 

The guidance instructions for ShellCommandFencer are missing here.
If it is improved, the robustness of the distributed system can be increased.
Bug: AWS SDK v2: add path capability probe "fs.s3a.capability.aws.v2"
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Oct/23 10:13
Affects Version/s: 3.4.0

Add a "hasPathCapability()" probe for s3a v2 builds to aid diagnostics -avoids needing to look for specific s3a files on classpath. 

plus bucket-info to enum all capabilities which may be present
Bug: Allow no-downtime migration of HDFS clusters into secure mode
Status: Patch Available | Priority: Minor | Resolution: 
Created: 09/Nov/23 20:13

My employer (HubSpot) recently completed transitioning all of the Hadoop clusters underlying our HBase databases into secure mode. It was important to us that we be able to make this change without impacting the functionality of our SaaS product. To accomplish this, we added some new settings to our fork of Hadoop, and fixed a latent bug (HADOOP-18972). This ticket is my intention to contribute these changes back to the mainline code, so others can benefit. A patch will be incoming.

It was only necessary to change the HDFS code, because other Hadoop components are already able to seamlessly switch into secure mode.

The basic theme of the new functionality is the ability to accept incoming secure connections without requiring them or making them outgoing. Secure mode enablement will then be done in two stages.
 * First, all nodes are given configuration to accept secure connections, and are gracefully rolling-restarted to adopt this new functionality. I'll be adding the new settings to make this stage possible.
 * Second, all nodes are told to require incoming connections be secure, and to make secure outgoing connections, and the settings added in the first stage are removed. Nodes are again rolling-restarted to adopt this functionality. The settings in this final state will look the same as in any secure Hadoop cluster today.

I'll include documentation changes explaining how to do this.