av9ash/GitBugs huggingface dataset:
Bug: Disable abfs prefetching by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Nov/22 13:30
Affects Version/s: 3.3.4

After the addition of HADOOP-18517, we should disable readAhead by default to mitigate inconsistent read results caused by ABFS prefetching, HADOOP-18521.

As an urgent fix: Disable readAhead/prefetch, tracked for 3.3.5.
Long-term fix: HADOOP-18521, tracked for 3.3.6.
Bug: assertion failure in ITestS3APrefetchingInputStream
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Nov/22 16:47
Affects Version/s: 3.4.0

assert failure in {{ITestS3APrefetchingInputStream.testReadLargeFileFullyLazySeek}}; looks like the executor was acquired faster than the test expected.

{code}
java.lang.AssertionError: 
[Maxiumum named action_executor_acquired.max] 
Expecting:
 <0L>
to be greater than:
 <0L> 

{code}

proposed: cut that assert as it doesn't seem needed
Bug: Implement token storage solution based on MySQL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/22 01:16
Affects Version/s: 3.3.6

Hadoop RBF supports custom implementations of secret managers. At the moment, the only available implementation is ZKDelegationTokenSecretManagerImpl, which stores tokens and delegation keys in Zookeeper.

During our investigation, we found that the performance of routers is limited by the writes to the Zookeeper token store, which impacts requests for token creation, renewal and cancellation. An alternative secret manager implementation has been created, based on MySQL, to handle a higher number of writes.

We measured the throughput of each token operation (create/renew/cancel) on different setups and obtained the following results:
 # Sending requests directly to Namenode (no RBF):
Token creations: 290 reqs per sec
Token renewals: 86 reqs per sec
Token cancellations: 97 reqs per sec

 # Sending requests to routers using Zookeeper based secret manager:
Token creations: 31 reqs per sec
Token renewals: 29 reqs per sec
Token cancellations: 40 reqs per sec
 # Sending requests to routers using SQL based secret manager:
Token creations: 241 reqs per sec
Token renewals: 103 reqs per sec
Token cancellations: 114 reqs per sec

We noticed a significant improvement when using a SQL secret manager, comparable to the throughput offered by Namenodes.
Bug: Upgrade Bouncy Castle to 1.70
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/22 17:27
Affects Version/s: 3.4.0

Upgrade Bouncycastle to 1.70 to resolve

 
|[[sonatype-2021-4916] CWE-327: Use of a Broken or Risky Cryptographic Algorithm|https://ossindex.sonatype.org/vulnerability/sonatype-2021-4916?component-type=maven&component-name=org.bouncycastle/bcprov-jdk15on]|

|[[sonatype-2019-0673] CWE-400: Uncontrolled Resource Consumption ('Resource Exhaustion')|https://ossindex.sonatype.org/vulnerability/sonatype-2019-0673?component-type=maven&component-name=org.bouncycastle/bcprov-jdk15on]|
Bug: Hadoop Archive tool (HAR) should acquire delegation tokens from source and destination file systems
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Dec/22 04:28
Affects Version/s: 3.4.0

Running Hadoop Archive tool (HAR) against source/dest that are not HDFS does not work today.

Example:
{noformat}
hadoop archive -archiveName foo.har -p ofs://ozone1/vol2/bucket2/src/ ofs://ozone1/vol2/bucket2/dst/
{noformat}
The command fails because it's essentially a MapReduce job and for some reason it does not obtain Ozone delegation token (only HDFS and KMS)

The failure:
{noformat}
22/11/30 18:12:10 INFO mapreduce.Job: Job job_1669745757520_0001 failed with state FAILED due to: Job setup failed : java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:789)
        at java.security.AccessController.doPrivileged(Native Method)
...
{noformat}

The workaround is to add the ozone path as the default file system:
{noformat}
export HADOOP_OPTS="$HADOOP_OPTS -Dfs.defaultFS=ofs://ozone1/vol2/bucket2"
{noformat}

A proper fix should make sure HAR requests delegation tokens from src/dest file system too.
Bug: LogThrottlingHelper: the dependent recorder is not triggered correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/22 06:46
Affects Version/s: 3.3.4

The current implementation of {{LogThrottlingHelper}} works well most of the time, but it missed out one case, which appears quite common in the production codes:
- if the dependent recorder was not suppressed before the primary one is triggered on the next period, then the next logging of the dependent recorder will be unexpectedly suppressed.

{code:java}
    helper = new LogThrottlingHelper(LOG_PERIOD, "foo", timer);

    assertTrue(helper.record("foo", 0).shouldLog());
    assertTrue(helper.record("bar", 0).shouldLog());

    // Both should log once the period has elapsed
    // <pos1>
    assertTrue(helper.record("foo", LOG_PERIOD).shouldLog());
    assertTrue(helper.record("bar", LOG_PERIOD).shouldLog()); <--- This assertion will now fail
{code}

Note if we call {{helper.record("bar", LOG_PERIOD * 2)}} in <pos1>, as the existing test cases do, it will work as expected.
Bug: AWS SDK V2 - Qualify the upgrade. 
Status: Resolved | Priority: Major | Resolution: Done
Created: 12/Dec/22 10:02
Affects Version/s: 3.4.0

Run tests as per [qualifying aws ask update|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md#-qualifying-an-aws-sdk-update]
Bug: Make XML transformer factory more lenient
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/22 18:21
Affects Version/s: 3.3.5

Issues raised today in HADOOP-18469 about particular attributes that are not supported by Saxon transformer.

 

note: there are two PRs here, the initial patch and a followup which improves performance on incompatible transformers. backports/cherrypicks should include both.
Bug: Java 11 JavaDoc fails due to missing package comments
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Dec/22 11:37
Affects Version/s: 3.4.0

yetus is failing javadoc builds with import issues on package-info.java files related to @InterfaceAudience/stability annotations

We need to change how they are imported to fix this. Other modules have the same issue

This is happening because java11 javadoc requires a doc comment for all packages with annotations.
Bug: ABFS: add probes of readahead fix
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Dec/22 17:04
Affects Version/s: 3.3.5

Followup patch as to  HADOOP-18456 to
aid in probing for read correctness of
hadoop ABFS client across different releases.

* ReadBufferManager constructor logs the fact it is safe at TRACE
* AbfsInputStream declares it is fixed in toString()
  by including fs.azure.capability.readahead.safe" in the
  result.

The ABFS FileSystem hasPathCapability("fs.azure.capability.readahead.safe")
probe returns true to indicate the client's readahead manager has been fixed
to be safe when prefetching.

All Hadoop releases for which probe this returns false
and for which the probe "fs.capability.etags.available"
returns true at risk of returning invalid data when reading
ADLS Gen2/Azure storage data.
Bug: Handle Server KDC re-login when Server and Client run in same JVM.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/22 10:01
Affects Version/s: 3.1.1

Handle re-login in Server when client, server running in same JVM and client trying to re-login, but it fails.

For example, NameNode is server but in same JVM journal node client also running to push to edit logs. When JN client try to re-login and it fails, it will destroy server service ticket also and NameNode not able to server client request. We can see the below error logs in NameNode log file.

 
{noformat}
Auth failed for x.x.x.x:42199:null (GSS initiate failed) with true cause: (GSS initiate failed)
Auth failed for x.x.x.x:42199:null (GSS initiate failed) with true cause: (GSS initiate failed)
Auth failed for x.x.x.x:42199:null (GSS initiate failed) with true cause: (GSS initiate failed){noformat}
Same discussion happened in HADOOP-17996.
Bug: No need to clean tmp files in distcp direct mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/22 04:33
Affects Version/s: 3.3.4

it not necessary to do `cleanupTempFiles`  while ditcp commit job in direct  mode, because it there is no temp files in direct mode.

This clean operation will increase the task execution time, because it will get the list of files in the target path. When the number of files in the target path is very large, this operation will be very slow.

*note* there are two patches which need to be cherrypicked when picking this up; the original patch and a followup, both with HADOOP-18582 in the title


{code}
3b7b79b37ae HADOOP-18582. skip unnecessary cleanup logic in distcp (#5251)
e8a6b2c2c4e HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.
{code}
Bug: [NFS GW] Fix regression after netty4 migration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/22 17:53
Affects Version/s: 3.4.0

Two bugs were found recently. Both related to hadoop portmap utility.

(1) The ChannelHandler that can be added to a ChannelPipeline in Netty4 must be Shareable. However, IdleStateHandler, used by Portmap is not. We need to fix it otherwise hadoop portmap may fail to start.

(2) RpcUdpResponseStage does not get the request from client (hadoop mountd) because of response type mismatch, and therefore it does not respond back to client request.

This is a regression from HADOOP-11245 but I consider it not a blocker, because most recent Linux distro supports rpcbind and there's little reason to use Hadoop's portmap.
Bug: update jackson-databind 2.12.7.1 due to CVE fixes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/22 08:17
Affects Version/s: 3.3.5

* [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-42003]
 * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-42004]
 * both fixes have been backported (the CVEs themselves need to be updated to reflect this)
 * [https://github.com/FasterXML/jackson-databind/pull/3622]
Bug: Azure RefreshTokenBasedTokenProvider is only supporting public client
Status: Open | Priority: Major | Resolution: 
Created: 27/Oct/22 08:07
Affects Version/s: 3.3.4

The Azure RefreshTokenBasedTokenProvider is assuming the client is public, meaning it's not exchanging the refresh token to an access token with the client secret.

 

This limitation is not really justify and the RefreshTokenBasedTokenProvider should use the client secret if present.

 

From my understanding, there is no particular reason to think that hadoop is not able to store secrets securely, especially as I see the client credential flow, which require a confidential client, is supported by the library.

 

The fix is to simply inject the client secret in the request, using client basic auth method or client Post auth method, when the client secret is present.

 

https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/RefreshTokenBasedTokenProvider.java#L61
Bug: Backport HADOOP-18427 and HADOOP-18452 to branch-3.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Nov/22 00:59
Affects Version/s: 3.3.5

This is a sub-task of HADOOP-18518 to upgrade zk&curator on 3.3 branches.

It is a clean cherry pick from [https://github.com/apache/hadoop/pull/4812] which solved the deprecation of EnsurePath in new Curator. Note, as this change contained a bug, fixed by [https://github.com/apache/hadoop/pull/4885] , we thus need to cherry pick this bug fix PR as well.
Bug: Bump Zookeeper to 3.6.3 and Curator to 5.2.0 in branch 3.3
Status: Open | Priority: Major | Resolution: 
Created: 03/Nov/22 00:39

This is the parent ticket. We plan to backport the upgrade from trunk to branch-3.3 in 3 PRs:
PR1(HADOOP-18515). Cherry pick [https://github.com/apache/hadoop/pull/3241] which simply bump the version.

PR2(HADOOP-18519). Cherry pick [https://github.com/apache/hadoop/pull/3266] which solved the deprecation of PathChildrenCache/TreeCache in new ZK. Note, we need to backport https://issues.apache.org/jira/browse/HDFS-15383 along because PR-3266 is based on this earlier change, specifically isTokenWatcherEnabled was introduced in [ZKDelegationTokenSecretManager.java|https://github.com/apache/hadoop/pull/2047/files#diff-f65a8ac81e253e85af159ba041fbad62fbb34b5bd909c1e9fc93d58222f406b9]

PR3(HADOOP-18520).Cherry pick [https://github.com/apache/hadoop/pull/4812] which solved the deprecation of EnsurePath in new Curator. Note, as this change contained a bug, fixed by [https://github.com/apache/hadoop/pull/4885] , we thus need to cherry pick this bug fix PR as well.
Bug: upgrade kotlin-stdlib due to CVEs
Status: Open | Priority: Major | Resolution: 
Created: 13/Oct/22 18:54

I'm not an expert on Kotlin but dependabot show these 2 CVEs with the version of kotlin-stdlib used in Hadoop.
 * [https://github.com/advisories/GHSA-cqj8-47ch-rvvq]
 * [https://github.com/advisories/GHSA-2qp4-g3q3-f92w]

kotlin-stlib 1.6.0 is the minimum version needed to fix both. It might be better to use latest v1.6 jar (currently 1.6.21) or even use latest jar altogether (currently 1.7.20).
Bug: Bump netty to the latest 4.1.86
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 16/Dec/22 07:19
Affects Version/s: 3.2.4

Netty 4.1.86 fixes the following vulnerabilities.
 * HAProxyMessageDecoder Stack Exhaustion DoS (CVE-2022-41881)
 * HTTP Response splitting from assigning header value iterator (CVE-2022-41915)

For more details: https://netty.io/news/2022/12/12/4-1-86-Final.html
Bug: AWS SDK V2 - Complete outstanding items
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/22 15:34
Affects Version/s: 3.4.0

The following work remains to complete the SDK upgrade work:
 * S3A allows users configure to custom signers, add in support for this.
 * Remove SDK V1 bundle dependency
 * Update `getRegion()` logic to use retries. 
 * Add in progress listeners for `S3ABlockOutputStream`
 * Fix any failing tests.
Bug: AWS SDK V2 - Fix failing tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/22 10:21
Affects Version/s: 3.4.0

We have a few failing tests for various reasons. Some are dependent on the TM, but others can be looked into and fixed. 


|TestS3AExceptionTranslation|test301ContainsEndpoint|Missing endpoint in SDK exception ([aws/aws-sdk-java-v2#3048|https://github.com/aws/aws-sdk-java-v2/issues/3048])|
|TestStreamChangeTracker|testCopyETagRequired, testCopyVersionIdRequired|Transfer Manager response does not yet have {{CopyObjectResult}}|
|ITestS3AFileContextStatistics|testStatistics|ProgressListeners not attached to non-TM uploads|
|ITestS3AEncryptionSSEC|multiple tests (14 out of 24)|Transfer Manager issue with SSE-C|
|ITestXAttrCost|testXAttrRoot.|{{headObject()}} with empty key fails|
|ITestSessionDelegationInFileystem|testDelegatedFileSystem|Succeeds, but {{headObject()}} with empty key commented out|
|ITestS3ACannedACLs|testCreatedObjectsHaveACLs|AWSCannedACL.LogDeliveryWrite not supported in SDK v2|
Bug: AWS SDK V2 - Update region logic
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/22 10:01
Affects Version/s: 3.4.0

SDK V2 will no longer resolve a buckets region if it is not set when initialising the client. 

 

Current logic will always make a head bucket call on FS initialisation. We should review this. Possible solution:
 * Warn if region is not set.
 * If no region, try and resolve. If resolution fails, throw an exception. Cache the region to optimise for short lived FS.
Bug: Upgrade grizzly version to 2.4.4
Status: Open | Priority: Major | Resolution: 
Created: 27/Nov/22 19:07
Affects Version/s: 3.4.0

Upgrade grizzly version to 2.4.4 to resolve
|[[sonatype-2016-0415] CWE-79: Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')|https://ossindex.sonatype.org/vulnerability/sonatype-2016-0415?component-type=maven&component-name=org.glassfish.grizzly/grizzly-http-server]|

[CVE-2014-0099|https://nvd.nist.gov/vuln/detail/CVE-2014-0099], [CVE-2014-0075|https://nvd.nist.gov/vuln/detail/CVE-2014-0075], [CVE-2017-1000028|https://nvd.nist.gov/vuln/detail/CVE-2017-1000028]
Bug: VectorIO FileRange type to support a "reference" field
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Oct/22 17:00
Affects Version/s: 3.3.5

to use in libraries, it is really good to be able to connect a FileRange back to the application/library level structure (chunk/split data, usually). 

Proposed: add an {{Object reference)) field which can be given arbitrary data or null, and queried for by app. it is not used in the API at all
Bug: AliyunOSS: AliyunOSSFileSystem#open(Path path, int bufferSize) should use buffer size as its downloadPartSize
Status: Open | Priority: Major | Resolution: 
Created: 28/Nov/22 13:57

In our application, different components have their own suitable buffer size to download.

But currently, AliyunOSSFileSystem#open(Path path, int bufferSize) just get downloadPartSize from configuration.

We cannnot use different value for different components in our programs.

I think we should the method should use the buffer size from the paramater.

AliyunOSSFileSystem#open(Path path) could have default value as current default downloadPartSize.
Bug: KeyVault CrednetialsProvider
Status: Open | Priority: Major | Resolution: 
Created: 11/Nov/22 15:05

Azure KeyVault is a secure key management tool that keys, secrets , etc could securely store in it. Also azure KeyVault is a secure place to store storage account access key.

By providing a new CredentialsProvider, it's possible to easily fetch access key from KeyVault and this would helps a lot of developer especially in the local environment
Bug: S3A create(Path f, FsPermission permission)  need create target middle directories?
Status: Resolved | Priority: Major | Resolution: Information Provided
Created: 25/Nov/22 08:22

I see the code of s3a create(Path f, etc) which does not create target middle directories,

So whether exist this one case in which I create only one file  "/a/b/c.txt", It can get "b" as a directory by list "/a/", but If we delete this key "/a/b/c.txt",  "a" and "b" directories are all gone disappear.  

In the end, I want to ask two questions

1) Whether HCFS interface create(Path f, etc) need to create target middle directories inside?(sorry did not find doc to say about this)

2) If not create target middle directories inside, Is the object storage(s3a) scenario described above plausible?
Bug: Backport HDFS-15383 and HADOOP-17835 to branch-3.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Nov/22 00:55
Affects Version/s: 3.3.5

This is a sub-task of HADOOP-18518 to upgrade zk&curator on 3.3 branches.

It contains clean cherry pick from [https://github.com/apache/hadoop/pull/3266] which solved the deprecation of PathChildrenCache/TreeCache in new ZK. 
Also clean cherry pick from https://issues.apache.org/jira/browse/HDFS-15383 because PR-3266 is based on this earlier change, specifically isTokenWatcherEnabled was introduced in [ZKDelegationTokenSecretManager.java|https://github.com/apache/hadoop/pull/2047/files#diff-f65a8ac81e253e85af159ba041fbad62fbb34b5bd909c1e9fc93d58222f406b9]
Bug: Backport HADOOP-17612 to branch-3.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Nov/22 00:55
Affects Version/s: 3.3.5

This is a sub-task of HADOOP-18518 to upgrade zk&curator on 3.3 branches. 

It is a clean cherry pick from [https://github.com/apache/hadoop/pull/3241] .
Bug: ABFS: Support client-side encryption
Status: Open | Priority: Major | Resolution: 
Created: 28/Oct/22 13:15
Affects Version/s: 3.3.4

ABFS currently supports [customer-provided keys|https://azure.microsoft.com/en-us/blog/customer-provided-keys-with-azure-storage-service-encryption/], but does not support [client-side encryption|https://learn.microsoft.com/en-us/azure/storage/blobs/client-side-encryption?tabs=dotnet]. With client-side encryption, the encryption key never leaves the client.
Bug: ArrayIndexOutOfBoundsException in TestCoderBase.java
Status: Open | Priority: Major | Resolution: 
Created: 22/Oct/22 10:54
Affects Version/s: 3.3.4

The code throws an unhandled ArrayIndexOutOfBoundsException in the method _backupAndEraseChunks_ of _TestCoderBase_ class when it tries to access the out of bound indexes in _dataChunks_ and _parityChunks_ array.
Bug: The check logic for erasedIndexes in XORRawDecoder is buggy
Status: Open | Priority: Major | Resolution: 
Created: 11/Oct/22 11:59
Affects Version/s: 3.3.4

In the method _doDecode_ of class {_}XORRawDecoder{_}, the code does not handle all the erased and null marked locations in the array ({_}inputs{_}) but only skips the first erased location ({_}erasedIndexes[0]{_}). The missing handling results in an unhandled NullPointerException.
Bug: upgrade commons-text to 1.10.0
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Oct/22 14:31

Extends HADOOP-18341

[https://commons.apache.org/proper/commons-text/changes-report.html#a1.10.0]

StringInterpolator prior to v1.10.0 allowed scripting that could be problematic – a similar issue to one that led to CVE in commons-configuration2.
Bug: AWS v2 SDK upgrade log to not warn of use standard AWS Credential Providers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/22 14:10
Affects Version/s: 3.4.0


looking at test output with the sdk warnings enabled, it is now always warning of a v1 provider reference, even if the user hasn't set any fs.s3a.credential.provider option


{code}
2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
2022-10-05 14:09:09,734 [setup] WARN  s3a.SDKV2Upgrade (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK V1 credential providers will be removed once S3A is upgraded to SDK V2
2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider

{code}

This is because the EnvironmentVariableCredentialsProvider provider is on the default list of providers.

Everybody who is using the S3 a connector and who has not explicitly declared a new set of providers excluding this one will be seeing the error message.

Proposed:

Don't warn on this provider. Instead with the v2 move the classname can be patched to switch to a modified one.

The alternative would be to provide an s3a specific env var provider subclass of this; and while that is potentially good in future it is a bit more effort for the forthcoming 3.3.5 release.
And especially because and it will not be in previous versions people cannot explicitly switch to it in their configs and be confident it will always be there,