av9ash/GitBugs huggingface dataset:
Bug: get local file system fails with a casting error
Status: Open | Priority: Trivial | Resolution: 
Created: 14/Sep/23 18:36
Affects Version/s: 3.3.6

h2. What happened

After setting {{{}fs.file.impl=org.apache.hadoop.fs.RawLocalFileSystem{}}}, trying to acquire local file system using the {{getLocal}} in {{org.apache.hadoop.fs}} fails with {{java.lang.ClassCastException}}
h2. Where's the bug

In the function {{getLocal}} of {{FileSystem}} in HCommon:
{code:java}
  public static LocalFileSystem getLocal(Configuration conf)
    throws IOException {
    return (LocalFileSystem)get(LocalFileSystem.NAME, conf);
  } {code}
the returned file system is directly cast to LocalFileSystem without checking. If the user set the implementation of the local filesystem to be Raw rather than Checksum, this type cast would fail.
h2. How to reproduce
 # Set {{fs.file.impl=org.apache.hadoop.fs.RawLocalFileSystem}}
 # Run the following test in HBase: {{org.apache.hadoop.hbase.TestHBaseTestingUtility#testMiniDFSCluster}}
and the following exception should be observed:

{code:java}
java.lang.ClassCastException: class org.apache.hadoop.fs.RawLocalFileSystem cannot be cast to class org.apache.hadoop.fs.LocalFileSystem (org.apache.hadoop.fs.RawLocalFileSystem and org.apache.hadoop.fs.LocalFileSystem are in unnamed module of loader 'app')
	at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:441)
	at org.apache.hadoop.hbase.HBaseTestingUtility.getNewDataTestDirOnTestFS(HBaseTestingUtility.java:550)
...{code}
Or simply set the configuration parameter and call the method using a {{Configuration}} object and the exception would be triggered.
Bug: S3A committer NPE in spark job abort
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 11/Aug/22 17:31
Affects Version/s: 3.3.5

NPE happening in spark {{HadoopMapReduceCommitProtocol.abortJob}} when jobID is null


{code}
- save()/findClass() - non-partitioned table - Overwrite *** FAILED ***
  java.lang.NullPointerException:
  at org.apache.hadoop.fs.s3a.commit.impl.CommitContext.<init>(CommitContext.java:159)
  at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.createCommitContext(CommitOperations.java:652)
  at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.initiateJobOperation(AbstractS3ACommitter.java:856)
  at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.abortJob(AbstractS3ACommitter.java:909)
  at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.abortJob(HadoopMapReduceCommitProtocol.scala:252)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:268)
  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:191)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
  ...

{code}
Bug: S3AInputStream.unbuffer() async drain not releasing http connections
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 19/Aug/22 10:00
Affects Version/s: 3.3.5

Impala tcp-ds setup to s3 is hitting problems with timeout fetching http connections from the s3a fs pool. Disabling s3a async drain makes this problem *go away*. assumption, either those async ops are blocking, or they are not releasing references properly.
Bug: NullPointerException in ObjectListingIterator's constructor
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 19/Sep/22 02:31
Affects Version/s: 3.3.5

We saw NullPointerExceptions in Impala's S3 tests: IMPALA-11592. It's thrown from the hadoop jar:
{noformat}
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.<init>(Listing.java:621)
        at org.apache.hadoop.fs.s3a.Listing.createObjectListingIterator(Listing.java:163)
        at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:144)
        at org.apache.hadoop.fs.s3a.Listing.getListFilesAssumingDir(Listing.java:212)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListFiles(S3AFileSystem.java:4790)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listFiles$37(S3AFileSystem.java:4732)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2363)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2382)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.listFiles(S3AFileSystem.java:4731)
        at org.apache.impala.common.FileSystemUtil.listFiles(FileSystemUtil.java:754)
        ... {noformat}
We are using a private build of the hadoop jar. Version: CDP 3.1.1.7.2.16.0-164
Code snipper of where the NPE throws:
{code:java}
604     @Retries.RetryRaw
605     ObjectListingIterator(
606         Path listPath,
607         S3ListRequest request,
608         AuditSpan span) throws IOException {
609       this.listPath = listPath;
610       this.maxKeys = listingOperationCallbacks.getMaxKeys();
611       this.request = request;
612       this.objectsPrev = null;
613       this.iostats = iostatisticsStore()
614           .withDurationTracking(OBJECT_LIST_REQUEST)
615           .withDurationTracking(OBJECT_CONTINUE_LIST_REQUEST)
616           .build();
617       this.span = span;
618       this.s3ListResultFuture = listingOperationCallbacks
619           .listObjectsAsync(request, iostats, span);
620       this.aggregator = IOStatisticsContext.getCurrentIOStatisticsContext()
621           .getAggregator();           // <---- thrown here
622     }
{code}
Bug: Upgrade Maven Surefire plugin to 3.0.0-M7
Status: Patch Available | Priority: Blocker | Resolution: 
Created: 23/Aug/22 19:06
Affects Version/s: 3.3.5

The Maven Surefire plugin 3.0.0-M1 doesn't always include the launcher as part of it's setup, which can cause problems with Yarn tests. Some of the Yarn modules use Jupiter, which may be a complicating factor.  Switching to 3.0.0-M7 fixes the issue.

This is currently blocking MAPREDUCE-7386
Bug: s3a FS init logs at warn if fs.s3a.create.storage.class is unset
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 27/Jul/22 11:00
Affects Version/s: 3.3.5

if you don't have an s3a storage class set in {{fs.s3a.create.storage.class}}, then whenever you create an S3A FS instance, it logs at warn

{code}

bin/hadoop s3guard bucket-info $BUCKET

2022-07-27 11:53:11,239 [main] INFO  Configuration.deprecation (Configuration.java:logDeprecation(1459)) - fs.s3a.server-side-encryption.key is deprecated. Instead, use fs.s3a.encryption.key
2022-07-27 11:53:11,240 [main] INFO  Configuration.deprecation (Configuration.java:logDeprecation(1459)) - fs.s3a.server-side-encryption-algorithm is deprecated. Instead, use fs.s3a.encryption.algorithm
2022-07-27 11:53:11,396 [main] WARN  s3a.S3AFileSystem (S3AFileSystem.java:createRequestFactory(1004)) - Unknown storage class property fs.s3a.create.storage.class: ; falling back to default storage class
2022-07-27 11:53:11,839 [main] INFO  impl.DirectoryPolicyImpl (DirectoryPolicyImpl.java:getDirectoryPolicy(189)) - Directory markers will be kept
Filesystem s3a://stevel-london
Location: eu-west-2


{code}

note, this is why part of quaifying an sdk update involves looking at the logs and running the CLI commands by hand...you see if new messages have crept in
Bug:  Fix file split duplicating records from a succeeding split when reading BZip2 text files 
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 11/Aug/22 01:24
Affects Version/s: 3.3.3

Fix data correctness issue with TextInputFormat that can occur when reading BZip2 compressed text files. When a file split's range does not include the start position of a BZip2 block, then it is expected to contain no records (i.e. the split is empty). However, if it so happens that the end of this split (exclusive) is at the start of a BZip2 block, then LineRecordReader ends up returning all the records for that BZip2 block. This ends up duplicating records read by a job because the next split would also end up returning all the records for the same block (since its range would include the start of that block).

This bug does not get triggered when the file split's range does include the start of at least one block and ends just before the start of another block. The reason for this has to do with when BZip2CompressionInputStream updates its position when using the BYBLOCK READMODE. Using this read mode, the stream's position while reading only gets updated when reading the first byte past an end of a block marker. The bug is that if the stream, when initialized, was adjusted to be at the end of one block, then we don't update the position after we read the first byte of the next block. Rather, we keep the position to be equal to the next block marker we've initialized to. If the exclusive end position of the split is equal to stream's position, LineRecordReader will continue to read lines until the position is updated (an an additional record in the next block is read if needed).
Bug: JavaKeyStoreProvider should throw FileNotFoundException in renameOrFail
Status: Open | Priority: Critical | Resolution: 
Created: 12/Sep/22 01:43
Affects Version/s: 3.3.5

Attempting to create a key a KMS is configured with the JavaKeystoreProvider and an HDFS store.  The calls to:
{noformat}
renameOrFail(Path src, Path dest) throws IOException {noformat}
... fails with an IOException when it attempts to rename a file.  The calling code catches FileNotFoundException since the src file may not exist.

 

Example:
{noformat}
$ hadoop key create sample
java.io.IOException: Rename unsuccessful : 'hdfs://mycluster/security/kms.jks_NEW' to 'hdfs://mycluster/security/kms.jks_NEW_ORPHANED_1662946593691'{noformat}
 

Update the implementation to check for the file, throwing a FileNotFoundException.
Bug: Don't fail when using -DNoUnitTests
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 24/Aug/22 16:10
Affects Version/s: 3.4.0

There are commands in hadoop.sh that use -Dtest=NoUnitTests.  I believe this is intended to replace -DskipTest, by not matching any tests.  This causes an issue since the default for surefire.failIfNoSpecifiedTests is true.

Updating those commands to override the default will address the issue.
Bug: Suggest to update the IP address of ConnectionId when server of Connection got updated
Status: Open | Priority: Critical | Resolution: 
Created: 18/Jul/22 19:04
Affects Version/s: 3.3.1

When use ProtobufRpcEngine to get RPC proxy, we apply the InetSocketAddress addr, and this address is then kept in the ConnectionId (see [here|https://github.com/apache/hadoop/blob/rel/release-3.3.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java#L139] and [here|https://github.com/apache/hadoop/blob/rel/release-3.3.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1681]).

When the destination IP of the connection changes, we have code to update the server variable in the connection (see [here|https://github.com/apache/hadoop/blob/rel/release-3.3.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L641]). This makes the retry of the connection to succeed. 

However, the above logic only temporary fix the connection. After the RPC call of the client, the connection will be closed. Next time when we use the same proxy to do the RPC call, we will again to fail the first connection and succeed the second one after the server IP got updates. 

My suggestion is to not only update the server variable in the connection, but also update the address of the ConnectionId in the Invoker. This way, we can reuse the proxy with the latest IP address.

The use case for this is: in Apache ozone project, we initialize the RPC proxy with a fixed InetSocketAddress (for example, datanode needs to set up the RPC proxy to scm). But if lately the server IP changes (e.g. in k8s, the IP of a pod is very likely to change), every time the proxy needs two calls (1st fails and the 2nd succeeds with the updated IP). By using the above solution, we can eliminate this issue.
Bug: Improve Pre-Commit Time
Status: Open | Priority: Critical | Resolution: 
Created: 12/Jul/22 07:08

As of now the complete build time has reached ~24 hours, which makes tracking Jira with root level changes very tough.

Even at module level, it is on the higher side, kind of 6 hours or more, which is typically equivalent to 1 working day hours:

Some areas to explore:
 * Enable Parallel-Test profile for the modules which don't have it.
 * Explore improvements in the area of our Test setup, increase memory, number of threads, or some parallel executions?
 * Remove the modules or atleast disable the test suites of modules which are no longer being used or maintained (Separate Mail Chain is there for this)
 * Spin Two Pre-Commit jobs rather than 1 and split some task between them, kind of one for Java-11 build & Javadoc, Checkstyle and related stuff.
 * Figure out & remove irrelevant or similar tests.
 * Improve existing tests, Like reusing MiniDfsClusters or so, rather than a bunch of test spinning one for themselves.
 * If possible, if there are multiple modules to be tested, run those modules test in parallel (Exploratory: may lead to OOM)
Bug: Updated addresses are still accessed using the old IP address
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/22 13:42
Affects Version/s: 3.3.5

When the IPC Client recognizes that an IP address has changed, it updates the server field and logs a message:
Address change detected. Old: journalnode-1.journalnode.hdfs.svc.cluster.local/10.1.0.178:8485 New: journalnode-1.journalnode.hdfs.svc.cluster.local/10.1.0.182:8485
Although the change is detected, the client will continue to connect to the old IP address, resulting in repeated log messages.  This is seen in managed environments when JournalNode syncing is enabled and a JournalNode is restarted, with the remaining nodes in the set repeatedly logging this message when syncing to the restarted JournalNode.

The source of the problem is that the remoteId.address is not updated.
Bug: Remove fs.s3a.executor.capacity
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/22 00:35

When s3guard was part of s3a, DynamoDBMetadataStore was the only consumer of StoreContext that used throttled executor provided by StoreContext, which internally uses fs.s3a.executor.capacity to determine executor capacity for SemaphoredDelegatingExecutor. With the removal of s3guard from s3a, we should also remove fs.s3a.executor.capacity and it's usages as it's no longer being used by any StoreContext consumers. The config's existence and its description can be really confusing for the users.
Bug: Optimise S3A’s recursive delete to drop successful S3 keys on retry of S3 DeleteObjects
Status: Open | Priority: Major | Resolution: 
Created: 24/Aug/22 16:20

S3A users with large filesystems performing renames or deletes can run into throttling when S3A performs a bulk delete on keys. These are currently batches of 250 ([https://github.com/apache/hadoop/blob/c1d82cd95e375410cb0dffc2931063d48687386f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L319-L323]).

When the bulk delete ([S3 DeleteObjects|https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html]) fails, it provides a list of keys that failed and why. Today, S3A recovers from throttles by sending the DeleteObjects request again with no change. This can result in additional deletes and counts towards throttling limits.

Instead, S3A should retry only the keys that failed, limiting the number of mutations against the S3 bucket, and hopefully mitigate errors when deleting a large number of objects.
Bug: Add XMLUtils methods to centralise code that creates secure XML parsers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/22 10:30
Affects Version/s: 3.3.4

Relates to HDFS-16766

There are other places in the code where DocumentBuilderFactory instances are created that could benefit from the same changes as HDFS-16766

h3. sonatype-2022-5820

If anyone is landing on this page following the sonatype-2022-5820 alert, know that there is no known issue here, just a centralisation of all construction of XML parsers with lockdown of all the features.
Bug: Remove rs-api dependency by downgrading jackson to 2.12.7
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jul/22 11:56
Affects Version/s: 3.3.4

This jsr311-api jar seems to conflict with newly added rs-api jar dependency - they have many of the same classes (but conflicting copies) - jersey-core 1.19 needs jsr311-api to work properly (and fails if rs-api used instead)

* https://mvnrepository.com/artifact/javax.ws.rs/jsr311-api
* https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api

Seems we will need to downgrade jackson to 2.12.7 because of jax-rs compatibility issues in jackson 2.13 (see https://github.com/FasterXML/jackson-jaxrs-providers/issues/134)
Bug: Hadoop 3.3.2 has CVEs coming from dependencies
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 05/Aug/22 05:23
Affects Version/s: 3.3.2

Hi Team,

 

Hadoop version 3.3.1 which is compatible for our application have Vulnerebilities:

Is there any plan to fix this

CVE-2021-37404 hadoop versions < 3.3.2 Apache Hadoop potential heap buffer overflow in libhdfs.
CVE-2020-10650 jackson < 2.9.10.4
CVE-2021-33036 hadoop < 3.3.2
CVE-2022-31159 aws xfer manager download < 1.12.262
Bug: S3A supports S3 on Outposts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/22 02:48
Affects Version/s: 3.3.9

Currently, the endpoint for using S3 accesspoint is set as "s3-accesspoint.%s.amazonaws.com" as follows.
[https://github.com/apache/hadoop/blob/3ec4b932c179d9ec6c4e465f25e35b3d7eded08b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/ArnResource.java#L29]

However, "s3-outposts.%s.amazonaws.com" is the preferred endpoint when accessing S3 on Outposts bucket by accesspoint.
This ticket improves them.
Bug: Add support for IBM Semeru OE JRE 11.0.15.0 and greater
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/22 10:18
Affects Version/s: 3.0.0

There are checks within the PlatformName class that use the Vendor property of the provided runtime JVM specifically looking for `IBM` within the name. Whilst this check worked for IBM's [java technology edition|https://www.ibm.com/docs/en/sdk-java-technology] it fails to work on [Semeru|https://developer.ibm.com/languages/java/semeru-runtimes/] since 11.0.15.0 due to the following change:
h4. java.vendor system property

In this release, the {{java.vendor}} system property has been changed from "International Business Machines Corporation" to "IBM Corporation".

Modules such as the below are not provided in these runtimes.
com.ibm.security.auth.module.JAASLoginModule
Bug: hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded jetty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jul/22 23:26
Affects Version/s: 3.3.3

CVE-2022-2047 and CVE-2022-2048 is recently found for Eclipse Jetty, and impacts 9.4.0 thru 9.4.46.

In latest 3.3.3 of hadoop-client-runtime, it shaded 9.4.43.v20210629 version jetty which is impacted.

In Trunk, Jetty is in version 9.4.44.v20210927, which is still impacted.

Need to upgrade Jetty Version.
Bug: S3A storage class option only picked up when buffering writes to disk
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/22 10:20
Affects Version/s: 3.3.5

when you switch s3a output stream buffering to heap or byte buffer, the storage class option isn't added to the put request


{code}

  <property>
    <name>fs.s3a.fast.upload.buffer</name>
    <value>bytebuffer</value>
  </property>

{code}

and the ITestS3AStorageClass tests fail.
{code}

java.lang.AssertionError: [Storage class of object s3a://stevel-london/test/testCreateAndCopyObjectWithStorageClassGlacier/file1] 
Expecting:
 <null>
to be equal to:
 <"glacier">
ignoring case considerations

	at org.apache.hadoop.fs.s3a.ITestS3AStorageClass.assertObjectHasStorageClass(ITestS3AStorageClass.java:215)
	at org.apache.hadoop.fs.s3a.ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier(ITestS3AStorageClass.java:129)


{code}

we noticed this in a code review; the request factory only sets the option when the source is a file, not memory.

proposed: parameterize the test suite on disk/byte buffer, then fix
Bug: upgrade commons-configuration2 to 2.8.0 and commons-text to 1.9
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/22 09:28
Affects Version/s: 3.3.5

Current version 2.1.1 has no CVEs but all higher versions have CVEs except for the latest release 2.8.0. Still feels like it would be safer to upgrade.

Currently, causes issues - that will need to be fixed:

```
[ERROR] testBlockReaderLocalWithMlockChanges(org.apache.hadoop.hdfs.client.impl.TestBlockReaderLocal)  Time elapsed: 0.414 s  <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.configuration2.interpol.ConfigurationInterpolator$DefaultPrefixLookupsHolder
	at org.apache.commons.configuration2.interpol.ConfigurationInterpolator.getDefaultPrefixLookups(ConfigurationInterpolator.java:290)
	at org.apache.commons.configuration2.AbstractConfiguration.installDefaultInterpolator(AbstractConfiguration.java:375)
	at org.apache.commons.configuration2.AbstractConfiguration.<init>(AbstractConfiguration.java:122)
	at org.apache.commons.configuration2.BaseConfiguration.<init>(BaseConfiguration.java:37)
	at org.apache.commons.configuration2.PropertiesConfiguration.<init>(PropertiesConfiguration.java:1059)
	at org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:114)
	at org.apache.hadoop.metrics2.impl.MetricsConfig.create(MetricsConfig.java:97)
	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:482)
	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188)
	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163)
	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:62)
	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:58)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1780)
```
Bug: AWS SDK update to 1.12.262 to address jackson  CVE-2018-7489 and AWS CVE-2022-31159
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/22 11:06
Affects Version/s: 3.3.4

The CVE [CVE-2022-31159|https://nvd.nist.gov/vuln/detail/CVE-2022-31159] is a vulnerability in path resolution in the AWS SDK transfer manager during downloads.

*the s3a client is not exposed to this*. it uses the class for local file upload and for object copying, but not download.

it may affect downstream use by other applications.

 yet another jackson CVE in aws sdk
https://github.com/apache/hadoop/pull/4491/commits/5496816b472473eb7a9c174b7d3e69b6eee1e271

maybe we need to have a list of all shaded jackson's we get on the CP and have a process of upgrading them all at the same time