av9ash/GitBugs huggingface dataset:
Bug: NPE in LdapAuthenticationHandler as disableHostNameVerification is never initialized
Status: Open | Priority: Major | Resolution: 
Created: 25/Jan/23 06:35

Steps to reproduce this issue:

Enable ldap auth with tls by configuring these in core-site.xml
1. hadoop.http.authentication.multi-scheme-auth-handler.schemes = basic
2. hadoop.http.authentication.multi-scheme-auth-handler.schemes.basic.handler = ldap
3. hadoop.http.authentication.ldap.enablestarttls = true

 

Trace:
{noformat}
java.lang.NullPointerException
        at org.apache.hadoop.security.authentication.server.LdapAuthenticationHandler.authenticateWithTlsExtension(LdapAuthenticationHandler.java:261)
        at org.apache.hadoop.security.authentication.server.LdapAuthenticationHandler.authenticateUser(LdapAuthenticationHandler.java:238)
        at org.apache.hadoop.security.authentication.server.LdapAuthenticationHandler.authenticate(LdapAuthenticationHandler.java:202)
{noformat}
Bug: Remove unnecessary dependency on json-smart
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Mar/23 11:16
Affects Version/s: 3.3.5

hadoop-auth has a dependency on net.minidev:json-smart 2.4.7, but this dependency is never used.

This dependency was originally included because the transitive dependency that nimbus-jose-jwt had did not work properly (see https://issues.apache.org/jira/browse/HADOOP-14903). Since version 9.* nimbus-jose-jwt is using its own shaded version of json-smart, so the version declared in hadoop-auth is never actually used.

json-smart 2.4.7 shows up in CVE scans for CVE-2023-1370. It is still used as a transitive dependency in hadoop-hdfs
Bug: ABFS: Customize and optimize timeouts made based on each separate request
Status: Open | Priority: Major | Resolution: 
Created: 15/Feb/23 09:21

In present day ABFS Driver functioning, all API request calls use the same values of default timeouts. This is sub-optimal in the scenarios where a request is failing due to hitting a particular busy node, and would benefit simply by retrying quicker.

For this, the change to be brought in chooses customized timeouts based on which API call is being made. Further, starting with smaller, optimized values of timeouts, the timeout values would increase by a certain incremental factor for subsequent retries to ensure quicker retries and success.
Bug: upgrade to jettison 1.5.3 to fix CVE-2022-40150
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/23 22:50
Affects Version/s: 3.3.5

[https://github.com/advisories/GHSA-x27m-9w8j-5vcw]

 

[https://github.com/jettison-json/jettison/releases]

v1.5.2 is flagged as fixing a CVE but a v1.5.3 was quickly released and appears ti fix some regressions caused by v1.5.2.
Many hadoop tests fail when jettison 1.5.2 is used.
Bug: Fix build failure with docs profile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/23 08:09
Affects Version/s: 3.4.0

Build with docs profile failed due to depcheck error.

{noformat}
$ mvn clean install -DskipTests -DskipShade -Pdocs
...
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (depcheck) @ hadoop-hdfs ---
[WARNING]
Dependency convergence error for xerces:xercesImpl:jar:2.12.2:provided paths to dependency are:
+-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
  +-org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:provided
    +-xerces:xercesImpl:jar:2.12.2:provided
and
+-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
  +-org.apache.hadoop:hadoop-common:test-jar:tests:3.4.0-SNAPSHOT:test
    +-xerces:xercesImpl:jar:2.12.2:test
and
+-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
  +-org.apache.hadoop:hadoop-hdfs-client:jar:3.4.0-SNAPSHOT:provided
    +-xerces:xercesImpl:jar:2.12.2:provided
and
+-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
  +-com.google.code.findbugs:findbugs:jar:3.0.1:provided
    +-jaxen:jaxen:jar:1.1.6:provided
      +-xerces:xercesImpl:jar:2.6.2:provided
and
+-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
  +-xerces:xercesImpl:jar:2.12.2:compile

[WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with message:
Failed while enforcing releasability. See above detailed error message.
[INFO] ------------------------------------------------------------------------
{noformat}
Bug: Avoid using grizzly-http-* APIs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/23 21:35
Affects Version/s: 3.3.6

As discussed on the parent Jira HADOOP-15984, we do not have any grizzly-http-servlet version available that uses Jersey 2 dependencies. 

version 2.4.4 contains Jersey 1 artifacts: [https://repo1.maven.org/maven2/org/glassfish/grizzly/grizzly-http-servlet/2.4.4/grizzly-http-servlet-2.4.4.pom]

The next higher version available is 3.0.0-M1 and it contains Jersey 3 artifacts: [https://repo1.maven.org/maven2/org/glassfish/grizzly/grizzly-http-servlet/3.0.0-M1/grizzly-http-servlet-3.0.0-M1.pom]

 

Moreover, we do not use grizzly-http-* modules extensively. We use them only for few tests such that we don't have to implement all the methods of HttpServletResponse for our custom test classes.

We should get rid of grizzly-http-servlet, grizzly-http and grizzly-http-server artifacts of org.glassfish.grizzly and rather implement HttpServletResponse directly to avoid having to depend on grizzly upgrades as part of overall Jersey upgrade.
Bug: Upgrade ant to 1.10.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/23 12:03
Affects Version/s: 3.3.6

lnerabilities reported in org.apache.ant:ant:1.10.11
 * [CVE-2022-23437|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23437]
 * [CVE-2020-14338|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-14338]

suggested: org.apache.ant:ant ~> 1.10.13
Bug: Add gh-pages in asf.yaml to deploy the current trunk doc
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/23 10:50
Affects Version/s: 3.4.0

The docs are now pushed to gh-pages.
Add the entry in asf.yaml to publish as mentioned in 
https://issues.apache.org/jira/browse/INFRA-24202
Bug: Migrate Async appenders to log4j properties
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Feb/23 06:42
Affects Version/s: 3.4.0

Before we can upgrade to log4j2, we need to migrate async appenders that we add "dynamically in the code" to the log4j.properties file. Instead of using core/hdfs site configs, log4j properties or system properties should be used to determine if the given logger should use async appender.
Bug: S3A to support upload of files greater than 2 GB using DiskBlocks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/23 09:05
Affects Version/s: 3.3.6

Use S3A Diskblocks to support the upload of files greater than 2 GB using DiskBlocks. Currently, the max upload size of a single block is ~2GB. 

cc: [~mthakur] [~stevel@apache.org] [~mehakmeet]
Bug: Add bswap support for LoongArch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/23 10:08
Affects Version/s: 3.4.0

The LoongArch architecture (LoongArch) is an Instruction Set Architecture (ISA) that has a RISC style.
Documentations:
ISA:
[https://loongson.github.io/LoongArch-Documentation/LoongArch-Vol1-EN.html]
ABI:
[https://loongson.github.io/LoongArch-Documentation/LoongArch-ELF-ABI-EN.html]
More docs can be found at:
[https://loongson.github.io/LoongArch-Documentation/README-EN.html]
Bug: Upgrade Netty to 4.1.89.Final
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/23 07:12
Affects Version/s: 3.3.4

h4. Netty version - 4.1.89 has fix  CVEs: [CVE-2022-41881|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-41881]
Bug: Avoid loading kms log4j properties dynamically by KMSWebServer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/23 00:19
Affects Version/s: 3.4.0

Log4j2 does not support loading of log4j properties (/xml/json/yaml) dynamically by applications. It no longer supports overriding the loading of properties using "log4j.defaultInitOverride" the way log4j1 does.

For KMS, instead of loading the properties file dynamically, we should add the log4j properties file as part of HADOOP_OPTS.
Bug: CLA and CRLA appenders to be replaced with RFA
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/23 07:26
Affects Version/s: 3.4.0

ContainerLogAppender and ContainerRollingLogAppender both have quite similar functionality as RollingFileAppender. Maintenance of custom appenders for Log4J2 is costly when there is very minor difference in comparison with built-in appender provided by Log4J. 

The goal of this sub-task is to replace both ContainerLogAppender and ContainerRollingLogAppender custom appenders with RollingFileAppender without changing any system properties already being used to determine file name, file size, backup index, pattern layout properties etc.
Bug: LogLevel servlet to determine log impl before using setLevel
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/23 06:03
Affects Version/s: 3.4.0

LogLevel GET API is used to set log level for a given class name dynamically. While we have cleaned up the commons-logging references, it would be great to determine whether slf4j log4j adapter is in the classpath before allowing client to set the log level.

Proposed changes:
 * Use slf4j logger factory to get the log reference for the given class name
 * Use generic utility to identify if the slf4j log4j adapter is in the classpath before using log4j API to update the log level
 * If the log4j adapter is not in the classpath, report error in the output
Bug: Remove unused custom appender TaskLogAppender
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Mar/23 03:12
Affects Version/s: 3.4.0

TaskLogAppender is no longer being used in codebase. The only past references we have are from old releasenotes (HADOOP-7308, MAPREDUCE-3208, MAPREDUCE-2372, HADOOP-1355).

Before we migrate to log4j2, it would be good to remove TaskLogAppender.
Bug: snakeyaml dependency: upgrade to v2.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/23 11:09
Affects Version/s: 3.3.6

* [https://github.com/advisories/GHSA-mjmj-j48q-9wg2]
 * I don't think this needs to go in v3.3.5 - since this CVE affects part of snakeyaml that hadoop doesn't use
Bug: ListFiles with recursive fails with FNF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/23 08:07
Affects Version/s: 3.3.6

Problem triggers in HDFS, but the change is in Hadoop-Common, Since the listFiles is defined in Hadoop-Common.

Scenario:

ListFiles With recursive: 
 * Fetches a dir say /dir, which has some /dir/s1...s10
 * Recursive is set to true: It goes and tries on say /dir/s5 and /dir/s5 got deleted by that time
 * The entire operation fails with FNF

Hive Cleaner uses listFiles with recursive true and this impacts that
{noformat}
2023-03-06 07:45:48,331 ERROR org.apache.hadoop.hive.ql.txn.compactor.Cleaner: [Cleaner-executor-thread-12]: Caught exception when cleaning, unable to complete cleaning of id:39762523,dbname:test,tableName:test_table,partName:null,state:,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:989,errorMessage:null,workerId: null,initiatorId: null java.io.FileNotFoundException: File hdfs:/cluster/warehouse/tablespace/managed/hive/test.db/test_table/.hive-staging_hive_2023-03-06_07-45-23_120_4659605113266849995-73550 does not exist.
    at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1275)
    at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1249)
    at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
    at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
    at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144)
    at org.apache.hadoop.fs.FileSystem$5.handleFileStat(FileSystem.java:2332)
    at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:2309)
    at org.apache.hadoop.util.functional.RemoteIterators$WrappingRemoteIterator.sourceHasNext(RemoteIterators.java:432)
    at org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.fetch(RemoteIterators.java:581)
    at org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.hasNext(RemoteIterators.java:602)
    at org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1435)
    at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:287)
    at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:214)
    at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.lambda$run$0(Cleaner.java:114)
    at org.apache.hadoop.hive.ql.txn.compactor.CompactorUtil$ThrowingRunnable.lambda$unchecked$0(CompactorUtil.java:54)
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750){noformat}
Bug: A whitelist of endpoints to skip Kerberos authentication doesn't work for ResourceManager and Job History Server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/23 07:44
Affects Version/s: 3.4.0

Thanks to HADOOP-16527, we can add a whitelist of endpoints to skip Kerberos authentication such as {{/isActive}}, {{/jmx}}, {{/prom}}.
However, I found that ResourceManager and Job History Server doesn't repect {{hadoop.http.authentication.kerberos.endpoint.whitelist}}.

To workaround this issue for ResourceManager, set {{yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled=true}} in yarn-site.xml.
However, there is no workaround for Job History Server.

This bug is caused by {{HttpServer2#initSpnego}} call without proper configurations which starts with "{{hadoop.http.authentication.}}".

I will make a PR soon.
Bug: Path capability probe for truncate is only honored by RawLocalFileSystem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/23 22:56
Affects Version/s: 3.3.4

FileSystem#hasPathCapability returns true for probing "fs.capability.paths.truncate" only by RawLocalFileSystem. It should be honored by all file system implementations that support truncate.
Bug: Remove Log4Json Layout
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/23 05:39
Affects Version/s: 3.4.0

Log4Json extends org.apache.log4j.Layout to provide log layout for Json. This utility is not being used anywhere in Hadoop. It is IA.Private (by default).

Log4j2 has introduced drastic changes to the Layout. It also converted it as an interface. Log4j2 also has JsonLayout, it provides options like Pretty vs. compact JSON, Encoding UTF-8 or UTF-16, Complete well-formed JSON vs. fragment JSON, addition of custom fields into generated JSON

[https://github.com/apache/logging-log4j2/blob/2.x/log4j-core/src/main/java/org/apache/logging/log4j/core/layout/JsonLayout.java]

 

This utility is more suitable to be part of log4j project rather than hadoop because the maintenance cost in hadoop would be higher with any more upgrades introducing changes to the Layout format.
Bug: Include jettison as direct dependency of hadoop-common
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/23 16:23
Affects Version/s: 3.4.0

When hadoop common is pulled in outside of hadoop project, the wrong version of jettison is coming as the dependency management of hadoop-project doesn't apply this case.

So it's not enough just to upgrade the version in dependency management. When jettison is coming only as transitive, whoever is pulling in that hadoop library, will still get the wrong jettison.
example:
hadoop-common
{noformat}
org.example:untitled:jar:1.0-SNAPSHOT
\- org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:compile
...
   +- com.github.pjfanning:jersey-json:jar:1.20:compile
   |  +- org.codehaus.jettison:jettison:jar:1.1:compile
...
{noformat}
When the module is a library (so it will be used outside of the actual project), the correct dependency must be declared as direct dependency (and optionally excluding from the dependency where it came from originally).


jettison should be added as direct dependency
Bug: S3A filesystem to support binding to other URI schemes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/23 13:35
Affects Version/s: 3.3.5

Allow s3a filesystem to be bindable to other filesystem schemas, especially s3://

* FileContext API has hard coded use of "s3a"
* S3AFileSystem.getScheme() needs to pick up the scheme of the URI passed to initialize()
* plus tests
Bug: Sasl connection failure should log remote address
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 02:53
Affects Version/s: 3.3.4

If Sasl connection fails with some generic error, we miss logging remote server that the client was trying to connect to.

Sample log:
{code:java}
2023-01-12 00:22:28,148 WARN  [20%2C1673404849949,1] ipc.Client - Exception encountered while connecting to the server 
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:197)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
    at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
    at java.io.DataInputStream.readInt(DataInputStream.java:387)
    at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1950)
    at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367)
    at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
    at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
...
... {code}
We should log the remote server address.
Bug: Expose distcp counters to user via config parameter and distcp contants
Status: Open | Priority: Major | Resolution: 
Created: 15/Feb/23 14:19
Affects Version/s: 3.4.0

Currently users or application such as Hive cannot access directly the distcp counters such as total number of bytes copied by distcp operation. 

This Jira is to enable this functionality in distcp tool.
Bug: replace jsr311-api dependency with rs-api
Status: Open | Priority: Major | Resolution: 
Created: 06/Feb/23 20:00

[jsr311-api|https://mvnrepository.com/artifact/javax.ws.rs/jsr311-api] is unmaintained and causes issues when jars bring in a dependency on the newer [rs-api|https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api/2.1.1] jar - that uses the same package name but has incompatible code

To make things worse, there is now a jakarta fork of rs-api but I suggest we worry about that later.

jersey-core 1.19.x gives us the jsr311-api dependency. 

The upgrade to HADOOP-15984 is currently blocked and looks hard.

HADOOP-15983 is a workaround that allows us to keep jersey 1.x but removes the issue where we end up relying on the unmaintained Jackson 1.9 jars.

We may now need a similar fork of jersey-core 1.19 to build a version of that jar that uses rs-api instead of jsr311.

The main benefit here is get around the fact that jackson jaxrs 2.13+ has dropped support for jsr311 and now only supports rs-api. (see HADOOP-18332)
Bug: Distcp -update between different cloud stores to use modification time while checking for file skip.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/23 05:51

Distcp -update currently relies on File size, block size, and Checksum comparisons to figure out which files should be skipped or copied. 
Since different cloud stores have different checksum algorithms we should check for modification time as well to the checks.

This would ensure that while performing -update if the files are perceived to be out of sync we should copy them. The machines between which the file transfers occur should be in time sync to avoid any extra copies.

Improving testing and documentation for modification time checks between different object stores to ensure no incorrect skipping of files.
Bug: Java 11 JavaDoc fails due to missing package comments
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 03/Feb/23 02:14
Affects Version/s: 3.3.5

Submissions to `hadoop-common` fail in Yetus due to Java 11 JavaDoc errors:
```
[ERROR] /home/builder/src/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/package-info.java:21: error: unknown tag: InterfaceAudience.Private
[ERROR] @InterfaceAudience.Private
[ERROR] ^
[ERROR] /home/builder/src/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/package-info.java:22: error: unknown tag: InterfaceStability.Unstable
[ERROR] @InterfaceStability.Unstable
[ERROR] ^
```
Bug: fix test AbstractContractDistCpTest#testDistCpUpdateCheckFileSkip 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Feb/23 09:50

In the newly introduced test testDistCpUpdateCheckFileSkip, for the first pass of "distcp -update", target file should not be present so that the copy takes place and creates the target file. 

Currently, we create both the source and target file with same block size from the start which can lead to flakiness due to race condition causing the modification time of the target file to be greater than/equal to the source and not copy the file at all. This can be seen more in the TestLocalContractDistCp due to no remote calls to create the target.
{code:java}
java.lang.AssertionError: Mismatch in COPY counter value expected:<1> but was:<0>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.apache.hadoop.tools.contract.AbstractContractDistCpTest.verifySkipAndCopyCounter(AbstractContractDistCpTest.java:1000)
	at org.apache.hadoop.tools.contract.AbstractContractDistCpTest.testDistCpUpdateCheckFileSkip(AbstractContractDistCpTest.java:919) {code}
Bug: Tune ABFS create() retry logic
Status: Open | Priority: Major | Resolution: 
Created: 08/Mar/23 14:23
Affects Version/s: 3.3.5

Based on experience trying to debug this happening
# add debug statements when create() fails
# generated exception text to reference string shared with tests, path and error code
# generated exception to include inner exception for full stack trace

Currently the retry logic is
# create(overwrite=false)
# if HTTP_CONFLICT/409 raised; call HEAD
# use etag in create(path, overwrite=true, etag)
# special handling of error HTTP_PRECON_FAILED = 412

There's a race condition here, which is if between 1 and 2 the file which exists is deleted. The retry should succeed, but currently a 404 from the head is escalated to a failure

proposed changes
# if HEAD is 404, leave etag == null and continue
# special handling of 412 also to handle 409
Bug: Upgrade hadoop3 docker scripts to use 3.3.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/23 12:49

Hadoop 3.3.5 is released, the docker3 still points to 3.3,1, Lets upgrade it to 3.3.5

It doesn't work as of now due to broken link to apache-rat-0.13