av9ash/GitBugs huggingface dataset:
Bug: ITestS3AContractCreate#testOverwrite*EmptyDirectory fail on ozone when create performance enabled
Status: Resolved | Priority: Minor | Resolution: Not A Bug
Created: 30/Nov/24 09:36
Affects Version/s: 3.4.1

See [https://github.com/apache/hadoop/pull/6168#issuecomment-1756082091] 
They also fail with Apache Ozone. See https://issues.apache.org/jira/browse/HDDS-11663

Both of these tests expect to throw a FileAlreadyExistsException, while creating a file that overwrites an existing directory.

The following is a step-by-step description of the test.
 # mkdir test/testOverwriteNonEmptyDirectory/         // existing directory
 # get test/testOverwriteNonEmptyDirector                // start to create file to overwrite directory
 # check the file status of test/testOverwriteNonEmptyDirector
 ** If it throws FileNotFound then goes to step 4
 ** check if it is a directory then throw FileAlreadyExistsException              // expect to throw [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2196-L2199]
 ** check if the overwrite is false then throw FileAlreadyExistsException    
 # put test/testOverwriteNonEmptyDirector

However, in step 3 it actually goes to the first check, and put a new object.

Because S3a is expecting:

Get test/testOverwriteNonEmptyDirector    // does not exist
Get test/testOverwriteNonEmptyDirector/   // exists

And it will still throw an exception but not FileAlreadyExistsException.

 

How to fix:

Maybe we also get the path test test/testOverwriteNonEmptyDirector/ and check if the directory exists.
Bug: ViewFileSystem.InnerCache: Replaced ReentrantReadWriteLock with ConcurrentHashMap/putIfAbsent()
Status: Open | Priority: Minor | Resolution: 
Created: 26/Nov/24 04:38

Use of ReentrantReadWriteLock  + Map can be replaced/simplified with a ConcurrentHashMap. 

Right now, we are using a ReentrantReadWriteLock to protect concurrent updates to InnerCache.map(). 

As a performance optimization, to reduce acquiring writeLock, we first acquire readLock, check existence of the key and then acquire writeLock, to insert the new key.
{code:java}
    FileSystem get(URI uri, Configuration config) throws IOException {
      Key key = new Key(uri);
      FileSystem fs = null;
      try {
        rwLock.readLock().lock();
        fs = map.get(key);
        if (fs != null) {
          return fs;
        }
      } finally {
        rwLock.readLock().unlock();
      }
      try {
        rwLock.writeLock().lock();
        fs = map.get(key);
        if (fs != null) {
          return fs;
        }
        fs = fsCreator.getNewInstance(uri, config);
        map.put(key, fs);
        return fs;
      } finally {
        rwLock.writeLock().unlock();
      }
    }
{code}

The above function is already available as computeIfAbsent() from ConcurrentHashMap, which atomically checks existence of a key and inserts the new key if not there. This greatly simplifies the code. On the other hand, it should improve performance as well: concurrentHashMap supports finer-grain concurrency vs. a single ReadWrite Lock. 

We can replace the above code with the following one and it is much simpler.
{code:java}
    FileSystem get(URI uri, Configuration config) throws IOException {
      Key key = new Key(uri);

      FileSystem fs = map.computeIfAbsent(key, k -> getNewFileSystem(uri, config));
      if (fs == null) {
        throw new IOException("Failed to create new FileSystem instance for " + uri);
      }
      return fs;
{code}
Bug: AWS SDK deleteObjects() and S3Store.deleteObjects() don't handle 500 failures of individual objects
Status: Open | Priority: Minor | Resolution: 
Created: 26/Nov/24 14:50
Affects Version/s: 3.4.1

S3Store.deleteObjects() encountered 500 error and didn't recover.

We normally assume that 500 errors are already retried by the SDK so our own retry logic doesn't bother

The root cause is that the 500 errors can surface within the bulk delete.
* The delete POST returns 200, so SDK is happy
* but one of the rows in the request is reports the S3Error "InternalError":
{{Code=InternalError, Message=We encountered an internal error. Please try again.)]}}


Proposed.
* bulk delete invoker must map "InternalError" to AWSStatus500Exception and throw that.
* Add a retry policy for bulk deletes which considers AWSStatus500Exception as retriable. retry. We currently don't on the assumption that the SDK will retry, which it does for base retries, but clearly not for multiobject delete.
* Maybe also consider possibility that a partial 503 response could be generated? that is: only part of the delete throttled?

{code}

Caused by: org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteException: [S3Error(Key=table/warehouse/tablespace/external/hive/table/-tmp.-ext-10000/file/, Code=InternalError, Message=We encountered an internal error. Please try again.)]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:3186)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeysS3(S3AFileSystem.java:3422)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3481)
	at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.removeKeys(S3AFileSystem.java:2558)
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$removeSourceObjects$3(RenameOperation.java:625)
	at org.apache.hadoop.fs.s3a.Invoker.lambda$once$0(Invoker.java:165)
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:122)
  
{code}
Bug: AWS V1 SDK now tells users off for using a deprecated SDK
Status: Open | Priority: Minor | Resolution: 
Created: 21/Oct/24 11:50
Affects Version/s: 3.3.9

This is for anyone upgrading their v1 SDK to any release after July 2024:

The fix is to set the log to fatal or the sysprop aws.java.v1.disableDeprecationAnnouncement to true

I think I will go with #2 everywhere, and add it to a copy of the AwsWorkarounds class for v1. New warning messages from the SDK telling off end users are not permitted.

{code}
logger.log.com.amazonaws.util.VersionInfoUtils=FATAL
{code}

{code}
WARN util.VersionInfoUtils: The AWS SDK for Java 1.x entered maintenance mode starting July 31, 2024 and will reach end of support on December 31, 2025. For more information, see https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-1-x-is-in-maintenance-mode-effective-july-31-2024/
You can print where on the file system the AWS SDK for Java 1.x core runtime is located by setting the AWS_JAVA_V1_PRINT_LOCATION environment variable or aws.java.v1.printLocation system property to 'true'.
This message can be disabled by setting the AWS_JAVA_V1_DISABLE_DEPRECATION_ANNOUNCEMENT environment variable or aws.java.v1.disableDeprecationAnnouncement system property to 'true'.
{code}
Bug: Fix ZKFailoverController NPE issue due to integer overflow in parseInt when initHM.
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 16/Nov/24 01:46

h3. *What Happened:* 

A null pointer exception occurs when trying to shutdown healthMonitor. healthMonitor is not initialized if the ha.health-monitor.rpc-timeout.ms is set to 4294967295. The healthMonitor constructor uses parseInt() to parse configuration values and this value overflows for parseInt and throws an exception during healthMonitor initialization. 
h3. *Buggy Code:* 

 
{code:java}
try {
  initRPC();
  initHM(); // -> This throws a java.Lang.NumberFormatException and   healthMonitor is not intialized
  startRPC();
  mainLoop();
} catch (Exception e) {
  LOG.error("The failover controller encounters runtime error: ", e);
  throw e;
} 

  ...
  
  healthMonitor.shutdown(); // -> NPE if healthMonitor is not intialized
  healthMonitor.join();
} {code}
 

 
{code:java}
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:266)
        at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:65)
        at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:186)
        at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:182)
        at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:520)
        at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:182)
        at org.apache.hadoop.ha.MiniZKFCCluster$DummyZKFCThread.doWork(MiniZKFCCluster.java:301)
        at org.apache.hadoop.test.MultithreadedTestUtil$TestingThread.run(MultithreadedTestUtil.java:189)
 {code}
h3. *How to Reproduce:*

(1) Set ha.health-monitor.rpc-timeout.ms to 4294967295.  

(2) Run: test: org.apache.hadoop.ha.TestZKFailoverController#testVerifyObserverState
Bug: Mark FutureDataInputStreamBuilderImpl as VisibleForTesting
Status: Open | Priority: Trivial | Resolution: 
Created: 03/Dec/24 15:21
Affects Version/s: 3.4.1

declare {{org.apache.hadoop.fs.impl.FutureDataInputStreamBuilderImpl}} as
* VisibleForTesting
* Evolving

This is because it is class is the only way to reliably create stub implementations of the builder API. We also know that it's pretty stable, though some methods have been added over time -a change which makes writing a stub implementation on its own really hard.
Bug: MutableQuantiles.getQuantiles() should be made a static method
Status: Open | Priority: Trivial | Resolution: 
Created: 01/Oct/24 22:04

MutableQuantiles.getQuantiles() returns the static member variable QUANTILES, so this method should be a static method too.

https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java#L157
Bug: Add back the exceptions removed by HADOOP-17432 for compatibility
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 18/Apr/21 08:46
Affects Version/s: 3.3.1

As [~ayushsaxena] commented in https://issues.apache.org/jira/browse/HADOOP-17432?focusedCommentId=17324284&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17324284, we need to add back the exceptions for compatibility.
Bug: Upgrade Jetty to 9.4.40
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 23/Apr/21 07:17
Affects Version/s: 3.3.1

Jetty 9.4.39 has a critical bug: https://github.com/eclipse/jetty.project/issues/6152
It causes http put failure in HttpFS with SSL enabled.

{noformat}
curl --negotiate -u : "https://<HttpFS host>:<port>/webhdfs/v1/<output>?op=CREATE&data=true" -X PUT -H Content-Type:application/octet-stream -T <input>
curl: (55) TCP connection reset by peer
{noformat}
Bug: Explicitly set locale in the Dockerfile
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 20/May/21 11:50
Affects Version/s: 3.3.1

When producing the RC bits for 3.3.1, the releasedocmaker step failed.

{noformat}
[INFO] --- exec-maven-plugin:1.3.1:exec (releasedocs) @ hadoop-common ---
Traceback (most recent call last):
  File "/build/source/patchprocess/apache-yetus-0.13.0/bin/../lib/releasedocmaker/releasedocmaker.py", line 25, in <module>
    releasedocmaker.main()
  File "/build/source/patchprocess/apache-yetus-0.13.0/lib/releasedocmaker/releasedocmaker/__init__.py", line 979, in main
    JIRA_BASE_URL)
  File "/build/source/patchprocess/apache-yetus-0.13.0/lib/releasedocmaker/releasedocmaker/utils.py", line 199, in write_list
    self.write_key_raw(jira.get_project(), line)
  File "/build/source/patchprocess/apache-yetus-0.13.0/lib/releasedocmaker/releasedocmaker/utils.py", line 170, in write_key_raw
    self.base.write(input_string)
UnicodeEncodeError: 'ascii' codec can't encode character '\xdc' in position 71: ordinal not in range(128)

{noformat}

It turns out if the script reads jiras containing ascended characters, it can't write the report.

Inside docker container, the default locale is "ANSI_X3.4-1968". Must set it to utf-8 to support special characters.

Curious why it wasn't a problem before.

More details: https://stackoverflow.com/questions/43356982/docker-python-set-utf-8-locale
Bug: [build] fix the Dockerfile for ARM
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 21/May/21 09:12
Affects Version/s: 3.3.1

Running the create-release script for Hadoop 3.3.1 on an ARM machine, docker image fails to build:

{noformat}
    aarch64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iast27/Include -I/usr/include/python3.6m -c ast27/Parser/acceler.c -o build/temp.linux-aarch64-3.6/ast27/Parser/acceler.o                                                                                                            In file included from ast27/Parser/acceler.c:13:0:                                                                                ast27/Parser/../Include/pgenheaders.h:8:10: fatal error: Python.h: No such file or directory                                       #include "Python.h"                                                                                                                        ^~~~~~~~~~                                                                                                              compilation terminated.                                                                                                           error: command 'aarch64-linux-gnu-gcc' failed with exit status 1


{noformat}

The missing Python3.h requires python3-dev package: https://stackoverflow.com/questions/21530577/fatal-error-python-h-no-such-file-or-directory

The PhantomJS binary was built for Xenial, doesn't run after the Dockerfile migrated to Bionic/Focal. Fortunately Bionic/Focal has official PhantomJS packages.
Bug: S3AFS creation fails "Unable to find a region via the region provider chain."
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 22/Jun/21 09:20
Affects Version/s: 3.3.1

If you don't have {{fs.s3a.endpoint}} set and lack a region set in
env var {{AWS_REGION_ENV_VAR}}, system property {{aws.region}} or the file  ~/.aws/config
then S3A FS creation fails with  the message
"Unable to find a region via the region provider chain."

This is caused by the move to the AWS S3 client builder API in HADOOP-13551

This is pretty dramatic and no doubt everyone will be asking "why didn't you notice this?",


But in fact there are some reasons.
# when running in EC2, all is well. Meaning our big test runs were all happy.
# if a developer has fs.s3a.endpoint set for the test bucket, all is well.
   Those of us who work with buckets in the "regions tend to do this, not least because it can save a HEAD request every time an FS is created.
# if you have a region set in ~/.aws/config then all is well

reason #3 is the real surprise and the one which has really caught us out. Even my tests against buckets in usw-2 through central didn't fail because of course I, like my colleagues, have the AWS cli client installed locally. This was sufficient to make the problem go away. It is also why this has been an intermittent problem on test clusters outside AWS infra: it really depended on the VM/docker image whether things worked or not.

h2. Quick Fix: set {{fs.s3a.endpoint}} to {{s3.amazonaws.com}} 

If you have found this JIRA because you are encountering this problem, you can fix it in by explicitly declaring the endpoint in {{core-site.xml}}

{code}
<property>
  <name>fs.s3a.endpoint</name>
  <value>s3.amazonaws.com</value>
</property>
{code}

For Apache Spark, this can be done in {{spark-defaults.conf}}

{code}
spark.hadoop.fs.s3a.endpoint s3.amazonaws.com
{code}

If you know the exact AWS region your data lives in, set the endpoint to be that region's endpoint, and so save an HTTPS request to s3.amazonaws.com every time an S3A Filesystem instance is created.
Bug: mvn versions:set fails to parse pom.xml
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Apr/21 05:14
Affects Version/s: 3.2.3

Run the following command on trunk
mvn versions:set -DnewVersion=3.4.0

{noformat}
[ERROR] Failed to execute goal org.codehaus.mojo:versions-maven-plugin:2.8.1:set (default-cli) on project hadoop-main: Execution default-cli of goal org.codehaus.mojo:versions-maven-plugin:2.8.1:set failed: Error parsing /Users/weichiu/sandbox/hadoop/hadoop-project/pom.xml: Unexpected character '-' (code 45) (expected a name start character)
[ERROR]  at [row,col {unknown-source}]: [2288,12]
{noformat}
Someone else reported the same issue https://github.com/mojohaus/versions-maven-plugin/issues/429

Maven: 3.3.9 ~ 3.8.1
Affects trunk ~ branch-3.0. (But branch-2.10 is not affected)

We don't use versions-maven-plugin directly, and I couldn't find out when it was updated.
Bug: HADOOP-16916 changed interface SASTokenProvider fields, breaking compatibility between 3.3.0 and 3.3.1 
Status: Resolved | Priority: Blocker | Resolution: Won't Fix
Created: 17/May/21 10:52
Affects Version/s: 3.3.1

I understand HADOOP-16730/HADOOP-16916 is specifically made for Ranger, but I am not sure how Ranger consumes this feature. The interface SASTokenProvider has a number of member fields that changed variable names in HADOOP-16916, breaking the compatibility between 3.3.0 and 3.3.1.

As a matter of fact, the feature HADOOP-16730 itself was merged in 3.3.0 not 3.3.1. I just corrected it today.

Raise this jira and mark it as a blocker for 3.3.1. But if this isn't a big deal then we can downgrade, because, well, this feature was not officially in the 3.3.0 release.
Bug: [thirdparty] Fix the docker image
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 12/May/21 05:01
Affects Version/s: thirdparty-1.1.0

The hadoop-thirdparty release script does not work out of box.

(1) Hard code a python dependency version to avoid image build error.
(2) Remove npm which is not used for building hadoop-thirdparty to avoid image build error.
(2) Use GPG 2.2 (LTS) instead of 1.x for signing. GPG 1.x is quite old and is not compatible with GPG 2.x (https://superuser.com/questions/1112673/gpg2-no-secret-key). The build script runs a gpg agent inside docker that shares the gpg keys with the host machine. Most host machines today ships with GPG 2. Ubuntu 18.04 installs gpg 2.2 by default.

(gpg2.1, which is supported by Ubuntu 16.04, is not compatible with gpg 2.2: https://github.com/sobolevn/git-secret/issues/136)
Bug: IOStatistics API in branch-3.3 break compatibility
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 27/Apr/21 10:15

The S3 delegation token feature (3.3.0) added API 
{code:java}
  AmazonS3 createS3Client(URI name,
      String bucket,
      AWSCredentialsProvider credentialSet,
      String userAgentSuffix) throws IOException;
 {code}
However, the IOStatistics API (HADOOP-17271, HADOOP-13551. in 3.3.1) changed it to
{code:java}

  AmazonS3 createS3Client(URI name,
      String bucket,
      AWSCredentialsProvider credentialSet,
      String userAgentSuffix, 
    StatisticsFromAwsSdk statisticsFromAwsSdk) throws IOException; {code}

The API is declared evolving, so we're not supposed to break compat between maintenance releases.

[~stevel@apache.org]
Bug: DistCp job fails when AM is killed
Status: Patch Available | Priority: Major | Resolution: 
Created: 16/Jun/21 05:36

Job fails as tasks fail with below exception
{code:java}
2021-06-11 18:48:47,047 | ERROR | IPC Server handler 0 on 27101 | Task: attempt_1623387358383_0006_m_000000_1000 - exited : java.io.FileNotFoundException: File does not exist: hdfs://hacluster/staging-dir/dsperf/.staging/_distcp-646531269/fileList.seq
 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1637)
 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1630)
 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1645)
 at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1863)
 at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1886)
 at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54)
 at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
 at org.apache.hadoop.mapred.YarnChild$1.run(YarnChild.java:183)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:177)
 | TaskAttemptListenerImpl.java:304{code}
Bug: A divide by zero bug in LoadBalancingKMSClientProvider.java
Status: Open | Priority: Major | Resolution: 
Created: 19/May/21 06:52

In the file _kms/LoadBalancingKMSClientProvider.java_, the function _rollNewVersion_ has the following [code|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/LoadBalancingKMSClientProvider.java#L509-#L516]:

 
{code:java}
@Override
public KeyVersion rollNewVersion(final String name, final byte[] material)
    throws IOException {
    final KeyVersion newVersion = doOp(new ProviderCallable<KeyVersion>() {
      @Override
      public KeyVersion call(KMSClientProvider provider) throws IOException {
        return provider.rollNewVersion(name, material);
      }
    }, nextIdx(), false);
    ...
}
{code}
The function _nextIdx_ uses _providers.length_ as a divisor:
{code:java}
private int nextIdx() {
    while (true) {
      int current = currentIdx.get();
      int next = (current + 1) % providers.length;
      ......
}{code}
However, _providers.length_ may be equal to zero, since the function _doOp_ explicitly checks that and throws an exception when it happens:
{code:java}
private <T> T doOp(ProviderCallable<T> op, int currPos,
      boolean isIdempotent) throws IOException {
    if (providers.length == 0) {
      throw new IOException("No providers configured !");
    }
    ...
}
{code}
 

The problem is that when _providers.length_ is 0, a divide by zero problem will happen when computing the argument for _doOp_ (inside the function _nextIdx_) before reaching the protection check above, causing an ArithmeticException.