av9ash/GitBugs huggingface dataset:
Bug: Configuration Resolver not working on env.vars through namenode/conf or spark
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 13/Oct/22 09:57
Affects Version/s: 3.3.4

- Following [Configuration javadoc|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java]
- Using property value as 
{code:xml}
${env.ENV_VARIABLE} 
{code}
- Cluster Startup well, and property is correctly replaced

Launching a Spark Job or accessing configuration throught http://namenode/conf property value is not replaced and stile provided as value ${env.ENV_VARIABLE}
Bug: RawErasureCoderBenchmark.configure() returns negative values of data and throughput in the output
Status: Open | Priority: Major | Resolution: 
Created: 11/Oct/22 12:20
Affects Version/s: 3.3.4

The method _configure_ of class BenchData inside RawErasureCoderBenchmark.java, when provided with a negative value for its input parameter ({_}int dataSizeMB{_}) returns negative values of data and throughput instead of logging an error or throwing an exception.
Bug: ClassNotFoundException due to org.apache.hadoop.shaded prepended to class name
Status: Open | Priority: Major | Resolution: 
Created: 03/Oct/22 19:00
Affects Version/s: 3.3.1

I have a scenario where I am doing Dynamic Class loading . I do not have any dependencies in pom.xml.  Class is supposed to be loaded when hadoop runs. I am facing class not found exception. This is happening on Hadoop 3.x. It used to work fine on hadoop 2.9

java.io.IOException: ClassNotFoundExceptionjava.lang.ClassNotFoundException: org.apache.hadoop.shaded.com.microsoft.MyClustomClass
Bug: S3A: support custom S3 and STS headers
Status: Open | Priority: Minor | Resolution: 
Created: 07/Dec/22 18:29

Some users have the use case where for a particular S3A filesystem, they would like to set one or more headers to a static value.

We might imagine a set of configurations properties with a common prefix, where the suffix determines the header name and the value of the property determines the value of the header.

Per-filesystem headers can be set using the per-bucket configuration.
Bug: Warn when no region is configured
Status: Open | Priority: Minor | Resolution: 
Created: 16/Dec/22 11:36
Affects Version/s: 3.3.9

The AWS Java SDK V1 allows for cross region access. This means that even if you instantiate the S3 client with US_EAST_1 (or any region different to your actual bucket's region), the SDK will figure out the region. 

 

With the upgrade to SDK V2, this is no longer supported and the region should be set explicitly. Requests with the incorrect region will fail. To prepare for this change, S3A should warn when a region is not set via fs.s3a.endpoint.region. 

 

We should warn even if fs.s3a.endpoint is set and region can be parsed from this. This is because it is recommended to let the SDK V2 figure out the endpoint to use from the region, and so S3A should discourage from setting the endpoint unless absolutely required (eg for third party stores). 

 

Ideally rename fs.s3a.endpoint.region to fs.s3a.region, but not sure if this is ok to do.
Bug: support multiple s3a integration test runs on same bucket in parallel
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Oct/22 13:25
Affects Version/s: 3.3.9

It is now possible to provide a job ID in the maven "job.id" property
hadoop-aws test runs to isolate paths under a the test bucket
under which all tests will be executed.

This will allow independent builds *in different source trees*
to test against the same bucket in parallel, and is designed for
CI testing.

Example:

mvn verify -Dparallel-tests -Droot.tests.enabled=false -Djob.id=1
mvn verify -Droot.tests.enabled=false -Djob.id=2

- Root tests must be be disabled to stop them cleaning up
  the test paths of other test runs.
- Do still regularly run the root tests just to force cleanup
  of the output of any interrupted test suites.
Bug: ITestS3AInputStreamPerformance#testDecompressionSequential128K NPE if no CSV file available
Status: Open | Priority: Minor | Resolution: 
Created: 04/Oct/22 12:23

I am running the integration tests against a store with no CSV file available for scale testing. (scale enabled, csv empty).

NPE is here: https://github.com/apache/hadoop/blob/682931a6ace460d829954398eddecefeeac82b34/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3AInputStreamPerformance.java#L397

We should add a call to this somewhere - not entirely sure where right now: https://github.com/apache/hadoop/blob/682931a6ace460d829954398eddecefeeac82b34/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3AInputStreamPerformance.java#L166-L171

Stack trace:
{code:java}
[ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 6, Time elapsed: 43.985 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[ERROR] testDecompressionSequential128K(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)  Time elapsed: 1.413 s  <<< ERROR!
java.lang.NullPointerException
    at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodec(CompressionCodecFactory.java:202)
    at org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.executeDecompression(ITestS3AInputStreamPerformance.java:397)
    at org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testDecompressionSequential128K(ITestS3AInputStreamPerformance.java:383)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:750) {code}
Bug: [ABFS]: Support fixed SAS token config in addition to Custom SASTokenProvider Implementation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Nov/22 06:31
Affects Version/s: 3.4.0

This PR introduces a new configuration for Fixed SAS Tokens: *"fs.azure.sas.fixed.token"*

Using this new configuration, users can configure a fixed SAS Token in the account settings files itself. Ideally, this should be used with SAS Tokens that are scoped at a container or account level (Service or Account SAS), which can be considered to be a constant for one account or container, over multiple operations.

The other method of using a SAS Token remains valid as well, where a user provides a custom implementation of the SASTokenProvider interface, using which a SAS Token are obtained.

When an Account SAS Token is configured as the fixed SAS Token, and it is used, it is ensured that operations are within the scope of the SAS Token.

The code checks for whether the fixed token and the token provider class implementation are configured. In the case of both being set, preference is given to the custom SASTokenProvider implementation. It must be noted that if such an implementation provides a SAS Token which has a lower scope than Account SAS, some filesystem and service level operations might be out of scope and may not succeed.
Bug: ITestS3APrefetchingInputStream does not skip if no CSV test file available
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Oct/22 09:46
Affects Version/s: 3.3.6

We should use S3ATestUtils.getCSVTestFile(conf) to skip if the property is empty (single space).

Today, when I set _fs.s3a.scale.test.csvfile_ to empty space, all but this test that rely on the file are skipped.
Bug: [ABFS]: Error introduced when SAS Token containing '?' prefix is passed
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Oct/22 18:32
Affects Version/s: 3.3.5

Error Description:
At present, SAS Tokens generated from the Azure Portal may or may not contain a ? as a prefix. SAS Tokens that contain the ? prefix will lead to an error in the driver due to a clash of query parameters. This leads to customers having to manually remove the ? prefix before passing the SAS Tokens.

Mitigation:
After receiving the SAS Token from the provider, check if any prefix ? is present or not. If present, remove it and pass the SAS Token.
Bug: Upgrade maven-shade-plugin to 3.3.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Oct/22 19:35
Affects Version/s: 3.4.0

Maven-shade-plugin rewrites classes when moving them into {{hadoop-client}} JARs. That's true even when it doesn't actually need to modify the byte code of the classes, say for shading.

We use a tool that checks for classpath duplicates that don't have equal byte code. This tool flags classes brought in via Hadoop. The classes it flagged came on one side from 
a JAR containing relocated classes ({{hadoop-client-api}} or {{-runtime}}) and the other from the relocated JAR ({{hadoop-common}} or {{hadoop-shaded-guava}}). We checked and the byte code for the same class is indeed different between the relocated and non-relocated JARs.

This is because maven-shade-plugin, before 3.3.0, was rewriting class files even when the relocation was a "no-op". See MSHADE-391 and [apache/maven-shade-plugin#95|https://github.com/apache/maven-shade-plugin/pull/95].
{quote}Maven Shade internally uses [ASM's {{ClassRemapper}}|https://asm.ow2.io/javadoc/org/objectweb/asm/commons/ClassRemapper.html] and defines a custom {{Remapper}} subclass, which takes care of relocation, partially doing the work by itself and partially delegating to the ASM parent class. An ASM {{ClassReader}} reads each class file from the original JAR and *unconditionally* writes it into a {{{}ClassWriter{}}}, plugging in the transformer.

This transformation, even if not a single relocation (package name mapping) takes place, often leads to binary differences between original class and transformed class, because constant pool or stack map frames have been adjusted, not changing the functionality of the class, but making it look like something changed when comparing class files before and after the relocation process.
{quote}
Upgrading to maven-shade-plugin 3.3.0 fixes the unnecessary rewrite of classes.
Bug: RPC Client performance improvement
Status: Open | Priority: Minor | Resolution: 
Created: 19/Nov/22 07:03

  The current implementation copies the rpcRequest and header to a ByteArrayOutputStream in order to calculate the total length of the sent request, and then writes it to the socket buffer.

  But if the rpc engine is ProtobufRpcEngine2, we can pre-calculate the request size, and then send the request directly to the socket buffer, reducing a memory copy. And avoid allocating 1024 bytes of ResponseBuffer each time a request is sent.
Bug: Propose a mechanism to free the direct memory occupied by RPC Connections
Status: Open | Priority: Minor | Resolution: 
Created: 21/Nov/22 05:22

  In the RPC Client, a thread called RpcRequestSender is responsible for writing the connection request to the socket. Every time a request is sent, a direct memory is applied for in sun.nio.ch.IOUtil#write() and cached.

  If Connection and RpcRequestSender are promoted to the old generation, they will not be recycled when full gc is not performed, resulting in the DirectByteBuffer cached in sun.nio.ch.Util not being recycled. When the memory occupied by DirectByteBuffer is too large, the jvm process may not have the opportunity to do full gc and is killed.

  Unfortunately, there is no easy way to free these DirectByteBuffers. Perhaps, we can manually free these DirectByteBuffers by the following methods when the Connection is closed.
{code}
private void freeDirectBuffer() {
  try {
    DirectBuffer buffer = (DirectBuffer) Util.getTemporaryDirectBuffer(1);
    buffer.cleaner().clean();
  } catch (Throwable t) {
    LOG.error("free direct memory error, connectionId: " + remoteId, t);
  }
}{code}
Bug: Magic Committer optional clean up 
Status: Open | Priority: Minor | Resolution: 
Created: 12/Dec/22 08:53
Affects Version/s: 3.3.3

It seems that deleting the `__magic` folder, depending on the number of tasks/partitions used on a given spark job, can take really long time. I'm having the following behavior on a given Spark job (processing ~30TB, with ~420k tasks) using the magic committer:
{code:java}
2022-12-10T21:25:19.629Z pool-3-thread-32 INFO MagicS3GuardCommitter: Starting: Deleting magic directory s3a://my-bucket/random_hash/__magic
2022-12-10T21:52:03.250Z pool-3-thread-32 INFO MagicS3GuardCommitter: Deleting magic directory s3a://my-bucket/random_hash/__magic: duration 26:43.620s {code}
I don't see a way out of it since the deletion of s3 objects needs to list all objects under a prefix and this is what may be taking too much time. Could we somehow make this cleanup optional? (the idea would be to delegate it through s3 lifecycle policies in order to not create this overhead on the commit phase).
Bug: Special characters handling in Azure file system is different from others
Status: Open | Priority: Minor | Resolution: 
Created: 16/Dec/22 12:55
Affects Version/s: 3.2.0

Special characters handling in [AzureBlobFileSystem|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java] looks different from other file systems. e.g. GoogleHadoopFileSystem.

For instance, FileSystem.open with ".../part=a%25percent/test.parquet" works fine in other file systems, but ABFS requires ".../part=a%percent/test.parquet" (%25 → %) because the path is URL encoded by AbfsClient#createRequestUrl internally. Can we change the behavior? Or can I request to add javadoc to explain the behavior if this is the expected behavior?
Bug: [ABFS]: Partial Read should add to throttling metric
Status: Open | Priority: Minor | Resolution: 
Created: 20/Oct/22 09:29
Affects Version/s: 3.3.4

Error Description:
For partial read (due to account backend throttling), the ABFS driver retry but doesn't add up in the throttling metrics.
In case of partial read with connection-reset exception, ABFS driver retry for the full request and doesn't add up in throttling metrics.

Mitigation:
In case of partial read, Abfs Driver should retry for the remaining bytes and it should be added in throttling metrics.
Bug: Check if config value is not empty string in AbfsConfiguration.getMandatoryPasswordString()
Status: Open | Priority: Minor | Resolution: 
Created: 01/Dec/22 06:22
Affects Version/s: 3.3.4

The method `getMandatoryPasswordString` is called in `AbfsConfiguration.getTokenProvider()' to check if following configs are non-null (diff keys applicable for different implementation of AccessTokenProvider):

1. fs.azure.account.oauth2.client.endpoint: in ClientCredsTokenProvider
2. fs.azure.account.oauth2.client.id: in ClientCredsTokenProvider, MsiTokenProvider, RefreshTokenBasedTokenProvider
3. fs.azure.account.oauth2.client.secret: in ClientCredsTokenProvider
4. fs.azure.account.oauth2.client.endpoint: in UserPasswordTokenProvider
5. fs.azure.account.oauth2.user.name: in UserPasswordTokenProvider
6. fs.azure.account.oauth2.user.password: in  UserPasswordTokenProvider
7. fs.azure.account.oauth2.msi.tenant: in MsiTokenProvider
8. fs.azure.account.oauth2.refresh.token: in RefreshTokenBasedTokenProvider

Right now, this method checks if its non-null and not non-empty. This task needs to add check on non-empty config values.
Bug: DataNode's internal infoserver redirects with http scheme, not https when https enabled.
Status: Open | Priority: Minor | Resolution: 
Created: 24/Dec/22 09:28
Affects Version/s: 3.3.4

After HADOOP-16314, WebServlet.java was added. On WebServlet#doGet, it redirects '/' to '/index.html'. However, if a client connects to DataNode with https scheme, it fails to connect because it responds 302 with Location header which has http scheme.

(Hostname is modified.)

{code:java}
$ curl https://dn1.example.com:50475/ -v 2>&1 | grep Location
< Location: http://dn1.example.com:50475/index.html
{code}

I can't ensure that which solution is the best among:

- Use DefaultServlet instead of WebServlet. DataNode can answer with index.html when accessed in '/'.
- According to "dfs.http.policy" in hdfs-site.xml, run internal infoserver as https or http server each.
- Make redirection on URLDispatcher.java
Bug: Upgrade jackson-databind to a version with fixes for CVE-2022-42003 and CVE-2022-42004
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 15/Nov/22 08:35
Affects Version/s: 3.3.4

|CVE-2022-42003|
|CVE-2022-42004|

These HIGH severity CVEs are reported against hadoop-client-runtime jars of hadoop 3.3.4. These are from Twistlock security scans
Bug: TestDecayRpcScheduler.testPriority is a flaky test
Status: Open | Priority: Trivial | Resolution: 
Created: 27/Oct/22 19:08

 
{code:java}
org.apache.hadoop.ipc.TestDecayRpcScheduler.testPriority{code}
This is a flaky test, it can pass maven-test while showing non-deterministic behavior under [NonDex|[https://github.com/TestingResearchIllinois/NonDex]] and thus failed.

The test shows below:
{code:java}
java.lang.AssertionError: Get expected JMX of CallVolumeSummary before decay
    at org.junit.Assert.fail(Assert.java:89)
    at org.junit.Assert.assertTrue(Assert.java:42)
    at org.apache.hadoop.ipc.TestDecayRpcScheduler.testPriority(TestDecayRpcScheduler.java:295){code}
*Steps to reproduce the failure:*
 # install [NonDex|https://github.com/TestingResearchIllinois/NonDex]
 # run the following command in hadoop: 
{code:java}
MODULE=hadoop-common-project/hadoop-common

TEST=org.apache.hadoop.ipc.TestDecayRpcScheduler#testPriority   

mvn install -pl $MODULE -am -DskipTests 

mvn -pl $MODULE edu.illinois:nondex-maven-plugin:1.1.2:nondex -Dtest=$TEST{code}

      3. the result will be saved under the package folder in .nondex
Bug: Misleading AWS SDK S3 timeout configuration comment
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 08/Dec/22 13:10
Affects Version/s: 3.3.4

[https://github.com/apache/hadoop/blob/branch-3.3.5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L254]
{code:java}
// seconds until we give up trying to establish a connection to s3
  public static final String ESTABLISH_TIMEOUT =
      "fs.s3a.connection.establish.timeout";
  public static final int DEFAULT_ESTABLISH_TIMEOUT = 50000;

  // seconds until we give up on a connection to s3
  public static final String SOCKET_TIMEOUT = "fs.s3a.connection.timeout";
  public static final int DEFAULT_SOCKET_TIMEOUT = 200000;{code}
 
{code:java}
public static void initConnectionSettings(Configuration conf,
      ClientConfiguration awsConf) throws IOException {
    awsConf.setMaxConnections(intOption(conf, MAXIMUM_CONNECTIONS,
        DEFAULT_MAXIMUM_CONNECTIONS, 1));
    initProtocolSettings(conf, awsConf);
    awsConf.setMaxErrorRetry(intOption(conf, MAX_ERROR_RETRIES,
        DEFAULT_MAX_ERROR_RETRIES, 0));
    awsConf.setConnectionTimeout(intOption(conf, ESTABLISH_TIMEOUT,
        DEFAULT_ESTABLISH_TIMEOUT, 0));
    awsConf.setSocketTimeout(intOption(conf, SOCKET_TIMEOUT,
        DEFAULT_SOCKET_TIMEOUT, 0));
    int sockSendBuffer = intOption(conf, SOCKET_SEND_BUFFER,
{code}
AWS SDK S3 client actually consumes timeout in milliseconds
[https://github.com/aws/aws-sdk-java/blob/1.11.600/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java#L185]
Bug: Apache Hadoop support for log4j >= 2.17
Status: Resolved | Priority: Blocker | Resolution: Duplicate
Created: 20/Dec/21 16:41
Affects Version/s: 2.10.1

Federal Agencies are being given [CISA|https://www.cisa.gov/] directives requiring all  agencies to upgrade log4j 1.x applications to versions supporting log4j version 2.16.0 or higher (as of last Friday) or remove the jar files from our machines.

1.x versions of log4j are EOL, are vulnerable to multiple existing CVEs (9.8 Critical severity RCE<[https://nvd.nist.gov/vuln/detail/CVE-2019-17571]> and 8.1 High severity RCE<[https://nvd.nist.gov/vuln/detail/CVE-2021-4104]>), and due to increased scrutiny have already had a new CVE reported this week ([https://nvd.nist.gov/vuln/detail/CVE-2021-4104]<(https:/nvd.nist.gov/vuln/detail/CVE-2021-4104>).

The CISA guidance will continue to grow and improve overtime, and as of Friday 12/17/2021 CISA stated that log4j needs to be upgraded to 2.16.0 or higher.

I'm afraid Apache's statement <https://hadoop.apache.org/news/2021-12-17-log4jshell.html> will not meet the federal requirement. Please consider this an urgent request to release updated versions of Hadoop 2.x / 3.x which support log4j 2.17 or higher. Patches or workarounds would be helpful in the short term.
Bug: Document Hadoop's stance on the log4jshell vulnerability
Status: Resolved | Priority: Blocker | Resolution: Done
Created: 17/Dec/21 13:50

As of today, Hadoop relies on log4j-1, not log4j2. It is understood that the log4jshell vulnerability (CVE-2021-44228) does not impact log4j-1. Given the widespread attention to the incidence, we should make it clear that Hadoop is not susceptible to the attack.
Bug: Disable JIRA plugin for YETUS on Hadoop
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 02/Nov/21 18:05
Affects Version/s: 3.3.5

I’ve been noticing an issue with Jenkins CI where a file jira-json goes missing all of a sudden – jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)
{code}
[2021-10-27T17:52:58.787Z] Processing: https://github.com/apache/hadoop/pull/3588
[2021-10-27T17:52:58.787Z] GITHUB PR #3588 is being downloaded from
[2021-10-27T17:52:58.787Z] https://api.github.com/repos/apache/hadoop/pulls/3588
[2021-10-27T17:52:58.787Z] JSON data at Wed Oct 27 17:52:55 UTC 2021
[2021-10-27T17:52:58.787Z] Patch data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:58.787Z] Diff data at Wed Oct 27 17:52:56 UTC 2021
[2021-10-27T17:52:59.814Z] awk: cannot open /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3588/centos-7/out/jira-json (No such file or directory)
[2021-10-27T17:52:59.814Z] ERROR: https://github.com/apache/hadoop/pull/3588 issue status is not matched with "Patch Available".
[2021-10-27T17:52:59.814Z]
{code}
This causes the pipeline run to fail. I’ve seen this in my multiple attempts to re-run the CI on my PR –
 # After 45 minutes – [jenkins / hadoop-multibranch / PR-3588 / #1 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/1/pipeline/]
 # After 1 minute – [jenkins / hadoop-multibranch / PR-3588 / #2 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/2/pipeline/]
 # After 17 minutes – [jenkins / hadoop-multibranch / PR-3588 / #3 (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/3/pipeline/]

The hadoop-multibranch pipeline doesn't use ASF JIRA, thus, we're disabling the *jira* plugin to fix this issue.