av9ash/GitBugs huggingface dataset:
Bug: RBF: Namespace usage of mount table with multi subclusters can exceed quota
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 09/Nov/21 02:24

# mkdir on each subcluster /test10
 # mount table: hdfs dfsrouteradmin -add /test10 ns1,ns2 /test10
 # set quota: hdfs dfsrouteradmin -setQuota /test10 -nsQuota 4
 # touch two files under hdfs://\{fed}/test10, now the namespace usage is 4(2 directories and 2 files). 
 # refresh and put a new file, the put is successfull, and the namespace usage comes to 5.
!1636424307319.jpg!


The router checks quota without considering the increments, It is difficult to limit quota precisely without increments , but throw exception when usage >= quota will be better.
Bug: Build arm64 (aarch64) and x86_64 image with the same Dockerfile
Status: Open | Priority: Major | Resolution: 
Created: 02/Dec/21 10:01

-Support building Linux arm64 (aarch64) Docker images. And bump up some dependency versions.-

-Note: This only provides a arm64 *runtime* environment for Hadoop 3. But not a full environment for compiling Hadoop 3 in arm64 yet. For the latter, gRPC may well need to be compiled from source because it hasn't started distributing Linux arm64 binaries yet.-

-The patch for branch-3.3 is ready. I developed this patch on branch-3.3.1 when I was trying to build arm64 Linux Hadoop Docker image. For trunk (3.4.0), due to HADOOP-17727, I need to post a different PR.-

Just realized we already had {{Dockerfile_aarch64}}. Will try it out.

My approach builds the Docker images for both architectures (x86_64 and aarch64) with the same {{Dockerfile}}.

We should push the built arm64 image to Docker Hub. I only see amd64 [there|https://hub.docker.com/r/apache/hadoop/tags] so I assumed we didn't have arm64 Docker image, hmm.
Bug: Support etag-assisted renames in FileOutputCommitter
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 28/Oct/21 15:24
Affects Version/s: 3.4.0

To deal with some throttling/retry issues in object stores,
pass the FileStatus entries retrieved during listing
into a private interface ResilientCommitByRename which filesystems
may implement to use extra attributes in the listing (etag, version)
to constrain and validate the operation.

Although targeting azure, GCS and others could use. no point in S3A as they shouldn't use this committer.

# And we are not going to do any changes to FileSystem as there are explicit guarantees of public use and stability.
I am not going to make that change as the hive thing that will suddenly start expecting it to work forever.
# I'm not planning to merge this in, as the manifest committer is going to include this and more (MAPREDUCE-7341)

However, I do need to get this in on a branch, so am doing this work on trunk for dev & test and for others to review
Bug: S3AFileSystem.s3GetFileStatus() doesn't find dir markers on minio
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 19/Nov/21 21:17
Affects Version/s: 3.3.0

Repro code:
{code:java}
val conf = new Configuration()  
conf.set("fs.s3a.endpoint", "http://127.0.0.1:9000") conf.set("fs.s3a.path.style.access", "true") 
conf.set("fs.s3a.access.key", "user_access_key") 
conf.set("fs.s3a.secret.key", "password")  

val path = new Path("s3a://comcast-test")  
val fs = path.getFileSystem(conf)  
fs.mkdirs(new Path("/testdelta/_delta_log"))  
fs.getFileStatus(new Path("/testdelta/_delta_log")){code}
Fails with *FileNotFoundException fails* on Minio. The same code works in real S3.
It also works in Hadoop 3.2 with Minio and earlier versions.

Only fails on 3.3 and newer Hadoop branches.

The reason as discovered by [~sadikovi] is actually a more fundamental one - Minio does not have empty directories (sort of), see [https://github.com/minio/minio/issues/2423].

This works in Hadoop 3.2 because of this infamous "Is this necessary?" block of code
[https://github.com/apache/hadoop/blob/branch-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2204-L2223]

that was removed in Hadoop 3.3 -
[https://github.com/apache/hadoop/blob/branch-3.3.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2179]

and this causes the regression
Bug: Provide a public wrapper of Configuration#substituteVars
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/21 07:20
Affects Version/s: 3.4.0

YARN-10838 and YARN-10911 introduced an unfortunate change in Configuration that could potentially be backward incompatible (visibility of substituteVars). Since it is easily circumvented, my proposal is to avoid breaking changes if possible.

One found issue is that Oozie defines substituteVars as a private method.
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/util/XConfiguration.java#L186
Bug: Use platform specific endpoints for CI report
Status: Open | Priority: Major | Resolution: 
Created: 12/Nov/21 16:04
Affects Version/s: 3.4.0

Consider the Github comment by hadoop-yetus for different platform runs -

||Platform||hadoop-yetus report||
|Centos 7|https://github.com/apache/hadoop/pull/3563#issuecomment-947151110|
|Centos 8|https://github.com/apache/hadoop/pull/3563#issuecomment-947239385|
|Debian 10|https://github.com/apache/hadoop/pull/3563#issuecomment-947340406|
|Ubuntu Focal|https://github.com/apache/hadoop/pull/3563#issuecomment-947464004|

 
Notice that the Docker subsystem points to the same file for all the platforms -

!image-2021-11-12-21-13-18-997.png!

To illustrate the issue here, upon clicking on the link to Dockerfile in [Centos 7's hadoop-yetus report|https://github.com/apache/hadoop/pull/3563#issuecomment-947151110] will navigate to Ubuntu Focal's Dockerfile (since Ubuntu Focal wass the most recently run platform for that pipeline run).
Please note that this issue applies to all the links that contain "out/" that appear in the Yetus summary.

As a side note, this issue doesn't happen when the Yetus run fails. Consider https://github.com/apache/hadoop/pull/3563#issuecomment-946215556. Here the Yetus run has failed for Centos 7. The link to Dockerfile points to Centos 7's Dockerfile since Centos 7 is the last platform that was run for that pipeline.


----


To fix this issue, we need to do the following two things -
*+Archive the artifacts even when the Yetus run is successful+*
Currently, we're archiving the artifacts only when the run fails - https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/Jenkinsfile#L142-L148 (Hence, we're able to navigate to the right Dockerfile when the Yetus run fails for a platform). We need to archive the artifacts even when the run succeeds so that we're able to navigate to it irrespective of success/failure of the Yetus run.

*+Expose different endpoints for each platform+*
At the end of the Yetus's pre-commit run, we see the following table -

|| Subsystem || Report/Notes ||
|  cc  | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3650/1/*artifact/out*/results-compile-cc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt |
|  unit  | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3650/1/*artifact/out*/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |

We need a way to change the URL to /artifact/*centos-7*/out/, /artifact/*ubuntu-focal*/out/ and so on.
Bug: TestIPC#testIOEOnListenerAccept fails
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 13/Dec/21 08:05

{code}
[ERROR] testIOEOnListenerAccept(org.apache.hadoop.ipc.TestIPC)  Time elapsed: 0.007 s  <<< FAILURE!
java.lang.AssertionError: Expected an EOFException to have been thrown
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.hadoop.ipc.TestIPC.testIOEOnListenerAccept(TestIPC.java:652)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)
{code}
Bug: RPCMetrics increases the number of handlers in processing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/21 04:04
Affects Version/s: 2.9.2

When using RPC, we recorded a lot of useful information, such as Queue time, Processing time. These are very helpful.
But we can't know how many handlers are actually working now (only those that handle Call), especially when the Call Queue is very high. This is also not conducive to us optimizing the cluster.
It would be very helpful if we can see the number of handlers being processed in RPCMetrics.
Bug: [branch-3.3] Dockerfile_aarch64 build fails with fatal error: Python.h: No such file or directory
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/21 22:48

See previous discussion: https://issues.apache.org/jira/browse/HADOOP-17723?focusedCommentId=17452329&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17452329
Bug: Command to refresh auth to local rules without having to restart the cluster
Status: Open | Priority: Major | Resolution: 
Created: 10/Dec/21 03:02

when two kerberos cluster establish cross realm trust, we should set up hadoop.security.auth_to_local parameter in both clusters, and then restart cluster。restart namenode is very challenge thing for large cluster, so we can add admin command to refresh auth to local rules without restart cluster.
Bug: Skip unit test failures to run all the unit tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/21 10:31

In branch-3.3 and upper, unit tests failures are ignored in HADOOP-16596. That way we can run all the unit tests in all the modules by simply modifying a file under the project root directory. Without the feature, if there is a test failure in a module (it is likely happen due to the flaky jobs), the tests in the subsequent modules are not executed. I want to introduce the feature in the other branches.
Bug: Authentication cookie will never expire by default after HADOOP-12049
Status: Open | Priority: Major | Resolution: 
Created: 02/Dec/21 06:37
Affects Version/s: 3.1.0

Whlie create auth cookie for client, AuthenticationFilter will add "Expires"  attribute for the cookie if needed. But after https://issues.apache.org/jira/browse/HADOOP-12049, it never enter the code block by default.

 

 
{code:java}
// AuthenticationFilter

public static void createAuthCookie(HttpServletResponse resp, String token,
                                    String domain, String path, long expires,
                                    boolean isCookiePersistent,
                                    boolean isSecure) {
  //...
  //By default, isCookiePersistent = false
  if (expires >= 0 && isCookiePersistent) {
    Date date = new Date(expires);
    SimpleDateFormat df = new SimpleDateFormat("EEE, " +
            "dd-MMM-yyyy HH:mm:ss zzz");
    df.setTimeZone(TimeZone.getTimeZone("GMT"));
    sb.append("; Expires=").append(df.format(date));
  }

  //...
}{code}
Bug: ITestAzureBlobFileSystemDelete failing "Operations has null HTTP response"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/21 13:37
Affects Version/s: 3.3.2

ITestAzureBlobFileSystemDelete .testDeleteIdempotency failing



looks like caused by HADOOP-17934 and not a bug in that patch, just a mockito based test case which needs fixing now a new method is called.


{code} 

java.lang.IllegalArgumentException: Operations has null HTTP response

	at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:144)
	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.deleteIdempotencyCheckOp(AbfsClient.java:830)
	at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelete.testDeleteIdempotency(ITestAzureBlobFileSystemDelete.java:196)

{code}
Bug: Make certain methods LimitedPrivate in S3AUtils.java 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/21 07:02
Affects Version/s: 3.4.0

S3AUtils(Private) have certain methods which can be used in external projects to provide proxy support, since it is private we can't call them directly and have to copy the whole methods to provide the support. Making these methods LimitedPrivate so that these can be called directly with a reason.
Bug: Public Interface to use S3AUtils.java methods in external projects
Status: Open | Priority: Major | Resolution: 
Created: 19/Nov/21 06:58
Affects Version/s: 3.4.0

S3AUtils is marked Private, this makes it difficult to use the methods in it and the projects have to copy the whole method rather than simply calling them. Having a public Interface helper class would act as a gateway to these methods which can be directly called now.
Bug: some time delay (0.3s) for swebhdfs + kerberos + observer setting.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 14/Nov/21 22:37
Affects Version/s: 3.3.1

Settings:

1 master namenode (A), 1 standby namenode (B), 1 observer namenode (C).

following 

[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html]

except that 

dfs.client.failover.observer.auto-msync-period.<nameservice>

is set to -1 (not auto -msync)

 

uable to do curl - - negotiate -u ':' 'https://<observer>:<port>/webhdfs/v1/...'

because it seems like due to the following issue:

https://issues.apache.org/jira/browse/HDFS-14443

using curl --negotiate -u ':' 'https://<master>:<port>/webhdfs/v1/...'

can successfully get 307 redirect with the corresponding Location.

but got 

token (token for xxx HDFS_DELEGATION_TOKEN owner=xxx renewer=xxx masterKeyID=ooo) can't be found in cache"

if redirect the url within 300ms.

 

Not issue if waiting for more than 300ms and then do the redirect.

No issue if changing (C) to Standby (no observers) (and redirect within 10 ms)
Bug: Failing concurrent FS.initialize commands when fs.azure.createRemoteFileSystemDuringInitialization is enabled on hadoop-azure ABFS
Status: Open | Priority: Major | Resolution: 
Created: 04/Nov/21 16:45
Affects Version/s: 3.3.1

*Bug description:*

When {{fs.azure.createRemoteFileSystemDuringInitialization}} is enabled, the filesystem will create a container if it does not already exist inside the {{initialize}} method. The current flow of creating the container will fail in the case of concurrent {{initialize}} methods being executed simultaneously (only one request can create the container, the rest will fail instead of moving on). This is happen due to the `checkException` method that is not catching the Hadoop `FileAlreadyExists` exception.

Stacktrace:

{{Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed: "The specified filesystem already exists.", 409, PUT, https://<REDACTED>.dfs.core.windows.net/project?resource=filesystem, FilesystemAlreadyExists, "The specified filesystem already exists. RequestId:<REDACTED> Time:2021-10-18T13:46:05.7504906Z"}}
 {{ {{at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1182)}}}}
 {{ {{at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.createFileSystem(AzureBlobFileSystem.java:1067)}}}}
 {{ {{at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:126)}}}}
 {{ {{at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)}}}}

*To reproduce:*
 * Set `fs.azure.createRemoteFileSystemDuringInitialization` to `true`
 * Run two concurrent `initialize` commands with the root to the non existing container/filesystem.

 

*Proposed fix:*

[https://github.com/apache/hadoop/pull/3620]
Bug: Exclude ASF license check for pkg-resolver JSON
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Oct/21 14:14
Affects Version/s: 2.10.0

There's no way to add comments to a JSON file. Need to exclude the following files from ASF license checks since they're JSON files -

1. dev-support/docker/pkg-resolver/packages.json
2. dev-support/docker/pkg-resolver/platforms.json
Bug: Improve CustomTokenProviderAdapter to import VisibleForTesting
Status: Open | Priority: Major | Resolution: 
Created: 05/Nov/21 14:06

Recently, some new features have been added to CustomTokenProviderAdapter, which is of course very good.
Here is the introduction of'org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting'. When using Maven to compile the hadoop project, an error occurred as expected.
E.g:
org.apache.maven.enforcer.rule.api.EnforcerRuleException: 
Banned imports detected:

Reason: Use hadoop-annotation provided VisibleForTesting rather than the one provided by Guava
	in file: org/apache/hadoop/fs/azurebfs/oauth2/CustomTokenProviderAdapter.java
		org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting (Line: 25, Matched by: org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting)

Analysis took 0 seconds

    at de.skuzzle.enforcer.restrictimports.rule.RestrictImports.execute (RestrictImports.java:70)
    at org.apache.maven.plugins.enforcer.EnforceMojo.execute (EnforceMojo.java:202)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

In the end it was unsuccessful.
In addition, I want to explain that this happened in a Mac environment.

Obviously, we should introduce'org.apache.hadoop.classification.VisibleForTesting'.
Bug: S3A SSE-KMS inconsistency issue during rename
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 13/Oct/21 04:58
Affects Version/s: 3.1.2

According to the document:
 [https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/encryption.html#S3_Default_Encryption]
 "Organizations may define a default key in the Amazon KMS; if a default key is set, then it will be used whenever SSE-KMS encryption is chosen and the value of fs.s3a.server-side-encryption.key is empty."

So basically two conditions to make the object with default KMS: 1. Set SSE-KMS encryption 2. Did not set fs.s3a.server-side-encryption.key

But there is another confusing scenario below:

1. User want to rely on s3 bucket side encryption using their customer KMS key(kms-keyA, for example), so user did not set fs.s3a.server-side-encryption-algorithm or fs.s3a.server-side-encryption.key, and the files uploaded to this bucket will use bucket custom KMS key kms-keyA
 2. Next step, user want to copy the file to other file using s3a, the process will invoke copyFile() in S3AFileSystem, during the copy, s3a will clone the meta data of the source in cloneObjectMetadata(), in the clone, there is copy of SSE algorithm but no specific kms key copy for the SSE-KMS, it will cause the destination using SSE-KMS without any key id, the final file will use account level default key under aws/s3(
 [https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object.html),]
 lets say its kms-keyB.

It means when ever there is a copy, the kms key will be changed from customer key kms-keyA to kms-keyB, which will cause inconsistency, for example:

hdfs dfs -put test s3://ssetest/

During this put, there will be rename processing from test.__COPYING__ to test, it will cause the final test file encrypted with account default key kms-keyB instead of s3 bucket customer key kms-keyA which is expected. Should we consider to clone the KMS key id also to keep the consistency?
Bug: Backport HADOOP-17683 for branch-3.2 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/21 05:09
Affects Version/s: 3.2.2

Our security tool raised the following security flaw on Hadoop 3.2.2: 

CVE-2021-29425: [https://nvd.nist.gov/vuln/detail/CVE-2021-29425]
Bug: [hadoop-azure] Support Azure Active Directory auth with hadoop config (azureblob storage)
Status: Open | Priority: Major | Resolution: 
Created: 21/Oct/21 13:28
Affects Version/s: 3.3.1

For azure blob storage there's 3 auth types: Shared key, SAS and Active Directory (via app, using tenantId, appID and appSecret)

 

Currently it's not possible to configure hadoopConfiguration to use the third way in hadoop-azure driver for azure blob storage. Oauth is supported only for azure data lake storage gen2, but not for wasb(s) containers.

 

Is it possible to implement it either?
Bug: Increase Java heap size for running Maven in Dockerfile of branch-2.10
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/21 15:02

I got OOM on running create-release script in branch-2.10.

{noformat}
$ ./dev-support/bin/create-release --docker --dockercache --native
$ tail -n 5 patchprocess/mvn_install.log
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:06 min
Exception in thread "main"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
{noformat}