av9ash/GitBugs huggingface dataset:
Bug: Hadoop GPG key is not valid
Status: Resolved | Priority: Blocker | Resolution: Done
Created: 09/Nov/20 09:08

Hi,

I build regular hadoop images.

At the moment the KEYS file ([https://dist.apache.org/repos/dist/release/hadoop/common/KEYS]) is invalid.

Error-Message:
{code:java}
gpg: key FC8D04357BB49FF0: public key "Sammi Chen (CODE SIGNING KEY) <sammichen@apache.org>" imported
gpg: invalid armor header: mQINBF9U5ZcBEADJS2a8ihhZtN1wXOJfyLZreuHL9HJxRvogQbhrhpFQrKAusdf2\n
gpg: CRC error; 95D523 - 51AC03
gpg: packet(7) with unknown version 103
gpg: read_block: read error: Unknown version in packet
gpg: import from '/tmp/KEYS' failed: Invalid keyring
gpg: Total number processed: 60
gpg:               imported: 60
gpg: no ultimately trusted keys found
{code}
Steps to reproduce:
 * Install Docker
 * Create a Dockerfile with the following Content
 ** 
{code:java}
FROM alpine:3.12
RUN set -ex && \
    /sbin/apk add --no-cache wget gnupg tar && \
    # Install Hadoop
    /usr/bin/wget -nv https://dist.apache.org/repos/dist/release/hadoop/common/KEYS -O /tmp/KEYS && \
    /usr/bin/gpg --import /tmp/KEYS
 {code}

 * Run docker build
 ** 
{code:java}
 docker build -t hadoop-test -f Dockerfile .
{code}

Other users complaining about this error:
 - [https://askubuntu.com/questions/1290190/hadoop-gpg-key-is-not-valid-no-ultimately-trusted-keys-found]
 - [https://stackoverflow.com/questions/64719392/invalid-keyring-are-hadoop-gpg-keys-are-wrong]

I hope for a quick fix, because automatic builds are currently blocked.

If this duplicates another ticket, please close it and link it to the original.

Best Regards
 Reamer
Bug: Don't relocate org.bouncycastle in shaded client jars
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 22/Oct/20 21:33
Affects Version/s: 3.3.0

When downstream apps depend on {{hadoop-client-api}}, {{hadoop-client-runtime}} and {{hadoop-client-minicluster}}, it seems the {{MiniYARNCluster}} could have issue because {{org.apache.hadoop.shaded.org.bouncycastle.operator.OperatorCreationException}} is not in any of the above jars. 

{code}
Error:  Caused by: sbt.ForkMain$ForkError: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.org.bouncycastle.operator.OperatorCreationException
Error:  	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
Error:  	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
Error:  	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
Error:  	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
Error:  	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:862)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1296)
Error:  	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:339)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:353)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster.access$200(MiniYARNCluster.java:127)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceInit(MiniYARNCluster.java:488)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:321)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.spark.deploy.yarn.BaseYarnClusterSuite.beforeAll(BaseYarnClusterSuite.scala:94)
Error:  	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
Error:  	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
Error:  	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
Error:  	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:61)
Error:  	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
Error:  	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
Error:  	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
Error:  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Error:  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Error:  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Error:  	at java.lang.Thread.run(Thread.java:748)
{code}
Bug: NPE when starting MiniYARNCluster from hadoop-client-minicluster
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 24/Oct/20 00:16
Affects Version/s: 3.3.0

When starting MiniYARNCluster one could get the following exception:
{code}
  java.lang.NullPointerException:
  at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:72)
  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:122)
  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  at org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper.serviceStart(MiniYARNCluster.java:616)
  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:122)
  at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:327)
  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  at org.apache.spark.deploy.yarn.BaseYarnClusterSuite.beforeAll(BaseYarnClusterSuite.scala:96)
  ...
{code}

Looking into the code, this is because we explicitly exclude resource files under {{hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL}}, and therefore this code in {{WebServer}} fails with NPE:
{code}
    terminalParams.put("resourceBase", WebServer.class
        .getClassLoader().getResource("TERMINAL").toExternalForm());
{code}

Those who use {{hadoop-minicluster}} may not be affected because they'll also need {{hadoop-yarn-server-nodemanager}} as an extra dependency, which includes the resource files. On the other hand {{hadoop-client-minicluster}} packages both test classes (e.g., {{MiniYARNCluster}}) as well as main classes (e.g., {{ResourceManager}} and {{NodeManager}}) into a single shaded jar. It should include these resource files for testing as well. Otherwise, {{MiniYARNCluster}} is unusable.
Bug: WASB : PageBlobOutputStream succeeding hflush even when underlying flush to storage failed 
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 16/Oct/20 10:33
Affects Version/s: 2.7.0

In PageBlobOutputStream, write()  APIs will fill the buffer and hflush/hsync/flush call will flush the buffer to underlying storage. Here the Azure calls are handled in another thread 
{code}
private synchronized void flushIOBuffers()  {
    ...
    lastQueuedTask = new WriteRequest(outBuffer.toByteArray());
    ioThreadPool.execute(lastQueuedTask);
  ....
 }
private class WriteRequest implements Runnable {
    private final byte[] dataPayload;
    private final CountDownLatch doneSignal = new CountDownLatch(1);

    public WriteRequest(byte[] dataPayload) {
      this.dataPayload = dataPayload;
    }

    public void waitTillDone() throws InterruptedException {
      doneSignal.await();
    }

    @Override
    public void run() {
      try {
        LOG.debug("before runInternal()");
        runInternal();
        LOG.debug("after runInternal()");
      } finally {
        doneSignal.countDown();
      }
    }

    private void runInternal() {
      ......
      writePayloadToServer(rawPayload);
      ...........
    }

    private void writePayloadToServer(byte[] rawPayload) {
      ......
      try {
        blob.uploadPages(wrapperStream, currentBlobOffset, rawPayload.length,
            withMD5Checking(), PageBlobOutputStream.this.opContext);
      } catch (IOException ex) {
        lastError = ex;
      } catch (StorageException ex) {
        lastError = new IOException(ex);
      }
      if (lastError != null) {
        LOG.debug("Caught error in PageBlobOutputStream#writePayloadToServer()");
      }
    }
  }
{code}
The flushing thread will wait for the other thread to complete the Runnable WriteRequest. Thats fine. But when some exception happened while blob.uploadPages, we just set that to lastError state variable.  This variable is been checked for all subsequent ops like write, flush etc.  But what about the current flush call? that is silently being succeeded.!!  
In standard Azure backed HBase clusters WAL is on page blob. This issue causes a serious issue in HBase and causes data loss! HBase think a WAL write was hflushed and make row write successful. In fact the row was never gone to storage.

Checking the lastError variable at the end of flush op will solve the issue. Then we will throw IOE from this flush() itself.
Bug: ABFS: read-ahead error reporting breaks buffer management
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 07/Oct/20 23:08
Affects Version/s: 3.3.0

When reads done by readahead buffers failed, the exceptions where dropped and the failure was not getting reported to the calling app. 

Jira HADOOP-16852: Report read-ahead error back

tried to handle the scenario by reporting the error back to calling app. But the commit has introduced a bug which can lead to ReadBuffer being injected into read completed queue twice.
Bug: Use S3 content-range header to update length of an object during reads
Status: Open | Priority: Major | Resolution: 
Created: 08/Dec/20 13:39
Affects Version/s: 3.3.0

As part of all the openFile work, knowing full length of an object allows for a HEAD to be skipped. But: code knowing only the splits don't know the final length of the file.

If the content-range header is used, then as soon as a single GET is initiated against an object, if the field is returned then we can update the length of the S3A stream to its real/final length

Also: when any input stream fails with an EOF exception, we can distinguish stream-interrupted from "no, too far"
Bug: [JDK 11] Upgrade dnsjava to remove illegal access warnings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/20 11:00
Affects Version/s: 3.4.0

Originally reported by [~kihwal] in https://issues.apache.org/jira/browse/HADOOP-15338?focusedCommentId=17129854&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17129854

When running FsShell commands, there are some warning messages as follows:
{noformat}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.xbill.DNS.ResolverConfig  to method sun.net.dns.ResolverConfiguration.open()
WARNING: Please consider reporting this to the maintainers of org.xbill.DNS.ResolverConfig
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
{noformat}
Bug: Upgrade to jQuery 3.5.1 in hadoop-sls
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/20 07:29
Affects Version/s: 3.3.1

jQuery is upgraded from 3.3.1 to 3.5.1 at 

hadoop/hadoop-tools/hadoop-sls/src/main/html/js/thirdparty
Bug: [JDK 15] TestDNS fails by UncheckedIOException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/20 11:01
Affects Version/s: 3.4.0

After [JDK-8235783|https://bugs.openjdk.java.net/browse/JDK-8235783], DatagramSocket::connect throws UncheckedIOException if connect fails.
{noformat}
[INFO] Running org.apache.hadoop.net.TestDNS
[ERROR] Tests run: 12, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 0.403 s <<< FAILURE! - in org.apache.hadoop.net.TestDNS
[ERROR] testNullDnsServer(org.apache.hadoop.net.TestDNS)  Time elapsed: 0.134 s  <<< ERROR!
java.io.UncheckedIOException: java.net.SocketException: Unsupported address type
	at java.base/sun.nio.ch.DatagramSocketAdaptor.connect(DatagramSocketAdaptor.java:120)
	at java.base/java.net.DatagramSocket.connect(DatagramSocket.java:341)
{noformat}
Full error log: https://gist.github.com/aajisaka/2a24cb2b110cc3d19f7dec6256db6844
Bug: hadoop-client-integration-tests doesn't work when building with skipShade
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/20 00:04
Affects Version/s: 3.3.1

Compiling with skipShade:
{code}
mvn clean install -Pdist -DskipShade -DskipTests -Dtar -Danimal.sniffer.skip -Dmaven.javadoc.skip=true
{code}

fails with
{code}
[ERROR] /Users/chao/git/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java:[47,37] package org.apache.hadoop.yarn.server does not exist
[ERROR] /Users/chao/git/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java:[59,11] cannot find symbol
[ERROR]   symbol:   class MiniYARNCluster
[ERROR]   location: class org.apache.hadoop.example.ITUseMiniCluster
[ERROR] /Users/chao/git/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java:[82,23] cannot find symbol
[ERROR]   symbol:   class MiniYARNCluster
[ERROR]   location: class org.apache.hadoop.example.ITUseMiniCluster
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hadoop-client-integration-tests
{code}
Bug: LazyPersist Overwrite fails in direct write mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Oct/20 17:09
Affects Version/s: 3.3.1

{noformat}
hadoop fs -put -f -d -l Dockerfile /vol/buck/key/file1{noformat}
fails with {{File exists}} if the file is already there despite of the overwrite flag 

The overwrite flag is missing in create in case of {{LazyPersist}}
Bug: Touch command with -c  option is broken
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/20 17:58
Affects Version/s: 3.2.3

The touch command has an option -c :
{noformat}
touch
Usage: hadoop fs -touch [-a] [-m] [-t TIMESTAMP] [-c] URI [URI ...]{noformat}
Ref : [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#touch]

But on using it, it gives error as :
{noformat}
-touch: Illegal option -c
Usage: hadoop fs [generic options]{noformat}
Bug: KMS should log full UGI principal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/20 21:51
Affects Version/s: 3.2.2

[~daryn] reported that the kms-audit log only logs the short username:
{{OK[op=GENERATE_EEK, key=key1, user=hdfs, accessCount=4206, interval=10427ms]}}

In this example, it's impossible to tell which NN(s) requested EDEKs when they are all lumped together.
Bug: KMS ACL: Allow DeleteKey Operation to Invalidate Cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/20 19:33
Affects Version/s: 3.4.0

HADOOP-17208 send invalidate cache for key being deleted. The invalidate cache operation itself requires ROLLOVER permission on the key. This ticket is opened to fix the issue caught by TestKMS.testACLs.
Bug: Use Yetus before YETUS-994 to enable adding comments to GitHub
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/20 02:36
Affects Version/s: 3.4.0

After YETUS-994, test-patch updates the commit status instead of adding a comment in GitHub. Use the commit hash before YETUS-994 because of the following reasons:

* The Yetus feature seems to be broken for now (Trying and debugging in https://github.com/apache/hadoop/pull/2348)
* Use the fixed version to avoid surprising the developers
* The latest Yetus version is 0.12.0 but it does not support YETUS-972, which is required to run javadoc with Java 11.
Bug: Update the checkstyle config to ban some guava functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/20 07:34
Affects Version/s: 3.4.0

Some guava functions are banned in HADOOP-17111, HADOOP-17099, and HADOOP-17101 via checkstyle, however, the checkstyle configuration does not work after HADOOP-17288 because the package names have been changed.

Originally reported by [~ahussein] in HADOOP-17315.
Bug: mvn site commands fails due to MetricsSystemImpl changes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Oct/20 15:20
Affects Version/s: 3.3.1

When prepare for branch-3.2.2 release, i found there is one issue while create release. And it also exist in trunk.
command line: mvn install site site:stage -DskipTests -DskipShade -Pdist,src -Preleasedocs,docs
failed log show as the following: 
{quote}[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.6:site (default-site) on project hadoop-common: failed to get report for org.apache.maven.plugins:maven-dependency-plugin: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-common: Compilation failure
[ERROR] /Users/hexiaoqiao/Source/hadoop-common/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java:[298,5] method <T>register(java.lang.String,java.lang.String,T) is already defined in class org.apache.hadoop.metrics2.impl.MetricsSystemImpl{quote}

I am not sure why source code of class MetricsSystemImpl will be changed while building, I try to revert HADOOP-17081 everything seems OK.
Bug: Harmonize guava version and shade guava in yarn-csi
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Nov/20 17:33
Affects Version/s: 3.4.0

yarn-csi defines a separate guava version [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml#L30]. 

 

We should harmonize the guava version (pull it from hadoop-project/pom.xml) and use the shaded guava classes.
Bug: Move Jenkinsfile outside of the root directory
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/20 05:56
Affects Version/s: 3.3.1

The Jenkinsfile is placed under the project root directory, so when the Jenkinsfile is changed, all the Hadoop unit tests will run and it wastes a lot of time and resources. Let's move the file outside of the root directory.
Bug: Replace HTrace with No-Op tracer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/20 17:41
Affects Version/s: 3.3.2

Remove HTrace dependency as it is depending on old jackson jars. Use a no-op tracer for now to eliminate potential security issues.

The plan is to move part of the code in [PR#1846|https://github.com/apache/hadoop/pull/1846] out here for faster review.
Bug: [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/20 01:49
Affects Version/s: 3.3.1

JEP 396 (Strongly Encapsulate JDK Internals by Default) has been migrated since Java 16 EA Build 28. Calling the internal APIs (except the critical APIs such as sun.misc.Unsafe) are banned by default.
{noformat}
[INFO] Running org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
[ERROR] Tests run: 16, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 17.323 s <<< FAILURE! - in org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
[ERROR] testNonDefaultNonBrowserUserAgentAsNonBrowser(org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler)  Time elapsed: 1.272 s  <<< ERROR!
java.lang.IllegalAccessException: class org.apache.hadoop.security.authentication.util.KerberosUtil cannot access class sun.security.jgss.GSSUtil (in module java.security.jgss) because module java.security.jgss does not export sun.security.jgss to unnamed module @48967c8b
	at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:385)
	at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:687)
	at java.base/java.lang.reflect.Field.checkAccess(Field.java:1096)
	at java.base/java.lang.reflect.Field.get(Field.java:417)
	at org.apache.hadoop.security.authentication.util.KerberosUtil.getOidInstance(KerberosUtil.java:90)
{noformat}
Bug: fs.s3a.buffer.dir to be under Yarn container path on yarn applications
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/20 16:55
Affects Version/s: 3.3.0

# fs.s3a.buffer.dir defaults to hadoop.tmp.dir which is /tmp or similar
# we use this for storing file blocks during upload
# staging committers use it for all files in a task, which can be a lot more
# a lot of systems don't clean up /tmp until reboot -and if they stay up for a long time then they accrue files written through s3a staging committer from spark containers which fail

Fix: use ${env.LOCAL_DIRS:-${hadoop.tmp.dir}}/s3a as the option so that if env.LOCAL_DIRS is set is used over hadoop.tmp.dir. YARN-deployed apps will use that for the buffer dir. When the app container is destroyed, so is the directory.