av9ash/GitBugs huggingface dataset:
Bug: Optimize NetworkTopology while sorting of block locations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/20 15:29

In {{NetworkTopology}}, I noticed that there are some hanging fruits to improve the performance.

Inside {{sortByDistance}}, collections.shuffle is performed on the list before calling {{secondarySort}}.

{code:java}
Collections.shuffle(list, r);
if (secondarySort != null) {
  secondarySort.accept(list);
}
{code}

However, in different call sites, {{collections.shuffle}} is passed as the secondarySort to {{sortByDistance}}. This means that the shuffle is executed twice on each list.
Also, logic wise, it is useless to shuffle before applying a tie breaker which might make the shuffle work obsolete.

In addition, [~daryn] reported that:
* topology is unnecessarily locking/unlocking to calculate the distance for every node
* shuffling uses a seeded Random, instead of ThreadLocalRandom, which is heavily synchronized
Bug: S3AInputStream to be resilient to faiures in abort(); translate AWS Exceptions
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 19/Oct/20 11:01
Affects Version/s: 3.2.1

Stack overflow issue complaining about ConnectionClosedException during S3AInputStream close(), seems triggered by an EOF exception in abort. That is: we are trying to close the stream and it is failing because the stream is closed. oops.

https://stackoverflow.com/questions/64412010/pyspark-org-apache-http-connectionclosedexception-premature-end-of-content-leng

Looking @ the stack, we aren't translating AWS exceptions in abort() to IOEs, which may be a factor.
Bug: Update Jetty hadoop dependency
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Dec/20 11:12
Affects Version/s: 3.2.1

Vulnerability fixes needed for Jetty hadoop dependency library

The jetty jars where CVEs are found are ,

================ =====

Jetty [version 9.4.20.v20190813 ]

jetty-server-9.4.20.v20190813.jar
CVE details :- [ CVE-2020-27216 ]
================ =====

Jetty-http [version 9.4.20.v20190813 ]

jetty-http-9.4.20.v20190813.jar
CVE details :- [ CVE-2020-27216 ]
Bug: Jetty 9.4.20 can't generate resourceBase with NPE
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 29/Dec/20 22:40

While I was looking into TestDistributedShell logs, I noticed the following {{Warning}}


{code:bash}
2020-12-29 16:22:26,379 INFO  [Time-limited test] handler.ContextHandler (ContextHandler.java:doStart(824)) - Started o.e.j.s.ServletContextHandler@75389179{logs,/logs,file:///hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/log,AVAILABLE}
2020-12-29 16:22:26,380 INFO  [Time-limited test] handler.ContextHandler (ContextHandler.java:doStart(824)) - Started o.e.j.s.ServletContextHandler@116ed75c{static,/static,jar:file:~/.m2/repository/org/apache/hadoop/hadoop-yarn-common/3.4.0-SNAPSHOT/hadoop-yarn-common-3.4.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
2020-12-29 16:22:26,390 WARN  [Time-limited test] webapp.WebInfConfiguration (WebInfConfiguration.java:getCanonicalNameForWebAppTmpDir(794)) - Can't generate resourceBase as part of webapp tmp dir name: java.lang.NullPointerException
2020-12-29 16:22:26,469 INFO  [Time-limited test] util.TypeUtil (TypeUtil.java:<clinit>(201)) - JVM Runtime does not support Modules
{code}

For OS X, it looks like {{webAppContext.setBaseResource}} and accessing the sources from a jar file will cause {{file.resource.toURI().getPath()}} to return {{null}} for {{jar:-urls}}

I checked that changing the jetty-version  from {{9.4.20.v20190813}} to something above {{9.4.21}} (aka., 9.4.23.v20191118) fixes the warning.

[~inigoiri], [~aajisaka], [~weichiu], [~ayushtkn]
Do you guys think we should consider upgrading Jetty to the [latest versions of 9.4.x|https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-webapp] like 9.4.35?
Bug: Skip license check on lz4 code files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/20 00:33

There are some files against the asflicense check which will make _This commit cannot be built_ when building,  this is caused by the following two files:
{noformat}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/lz4/lz4.c  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/lz4/lz4.h{noformat}
The two do not have an Apache license header, maybe we should skip check that to let the buildings go.
Bug: ABFS: Set default ListMaxResults to max server limit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/20 07:33
Affects Version/s: 3.3.0

{{Changing the default value of maximum size of }}{{results to be}}{{ returned by ListStatus from 500 to 5000, since the maximum number of items supported by a listStatus server call is 5000.}}
Bug: Remove Hadoop 2.9.2 from the download page
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/20 07:14

2.9.x is EoL: https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
Let's remove 2.9.2 from https://hadoop.apache.org/releases.html
Bug: When `fs.s3a.connection.ssl.enabled=true`,   Error when visit S3A with AKSK
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 07/Dec/20 02:23

When we update hadoop version from hadoop-3.2.1 to hadoop-3.3.0, Use AKSK access s3a with ssl enabled, then this error happen
{code:java}
<property>   
    <name>ipc.client.connection.maxidletime</name> 
   <value>20000</value> 
</property> 
<property>
    <name>fs.s3a.secret.key</name> 
    <value></value> 
</property> 
<property>   
    <name>fs.s3a.access.key</name> 
    <value></value>
</property> 
<property> 
    <name>fs.s3a.aws.credentials.provider</name> 
    <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
</property>
{code}
!image-2020-12-07-10-25-51-908.png!
Bug: Skipping network I/O in S3A getFileStatus(/) breaks some tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/20 09:11
Affects Version/s: 3.3.1

[*ERROR*]   *ITestS3ABucketExistence.testNoBucketProbing:65->expectUnknownStore:102 Expected a org.apache.hadoop.fs.s3a.UnknownStoreException to be thrown, but got the result: : S3AFileStatus\{path=s3a://random-bucket-57a85110-c715-4db2-a049-e43999d7e51b/; isDirectory=true; modification_time=0; access_time=0; owner=mthakur; group=mthakur; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null*

[*ERROR*]   *ITestS3AInconsistency.testGetFileStatus:114->Assert.fail:88 getFileStatus should fail due to delayed visibility.*

 

*ITestMarkerTool.testRunWrongBucket:227->AbstractMarkerToolTest.runToFailure:276 » UnknownStore*
Bug: Reduce UGI overhead in token ops 
Status: Resolved | Priority: Major | Resolution: Later
Created: 08/Dec/20 17:46

{{DelegationTokenIdentifier}} has a {{ugiCache}} but  AbstractDelegationTokenManager calls a static method {{getRemoteUser()}} which would bypass the cache.

Performance analysis of the KMS revealed the RPC server layer is creating many and redundant UGI instances.. UGIs are not cheap to instantiate, require synchronization, and waste memory. Reducing instantiations will improve the performance of the ipc readers.
Bug: Add reload option to RefreshCallQueue
Status: Patch Available | Priority: Major | Resolution: 
Created: 11/Dec/20 08:22

Currently the can use the dfsadmin command of "refreshCallQueue" to refresh all configs of FairCallQueue, but it will cause call queue spike during the refresh on NameNode.

In  [HADOOP-17421|https://issues.apache.org/jira/projects/HADOOP/issues/HADOOP-17421?filter=allopenissues], we added some configurations to specify queues for some static users, with which we can have fine-grained control on the behavior of different users manually, this feature will require us to refresh call queue of NameNodes constantly, which will sure cause bad impacts to NameNode's performance.

This ticket was to propose adding a reload option for the command of refreshCallQueue, which will only trigger the _callQueueManager_ to reload something, instead of reconstructing new scheduler and new queues.

The basic design is :
 # Add a method of "reload" in the interface of _Scheduler._
 # Add different _RefreshCallQueueTypes_ to trigger different refresh operations.
 # Add the argument in DFSAdmin for *reload* option.
Bug: Specify user's queue via configuration in FairCallQueue 
Status: Patch Available | Priority: Major | Resolution: 
Created: 09/Dec/20 07:26

The feature of FairCallQueue helps a lot in maintaining a fair and good service in a multi-tenant cluster, each user is assigned to queues with different priority to reach this goal. But in production, we met some problems that the automatic assignment won't fit, the problems are as follows:
 # We have a service account that would send more NN requests, for some reasons, we would like to keep this user and allow this user to keep this volume of operations. When we deployed FairCallQueue, this service user would be treated as a bad user and assigned to a lower queue, causing some slowness on the service account.
 # We are having more Flink jobs writing checkpoints to our NN, and the checkpoint operations have a characteristic that they would have a periodically high cost on the NN with an interval of several minutes. FairCallQueue (with cost-based enabled) doesn't have good control of this kind of operations because when this kind of operations starts, the cost in the decay window of this user is quite low, so the user will be assigned to queue 0, after some windows, when the users' high cost has got the attention and assigned to a lower queue, the user's operations are already finished. 

For problem 1, we noticed that there is already an option mentioned in HADOOP-17165, but in our case, the service account isn't that important that we'd allow it to always be assigned to queue 0. 

To solve these problems, we'd like to raise a solution by specifying the queue for some static users via config. The basic design is as follows:
 * Specify the static users in config for each queue.
 * Load the mapping from the config while initializing the callqueue.
 * Check the configured queue for each user when assigning the queue.
 * The cost time of the static users would not be count in our decay calculation to mitigate the impacts on other normal users' costs.
Bug: Replace HTrace with NoOp tracer
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 18/Nov/20 22:01

HADOOP-17171 raised the concern that the deprecated htrace binaries has a few CVEs in its dependency jackson-databind. Not that HADOOP-15566 may not be merged any time soon. We can replace the existing htrace impl with a noop tracer (dummy).

This could be realized by reusing some code in HADOOP-15566's PR.
Bug: Javadoc failure in hadoop-aws (branch-3.3)
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Dec/20 09:29

In branch-3.3, javadoc is failing in hadoop-aws module.
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws.txt
{noformat}
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2522/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ListingOperationCallbacks.java:81: error: unexpected text
[ERROR] * {@link this.getMaxKeys()}.
[ERROR] ^
{noformat}
Found in https://github.com/apache/hadoop/pull/2522#issuecomment-741996431
Bug: ERROR conf.Configuration: error parsing conf core-site.xml com.ctc.wstx.exc.WstxParsingException: Illegal processing instruction target ("xml"); xml (case insensitive) is reserved by the specs.
Status: Open | Priority: Major | Resolution: 
Created: 09/Dec/20 16:54
Affects Version/s: 2.9.2

ERROR conf.Configuration: error parsing conf core-site.xml
com.ctc.wstx.exc.WstxParsingException: Illegal processing instruction target ("xml"); xml (case insensitive) is reserved by the specs.
Bug: ABFS: SAS Test updates for version and permission update
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/20 22:02
Affects Version/s: 3.3.0

This Jira will track the below 2 updates to SAS test code:
 # Upgrading the SAS version in Service SAS generator (test code)
 # Updating the permission in Delegation SAS to "op" from "p" for ACL operation as identities added as suoid/saoid added by tests are not owners of test path (Again test code).
 [Relevant public documentation: https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview]
Bug: WASB: Test failures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/20 04:49
Affects Version/s: 3.3.0

WASB tests are failing in Apache trunk resulting in Yetus run failures for PRs.

 
||Reason||Tests||
|Failed junit tests|hadoop.fs.azure.TestNativeAzureFileSystemMocked|
| |hadoop.fs.azure.TestNativeAzureFileSystemConcurrency|
| |hadoop.fs.azure.TestWasbFsck|
| |hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked|
| |hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck|
| |hadoop.fs.azure.TestNativeAzureFileSystemContractMocked|
| |hadoop.fs.azure.TestOutOfBandAzureBlobOperations|
| |hadoop.fs.azure.TestBlobMetadata|

Many PRs are hit by this. Test report link from one of the PRs:
[https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2368/5/testReport/]
Bug: ABFS: testRenameFileOverExistingFile Fails after Contract test update
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/20 19:51
Affects Version/s: 3.3.1

Post updates to rename on existing file test, ABFS contract test is having failure.

Updates were made in the AbstractContractTest class in https://issues.apache.org/jira/browse/HADOOP-17365.

To align to test expectation, ABFS tests need config "fs.contract.rename-returns-false-if-dest-exists" set to true.
Bug: ITestS3ADeleteCost.testDirMarkersFileCreation failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/20 15:51
Affects Version/s: 3.4.0

Test runs after apply of recent patches HADOOP-17318 and HADOOP-17244 are showing a failure in ITestS3ADeleteCost.testDirMarkersFileCreation

Either one of the patches is failing and wasn't picked up, or the combined changes are conflicting in some way. Not backporting HADOOP-17318 to branch-3.3 until this is addressed.
Bug: [JDK 11] mvn package -Pdocs fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/20 09:02

"mvn package -Pdocs -DskipTests -DskipShade" fails with Java 11.
{noformat}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc (default) on project hadoop-common: An error has occurred in Javadoc report generation: 
[ERROR] Exit code: 1 - javadoc: warning - You have specified the HTML version as HTML 4.01 by using the -html4 option.
[ERROR] The default is currently HTML5 and the support for HTML 4.01 will be removed
[ERROR] in a future release. To suppress this warning, please ensure that any HTML constructs
[ERROR] in your comments are valid in HTML5, and remove the -html4 option.
[ERROR] /Users/aajisaka/git/hadoop/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/RpcHeaderProtos.java:3467: error: cannot find symbol
[ERROR]       com.google.protobuf.GeneratedMessageV3 implements
[ERROR]                          ^
(snip)
{noformat}
The goal "javadoc" should be "javadoc-no-fork".
Bug: AbfsInputStream#seek throws EOFException when seek past the end of stream
Status: Open | Priority: Major | Resolution: 
Created: 17/Nov/20 23:13

Currently AbfsInputStream#seek has below check:
{code}
public synchronized void seek(long n) throws IOException {
...
if (n > contentLength) {
 throw new EOFException(FSExceptionMessages.CANNOT_SEEK_PAST_EOF);
}
...
}
{code}
So if a the seek is called with a position past the end of the stream, a EOFException gets thrown. However, it appears to me this behavior is different from some other inputstream impls, such as \{{S3AInputStream}} and \{{LocalFSFileInputStream}}, where seek passing the end is allowed, only that a later read call will return -1 indicating EOF. I think it would be better to have AbfsInputStream do the same thing.
Bug: AbstractS3ATokenIdentifier to provide issue date == created && max date == expiry time
Status: Open | Priority: Major | Resolution: 
Created: 18/Nov/20 04:22
Affects Version/s: 3.3.0

As we've seen the behavior from HADOOP-17379, AbstractS3ATokenIdentifier doesn't guarantee providing correct "issue date" and "max date" before HADOOP-17379, and even after, "max date" is not a thing we can make a guarantee only with AbstractS3ATokenIdentifier, as there's no information.

In the meanwhile, I realized there're two methods in AbstractS3ATokenIdentifier, getCreated() and getExpiryTime(), which feel like the same purpose with getIssueDate() and getMaxDate() in AbstractDelegationTokenIdentifier.

If we think these pairs are doing the same, we can just override AbstractS3ATokenIdentifier.getIssueDate() to call getCreated() && AbstractS3ATokenIdentifier.getMaxDate() to call getExpiryTime().
Bug: Ability to specify separate compression settings when intermediate and final output use the same codec
Status: Open | Priority: Major | Resolution: 
Created: 17/Nov/20 21:10

The ZStandard codec may become a codec that users will want to use for both intermediate data and for final output data yet specify different compression levels for those use cases.

It would be nice if there was a way we could create a "meta codec" like IntermediateCodec that used conf prefix techniques, like Oozie does with oozie.launcher for the Oozie launcher configs, to create a custom config namespace of sorts for setting arbitrary codec settings specific to the intermediate codec separate from the final output codec even if the same underlying codec is used for both.

However Codecs don't allow a configuration to be passed when obtaining a codec stream, and I think we would have to bypass the CodecPool entirely to be able to pass a custom conf to an arbitrary Codec.

Another approach is to skip trying to generalize the solution and specifically focus on ZStandard. It would be easy to create a wrapper codec around the existing ZStandardCompressor and ZStandardDecompressor which take the relevant parameters directly in their constructors.
Bug: java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 13/Nov/20 04:21
Affects Version/s: 3.0.0

Hi. I need help.

now Im trying to migration Cloudera flume to Apache flume. 

(which means no use XXX _chd5.16 any more)

during the test, when i stored data in HDFS i faced this problem below.

java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHostjava.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2816) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at com.poscoict.posframe.bdp.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:238) at com.poscoict.posframe.bdp.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:230) at com.poscoict.posframe.bdp.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:675) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:46) at com.poscoict.posframe.bdp.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:672) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tracing.SpanReceiverHost at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 

 

And i don't know fixed it .

please help me. 

thank you!
Bug: Zookeeper secret manager attempts to reuse token sequence numbers
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Nov/20 22:24

[~daryn] reported that the ZK delegation token secret manager uses a {{SharedCounter}} to synchronize increments of a monotonically increasing sequence number for new tokens. Yet the KMS logs occasionally, depending on load, contains an odd error indicating collisions: 


{code:bash}
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_137547444
{code}


ZKDTSM does a CAS get and set of the sequence number. Rather than return the value it set, it returns the current value which may have already been incremented by another KMS.
Bug: RawLocalFileSystem's lastModifiedTime() loses milli seconds in JDK < 10.b09
Status: Reopened | Priority: Major | Resolution: 
Created: 15/Oct/20 06:38

RawLocalFileSystem's FileStatus uses {{File.lastModified()}} api from JDK.

This api looses milliseconds due to JDK bug.

[https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8177809]

This bug fixed in JDK 10 b09 onwards and still exists in JDK 8 which is still being used in many productions.

Apparently, {{Files.getLastModifiedTime()}} from java's nio package returns correct time.

Use {{Files.getLastModifiedTime()}} instead of {{File.lastModified}} as workaround.
Bug: TestLdapGroupsMapping is failing in trunk
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 06/Nov/20 17:34

 

[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/317/testReport/junit/org.apache.hadoop.security/TestLdapGroupsMapping/testLdapReadTimeout/]

 

[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/317/testReport/junit/org.apache.hadoop.security/TestLdapGroupsMapping/testLdapConnectionTimeout/]

 

The tests are failing in open-jdk due to change in exception message(One space character). The tests passes in oracle java(for me)

 
{noformat}
 Expected to find 'LDAP response read timed out, timeout used:3000ms' but got unexpected exception: javax.naming.NamingException: LDAP response read timed out, timeout used: 3000 ms.{noformat}