av9ash/GitBugs huggingface dataset:
Bug: ABFS: Long waits and unintended retries when multiple threads try to fetch token using ClientCreds
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/20 03:49
Affects Version/s: 3.3.0

Issue reported by DB:

we recently experienced some problems with ABFS driver that highlighted a possible issue with long hangs following synchronized retries when using the _ClientCredsTokenProvider_ and calling _AbfsClient.getAccessToken_. We have seen [https://github.com/apache/hadoop/pull/1923|https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhadoop%2Fpull%2F1923&data=02%7c01%7csnvijaya%40microsoft.com%7c7362c5ba4af24a553c4308d807ec459d%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c637268058650442694&sdata=FePBBkEqj5kI2Ty4kNr3a2oJgB8Kvy3NvyRK8NoxyH4%3D&reserved=0], but it does not directly apply since we are not using a custom token provider, but instead _ClientCredsTokenProvider_ that ultimately relies on _AzureADAuthenticator_. 

 

The problem was that the critical section of getAccessToken, combined with a possibly redundant retry policy, made jobs hanging for a very long time, since only one thread at a time could make progress, and this progress amounted to basically retrying on a failing connection for 30-60 minutes.
Bug: ITestS3AConfiguration proxy tests fail when bucket probes == 0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/20 13:08
Affects Version/s: 3.3.0


when bucket probes are disabled, proxy config tests in ITestS3AConfiguration fail because the probes aren't being attempted in initialize()

{code}
  <property>
    <name>fs.s3a.bucket.probe</name>
    <value>0</value>
  </property>   
{code}

Cause: HADOOP-16711
Fix: call unsetBaseAndBucketOverrides for bucket probe in test conf, then set the probe value to 2 just to be resilient to future default changes.
Bug: Optimise s3a Listing to be fully asynchronous.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/20 12:31
Affects Version/s: 3.3.0

Listing api returns a Remote iterator. Currently it is blocking in the constructor Listing class. Also it is done in batches, so if a batch is exhausted , the next batch will be fetched from s3 

thus causing small pauses on the client side every time a batch is fetched. We can add the following optimisations to fix these:
 * Make the constructor list call to s3 asynchronous.
 * Fetch the next batch asynchronously while the current batch is getting processed.
Bug: Tune listStatus() api of s3a.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/20 12:50
Affects Version/s: 3.2.1

Similar optimisation which was done for listLocatedStatus() HADOOP-16465  can be done for listStatus() api as well. 

This is going to reduce the number of remote calls in case of directory listing.

 

CC [~stevel@apache.org] [~shwethags]
Bug: Tune S3A listFiles() api.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/20 12:49
Affects Version/s: 3.2.1

Similar optimisation which was done for listLocatedStatus() https://issues.apache.org/jira/browse/HADOOP-16465  can done for listFiles() and listStatus() api as well. 

This is going to reduce the number of remote calls in case of directory listing.

 

CC [~stevel@apache.org] [~shwethags]
Bug: [JDK11] Fix Javadoc errors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/20 06:17

{noformat}
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  17.982 s
[INFO] Finished at: 2020-06-20T01:56:28Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc (default-cli) on project hadoop-hdfs: An error has occurred in Javadoc report generation: 
[ERROR] Exit code: 1 - javadoc: warning - You have specified the HTML version as HTML 4.01 by using the -html4 option.
[ERROR] The default is currently HTML5 and the support for HTML 4.01 will be removed
[ERROR] in a future release. To suppress this warning, please ensure that any HTML constructs
[ERROR] in your comments are valid in HTML5, and remove the -html4 option.
[ERROR] /home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-2084/src/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/server/namenode/FsImageProto.java:25197: error: cannot find symbol
[ERROR]       com.google.protobuf.GeneratedMessageV3 implements
[ERROR]                          ^
[ERROR]   symbol:   class GeneratedMessageV3
[ERROR]   location: package com.google.protobuf
[ERROR] /home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-2084/src/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/server/namenode/FsImageProto.java:25319: error: cannot find symbol
[ERROR]         com.google.protobuf.GeneratedMessageV3 implements
[ERROR]                            ^
[ERROR]   symbol:   class GeneratedMessageV3
[ERROR]   location: package com.google.protobuf
[ERROR] /home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-2084/src/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/server/namenode/FsImageProto.java:26068: error: cannot find symbol
[ERROR]         com.google.protobuf.GeneratedMessageV3 implements
[ERROR]                            ^
[ERROR]   symbol:   class GeneratedMessageV3
[ERROR]   location: package com.google.protobuf
[ERROR] /home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-2084/src/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/server/namenode/FsImageProto.java:26073: error: package com.google.protobuf.GeneratedMessageV3 does not exist
[ERROR]       private PersistToken(com.google.protobuf.GeneratedMessageV3.Builder<?> builder) {
{noformat}
Bug: findbugs warnings building branch-2.10
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/20 19:16
Affects Version/s: 2.10.0

precommit build for branch-2.10 generates finbugs warnings in several components.
 This is an umbrella to analyze those warnings and fix/ignore them as necessary.

 
|{color:#FF0000}-1{color}|{color:#FF0000}findbugs{color}|{color:#FF0000}2m 1s{color}|{color:#FF0000}hadoop-common-project/hadoop-common in branch-2.10 has 14 extant findbugs warnings.{color}|
|{color:#FF0000}-1{color}|{color:#FF0000}findbugs{color}|{color:#FF0000}2m 54s{color}|{color:#FF0000}hadoop-hdfs-project/hadoop-hdfs in branch-2.10 has 10 extant findbugs warnings.{color}|
|{color:#FF0000}-1{color}|{color:#FF0000}findbugs{color}|{color:#FF0000}2m 12s{color}|{color:#FF0000}hadoop-hdfs-project/hadoop-hdfs-client in branch-2.10 has 1 extant findbugs warnings.{color}|
|{color:#FF0000}-1{color}|{color:#FF0000}findbugs{color}|{color:#FF0000}1m 35s{color}|{color:#FF0000}hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in branch-2.10 has 3 extant findbugs warnings.{color}|
|{color:#FF0000}-1{color}|{color:#FF0000}findbugs{color}|{color:#FF0000}1m 50s{color}|{color:#FF0000}hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in branch-2.10 has 1 extant findbugs warnings.{color}|
Bug: ABFS: FS initialize fails for incompatible account-agnostic Token Provider setting 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/20 16:00
Affects Version/s: 3.2.1

When AuthType and Auth token provider configs are set for both generic and account specific config, as below:

// account agnostic

fs.azure.account.auth.type=CUSTOM

fs.azure.account.oauth.provider.type=ClassExtendingCustomTokenProviderAdapter

// account specific

fs.azure.account.auth.type.account_name=OAuth

fs.azure.account.oauth.provider.type.account_name=ClassExtendingAccessTokenProvider

 For account_name, OAuth with provider as ClassExtendingAccessTokenProvider is expected to be in effect.

When the token provider class is being read from the config, account agnostic config setting is read first in the assumption that it can serve as default if account-specific config setting is absent. But this logic leads to failure when AuthType set for account specific and otherwise are different as the Interface implementing the token provider is different for various Auth Types. This leads to a Runtime exception when trying to create the oAuth access token provider.

This Jira is to track the fix for it.
Bug: ABFS: Fix idempotency test failures when SharedKey is set as AuthType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/20 16:11
Affects Version/s: 3.2.1

Idempotency related tests added as part of 

https://issues.apache.org/jira/browse/HADOOP-17015

create a test AbfsClient instance. This mock instance wrongly accepts valid sharedKey and oauth token provider instance. This leads to test failures with exceptions:
[ERROR] testRenameRetryFailureAsHTTP404(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename)  Time elapsed: 9.133 s  <<< ERROR!
 Invalid auth type: SharedKey is being used, expecting OAuth
         at org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:643)
This Jira is to fix these tests.
Bug: Update guava to 27.0-jre in hadoop branch-2.10
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 23/Jun/20 16:14
Affects Version/s: 2.10.0

com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found [CVE-2018-10237|https://nvd.nist.gov/vuln/detail/CVE-2018-10237].

 

The upgrade should not affect the version of java used. branch-2.10 still sticks to JDK7
Bug: Replace Guava Supplier with Java8+ Supplier in MAPREDUCE
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 30/Jun/20 15:39

Replacing Usage of Guava supplier are in Unit tests {{GenereicTestUtils.waitFor()}} in hadoop-mapreduce-project subdirectory.
{code:java}
Targets
hadoop-mapreduce-project with mask '*.java'
Found Occurrences  (8 usages found)
    org.apache.hadoop.mapred  (2 usages found)
        TestTaskAttemptListenerImpl.java  (1 usage found)
            20 import com.google.common.base.Supplier;
        UtilsForTests.java  (1 usage found)
            64 import com.google.common.base.Supplier;
    org.apache.hadoop.mapreduce.v2.app  (4 usages found)
        TestFetchFailure.java  (1 usage found)
            29 import com.google.common.base.Supplier;
        TestMRApp.java  (1 usage found)
            31 import com.google.common.base.Supplier;
        TestRecovery.java  (1 usage found)
            31 import com.google.common.base.Supplier;
        TestTaskHeartbeatHandler.java  (1 usage found)
            28 import com.google.common.base.Supplier;
    org.apache.hadoop.mapreduce.v2.app.rm  (1 usage found)
        TestRMContainerAllocator.java  (1 usage found)
            156 import com.google.common.base.Supplier;
    org.apache.hadoop.mapreduce.v2.hs  (1 usage found)
        TestJHSDelegationTokenSecretManager.java  (1 usage found)
            30 import com.google.common.base.Supplier;

{code}
Bug: WASB : NativeAzureFsOutputStream#close() throwing IllegalArgumentException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Apr/20 08:03

During HFile create, at the end when called close() on the OutputStream, there is some pending data to get flushed. When this flush happens, an Exception is thrown back from Storage. The Azure-storage SDK layer will throw back IOE. (Even if it is a StorageException thrown from the Storage, the SDK converts it to IOE.) But at HBase, we end up getting IllegalArgumentException which causes the RS to get aborted. If we get back IOE, the flush will get retried instead of aborting RS.
The reason is this
NativeAzureFsOutputStream uses Azure-storage SDK's BlobOutputStreamInternal. But the BlobOutputStreamInternal is wrapped within a SyncableDataOutputStream which is a FilterOutputStream. During the close op, NativeAzureFsOutputStream calls close on SyncableDataOutputStream and it uses below method from FilterOutputStream
{code}
public void close() throws IOException {
  try (OutputStream ostream = out) {
              flush();
  }
}
{code}
Here the flush call caused an IOE to be thrown to here. The finally will issue close call on ostream (Which is an instance of BlobOutputStreamInternal)
When BlobOutputStreamInternal#close() is been called, if there was any exception already occured on that Stream, it will throw back the same Exception
{code}
public synchronized void close() throws IOException {
  try {
              // if the user has already closed the stream, this will throw a STREAM_CLOSED exception
              // if an exception was thrown by any thread in the threadExecutor, realize it now
              this.checkStreamState();
              ...
}
private void checkStreamState() throws IOException {
  if (this.lastError != null) {
              throw this.lastError;
  }
}
{code}
So here both try and finally block getting Exceptions and Java uses Throwable#addSuppressed() 
Within this method if both Exceptions are same objects, it throws back IllegalArgumentException
{code}
public final synchronized void addSuppressed(Throwable exception) {
              if (exception == this)
                             throw new IllegalArgumentException(SELF_SUPPRESSION_MESSAGE, exception);
              ....
}
{code}
Bug: No Log compression and retention in Hadoop
Status: Patch Available | Priority: Major | Resolution: 
Created: 20/Apr/20 16:15

Hadoop logging lacks several important features. Logs generated end up eating disk space
We need an implementation that satisfies the following three features:  1) time-based rolling, 2) retention and 3) compression.

For example KMS logs have no retention or compression. 
{code:bash}
-rw-r--r-- 1 hkms users 704M Mar 20 23:59 kms.log.2020-03-20
-rw-r--r-- 1 hkms users 731M Mar 21 23:59 kms.log.2020-03-21
-rw-r--r-- 1 hkms users 750M Mar 22 23:59 kms.log.2020-03-22
-rw-r--r-- 1 hkms users 757M Mar 23 23:59 kms.log.2020-03-23
-rw-r--r-- 1 hkms users 805M Mar 24 23:59 kms.log.2020-03-24
-rw-r--r-- 1 hkms users 858M Mar 25 23:59 kms.log.2020-03-25
-rw-r--r-- 1 hkms users 875M Mar 26 23:59 kms.log.2020-03-26
-rw-r--r-- 1 hkms users 754M Mar 27 23:59 kms.log.2020-03-27
{code}
Bug: ABFS: GetAccessToken unrecoverable failures are being retried
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 25/Jun/20 03:56

When there is an invalid config set, call to fetch token fails with exception:

throw new UnexpectedResponseException(httpResponseCode,
 requestId,
 operation
 + " Unexpected response."
 + " Check configuration, URLs and proxy settings."
 + " proxies=" + proxies,
 authEndpoint,
 responseContentType,
 responseBody);
 }

Issue here is that UnexpectedResponseException is not recognized as irrecoverable state and ends up being retried. This needs to be fixed.
Bug: ABFS: Reuse DSAS fetched in ABFS Input and Output stream
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 18/Apr/20 15:31
Affects Version/s: 3.2.1

This Jira will track the update where ABFS input and output streams can re-use D-SAS token fetched. If the SAS is within 1 minute of expiry, ABFS will request a new SAS.  When the stream is closed the SAS will be released.
Bug: Improvement to the AccessControlException thrown by Azure abfs driver
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 17/Jun/20 16:10

Currently when an AccessControlException happens in Abfs driver call it prints the entire stack trace. To be consistent with the HDFS way of showing the Permission denied, could we modify this in ABFSClient?
 
e.g:
>$ hdfs dfs -ls /hbase/mobdir
ls: Permission denied: user=user1, access=READ_EXECUTE, inode="/hbase/mobdir":hbase:hbase:drwx------
Bug: vulnerabilities reported in jackson and jackson-databind in branch-2.10
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/20 17:24
Affects Version/s: 2.10.0

There are known vulnerabilities in the com.fasterxml.jackson.core:jackson-databind package [,2.9.10.5).

[List of vulnerabilities|https://snyk.io/vuln/maven:com.fasterxml.jackson.core%3Ajackson-databind].

Upgrading jackson and jackson-databind to 2.10 should get rid of those vulnerabilities.
Bug: Add EC flag to stat commands
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 24/Jun/20 10:44

We currently do not have a brief way to judge an ec file.  {{hdfs fsck}}  can do but shows too much information. Neither {{du}} nor {{ls}} can accurately judge the ec file. 

So I added ec flag to stat cli.

old result: 
{code:java}
$ hadoop fs -stat "%F" /user/ec/ec.txt
regular file
$ hadoop fs -stat "%F" /user/rep/rep.txt 
regular file
{code}
new result:
{code:java}
$ hadoop fs -stat "%F" /user/ec/ec.txt 
erasure coding file
$ hadoop fs -stat "%F" /user/rep/rep.txt 
replica file
{code}
Bug: Add getClusterRoot and getClusterRoots methods to FileSystem and ViewFilesystem
Status: Open | Priority: Major | Resolution: 
Created: 17/Jun/20 02:26

In a federated setting (HDFS federation, federation across multiple buckets on S3, multiple containers across Azure storage), certain system tools/pipelines require the ability to map paths to the clusters/accounts.

Consider the example of GDPR compliance/retention jobs that need to go over various datasets, ingested over a period of T days and remove/quarantine datasets that are not properly annotated/have reached their retention period. Such jobs can rely on renames to a global trash/quarantine directory to accomplish their task. However, in a federated setting, efficient, atomic renames (as those within a single HDFS cluster) are not supported across the different clusters/shards in federation. As a result, such jobs will need to leverage a trash/quarantine directory per cluster/shard. Further, they would need to map from a particular path to the cluster/shard that contains this path.

To address such cases, this JIRA proposes to get add two new methods to {{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
Bug: The default password(none) in JavaKeyStoreProvider, is no-longer useful. 
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 08/Jun/20 14:24
Affects Version/s: 3.2.1

Since, the java keytool does not allow us to create a keystore with password length less than 6 characters(i.e. none), we should consider updating the password to a 6 char length (e.g. nopass).

{code}
$ keytool -genkeypair -storetype jceks -keyalg RSA -alias kms -keystore `pwd`/keystore4 -storepass none
keytool error: java.lang.Exception: Keystore password must be at least 6 characters
$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1ubuntu1-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
{code}
Bug: Drop MRv1 binary compatibility in 4.0.0
Status: Open | Priority: Major | Resolution: 
Created: 04/Jun/20 07:04

A code comment suggests making the setJobConf method deprecated along with mapred package HADOOP-1230. HADOOP-1230 has been closed a long time ago, but the method is still not annotated as deprecated.
{code:java}
 /**
   * This code is to support backward compatibility and break the compile  
   * time dependency of core on mapred.
   * This should be made deprecated along with the mapred package HADOOP-1230. 
   * Should be removed when mapred package is removed.
   */ {code}
Comment location: [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java#L88]

From the previous discussion, it seems that this method is still required if we ensure binary compatibility with MRv1. 
 https://issues.apache.org/jira/browse/HADOOP-17047?focusedCommentId=17111702&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17111702

Mingliang Liu suggested to Drop MRv1 binary compatibility in 4.0.0
 https://issues.apache.org/jira/browse/HADOOP-17047?focusedCommentId=17112442&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17112442
Bug: AuthenticatedURL.Token does not store the token if there are redirects.
Status: Open | Priority: Major | Resolution: 
Created: 05/Jun/20 09:55

When a http call which requires SPNEGO auth is redirected to another server which also required SPNEGO auth. The token is overwritten in second call which invalidates the token in the first call. This causes a performance impact in all redirect cases and the authentication to fail in some cases.
{code:java}
    AuthenticatedURL.Token token = new AuthenticatedURL.Token();
    // Is this right, can you have spnego over http, which would mean we'll have sslFactory for all cases.
    AuthenticatedURL authenticatedURL = new AuthenticatedURL(null, sslFactory);
    conn = authenticatedURL.openConnection(url, token);
{code}
The above code results in 2 OPTION calls to the server, the first call gets the token for first server and stores it into cookieHandler in token and then then the client redirects to server2, which overwrites the token first call made. When getInputStream is called on the conn, the jdk does a SPNEGO auth again since the tokens sent to the servers are not valid anymore. This mean the KerberosAuthenticator authenticate method is skipped for the second call and works only with the jdk call. I believe the fix should be to make the cookie handler multi-domain aware and also add APIs to extract token for a given domain.

 

I've attached the logs where I saw this behavior. There is another issue which can be seen in the log, where the token type is sent as alt-kerberos by the server, which is ignore by the handler and make a call again.

 

Effectively, we are making 6 http calls, whereas with token reuse it should have been 2 calls, except for the first call which will be 4 calls.
Bug: Adding Common Counters in ABFS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/20 07:36
Affects Version/s: 3.3.0

Common Counters to be added to ABFS:
|OP_CREATE|
|OP_OPEN|
|OP_GET_FILE_STATUS|
|OP_APPEND|
|OP_CREATE_NON_RECURSIVE|
|OP_DELETE|
|OP_EXISTS|
|OP_GET_DELEGATION_TOKEN|
|OP_LIST_STATUS|
|OP_MKDIRS|
|OP_RENAME|

|DIRECTORIES_CREATED|
|DIRECTORIES_DELETED|
|FILES_CREATED|
|FILES_DELETED|
|ERROR_IGNORED|

 propose:
 * Have an enum class to define all the counters.
 * Have an Instrumentation class for making a MetricRegistry and adding all the counters.
 * Incrementing the counters in AzureBlobFileSystem.
 * Integration and Unit tests to validate the counters.
Bug: NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/20 06:21
Affects Version/s: 2.10.0

Hadoop components are increasingly being deployed on VMs and containers. One aspect of this environment is that DNS is dynamic. Hostname records get modified (or deleted/recreated) as a container in Kubernetes (or even VM) is being created/recreated. In such dynamic environments, the initial DNS resolution request might return resolution failure briefly as DNS client doesn't always get the latest records. This has been observed in Kubernetes in particular. In such cases NetUtils.connect() appears to throw java.nio.channels.UnresolvedAddressException.  In much of Hadoop code (like DFSInputStream and DFSOutputStream), the code is designed to retry IOException. However, since UnresolvedAddressException is not child of IOException, no retry happens and the code aborts immediately. It is much better if NetUtils.connect() throws java.net.UnknownHostException as that is derived from IOException and the code will treat this as a retry-able error.