av9ash/GitBugs huggingface dataset:
Bug: Upgrade Netty version to 4.1.45.Final to handle CVE-2019-20444,CVE-2019-16869
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Feb/20 05:03
Affects Version/s: 3.3.0

[CVE-2019-20444 |[https://rnd-vulncenter.huawei.com/vuln/toViewOfficialDetail?cveId=CVE-2019-20444]]

[CVE-2019-16869|[https://rnd-vulncenter.huawei.com/vuln/toViewOfficialDetail?cveId=CVE-2019-16869]]

We should upgrade the netty dependency to 4.1.45.Final version
Bug: Declare 2.8.x release line EOL
Status: Resolved | Priority: Major | Resolution: Done
Created: 24/Feb/20 18:24
Affects Version/s: 2.8.6

The Hadoop community discussed the 2.8.x release line EOL plan:
https://s.apache.org/hadoop2.8eol

The first 2.8 release, 2.8.0 was released on 03/27/2017
The last 2.8 release, 2.8.5, was released on 09/15/2018

Per the [Hadoop EOL policy|https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches] 2.8.5 was released nearly 18 months ago and with no volunteer taking up the RM for 2.8.6, it is time to start the EOL process.

I am going to update the wiki and post an announcement to the user/general mailing list declaring 2.8 EOL.
Bug: Batch listing of multiple directories to be an unstable interface
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/20 14:11
Affects Version/s: 3.3.0

HDFS-13616 added a new API for batch listing of multiple directories, but
it isn't yet ready for tagging as stable & doesn't suit object stores.


* the new API is pulled into a new interface marked unstable;
* new classes (PartialListing) also tagged unstable.
* Define a new path capability.

HDFS will implement, but not filter/HarFS; it is an HDFS exclusive implementation for now.
Bug: update jackon-databind version
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Feb/20 10:28

according to [CVE-2020-8840|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8840], maybe we should update  jackson-databind to 2.9.10.3 or 2.10.x?*
Bug: ABFS test failures when running with a storage account without a hierarchical namespace
Status: Open | Priority: Major | Resolution: 
Created: 04/Mar/20 16:01
Affects Version/s: 3.3.0

Some tests are failing when running on a storage account without hierarchical namespace: 

{noformat}
[ERROR] Failures:
[ERROR]   ITestAzureBlobFileSystemAuthorization.testSetOwnerUnauthorized:212->runTest:274 Expected an exception of type class org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException
[ERROR]   ITestAzureBlobFileSystemAuthorization.testSetPermissionUnauthorized:217->runTest:274 Expected an exception of type class org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException
[ERROR]   ITestGetNameSpaceEnabled.testXNSAccount:51->Assert.assertTrue:41->Assert.fail:88 Expecting getIsNamespaceEnabled() return true
[ERROR] Errors:
[ERROR]   ITestAzureBlobFileSystemAuthorization.testGetAclStatusAuthorized:247->runTest:278->executeOp:298 ? UnsupportedOperation
[ERROR]   ITestAzureBlobFileSystemAuthorization.testGetAclStatusUnauthorized:252->runTest:274->lambda$runTest$4:275->executeOp:298 ? UnsupportedOperation
[ERROR]   ITestAzureBlobFileSystemAuthorization.testModifyAclEntriesUnauthorized:222->runTest:274->lambda$runTest$4:275->executeOp:335 ? UnsupportedOperation
[ERROR]   ITestAzureBlobFileSystemAuthorization.testRemoveAclEntriesUnauthorized:227->runTest:274->lambda$runTest$4:275->executeOp:331 ? UnsupportedOperation
[ERROR]   ITestAzureBlobFileSystemAuthorization.testRemoveAclUnauthorized:237->runTest:274->lambda$runTest$4:275->executeOp:339 ? UnsupportedOperation
[ERROR]   ITestAzureBlobFileSystemAuthorization.testRemoveDefaultAclUnauthorized:232->runTest:274->lambda$runTest$4:275->executeOp:342 ? UnsupportedOperation
[ERROR]   ITestAzureBlobFileSystemAuthorization.testSetAclUnauthorized:242->runTest:274->lambda$runTest$4:275->executeOp:304 ? UnsupportedOperation
[ERROR]   ITestAzureBlobFileSystemCheckAccess.testWhenCheckAccessConfigIsOff:125->setupTestDirectoryAndUserAccess:313->modifyAcl:304 ? UnsupportedOperation
{noformat}

This should be fixed, so these tests will be skipped or will fail with a more meaningful error.
Bug: Announce user-zh mailing list
Status: Resolved | Priority: Major | Resolution: Done
Created: 28/Feb/20 16:49

A user-zh mailing list for Mandarin-speaking Hadoop users is set up now. Let's make it public:

(1) Add the mailing list to https://hadoop.apache.org/mailing_lists.html
(2) Announce the ML at the user@, general@
(3) Send notification to the ASF slack channel.
Bug: TestIPC#testProxyUserBinding failure
Status: Open | Priority: Major | Resolution: 
Created: 28/Feb/20 21:51
Affects Version/s: 3.1.3

mvn clean install -Pdist -Pnative -Dtar -DskipTests in hadoop-common fails due to TestIPC failure, specifically testProxyUserBinding. Ive tested separately and the different components seem to work just fine, so Im not sure why its failing.
Bug: S3Guard: add support for other MetadataStores
Status: Open | Priority: Major | Resolution: 
Created: 20/Feb/20 14:27
Affects Version/s: 3.2.1

Hi all,

 

Are there any plans to add other MetadataStore implementations for S3Guard? DynamoDB costs are too high when the read capacity/write capacity are high.

 

Maybe a Postgres/MySQL implementation is simple enough to implement and offer strong consistency.

Another idea is to implement a Cassandra/Scylla MetadataStore(for better write scalability), but we should pay attention to consistency.

 

Any thoughts?
Bug: ipc.Server readAndProcess threw NullPointerException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/20 21:34

{code}
2020-01-18 10:19:02,109 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020: readAndProcess from client xx.xx.xx.xx threw exception [java.lang.NullPointerException]
java.lang.NullPointerException
	at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1676)
	at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:935)
	at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:791)
	at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:762)
{code}
Bug: InstrumentedLock should log lock queue time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/20 16:53
Affects Version/s: 3.3.0

Within the Datanode, we often see contention around the FsDatasetImpl lock. This can be for various reasons, eg the DN is under IO load, and other bugs, eg HDFS-15131.

When DN slow downs happen, it is very difficult to debug what is causing it, as there are few messages in the logs which indicate what is happening.

In my experience, the presence of this log is informative:

{code}
2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Took 21540ms to process 1 commands from NN
{code}

This comes from the datanode heartbeat thread, and it is logged when the commands returned from the heartbeat cannot be enqueued, and usually this is because the thread cannot get the FsDatasetImpl lock due to contention with other slow threads.

HDFS-14997 moved the command processing to an async thread, and hence this useful message will disappear.

InstrumentedLock introduced a feature to the datanode, where it will log a stack trace if any thread holds the lock for over 300ms. However this will not catch a scenario where 10 threads each hold a lock for 200ms, leading to the next in the queue having waited over 2 seconds.

I believe it would be useful to extend InstrumentedLock to log if a thread has to wait for over some threshold. That way, we could be able to catch scenarios like the heartbeat thread shows us, but in a much more obvious way, provided lock contention is involved.
Bug: Trunk is broken on OS X
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 13/Feb/20 23:43

Trunk does not compiler on Mac OS X with Java "1.8.0_242".


{code:bash}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-maven-plugins: Compilation failure
[ERROR] /home/ahussein/hadoop/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc/ProtocRunner.java:[16,39] package com.fasterxml.jackson.core.type does not exist
{code}
Bug: Support getting thread info from thread group for JvmMetrics to improve the performance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/20 04:54
Affects Version/s: 2.10.1

Recently we found jmx request taken almost 5s+ to be done when there were 1w+ threads in a stressed datanode process, meanwhile other http requests were blocked and some disk operations were affected (we can see many "Slow manageWriterOsCache" messages in DN log, and these messages were hard to be seen again after we stopped sending jxm requests)

The excessive time is spent in getting thread info via ThreadMXBean inside which ThreadImpl#getThreadInfo native method is called, the time complexity of ThreadImpl#getThreadInfo is O(n*n) according to [JDK-8185005|https://bugs.openjdk.java.net/browse/JDK-8185005] and it holds global thread lock and prevents creation or termination of threads.

To improve this, I propose to support getting thread info from thread group which will improve a lot by default, also support using original approach when "-Dhadoop.metrics.jvm.use-thread-mxbean=true" is configured in the startup command.

An example of performance tests between these two approaches is as follows:
{noformat}
#Threads=100, ThreadMXBean=382372 ns, ThreadGroup=72046 ns, ratio: 5
#Threads=200, ThreadMXBean=776619 ns, ThreadGroup=83875 ns, ratio: 9
#Threads=500, ThreadMXBean=3392954 ns, ThreadGroup=216269 ns, ratio: 15
#Threads=1000, ThreadMXBean=9475768 ns, ThreadGroup=220447 ns, ratio: 42
#Threads=2000, ThreadMXBean=53833729 ns, ThreadGroup=579608 ns, ratio: 92
#Threads=3000, ThreadMXBean=196829971 ns, ThreadGroup=1157670 ns, ratio: 170
{noformat}
Bug: Prune -tombstones to remove children entries
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 13/Feb/20 16:04
Affects Version/s: 3.3.0

When you prune a directory in S3Guard the children entries are not pruned with the tombstoned parent (directory).
I propose the solution to remove all children entries (the whole hierarchy) to avoid orphaned entries once the parent (directory) tombstone is removed.
Bug: Support for `fs.s3a.endpoint.region`
Status: Resolved | Priority: Major | Resolution: Works for Me
Created: 04/Feb/20 13:02

Currently it is not possible to connect S3 Compatible services like MinIO, Ceph, etc (running with a custom region) to Spark with s3a connector. For example, if MinIO is running on a Server with
 * IP Address: 192.168.0.100
 * Region: ap-southeast-1

The s3a connector can't be configured to use the region `ap-southeast-1`. 

It would be great to have a configuration field like `fs.s3a.endpoint.region`. This will be very helpful for users deploying Private Cloud and who intend to use S3 like services on premises.
Bug: ITestAbfsClient.testContinuationTokenHavingEqualSign failing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/20 20:55
Affects Version/s: 3.3.0

Testcase testContinuationTokenHavingEqualSign is failing as request that was expected to fail is passing.

There is change in the queryparam validation in ContinuationToken at server end wihch has resulted in this behaviour. 

Server request trace:

2020-02-05 16:59:17,001 DEBUG [JUnit-testContinuationTokenHavingEqualSign]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(263)) - HttpRequest: 200,,cid=87c3ebea-def7-4fdd-a21a-a56c63a59387,rid=0931c565-201f-004c-1317-dcdd90000000,sent=0,recv=0,GET,[https://snvijayaabfsns.dfs.core.windows.net/abfs-testcontainer-85bb9523-fccd-45f3-ae6d-37622d8231e5?upn=false&resource=filesystem&maxResults=500&directory=/&continuation=%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D&timeout=90&recursive=true]

 

Disabling the test until the server fix is in and deployed on all regions.
Bug: S3Guard testing doc: Add required parameters for S3Guard testing in IDE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/20 15:29
Affects Version/s: 3.2.1

When I tried running the S3 guard tests using my IDE, they were getting skipped saying s3 guard is not enabled. To enable S3 guard test you have to configure the following property in auth-keys.xml

<property>
 <name>fs.s3a.s3guard.test.enabled</name>
 <value>true</value>
</property>

Once you have done this tests will still get skipped saying "Test only applies when DynamoDB is used for S3Guard Store"

To configure dynamo db you have  to set this property in auth-keys.xml

<property>
 <name>fs.s3a.s3guard.test.implementation</name>
 <value>dynamo</value>
</property>
Bug: Spark-SQL test running on Windows: hadoop chgrp warnings	
Status: Open | Priority: Major | Resolution: 
Created: 04/Feb/20 07:28
Affects Version/s: 2.6.5

Running SparkSQL local embedded unit tests on Win10, using winutils.

Got warnings about 'hadoop chgrp'.

See environment info.
{code:bash}
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
{code}
Related info on SO: [https://stackoverflow.com/questions/48605907/error-in-pyspark-when-insert-data-in-hive]

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShellPermissions.java:210

The problem is: backslash character isn't included to allowedChars, see attached HadoopGroupTest.java

Original issue in Spark: https://issues.apache.org/jira/browse/SPARK-30701
Bug: TestHarFileSystem.testInheritedMethodsImplemented broken
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/20 14:30
Affects Version/s: 3.3.0

caused by HADOOP-16759.

I'm surprised this didn't surface earlier -or embarrassed that if it did, somehow I missed it. Will fix.

 

will also review the checksum FS to make sure it's gone through there too. 

 

Given I was using the IDE to refactor, it should have all been automatic
Bug: Let s3 clients configure request timeout
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/20 01:04
Affects Version/s: 3.3.0

S3 does not guarantee latency. Every once in a while a request may straggle and drive latency up for the greater procedure. In these cases, simply timing-out the individual request is beneficial so that the client application can retry. The retry tends to complete faster than the original straggling request most of the time. Others experienced this issue too: [https://arxiv.org/pdf/1911.11727.pdf] .

S3 configuration already provides timeout facility via `ClientConfiguration#setTimeout`. Exposing this configuration is beneficial for latency sensitive applications. S3 client configuration is shared with DynamoDB client which is also affected from unreliable worst case latency.
Bug: [pb-upgrade] Use 'o.a.h.thirdparty.protobuf' shaded prefix instead of 'protobuf_3_7'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/20 10:07

As per discussion  [here|https://github.com/apache/hadoop/pull/1635#issuecomment-576247014], versioned package name may make upgrade of library to a non-trivial task. package name needs to be updated in all usages in all modules. 

So common package name is preferred.
Bug: Please remove or minimize need for native libraries "winutils" to run Spark locally on Windows.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Jan/20 01:23
Affects Version/s: 3.1.3

Please remove the need for building and using custom native libraries "winutils" to run Spark on Windows systems just to run Spark & similar.

Reference: [https://github.com/steveloughran/winutils]

If native libraries are needed for performance benefits then please try to load the native libraries and then if not found then provide and use a functional equivalent Java implementation such that Spark will build and run applications using only Java.
Bug: Unable to build 3.1.3 winutils and libwinutils native x64 using VisualStudio 2019: winutils.sln, Platform Toolset v142, SDK v10.0, C++ LangStd: Default, and other default settings
Status: Open | Priority: Major | Resolution: 
Created: 19/Jan/20 23:01
Affects Version/s: 3.1.3

libwinutils.c  line 40 gives several build errors:

C2065  'L': undeclared identifier

E0065 expected a ';'

E0020 identifier "L" is undefined

C2099 initializer is not a constant

C2143 syntax error: missing ';' before 'string'
Bug: Javadoc errors are always ignored in the precommit jobs
Status: Open | Priority: Major | Resolution: 
Created: 15/Jan/20 04:21

There are many javadoc errors reported and fixed. This is mostly because the precommit jobs cannot capture the errors. There is an issue (YETUS-554) in Yetus but there is something to do in Hadoop-side.

Examples:
* MAPREDUCE-7256
* HADOOP-15904
* HADOOP-16390
* HADOOP-16654
* HADOOP-16681
* YARN-8939
* YARN-9812
* and many more
Bug: wasb tests can't work with https only wasb storage account
Status: Open | Priority: Major | Resolution: 
Created: 03/Jan/20 16:32
Affects Version/s: 3.3.0

If you create a wasb storage account which is https only, the wasb tests all fail to instantiate the FS.

Either we fix the tests to support wasbs:// schemas in place of the hard-coded wasb ones, or we just add a switch to say "all wasb URLs are HTTPS"

happy either way
Bug: Checksum FS #hsync does not sync to disk
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 03/Jan/20 11:02

Recently I created a SequenceFile Writer and was surprised to see that the file size was not changing despite hsync being called after each append.

After some fiddling around, I noticed that if write checksums are disabled then the hsync is effective.

I checked the documentation and could not find any mention of the fact that hsync could be ineffective when write checksums are on.

The Writer was using the LocalFilesystem.
Bug: Add com.amazonaws.auth.profile.ProfileCredentialsProvider to hadoop-aws docs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Mar/20 02:24
Affects Version/s: 3.2.1

There is a very, very useful S3A authentication method that is not currently documented: {{com.amazonaws.auth.profile.ProfileCredentialsProvider}}

This provider lets you source your AWS credentials from a shared credentials file, typically stored under {{~/.aws/credentials}}, using a [named profile|https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html]. All you need is to set the {{AWS_PROFILE}} environment variable, and the provider will get the appropriate credentials for you.

I discovered this from my coworkers, but cannot find it in the docs for hadoop-aws. I'd expect to see it at least mentioned in [this section|https://hadoop.apache.org/docs/r2.9.2/hadoop-aws/tools/hadoop-aws/index.html#S3A_Authentication_methods]. It should probably be added to the docs for every minor release that supports it, which I'd guess includes 2.8 on up.

(This provider should probably also be added to the default list of credential provider classes, but we can address that in another ticket. I can say that at least in 2.9.2, it's not in the default list.)

(This is not to be confused with {{com.amazonaws.auth.InstanceProfileCredentialsProvider}}, which serves a completely different purpose.)
Bug: Add AWS S3 Transfer acceleration support
Status: Open | Priority: Minor | Resolution: 
Created: 08/Jan/20 13:48
Affects Version/s: 2.8.5

It would be great to be able to use [S3 Transfer acceleration |[https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html]]especially when reading data from multiple locations around the globe as there is a significant performance improvement be routing traffic over the AWS backbone.
Bug: AWS AssumedRoleCredentialProvider needs ExternalId add
Status: Open | Priority: Minor | Resolution: 
Created: 15/Jan/20 18:58
Affects Version/s: 3.2.1

AWS has added a security feature to the assume-role function in the form of the "ExternalId" key in the AWS Java SDK {{STSAssumeRoleSessionCredentialsProvider.Builder}} class.  To support this security feature, the hadoop aws {{AssumedRoleCredentialProvider}} needs a patch to include this value from the configuration as well as an added Constant to the {{org.apache.hadoop.fs.s3a.Constants}} file.

The ExternalId is not a required security feature, it is an augmentation of the current assume role configuration. 

Proposed: 
 * Get the assume-role ExternalId token from the configuration for the configuration key {{fs.s3a.assumed.role.externalid}}
 * Use the configured ExternalId value in the {{STSAssumeRoleSessionCredentialsProvider.Builder}}   

e.g.

{{if (StringUtils.isNotEmpty(externalId)) {}}
 {{    builder.withExternalId(externalId); // include the token for cross-account assume role}}
 {{}}}

 Tests:
 * +Unit test+ which verifies the ExternalId state value of the {{AssumedRoleCredentialProvider}} is consistent with the configured value - either empty or populated
 * Question: not sure about how to write the +integration test+ for this feature.  We have an account configured for this use-case that verifies this feature but I don't have much context on the Hadoop project AWS S3 integration tests, perhaps a pointer could help.
Bug: ITestCustomSigner uses absolute paths off the bucket root rather than fork-relative
Status: Open | Priority: Minor | Resolution: 
Created: 25/Mar/20 17:06
Affects Version/s: 3.3.0

ITestCustomSigner.testCustomSignerAndInitializer is creating a a couple of paths (/customsignerpath1, /customsignerpath2) rather than in a specific fork.

the paths are uniquely named enough to be low risk, but they should be moved to being fork-relative
Bug: S3A DT marshalling to  include nested error text in wrapped message
Status: Open | Priority: Minor | Resolution: 
Created: 05/Feb/20 09:48
Affects Version/s: 3.3.0

When you get an unmarshalling error with S3A DTs, the thrown exception wraps the cause, but doesn't include it's text in the message, so on a long trace the cause can get lost
Bug: abfs can't access storage account if soft delete is enabled
Status: Open | Priority: Minor | Resolution: 
Created: 05/Feb/20 09:44
Affects Version/s: 3.1.3

Facing the issue in which if soft delete is enabled on storage account.

Hadoop fs -ls command fails with 
{noformat}
 Operation failed: "This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.", 409, HEAD, https://<account_name>.[dfs.core.windows.net/test-container-1//?upn=false&action=getAccessControl&timeout=90|http://dfs.core.windows.net/test-container-1//?upn=false&action=getAccessControl&timeout=90]
{noformat}
Trying to access storage account by issuing below command :
{noformat}
 hadoop fs -Dfs.azure.account.auth.type.<account_name>.[dfs.core.windows.net|http://dfs.core.windows.net/]=OAuth -Dfs.azure.account.oauth.provider.type.<account_name>.[dfs.core.windows.net|http://dfs.core.windows.net/]=org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider -ls [abfs://test-container-1]@<account_name>.[dfs.core.windows.net/|http://dfs.core.windows.net/]
{noformat}
Bug: Upgrade spotbugs to 4.0.6
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Feb/20 04:53
Affects Version/s: 3.4.0

[https://github.com/spotbugs/spotbugs/releases]

spotbugs 4.0.0 is now released. 

 

We can upgrade spotbugs' version to:

1. 3.1.12  (conservative option)

2. 4.0.0 (which might includes incompatible changes, according to the migration guide: [https://spotbugs.readthedocs.io/en/stable/migration.html])

 

Step by step approach is also acceptable.