av9ash/GitBugs huggingface dataset:
Bug: Optimize UGI#getGroups by adding UGI#getGroupsSet
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/20 02:28
Affects Version/s: 3.4.0

UGI#getGroups has been optimized with HADOOP-13442 by avoiding the List->Set->List conversion. However the returned list is not optimized to contains lookup, especially the user's group membership list is huge (thousands+) . This ticket is opened to add a UGI#getGroupsSet and use Set#contains() instead of List#contains() to speed up large group look up while minimize List->Set conversions in Groups#getGroups() call.
Bug: Increase precommit job timeout from 5 hours to 20 hours
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/20 17:12
Affects Version/s: 3.3.1

Now we frequently increase the timeout for testing and undo the change before committing.

* https://github.com/apache/hadoop/pull/2026
* https://github.com/apache/hadoop/pull/2051
* https://github.com/apache/hadoop/pull/2012
* https://github.com/apache/hadoop/pull/2098
* and more...

I'd like to increase the timeout by default to reduce the work.
Bug: Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/20 06:22
Affects Version/s: 3.3.0

After upgrade/shade of protobuf to 3.7 version, existing Hadoop-Rpc client-server implementations using ProtobufRpcEngine will not work.

So, this Jira proposes to keep existing ProtobuRpcEngine as-is (without shading and with protobuf-2.5.0 implementation) to support downstream implementations.

Use new ProtobufRpcEngine2 to use shaded protobuf classes within Hadoop and later projects who wish to upgrade their protobufs to 3.x.
Bug: Umbrella Jira for improving the Hadoop-cos support in Hadoop
Status: Patch Available | Priority: Major | Resolution: 
Created: 02/Apr/20 17:43

This Umbrella Jira focus on fixing some known bugs and adding some important features.

 

bugfix:
 # resolve the dependency conflict;
 # fix the upload buffer returning failed when some exceptions occur;
 # fix the issue that the single file upload can not be retried;
 # fix the bug of checking if a file exists through listing the file frequently.

features:
 # support SessionCredentialsProvider and InstanceCredentialsProvider, which allows users to specify the credentials in URI or get it from the CVM (Tencent Cloud Virtual Machine) bound to the CAM role that can access the COS bucket;
 # support the server encryption  based on SSE-COS and SSE-C;
 # support the HTTP proxy settings;
 # support the storage class settings;
 # support the CRC64 checksum.
Bug: S3A delegation token binding to support secondary binding list
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 18/Jun/20 17:00
Affects Version/s: 3.3.0

(followon from HADOOP-17050)

Add the ability of an S3A FS instance to support multiple instances of delegation token bindings.

The property "fs.s3a.delegation.token.secondary.bindings" will list the classnames of all secondary bindings.

for each one, an instance shall be created with the canonical service name being: fs URI + [ tokenKind ]. This is to ensure that the URIs are unique for each FS instance -but also that a single fs instance can have multiple tokens in the credential list.

the instance is just a AbstractDelegationTokenBinding provider of an AWS credential provider chain, with the normal lifecycle and operations to bind to a DT, issue tokens, etc

* the final list of AWS Credential providers will be built by appending those provided by each binding in turn.

Token binding at launch

If the primary token binding binds to a delegation token, then the whole binding is changed such that all secondary tokens MUST also bind. That is: it will be an error if one cannot be found. This is  possibly overstrict-but it avoids situations where an incomplete set of tokens are retrieved and This does not surface until later.

Only the encryption secrets in the primary DT will be used for FS encryption settings.

Testing: yes.

Probably also by adding a test-only DT provider which doesn't actually issue any real credentials and so which can be deployed in both ITests and staging tests where we can verify that the chained instantiation works.

Compatibility: the goal is to be backwards compatible with any already released token provider plugin.
Bug: ABFS: Make PUT and POST operations idempotent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/20 03:54
Affects Version/s: 3.2.1

Initially changes were made as part of this PR to handle idempotency including rename operation with the understanding that last modified time gets updated. But that assumption was wrong and the rename idempotency handling has since evolved.


For a job clean up, if the Manifest Committer in below Jira is used, then rename idempotency works using the previously fetched etag :

[HADOOP-18163] hadoop-azure support for the Manifest Committer of MAPREDUCE-7341 - ASF JIRA (apache.org)

 

A part of the commit tracked under current Jira to handle DELETE idempotency is still relevant.

A means to handle idempotency between driver and backend inherently is being worked upon.

-- Older notes


Currently when a PUT or POST operation timeouts and the server has already successfully executed the operation, there is no check in driver to see if the operation did succeed or not and just retries the same operation again. This can cause driver to through invalid user errors.

 

Sample scenario:
 # Rename request times out. Though server has successfully executed the operation.
 # Driver retries rename and get source not found error.

In the scenario, driver needs to check if rename is being retried and success if source if not found, but destination is present.
Bug: Handle release package related issues
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/20 12:29

Same issue as mentioned in HADOOP-16919 is present in hadoop distribution generation as well.

Handle following comments from [~elek] in 1.0.0-RC0 voting mail thread here[[https://lists.apache.org/thread.html/r1f2e8325ecef239f0d713c683a16336e2a22431a9f6bfbde3c763816%40%3Ccommon-dev.hadoop.apache.org%3E]]
{quote}3. Yetus seems to be included in the source package. I am not sure if
 it's intentional but I would remove the patchprocess directory from the
 tar file.

7. Minor nit: I would suggest to use only the filename in the sha512
 files (instead of having the /build/source/target prefix). It would help
 to use `sha512 -c` command to validate the checksum.
{quote}
Bug: Introduce StreamContext for Abfs Input and Output streams.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/20 06:14

The number of configuration keeps growing in AbfsOutputStream and AbfsInputStream as we keep on adding new features. It is time to refactor the configurations in a separate class like StreamContext and pass them around. 

This is will improve the readability of code and reduce cherry-pick-backport pain.
Bug: Reduce Guava dependency in Hadoop source code
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Jun/20 15:59

Relying on Guava implementation in Hadoop has been painful due to compatibility and vulnerability issues.
 Guava updates tend to break/deprecate APIs. This made It hard to maintain backward compatibility within hadoop versions and clients/downstreams.

With 3.x uses java8+, the java 8 features should preferred to Guava, reducing the footprint, and giving stability to source code.

This jira should serve as an umbrella toward an incremental effort to reduce the usage of Guava in the source code and to create subtasks to replace Guava classes with Java features.

Furthermore, it will be good to add a rule in the pre-commit build to warn against introducing a new Guava usage in certain modules.

Any one willing to take part in this code refactoring has to:
 # Focus on one module at a time in order to reduce the conflicts and the size of the patch. This will significantly help the reviewers.
 # Run all the unit tests related to the module being affected by the change. It is critical to verify that any change will not break the unit tests, or cause a stable test case to become flaky.
 # Merge should be done to the following branches:  trunk, branch-3.3, branch-3.2, branch-3.1

 

A list of sub tasks replacing Guava APIs with java8 features:
{code:java}
com.google.common.io.BaseEncoding#base64()	java.util.Base64
com.google.common.io.BaseEncoding#base64Url()	java.util.Base64
com.google.common.base.Joiner.on()	                        java.lang.String#join() or 
                                                                                         java.util.stream.Collectors#joining()
com.google.common.base.Optional#of()	                java.util.Optional#of()
com.google.common.base.Optional#absent()	        java.util.Optional#empty()
com.google.common.base.Optional#fromNullable()	java.util.Optional#ofNullable()
com.google.common.base.Optional	                        java.util.Optional
com.google.common.base.Predicate	                        java.util.function.Predicate
com.google.common.base.Function	                        java.util.function.Function
com.google.common.base.Supplier	                        java.util.function.Supplier
{code}
 

I also vote for the replacement of {{Precondition}} with either a wrapper, or Apache commons lang.

I believe you guys have dealt with Guava compatibilities in the past and probably have better insights. Any thoughts? [~weichiu], [~gabor.bota], [~stevel@apache.org], [~ayushtkn], [~busbey], [~jeagles], [~kihwal]
Bug: Add checkstyle rule to prevent further usage of Guava classes
Status: Resolved | Priority: Major | Resolution: Done
Created: 30/Jun/20 15:08

We should have precommit rules to prevent further usage of Guava classes that are available in Java8+


A list replacing Guava APIs with java8 features:
{code:java}
com.google.common.io.BaseEncoding#base64()	java.util.Base64
com.google.common.io.BaseEncoding#base64Url()	java.util.Base64
com.google.common.base.Joiner.on()	                        java.lang.String#join() or 
                                                                                         java.util.stream.Collectors#joining()
com.google.common.base.Optional#of()	                java.util.Optional#of()
com.google.common.base.Optional#absent()	        java.util.Optional#empty()
com.google.common.base.Optional#fromNullable()	java.util.Optional#ofNullable()
com.google.common.base.Optional	                        java.util.Optional
com.google.common.base.Predicate	                        java.util.function.Predicate
com.google.common.base.Function	                        java.util.function.Function
com.google.common.base.Supplier	                        java.util.function.Supplier
{code}
Bug: ViewFS should initialize target filesystems lazily
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/20 06:09
Affects Version/s: 3.2.1

Currently viewFS initialize all configured target filesystems when viewfs#init itself.

Some target file system initialization involve creating heavy objects and proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient object which will create proxy connections to NN etc.
For example: if ViewFS configured with 10 target fs with hdfs uri and 2 targets with s3a.

If one of the client only work with s3a target, But ViewFS will initialize all targets irrespective of what clients interested to work with. That means, here client will create 10 DFS initializations and 2 s3a initializations. Its unnecessary to have DFS initialization here. So, it will be a good idea to initialize the target fs only when first time usage call come to particular target fs scheme.
Bug: Enable setting clientId and tenantId for ADL MsiTokenProvider
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 07/Apr/20 13:50

AdlFileSystem is [currently using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] a [deprecated constructor for MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], only passing a port setting which is not used anymore.

The purpose of this improvement is to update AdlFileSystem to pass optional clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI authentication on VMs with attached multiple MSIs.
Bug: Remove source code from branch-2
Status: Resolved | Priority: Major | Resolution: Done
Created: 15/Apr/20 09:03

Now, branch-2 is dead and unused. I think we can delete the entire source code from branch-2 to avoid committing or cherry-picking to the unused branch.

Chen Liang asked ASF INFRA for help but it didn't help for us: INFRA-19581
Bug: hadoop-shaded-protobuf_3_7 depends on the wrong version.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 20/Jun/20 16:00
Affects Version/s: 3.0.1

When using maven to compile hadoop source code, the following exception message appears:

[*INFO*] *------------------------------------------------------------------------*

[*INFO*] *BUILD FAILURE*

[*INFO*] *------------------------------------------------------------------------*

[*INFO*] Total time:  29.546 s

[*INFO*] Finished at: 2020-06-20T23:57:59+08:00

[*INFO*] *------------------------------------------------------------------------*

[*ERROR*] Failed to execute goal on project hadoop-common: *Could not resolve dependencies for project org.apache.hadoop:hadoop-common:jar:3.3.0-SNAPSHOT: Could not find artifact org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.0.0-SNAPSHOT in apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots)* -> *[Help 1]*

[*ERROR*] 

[*ERROR*] To see the full stack trace of the errors, re-run Maven with the *-e* switch.

[*ERROR*] Re-run Maven using the *-X* switch to enable full debug logging.

[*ERROR*] 

[*ERROR*] For more information about the errors and possible solutions, please read the following articles:

[*ERROR*] *[Help 1]* http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[*ERROR*] 

[*ERROR*] After correcting the problems, you can resume the build with the command

[*ERROR*]   *mvn <args> -rf :hadoop-common*
Bug: Fix shelldocs path in Jenkinsfile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/20 06:31

Shelldocs check is not enabled in the precommit jobs.
|{color:#0000FF}0{color}|{color:#0000FF}shelldocs{color}|{color:#0000FF}0m 1s{color}|{color:#0000FF}Shelldocs was not available.{color}|

Console log https://builds.apache.org/job/hadoop-multibranch/job/PR-2045/1/console
{noformat}
WARNING: shellcheck needs UTF-8 locale support. Forcing C.UTF-8.
executable '/testptch/hadoop/dev-support/bin/shelldocs' for 'shelldocs' does not exist.
{noformat}
Bug: Handle an internal dir in viewfs having multiple children mount points pointing to different filesystems
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/20 21:42

In case the viefs mount table is configured in a way where multiple child mount points are pointing to different file systems, the getContentSummary or getStatus don't return the expected result
{code:java}
mount link /a/b/ → hdfs://nn1/a/b
 mount link /a/d/ → file:///nn2/c/d{code}
b has two files and d has 1 file. So getContentSummary on / should return 3 files.

It also fails for the following scenario:
{code:java}
mount link  /internalDir -> /internalDir/linternalDir2
mount link  /internalDir -> /internalDir/linkToDir2 -> hdfs://nn1/dir2{code}
Exception:
{code:java}
java.io.IOException: Internal implementation error: expected file name to be /java.io.IOException: Internal implementation error: expected file name to be /
 at org.apache.hadoop.fs.viewfs.InternalDirOfViewFs.checkPathIsSlash(InternalDirOfViewFs.java:88) at org.apache.hadoop.fs.viewfs.InternalDirOfViewFs.getFileStatus(InternalDirOfViewFs.java:154) at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1684) at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1695) at org.apache.hadoop.fs.viewfs.ViewFileSystem.getContentSummary(ViewFileSystem.java:918) at org.apache.hadoop.fs.viewfs.ViewFileSystemBaseTest.testGetContentSummary(ViewFileSystemBaseTest.java:1106){code}
Bug: shelldoc fails in hadoop-common
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/20 07:50

{noformat}
[INFO] --- exec-maven-plugin:1.3.1:exec (shelldocs) @ hadoop-common ---
> ERROR: yetus-dl: gpg unable to import
> /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/patchprocess/KEYS_YETUS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time:  9.377 s
> [INFO] Finished at: 2020-05-28T17:37:41Z
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal
> org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (shelldocs) on project
> hadoop-common: Command execution failed. Process exited with an error: 1
> (Exit value: 1) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
{noformat}
* https://builds.apache.org/job/PreCommit-HADOOP-Build/16957/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt
* https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/155/artifact/out/patch-mvnsite-root.txt
* https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/157/artifact/out/patch-mvnsite-root.txt
Bug: ViewFS does not return correct user/group and ACL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/20 20:10

When doing ls on a mount point parent, the returned user/group ACL is incorrect. It always showing the user and group being current user, with some arbitrary ACL. Which could misleading any application depending on this API.

cc [~cliang] [~virajith]
Bug: listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectory
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/20 13:38
Affects Version/s: 3.0.0

listStatus implementation in ViewFs and getFileStatus does not return consistent values for an element on isDirectory value. listStatus returns isDirectory of all softlinks as false and getFileStatus returns isDirectory as true.
{code:java}
[hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus "/"
FileStatus of viewfs://c3121/testme21may isDirectory:false
FileStatus of viewfs://c3121/tmp isDirectory:false
FileStatus of viewfs://c3121/foo isDirectory:false
FileStatus of viewfs://c3121/tmp21may isDirectory:false
FileStatus of viewfs://c3121/testme isDirectory:false
FileStatus of viewfs://c3121/testme2 isDirectory:false <--- returns false
FileStatus of / isDirectory:true
[hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus /testme2
FileStatus of viewfs://c3121/testme2/dist-copynativelibs.sh isDirectory:false
FileStatus of viewfs://c3121/testme2/newfolder isDirectory:true
FileStatus of /testme2 isDirectory:true <--- returns true
[hdfs@c3121-node2 ~]$ {code}
Bug: Revert "HADOOP-8143. Change distcp to have -pb on by default"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/20 13:05
Affects Version/s: 3.0.3

revert the HADOOP-8143. "distcp -pb as default" feature as it was

* breaking s3a uploads
* breaking incremental uploads to any object store
Bug: ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root).
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/20 01:52
Affects Version/s: 3.2.2

As part of the design doc HDFS-15289, [~sanjay.radia] and me discussed the following scenarios when fallback enabled.

*Behavior when fallback enabled:*

       Assume FS trees and mount mappings like below:

           mount link /a/b/c/d  → hdfs://nn1/a/b

           mount link /a/p/q/r  → hdfs://nn2/a/b    

           fallback → hdfs://nn3/  $  /a/c
                                                 /x/z
 # Open(/x/y) then it goes to nn3 (fallback)      - WORKS
 # Create(/x/foo) then foo is created in nn3 in dir /x   - WORKS
 # ls /  should list   /a  /x .Today this does not work and IT IS A BUG!!! Because it conflicts with the open(/x/y)
 # Create /y  : fails  - also fails when not using  fallback  - WORKS
 # Create /a/z : fails - also fails when not using  fallback - WORKS
 # ls /a should list /b /p  as expected and will not show fallback in nn3 - WORKS

 

This Jira will fix issue of #3. So, when fallback enabled it should show merged ls view with mount links + fallback root. ( this will only be at root level)
Bug: ArrayIndexOfboundsException in ViewFileSystem#listStatus
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/20 17:10

In Viewfilesystem#listStatus , we get groupnames of ugi , If groupnames doesn't exists  it will throw AIOBE
{code:java}
        else {
          result[i++] = new FileStatus(0, true, 0, 0,
            creationTime, creationTime, PERMISSION_555,
            ugi.getShortUserName(), ugi.getGroupNames()[0],
            new Path(inode.fullPath).makeQualified(
                myUri, null));
        } {code}
Bug: Adding Network Counters in ABFS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jun/20 13:51
Affects Version/s: 3.3.0

Network Counters to be added in ABFS:
|CONNECTIONS_MADE|Number of times connection was made with Azure Data Lake|
|SEND_REQUESTS|Number of send requests|
|GET_RESPONSE|Number of response gotten|
|BYTES_SEND|Number of bytes send|
|BYTES_RECEIVED|Number of bytes received|
|READ_THROTTLE|Number of times throttled while read operation|
|WRITE_THROTTLE|Number of times throttled while write operation|

propose:
 * Adding these counters as part of AbfsStatistic already made in HADOOP-17016.
 * Increment of counters across Abfs Network services.
Bug: Update JaegerTracing
Status: Open | Priority: Major | Resolution: 
Created: 15/Apr/20 16:35
Affects Version/s: thirdparty-1.0.0

We currently use JaegerTracing 0.34.0. The latest is 1.2.0. We are several versions behind and should update. Note this update requires the latest version fo OpenTracing and has several breaking changes.