av9ash/GitBugs huggingface dataset:
Bug: Install Python 3 for Windows 10 docker image
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/23 07:52
Affects Version/s: 3.4.0

Currently, mvnsite build phase fails due to the following error -

{code}
[INFO] ------------------< org.apache.hadoop:hadoop-common >-------------------
[INFO] Building Apache Hadoop Common 3.4.0-SNAPSHOT                    [11/114]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ hadoop-common ---
[INFO] Deleting C:\hadoop\hadoop-common-project\hadoop-common\target
[INFO] Deleting C:\hadoop\hadoop-common-project\hadoop-common\src\site\markdown (includes = [UnixShellAPI.md], excludes = [])
[INFO] Deleting C:\hadoop\hadoop-common-project\hadoop-common\src\site\resources (includes = [configuration.xsl, core-default.xml], excludes = [])
[INFO] 
[INFO] --- exec-maven-plugin:1.3.1:exec (shelldocs) @ hadoop-common ---
tar: apache-yetus-0.14.0/lib/precommit/qbt.sh: Cannot create symlink to 'test-patch.sh': No such file or directory
tar: Exiting with failure status due to previous errors
/usr/bin/env: 'python3': No such file or directory
{code}

Thus, we need to install Python 3 in the Windows 10 Hadoop builder docker image to fix this.
Bug: AWS SDK V2 - ITestS3AHugeFilesNoMultipart failure
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 22/May/23 13:14
Affects Version/s: 3.4.0

ITestS3AHugeFilesNoMultipart fails with

java.lang.AssertionError: Expected a org.apache.hadoop.fs.s3a.api.UnsupportedRequestException to be thrown, but got the result: : true

Happens because the transfer manager currently does not do any MPU when used with the Java async client, so the UnsupportedRequestException never gets thrown.
Bug: CachingBlockManager to use AtomicBoolean for closed flag
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/23 14:12
Affects Version/s: 3.3.9

the {{CachingBlockManager}} uses the boolean field {{closed)) in various operations, including a do/while loop. to ensure the flag is correctly updated across threads, it needs to move to an atomic boolean.
Bug: Delete path directly when it can not be parsed in trash
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jun/23 07:35
Affects Version/s: 3.4.0

If we move path to trash dir directly rather than use delete API or rm command, when 

invoke deleteCheckpoint method, it will catch ParseException and ignore deleting the path. It will never be deleted, so we should do something to prevent or monitor it.

Some logs are listed below.

 
{code:java}
WARN org.apache.hadoop.fs.TrashPolicyDefault: Unexpected item in trash: /user/de_eight/.Trash/college_geek_job_recall_als_modelres_5_2_6.del. Ignoring.

WARN org.apache.hadoop.fs.TrashPolicyDefault: Unexpected item in trash: /user/de_eight/.Trash/college_geek_addf_vector. Ignoring.
 {code}
Bug: Upgrade Apache Derby from 10.10.2.0 to 10.14.2.0 due to CVEs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Apr/23 11:06
Affects Version/s: 3.4.0

[https://github.com/advisories/GHSA-wr69-g62g-2r9h]

[https://github.com/advisories/GHSA-42xw-p62x-hwcf]

[https://github.com/apache/hadoop/pull/5427]

Only seems to be used in test scope but it would be nice to silence the dependabot warnings by merging the PR.
Bug: s3a prefetch cache blocks should be accessed by RW locks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/May/23 21:24
Affects Version/s: 3.3.6

In order to implement LRU or LFU based cache removal policies for s3a prefetched cache blocks, it is important for all cache reader threads to acquire read lock and similarly cache file removal mechanism (fs close or cache eviction) to acquire write lock before accessing the files.

As we maintain the block entries in an in-memory map, we should be able to introduce read-write lock per cache file entry, we don't need coarse-grained lock shared by all entries.

 

This is a prerequisite to HADOOP-18291.
Bug: openFile builder new optLong() methods break hbase-filesystem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 17:23
Affects Version/s: 3.3.6

The new methods of HADOOP-18724 break builds of hbase filesystem because HBASE-26483 added an implementation of the interface, which,
because it lacks the complete methods, doesn't compile any more.

It also marked the (deprecated) methods as final in the base implementation, which is also a source of pain.
Bug: Upgrade aws-java-sdk to 1.12.367+
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/23 10:50
Affects Version/s: 3.3.5


aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is pulling in high severity CVE and creating unhappiness in security scans, even if s3a doesn't use that lib. 

The safe version for netty is netty:4.1.86.Final and this is used by aws-java-adk:1.12.367+
Bug: build/release a shaded version of the AWS SDK
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 01/Jun/23 12:42
Affects Version/s: 3.4.0

we need a shaded version of the AWS SDK to stay in control of our dependencies.

Either we somehow get the AWS dev team to do it, or we do it ourselves in the hadoop-thirdparty repository.

Doing it ourselves would actually give us a bit more control as we have the possibility of making a lighter weight release. It would reduce the control of people downstream though, as they wouldn't be able to "just" drop in a new JAR
Bug: convert declarations of AWS v1 SDK EnvironmentVariableCredentialsProvider to v2 version
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 17/May/23 13:44

As I play with the v2 sdk I've cut the v1 sdk from the tools/lib and now I'm getting stack traces about missing class EnvironmentVariableCredentialsProvider

{code}
java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class com.amazonaws.auth.EnvironmentVariableCredentialsProvider not found
        at org.apache.hadoop.fs.s3a.auth.AwsCredentialListProvider.loadAWSProviderClasses(AwsCredentialListProvider.java:128)
        at org.apache.hadoop.fs.s3a.auth.AwsCredentialListProvider.buildAWSProviderList(AwsCredentialListProvider.java:167)
        at org.apache.hadoop.fs.s3a.auth.AwsCredentialListProvider.createAWSCredentialProviderSet(AwsCredentialListProvider.java:102)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:945)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:597)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3595)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:171)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3696)
        at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3653)
        at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:608)
        at org.apache.hadoop.fs.store.diag.StoreDiag.executeFileSystemOperations(StoreDiag.java:753)
 
this provider is listed in the fs.s3a.aws.credentials.provider chain

{code}
<property>
  <name>fs.s3a.aws.credentials.provider</name>
  <value>
    org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
    org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
    com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
    org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
  </value>
...

{code}

Ideally we should remove all v1 dependencies, explicitly (here) and implicitly.

maybe for the env vars, we should consider adding our own env var provider which could go into branch-3.3 *now* as a single patch, which can then be cherrypicked by anyone who wants it in older releases; the v2 version will work with the new api
Bug: remove use of com.google.common.base.Charsets;
Status: Open | Priority: Major | Resolution: 
Created: 14/Jun/23 12:32
Affects Version/s: 3.3.9

just noticed there's references to com.google.common.base.Charsets /thirdparty equivalent just for Charsets.UTF_8 references.

we can cut these and remove one more guava version pain point
Bug: Add OptimizedS3AMagicCommitter For Zero Rename Commits to S3 Endpoints
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 19/Jun/23 08:45

The goal is to add a new S3A committer named *OptimizedS3AMagicCommitter* which is an another type of S3 Magic committer but with a better performance by taking in few tradeoffs.

The following are the differences in MagicCommitter vs OptimizedMagicCommitter

 
||Operation||Magic Committer||*OptimizedS3AMagicCommitter*||
|commitTask    |1. Lists all {{.pending}} files in its attempt directory.
 
2. The contents are loaded into a list of single pending uploads.
 
3. Saved to a {{.pendingset}} file in the job attempt directory.|1. Lists all {{.pending}} files in its attempt directory
 
2. The contents are loaded into a list of single pending uploads.
 
3. For each pending upload, commit operation is called (complete multiPartUpload)|
|commitJob|1. Loads all {{.pendingset}} files in its job attempt directory
 
2. Then every pending commit in the job will be committed.
 
3. "SUCCESS" marker is created (if config is enabled)
 
4. "__magic" directory is cleaned up.|1. "SUCCESS" marker is created (if config is enabled)
 
2.  "__magic" directory is cleaned up.|

 

*Performance Benefits :-*
 # The primary performance boost due to distributed complete multiPartUpload call being made in the taskAttempts(Task containers/Executors) rather than a single job driver. In case of MagicCommitter it is O(files/threads).
 # It also saves a couple of S3 calls needed to PUT the "{{{}.pendingset{}}}" files and READ call to read them in the Job Driver.

 

*TradeOffs :-*

The tradeoffs are similar to the one in FileOutputCommitter V2 version. Users migrating from FileOutputCommitter V2 to OptimizedS3AMagicCommitter will no see behavioral change as such
 # During execution, intermediate data becomes visible after commitTask operation
 # On a failure, all output must be deleted and the job needs to be restarted.

 

*Performance Benchmark :-*

Cluster : c4.8x large (ec2-instance)
Instance : 1 (primary) + 5 (core)
Data Size : 3TB Partitioned(TPC-DS store_sales data)
Engine     : Apache Spark 3.3.1 / Hadoop 3.3.3

Query: The following query inserts around 3000+ files into the table directory (ran for 3 iterations)
{code:java}
insert into <table> select ss_quantity from store_sales; {code}
||Committer||Iteration 1||Iteration 2||Iteration 3||
|Magic|126|127|122|
|OptimizedMagic|50|51|58|

So on an average, OptimizedMagicCommitter was *~2.3x* faster as compared to MagicCommitter.

 

_*Note: Unlike MagicCommitter , OptimizedMagicCommitter is not suitable for all the cases where in user requires the guarantees of file not being visible in failure scenarios. Given the performance benefit, user can may choose to use this if they don't require any guarantees or have some mechanism to clean up the data before retrying.*_
Bug: Parallelize concatenation of distcp chunks of separate files in CopyCommitter
Status: Open | Priority: Major | Resolution: 
Created: 10/May/23 20:30
Affects Version/s: 3.3.6

While copying a folder containing large files consisting of multiple distcp chunks, copy committer synchronously picks chunks of each file and concatenates them. This part can be improved by parallelizing the concatenation of distcp chunks of separate files. We are able to save 2-3 minutes while copying a folder of 100 GB containing 20 files of 5GB size with this improvement.

Contributing a patch for this.
Bug: Fix '-Dbundle.pmdk' does not take effect, checknative pmdk shows error
Status: Open | Priority: Major | Resolution: 
Created: 09/May/23 09:27

Fix '-Dbundle.pmdk' does not take effect, checknative pmdk shows error

before fix:

!image-2023-05-10-14-33-49-601.png!

after fix:

!image-2023-05-10-14-34-02-922.png|width=583,height=32!
Bug: Improve hadoop-function.sh#status script
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/23 09:34
Affects Version/s: 3.4.0

When we use yarn --daemon status router/nodemanager, I found that no information can be displayed. I checked the script and modified the script of hadoop-functions.sh#status to increase the output information.

yarn --daemon status router

- If the service is started
router is running as process 47626.

- If the service is stop
router is stopped.
Bug: Add tags 3.3/3.3.5 to the official Hadoop docker image
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/23 23:29

Our official docker repository has just two tags for the images: 2 and 3.

Shall we add more tags to the images? For example Hadoop 3 has two release lines: 3.2 and 3.3. And probably even maintenance release tags: 3.3.5, 3.3.6. This way, downstream users will not have surprises when we make a new release.
Bug: Update jackson2 version from 2.12.7.1 to 2.15.0
Status: Open | Priority: Major | Resolution: 
Created: 20/Jun/23 10:31

can the jackson2 version in hadoop-project be updated from 2.12.7.1 to 2.15.*

This is to rectify the following vulnerability

[https://github.com/FasterXML/jackson-core/pull/827]

https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L72
Bug: Integrating Apache Hadoop into OSS-Fuzz
Status: Open | Priority: Major | Resolution: 
Created: 12/Jun/23 16:09

Hi all,

We have prepared the [initial integration|https://github.com/google/oss-fuzz/pull/10511] of Apache Hadoop into [Google OSS-Fuzz|https://github.com/google/oss-fuzz] which will provide more security for your project.

 

*Why do you need Fuzzing?*
The Code Intelligence JVM fuzzer [Jazzer|https://github.com/CodeIntelligenceTesting/jazzer] has already found [hundreds of bugs|https://github.com/CodeIntelligenceTesting/jazzer/blob/main/docs/findings.md] in open source projects including for example [OpenJDK|https://nvd.nist.gov/vuln/detail/CVE-2022-21360], [Protobuf|https://nvd.nist.gov/vuln/detail/CVE-2021-22569] or [jsoup|https://github.com/jhy/jsoup/security/advisories/GHSA-m72m-mhq2-9p6c]. Fuzzing proved to be very effective having no false positives. It provides a crashing input which helps you to reproduce and debug any finding easily. The integration of your project into the OSS-Fuzz platform will enable continuous fuzzing of your project by [Jazzer|https://github.com/CodeIntelligenceTesting/jazzer].

 

*What do you need to do?*
The integration requires the maintainer or one established project committer to deal with the bug reports.

You need to create or provide one email address that is associated with a google account as per [here|https://google.github.io/oss-fuzz/getting-started/accepting-new-projects/]. When a bug is found, you will receive an email that will provide you with access to ClusterFuzz, crash reports, code coverage reports and fuzzer statistics. More than 1 person can be included.

 

*How can Code Intelligence support you?*
We will continue to add more fuzz targets to improve code coverage over time. Furthermore, we are permanently enhancing fuzzing technologies by developing new fuzzers and bug detectors.

 

Please let me know if you have any questions regarding fuzzing or the OSS-Fuzz integration.
Bug: Incorporate Qiniu Cloud Kodo File System Implementation
Status: Open | Priority: Major | Resolution: 
Created: 09/Jun/23 04:41
Affects Version/s: 3.3.9

Qiniu Kodo is a self-developed unstructured data storage management platform by Qiniu Cloud Storage that supports center and edge storage. The platform has been verified by a large number of users for many years and has been widely used in various scenarios of massive data management. It is widely used in many cloud service users in China, but currently in the Apache Hadoop project, there is a lack of a solution that supports Kodo through Hadoop/Spark directly.

The purpose of this project is to integrate Kodo into Hadoop/Spark projects, so that users can operate Kodo through the API of Hadoop/Spark without additional learning costs.
Bug: Fix transient failure of ITestS3APrefetchingInputStream#testRandomReadLargeFile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/23 18:51
Affects Version/s: 3.4.0

intermittent failure of test; caller in uk, no VPN involved. does not always surface

{code}
testRandomReadLargeFile(org.apache.hadoop.fs.s3a.ITestS3APrefetchingInputStream)  Time elapsed: 19.913 s  <<< FAILURE!
org.junit.ComparisonFailure: [Gauge named stream_read_blocks_in_cache with expected value 3] expected:<[3]L> but was:<[2]L>

{code}
Bug: Backport S3A prefetching stream to branch-3.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/23 16:12
Affects Version/s: 3.3.9

Jira to cover work for backporting the prefetching into branch-3.3

I'm thinking of the best way to do do this. Proposed

# new feature branch
# cherrypick from trunk all current changes for it
# squash into one single pr, the backport change
# merge that into branch-3.3
# then for all the outstanding issues add them separately to trunk and branch-3.3
Bug: S3A ITestS3ABucketExistence access point test failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/23 18:44
Affects Version/s: 3.3.9

this is inevitably me having some config option in the way; just need to find out what and clear it for the test case so the probes are to the same region/endpont as the mock bucket

{code}
[ERROR] testAccessPointProbingV2(org.apache.hadoop.fs.s3a.ITestS3ABucketExistence)  Time elapsed: 1.748 s  <<< ERROR!
java.lang.IllegalArgumentException: The region field of the ARN being passed as a bucket parameter to an S3 operation does not match the region the client was configured with. Provided region: 'eu-west-1'; client region: 'eu-west-2'.
        at com.amazonaws.services.s3.AmazonS3Client.validateIsTrue(AmazonS3Client.java:6588)


{code}
Bug: S3AFileSystem doesn't consistently handle prefixes that are both files and directories between versions
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 24/May/23 18:36
Affects Version/s: 3.3.4

We have a prefix structure where the prefix Spark reads is both a file and a directory. So s3://a/b is the file we are trying to read, but s3://a/b/c is also a file. In 3.2.1, listStatuses identifies a/b as a File, but a change in 3.3.4 now identifies a/b as a directory and tries to read a/b/c instead of a/b.

When s3GetFileStatus is called on the path with StatusProbeEnum HEAD, the path does return as "File". However innerListStatus first assumes that any prefix that is "nonempty" is a directory; it only calls s3GetFileStatus on empty directories and on listObjects results of the prefix.

Wonder if this is known/if there are any suggestions to get around this without changing the prefix structure?
Bug: Error message when multipart disabled and buffering is array not useful
Status: Open | Priority: Major | Resolution: 
Created: 10/May/23 18:50
Affects Version/s: 3.3.9

testing branch-3.3 on a bucket with array buffering, I get a fairly useless error message.
Problem here is that AbstractCommitITest forces fs.s3a.fast.upload.buffer to array and somehow multipart is disabled.

# error text needs fixing (move to %s)
# s3a fs should fail in init if the threshold is isn't allowed for the store type
# then determine why the test is failing, and fix.


The tests do work in trunk, so something in branch-3.3 is stopping this


Also 

* correct the name of the constant STORE_CAPABILITY_DIRECTORY_MARKER_MULTIPART_UPLOAD_ENABLED to match the actual string "fs.s3a.capability.multipart.uploads.enabled"; accidental copy there

* capability FS_MULTIPART_UPLOADER must declare itself as disabled when multipart is off
    

{code}

[ERROR] testRevertCommit(org.apache.hadoop.fs.s3a.commit.ITestCommitOperations)  Time elapsed: 0.688 s  <<< ERROR!
java.lang.IllegalArgumentException: Invalid block size: %d [-1]
        at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:205)
        at org.apache.hadoop.fs.s3a.S3ADataBlocks$ArrayBlockFactory.create(S3ADataBlocks.java:397)
        at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:235)
        at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.<init>(S3ABlockOutputStream.java:217)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerCreateFile(S3AFileSystem.java:1887)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$create$7(S3AFileSystem.java:1789)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2479)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2498)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:1788)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1210)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1091)
        at org.apache.hadoop.fs.contract.ContractTestUtils.createFile(ContractTestUtils.java:650)
        at org.apache.hadoop.fs.contract.ContractTestUtils.touch(ContractTestUtils.java:686)
        at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.testRevertCommit(ITestCommitOperations.java:543)
{code}
Bug: Support a "permissive" mode for secure clusters to allow "simple" auth clients
Status: Open | Priority: Major | Resolution: 
Created: 17/Apr/23 02:25
Affects Version/s: 2.10.3

Similar to HBASE-14700, would like to add support for Secure Server to fallback to simple auth for non-secure clients.



Secure Hadoop to support a permissive mode to allow mixed secure and insecure clients. This allows clients to be incrementally migrated over to a secure configuration. To enable clients to continue to connect using SIMPLE authentication when the cluster is configured for security, set "hadoop.ipc.server.fallback-to-simple-auth-allowed" equal to "true" in hdfs-site.xml. NOTE: This setting should ONLY be used as a temporary measure while converting clients over to secure authentication. It MUST BE DISABLED for secure operation.
Bug: Support VFSFileSystem based on apache commons vfs
Status: Open | Priority: Major | Resolution: 
Created: 06/May/23 01:25
Affects Version/s: 3.4.0

Common vfs supports multiple types of storage, different archives, and a variety of compression algorithms

 

simple demo:  [https://github.com/melin/spark-jobserver/blob/master/jobserver-extensions/src/test/scala/com/github/melin/jobserver/extensions/sql/]

spark.read.option("header", "true")
      .csv("vfs://tgz:[ftp://fcftp:fcftp@172.18.1.52/csv.tar.gz!/csv]").show()
spark.read.option("header", "true")
.csv("vfs://tgz:s3://BxiljVd5YZa3mRUn:3Mq9dsmdMbN1JipE1TlOF7OuDkuYBYpe@cdh1:9300/demo-bucket/csv.tar.gz!/csv").show()
 
spark.read.option("header", "true")
.csv("vfs://tgz:sftp:///test:test2023@172.18.5.46:22/ftpdata/csv.tar.gz!/csv").show()

 

[~prabhujoseph]