av9ash/GitBugs huggingface dataset:
Bug: Backslash in username causes build failure in the environment started by start-build-env.sh.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Oct/20 09:03

If a username includes a backslash, `mvn clean install` fails in an environment started by start-build-env.sh.

Here is my result in Amazon WorkSpaces.

 
{code:java}
CORPbtkuramototkr@b8e750b1e386:/home/CORP\btkuramototkr/hadoop/hadoop-build-to
ols$ mvn clean install
/usr/bin/mvn: 1: cd: can't cd to /home/CORtkuramototkr/hadoop/hadoop-build-tools/..
[INFO] Scanning for projects...
[INFO] 
[INFO] ----------------< org.apache.hadoop:hadoop-build-tools >----------------
[INFO] Building Apache Hadoop Build Tools 3.4.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-build-tools ---
[INFO] 
[INFO] --- maven-resources-plugin:3.0.1:copy-resources (copy-resources) @ hadoop-build-tools ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.074 s
[INFO] Finished at: 2020-10-05T02:51:53Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.0.1:copy-resources (copy-resources) on project hadoop-build-tools: Cannot create resource output directory: /home/CORP/btkuramototkr/hadoop/hadoop-build-tools/target/extra-resources -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
{code}
 

This problem can be solved by adding an option to change the path to maven's local repository in the container so that users can remove backslashes from their username.
Bug: Improve BUILDING.TXT file to clarify instructions for new users
Status: Resolved | Priority: Trivial | Resolution: Won't Fix
Created: 11/Dec/20 21:12
Affects Version/s: 2.10.1

In the BUILDING.TXT file on the apache/hadoop git repo, the instructions for Linux/macOS users needs an addition to show how to set a shell script as executable using sudo and then running the script. This should help avoid potential confusion for users who are unfamiliar with using executable shell scripts in the Linux/macOS terminal.
Bug: S3A marker tool mixes up -min and -max
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 27/Oct/20 18:12
Affects Version/s: 3.3.1

HADOOP-17227 manages to get -min and -max mixed up through the call chain,. 

{code}
hadoop s3guard markers -audit -max 2000  s3a://stevel-london/
{code}
leads to
{code}
2020-10-27 18:11:44,434 [main] DEBUG s3guard.S3GuardTool (S3GuardTool.java:main(2154)) - Exception raised
46: Marker count 0 out of range [2000 - 0]
	at org.apache.hadoop.fs.s3a.tools.MarkerTool$ScanResult.finish(MarkerTool.java:489)
	at org.apache.hadoop.fs.s3a.tools.MarkerTool.run(MarkerTool.java:318)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:505)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:2134)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:2146)
2020-10-27 18:11:44,436 [main] INFO  util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 46: 46: Marker count 0 out of range [2000 - 0]
{code}

Trivial fix.
Bug: S3A: IAMCredentialsProvider throttling results in AWS auth failures
Status: Open | Priority: Blocker | Resolution: 
Created: 20/May/24 10:01
Affects Version/s: 3.4.0

Tests report throttling errors in IAM being remapped to noauth and failure

Again, impala tests, but with multiple processes on same host. this means that HADOOP-18945 isn't sufficient as even if it ensures a singleton instance for a process
* it doesn't if there are many test buckets (fixable)
* it doesn't work across processes (not fixable)

we may be able to 
* use a singleton across all filesystem instances
* once we know how throttling is reported, handle it through retries + error/stats collection
Bug: EC: Fix calculation errors caused by special index order
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 11/May/24 10:16

I found that if the erasedIndexes distribution is such that the parity index is in front of the data index, ec will produce wrong results when decoding.

In fact, HDFS-15186 has described this problem, but does not fundamentally solve it.

The reason is that the code assumes that erasedIndexes is preceded by the data index and followed by parity index. If there is a parity index placed in front of the data index, a calculation error will occur.
Bug: TestStagingCommitter.testJobCommitFailure failing 
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 22/May/24 21:49

{code:java}
[INFO] 
[ERROR] Failures: 
[ERROR]   TestStagingCommitter.testJobCommitFailure:662 [Committed objects compared to deleted paths org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase$ClientResults@1b4ab85{ requests=12, uploads=12, parts=12, tagsByUpload=12, commits=5, aborts=7, deletes=0}] 
Expecting:
  <["s3a://bucket-name/output/path/r_0_0_0e1f4790-4d3f-4abb-ba98-2b39ec8b7566",
    "s3a://bucket-name/output/path/r_0_0_92306fea-0219-4ba5-a2b6-091d95547c11",
    "s3a://bucket-name/output/path/r_1_1_016c4a25-a1f7-4e01-918e-e24a32c7525f",
    "s3a://bucket-name/output/path/r_0_0_b2698dab-5870-4bdb-98ab-0ef5832eca45",
    "s3a://bucket-name/output/path/r_1_1_600b7e65-a7ff-4d07-b763-c4339a9164ad"]>
to contain exactly in any order:
  <[]>
but the following elements were unexpected:
  <["s3a://bucket-name/output/path/r_0_0_0e1f4790-4d3f-4abb-ba98-2b39ec8b7566",
    "s3a://bucket-name/output/path/r_0_0_92306fea-0219-4ba5-a2b6-091d95547c11",
    "s3a://bucket-name/output/path/r_1_1_016c4a25-a1f7-4e01-918e-e24a32c7525f",
    "s3a://bucket-name/output/path/r_0_0_b2698dab-5870-4bdb-98ab-0ef5832eca45",{code}
Bug: [JDK23] org.apache.hadoop.security.UserGroupInformation use of Subject needs to move to replacement APIs
Status: Open | Priority: Major | Resolution: 
Created: 28/Jun/24 08:02
Affects Version/s: 3.5.0

`javax.security.auth.Subject.getSubject` and `Subject.doAs` were deprecated for removal in JDK 17. The replacement APIs are `Subject.current` and `callAs`. See [JEP 411]([https://openjdk.org/jeps/411]) for background.

The `Subject.getSubject` API has been "degraded" in JDK 23 to throw `UnsupportedOperationException` if not running with the option to allow a SecurityManager. In a future JDK release, the `Subject.getSubject` API will be degraded further to throw`UnsupportedOperationException` unconditionally.

[renaissance/issues/439]([https://github.com/renaissance-benchmarks/renaissance/issues/439]) is a failure with a Spark benchmark due to the code in `org.apache.hadoop.security.UserGroupInformation` using the deprecated `Subject.getSubject` method. The maintainers of this code need to migrate this code to the replacement APIs to ensure that this code will continue to work once the security manager feature is removed.
Bug: ABFS: Support FNS Accounts over BlobEndpoint
Status: Open | Priority: Major | Resolution: 
Created: 15/May/24 19:44
Affects Version/s: 3.4.0

As a pre-requisite to deprecating WASB Driver, ABFS Driver will need to match FNS account support as intended by WASB driver. This will provide an official migrating means for customers still using the legacy driver to ABFS Driver. 

 

Parent Jira for WASB deprecation: [HADOOP-19178] WASB Driver Deprecation and eventual removal - ASF JIRA (apache.org)
Bug: S3A prefetching to support Vector IO
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 04/Apr/24 16:42
Affects Version/s: 3.4.0

Add explicit support for vector IO in s3a prefetching stream.

* if a range is in 1+ cached block, it SHALL be read from cache and returned
* if a range is not in cache : TBD
* If a range is partially in cache: TBD

these are the same decisions that abfs has to make: should the client fetch/cache block or just do one or more GET requests

A big issue is: does caching of data fetched in a range request make any sense at all? Or more specifically: does fetching the blocks in which range requests are found make sense

Simply going to the store is a lot simpler
Bug: ABFS: [FnsOverBlob] Response Handling of Blob Endpoint APIs and Metadata APIs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/24 10:28
Affects Version/s: 3.4.0

Blob Endpoint APIs has a different format for response than DFS Endpoint APIs.
There are some behavioral differences as well that need to be handled at client side.
Bug: ABFS: [FnsOverBlob] Making AbfsClient Abstract for supporting both DFS and Blob Endpoint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/24 05:59
Affects Version/s: 3.4.0

Azure Services support two different set of APIs.
Blob: [https://learn.microsoft.com/en-us/rest/api/storageservices/blob-service-rest-api] 
DFS: [https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/operation-groups] 

As per the plan in HADOOP-19179, this task enables ABFS Driver to work with both set of APIs as per the requirement.

Scope of this task is to refactor the ABfsClient so that ABFSStore can choose to interact with the client it wants based on the endpoint configured by user.

The blob endpoint support will remain "Unsupported" until the whole code is checked-in and well tested.
Bug: Include FileStatus when opening a file from FileSystem
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 08/Jun/24 11:07
Affects Version/s: 3.4.0

The FileSystem abstract class prevents that if you have information about the FileStatus of a file, you use it to open that file, which means that in the implementations of the open method, they have to request the FileStatus of the same file again, making unnecessary requests.

A very clear example is seen in today's latest version of the parquet-hadoop implementation, where:

https://github.com/apache/parquet-java/blob/apache-parquet-1.14.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/util/HadoopInputFile.java

Although to create the implementation you had to consult the file to know its FileStatus, when opening it only the path is included, since the FileSystem implementation is the only thing it allows you to do. This implies that the implementation will surely, in its open function, verify that the file exists or what information the file has and perform the same operation again to collect the FileStatus.

 

This would simply be resolved by taking the latest current version:

 

[https://github.com/apache/hadoop/blob/release-3.4.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java]

and including the following:

 

  public FSDataInputStream open(FileStatus f) throws IOException {
        return this.open(f.getPath(), this.getConf().getInt("io.file.buffer.size", 4096));
    }

 

This would imply that it is backward compatible with all current Filesystems, but since it is in the implementation it could be used when this information is already known.
Bug: S3A: Support AWS KMS Encryption Context
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/24 23:33
Affects Version/s: 3.4.0

S3A properties allow users to choose the AWS KMS key ({_}fs.s3a.encryption.key{_}) and S3 encryption algorithm to be used (f{_}s.s3a.encryption.algorithm{_}). In addition to the AWS KMS Key, an encryption context can be used as non-secret data that adds additional integrity and authenticity to check the encrypted data. However, there is no option to specify the [AWS KMS Encryption Context|https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context] in S3A.

In AWS SDK v2 the encryption context in S3 requests is set by the parameter [ssekmsEncryptionContext.|https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/model/CreateMultipartUploadRequest.Builder.html#ssekmsEncryptionContext(java.lang.String)] It receives a base64-encoded UTF-8 string holding JSON with the encryption context key-value pairs. The value of this parameter could be set by the user in a new property {_}*fs.s3a.encryption.context*{_}, and be stored in the [EncryptionSecrets|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/EncryptionSecrets.java] to later be used when setting the encryption parameters in [RequestFactoryImpl|https://github.com/apache/hadoop/blob/f92a8ab8ae54f11946412904973eb60404dee7ff/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java].
Bug: S3A: handle alternative forms of connection failure
Status: Open | Priority: Major | Resolution: 
Created: 13/May/24 13:15
Affects Version/s: 3.3.6

We've had reports of network connection failures surfacing deeper in the stack where we don't convert to AWSApiCallTimeoutException so they aren't retried properly (retire connection and repeat)


{code}
Unable to execute HTTP request: Broken pipe (Write failed)
{code}


{code}
 Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. (Service: Amazon S3; Status Code: 400; Error Code: RequestTimeout
{code}

note, this is v1 sdk but the 400 error is treated as fail-fast in all our versions and I don't think we do the same for the broken pipe. that one is going to be trickier to handle as unless that is coming from the http/tls libraries "broken pipe" may not be in the newer builds. We'd have to look for the string in the SDKs to see what causes it and go from there
Bug: S3A Xattr headers need hdfs-compatible prefix
Status: Open | Priority: Major | Resolution: 
Created: 15/May/24 13:02
Affects Version/s: 3.3.6

x3a xattr list needs a prefix compatible with hdfs or existing code which tries to copy attributes between stores can break

we need a prefix of {user/trusted/security/system/raw}.

now, problem: currently xattrs are used by the magic committer to propagate file size progress; renaming the prefix will break existing code. But as it's read only we could modify spark to look for both old and new values.

{code}

org.apache.hadoop.HadoopIllegalArgumentException: An XAttr name must be prefixed with user/trusted/security/system/raw, followed by a '.'
	at org.apache.hadoop.hdfs.XAttrHelper.buildXAttr(XAttrHelper.java:77) 
	at org.apache.hadoop.hdfs.DFSClient.setXAttr(DFSClient.java:2835) 
	at org.apache.hadoop.hdfs.DistributedFileSystem$59.doCall(DistributedFileSystem.java:3106) 
	at org.apache.hadoop.hdfs.DistributedFileSystem$59.doCall(DistributedFileSystem.java:3102) 
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.setXAttr(DistributedFileSystem.java:3115) 
	at org.apache.hadoop.fs.FileSystem.setXAttr(FileSystem.java:3097)

{code}
Bug: Test ITestAbfsRestOperationException#testAuthFailException is broken. 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Apr/24 17:47

{code:java}
intercept(Exception.class,
        () -> {
          fs.getFileStatus(new Path("/"));
        }); {code}
Intercept shouldn't be used as there are assertions in catch statements. 

 

CC [~stevel@apache.org]  [~anujmodi2021] [~asrani_anmol]
Bug: ABFS: Fixing logic to determine HNS nature of account to avoid extra getAcl() calls
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/24 16:56
Affects Version/s: 3.4.0

ABFS driver needs to know the type of account being used. It relies on the user to inform the account type using the config `fs.azure.account.hns.enabled`.
If not configured, driver makes a getAcl call to determine the account type.

Expectation is getAcl() will fail with 400 Bad Request if made on the FNS Account.
For any other case including 200, 404 it will indicate account is HNS.

Today, when determining this, the logic only checks status code to be either 200 or 400. In case of 404, nothing is inferred, and this leads to more getAcl again and again until 200 or 400 comes.

Fix is to update the logic such that if getAcl() fails with 400, it is FNS account. For all other cases it will be an HNS account. In case of throttling, if all retries are exhausted, FS init itself will fail.

This is also to fix a test case failing on trunk. {{testGetAclCallOnHnsConfigAbsence(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate)}}
Bug: Support configurable SASL mechanism
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Apr/24 18:23

Currently, the SASL mechanism is hard coded to DIGEST-MD5.  As mentioned in HADOOP-14811, DIGEST-MD5 is known to be insecure; see [rfc6331|https://datatracker.ietf.org/doc/html/rfc6331].

In this JIRA, we will make the SASL mechanism configurable.  The default mechanism will still be DIGEST-MD5 in order to maintain compatibility.
Bug: upgrade bouncy castle to 1.78.1 due to CVEs
Status: Open | Priority: Major | Resolution: 
Created: 19/Apr/24 09:10
Affects Version/s: 3.3.6

[https://www.bouncycastle.org/releasenotes.html#r1rv78]

There is a v1.78.1 release but no notes for it yet.

For v1.78
h3. 2.1.5 Security Advisories.

Release 1.78 deals with the following CVEs:
 * CVE-2024-29857 - Importing an EC certificate with specially crafted F2m parameters can cause high CPU usage during parameter evaluation.
 * CVE-2024-30171 - Possible timing based leakage in RSA based handshakes due to exception processing eliminated.
 * CVE-2024-30172 - Crafted signature and public key can be used to trigger an infinite loop in the Ed25519 verification code.
 * CVE-2024-301XX - When endpoint identification is enabled and an SSL socket is not created with an explicit hostname (as happens with HttpsURLConnection), hostname verification could be performed against a DNS-resolved IP address. This has been fixed.
Bug: Explore dropping protobuf 2.5.0 from the distro
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/24 17:32

explore if protobuf-2.5.0 can be dropped from distro, it is a transitive dependency from HBase, but HBase doesn't use it in the code.

Check if it is the only one pulling it into the distro & will something break if we exclude that, if none lets get rid of it
Bug: Hadoop CLI MiniCluster is broken
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/24 20:52

Documentation is also broken & it doesn't work either

(https://apache.github.io/hadoop/hadoop-project-dist/hadoop-common/CLIMiniCluster.html)

*Fails with:*
{noformat}
Exception in thread "main" java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
	at org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2666)
	at org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2680)
	at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1510)
	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:989)
	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:588)
	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:530)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:160)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	... 9 more{noformat}
{*}Command executed:{*}
{noformat}
bin/mapred minicluster -format{noformat}
*Documentation Issues:*
{noformat}
bin/mapred minicluster -rmport RM_PORT -jhsport JHS_PORT{noformat}

Without -format option it doesn't work the first time telling Namenode isn't formatted, So, this should be corrected.


{noformat}
2024-05-04 00:35:52,933 WARN namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:253)
{noformat}

This isn't required either:

{noformat}
NOTE: You will need protoc 2.5.0 installed.
{noformat}
Bug: S3A: Support external id in assume role
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/24 13:53
Affects Version/s: 3.4.0

Extend IAM role suport with external IDs which can be set in fs.s3a.assumed.role.external.id
Support external id in AssumedRoleCredentialProvider.

 

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html
Bug: S3A initialization/close slower than with v1 SDK
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/24 16:23
Affects Version/s: 3.4.0

Hive QE have observed slowdown in LLAP queries due to time to create and close s3a filesystems instances. A key aspect of that is they keep closing the fs instances (HIVE-27884), but looking at the profiles, the reason things seem to have regressed is

* two s3 clients are being created (sync and async)
* these seem to take a lot of time scanning the classpath for "global interceptors", which is at least an O(jars) operation; #of index entries in the zip files may factor too.

Proposed:
* create async client on demand when the transfer manager is invoked
* look at why passwords are being scanned for if InstanceProfileCredentialsProvider is in use...that seems slow too

SDK wishes
* SDK maybe allow us to turn off that scan for interceptors?

attaching screenshots of the profile. storediag snippet:
{code}

[001]  fs.s3a.access.key = (unset)
[002]  fs.s3a.secret.key = (unset)
[003]  fs.s3a.session.token = (unset)
[004]  fs.s3a.server-side-encryption-algorithm = (unset)
[005]  fs.s3a.server-side-encryption.key = (unset)
[006]  fs.s3a.encryption.algorithm = (unset)
[007]  fs.s3a.encryption.key = (unset)
[008]  fs.s3a.aws.credentials.provider = "com.amazonaws.auth.InstanceProfileCredentialsProvider" [core-site.xml]

{code}
Bug: S3A: option "fs.s3a.performance.flags" to take list of performance flags
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/24 17:33
Affects Version/s: 3.4.1

HADOOP-19072 shows we want to add more optimisations than that of HADOOP-18930.

* Extending the new optimisations to the existing option is brittle
* Adding explicit options for each feature gets complext fast.

Proposed
* A new class S3APerformanceFlags keeps all the flags
* it build this from a string[] of values, which can be extracted from getConf(),
* and it can also support a "*" option to mean "everything"
* this class can also be handed off to hasPathCapability() and do the right thing.

Proposed optimisations
* create file (we will hook up HADOOP-18930)
* mkdir (HADOOP-19072)
* delete (probe for parent path)
* rename (probe for source path)

We could think of more, with different names, later.
The goal is make it possible to strip out every HTTP request we do for safety/posix compliance, so applications have the option of turning off what they don't need.