av9ash/GitBugs huggingface dataset:
Bug: hadoop-auth module cannot import non-guava implementation in hatoop util
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 10/Oct/21 19:47

hadoop-common  provides several util implementations in {{org.apache.hadoop.util.*}}. Since hadoop-common depends on hadoop-auth, all the utility implementations cannot be used within hadoop-auth.

There are several options:
* similar to {{hadoop-annotations}} generic and utility implementations such as maps, Strings, Preconditions, ..etc could be moved to a new common-util module that has no dependency on other modules.
* easier fix is to manually replace the guava calls in hadoop-auth module without importing {{hadoop.util.*}}. Only few calls need to be manually replaced: {{Splitter}}, {{Preconditions.checkNotNull}}, and {{Preconditions.checkArgument}}

CC: [~vjasani] , [~stevel@apache.org], [~tasanuma]
Bug: Incorrect Class Name supplied to Logger in AbfsRestOperation.java & TracingContext.java
Status: Patch Available | Priority: Major | Resolution: 
Created: 07/Oct/21 13:31

The class name provided while initializing org.slf4j.Logger (using LoggerFactory.getLogger(...) method) is incorrect for following classes.
 # [AbfsRestOperation.java|https://github.infra.cloudera.com/CDH/hadoop/blob/cdpd-master/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L62]
 # [TracingContext.java|https://github.infra.cloudera.com/CDH/hadoop/blob/cdpd-master/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java#L66]

This makes debugging logs difficult.
Bug: Improve logging for abfs rename failures.
Status: Open | Priority: Major | Resolution: 
Created: 01/Oct/21 07:42

[https://github.infra.cloudera.com/CDH/hadoop/blob/cdpd-master/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java#L431] should be warn.
Bug: s3 and abfs incremental listing: use SAX parsers to stream results to list iterators
Status: Open | Priority: Minor | Resolution: 
Created: 11/Oct/21 15:27
Affects Version/s: 3.3.2

With code gradually adopting listStatusIncremental(), asking for a smaller initial batch could permit faster ramp of result processing.

probably most significant on an s3 versioned bucket, as there the need to skip tombstones can result in significantly slower listings -but could benefit ABFS too
Bug: Add a method appendIfAbsent for CallerContext
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Nov/21 01:47
Affects Version/s: 3.3.5

As we discussed here [#3635.|#discussion_r746873078]

In some cases, when we need to add a _key:value_ to the {_}CallerContext{_}, we need to check whether the _key_ already exists in the outer layer, which is a bit of a hassle. To solve this problem, we can add a new method {_}CallerContext#appendIfAbsent{_}.
Bug: CachedDNSToSwitchMapping#reloadCachedMappings is not consistent with CachedDNSToSwitchMapping#resolve
Status: Open | Priority: Minor | Resolution: 
Created: 28/Dec/21 08:20

The follow test case failed.

{code}
public class TestCachedDNSToSwitchMapping {

  @Test
  public void testReloadCachedMappings() {
    StaticMapping.resetMap();
    StaticMapping.addNodeToRack("127.0.0.1", "/rack0");
    StaticMapping.addNodeToRack("notexisit.host.com", "/rack1");
    CachedDNSToSwitchMapping cacheMapping =
        new CachedDNSToSwitchMapping(new StaticMapping());
    List<String> names = new ArrayList<>();
    names.add("localhost");
    names.add("notexisit.host.com");
    cacheMapping.resolve(names);
    Assert.assertTrue(cacheMapping.getSwitchMap().containsKey("127.0.0.1"));
    Assert.assertTrue(cacheMapping.getSwitchMap().containsKey("notexisit.host.com"));
    cacheMapping.reloadCachedMappings(names);
    // failed here
    Assert.assertEquals(0, cacheMapping.getSwitchMap().keySet().size());
  }
}
{code}
Bug: Update CompressionCodecFactory to handle uppercase file extensions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Dec/21 00:11

I've updated the CompressionCodecFactory to be able to handle filenames with capitalized compression extensions. Two of the three maps internal to the class which are used to store codecs have existing lowercase casts, but it is absent from the call inside getCodec() used for comparing path names.

I updated the corresponding unit test in TestCodecFactory to include intended use cases, and confirmed the test passes with the change. I also updated the error message in the case of a null from an NPE to a rich error message. I've resolved all checkstyle violations within the changed files.
Bug: "hdfs --daemon start" command may write invalid PID to file
Status: Open | Priority: Minor | Resolution: 
Created: 07/Dec/21 16:47
Affects Version/s: 3.2.2

Starting a daemon with {{*hdfs --daemon start ...*}} (and also {{{}*yarn --daemon start ...*{}}}) might result in writing invalid PID to PIDfile.

Scenario: run {{*hdfs --daemon start namenode*}} (or any other hadoop daemon).

Expected result: PID of running namenode java process gets written to PID file.

Actual result (non-deterministic): PID of exited bash process get written to PID file.

 

Root cause of the issue is a fact that both daemon launching bash functions - {{*hadoop_start_daemon*}} and {{*hadoop_start_daemon_wrapper*}} - are concurrently writing different PIDs to the same file, and only PID written by {{*hadoop_start_daemon_wrapper*}} is correct. Order of those writes is weakly synchronised (with hardcoded 5s timeout). Under specific circumstances (like heavy CPU load) this ordering might not be preserved resulting in invalid PID ending up in PIDfile.

 

Possible solution: It seems that it's unnecessary for {{*hadoop_start_daemon*}} to write to pidfile if it's being called from {{*hadoop_start_daemon_wrapper*}} - it should skip this step in this scenario.
Bug: Reset state after running TestSecureRegistry
Status: Open | Priority: Minor | Resolution: 
Created: 13/Nov/21 22:49

If test class TestRegistryOperationUtils runs after TestSecureRegistry in the same JVM, test TestRegistryOperationUtils#testUsernameExtractionEnvVarOverrride fails due to shared state between tests resulting in a wrong returned username. While currently the build process runs every test class in its own, separate JVM, tests could be sped up by running in the same JVM, and the state should be easy to reset.
Bug: Add metrics doc for ReadLockLongHoldCount and WriteLockLongHoldCount
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 12/Nov/21 22:19

Add metrics doc for ReadLockLongHoldCount and WriteLockLongHoldCount. See [HDFS-15808|https://issues.apache.org/jira/browse/HDFS-15808].
Bug: abfs and s3a disk buffer factories to use UUIDs for file prefixes
Status: Open | Priority: Minor | Resolution: 
Created: 11/Nov/21 12:55
Affects Version/s: 3.3.2

the disk buffers created in s3a and abfs output streams use a simple String.format("datablock-%04d-",  index) pattern for the prefix for File.tmpFile

this means there will be contention for filenames across streams, especially across processes. 

if each stream had a uuid prefix there'd be no contention. That'd change the API though. Alternatively: each disk block factory has the uuid, and the index is simply total number blocks created.
Bug: S3AFileSystem to add object tags on a PutObjectRequest to S3
Status: Open | Priority: Minor | Resolution: 
Created: 28/Oct/21 20:41
Affects Version/s: 3.3.1

AWS S3 lifecycle rules can filter on object tags allowing for data retention policies to be more granular than a bucket or prefix.

 

It would be useful if during write, tags could be applied to the PutObjectRequest.

 

For our use-case, we are simply writing Parquet files through S3A to a target bucket, without a cluster. I believe a Configuration property would be sufficient for our needs.

 

But i'm unsure what the general use case would be for a HDFS cluster and the user wanted unique tags per job.
Bug: Fix the import statements in hadoop-aws module
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Oct/21 08:57
Affects Version/s: 3.3.1

The install target with the activated clover profile fails with an error on the hadoop-aws module.


{code:java}
mvn -Pclover clean install -DskipTests --projects '!hadoop-client-modules/hadoop-client-check-invariants,!hadoop-client-modules/hadoop-client-check-test-invariants' {code}
{code:java}
com.atlassian.clover.api.CloverException: /Users/tdomok/Work/hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/DeleteOperation.java:31:85:unexpected token: ; {code}
Bug: S3A: ITestS3AFileContextStatistics test to lookup global or per-bucket configuration for encryption algorithm
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Oct/21 10:37
Affects Version/s: 3.4.0

ITestS3AFileContextStatistics uses the conf.get() method to get the global configuration for encryption algorithm and keys, but the per-bucket configuration would be ignored in this case.
Bug: NFS unable to bind to chosen address
Status: Open | Priority: Minor | Resolution: 
Created: 18/Oct/21 15:23
Affects Version/s: 3.1.1

[https://lists.apache.org/list.html?user@hadoop.apache.org:2021-10]

 

NFS Gateway appears not have a configuration setting for selecting which IP(s) to which to bind on a multihomed machine. I would like to offer a patch to add this feature in line with other similar settings for HDFS components that can bind to a chosen address. Or even better, I would _love_ to be told I am wrong and that this config setting _does_ exist. :) I will attempt to create a patch, but I have not contributed to Hadoop before, so I will need handholding.
Bug: OpensslCipher initialization error should log a WARN message
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 28/Oct/21 20:34

We spent months troubleshooting a RangerKMS performance problem, only to realize that the openssl library wasn't even loaded properly.

The failure to load openssl lib is currently logged as a debug message during initialization. We really should upgrade it to at least INFO/WARN. 

{code}
static {
    String loadingFailure = null;
    try {
      if (!NativeCodeLoader.buildSupportsOpenssl()) {
        PerformanceAdvisory.LOG.debug("Build does not support openssl");
        loadingFailure = "build does not support openssl.";
      } else {
        initIDs();
      }
    } catch (Throwable t) {
      loadingFailure = t.getMessage();
      LOG.debug("Failed to load OpenSSL Cipher.", t);
    } finally {
      loadingFailureReason = loadingFailure;
    }
  }
{code}
Bug: AccessPoint verifyBucketExistsV2 always returns false
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 04/Oct/21 11:12
Affects Version/s: 3.4.0

Turns out the implementation added for accesspoints to check that they exist was always returning "false". (Is it an improvement from the SDK which always returned "true"? nope).

The fix for this is trivial as we need to check for the presence of a message "Could not access through this access point" or 404 if the AP doesn't exist. Otherwise it does.
Bug: hostnmae is shown in LowerCase in SecurityUtil#replacePattern 
Status: Resolved | Priority: Trivial | Resolution: Invalid
Created: 14/Dec/21 22:19
Affects Version/s: 3.3.1

Leading hostname to lower case causes test failures, when original hostname in uppercase. 

[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L230]

What is the intention of using lowercase there?
Bug: Cut excess dependencies from hadoop-azure, hadoop-aliyun transitive imports; fix LICENSE-binary
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 23/Feb/23 17:11
Affects Version/s: 3.3.5

There's too many dependencies coming through transitively from libraries getting into the hadoop-cloud-storage POM, which, while being resolved in hadoop imports, can cause problems elsewhere. Cut those we know are duplicated in hadoop-common

also, changes in direct dependencies have added new transitive ones which aren't covered in the license file. identify, verify license is valid, remind PR authors of their homework. That change makes this a blocker.

this pr went in with HADOOP-18641 in the commit text
Bug: Upgrade Kerby to 2.0.3 due to CVE-2023-25613
Status: Patch Available | Priority: Blocker | Resolution: 
Created: 07/Mar/23 09:24
Affects Version/s: 3.3.4

An LDAP Injection vulnerability exists in the LdapIdentityBackend of Apache Kerby before 2.0.3.

CVSSv3 Score:- 9.8(Critical)

[https://nvd.nist.gov/vuln/detail/CVE-2023-25613]
Bug: Spark application's dependency conflicts with Hadoop's dependency
Status: Resolved | Priority: Blocker | Resolution: Invalid
Created: 20/Mar/23 03:41
Affects Version/s: 3.3.2

The issue I'm going to describe happens with the distribution: Spark 3.3.2 (git revision 5103e00c4c) built for Hadoop 3.3.2

Based on [this ticket|https://issues.apache.org/jira/browse/HADOOP-11804], as per my understanding, from Hadoop v3, there shouldn't be any conflict between the Hadoop's and Spark app's dependencies. But, I see a runtime failure with my spark app because of this conflict. Pasting the stack trace below:

{{Caused by: java.lang.NoSuchMethodError: com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;}}
{{    at org.apache.cassandra.config.Config.<init>(Config.java:102)}}
{{    at org.apache.cassandra.config.DatabaseDescriptor.clientInitialization(DatabaseDescriptor.java:288)}}
{{    at org.apache.cassandra.io.sstable.CQLSSTableWriter.<clinit>(CQLSSTableWriter.java:109)}}
{{    at com.<redacted>.spark.cassandra.bulkload.GameRecommendationsSSTWriter.init(GameRecommendationsSSTWriter.java:60)}}
{{    at com.<redacted>.spark.cassandra.bulkload.GameRecommendationsSSTWriter.<init>(GameRecommendationsSSTWriter.java:23)}}
{{    at com.<redacted>.spark.cassandra.bulkload.CassandraBulkLoad.execute(CassandraBulkLoad.java:93)}}
{{    at com.<redacted>.spark.cassandra.bulkload.CassandraBulkLoad.main(CassandraBulkLoad.java:60)}}
{{    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
{{    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
{{    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
{{    at java.lang.reflect.Method.invoke(Method.java:498)}}
{{    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:740)}}

My Spark app has a transitive dependency on Guava library. It depends on cassandra-all lib, which does on guava lib. The jar guava-14.0.1 that comes in "spark-3.3.2-bin-hadoop3/jars" directory is a decade old and doesn't have Sets.newConcurrentHashSet() method. I'm able to run the spark app successfully by deleting that old version of guava jar from /jar directory and by including a recent version in my project's pom.xml.
Bug: ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 02/Feb/23 04:18
Affects Version/s: 3.3.4

In Jan 2023, Microsoft Azure AKS replaced its original pod-managed identity with with [Azure Active Directory (Azure AD) workload identities|https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-identities-overview] (preview), which integrate with the Kubernetes native capabilities to federate with any external identity providers. This approach is simpler to use and deploy.

Refer to [https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview|https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview.] and [https://azure.github.io/azure-workload-identity/docs/introduction.html] for more details.

The basic use scenario is to access Azure cloud resources (such as cloud storage) from Kubernetes (such as AKS) workload using Azure managed identity federated with Kubernetes service account. The credential environment variables in pod projected by Azure AD workload identity are like following:

AZURE_AUTHORITY_HOST: (Injected by the webhook, [https://login.microsoftonline.com/])

AZURE_CLIENT_ID: (Injected by the webhook)

AZURE_TENANT_ID: (Injected by the webhook)

AZURE_FEDERATED_TOKEN_FILE: (Injected by the webhook, /var/run/secrets/azure/tokens/azure-identity-token)

The token in the file pointed by AZURE_FEDERATED_TOKEN_FILE is a JWT (JASON Web Token) client assertion token which we can use to request to AZURE_AUTHORITY_HOST (url is  AZURE_AUTHORITY_HOST + tenantId + "/oauth2/v2.0/token")  for a AD token which can be used to directly access the Azure cloud resources.

This approach is very common and similar among cloud providers such as AWS and GCP. Hadoop AWS integration has WebIdentityTokenCredentialProvider to handle the same case.

The existing MsiTokenProvider can only handle the managed identity associated with Azure VM instance. We need to implement a WorkloadIdentityTokenProvider which handle Azure Workload Identity case. For this, we need to add one method (getTokenUsingJWTAssertion) in AzureADAuthenticator which will be used by WorkloadIdentityTokenProvider.
Bug: Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/23 23:23
Affects Version/s: 3.3.6

We are in the midst of enabling HBase and Solr to run on Ozone.

An obstacle is that HBase relies heavily on HDFS APIs and semantics for its Write Ahead Log (WAL) file (similarly, for Solr's transaction log). We propose to push up these HDFS APIs, i.e. recoverLease(), setSafeMode(), isFileClosed() to FileSystem abstraction so that HBase and other applications do not need to take on Ozone dependency at compile time. This work will (hopefully) enable HBase to run on other storage system implementations in the future.

There are other HDFS features that HBase uses, including hedged read and favored nodes. Those are FS-specific optimizations and are not critical to enable HBase on Ozone.
Bug: improve s3a committer stats collected
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 03/Mar/23 18:22
Affects Version/s: 3.3.5

we can improve stats collected in the s3a committer and saved to the JSON.

key ones
# of task manifests read; duration of loads
# size of each manifest

I think we would also benefit if we could set the commit thread pools to be big -but then shared across all jobs (i.e. demand-created thread pool in s3a fs). that would allow for a pool size of say, 500, but still support many jobs actively committing at same time (busy spark driver)
finally: should file commit pool size be > size of pool of manifest readers. I think it could be, but the ratio should be fairly low.
Bug: Add BulkDelete API for paged delete of files and objects
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/23 14:49
Affects Version/s: 3.3.5

iceberg and hbase could benefit from being able to give a list of individual files to delete -files which may be scattered round the bucket for better read peformance.

Add some new optional interface for an object store which allows a caller to submit a list of paths to files to delete, where
the expectation is
 * if a path is a file: delete
 * if a path is a dir, outcome undefined
For s3 that'd let us build these into DeleteRequest objects, and submit, without any probes first.

h2. Cherrypicking
{quote}
when cherrypicking, you must include
 * followup commit #6854
 * https://issues.apache.org/jira/browse/HADOOP-19196
 * test fixes HADOOP-19814 and HADOOP-19188
Bug: define s3a encryption behaviour on copy
Status: Open | Priority: Major | Resolution: 
Created: 21/Feb/23 16:20

When doing a copy, S3A always uses encryption configuration of the filesystem, rather than the source object. This behaviour may not have been intended, as in `RequestFactoryImpl.copyEncryptionParameters()`  it does copy source object encryption properties [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java#L336] , but a missing return statement means it ends up using the FS settings anyway. 

 

Proposed:
 * If the copy is called by rename, always preserve source object encryption properties. 
 * For all other copies, use current FS encryption settings.
Bug: Add "versions" tool to s3a command line entry point
Status: Open | Priority: Major | Resolution: 
Created: 04/Mar/23 12:54
Affects Version/s: 3.3.9

having just implemented some version command support in the cloudstore jar, I can see benefit in actually implementing it in hadoop-aws module

https://github.com/steveloughran/cloudstore/blob/trunk/src/main/site/versioned-objects.md

https://github.com/steveloughran/cloudstore/blob/trunk/src/main/extra/org/apache/hadoop/fs/s3a/extra/) 

this code
* uses v1 sdk by asking the s3a fs for it; this will break with the move to v2 sdk
* doesn't have any tests
* doesn't have any review, maintenance plan
* bypasses audit log/referrer header creation

we could just say "use the aws CLI", but there are some benefits in using the s3a connector code
* support for s3a:// urls
* can use the s3a auth/signing chain (knox, etc)
* plus proxy, region settings etc.
* could integrate with other bits of the stack (e.g spark RDD to get at all versions of objects)
* would be really useful to have a tool to purge all directory delete markers down a path, to speed up listing on versioned buckets.
* gets bundled everywhere

For use by downstream code we would want to have a public/evolving API to access operations, e.g. 

# taking an S3AFileStatus for rename/purge/restore operations
# listing all versions of objects under a path within a given time range and mapping to RemoteIterator.
# HADOOP-16387. S3A openFile() options to allow etag/version to be set

Core code straightforward (it takes exactly two days to write, *excluding tests*), public API and tests more work.

note, we should also move the entry point to being "s3a" with "s3guard" retained for compatibility)
Bug: Move hadoop docker scripts under the main source code
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/23 13:01

Exploratory:
Coming from https://github.com/apache/hadoop/pull/5514
We have docker scripts maintained in a different branch. We can explore them to have as part of dev-support in our main source code.

They can be used for new users to try the code without the headache of building and doing crazy stuff, and can help in dev testing as well
Bug: Leaked calls may cause ObserverNameNode OOM.
Status: Open | Priority: Major | Resolution: 
Created: 08/Feb/23 03:46

Leaked calls may cause ObserverNameNode OOM.

 

During Observer Namenode tailing edits from JournalNode, it will cancel slow request with an interruptException if there are a majority of successful responses. 

There is a bug in Client.java, it will not clean the interrupted call from the calls. The leaked calls may cause ObserverNameNode OOM.
Bug: cyclonedx maven plugin breaks builds on recent maven releases (3.9.0)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/23 11:13
Affects Version/s: 3.3.5

having upgraded maven to get spark to build, the cyclonedx plugin for HADOOP-18590 is failing. see SPARK-42380 for the same.

the 3.3.5 RCs are building with the older maven version of our docker images, but if we release as is we will ship something which will get harder over time to build.

reverting HADOOP-18590 everywhere. sorry.

*note* commits for HADOOP-18642, _HADOOP-18641. Cloud connector dependency and LICENSE fixup._ have gone in using this JIRA iD.