av9ash/GitBugs huggingface dataset:
Bug: ask: abfs connector to support checksum
Status: Resolved | Priority: Major | Resolution: Information Provided
Created: 21/Mar/23 03:08

Hi Hadoop-Azure community,

I cannot find much information on reason why abfs connector file level checksum is not supported, could you share some insights on why it doesn't support and is there plan to support in the future ? 

having this would be helpful for migrating data from on-prem to Azure storage using abfs connector

ref https://hadoop.apache.org/docs/stable/hadoop-azure/abfs.html
Bug: Broken links for arm build of hadoop 3.3.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/23 07:57
Affects Version/s: 3.3.5

The 3.3.5 release include arm builds under the link [https://dlcdn.apache.org/hadoop/common/hadoop-3.3.5/hadoop-arm64-3.3.5.tar.gz] notice the `arm64` suffix before the version number. But the download links on the release page [https://hadoop.apache.org/releases.html]  `hadoop/common/hadoop-3.3.5/hadoop-3.3.5-aarch64.tar.gz` using `aarch64` suffix after the version number.  This is also the suffix used the last time an arm build was published in release 3.3.1 [https://dlcdn.apache.org/hadoop/common/hadoop-3.3.1/]
Bug: CachedSASToken noisy log errors when SAS token has YYYY-MM-DD expiration
Status: Open | Priority: Major | Resolution: 
Created: 23/Mar/23 15:59

Error Description:

When using SAS tokens with expiration dates in the format YYYY-MM-DD, a frequent error appears in the logs related to the date format. The error expects an ISO_DATE_TIME. See [existing implementation|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/CachedSASToken.java#LL112-L119]. The error is noisy in the logs, but does not cause issues. 

Example stacktrace:
{code:java}
23/03/23 15:40:06 ERROR CachedSASToken: Error parsing se query parameter (2023-11-05) from SAS.
java.time.format.DateTimeParseException: Text '2023-11-05' could not be parsed at index 10
	at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949)
	at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851)
	at java.time.OffsetDateTime.parse(OffsetDateTime.java:402)
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.utils.CachedSASToken.getExpiry(CachedSASToken.java:116)
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.utils.CachedSASToken.update(CachedSASToken.java:168)
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.readRemote(AbfsInputStream.java:670)
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.lambda$run$0(ReadBufferWorker.java:66)
	at com.databricks.common.SparkTaskIOMetrics.withTaskIOMetrics(SparkTaskIOMetrics.scala:43)
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:65)
	at java.lang.Thread.run(Thread.java:750) {code}
Desired Resolution:

Expiration code can read YYYY-MM-DD format as well as existing format.
Bug: DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.
Status: Open | Priority: Major | Resolution: 
Created: 22/Feb/23 13:43
Affects Version/s: 3.1.2

YARN NodeManager's deletion service has two types of deletion tasks: the FileDeletionTask for deleting log, usercache, appcache files and the DockerContainerDeletionTask for deleting Docker containers.
 
The FileDeletionTask is removed from the statestore when the task is completed, but the DockerContainerDeletionTask is not.
Therefore, the DockerContainerDeletionTask accumulates continuously in the statestore.
 
This causes the NodeManager's deletion service to run the accumulated DockerContainerDeletionTask in the statestore when the NodeManager restarts.

As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed unnecessarily while processing accumulated tasks, which can cause disk full issues in environments where a large number of containers are allocated and released.

I will attach a patch soon
Bug: Hadoop "current" documentation link broken after release 3.3.5.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/23 20:17

From hadoop.apache.org, access Documentation -> Current, leading to:

https://hadoop.apache.org/docs/current/

This results in a Forbidden response, seemingly since completion of the 3.3.5 release. (To see this, you might need to refresh your browser if it's still serving a cached copy of the 3.3.4 docs from this link.)
Bug: IOUtils.wrapWithMessage can't wrap exceptions without string constructor
Status: Open | Priority: Major | Resolution: 
Created: 14/Mar/23 18:32
Affects Version/s: 3.3.4

When and attempt is made to wrap {{AbfsRestOperationException}}; it fails as there's no string constructor there.


{code}
java.lang.NoSuchMethodException: org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException.<init>(java.lang.String)
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getConstructor(Class.java:1825)
        at org.apache.hadoop.io.IOUtils.wrapWithMessage(IOUtils.java:507)
        at org.apache.hadoop.io.IOUtils.wrapException(IOUtils.java:491)
        at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.close(AbfsOutputStream.java:495)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)

{code}

proposed: handle NoSuchMethodException and downgrade to wrapping with a simple iOE. reluctant to not wrap in close() as we've had bizarre failures in the past there with try-with-resources failures
Bug: DecayRpcSchedulerDetailedMetrics display name numbers incremented by 1
Status: Open | Priority: Major | Resolution: 
Created: 14/Mar/23 10:58

It should keep names consistent with {color:#00875a}DecayRpcSchedulerMetrics{color}, to avoid confusion.


What might be displayed as Priority.0 and Priority.1 for {color:#00875a}DecayRpcSchedulerMetrics{color}, it's Priority.1 and Priority.2 for {color:#00875a}DecayRpcSchedulerDetailedMetrics{color}.
Bug: ABFS to add Auditing support and attach http referrer to requests
Status: Open | Priority: Major | Resolution: 
Created: 09/Mar/23 14:19
Affects Version/s: 3.3.5

 the s3a auditing feature of HADOOP-17511 is wonderful in production, as we can
* scan for all IO done by a single user, job, operation
* get a comprehensive view of what was done to the store, in the order it saw it, which doesn't always match the order the client logged it, especially logged outcomes as that only happens when the threads get scheduled.

the abfs TracingContext effectively does most of this. All that is needed is to integrate with the hadoop-common audit interfaces/apis
* TracingContext to pick up global/thread-local audit contexts in construction and so extract information (tool class; principal; spark job id. etc)

The manifest committer already sets task id on the active thread whenever entered; this persists through all the IO done by the spark worker.
Bug: Hadoop DistCp supports specifying favoredNodes for data copying
Status: Open | Priority: Major | Resolution: 
Created: 14/Feb/23 03:12
Affects Version/s: 3.3.4

When importing large scale data to HBase, we always generate the hfiles with other Hadoop cluster, use the Distcp tool to copy the data to the HBase cluster, and bulkload data to HBase table. However, the data locality is rather low which may result in high query latency. After taking a compaction it will recover. Therefore, we can increase the data locality by specifying the favoredNodes in Distcp.

Could I submit a pull request to optimize it?
Bug: S3A proxy to use proxy protocol to set port
Status: Open | Priority: Major | Resolution: 
Created: 24/Feb/23 07:27

In the current code, we use the Cloud store's protocol to set the proxy's port({*}If no port is set{*}).
{code:java}
if (conf.getBoolean(SECURE_CONNECTIONS, DEFAULT_SECURE_CONNECTIONS)) {
          LOG.warn("Proxy host set without port. Using HTTPS default 443");
          awsConf.setProxyPort(443);
        } else {
          LOG.warn("Proxy host set without port. Using HTTP default 80");
          awsConf.setProxyPort(80);
        } {code}
We should use the proxy protocol instead.
Bug: [Azure FS] Add new property to configure credential provider path for azure storage.
Status: Open | Priority: Major | Resolution: 
Created: 10/Feb/23 05:06
Affects Version/s: 3.1.1

Azure storage supports account-specific credentials, and thus the credential provider should permit the configuration of separate JCEKS files for each account, such as the property "{*}fs.azure.account.credential.provider.path.<account>.blob.core.windows.net{*}".

 

This Jira utilize new API supported in HADOOP-18618 for external provider path.
Bug: Make IOStatisticsStore and binding APIs public for use beyond our code
Status: Open | Priority: Major | Resolution: 
Created: 03/Feb/23 19:19
Affects Version/s: 3.3.5

it's really useful to be able to collect iostats in things other than the FS classes -we do it in the S3A and manifest committers.

But external code -such as the spark committers can't use the methods in {{org.apache.hadoop.fs.statistics.impl))

Proposed

Make some classes/interfaces public

* IOStatisticsBinding
* IOStatisticsStore
* IOStatisticsStoreBuilder

Ideally we should actually move the IOStatisticsStore interface into org.apache.hadoop.fs.statistics and the builder to match -but we can't do that without causing trauma elsewhere (google gcs).

Strategy there: Add a new interface IOStatisticsCollector in .impl which is then implemented by IOStatisticsStore, and a new builder API which forwards to IOStatisticsStoreBuilder.

Side issue: we don't make any use of the "clever, elegant functional" bit of DynamicIOStatisticsBuilder/DynamicIOStatistics, where every counter is mapped to a function which is then invoked to get at the atomic longs. It's used in IOStatisticsStoreImpl, but only with AtomicLong and MeanStatistic instances. If we just move to simple maps we will save on lambda-expressions and on lookup overhead. The original intent was something like coda hale metrics where we could add dynamic lookup to other bits of instrumentation; in practise we measure durations and build counts/min/max.
Bug: Expose `listStatus(Path path, String startFrom)` on `AzureBlobFileSystem`
Status: Open | Priority: Major | Resolution: 
Created: 18/Jan/23 14:16
Affects Version/s: 3.3.2

When working with Azure blob storage listing operations can often be quite slow even on storage accounts with the hierarchical namespace. 

This can be mitigated by listing only a specific subset of directories using a function like [https://hadoop.apache.org/docs/r3.3.4/api/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.html#listStatus-org.apache.hadoop.fs.Path-java.lang.String-org.apache.hadoop.fs.azurebfs.utils.TracingContext-]

Which accepts a `startFrom` argument and lists all files in order starting from there.

I'm wondering if we could add a method to the `AzureBlobFileSystem`

Something like:

```
public FileStatus[] listStatus(final Path f, final String startFrom) throws IOException
```

This exposes the functionality that already exists on the underlying `AzureBlobFileSystemStore`. My understanding from reading a bit of the code is that users should mainly be dealing with `AzureBlobFileSystem`s and `AzureBlobFileSystem` seem easier to use to me hence the benefit of exposing it on the `AzureBlobFileSystem`.

 

I'm very un-familiar with java but I'm told that keeping strictly to interfaces is strongly preferred. However I can see some examples already on `AzureBlobFileSystem` that do not belong to any interface (e.g. `breakLease`) so I'm hoping its acceptable to add a method like I described only for the one `FileSystem` implementation.

 

The specific motivation for this is to unblock [https://github.com/delta-io/delta/issues/1568]

I would be willing to contribute this if maintainers think the plan is reasonable.
Bug: Hadoop 2.x should support s3a committers
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 19/Jan/23 12:48
Affects Version/s: 2.10.2

I think the feature about "Add S3A committers for zero-rename commits to S3 endpoints" (https://issues.apache.org/jira/browse/HADOOP-13786)  should be 蜜珀 in the merged in hadoop 2.10.2.
Bug: Reduce byte array allocation in DecompressorStreams
Status: Open | Priority: Major | Resolution: 
Created: 13/Jan/23 18:07

Compression codecs all implement CompressionCodec interface, which has a createInputStream method which takes an InputStream. These end up creating a class which extends DecompressorStream, which allocates 2 byte arrays of configurable size.

Currently its impossible to re-use these DecompressorStreams. Heavy users of decompression, can end up seeing large amounts of allocations coming from the creation of DecompressorStreams. Realtime systems like HBase have put a lot of work into reducing allocations, either through pooling and/or off-heaping. 

It would be very useful to be able to either re-use DecompressorStreams or add constructors which accept a ByteArrayManager or other source of byte array allocator. Either way could reduce allocations considerably in this case.
Bug: S3A Add new store vendor config option
Status: Open | Priority: Minor | Resolution: 
Created: 27/Mar/23 11:58
Affects Version/s: 3.3.5

Add in a new fs.s3a.store.vendor config, where users can specify the storage vendor they are using (eg: aws, netapp, minio).

This will allow us to configure S3A correctly per vendor. For example, if the vendor is not AWS, you probably want to use ListObjectsV1.
Bug: Amazon S3 disabling ACLs on all new buckets
Status: Open | Priority: Minor | Resolution: 
Created: 30/Mar/23 12:17

In April 2023, Amazon S3 will be disabling ACLs by default on *all* new buckets. [https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security-changes-are-coming-in-april-of-2023/.|https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security-changes-are-coming-in-april-of-2023/] Note, buckets created using the AWS Console are already created with ACLs disabled by default.

In S3A, we have tests that rely on ACLs being enabled. We should either update test documentation to tell developers to enable ACLs on buckets, or we should make the ACL tests opt-in if they are used infrequently by S3A users.
Bug: S3A AssumedRole credentials provider should use Instance Role credentials in chain for assuming role
Status: Open | Priority: Minor | Resolution: 
Created: 30/Jan/23 09:11

The AssumedRoleCredentialsProvider today creates its own credential provider set to use when getting session credentials from STS. This includes only environment variables and simple credentials provider. [https://github.com/apache/hadoop/blob/952d707240cb7dd088820d8b39e705b77fa3b6c4/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java#L114-L116]

 

It should at least include EC2 Instance Role credentials, possibly others, by default.
Bug: ABFS: Support for Pagination in Recursive Directory Delete 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Mar/23 05:31
Affects Version/s: 3.4.0

Today, when a recursive delete is issued for a large directory in ADLS Gen2 (HNS) account, the directory deletion happens in O(1) but in backend ACL Checks are done recursively for each object inside that directory which in case of large directory could lead to request time out. Pagination is introduced in the Azure Storage Backend for these ACL checks.

More information on how pagination works can be found on public documentation of [Azure Delete Path API|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete?view=rest-storageservices-datalakestoragegen2-2019-12-12].

This PR contains changes to support this from client side. To trigger pagination, client needs to add a new query parameter "paginated" and set it to true along with recursive set to true. In return if the directory is large, server might return a continuation token back to the caller. If caller gets back a continuation token, it has to call the delete API again with continuation token along with recursive and pagination set to true. This is similar to directory delete of FNS account.

Pagination is available only in versions "2023-08-03" onwards.
PR also contains functional tests to verify driver works well with different combinations of recursive and pagination features for HNS.
Full E2E testing of pagination requires large dataset to be created and hence not added as part of driver test suite. But extensive E2E testing has been performed.
Bug: Update solr from 8.8.2 to 8.11.2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Mar/23 10:42
Affects Version/s: 3.3.4

Solr is used in hadoop-yarn-applications-catalog-webapp.
{code:sh}
$ grep solr.version . -r
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml:            <version>${solr.version}</version>
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml:            <version>${solr.version}</version>
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml:            <version>${solr.version}</version>
./hadoop-project/pom.xml:    <solr.version>8.8.2</solr.version>
$
{code}
 

Solr-8.8.2's log4j2 version is 2.13.2.

Solr-8.11.2's log4j2 version is 2.17.1 ( https://issues.apache.org/jira/browse/SOLR-15871 )

Should we update solr to 8.11.2 correspondingly? 
Then maven will not try to download log4j2-2.13.2, which is forbidden by some maven repository.
Bug: LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Feb/23 11:40
Affects Version/s: 3.3.4

The  s3a and abfs clients use LocalDirAllocator for allocating files in local (temporary) storage for buffering blocks to write, and, for the s3a staging committer, files being staged. 
When initialized (or when the configuration key value is updated) LocalDirAllocator enumerates all directories in the list and calls {{mkdirs()}} to create them.

when you ask actually for a file, it will look for the parent dir, and will again call {{mkdirs()}}. 

But before it does that, it looks to see if the dir has any space...if not it is excluded from the list of directories with room for data.

And guess what: directories which don't exist report as having no space. So they get excluded -the recreation code doesn't get a chance to run.
Bug: Add reason in in x-ms-client-request-id on a retry API call.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jan/23 11:28
Affects Version/s: 3.3.6

In the header, x-ms-client-request-id contains informaiton on what retry this particular API call is: for ex: :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1.

We want to add the reason for the retry in the header_value:Now the same header would include retry reason in case its not the 0th iteration of the API operation. It would be like
:eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_RT. This corresponds that its retry number 1. The 0th iteration was failed due to read timeout.
Bug: Server connection should log host name before returning VersionMismatch error
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Feb/23 05:03
Affects Version/s: 3.3.4

In env with dynamically changing IP addresses, debugging issue with the logs with only IP address becomes a bit difficult at times.
{code:java}
2023-02-08 23:26:50,112 WARN  [Socket Reader #1 for port 8485] ipc.Server - Incorrect RPC Header length from {IPV4}:36556 expected length: java.nio.HeapByteBuffer[pos=0 lim=4 cap=4] got length: java.nio.HeapByteBuffer[pos=0 lim=4 cap=4] {code}
It would be better to log full hostname for the given IP address rather than only IP address.
Bug: Provide keytab file key name with ServiceStateException
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Feb/23 00:10
Affects Version/s: 3.4.0

 
{code:java}
util.ExitUtil - Exiting with status 1: org.apache.hadoop.service.ServiceStateException: java.io.IOException: Running in secure mode, but config doesn't have a keytab
1: org.apache.hadoop.service.ServiceStateException: java.io.IOException: Running in secure mode, but config doesn't have a keytab
  at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:264)
..
..
 {code}
 

 

When multiple downstreamers use different configs to present the same keytab file, if one of the config key gets missing or overridden as part of config generators, it becomes bit confusing for operators to realize which config is missing for a particular service, especially when keytab file value is already present with different config.

It would be nice to report config key with the stacktrace error message.
Bug: Path.suffix raises NullPointerException
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Mar/23 04:00
Affects Version/s: 3.3.6

Calling the Path.suffix method on root raises a NullPointerException. Tested with hadoop-client-api 3.3.2

Scenario:
{code:java}
import org.apache.hadoop.fs.*

Path root = new Path("/")
root.getParent == null  // true
root.suffix("bar")  // NPE is raised
{code}

Stack:
{code:none}
23/03/03 15:13:18 ERROR Uncaught throwable from user code: java.lang.NullPointerException
    at org.apache.hadoop.fs.Path.<init>(Path.java:104)
    at org.apache.hadoop.fs.Path.<init>(Path.java:93)
    at org.apache.hadoop.fs.Path.suffix(Path.java:361)
{code}
Bug: Insufficient heap during full test runs in Docker container.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Mar/23 16:22
Affects Version/s: 3.3.6

During verification of releases on the 3.3 line, I often run out of heap during full test runs inside the Docker container. Let's increase the default in {{MAVEN_OPTS}} to match trunk.

Additionally, on trunk, the settings are different in Dockerfile vs. Dockerfile_aarch64. We can align those.
Bug: Support custom property for credential provider path
Status: Open | Priority: Minor | Resolution: 
Created: 04/Feb/23 08:10
Affects Version/s: 3.1.3

Hadoop allows the configuration of a credential provider path through the property "{*}hadoop.security.credential.provider.path{*}", and the {{Configuration#getPassword()}} method retrieves the credentials from this provider.

However, using common credential provider properties for components like Hive, HDFS, and MapReduce can cause issues when they want to configure separate JCEKS files for credentials. For example, the value in the core-site.xml property file can be overridden by the hive-site.xml property file. To resolve this, all components should share a common credential provider path and add all their credentials.

Azure storage supports account-specific credentials, and thus the credential provider should permit the configuration of separate JCEKS files for each account, such as the property "{*}fs.azure.account.credential.provider.path.<account>.blob.core.windows.net{*}".

To accommodate this, the {{Configuration#getPassword()}} method should accept a custom property for the credential provider path and retrieve its value. The current default property can be overridden to achieve this.
{code:java}
public char[] getPassword(String name) throws IOException {
    ......
    ......
}


public char[] getPassword(String name, String providerKey) throws IOException {                  
    ......
    ......
 }{code}
 

One Example is, Ambari [CustomServiceOrchestrator|https://github.com/apache/ambari/blob/trunk/ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py#L312] service override the core-site.xml value for other component. This fix is very much needed for Ambari.