av9ash/GitBugs huggingface dataset:
Bug: Enhance FileSystem.Cache to honor security token and expiration
Status: Open | Priority: Major | Resolution: 
Created: 28/Jul/24 04:02
Affects Version/s: 3.3.4

We have an online service which uses Hadoop FileSystem to load files from a remote cloud storage.

The current cache in FileSystem is a [HashMap|https://github.com/apache/hadoop/blob/4525c7e35ea22d7a6350b8af10eb8d2ff68376e7/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L3635C1-L3635C62], and its key honors scheme, authority (like [user@host:port|https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Syntax]), ugi and a unique long for its [hash code|https://github.com/apache/hadoop/blob/4525c7e35ea22d7a6350b8af10eb8d2ff68376e7/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L3891C1-L3894C8]. And among those 4 fields, only "scheme" and "authority" could be controlled externally.

That results in a wrong case like: A FileSystem entry in the cache was created with schemeA + authorityA, and with read + write access, and an expiration. Later, an API to get FileSystem comes still using schemeA + authorityA, but with less access (maybe read only), or it already expires, that FileSystem entry in the cache is hit by mistake, while no new FilleSystem is created. It does not lead to a security issue, but subsequent calls (maybe to read the file) will be rejected with 403 by the remote storage.

Our proposal is like
 * Short term
 ** Add a new field in FileSystem.Cache.Key to affect hashCode() and equals(). This field could be specified when contructing a Key.
 ** Add a simple expiration mechanism in FileSystem.Cache
 *** Each cache entry is created with a expiration
 *** When getting a FileSystem, if the cache entry is hit but already expires, close it and remove it from the cache. And return a new created FileSystem.
 * Long term
 ** Replace the internal HashMap by a more modern and full functional cache framework, like [https://github.com/ben-manes/caffeine]
Bug: Google GCS changes fail due to VectorIO changes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/24 19:05
Affects Version/s: 3.4.1

the changes of HADOOP-19098 broken google gcs


{code}
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project gcs-connector: Compilation failure
[ERROR] /Users/mthakur/Sandbox/open_source/hadoop-connectors/gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/VectoredIOImpl.java:[317,60] incompatible types: java.util.List<capture#1 of ? extends org.apache.hadoop.fs.FileRange> cannot be converted to org.apache.hadoop.fs.FileRange[]
[ERROR]

{code}

failing line is

{code}
FileRange[] sortedRanges = VectoredReadUtils.sortRanges(input);

{code}


need to restore the original  {{sortRanges}}, renaming the changed signature one first. plus a test, obviously
Bug: Enhance FileSystem.Cache to honor a user defined field
Status: Open | Priority: Major | Resolution: 
Created: 28/Jul/24 05:22
Affects Version/s: 3.3.4

Add a new field in FileSystem.Cache.Key to affect hashCode() and equals(). This field could be specified when constructing a Key
 * For new users, they are able to control the hash code generation with respect to this field (could be a security string/token/object...), which may contain access (read/write/list...) and/or expiration time ...
 * For old users, this field defaults to null and is not honored in hashCode() and equals()
Bug: Update the yasm rpm download address
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/24 04:48

Some ci run failed, for example:

[https://ci-hadoop.apache.org/blue/rest/organizations/jenkins/pipelines/hadoop-multibranch/branches/PR-6972/runs/1/nodes/56/steps/61/log/?start=0]

I found the yasm rpm url `[https://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/y/yasm-1.2.0-4.el7.x86_64.rpm]` is outdated
Bug: IPC client uses CompletableFuture to support asynchronous operations.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/24 15:27

h3. Description

In the implementation of asynchronous Ipc.client, the main methods used include HADOOP-13226, HDFS-10224, etc.

However, the existing implementation does not support `CompletableFuture`; instead, it relies on setting up callbacks, which can lead to the "callback hell" problem. Using `CompletableFuture` can better organize asynchronous callbacks. Therefore, on the basis of the existing implementation, by using `CompletableFuture`, once the `client.call` is completed, the asynchronous thread handles the response of this call without blocking the main thread.

 

*Test*

new UT  TestAsyncIPC#testAsyncCallWithCompletableFuture()
Bug: Avoid DNS lookup while creating IPC Connection object
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/24 20:18

Been running HADOOP-18628 in production for quite sometime, everything works fine as long as DNS servers in HA are available. Upgrading single NS server at a time is also a common case, not problematic. Every DNS lookup takes 1ms in general.

However, recently we encountered a case where 2 out of 4 NS servers went down (temporarily but it's a rare case). With small duration DNS cache and 2s of NS fallback timeout configured in resolv.conf, now any client performing DNS lookup can encounter 4s+ delay. This caused namenode outage as listener thread is single threaded and it was not able to keep up with large num of unique clients (in direct proportion with num of DNS resolutions every few seconds) initiating connection on listener port.

While having 2 out of 4 DNS servers offline is rare case and NS fallback settings could also be improved, it is important to note that we don't need to perform DNS resolution for every new connection if the intention is to improve the insights into VersionMistmatch errors thrown by the server.

The proposal is the delay the DNS resolution until the server throws the error for incompatible header or version mismatch. This would also help with ~1ms extra time spent even for healthy DNS lookup.
Bug: ipc.Server accelerate token negotiation only for the default mechanism
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/24 06:20

{code}
//Server.java
      // accelerate token negotiation by sending initial challenge
      // in the negotiation response
      if (enabledAuthMethods.contains(AuthMethod.TOKEN)) {
        ...
      }
{code}
In Server.Connection.buildSaslNegotiateResponse() above, it accelerates token negotiation by sending initial challenge in the negotiation response.  However, it is a non-standard SASL
negotiation.  We should do it only for the default SASL mechanism.
Bug: Introduce getTrashPolicy to FileSystem API
Status: Open | Priority: Major | Resolution: 
Created: 02/Jul/24 10:32

Hadoop FileSystem supports multiple FileSystem implementations awareness (e.g. client is aware of both hdfs:// and ofs:// protocols).

However, it seems that currently Hadoop TrashPolicy remains the same regardless of the URI scheme. The TrashPolicy is governed by "fs.trash.classname" configuration and stays the same regardless of the FileSystem implementation. For example, HDFS defaults to TrashPolicyDefault and Ozone defaults to TrashPolicyOzone, but only one will be picked since the the configuration will be overwritten by the other.

Therefore, I propose to tie the TrashPolicy implementation to each FileSystem implementation by introducing a new FileSystem#getTrashPolicy interface. TrashPolicy#getInstance can call FileSystem#getTrashPolicy to get the appropriate TrashPolicy.
Bug: ABFS: Disabling Apache Http Client as Default Http Client for ABFS Driver
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Sep/24 17:46
Affects Version/s: 3.4.0

As part of work done under HADOOP-19120 [ABFS]: ApacheHttpClient adaptation as network library - ASF JIRA
Apache Http Client was introduced as an alternative Network Library that can be used with ABFS Driver. Earlier JDK Http Client was the only supported network library.

Apache Http Client was found to be more helpful in terms of controls and knobs it provides to better manage the Network aspects of driver. Hence it was made the default Network Client to be used with ABFS Driver.

Recently while running scale workloads, we observed a regression where some unexpected wait time was observed while establishing connections. A possible fix has been identified and we are working on getting it fixed.
One scenario identified during internal tests revealed that, with the current Apache client implementation, a connection cannot become stale. As a result, it is safe to remove the check for stale connections when closing them. This change will optimize the connection handling process by eliminating unnecessary delays and reducing the risk of potential failures caused by redundant checks.
There was also a possible NPE scenario which was identified on the new network client code recently.

Until we are done with the code fixes and have revalidated the whole Apache client flow, we would like to make JDK Client as default client again. The new support for Apache Http Client will still be there, but it will be disabled behind a config.
Bug: S3A: new test failures on branch-3.4.1
Status: Open | Priority: Minor | Resolution: 
Created: 13/Sep/24 16:58
Affects Version/s: 3.4.1

test failures on a -Dscale run; s3 usw-1, caller has IAM role.
{code}
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AClosedFS
[ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.497 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AClosedFS
[ERROR] testClosedInstrumentation(org.apache.hadoop.fs.s3a.ITestS3AClosedFS)  Time elapsed: 0.143 s  <<< FAILURE!
org.junit.ComparisonFailure: [S3AInstrumentation.hasMetricSystem()] expected:<[fals]e> but was:<[tru]e>
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at org.apache.hadoop.fs.s3a.ITestS3AClosedFS.testClosedInstrumentation(ITestS3AClosedFS.java:111)
{code}
Bug: S3A: ITestAssumeRole.testAssumeRoleBadInnerAuth failure
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Aug/24 19:13
Affects Version/s: 3.4.0

Not sure when this changed, but I've only just noticed today while setting up a new test config.

The test {{testAssumeRoleBadInnerAuth}} is failing because the error string coming back from STS is slightly different.

{code}
[ERROR] testAssumeRoleBadInnerAuth(org.apache.hadoop.fs.s3a.auth.ITestAssumeRole)  Time elapsed: 4.182 s  <<< FAILURE!
java.lang.AssertionError: 
 Expected to find 'not a valid key=value pair (missing equal-sign) in Authorization header' but got unexpected exception: org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on /: software.amazon.awssdk.services.sts.model.StsException: Invalid key=value pair (missing equal-sign) 
{code}

Rather than change the string to look for, lets just remove the string so it it less brittle in future
Bug: Files and directories mixed up in TreeScanResults#dump
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 16/Sep/24 13:48

{code}
AssertionError: listStatus(): directory count in 6 directories and 4 files
Files:
  "s3a://obs-bucket/test/testComplexDirActions/dir-0-0000
  "s3a://obs-bucket/test/testComplexDirActions/dir-0-0000
  "s3a://obs-bucket/test/testComplexDirActions/dir-0-0001
  "s3a://obs-bucket/test/testComplexDirActions/dir-0-0001
  "s3a://obs-bucket/test/testComplexDirActions/dir-0-0002
  "s3a://obs-bucket/test/testComplexDirActions/dir-0-0002
Directories:
  "s3a://obs-bucket/test/testComplexDirActions/file--0-0000.txt
  "s3a://obs-bucket/test/testComplexDirActions/file--0-0001.txt
  "s3a://obs-bucket/test/testComplexDirActions/file--0-0002.txt
  "s3a://obs-bucket/test/testComplexDirActions/file--0-0003.txt expected:<3> but was:<6>
{code}

"Files" and "Directories" are mixed up in the output.
Bug: Upgrade jackson-databind to 2.9.10.4
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 26/Apr/20 07:23
Affects Version/s: 3.1.4

trunk (3.3.0) now has HADOOP-16905. But branch 3.2/3.1/.. still uses jackson-databind 2.9.

I'm opening this jira since I'm unsure whether we are backporting HADOOP-16905 to lower version branches due to compatibility concern or whatever.

GH PR: https://github.com/apache/hadoop/pull/1981

CC [~weichiu]
Bug: ZStandardCodec compression mail fail(generic error) when encounter specific file
Status: Resolved | Priority: Blocker | Resolution: Duplicate
Created: 30/Jun/20 03:41
Affects Version/s: 2.6.5

*Problem:* 

In our production environment,  we put file in hdfs with zstd compressor, recently, we find that a specific file may leads to zstandard compressor failures. 

And we can reproduce the issue with specific file(attached file: badcase.data)

!image-2020-06-30-11-51-18-026.png|width=1031,height=230!

 

*Analysis*: 

ZStandarCompressor use buffersize( From zstd recommended compress out buffer size)  for both inBufferSize and outBufferSize 

!image-2020-06-30-11-35-46-859.png|width=1027,height=387!

but zstd indeed provides two separately recommending inputBufferSize and outputBufferSize  

!image-2020-06-30-11-39-17-861.png!

 

*Workaround*

One workaround,  using recommended in/out buffer size provided by zstd lib can avoid the problem, but we don't know why. 

zstd recommended input buffer size:  1301072 (128 * 1024)

zstd recommended ouput buffer size: 131591 

!image-2020-06-30-11-42-44-585.png|width=1023,height=196!
Bug: Ignore AuthenticationFilterInitializer for KMSWebServer
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 12/Apr/20 14:25
Affects Version/s: 3.3.0

KMS does not work if hadoop.http.filter.initializers is set to AuthenticationFilterInitializer since KMS uses its own authentication filter. This is problematic when KMS is on the same node with other Hadoop services and shares core-site.xml with them. The filter initializers configuration should be tweaked as done for httpfs in HDFS-14845.
Bug: HADOOP-16582 changed mkdirs() behavior
Status: Open | Priority: Critical | Resolution: 
Created: 08/Apr/20 18:21
Affects Version/s: 2.10.0

HADOOP-16582 changed behavior of {{mkdirs()}}
Some Hive tests depend on the old behavior and they fail miserably.

{quote}
earlier:

all plain mkdirs(somePath) were fast-tracked to FileSystem.mkdirs which have rerouted them to mkdirs(somePath, somePerm) method with some defaults (which were static)
an implementation of FileSystem have only needed implement "mkdirs(somePath, somePerm)" - because the other was not neccessarily called if it was always in a FilterFileSystem or something like that
now:

especially FilterFileSystem forwards the call of mkdirs(p) to the actual fs implementation...which may skip overriden mkdirs(somPath,somePerm) methods
...and could cause issues for existing FileSystem implementations
{quote}

File this jira to address this problem.

[~kgyrtkirk] [~stevel@apache.org] [~kihwal]
Bug: start-build-env.sh fails in branch-3.1
Status: Resolved | Priority: Critical | Resolution: Won't Fix
Created: 28/Jun/20 08:17

./start-build-env.sh fails to install ember-cli
{noformat}
npm ERR! Linux 5.4.0-37-generic
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "ember-cli"
npm ERR! node v4.2.6
npm ERR! npm  v3.5.2
npm ERR! code EMISSINGARG

npm ERR! typeerror Error: Missing required argument #1
npm ERR! typeerror     at andLogAndFinish (/usr/share/npm/lib/fetch-package-metadata.js:31:3)
npm ERR! typeerror     at fetchPackageMetadata (/usr/share/npm/lib/fetch-package-metadata.js:51:22)
npm ERR! typeerror     at resolveWithNewModule (/usr/share/npm/lib/install/deps.js:456:12)
npm ERR! typeerror     at /usr/share/npm/lib/install/deps.js:457:7
npm ERR! typeerror     at /usr/share/npm/node_modules/iferr/index.js:13:50
npm ERR! typeerror     at /usr/share/npm/lib/fetch-package-metadata.js:37:12
npm ERR! typeerror     at addRequestedAndFinish (/usr/share/npm/lib/fetch-package-metadata.js:82:5)
npm ERR! typeerror     at returnAndAddMetadata (/usr/share/npm/lib/fetch-package-metadata.js:117:7)
npm ERR! typeerror     at pickVersionFromRegistryDocument (/usr/share/npm/lib/fetch-package-metadata.js:134:20)
npm ERR! typeerror     at /usr/share/npm/node_modules/iferr/index.js:13:50
npm ERR! typeerror This is an error with npm itself. Please report this error at:
npm ERR! typeerror     <http://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /root/npm-debug.log
{noformat}
Bug: WASB: Update azure-storage-java SDK
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 24/Jun/20 13:14
Affects Version/s: 2.7.0

WASB depends on the Azure Storage Java SDK.  There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty.  This causes the Filesystem listStatus and similar APIs to return empty results.  This has been seen in Spark work loads when jobs use more than one executor core. 

See [https://github.com/Azure/azure-storage-java/pull/546] for details on the bug in the Azure Storage SDK.

This issue can cause data loss.
Bug: NPE when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 06/Apr/20 06:52
Affects Version/s: 3.2.1

During initialization, ZKFCRpcServer refreshes the service authorization ACL for the service handled by this server if config hadoop.security.authorization is enabled, by calling refreshServiceAcl with the input PolicyProvider and Configuration.
{code:java}
ZKFCRpcServer(Configuration conf,
 InetSocketAddress bindAddr,
 ZKFailoverController zkfc,
 PolicyProvider policy) throws IOException {
 this.server = ...
 
 // set service-level authorization security policy
 if (conf.getBoolean(
 CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
 server.refreshServiceAcl(conf, policy);
 }
}{code}

refreshServiceAcl calls ServiceAuthorizationManager#refreshWithLoadedConfiguration which directly gets services from the provider with provider.getServices(). When the provider is NULL, the code throws NPE without an informative message. In addition, the default value of config `hadoop.security.authorization.policyprovider` (which controls PolicyProvider here) is NULL and the only usage of ZKFCRpcServer initializer provides only an abstract method getPolicyProvider which does not enforce that PolicyProvider should not be NULL.

The suggestion here is to either add a guard check or exception handling with an informative logging message on ZKFCRpcServer to handle input PolicyProvider being NULL.

 

I am very happy to provide a patch for it if the issue is confirmed :)
Bug: Support disabling buffered reads in ABFS positional reads
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/20 19:00
Affects Version/s: 3.3.1

Right now it will do a seek to the position , read and then seek back to the old position.  (As per the impl in the super class)
In HBase kind of workloads we rely mostly on short preads. (like 64 KB size by default).  So would be ideal to support a pure pos read API which will not even keep the data in a buffer but will only read the required data as what is asked for by the caller. (Not reading ahead more data as per the read size config)

Allow an optional boolean config to be specified while opening file for read using which buffered pread can be disabled. 
FutureDataInputStreamBuilder openFile(Path path)
Bug: Replace Guava Function with Java8+ Function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/20 04:43
Affects Version/s: 3.2.2

{code:java}
Targets
    Occurrences of 'com.google.common.base.Function'
Found Occurrences  (7 usages found)
    hadoop-hdfs-project/hadoop-hdfs/dev-support/jdiff  (1 usage found)
        Apache_Hadoop_HDFS_2.6.0.xml  (1 usage found)
            13603 <field name="GET_START_TXID" type="com.google.common.base.Function"
    org.apache.hadoop.hdfs.server.blockmanagement  (1 usage found)
        HostSet.java  (1 usage found)
            20 import com.google.common.base.Function;
    org.apache.hadoop.hdfs.server.datanode.checker  (1 usage found)
        AbstractFuture.java  (1 usage found)
            58 * (ListenableFuture, com.google.common.base.Function) Futures.transform}
    org.apache.hadoop.hdfs.server.namenode.ha  (1 usage found)
        HATestUtil.java  (1 usage found)
            40 import com.google.common.base.Function;
    org.apache.hadoop.hdfs.server.protocol  (1 usage found)
        RemoteEditLog.java  (1 usage found)
            20 import com.google.common.base.Function;
    org.apache.hadoop.mapreduce.lib.input  (1 usage found)
        TestFileInputFormat.java  (1 usage found)
            58 import com.google.common.base.Function;
    org.apache.hadoop.yarn.api.protocolrecords.impl.pb  (1 usage found)
        GetApplicationsRequestPBImpl.java  (1 usage found)
            38 import com.google.common.base.Function;

{code}
Bug: Tolerate leading and trailing spaces in fs.defaultFS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/20 21:56
Affects Version/s: 3.4.0

*Problem:*

Currently, `getDefaultUri` is using `conf.get` to get the value of `fs.defaultFS`, which means that the trailing whitespace after a valid URI won’t be removed and could stop namenode and datanode from starting up.

 

*How to reproduce (Hadoop-2.8.5):*

Set the configuration
{code:java}
<property>
     <name>fs.defaultFS</name>
     <value>hdfs://localhost:9000 </value>
</property>{code}
In core-site.xml (there is a whitespace after 9000) and start HDFS.

Namenode and datanode won’t start and the log message is:
{code:java}
2020-04-23 11:09:48,198 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Illegal character in authority at index 7: hdfs://localhost:9000 
    at java.net.URI.create(URI.java:852)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.setClientNamenodeAddress(NameNode.java:440)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:897)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:885)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1694)
Caused by: java.net.URISyntaxException: Illegal character in authority at index 7: hdfs://localhost:9000 
    at java.net.URI$Parser.fail(URI.java:2848)
    at java.net.URI$Parser.parseAuthority(URI.java:3186)
    at java.net.URI$Parser.parseHierarchical(URI.java:3097)
    at java.net.URI$Parser.parse(URI.java:3053)
    at java.net.URI.<init>(URI.java:588)
    at java.net.URI.create(URI.java:850)
    ... 5 more
{code}
 

*Solution:*

Use `getTrimmed` instead of `get` for `fs.defaultFS`:
{code:java}
public static URI getDefaultUri(Configuration conf) {
  URI uri =
    URI.create(fixName(conf.getTrimmed(FS_DEFAULT_NAME_KEY, DEFAULT_FS)));
  if (uri.getScheme() == null) {
    throw new IllegalArgumentException("No scheme in default FS: " + uri);
  }
  return uri;
}
{code}
I have submitted a patch for trunk about this.