av9ash/GitBugs huggingface dataset:
Bug: Remove org.apache.hadoop.test#Whitebox Class
Status: In Progress | Priority: Minor | Resolution: 
Created: 07/Jun/22 03:17

org.apache.hadoop.test#Whitebox is marked as deprecated, I personally feel that this is unnecessary, which leads to a large number of junit test code appearing deprecated, and a large number of Warnings appear in the compilation, I checked the code and think the deprecated mark is unreasonable.
Bug: Do not perform a LIST call when creating a file
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 07/Jun/22 16:35
Affects Version/s: 3.3.3

Hello,

We've noticed that when creating a file, which does not exist in S3, we see an extra LIST call gets issued to see if it's a directory (i.e. if key = "bar", it will issue an object list request for "bar/"). 

Is this really necessary, shouldn't a HEAD request be sufficient to determine if it actually exists or not? As we're creating 1000s of files, this is quite expensive, as we're effectively doubling our costs for file creation. Curious if others have experienced similar or identical issues, or if there are any workarounds. 

[https://github.com/apache/hadoop/blob/516a2a8e440378c868ddb02cb3ad14d0d879037f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3359-L3369]

 

Thanks,

Sam
Bug: Tune S3A storage class support
Status: Open | Priority: Minor | Resolution: 
Created: 08/Jun/22 19:03
Affects Version/s: 3.3.5

Followup to HADOOP-12020, with work/review from rebasing HADOOP-17833 atop it.

* Can we merge ITestS3AHugeFilesStorageClass into one of the existing test cases? just because it is slow...ideally we want as few of those as possible, even if by testing multiple things at the same we break the rules of testing.
* move setting the storage class into
setOptionalMultipartUploadRequestParameters and setOptionalPutRequestParameters
* both newPutObjectRequest() calls to set storage class
* docs to list the valid option strings. I had to delve into the AWS SDK to work them out

Once HADOOP-17833 is in, make this a new option something which can be explicitly used in createFile().
I've updated PutObjectOptions to pass a value around, and made sure it gets down to to the request factory. that leaves
* setting the storage class from the options {{CreateFileBuilder}}
* testing!
* doc update
Bug: Update class names to be clear they belong to S3A prefetching
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Jun/22 15:53

tune classnames, e.g S3InputStream -> S3ABufferedStream, S3Reader -> StoreBlockReader, S3File -> OpenS3File. I think we just want to get the S3 prefixes off as all too often that means an AWS SDK class, not something in our own code
Bug: Use async drain threshold to decide b/w async and sync draining
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/May/22 15:42

[https://github.com/apache/hadoop/pull/4294] introduces changes to drain the stream asynchronously. [https://github.com/apache/hadoop/pull/2584] introduced ASYNC_DRAIN_THRESHOLD, use this value to decide when to drain asynchronously. This can be done once the prefetching branch has been rebased.
Bug: Add S3A configuration property for `no_proxy` hosts
Status: Open | Priority: Minor | Resolution: 
Created: 16/Jun/22 13:40

Seeing as there are configuration options for proxy host, port, username, and password, there should also be an option to be able to provide to the S3 client a list of hosts to not use the proxy for (i.e. `no_proxy`)

 

I'm happy to contribute the code, but figured I'd file a ticket first to see if this the hadoop community would be open to this idea or have any desire for this feature.
Bug: S3a: allow custom retry policies
Status: Open | Priority: Minor | Resolution: 
Created: 10/Jun/22 13:15
Affects Version/s: 3.3.3

This is related to HADOOP-18285

It would be great if the retry policies were pluggable so that one could inject their own implementation to overcome exceptional cases not correctly covered by the current policy. Currently {color:#24292f}S3ARetryPolicy{color} is hard-wired and can not be replaced w/o heavy sub-classing gymnastics.
Bug: update os-maven-plugin to 1.7.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Jun/22 18:13
Affects Version/s: 3.3.3

the os-maven-plugin we build with is 1.15; the release is up to 1.7.0 -update it.

when backporting this patch, YARN-11173 must be applied to keep yarn-csi in sync
Bug: Rename CHANGES to CHANGELOG in branch-2.10
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jun/22 01:16
Affects Version/s: 2.10.2

The file names of changelog generated by releasedocmaker of Yetus was changed from CHANGES.md to CHANGELOG.md. Hyperlinks in the generated index.html of "Changelog and Release Notes" page were also affected. The CHANGES*.md generated in previous releases must be renamed to avoid 404.
Bug: Misleading method name in DistCpOptions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Apr/22 01:25
Affects Version/s: 3.3.2

Currently method in DistCpOptions withCRC was used as the following
```
withCRC(true) means check without crc
withCRC(false) means check with crc
```

which mislead the developer when we pass the paramter, we can rename the method to clear that.after that it should be:
```
withSkipCRC(true) means check without crc
withSkipCRC(false) means check with crc
```

so it will be more understandable.
Bug: Remove duplicate locks in NetworkTopology
Status: Open | Priority: Minor | Resolution: 
Created: 16/May/22 03:34

During reading the hadoop NetworkTopology.java, I suspect there is a duplicate lock.

chooseRandom(line 532), and code is:
{code:java}
final int availableNodes;
if (excludedScope == null) {
  availableNodes = countNumOfAvailableNodes(scope, excludedNodes);
} else {
  netlock.readLock().lock();
  try {
    availableNodes = countNumOfAvailableNodes(scope, excludedNodes) -
        countNumOfAvailableNodes(excludedScope, excludedNodes);
  } finally {
    netlock.readLock().unlock();
  }
} {code}
All the places where called `chooseRandom` have the global read lock, so the internal read lock is duplicated.
Bug: s3a access point xml examples are wrong
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/May/22 10:24
Affects Version/s: 3.3.2

the examples of s3a access point bindings are wrong, as the .bucket prefix is missing


{code}
<property>
    <name>fs.s3a.sample-bucket.accesspoint.arn</name>
    <value> {ACCESSPOINT_ARN_HERE} </value>
    <description>Configure S3a traffic to use this AccessPoint</description>
</property>
{code}

the property should be fs.s3a.bucket.sample-bucket.accesspoint.arn
Bug: ZKDelegationTokenSecretManager should handle duplicate Token sequenceNums
Status: Open | Priority: Minor | Resolution: 
Created: 27/May/22 16:08

The ZKDelegationTokenSecretManager relies on the TokenIdentifier sequenceNumber to identify each Token in the ZK Store. It's possible for multiple TokenIdentifiers to share the same sequenceNumber, as this is an int that can overflow. 

The AbstractDelegationTokenSecretManager uses a Map<TokenIdent, DelegationTokenInformation> so all properties in the TokenIdentifier must match. ZKDelegationTokenSecretManager should follow the same logic.
Bug: ABFS: Two BlobCreated get triggered for writing one ABFS file
Status: Open | Priority: Minor | Resolution: 
Created: 06/May/22 17:03
Affects Version/s: 3.1.1

Using the new ABFS driver to write a file on ADLS gen storage account triggers 2 BlobCreated events in the Azure backend while we are expecting one.

Here is an example code snippet for creating a file (in scala):
{code:java}
import java.io._
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path, RemoteIterator}
val conf = new Configuration()
val path = new Path("abfss://container@some-ADLS-account.dfs.core.windows.net/test.txt")
val fs = path.getFileSystem(conf)
val bs = new BufferedOutputStream(fs.create(path, true))
bs.write("test".getBytes("UTF-8"))
bs.close() {code}
Bug: Publishing maven package in com.hadoop namespace
Status: Open | Priority: Minor | Resolution: 
Created: 05/May/22 17:33

*I'm looking to get opinions on Twitter publishing the hadoop-lzo project on maven central in the com.hadoop namespace.*

Hey, I work in the open source office at Twitter, and we publish the [hadoop-lzo project|https://github.com/twitter/hadoop-lzo], which is used by a handful of folks. Some of you may know the history better than I, but this was an old project from Google Code that got migrated to Twitter's GitHub org years ago, and is currently published on our maven host at maven.twttr.com (and only there).

I don't know the full history of why it is only published there, but we've had a few incidents of that service being unavailable, [breaking builds for downstream users|https://github.com/twitter/hadoop-lzo/issues/148]. I'd like to remove this reliance on the maven.twttr.com host by publishing this on maven central.

Ideally, I'd like to do it in a way that requires the least amount of work for downstream users, which would mean using the existing group and project ID: com.hadoop.gplcompression.hadoop-lzo. The com.hadoop namespace is mostly unused on maven ([https://repo1.maven.org/maven2/com/hadoop/])... interestingly, only used by what I believe is another fork of the same compression library. But since hadoop.com is presumably owned by the Hadoop project (whois records are hidden, but it [appears to have always redirected|https://web.archive.org/web/*/hadoop.com] to the project website), I wanted to run this by someone associated with the project before we did this.

Is there any objection to us publishing this maven library into that namespace?  The alternative of course is to switch to a com.twitter namespace, but that would require more changes for downstream users.
Bug: Improve build performance of modules
Status: Open | Priority: Minor | Resolution: 
Created: 03/May/22 17:12

Hi, I'm now using {*}hadoop{*}. I found that the build time of the project is not very fast when I used  *mvn -T 1C install -DskipTests* command to build the project during Github Actions. So I try to speed up the build performance of the project. The goal here is to clean up dependencies between maven modules in the project in order to improve the build performance.

Thread usage before cleaning up module dependencies was this:

!image-2022-05-04-01-03-59-169.png|width=559,height=116!

Using *mvn dependency:analyze* command I managed to have an overview of the unused dependencies of every module. I focused only on the dependencies between modules. The following dependencies can be cleaned up:
{noformat}
org.apache.hadoop:hadoop-client-minicluster(hadoop-client-modules/hadoop-client-minicluster) -> org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime)
org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime) -> org.apache.hadoop:hadoop-client-api(hadoop-client-modules/hadoop-client-api)
org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime) -> org.apache.hadoop:hadoop-client(hadoop-client-modules/hadoop-client)
org.apache.hadoop:hadoop-dist(hadoop-dist) -> org.apache.hadoop:hadoop-client-check-test-invariants(hadoop-client-modules/hadoop-client-check-test-invariants){noformat}
After clean up, the build time has been reduced by *3min* when I use parallel build during Github Actions.
Thread usage after cleaning up module dependencies was this:

!image-2022-05-04-01-02-55-509.png|width=553,height=102!

Could you help me review this issue? I can submit a PR to improve build performance of the project.

Thank you very much for your attention.
Best regards.
Bug: Improve S3A committers documentation clarity
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 20/Jun/22 15:13
Affects Version/s: 3.3.5

I recently was learning more about the S3A committers. I'm hoping to provide some improvements as someone who has recently read [this documentation|https://github.com/apache/hadoop/blob/1f157f802d2d6142d21482eaa86baf1bef458ed4/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md#L495] without fully understanding prior.

For instance, referencing different components more explicitly and adding pre-requisite info.
Bug: Fix typos in the definition of zstd
Status: Resolved | Priority: Trivial | Resolution: Duplicate
Created: 08/Jun/22 11:34

There are some typos in file *hadoop-common/pom.xml* and *ZStandardCompressor.c*
hadoop-common/pom.xml:
{code:xml|title=pom.xml|borderStyle=solid}
<require.ztsd>false</require.ztsd> 
<argument>/p:RequireZstd=${require.ztsd}</argument>
{code}
The correct print as follows:
{code:xml}
<require.zstd>false</require.zstd>
<argument>/p:RequireZstd=${require.zstd}</argument>
{code}
ZStandardCompressor.c:
{code:c|title=ZStandardCompressor.c|borderStyle=solid}
// Load the libztsd.so from disk 
{code}
The correct print as follows:
{code:c}
// Load the libzstd.so from disk
{code}
Bug: ZKDelegationTokenSecretManager causes ArithmeticException due to improper numRetries value checking
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 24/Jul/23 16:38
Affects Version/s: 3.4.0

h2. What happened

There is no value checking for parameter {{{}zk-dt-secret-manager.zkNumRetries{}}}. This may cause improper calculations and crashes the system like division by 0.
h2. Buggy code

In {{{}ZKDelegationTokenSecretManager.java{}}}, there is no value checking for {{numRetries}} which is passed directly in {{RetryNTimes}} constructor. When {{numRetries}} is mistakenly set to 0, the code would cause division by 0 and throw ArithmeticException to crash the system.
{noformat}
public ZKDelegationTokenSecretManager(Configuration conf) {
        ...
        int numRetries =
            conf.getInt(ZK_DTSM_ZK_NUM_RETRIES, ZK_DTSM_ZK_NUM_RETRIES_DEFAULT);
        builder =
            ...
                .retryPolicy(
                    new RetryNTimes(numRetries, sessionT / numRetries));
        ...{noformat}
h2. How to reproduce
 # set zk-dt-secret-manager.zkNumRetries=0
 # run org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager.testMultiNodeOperations
 # You will see the following stack trace.

{noformat}
java.lang.RuntimeException: Could not Load ZK acls or auth: java.lang.ArithmeticException: / by zero
    at org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.<init>(ZKDelegationTokenSecretManager.java:227)
    at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager.<init>(DelegationTokenManager.java:99)
    at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.<init>(DelegationTokenManager.java:120)
    at org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager.testMultiNodeOperations(TestZKDelegationTokenSecretManager.java:113)
        ...{noformat}
For an easy reproduction, run the reproduce.sh in the attachment. We are happy to provide a patch if this issue is confirmed.
Bug: S3A: V2 SDK client does not work with third-party store
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 12/Sep/23 14:58
Affects Version/s: 3.4.0

testing against an external store without specifying region now blows up because the region is queried off eu-west-1.

What are we do to here? require the region setting *which wasn't needed before? what even region do we provide for third party stores?
Bug: ClassCastException in test TestRPC#testWrappedStopProxy
Status: Open | Priority: Critical | Resolution: 
Created: 14/Jul/23 13:33

h2. What happened:

In HCommon, test TestRPC#testWrappedStopProxy tries to cast org.apache.hadoop.ipc.WritableRpcEngine$Invoker to org.apache.hadoop.ipc.TestRPC$StoppedInvocationHandler and causes ClassCastException.
h2. Buggy code:

In TestRPC.java:
{code:java}
@Test
public void testWrappedStopProxy() throws IOException {
  StoppedProtocol wrappedProxy = RPC.getProxy(StoppedProtocol.class,
      StoppedProtocol.versionID, null, conf);
  StoppedInvocationHandler invocationHandler = (StoppedInvocationHandler)    // <--- Here causes ClassCastException
      Proxy.getInvocationHandler(wrappedProxy);
  ...
} {code}
h2. How to reproduce:

(1) Just directly run test TestRPC#testWrappedStopProxy.

You can use the reproduce.sh in the attachment to easily reproduce the bug.

We are happy to provide a patch if this issue is confirmed.
Bug: NegativeArraySizeException thrown in FSOutputSummer.java given large file.bytes-per-checksum
Status: Open | Priority: Critical | Resolution: 
Created: 14/Sep/23 00:38
Affects Version/s: 3.3.6

Buffer size of FSOutputSummer equals to `file.bytes-per-checksum` times `BUFFER_NUM_CHUNKS`. A large `file.bytes-per-checksum` causes buffer size to overflow and crash with NegativeArraySizeException.

To reproduce:
1. set `file.bytes-per-checksum` to 238609295
2. `mvn surefire:test -Dtest=org.apache.hadoop.hdfs.TestDecommissionWithStriped#testFileSmallerThanOneStripe`

We created a PR that provides a fix which checks the buffer size is positive after multiplying `file.bytes-per-checksum` with `BUFFER_NUM_CHUNKS`
Bug: LdapGroupsMapping crashes with NullPointerException while going up the group hierarchy 
Status: Open | Priority: Critical | Resolution: 
Created: 20/Jul/23 16:36
Affects Version/s: 3.3.6

h2. What happened:

When set {{hadoop.security.group.mapping.ldap.search.group.hierarchy.levels}} to a value larger than 0, {{goUpGroupHierarchy}} in {{org/apache/hadoop/security/LdapGroupsMapping.java}} may return a null {{{}groupResults{}}}and use it without checking null.
h2. Buggy code:
{noformat}
  void goUpGroupHierarchy(Set<String> groupDNs, int goUpHierarchy, Set<String> groups) throws NamingException {
    if (goUpHierarchy <= 0 || groups.isEmpty()) {
      return;
    }
    ...
    NamingEnumeration<SearchResult> groupResults = context.search(groupbaseDN, filter.toString(), SEARCH_CONTROLS);
    while (groupResults.hasMoreElements()) {          // <--- Here groupResults may be null
    ...
    }
    ...
  }{noformat}
h2. How to reproduce:

(1) Set {{hadoop.security.group.mapping.ldap.search.group.hierarchy.levels}} to 1
(2) Run test {{org.apache.hadoop.security.TestLdapGroupsMapping#testGetGroupsWithConnectionClosed}}
h2. Stack trace:
{noformat}
java.lang.NullPointerException
        at org.apache.hadoop.security.LdapGroupsMapping.goUpGroupHierarchy(LdapGroupsMapping.java:612)
        at org.apache.hadoop.security.LdapGroupsMapping.lookupGroup(LdapGroupsMapping.java:489)
        at org.apache.hadoop.security.LdapGroupsMapping.doGetGroups(LdapGroupsMapping.java:552)
        at org.apache.hadoop.security.LdapGroupsMapping.getGroups(LdapGroupsMapping.java:365){noformat}
For an easy reproduction, run the reproduce.sh in the attachment.


We also create a PR provides a fix by checking the groupResults is not Null before it is accessed, similar to what's done in `org.apache.hadoop.security.LdapGroupsMapping#lookupGroup`
Bug: Missing null check when running doRun method
Status: Open | Priority: Critical | Resolution: 
Created: 27/Jul/23 14:44
Affects Version/s: 3.3.3

h2. What happened?

Got NullPointerException when running {{doRun}} method in {{{}ZKFailoverController.java{}}}.
h2. Where's the bug?

In line 258 of {{{}ZKFailoverController.java{}}},the code lacks a check to verify whether {{rpcServer}} is null or not.
{noformat}
private int doRun(String[] args)
    throws Exception {
      ...
    } catch (Exception e) {
          LOG.error("The failover controller encounters runtime error: ", e);
          throw e;
    } finally {
      rpcServer.stopAndJoin();
      ...
    }{noformat}
As a result, when the configuration provides a null rpcServer, the {{rpcServer.stopAndJoin()}} operation will throw a NullPointerException.
It is essential to add a null check for the rpcServer parameter before using it.
h2. How to reproduce?

(1) set {{ipc.server.handler.queue.size}} to {{0}}
(2) run {{org.apache.hadoop.ha.TestZKFailoverController#testAutoFailoverOnLostZKSession}}
h2. Stacktrace
{noformat}
Caused by: java.lang.NullPointerException
    at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:258)
    at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:63)
    at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:181)
    at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:177)
    at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:503)
    at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:177)
    at org.apache.hadoop.ha.MiniZKFCCluster$DummyZKFCThread.doWork(MiniZKFCCluster.java:301)
    at org.apache.hadoop.test.MultithreadedTestUtil$TestingThread.run(MultithreadedTestUtil.java:189){noformat}
For an easy reproduction, run the reproduce.sh in the attachment.

We are happy to provide a patch if this issue is confirmed.
Bug: Collision of config key name fs.viewfs.mounttable.default.name.key to other keys that specify the entry point to mount tables
Status: Open | Priority: Critical | Resolution: 
Created: 14/Jul/23 13:06

h2. What happened:

When manually set fs.viewfs.mounttable.default.name.key to default (the same as default value) in HCommon, test org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches would fail.
But the test can pass if this parameter is not manually set in the configuration file.
h2. Where's the bug:

In the constructor of InodeTree, the tree attempts to get all the mount table entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry<String, String> si : config) {       
    final String key = si.getKey();       
    if (!key.startsWith(mountTablePrefix)) {         
        continue;       
    }  
    
    gotMountTableEntry = true;       
    LinkType linkType;       
    String src = key.substring(mountTablePrefix.length());       
    ...
{code}
Here mountTablePrefix="fs.viewfs.mounttable.default.". However, it just so happens that the name of the configuration users use to specify the default mount table is fs.viewfs.mounttable.default.name.key. Thus, if a user specifies the default mount table and uses InodeTree the name.key would be falsely parsed as the entry point to one of the mount tables, which would cause InodeTree to throw an exception since name.key is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize: Invalid entry in Mount table in config: name.key    
        at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
        at org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
        at org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:

(1) Set fs.viewfs.mounttable.default.name.key to default
(2) Run test org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches

You can use the reproduce.sh in the attachment to easily reproduce the bug.

We are happy to provide a patch if this issue is confirmed.