av9ash/GitBugs huggingface dataset:
Bug: ChecksumFileSystem does not filter .crc files in listStatusIterator
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 18/May/21 16:28

The listStatusIterator does not filter the .crc files. It has to be overridden and implemented in ChecksumFileSystem. This affects all filesystems derived from ChecksumFileSystem, the most notable is LocalFileSystem which is used by default for file://.
Bug: IPC client is stuck, waiting for connections map to be empty
Status: Open | Priority: Major | Resolution: 
Created: 24/May/21 11:27
Affects Version/s: 2.10.1

In our cluster we have observed when close command is issued to Datanode, It is stuck indefinitely waiting for connections map to be empty for some reason the map is not empty even after the connections have closed. [code link|https://github.com/apache/hadoop/blob/branch-2.10.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1360]. 

Instead of infinite wait a timed wait (based on retries or timeout) would enable to shutdown the datanode in timely manner.

{code:java}
for (Connection conn : connections.values()) {
      conn.interrupt();
      conn.interruptConnectingThread();
    }
    
    // wait until all connections are closed
    while (!connections.isEmpty()) {
      try {
        Thread.sleep(STOP_SLEEP_TIME_MS);
      } catch (InterruptedException e) {
      }
    }
{code}
Bug: abfs: Unable to use OAuth authentication at storage account level if the default authn type is Custom
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/May/21 11:43
Affects Version/s: 3.3.0

If we set the default auth type as Custom and then decided to use OAuth type for some select storage accounts then the fs initialization for storage accounts with Oauth type authn fails.

Steps to recreate
{code:java}
conf.set("fs.abfss.impl", "org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem")
conf.set("fs.azure.account.auth.type", "Custom")
conf.set("fs.azure.account.oauth.provider.type", "xxx.yyy.zzz.ADTokenAdaptee")
conf.set("fs.azure.account.auth.type.abctest.dfs.core.windows.net", "OAuth")
conf.set("fs.azure.account.oauth.provider.type.abctest.dfs.core.windows.net",
  "org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider")

val fs = FileSystem.get(
  new URI("abfss://conatinerxyz@abctest.dfs.core.windows.net/arion-scribe-de-dev"),
  conf)
{code}
Error: java.lang.RuntimeException: class xxx.yyy.zzz.ADTokenAdaptee not org.apache.hadoop.fs.azurebfs.oauth2.AccessTokenProvider

Cause:

In [AbfsConfiguration. getTokenProvider|https://github.com/apache/hadoop/blob/aa96f1871bfd858f9bac59cf2a81ec470da649af/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java#L540] , after evaluating the auth type as OAuth, the program proceeds to get the implementing class using property `fs.azure.account.auth.type.abctest.dfs.core.windows.net`,  while doing so the first [step|https://github.com/apache/hadoop/blob/aa96f1871bfd858f9bac59cf2a81ec470da649af/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java#L321] is to get the default auth class (`fs.azure.account.oauth.provider.type`), which in our case is Custom. Here the problem is Default Auth class is CustomTokenProviderAdaptee implementation and not implementing  AccessTokenProvider.class, hence the program would fail.

proposed solution:
 In the getClass function in AbfsConfiguration, we split the logic and not use the default value property
{code:java}
public <U> Class<? extends U> getClass(String name, Class<? extends U> defaultValue, Class<U> xface) {

    Class<? extends U> klass = rawConfig.getClass(accountConf(name),
            null, xface);
    
    if(klass!=null){
        return klass;
    }else{
        return rawConfig.getClass(name, defaultValue, xface);
    }
}
{code}
Bug: Avoid potential NPE by using Path#getParentPath API in hadoop-huaweicloud
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/21 03:54

Hello,
Our code analyses found a following potential NPE:

 
{code:java}
public Path getParent() {
    String path = uri.getPath();
    int lastSlash = path.lastIndexOf('/');
    int start = startPositionWithoutWindowsDrive(path);
    if ((path.length() == start) ||               // empty path
        (lastSlash == start && path.length() == start+1)) { // at root
      return null; <--- Null returned
    }
{code}
 
{code:java}
private static void getDirectories(final String key, final String sourceKey,
      final Set<String> directories) {
    Path p = new Path(key);
    Path sourcePath = new Path(sourceKey);
    // directory must add first
    if (key.endsWith("/") && p.compareTo(sourcePath) > 0) {
      directories.add(p.toString());
    }
    while (p.compareTo(sourcePath) > 0) {
      p = p.getParent(); <--- NPE
      if (p.isRoot() || p.compareTo(sourcePath) == 0) {
        break;
      }
{code}
Given a root path, it will lead to NPE at method getDirectories

 

Full trace:

 

1. Return null to caller
[https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java#L432]

2. Function getParent executes and returns
[https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSObjectBucketUtils.java#L875]

3. The return value of function getParent is passed as the this pointer to function isRoot (the return value of function getParent can be null)
[https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSObjectBucketUtils.java#L876]

Commit: f40e3eb0590f85bb42d2471992bf5d524628fdd6
Bug: Potenail NPE in org.apache.hadoop.classification.tools
Status: Open | Priority: Major | Resolution: 
Created: 10/May/21 03:28

Hello,
Our code analyses found the following potential NPE:
{code:java}
  private static Object process(Object obj, Class<?> type) {
    if (obj == null) {
      return null;
    }{code}
{code:java}
  public static boolean start(RootDoc root) {
    System.out.println(
        ExcludePrivateAnnotationsStandardDoclet.class.getSimpleName());
    RootDoc excludedDoc = RootDocProcessor.process(root);
    if (excludedDoc.specifiedPackages().length == 0) { // NPE
      return true;
    }
{code}
 

Full Trace:

 

1. Return null to caller
https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java#L61

2. Return the return value of function process to caller
https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java#L56

3. Function process executes and stores the return value to excludedDoc (excludedDoc can be null)
https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/ExcludePrivateAnnotationsStandardDoclet.java#L41

4. excludedDoc is passed as the this pointer to function com.sun.javadoc.RootDoc.specifiedPackages (excludedDoc can be null)
https://github.com/apache/hadoop/blob/f40e3eb0590f85bb42d2471992bf5d524628fdd6/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/ExcludePrivateAnnotationsStandardDoclet.java#L42

Commit: f40e3eb0590f85bb42d2471992bf5d524628fdd6
Bug: LdapGroupsMapping$LdapSslSocketFactory ClassNotFoundException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/21 10:25
Affects Version/s: 3.2.2

Using LdapGroupsMapping with SSL enabled causes ClassNotFoundException when it is called through native threads, such as Apache Impala does.

When a thread is attached to the VM, the currentThread's context classloader is null, so when jndi internally tries to use the current thread's context classloader to load the socket factory implementation, the Class.forname(String, boolean, ClassLoader) method gets a null as the loader  uses the bootstrap classloader.
 Meanwhile the LdapGroupsMapping class and the SslSocketFactory defined in it is loaded by the application classloader from its classpath.

As the bootstrap classloader does not have hadoop-common in its classpath, when a native thread tries to use/load the LdapGroupsMapping class it can't because the bootstrap loader can't load anything from hadoop-common. The correct solution seems to be to set the currentThread's context classloader to the classloader of LdapGroupsMapping class before initializing the jndi internals, and then reset to the original value after, with that we can ensure that the behaviour of other things does not change, but this failure can be avoided as well.

Attached the complete stacktrace to this Jira.
Bug: Add precommit check for Hadoop Thirdparty repository
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/21 07:19

Not a blocker for thirparty-1.1.0 release, but it would help a lot if we have a precommit check for the hadoop-thirdparty. HBase thirdparty is a good example.

 

Reference: HBASE-24899 and [https://ci-hadoop.apache.org/view/HBase/job/HBase/job/HBase-Thirdparty-PreCommit/configure]
Bug: Remove hadoop-shaded-jaeger from hadoop-thirdparty 1.1.0 release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/21 00:39

The hadoop-shaded-jaeger was created to produce a shaded jaegertracer artifact used by the core Hadoop code. But the core Hadoop side isn't ready to consume it, and the version we shade is too old, need update.

So regardless, we can't release as is, therefore I propose to remove this jar from hadoop-thirdparty 1.1.0 for now.
Bug: Update dependency-check-maven version
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/21 00:41
Affects Version/s: thirdparty-1.1.0

The version of org.owasp:dependency-check-maven we use (1.4.3) is too old. The command (mvn dependency-check:aggregate) doesn't even run. Update to the latest version.
Bug: Hadoop Customed Counter
Status: Open | Priority: Major | Resolution: 
Created: 24/Apr/21 14:12

I have run a hadoop job and set multiple reducers in the reduce phase, then I have customize a counter to sum the results of multiple reducers. However, the data type of the result in each reducer is double, while the increment method of conuter only supports the long data type. If the conversion from double to long is forced, the precision will be lost. How can I solve this problem?
Bug: abfs incremental listing to support many active listings
Status: Open | Priority: Major | Resolution: 
Created: 22/Apr/21 13:23
Affects Version/s: 3.3.1

Each incremental iterator submits an async fetcher operation into the JVM's common ForkJoin thread pool, which defaults to # of cores -1., unless set iin "java.util.concurrent.ForkJoinPool.common.parallelism";

Given the LIST calls are going to be blocking, this may puts a limit on the performance of listing if you have many threads executing list requests, e.g spark workers.

Reviewing the code, the maximum number of list operations which can collect results will be limited to the #of cores -the others are going to block until the lists have been processed.

Which may also means: if you have multiple incremental iterators in the same thread (e.g. treewalking) there's a risk that you could actually deadlock. 

I'm not convinced this will happen, as once each listing has reached the end of its directory or there are 10 pages in the result queue, the submitted operation will complete.

But: we need a test for this. Is there any public abfs store with many, many objects we could use as a source for listings, similar to the AWS landsat repo we (ab)use for such purposes in the s3a ITests?
Bug: Restrict the "-skipTrash" param for accidentally deletes data
Status: Open | Priority: Major | Resolution: 
Created: 16/Apr/21 08:20

Suppose the user tries to delete the data from CLI with the "-skipTrash" param but by mistake, he deleted a couple of directories but actually, that directory user want to retain then their is no way to retrieve the delete data.

It will be good to have a confirm message like: "Skip the trash for the hdfs:///dri1/file.txt files? (Y or N)" ro we can completely disable the "-skipTrash" param.
Bug: Please upgrade the log4j dependency to log4j2
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 11/Apr/21 13:29
Affects Version/s: 3.3.0

The log4j dependency being use by hadoop-common is currently version 1.2.17. Our fortify scan picked up a couple of issues with this dependency. Please upgrade it to the latest version of log4j2 dependencies:


<dependency>
 <groupId>org.apache.logging.log4j</groupId>
 <artifactId>log4j-api</artifactId>
 <version>2.14.1</version>
</dependency>

<dependency>
 <groupId>org.apache.logging.log4j</groupId>
 <artifactId>log4j-core</artifactId>
 <version>2.14.1</version>
</dependency>

 

The slf4j dependency will need to be updated as well after you upgrade log4j to log4j2.
Bug: s3a listing IOStatistics to count #of entries returned per LIST call
Status: Open | Priority: Minor | Resolution: 
Created: 04/Jun/21 09:21
Affects Version/s: 3.3.1


Extend IOStatistics tracking for the S3A Listing operations to count the #of object entries returned in a LIST call. This would be interesting when working with versioned buckets, to see what the actual page size of a listing is, so understand how it behaves there, and maybe get some estimate of how many old versions/tombstones are being skipped. 

MUST: add to S3AFS stats, SHOULD add to Listing
Bug: TestS3AGetFileStatus:testNotFound() to use intercept()
Status: Open | Priority: Minor | Resolution: 
Created: 21/Jun/21 14:08
Affects Version/s: 3.3.1

The test TestS3AGetFileStatus:testNotFound() uses a JUnit rule of an ExpectedException, which is now deprecated. It should move to intercept()

This could be time to clean up that entire little test suite with a move to AssertJ asserts.
Bug: ABFS: Remove the workaround used for Http PATCH
Status: Open | Priority: Minor | Resolution: 
Created: 15/Apr/21 11:07
Affects Version/s: 3.3.0

JDK7 does not support PATCH, so to to achieve the same as a workaround we use PUT and specify the real method in the X-Http-Method-Override header.
This needs to be changed. Since we are using java 8.
Bug: ABFS: transient failure of TestAzureBlobFileSystemFileStatus.testLastModifiedTime
Status: Open | Priority: Minor | Resolution: 
Created: 16/Apr/21 15:03
Affects Version/s: 3.3.1

saw a transient failure of TestAzureBlobFileSystemFileStatus.testLastModifiedTime during a parallel (threads=8) test run against UK-west

{code}
java.lang.AssertionError: lastModifiedTime should be before createEndTime
{code}

assumption: the times are in fact equal, though the fact the assert doesn't include the values makes this hard to guarantee
Bug: ABFS IsNamespaceEnabled shouldn't use/log exceptions as default execution path
Status: Open | Priority: Minor | Resolution: 
Created: 21/Apr/21 09:59
Affects Version/s: 3.3.1

abfs isNameSpaceEnabled() call seems to raise then swallow TrileanConversionExceptions during the default execution path, logging the whole stack at debug

this is inefficient, noisy and distracting, as now there are stack traces to ignore when debugging things.

Better: treat Trilean.UKNOWN as exactly that and react accordingly
Bug: ABFS: rename ListResultSchemaTest to TestListResultSchema so maven tests run it
Status: Open | Priority: Minor | Resolution: 
Created: 15/Apr/21 15:54
Affects Version/s: 3.3.1

The test {{ListResultSchemaTest}} from  HADOOP-17086 won't actually be run in maven test runs because the Test- needs to come as a prefix.

Fix: rename
Bug: ABFS: Improve test scripts
Status: Patch Available | Priority: Minor | Resolution: 
Created: 21/Jun/21 07:19
Affects Version/s: 3.3.0

Current test run scripts need manual update across all combinations in runTests.sh for account name and is working off a single azure-auth-keys.xml file. While having to test across accounts that span various geo, the config file grows big and also needs a manual change for configs such as fs.contract.test.[abfs/abfss] which has to be uniquely set. To use the script across various combinations, dev to be aware of the names of all the combinations defined in runTests.sh as well.

 

These concerns are addressed in the new version of the scripts.
Bug: ABFS: Append blob tests with non HNS accounts fail
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/May/21 10:00
Affects Version/s: 3.3.2

Append blob tests with non HNS accounts fail.
 # The script to run the tests should ensure that append blob tests with non HNS account don't execute
 # Should have proper documentation mentioning that append blob is allowed only for HNS accounts
Bug: Fix test failures in org.apache.hadoop.fs.azure.ITestOutputStreamSemantics
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Apr/21 11:21
Affects Version/s: 3.3.1

Failures after HADOOP-13327
PageBlob and Compacting BlockBlob having only hflush and hsync capability. The test wrongly assert capability DROPBEHIND, READAHEAD, UNBUFFER
Bug: compatibility table in directory_markers.md doesn't render right
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Jun/21 19:44
Affects Version/s: 3.3.1


The compatibility table in the directory markers doc doesn't render right; looks like what is meant to be the top and bottom of the table are being interpreted as horizontal lines, which confuses the maven site code

Renders ~OK on github
https://github.com/apache/hadoop/blob/branch-3.3.1/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/directory_markers.md
Bug: Lock File System Creator Semaphore Uninterruptibly
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jun/21 00:58
Affects Version/s: 3.3.1

{{FileSystem}} "creator permits" are acquired in an interruptable way.  This changed the behavior of the call because previous callers were handling the IOException as a critical error.  An interrupt was handled in the typical {{InterruptedException}} way.  Lastly, there was no documentation of this new event so again, callers are not prepared.

Restore the previous behavior and lock the semaphore Uninterruptibly.
Bug: Configuration ${env.VAR:-FALLBACK} should eval FALLBACK when restrictSystemProps=true 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Apr/21 13:36
Affects Version/s: 3.3.0

When configuration reads in resources with a restricted parser, it skips evaluaging system ${env. } vars. But it also skips evaluating fallbacks

As a result, a property like {{fs.s3a.buffer.dir}}

{code}
${env.LOCAL_DIRS:-${hadoop.tmp.dir}} ends up evaluating as ${env.LOCAL_DIRS:-${hadoop.tmp.dir}}
{code}

It should instead fall back to the "env var unset" option of ${hadoop.tmp.dir}. This allows for configs (like for s3a buffer dirs) which are usable in restricted mode as well as unrestricted deployments.
Bug: Fix issue of the StatisticsDataReferenceCleaner cleanUp
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 21/May/21 14:03
Affects Version/s: 3.2.1

Cleaner thread will be blocked if we remove reference from ReferenceQueue unless the `queue.enqueue` called.
----
    As shown below, We call ReferenceQueue.remove() now while cleanUp, Call chain as follow：

                         *StatisticsDataReferenceCleaner#queue.remove()  ->  ReferenceQueue.remove(0)  -> lock.wait(0)*

    But, lock.notifyAll is called when queue.enqueue only, so Cleaner thread will be blocked.

 

ThreadDump:
{code:java}
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f7afc088800 nid=0x2119 in Object.wait() [0x00007f7b00230000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c00c2f58> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000000c00c2f58> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153){code}
Bug: DistCp: distcp fail when renaming within ftp filesystem
Status: Reopened | Priority: Minor | Resolution: 
Created: 02/Jun/21 11:13
Affects Version/s: 3.3.0

When I copy file from a ftp filesystem to other ftp filesystem by distCp, there are two problems. Firstly,  the tmp target path can't be found in the exception information. After my test, it was caused by the comma. Then when the uri of target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_000000_0.1622634094600 does not existjava.io.FileNotFoundException: Source path ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_000000_0.1622634094600 does not exist at org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)

{code}

When I update the tempFileName from '.distcp.tmp.XXX' in  org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to 'distcp.tmp.XXX', the question one is gone.

The targetPath can be renamed when only the path of uri exists in org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
Bug: ABFS: testBlobBackCompatibility, testRandomRead & WasbAbfsCompatibility tests fail when triggered with default configs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/May/21 09:56

testBlobBackCompatibility, testRandomRead & WasbAbfsCompatibility tests fail when triggered with default configs as http is not enabled on gen2 accounts by default.

 

Options to fix it:

tests' config should enforce https by default 

or the tests should be modified not execute http requests
Bug: Path.suffix() raises an exception when the path is in the root dir
Status: Open | Priority: Minor | Resolution: 
Created: 03/Jun/21 17:53
Affects Version/s: 3.3.1

Root cause of SPARK-34298.

If you have a Path (/something) and call suffix on it.

{code}
 new Path("/something").suffix("else")
{code}
you see an error because the path doesn't have a parent

{code}
Exception in thread "main" java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:168)[....]
at org.apache.hadoop.fs.Path.suffix(Path.java:446)
{code}
Bug: abfs test failure ITestSmallWriteOptimization.testSmallWriteOptimization
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 27/May/21 13:28
Affects Version/s: 3.4.0

(transient) failure of ITestSmallWriteOptimization.testSmallWriteOptimization; Azure cardiff over a long-haul link

This is on my manifest committer branch -but I've not done any changes to input streams there

{code}
[ERROR] Failures: 
[ERROR]   ITestSmallWriteOptimization.testSmallWriteOptimization:324->formulateSmallWriteTestAppendPattern:437->assertOpStats:499->AbstractAbfsIntegrationTest.assertAbfsStatistics:453->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in connections_made expected:<7> but was:<8>
{code}