av9ash/GitBugs huggingface dataset:
Bug: TestServiceInterruptHandling.testRegisterAndRaise failing intermittently
Status: Open | Priority: Minor | Resolution: 
Created: 25/Aug/22 08:26
Affects Version/s: 3.3.3

TestServiceInterruptHandling.testRegisterAndRaise faling with

 
|java.lang.AssertionError: interrupt data|
|at org.junit.Assert.fail(Assert.java:89)|
|at org.junit.Assert.assertTrue(Assert.java:42)|
|at org.junit.Assert.assertNotNull(Assert.java:713)|
|at org.apache.hadoop.service.launcher.TestServiceInterruptHandling.testRegisterAndRaise(TestServiceInterruptHandling.java:48)|
|at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)|
|at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)|
|at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)|
|at java.lang.reflect.Method.invoke(Method.java:498)|
|at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)|
|at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)|
|at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)|
|at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)|
|at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)|
|at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)|
|at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)|
|at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)|
|at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)|
|at java.util.concurrent.FutureTask.run(FutureTask.java:266)|
|at java.lang.Thread.run(Thread.java:748)|
| |
Bug: S3AFileSystem removes Path when calling createS3Client
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jul/22 14:36
Affects Version/s: 3.3.0

when using hadoop and spark to read/write data from an s3 bucket like -> s3a://bucket/path and using a custom Credentials Provider, the path is removed from the s3a URI and the credentials provider fails because the full path is gone.

In Spark 3.2,
It was invoked as -> s3 = ReflectionUtils.newInstance(s3ClientFactoryClass, conf)
.createS3Client(name, bucket, credentials); 

But In spark 3.3.3
It is invoked as s3 = ReflectionUtils.newInstance(s3ClientFactoryClass, conf).createS3Client(getUri(), parameters);
the getUri() removes the path from the s3a URI
Bug: ITestS3AContractVectoredRead.testStopVectoredIoOperationsUnbuffer failing
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Sep/22 12:20
Affects Version/s: 3.3.5

seeing a test failure in both parallel and single test case runs of {{ITestS3AContractVectoredRead.testStopVectoredIoOperationsUnbuffer))
Bug: Improve vectored IO api spec. 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Aug/22 21:38

Let's add more details to the vectored IO api spec for better clarity. 
 * the position returned by getPos(); is undefined afterwards.
 * note that if a file is changed during a read, the output is again undefined. some ranges may be old data, some may be new, *and some may be both
 * note that while reads are active, normal fs api calls may block.
Bug: Fix eval expression in hadoop-functions.sh
Status: Open | Priority: Minor | Resolution: 
Created: 07/Jul/22 01:08
Affects Version/s: 3.3.3

Need to fix the eval expression.

1. Prefix exec by eval in Hadoop bin scripts Prior to this change, if HADOOP_OPTS contains any arguments that include a space, the command is not parsed correctly. For example, if HADOOP_OPTS="... -XX:OnOutOfMemoryError=\"kill -9 %p\" ...", the bin/hadoop script will fail with the error "Unrecognized option: -9". No amount of clever escaping of the quotes or spaces in the "kill -9 %p" command will fix this. The only alternative appears to be to use 'eval'. Switching to use 'eval' *instead of* 'exec' also works, but it results in an intermediate bash process being left alive throughout the entire lifetime of the Java proces being started. Using 'exec' prefixed by 'eval' as has been done in this commit gets the best of both worlds, in that options with spaces are parsed correctly, and you don't end up with an intermediate bash process as the parent of the Java process.

2. We can replace single quote with escape-char and a double quote.
Bug: Improve ZKDelegationTokenSecretManager#startThead With recommended methods.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Aug/22 11:46
Affects Version/s: 3.4.0

When reading the code, I found a deprecated method to use. In ZKDelegationTokenSecretManager#startThead, the code here uses the Curator's EnsurePath,
But EnsurePath is deprecated, use the recommended method instead

public class EnsurePath
Deprecated.
Since 2.9.0 - Prefer CuratorFramework.create().creatingParentContainersIfNeeded() or CuratorFramework.exists().creatingParentContainersIfNeeded()
Bug: IOStatisticsContext tuning
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jul/22 13:34
Affects Version/s: 3.3.5

Tuning of the IOStatisticsContext code


h2. change property name  to fs.iostatistics....

there are other fs.iostatistics options, the new one needs consistent naming

h2. enable in hadoop-aws

edit core-site.xml in hadoop-aws/test/resources to always collect context iOStatistics

This helps qualify the code
{code}
	  <property>
	    <name>fs.thread.level.iostatistics.enabled</name>
	    <value>true</value>
	  </property>

{code}

h3.  IOStatisticsContext to add add static probe to see if it is enabled.

lets apps know not to bother collecting/reporting
Bug: ITestS3Select.testSelectSeekFullLandsat is timing out
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jul/22 14:18

ITestS3Select.testSelectSeekFullLandsat is timing out. When I run it separately on my IDE, I get the following output repeatedly till I stop the test manually:

 
{code:java}
2022-07-25 15:15:37,463 [JUnit-testSelectSeekFullLandsat] DEBUG s3a.Invoker (DurationInfo.java:close(101)) - read(): duration 0:00.000s
2022-07-25 15:15:37,463 [JUnit-testSelectSeekFullLandsat] DEBUG s3a.Invoker (DurationInfo.java:<init>(80)) - Starting: read()
2022-07-25 15:15:37,463 [JUnit-testSelectSeekFullLandsat] DEBUG s3a.Invoker (DurationInfo.java:close(101)) - read(): duration 0:00.000s
2022-07-25 15:15:37,463 [JUnit-testSelectSeekFullLandsat] DEBUG s3a.Invoker (DurationInfo.java:<init>(80)) - Starting: read()
2022-07-25 15:15:37,463 [JUnit-testSelectSeekFullLandsat] DEBUG s3a.Invoker (DurationInfo.java:close(101)) - read(): duration 0:00.000s {code}
When running the entire test suite on my EC2, I get:
{code:java}
Time elapsed: 600.01 s  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 600000 milliseconds {code}
Bug: ITestCustomSigner fails when access point name has '-' 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Jul/22 12:50

If you create an access point name which has hyphens, eg custom-signer-access-point, the ITestCustomSigner fails. This is because the array is split on "-" [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestCustomSigner.java#L231] and then joined, so access point name becomes "customsigneraccesspoint".
Bug: Supporting performant output committers for Aliyun
Status: Open | Priority: Minor | Resolution: 
Created: 13/Jul/22 08:24
Affects Version/s: 3.3.3

Due to the non-performant rename on OSS (similar to s3), zero-rename output committer (similar to HADOOP-13786 which provides this for S3) can be added for Aliyun as well. I see [similar output committer provided by Alibaba's E MapReduce platform|https://www.alibabacloud.com/help/en/e-mapreduce/latest/use-jindo-job-committer] but it's not available in open source
Bug: Performance improvement in org.apache.hadoop.io.Text#find
Status: Patch Available | Priority: Trivial | Resolution: 
Created: 07/Aug/22 07:18

The current implementation reset src and tgt to the mark and continues searching when tgt has remaining and src expired first. which is probably not necessary.
{code:java}
public int find(String what, int start) {
  try {
    ByteBuffer src = ByteBuffer.wrap(this.bytes, 0, this.length);
    ByteBuffer tgt = encode(what);
    byte b = tgt.get();
    src.position(start);

    while (src.hasRemaining()) {
      if (b == src.get()) { // matching first byte
        src.mark(); // save position in loop
        tgt.mark(); // save position in target
        boolean found = true;
        int pos = src.position()-1;
        while (tgt.hasRemaining()) {
          if (!src.hasRemaining()) { // src expired first
            tgt.reset();
            src.reset();
            found = false;
            break;
          }
          if (!(tgt.get() == src.get())) {
            tgt.reset();
            src.reset();
            found = false;
            break; // no match
          }
        }
        if (found) return pos;
      }
    }
    return -1; // not found
  } catch (CharacterCodingException e) {
    throw new RuntimeException("Should not have happened", e);
  }
} {code}
For example, when q is searched, it is found that src has no remaining, and src is reset to d to continue searching. But the remaining length of src is always smaller than tgt, at this point we can return -1 directly.
{code:java}
@Test
public void testFind() throws Exception {
  Text text = new Text("abcd\u20acbdcd\u20ac");
  assertThat(text.find("cd\u20acq")).isEqualTo(-1);
} {code}
Perhaps it could be:
{code:java}
public int find(String what, int start) {
  try {
    ByteBuffer src = ByteBuffer.wrap(this.bytes, 0, this.length);
    ByteBuffer tgt = encode(what);
    byte b = tgt.get();
    src.position(start);

    while (src.hasRemaining()) {
      if (b == src.get()) { // matching first byte
        src.mark(); // save position in loop
        tgt.mark(); // save position in target
        boolean found = true;
        int pos = src.position()-1;
        while (tgt.hasRemaining()) {
          if (!src.hasRemaining()) { // src expired first
            return -1;
          }
          if (!(tgt.get() == src.get())) {
            tgt.reset();
            src.reset();
            found = false;
            break; // no match
          }
        }
        if (found) return pos;
      }
    }
    return -1; // not found
  } catch (CharacterCodingException e) {
    throw new RuntimeException("Should not have happened", e);
  }
}{code}
Bug: Leak of S3AInstrumentation instances via hadoop Metrics references
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 10/Nov/22 10:19
Affects Version/s: 3.3.4

A heap dump of a process running OOM shows that if a process creates then destroys lots of S3AFS instances, you seem to run out of heap due to references to S3AInstrumentation and the IOStatisticsStore kept via the hadoop metrics registry

It doesn't look like S3AInstrumentation.close() is being invoked in S3AFS.close(). it should -with the IOStats being snapshotted to a local reference before this happens. This allows for stats of a closed fs to be examined.

If you look at org.apache.hadoop.ipc.DecayRpcScheduler.MetricsProxy it uses a WeakReference to refer back to the larger object. we should do the same for abfs/s3a bindings. ideally do some template proxy class in hadoop common they can both use.
Bug: disable purging list of in progress reads in abfs stream closed
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 30/Nov/22 14:27
Affects Version/s: 3.3.2

turn off the prune of in progress reads in ReadBufferManager::purgeBuffersForStream

this will ensure active prefetches for a closed stream complete. they wiill then get to the completed list and hang around until evicted by timeout, but at least prefetching will be safe.

When backporting
* include the followup test fix patch
* include HADOOP-18577.
Bug: AvroFSInput opens a stream twice and discards the second one without closing
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 05/Dec/22 15:55
Affects Version/s: 3.3.5

late breaking blocker for 3.3.5; AvroFsinput can leak input streams because the change of HADOOP-16202 failed to comment out the original open() call.

noticed during a code review with [~harshit.gupta]
Bug: CVE-2021-37533 on commons-net is included in hadoop common and hadoop-client-runtime
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 06/Dec/22 18:28
Affects Version/s: 3.3.4

Latest 3.3.4 version of hadoop-common and hadoop-client-runtime includes commons-net in version 3.6, which has vulnerability CVE-2021-37533. Need to upgrade it to 3.9 to fix. 

This is a due diligence patch only; by the time the caller encounters the CVE they must have already provided their username and password to a malicious ftp server.
Bug: NFS Gateway may release buffer too early
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 12/Dec/22 09:53
Affects Version/s: 3.2.5

After upgrading Netty from 4.1.68 to 4.1.77 (HADOOP-18079), NFS Gateway started crashing when writing data (can be easily reproduced by a few 10MB+ files).  The problem was triggered by [reduced default chunk size in PooledByteBufAllocator|https://github.com/netty/netty/commit/f650303911] (in 4.1.75), but it turned out to be caused by a buffer released too early in NFS Gateway.
Bug: Improve error reporting on non-standard kerberos names
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 14/Dec/22 12:18
Affects Version/s: 3.3.4

The kerberos RFC does not declare any restriction on
characters used in kerberos names, though
implementations MAY be more restrictive.

If the kerberos controller supports use non-conventional
user names *and the kerberos admin chooses to use them*
this can confuse some of the parsing.

The obvious solution is for the enterprise admins to "not do that"
as a lot of things break, bits of hadoop included.

Harden the hadoop code slightly so at least we fail more gracefully,
so people can then get in touch with their sysadmin and tell them
to stop it.

Note: given the kerberos admin is implicitly a superuser, being
able to create malformed principal names.
doesn't give them any privileges, just offers a different way
to stop the cluster working.
Bug: Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 04/Oct/22 14:26
Affects Version/s: 3.3.4

if you use the FileContext APIs to talk to abfs or s3a, it creates a new wrapped FileSystem implementation, and, because there is no close() call, never cleans up.

proposed: add finalizers for these two classes, which we know create helper threads, especially if plugins are added
Bug: ChecksumFileSystem::readVectored might return byte buffers not positioned at 0
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 16/Nov/22 12:36
Affects Version/s: 3.3.5

Checksystem::readVectored method returns the byte buffers that are not positioned at 0, which might be the underlying assumption for the readers like ORC.

 

cc:/ [~mthakur] [~rbalamohan] [~stevel@apache.org]
Bug: hadoop-aws tests to take a configurable subdir in the test bucket
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Oct/22 10:55
Affects Version/s: 3.3.5

For parallel jenkins runs (now possible with s3guard cut) we need to be able to run the failsafe test suites either in separate buckets brackets (which doesn't scale) or in subdirectories of a single bucket. That needs a way to pass in a base directory for the suites and to disable cleanup of the entire bucket afterwards. Option pick up is straightforward; we will do it the way we have done with the others –an auth-keys option which can be overridden on the command line. 

{{S3ATestUtils.createTestPath(Path defVal)}} is already used to create test paths; it can be extended to pick up a unique process id as well as thread id system property, and use that for the path it constructs. 

another system property/config option is needed to disable root dir tests (ITestMarkerToolRootOperations, ITestS3AContractRootDir). a config/property picked up by the S3AContract returning false when probed for "test.root-tests-enabled" should do most of that.
Bug: Über-jira: S3A Hadoop 3.3.9-3.4.1 features
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/22 16:21
Affects Version/s: 3.3.5

Changes related to s3a in the next branch-3.3 release. 
Presence in this list != any commitment to implement, unless there's active dev
Bug: S3A: add option to disable probe for dir marker recreation on delete/rename.
Status: Open | Priority: Major | Resolution: 
Created: 28/Nov/22 17:48
Affects Version/s: 3.3.4

In applications which do many single-file deletions on the same dir, a lot of time is wasted in {{maybeCreateFakeParentDirectory()}}.

Proposed: add an option to disable the probe, for use by applications which are happy for parent dirs to sometimes disappear after a cleanup.

file by file delete is still woefully inefficient because of the HEAD request on every file, but there's no need to amplify the damage.
Bug: hadoop checknative fails to load openssl 3.x
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/22 12:56
Affects Version/s: 3.3.4

After building Hadoop 3.3.4 from source on Ubuntu 22.04, `hadoop checknative` reports
{code:java}
$ hadoop checknative
2022-12-21 22:12:02,106 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2022-12-21 22:12:02,107 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2022-12-21 22:12:02,130 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false EVP_CIPHER_CTX_block_size
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.{code}
The issue seems to be at least two symbols that were removed from ABI in OpenSSL 3.x releases:
 * EVP_CIPHER_CTX_block_size (new name: EVP_CIPHER_CTX_get_block_size)
 * EVP_CIPHER_CTX_encrypting (new name: EVP_CIPHER_CTX_is_encrypting)

The attached patch [^100-hadoop-3.3.4-openssl-3.patch] works around the issue.
Bug: Upgrade ojalgo to  51.4.1
Status: Open | Priority: Major | Resolution: 
Created: 27/Nov/22 16:15
Affects Version/s: 3.4.0

Upgrade ojalgo to  51.4.1 to resolve CWE-327: [Use of a Broken or Risky Cryptographic Algorithm|https://cwe.mitre.org/data/definitions/327.html]
Bug: Make protobuf 2.5 an optional runtime dependency.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/22 15:26
Affects Version/s: 3.3.4

uses of protobuf 2.5 and RpcEnginej have been deprecated since 3.3.0 in HADOOP-17046

while still keeping those files around (for a long time...), how about we make the protobuf 2.5.0 export off hadoop common and hadoop-hdfs *provided*, rather than *compile*

that way, if apps want it for their own apis, they have to explicitly ask for it, but at least our own scans don't break.

i have no idea what will happen to the rest of the stack at this point, it will be "interesting" to see
Bug: Azure Token provider requires tenant and client IDs despite being optional
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/22 12:30
Affects Version/s: 3.3.2

The `AbfsConfiguration` class requires that we provide a tenant and client ID when using the `MsiTokenProvider` class to fetch an authentication token. The bug is that those fields are not required by the Azure API, which can infer those fields when the call is made from an Azure instance.

The fix is to make tenant and client ID optional when getting an Azure token from the Azure Metadata Service.

A fix has been submitted here: [https://github.com/apache/hadoop/pull/4262]

The bug was introduced with HADOOP-17725  ([https://github.com/apache/hadoop/pull/3041/files])
Bug: Upgrade to snakeyaml 1.33
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Oct/22 10:39
Affects Version/s: 3.3.6

Recent snakeyaml fixes missed a use case. Relates to HADOOP-18443

[https://bitbucket.org/snakeyaml/snakeyaml/wiki/Changes]
 * Fix [#553|https://bitbucket.org/snakeyaml/snakeyaml/issues/553/loaderoptionssetcodepointlimit-not-honored]: LoaderOptions.setCodePointLimit() not honored by loadAll() (thanks to Robert Patrick)
Bug: Exclude Dockerfile_windows_10 from hadolint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/22 00:33
Affects Version/s: 3.3.4

HADOOP-18133 tries to add Dockerfile for Windows 10 for building Hadoop. However, hadolint fails to run on *Dockerfile_windows_10* since the version of hadolint (1.1.1) used in Hadoop CI doesn't support parsing of the Windows command syntax.

HADOOP-18449 tries to upgrade the version of hadolint to the latest (2.10.0). However, it runs into some GPG issues on Centos 8.

Thus, we're going to exclude Dockerfile_windows_10 from getting hadolint-ed for the time being. There's a bug in Yetus that prevents exclusion of the file if the exclusion rule and the Dockerfile are added in the same PR - https://github.com/apache/yetus/pull/289#issuecomment-1263813381. Thus, this PR adds the exclusion rule first and then the Dockerfile will be added in another PR.
Bug: Upgrade commons-text version to fix CVE-2022-42889
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/22 07:49
Affects Version/s: 3.2.4

Upgrade commons-text version to ensure downstream applications are not at risk from CVE-2022-42889.

https://nvd.nist.gov/vuln/detail/CVE-2022-42889

The CVE is related to variable expansion through the utility class {{org.apache.commons.text.lookup.StringLookup}}.

# Hadoop does not use this in its codebase, and never has. Therefore it is not at direct risk from this. 
# We are not aware of any uses in its dependent libraries. Assuming this is true, hadoop is not at indirect risk from this.

Applications built using the hadoop libraries may be at risk if they use the class and get their version of commons-text set transitively from the hadoop build. Upgrading the dependency declared by hadoop ensures that these applications are not vulnerable.
Bug: Hadoop metrics should return 0 when there is no change
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/22 06:41
Affects Version/s: 3.4.0

When we try to switch active NN to standby, we find that the getContentSummary average time is always a very high value even if there is no more query. For us, the metrics return 0 is more reasonable. The monitor is as below:

!image-2022-10-21-14-41-43-105.png!
Bug: upgrade woodstox-core to 5.4.0 for security fix
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/22 21:05
Affects Version/s: 3.3.4

Per [issue|https://github.com/FasterXML/woodstox/issues/157], woodstox-core 5.3.0 has security vulnerability and need to upgrade to 5.4.0 for fix.

The Hadoop Configuration classes uses woodstox to parse the XML format (core-site.xml, ...) but

* people don't normally put in DTDs
* the XML format is not the wire format used when applications submit jobs to the yarn resource manager.
* when parsing untrusted XML configuration files in restricted mode (eg. oozie workflows), DTD support is already disabled
Bug: ABFS: Add fs.azure.enable.readahead option to disable readahead
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Nov/22 16:22
Affects Version/s: 3.3.4

Add an option fs.azure.enable.readahead to allow the abfs readahead to be disabled.

This has been shipping in cloudera releases, because workloads with heavy random IO don't always benefit from prefetching.

Pulling this in to ASF hadoop allows for
* everyone to experiment with/without prefetching
* ability to disable prefetching if you have problems with it