av9ash/GitBugs huggingface dataset:
Bug: Bad ipc.client.connection.idle-scan-interval.ms cause resource leaks
Status: Open | Priority: Critical | Resolution: 
Created: 12/Jul/23 02:55

When setting ipc.client.connection.idle-scan-interval.ms to a bad value (e.g. a negative value), Hadoop Server fails to schedule the idle connection scan task and causes resource leaks.

h2. Buggy code:
{code:java}
private void scheduleIdleScanTask() {
  ...
  TimerTask idleScanTask = new TimerTask(){
    @Override
    public void run() {
      ...
      try {
        closeIdle(false);
      } finally {
        // explicitly reschedule so next execution occurs relative
        // to the end of this scan, not the beginning
        scheduleIdleScanTask();
      }
    }
  };
  idleScanTimer.schedule(idleScanTask, idleScanInterval);   // <--- idleScanInterval is a negative value
}
{code}

In schedule, the task will not be scheduled if the delay is negative, which causes resource leaks due to unscheduled idleScanTask.
{code:java}
public void schedule(TimerTask task, long delay) {
    if (delay < 0)
        throw new IllegalArgumentException("Negative delay.");
    sched(task, System.currentTimeMillis()+delay, 0);        // <-- the task will not be scheduled when delay is negative
}
{code}

h2. How to reproduce:
We can use the test org.apache.hadoop.ipc.TestIPC#testSocketLeak to check the resource leaks.
(1) Set ipc.client.connection.idle-scan-interval.ms to -1;
(2) Run test org.apache.hadoop.ipc.TestIPC#testSocketLeak
(3) You will see the following message (note that the number of leaked descriptors can vary from run to run):
{code}
java.lang.AssertionError: Leaked 142 file descriptors
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.hadoop.ipc.TestIPC.testSocketLeak(TestIPC.java:1155)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.lang.Thread.run(Thread.java:829)
{code}

You can use the reproduce.sh in the attachment to easily reproduce the bug:

We are happy to provide a patch if this issue is confirmed.
Bug: unnecessary NullPointerException encountered when starting HttpServer2 with prometheus enabled 
Status: Open | Priority: Critical | Resolution: 
Created: 20/Jul/23 08:13
Affects Version/s: 3.3.3

h2. What happened?

Attempt to start an {{HttpServer2}} failed due to an NPE thrown in {{{}MetricsSystemImpl{}}}.
h2. Where's the bug?

In line 1278 of {{{}HttpServer2{}}}, if the support for prometheus is enabled the server registers a prometheus sink:
{noformat}
        if (prometheusSupport) {
          DefaultMetricsSystem.instance()
              .register("prometheus", "Hadoop metrics prometheus exporter",
                  prometheusMetricsSink);
        }{noformat}
However, a problem is that if the MetricsSystemImpl returned by the DefaultMetricsSystem.instance has not been start nor init, the config of the metric system would be set to null, thus failing the nullity check at the start of MetricsSystemImpl.registerSink. A better way of handling this would be to check in advance if the metric system has been initialized and initialize it if it has not been initialized.
h2. How to reproduce?

(1) set hadoop.prometheus.endpoint.enabled to true

(2) run org.apache.hadoop.http.TestHttpServer#testHttpResonseContainsDeny
h2. Stacktrace
{noformat}
java.io.IOException: Problem starting http server
        ...
Caused by: java.lang.NullPointerException: config
    at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:899)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSink(MetricsSystemImpl.java:298)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:277)
    at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1279)
    ... 34 more{noformat}
For an easy reproduction, run the reproduce.sh in the attachment.

We are happy to provide a patch if this issue is confirmed.
Bug: ZKDTSM could be stuck when meet znode version overflow
Status: Open | Priority: Critical | Resolution: 
Created: 03/Sep/23 16:21

ZKDTSM could be stuck when meet znode (/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot) version int overflow (2147483647). It can not recovery even restart Application which may include YARN Router, DFS Router, KMS and other modules who use zookeeper to manage Token. One solution (not very smooth) is delete this znode first and then restart Service.

The root cause is following code snippet and curator could not compatible with version overflow. I try to give a draft improvement at CURATOR-688. Welcome to any discussion if we could resolve it at Hadoop side smooth.

org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager#incrSharedCount

{code:java}
  private int incrSharedCount(SharedCount sharedCount, int batchSize)
      throws Exception {
    while (true) {
      // Loop until we successfully increment the counter
      VersionedValue<Integer> versionedValue = sharedCount.getVersionedValue();
      if (sharedCount.trySetCount(
          versionedValue, versionedValue.getValue() + batchSize)) {
        return versionedValue.getValue();
      }
    }
  }
{code}
Bug: Buggy ZKFCRpcServer constructor creates null object and crashes the rpcServer
Status: Open | Priority: Critical | Resolution: 
Created: 19/Jul/23 16:44

h2. What happened:

In ZKFailoverController.java, initRPC() function gets ZKFC RpcServer binding address and create a new ZKFCRpcServer object rpcServer. However rpcServer may be null when the ZKFCRpcServer constructor accepts a null policy provider and cause any later rpcServer usage a null pointer exception.
h2. Buggy code:

In ZKFailoverController.java
{code:java}
protected void initRPC() throws IOException {
  InetSocketAddress bindAddr = getRpcAddressToBindTo();
  LOG.info("ZKFC RpcServer binding to {}", bindAddr);
  rpcServer = new ZKFCRpcServer(conf, bindAddr, this, getPolicyProvider());  // <-- Here getpolicyProvider might be null
}
{code}
ZKFCRpcServer() eventually calls refreshWithLoadedConfiguration() function below. This function directly use provider without check null and this turns out making rpcServer above to be a null object.

In ServiceAuthorizationManager.java
{code:java}
  @Private
  public void refreshWithLoadedConfiguration(Configuration conf, PolicyProvider provider) {
    ...
    // Parse the config file
    Service[] services = provider.getServices();   // <--- provider might be null here
    ... {code}
h2. How to trigger this bug:

(1) Set hadoop.security.authorization to true

(2) Run test org.apache.hadoop.ha.TestZKFailoverControllerStress#testRandomExpirations

(3) You will see the following stack trace:
{code:java}
java.lang.NullPointerException                                                          
        at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:258)                                                                                                              
        at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:63)      
        at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:181)                                                                                                              
        at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:177)                                                                                                              
        at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:503)                                                                                                         
        at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:177)            
        at org.apache.hadoop.ha.MiniZKFCCluster$DummyZKFCThread.doWork(MiniZKFCCluster.java:301)   
        at org.apache.hadoop.test.MultithreadedTestUtil$TestingThread.run(MultithreadedTestUtil.java:189){code}
(4) The null pointer exception here is due to the null {{rpcServer}} object caused by the bug described above.

You can use the reproduce.sh in the attachment to easily reproduce the bug:

We are happy to provide a patch if this issue is confirmed.
Bug: Out of Memory when mistakenly set io.file.buffer.size to a large number
Status: Open | Priority: Critical | Resolution: 
Created: 14/Jul/23 14:26

h2. What happened:

When setting io.file.buffer.size to a large number, BufferedIOStatisticsOutputStream in Hcommon throws an out-of-memory exception due to inappropriate checking and handling.
The config is used to initialize a file system by passing it as one of the parameters bufferSize.
h2. Buggy code:

In RawLocalFileSystem.java
{code:java}
private FSDataOutputStream create(Path f, boolean overwrite,
      boolean createParent, int bufferSize, short replication, long blockSize,
      Progressable progress, FsPermission permission) throws IOException {
  ...
  return new FSDataOutputStream(new BufferedIOStatisticsOutputStream(
    createOutputStreamWithMode(f, false, permission), bufferSize, true), <<--- creates a BufferedIOStatisticsOutputStream with bufferSize, often set to config io.file.buffer.size
    statistics);
} {code}
In BufferedIOStatisticsOutputStream.java:
{code:java}
public class BufferedIOStatisticsOutputStream extends BufferedOutputStream
  implements IOStatisticsSource, Syncable, StreamCapabilities {
  ...
  public BufferedIOStatisticsOutputStream(
      final OutputStream out,
      final int size,
      final boolean downgradeSyncable) {
    super(out, size); <<--- init the BufferedOutputStream with a huge buffer size
    ...
  }{code}
h2. StackTrace:
{code:java}
java.lang.OutOfMemoryError: Java heap space
        at java.base/java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:75)
        at org.apache.hadoop.fs.statistics.BufferedIOStatisticsOutputStream.<init>(BufferedIOStatisticsOutputSt
ream.java:78)
        at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:428)
        at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:413)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
        at org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:183)
        at org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:152)
        at org.apache.hadoop.fs.contract.AbstractContractRenameTest.expectRenameUnderFileFails(AbstractContract
RenameTest.java:335)
...{code}
h2. Reproduce:

(1) Set io.file.buffer.size to a large value, e.g., 2112001717
(2) Run a simple test that exercises this parameter, e.g. org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename#testRenameFileUnderFile
Bug: S3A: AWS SDK V2 Migration: stabilization and S3Express
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Sep/23 13:35
Affects Version/s: 3.4.0

The final stabilisation changes to the V2 SDK MIgration; those moved off the HADOOP-18073 JIRA so we can close that.

also adds support to Amazon S3 Express One Zone storage
Bug: [JDK17] MiniYarnClusters don't launch in hadoop-aws integration tests
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 23/Aug/23 10:46
Affects Version/s: 3.4.0

I've tried running hadoop-aws tests under java17; everything which tries to launch a MiniYarnCluster fails because google guice is trying to stuff in java.land module that is now forbidden

{code}
Caused by: java.lang.ExceptionInInitializerError: Exception com.google.inject.internal.cglib.core.$CodeGenerationException: java.lang.reflect.InaccessibleObjectException-->Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @7ee7980d [in thread "Thread-109"]

{code}

short term fix is to add the params to the surefire and failsafe jvm launcher to allow access

{code}
--add-opens java.base/java.lang=ALL-UNNAMED

{code}

I don't know if updating guice will make it go away completely. if it doesn't then the history server itself needs to be launched with this

rather than just add an option for hadoop-aws, we ought to consider a general cross-module variable for junit.jvm.options which is set everywhere; the base impl is "" and a java profile could add the new stuff
Bug: add options to disable range merging of vectored io
Status: Resolved | Priority: Major | Resolution: Done
Created: 18/Aug/23 14:15
Affects Version/s: 3.3.5

I'm seeing test failures in my PARQUET-2171 pr because assertions about the #of bytes read isn't holding -small files are being read and the vector range merging is pulling in the whole file.

```
[ERROR]   TestInputOutputFormat.testReadWriteWithCounter:338 bytestotal != bytesread expected:<5510> but was:<11020>
```

I think for parquet i will add an option to disable vector io, but really the filesystems which support it should allow for merging to be disabled
Bug: Add ability to configure ConnectionTTL of http connections while creating S3 Client.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Aug/23 17:54
Affects Version/s: 3.3.6

The option fs.s3a.connection.ttl sets the maximum time an idle connection may be retained in the http connection pool. 

A lower value: fewer connections kept open, networking problems related to long-lived connections less likely
A higher value: less time spent negotiating TLS connections when new connections are needed
Bug: AWS SDK V2 - Move to S3 Java async client
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/23 08:49
Affects Version/s: 3.4.0

With the upgrade, S3A now has two S3 clients the Java async client and the Java sync client.

Java async is required for the transfer manager.

Java sync is used for everything else. 

 

* Move all operations to use the Java async client and remove the sync client. 

* Provide option to configure java async client with the CRT HTTP client. 

* Create a new interface for S3Client operations, move them out of S3AFS. interface will take request and span, and return response.
Bug: Add a way to get the IOStatistics of active filesystems in long-lived processes
Status: Open | Priority: Major | Resolution: 
Created: 08/Aug/23 17:27
Affects Version/s: 3.3.9

we can configure abfs and s3a to print out the filesystem IOStats in close(), but this doesn't let us see the state of long-lived processes, including latencies, error rates, and other important information.

Proposed: we add a means by which all cached filesystems which provide IOStatistics can log them to a file/console *while the process continues to run*


* a configurable scheduled thread which enumerates all open filesystem instances, gets their iostats and if non empty dumps to a log...log configuration can set that to the console or elsewhere. Good: can run without intervention; bad: one more thread to manage.
* option to add a signal handler which will do this whenever a SIGUSR2 is raised. JVM signal handling is a bit of a brittle feature, so it should be off by default.
{{org.apache.hadoop.service.launcher.IrqHandler}} supports signal wire-up. good: gives us an equivalent of kill -quit to get a view of iostats. bad: you need to be on the machine
Bug: remove/deprecate fs.s3a.multipart.purge
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/23 16:36
Affects Version/s: 3.3.6

the fs.s3a.multipart.purge option has been in for a long time, to help avoid running up costs from incomplete uploads, especially during testing.

but
* adds overhead on startup (list + delete)
* dangerous if one client has a very short lifespan; all active s3a committer jobs will be broken
* obsoleted by s3 lifecycle rules
* and "hadoop s3guard uploads" cli.

proposed: deprecate for the next release, delete after.
Bug: Support Overwrite Directory On Commit For S3A Committers
Status: Open | Priority: Major | Resolution: 
Created: 07/Aug/23 12:28
Affects Version/s: 3.4.0

The goal is to add a new kind of commit mechanism in which the destination directory is cleared off before committing the file.

*Use Case*

In case of dynamicPartition insert overwrite queries, The destination directory which needs to be overwritten are not known before the execution and hence it becomes a challenge to clear off the destination directory.

 

One approach to handle this is, The underlying engines/client will clear off all the destination directories before calling the commitJob operation but the issue with this approach is that, In case of failures while committing the files, We might end up with the whole of previous data being deleted making the recovery process difficult or time consuming.

 

*Solution*

Based on mode of commit operation either *INSERT* or *OVERWRITE* , During commitJob operations, The committer will map each destination directory with the commits which needs to be added in the directory and if the mode is *OVERWRITE* , The committer will delete the directory recursively and then commit each of the files in the directory. So in case of failures (worst case) The number of destination directory which will be deleted will be equal to the number of threads if we do it in multi-threaded way as compared to the whole data if it was done in the engine side.
Bug: [ABFS] Support VectorIO in ABFS Input Stream
Status: Open | Priority: Major | Resolution: 
Created: 07/Sep/23 16:39
Affects Version/s: 3.3.9

the hadoop vector IO APIs are supported in file;// and s3a://; there's a hive ORC patch for this and PARQUET-2171 adds it for parquet -after which all apps using the library with a matching hadoop version and the feature enabled will get a significant speedup.

abfs needs to support too, which needs support for parallel GET requests for different ranges
Bug: ABFS: Misreporting Retry Count for Sub-sequential and Parallel Operations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/23 07:26
Affects Version/s: 3.3.6

There was a bug identified where retry count in the client correlation id was wrongly reported for sub-sequential and parallel operations triggered by a single file system call. This was due to reusing same tracing context for all such calls.
We create a new tracing context as soon as HDFS call comes. We keep on passing that same TC for all the client calls.

For instance, when we get a createFile call, we first call metadata operations. If those metadata operations somehow succeeded after a few retries, the tracing context will have that many retry count in it. Now when actual call for create is made, same retry count will be used to construct the headers(clientCorrelationId). Alhough the create operation never failed, we will still see retry count from the previous request.

Fix is to use a new tracing context object for all the network calls made. All the sub-sequential and parallel operations will have same primary request Id to correlate them, yet they will have their own tracing of retry count.
Bug: ABFS: Adding Server returned request id in Exception method thrown to caller.
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 30/Aug/23 12:34

Each request made to Azure server has its unique ActivityId (rid) which is returned in response of the request whether is succeed or fails.
When a HDFS call fails due to an error from Azure service, An ABFSRestOperationException is throws to the caller. This task is to add a server returned activity id (rid) in the exception message which can be used to investigate the failure on service side.
Bug: ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport 
Status: Resolved | Priority: Major | Resolution: Done
Created: 21/Sep/23 05:12

Azure Storage Supports Content-MD5 Request Headers in Both Read and Append APIs.
Read: [Path - Read - REST API (Azure Storage Services) | Microsoft Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read]
Append: [Path - Update - REST API (Azure Storage Services) | Microsoft Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update]

This change is to make client-side changes to support them. In Read request, we will send the appropriate header in response to which server will return the MD5 Hash of the data it sends back. On Client we will tally this with the MD5 hash computed from the data received.

In Append request, we will compute the MD5 Hash of the data that we are sending to the server and specify that in appropriate header. Server on finding that header will tally this with the MD5 hash it will compute on the data received. 

This whole Checksum Validation Support is guarded behind a config, Config is by default disabled because with the use of "https" integrity of data is preserved anyways. This is introduced as an additional data integrity check which will have a performance impact as well.

Users can decide if they want to enable this or not by setting the following config to *"true"* or *"false"* respectively. *Config: "fs.azure.enable.checksum.validation"*
Bug: ABFS: Fixing Behavior of a File System APIs on root path
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 29/Aug/23 11:45
Affects Version/s: 3.3.6

Following HDFS Apis are failing when called on a root path.

{*}{*}{*}{*}{*}{*}
|FS Call|Status|Error thrown to caller|
|create()|Failing|Operation failed: "The request URI is invalid.", 400, PUT, https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-02076119-21ed-4ada-bcd0-14afaae54013/?resource=file&timeout=90, InvalidUri, "The request URI is invalid. RequestId:1d23f8c2-d01f-0059-61b6-c60c24000000 Time:2023-08-04T09:29:55.4813818Z"|
|createNonRecursive()|Failing|Runtime Exception: java.lang.IllegalArgumentException: null path (This is occuring because getParentPath is null and getFileStatus is called on null)|
|setXAttr()|Failing|Operation failed: "The request URI is invalid.", 400, HEAD, https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-491399b3-c3d0-4568-9d4a-a26e0aa8f000/?upn=false&timeout=90|
|getXAttr()|Failing|Operation failed: "The request URI is invalid.", 400, HEAD, https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-491399b3-c3d0-4568-9d4a-a26e0aa8f000/?upn=false&timeout=91|

h2. important: xattr support was removed in HADOOP-19089; include that change when cherrypicking this