av9ash/GitBugs huggingface dataset:
Bug: "Reversed (or previously applied) patch detected" for hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch 
Status: Open | Priority: Minor | Resolution: 
Created: 20/Nov/23 22:00
Affects Version/s: 3.4.0

tried to compile under Hadoop-common-project and noticed the following. 

trunk: commit f609460bda0c2bd87dd3580158e549e2f34f14d5
{code:java}
$ mvn clean package -Pdist,docs -DskipTests -Dtar
patching file 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
Reversed (or previously applied) patch detected!  Assume -R? [y]
1 out of 2 hunks failed--saving rejects to 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
patching file 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
Reversed (or previously applied) patch detected!  Assume -R? [y]
patching file 'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java{code}
also reproducible by 
{code:java}
xinglin@xinglin-mn2 ~/p/h/trunk (trunk)> patch -p1 < hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
patching file 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
Reversed (or previously applied) patch detected!  Assume -R? [y]
1 out of 2 hunks failed--saving rejects to 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
patching file 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
Reversed (or previously applied) patch detected!  Assume -R? [y]
patching file 'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java'{code}
Build succeeded but the jdiff-workaround patch probably needs a fix.
Bug: SingleCluster.html instructions are ordered incorrectly
Status: Open | Priority: Minor | Resolution: 
Created: 27/Nov/23 12:32
Affects Version/s: 3.3.6

I was going over the pesudo distributed cluster instructions at 

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation

The smoke test instructions direct you to execute a mapreduce job

{noformat}
  $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output 'dfs[a-z.]+'
{noformat}

but at this point Yarn hasn't been started, the instructions for that are in the next section.

Switch the YARN setup section and the smoke test section.
Bug: Bug in SaslPropertiesResolver allows mutation of internal state
Status: Patch Available | Priority: Minor | Resolution: 
Created: 13/Nov/23 19:59

When {{SaslDataTransferServer}} or {{SaslDataTranferClient}} want to get a SASL properties map to do a handshake, they call {{SaslPropertiesResolver#getServerProperties()}} or {{SaslPropertiesResolver#getClientProperties()}}, and they get back a {{Map<String, String>}}. Every call gets the same {{Map}} object back, and then the callers sometimes call [put()|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java#L385] on it. This means that future users of {{SaslPropertiesResolver}} get back the wrong information.

I propose that {{SaslPropertiesResolver}} should pass a copy of its internal map, so that users can safety modify them.

I discovered this problem in my company's testing environment as we began to enable {{dfs.data.transfer.protection}} on our DataNodes, while our NameNodes were using {{IngressPortBasedResolver}} to give out block tokens with different QOPs depending on the port used. Then our HDFS client applications became unable to read or write to HDFS because they could not find a QOP in common with the DataNodes during SASL handshake. With multiple threads executing SASL handshakes at the same time, the properties map used in {{SaslDataTransferServer}} in a DataNode could be clobbered during usage, since the same map was used by all threads. Also, future clients that do not have a QOP embedded in their block tokens would connect to a server with the wrong SASL properties map. I think that one or both of these issues explains the problem that I saw. I eliminated this unsafety and saw the problem go away.
Bug: s3a openfile is using readahead rather than async drain threshold to calculate drain threshold
Status: Open | Priority: Minor | Resolution: 
Created: 18/Oct/23 09:59
Affects Version/s: 3.3.6

we are using the wrong default value for the async drain threshold in   {{org.apache.hadoop.fs.s3a.impl.OpenFileSupport}}


{code}
       builderSupport.getPositiveLong(ASYNC_DRAIN_THRESHOLD,
                defaultReadAhead))

{code}

this means the drain threshold isn't being taken up in random io unless you have a very small readahead
Bug: Add journalnode maintenance node list
Status: Resolved | Priority: Trivial | Resolution: Abandoned
Created: 13/Oct/23 09:19
Affects Version/s: 3.3.6

* In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes are available and 1 journal node fails to start due to machine issues, it will result in a long initialization time for the namenode (around 30-40 minutes, depending on the IPC timeout and retry policy configuration). 
* The failed journal node cannot recover immediately, but HDFS can still function in this situation. In our production environment, we encountered this issue and had to reduce the IPC timeout and adjust the retry policy to accelerate the namenode initialization and provide services. 
* I'm wondering if it would be possible to have a journal node maintenance list to speed up the namenode initialization knowing that one journal node cannot provide services in advance?
Bug: Unrecognized SSL message error in LDAPGroupMappings
Status: Open | Priority: Critical | Resolution: 
Created: 18/Nov/24 03:11
Affects Version/s: 3.4.1

h3. What Happened: 

Got an unrecognized SSL message error instead of the expected LDAP response read timeout when hadoop.security.group.mapping.ldap.ssl is set to true. 
h3. Buggy Code: 

 
{code:java}
try (ServerSocket serverSock = new ServerSocket(0)) { // -> ServerSocket is not configured to accept SSL communication.
  final CountDownLatch finLatch = new CountDownLatch(1);

  final Thread ldapServer = new Thread(new Runnable() {
    @Override
    public void run() {
      try {
        try (Socket clientSock = serverSock.accept()) {
          IOUtils.skipFully(clientSock.getInputStream(), 1);
          clientSock.getOutputStream().write(AUTHENTICATE_SUCCESS_MSG);
          finLatch.await();
        }
      } catch (Exception e) {
        e.printStackTrace();
      }
    }
  });
  ldapServer.start(); {code}
 
h3. Stack Trace: 

 
{code:java}
Expected to find 'LDAP response read timed out, timeout used' but got unexpected exception: javax.naming.CommunicationException: localhost:36143 [Root exception is javax.net.ssl.SSLException: Unsupported or unrecognized SSL message]
        at java.naming/com.sun.jndi.ldap.Connection.<init>(Connection.java:250)
        at java.naming/com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137)
        at java.naming/com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1616)
        at java.naming/com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2847)
        at java.naming/com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:348)
        at java.naming/com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxFromUrl(LdapCtxFactory.java:266)
        at java.naming/com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:226)
        at java.naming/com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:284)
        at java.naming/com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:185)
        at java.naming/com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:115)
        at java.naming/javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:730)
        at java.naming/javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:305)
        at java.naming/javax.naming.InitialContext.init(InitialContext.java:236)
        at java.naming/javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154)
        at org.apache.hadoop.security.TestLdapGroupsMappingBase$DummyLdapCtxFactory.getInitialContext(TestLdapGroupsMappingBase.java:241)
{code}
h3. How to Reproduce: 

(1) Set hadoop.security.group.mapping.ldap.ssl to true 

(2) Run test: org.apache.hadoop.security.TestLdapGroupsMapping#testLdapReadTimeout
h3. Notes: 

I don't have a patch/fix for this yet, I am working on it.
Bug: Null Pointer Exception in KeyProviderCryptoExtension due to Class Not Found Exception
Status: Open | Priority: Critical | Resolution: 
Created: 16/Nov/24 23:29
Affects Version/s: 3.4.0

h3. What Happened: 

A null pointer exception occurs in KeyProviderExtension when trying to close a null CryptoCodec object. If supplied with an invalid class name for hadoop.security.crypto.codec.classes.aes.ctr.nopadding getClassbyName throws a ClassNotFound exception and consequently the CryptoCodec object is not created. 
h3. Buggy Code: 
{code:java}
CryptoCodec cc = CryptoCodec.getInstance(keyProvider.getConf()); // -> this does not initialize cc due to a ClassNotFound exception. 
try {
  final byte[] newKey = new byte[encryptionKey.getMaterial().length];
  cc.generateSecureRandom(newKey);
  final byte[] iv = new byte[cc.getCipherSuite().getAlgorithmBlockSize()];
  cc.generateSecureRandom(iv);
  Encryptor encryptor = cc.createEncryptor();
  return generateEncryptedKey(encryptor, encryptionKey, newKey, iv);
} finally {
  cc.close(); // -> this throws a NPE as cc is null
} {code}
h3. Stack Trace: 
{code:java}
java.lang.NullPointerException
        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension$DefaultCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:303)
        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:513)
        at org.apache.hadoop.crypto.key.TestKeyProviderCryptoExtension.testReencryptEncryptedKeys(TestKeyProviderCryptoExtension.java:229)
 {code}
h3. How to Reproduce: 

(1) Set hadoop.security.crypto.codec.classes.aes.ctr.nopadding to org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec/

(2) Run test: org.apache.hadoop.crypto.key.TestKeyProviderCryptoExtension#testReencryptEncryptedKeys
Bug: FastSaslClientFactory failing to initialise due to NPE
Status: Open | Priority: Critical | Resolution: 
Created: 11/Oct/24 08:44
Affects Version/s: 3.3.6

FastSaslClientFactory is instantiated in both SaslRpcClient and SaslParticipant as new FastSaslClientFactory(null)

In its constructor FastSaslClientFactory loads all the SaslFactories using Sasl.getSaslClientFactories();

Then it iterates through all the loaded factories and calls getMechanismNames method.

The null argument sent in the constructor is sent to the getMechanismNames call.

In some-cases, a loaded factory might not handle this null argument and might throw a NPE.

This can cause the entire FastSaslClientFactory to not instantiate leading to failure while instantiation of SaslRpcClient.

Code Pointer - [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/FastSaslClientFactory.java#L43]

Stack Trace - 
{code:java}
Cause3: java.lang.ExceptionInInitializerError: Exception java.lang.NullPointerException [in thread "ServiceScheduler:GridforceSfdcLogUploaderProcess"] Cause3-StackTrace:   at org.apache.qpid.client.security.amqplain.AmqPlainSaslClientFactory.getMechanismNames(AmqPlainSaslClientFactory.java:50)  at org.apache.hadoop.security.FastSaslClientFactory.<init>(FastSaslClientFactory.java:47)  at org.apache.hadoop.security.SaslRpcClient.<clinit>(SaslRpcClient.java:106)  at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:559)  at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:347)  at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:783)  at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:779)  at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)  at java.base/javax.security.auth.Subject.doAs(Subject.java:439)  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)  at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:779)  at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:347)  at org.apache.hadoop.ipc.Client.getConnection(Client.java:1632)  at org.apache.hadoop.ipc.Client.call(Client.java:1457)  at org.apache.hadoop.ipc.Client.call(Client.java:1410)  at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258)  at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139)  at jdk.proxy2/jdk.proxy2.$Proxy1500.getFileInfo(Unknown Source)  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.lambda$getFileInfo$41(ClientNamenodeProtocolTranslatorPB.java:811)  at org.apache.hadoop.ipc.internal.ShadedProtobufHelper.ipc(ShadedProtobufHelper.java:160)  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:811)  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)  ... 23 shared with parent` {code}
Bug: VectorIO API to support releasing buffers on failure
Status: Open | Priority: Major | Resolution: 
Created: 08/Oct/24 09:31
Affects Version/s: 3.4.1

extend for vector IO API with a method that takes a ByteBufferPool implementation rather than just an allocator. This allows for buffers to be returned to the pool when problems occur, before throwing an exception.

The Parquet API is already designed for this
Bug: S3A: Add initial support for analytics-accelerator-s3
Status: Open | Priority: Major | Resolution: 
Created: 26/Nov/24 15:31
Affects Version/s: 3.4.2

S3 recently released [Analytics Accelerator Library for Amazon S3|https://github.com/awslabs/analytics-accelerator-s3] as an Alpha release, which is an input stream, with an initial goal of improving performance for Apache Spark workloads on Parquet datasets. 

For example, it implements optimisations such as footer prefetching, and so avoids the multiple GETS S3AInputStream currently makes for the footer bytes and PageIndex structures.

The library also tracks columns currently being read by a query using the parquet metadata, and then prefetches these bytes when parquet files with the same schema are opened. 

This ticket tracks the work required for the basic initial integration. There is still more work to be done, such as VectoredIO support etc, which we will identify and follow up with.
Bug: S3A: InputStreams to be created by factory under S3AStore
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/24 14:11
Affects Version/s: 3.4.2

Migrate S3AInputStream creation into a factory pattern, push down into S3AStore.

Proposed factories
* default: whatever this release has as default
* classic: current S3AInputStream
* prefetch: prefetching
* analytics: new analytics stream
* other: reads a classname from another prop, instantiates.

Also proposed
* stream to implement some stream capability to declare what they are (classic, prefetch, analytics, other). 

h2. Implementation

All callbacks used by the stream also to call directly onto S3AStore.
S3AFileSystem must not be invoked at all (if it is needed: PR is still not ready).
Some interface from Instrumentation will be passed to factory; this shall include a way to create new per-stream 
The factory shall implement org.apache.hadoop.service.Service; S3AStore shall do same and become a subclass of CompositeService. It shall attach the factory as a child, so they can follow the same lifecycle. We shall do the same for anything else that gets pushed down.

Everything related to stream creation must go from s3afs; and creation of the factory itself. This must be done in S3AStore.initialize(). 

As usual, this will complicate mocking. But the streams themselves should not require changes, at least significant ones.

Testing.
* The huge file tests should be tuned so each of the different ones uses a different stream, always.
* use a -Dstream="factory name" to choose factory, rather than the -Dprefetch
* if not set, whatever is in auth-keys gets picked up.
Bug: [ABFS] Implement Backoff and Read Footer metrics using IOStatistics Class
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/24 14:12
Affects Version/s: 3.4.1

Current Flow: We have implemented metrics collection in ABFS flow. We have created a custom AbfsBackoffMetrics and AbfsReadFooterMetrics class which stores all the metrics on the file system level. Our objective is to move away from the custom class implementation and use IOStatisticsStore to store the metrics which is present in hadoop-common.

Changes Made: This PR contains the changes related to storing metrics related to above mentioned classes in IOStatisticStore which is present in hadoop-common. AbstractAbfsStatisticsSource abstract class is created which is implementing IOStatisticsSource interface. This will store IOStatistics of the child metrics class.

Both AbfsBackoffMetrics and AbfsReadFooterMetrics is inheriting AbstractAbfsStatisticsSource and store the respective metrics in IOStatisticsStore.
Bug: S3A: Test failures after CSE support added
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/24 13:04
Affects Version/s: 3.5.0

Testing hadoop trunk with CSE-KMS configured I get

* 400 error without region set (KMS providing nothing helpful). Proposed: move troubleshooting into encryption.md, cover 400 and this as a possible cuse

* test failures
{code}
[ERROR]   ITestS3AClientSideEncryptionKms>ITestS3AClientSideEncryption.testSizeOfEncryptedObjectFromHeaderWithV1Compatibility:345->ITestS3AClientSideEncryption.assertFileLength:447 [Length of s3a://stevel-london/job-00-fork-0009/test/testSizeOfEncryptedObjectFromHeaderWithV1Compatibility/file status: S3AFileStatus{path=s3a://stevel-london/job-00-fork-0009/test/testSizeOfEncryptedObjectFromHeaderWithV1Compatibility/file; isDirectory=false; length=1024; replication=1; blocksize=33554432; modification_time=1731674289000; access_time=0; owner=stevel; group=stevel; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag="0f343b0931126a20f133d67c2b018a3b" versionId=JyA1I_OW8osQTS3zWdn_Z0qlQYqBZ_7.] expected:<10[]L> but was:<10[24]L>
[ERROR]   ITestAwsSdkWorkarounds.testNoisyLogging:99 [LOG output does not contain the forbidden text. Has the SDK been fixed?] 
Expecting:
 <"">
to contain:
 <"The provided S3AsyncClient is an instance of MultipartS3AsyncClient"> 
[ERROR] Errors: 
[ERROR]   ITestUploadRecovery.testCommitOperations:234 » AWSClientIO upload part #1 uplo...
[ERROR]   ITestUploadRecovery.testMagicWriteRecovery[array-commit-true] » AWSClientIO up...
[ERROR]   ITestUploadRecovery.testMagicWriteRecovery[bytebuffer-commit-false] » AWSClientIO
[ERROR]   ITestUploadRecovery.testMagicWriteRecovery[disk-commit-false] » AWSClientIO up...
{code}
Bug: S3A Analytics-Accelerator: Add IoStatistics support
Status: Open | Priority: Major | Resolution: 
Created: 13/Dec/24 10:57

S3A provides InputStream statistics: [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/statistics/S3AInputStreamStatistics.java]

This helps track things like how many bytes were read from a stream etc. 

 

The current integration does not currently implement statistics. To start off with we should identify which of these statistics makes sense for us track in the new stream. Some examples are:

 

1/ bytesRead

2/ readOperationStarted

3/ initiateGetRequest

 

Some of these (1 and 2) are more straightforward, and should not require any changes to analytics-accelerator-s3, but tracking GET requests will require this. 

We should also add tests that make assertions on these statistics. See 
ITestS3APrefetchingInputStream for an example to do this. 

And see https://issues.apache.org/jira/browse/HADOOP-18190 for how this was done on the prefetching stream, and PR: https://github.com/apache/hadoop/pull/4458
Bug: Bump avro from 1.9.2 to 1.11.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Oct/24 15:21
Affects Version/s: 3.4.0



* All field access is now via setter/getter methods
* To use Avro to marshal Serializable objects,
  the packages they are in must be declared in the system property {{org.apache.avro.SERIALIZABLE_PACKAGES}}
  
This is required to address
* [CVE-2024-47561|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-47561]
* [CVE-2023-39410|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-39410]

This change is not backwards compatible.
Bug: AliyunOSS ：Optimize the process by reducing some extra QPS.
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/24 10:14

AliyunOSSConnector has not been updated for a long time. It implements basic functionality,  but its performance is not optimal, especially in terms of QPS requests. It generates a  significant number of invalid QPS, which increases the cost on the storage service side and  also reduces the efficiency on the connector side.
Bug: Support analytics-accelerator-s3 input streams for parquet read performance
Status: Open | Priority: Major | Resolution: 
Created: 13/Dec/24 08:11
Affects Version/s: 3.5.0

S3 recently released [Analytics Accelerator Library for Amazon S3|https://github.com/awslabs/analytics-accelerator-s3] as an Alpha release, which is an input stream, with an initial goal of improving performance for Apache Spark workloads on Parquet datasets. 

 

This tracks stabilisation work of this integration into S3A.
Bug: S3A: Add LeakReporter; use in S3AInputStream
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Nov/24 15:44
Affects Version/s: 3.4.1

A recurring problem is that applications forget to close their input streams; eventually the HTTP connection runs out.

Having the finalizer close streams during GC will ensure that after a GC the http connections are returned. While this is an improvement on today, it is insufficient
* only happens during GC, so may not fix problem entirely
* doesn't let developers know things are going wrong.
* doesn't let us differentiate well between stream leak and overloaded FS

proposed enhancements then
* collect stack trace in constructor
* log in finalize at warn including path, thread and stack
* have special log for this, so it can be turned off in production (libraries telling end users off for developer errors is simply an annoyance)

h2. Leak Reporting

* the log for  leak reporting is {{org.apache.hadoop.fs.resource.leaks}}
* An error message is reported at WARN, including the file name.
* A stack trace of where the stream was created is reported
  at INFO.
* A best-effort attempt is made to release any active HTTPS
  connection.
* The filesystem IOStatistic stream_leaks is incremented.

The intent is to make it easier to identify where streams
are being opened and not closed -as these consume resources
including often HTTPS connections from the connection pool
of limited size.

It MUST NOT be relied on as a way to clean up open
files/streams automatically; some of the normal actions of
the close() method are omitted.