av9ash/GitBugs huggingface dataset:
Bug: Upgrade Kafka to 2.8.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/21 08:48
Affects Version/s: 3.2.3

Kafka 2.4.0 has the following vulnerability.

[https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-38153]
Bug: lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Sep/21 21:33
Affects Version/s: 3.3.1

lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g. hadoop-client-api, if we don't exclude lz4 dependency, the downstream will still see the exception even they include lz4 dependency.
{code:java}
[info]   Cause: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
[info]   at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
[info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
[info]   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
[info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
[info]   at org.apache.hadoop.io.compress.lz4.Lz4Compressor.<init>(Lz4Compressor.java:66)
[info]   at org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
[info]   at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
[info]   at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
 {code}

Currently snappy-java is included and relocated in Hadoop shaded client libraries. But as it includes native methods, it should not be relocated too due to JNI method resolution. The downstream will see the exception:

{code}
[info]   Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
[info]   at org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native Method)                                                                                                 
[info]   at org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)                                                                                                        
[info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
[info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)

{code}
Bug: Test Result Not Working In Jenkins Result
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/21 04:32
Affects Version/s: 3.4.0

The jenkins used to show the Test Results, now the list is broken and No PR has that working.

eg:

[https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/]

 

For Daily Build No Test Result as well:

https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/
Bug: Upgrade ant to 1.10.11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/21 19:33
Affects Version/s: 2.10.2

Vulnerabilities reported in org.apache.ant:ant:1.10.9
 * [CVE-2021-36374|https://nvd.nist.gov/vuln/detail/CVE-2021-36374] moderate severity
 * [CVE-2021-36373|https://nvd.nist.gov/vuln/detail/CVE-2021-36373] moderate severity

suggested: org.apache.ant:ant ~> 1.10.11
Bug: Upgrade JSON smart to 2.4.7
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Aug/21 10:06
Affects Version/s: 3.2.3

Currently we are using JSON Smart 2.4.2 version which is vulnerable to - CVE-2021-31684.

We can upgrade the version to 2.4.7 (2.4.5 or later).
Bug: Upgrade jetty version to 9.4.43
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/21 04:34
Affects Version/s: 3.2.2

https://github.com/eclipse/jetty.project/security/advisories/GHSA-m6cp-vxjx-65j6
https://github.com/eclipse/jetty.project/security/advisories/GHSA-gwcr-j4wh-j3cq
Bug: Modify Text.ensureCapacity() to efficiently max out the backing array size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/21 11:06
Affects Version/s: 3.4.0

This is a continuation of HADOOP-17901.

Right now we use a factor of 1.5x to increase the byte array if it's full. However, if the size reaches a certain point, the increment is only (current size + length). This can cause performance issues if the textual data which we intend to store is beyond this point.

Instead, let's max out the array to the maximum. Based on different sources, a safe choice seems to be Integer.MAX_VALUE - 8 (see ArrayList, AbstractCollection, HashTable, etc).
Bug: Move ClusterStorageCapacityExceededException to Public from LimitedPrivate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/21 10:24
Affects Version/s: 3.4.0

As of now the exception is marked limited private
{code:java}
@InterfaceAudience.LimitedPrivate({ "HDFS", "MapReduce", "Tez" })
{code}
Doesn't allow other projects, Rather than individually adding project, Make it Public itself.

This exception can be used to act as a fail-fast marker for different operations.
Bug: Check real user ACLs in addition to proxied user ACLs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Aug/21 18:34
Affects Version/s: 2.10.1

In a secure cluster, it is possible to configure the services to allow a super-user to proxy to a regular user and perform actions on behalf of the proxied user (see [Proxy user - Superusers Acting On Behalf Of Other Users|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html]).

This is useful for automating server access for multiple different users in a multi-tenant cluster. For example, this can be used by a super user submitting jobs to a YARN queue, accessing HDFS files, scheduling Oozie workflows, etc, which will then execute the service as the proxied user.

Usually when these services check ACLs to determine if the user has access to the requested resources, the service only needs to check the ACLs for the proxied user. However, it is sometimes desirable to allow the proxied user to have access to the resources when only the real user has open ACLs.

For instance, let's say the user {{adm}} is the only user with submit ACLs to the {{dataload}} queue, and the {{adm}} user wants to submit apps to the {{dataload}} queue on behalf of users {{headless1}} and {{headless2}}. In addition, we want to be able to bill {{headless1}} and {{headless2}} separately for the YARN resources used in the {{dataload}} queue. In order to do this, the apps need to run in the {{dataload}} queue as the respective headless users. We could open up the ACLs to the {{dataload}} queue to allow {{headless1}} and {{headless2}} to submit apps. But this would allow those users to submit any app to that queue, and not be limited to just the data loading apps, and we don't trust the {{headless1}} and {{headless2}} owners to honor that restriction.

This JIRA proposes that we define a way to set up ACLs to restrict a resource's access to a  super-user, but when the access happens, run it as the proxied user.
Bug: hadoop-aws landsat-pds test bucket will be deleted after Jul 1, 2021
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 01/Jul/21 04:55

I found an anouncement that landsat-pds buket will be deleted on July 1, 2021

(https://registry.opendata.aws/landsat-8/)

and  I think this bucket  is used in th test of hadoop-aws module use

[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java#L93]

 

At this time I can access the bucket but we might have to change the test bucket in someday.
Bug: Provide fallbacks for callqueue.impl and scheduler.impl
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/21 16:27
Affects Version/s: 3.4.0

As mentioned in parent Jira, we should provide default properties for callqueue.impl and scheduler.impl such that if properties with port is not configured, we can fallback to default property. If "ipc.8020.callqueue.impl" is not present, fallback property could be "ipc.callqueue.impl" (without port). We can take up rest of the callqueue properties in separate sub-tasks.
Bug: Checkstyle - Allow line length: 100
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/21 02:45
Affects Version/s: 3.3.2

Update the checkstyle rule to allow for 100 or 120 characters.

Discussion thread: [https://lists.apache.org/thread.html/r69c363fb365d4cfdec44433e7f6ec7d7eb3505067c2fcb793765068f%40%3Ccommon-dev.hadoop.apache.org%3E]
Bug: Use CuratorCache implementation instead of PathChildrenCache / TreeCache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Aug/21 09:45
Affects Version/s: 3.4.0

As we have moved to Curator 5.2.0 for Hadoop 3.4.0, we should start using new CuratorCache service implementation in place of deprecated PathChildrenCache and TreeCache usecases.
Bug: Avoid possible class loading deadlock with VerifierNone initialization
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/21 08:17
Affects Version/s: 3.3.2

Superclass Verifier has a static initializer VERIFIER_NONE that initializes sub-class VerifierNone. This reference can result in deadlock during class loading as per [https://docs.oracle.com/javase/specs/jls/se8/html/jls-12.html#jls-12.4.2].

As of today, only RpcProgram use this instance and hence it is safe but if more clients start using this (specifically static ones), it has potential to bring deadlock. We should break this referencing before it is late.
Bug: ExceptionsHandler to add terse/suppressed Exceptions in thread-safe manner
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/21 19:46
Affects Version/s: 3.3.2

Even though we have explicit comments stating that we have thread-safe replacement of terseExceptions and suppressedExceptions, in reality we don't have it. As we can't guarantee only non-concurrent addition of Exceptions at a time from any Server implementation, we should make this thread-safe.
Bug: Provide alternative to Guava VisibleForTesting
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/21 09:54
Affects Version/s: 3.3.2

In an attempt to reduce the dependency on Guava, we should remove VisibleForTesting annotation usages as it has very high usage in our codebase. This Jira is to provide Hadoop's own alternative and use it in hadoop-common-project modules.
Bug: ABFS: Support for Encryption Context
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/21 10:55
Affects Version/s: 3.3.1

Support for customer-provided encryption keys at the file level, superceding the global (account-level) key use in HADOOP-17536.

ABFS driver will support an "EncryptionContext" plugin for retrieving encryption information, the implementation for which should be provided by the client. The keys/context retrieved will be sent via request headers to the server, which will store the encryption context. Subsequent REST calls to server that access data/user metadata of the file will require fetching the encryption context through a GetFileProperties call and retrieving the key from the custom provider, before sending the request.
Bug: Improve the GitHub pull request template
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/21 01:47

The current Hadoop pull request template can be improved.

- Require some information (e.g. https://github.com/apache/spark/blob/master/.github/PULL_REQUEST_TEMPLATE)
- Checklists (e.g. https://github.com/apache/nifi/blob/main/.github/PULL_REQUEST_TEMPLATE.md)
- Move current notice to comment (i.e. surround with <!-- and  -->)
Bug: improve YARN Registry DNS Server qps
Status: Open | Priority: Major | Resolution: 
Created: 24/Aug/21 06:55
Affects Version/s: 3.1.2

There are some points to improve the performance of YARN Registry DNS Server.

- Do not print unnecessary logs (It just needs change log4j.properties)
-- log4j.logger.org.apache.hadoop.registry.server.dns=WARN
- Change some loglevels at points which can affect performance degradation.
- Use "newFixedThreadPool" instead of "newCachedThreadPool" to prevent OOM
- Use Blocking on TCP handler. Using non-blocking and sleeping some time("Thread.sleep(500)") is meaningless.

In our environment, QPS of original yarn dns server is about 5000 (UDP), 100 (TCP).
Now, QPS of our improved yarn dns server is about 47000 (UDP), 500 (TCP).


I will make a pull request at https://github.com/apache/hadoop soon.
Bug: fs.s3a.acl.default not working after S3A Audit feature added
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/21 09:58
Affects Version/s: 3.3.2


After HADOOP-17511 the fs.s3a.acl.default propperty isn't being passed through to S3 PUT/COPY requests.

The new RequestFactory is being given the acl values from the S3A FS instance, but the factory is being created before the acl settings are loaded from the configuration.

Fix, and ideally, test (if the getXAttr lets us see this now)
Bug: S3AInstrumentation Closing output stream statistics while data is still marked as pending upload in OutputStreamStatistics
Status: Open | Priority: Major | Resolution: 
Created: 13/Aug/21 13:24
Affects Version/s: 3.2.1

When using hadoop s3a file upload for spark event Logs, the logs were queued up and not uploaded before the process is shut down:
{code:java}
// 21/08/13 12:22:39 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
21/08/13 12:22:39 WARN S3AInstrumentation: Closing output stream statistics while data is still marked as pending upload in OutputStreamStatistics{blocksSubmitted=1, blocksInQueue=1, blocksActive=0, blockUploadsCompleted=0, blockUploadsFailed=0, bytesPendingUpload=106716, bytesUploaded=0, blocksAllocated=1, blocksReleased=1, blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0, transferDuration=0 ms, queueDuration=0 ms, averageQueueTime=0 ms, totalUploadDuration=0 ms, effectiveBandwidth=0.0 bytes/s}{code}
details see logs attached
Bug: Upgrade third party protobuf-java-2.5.0.jar to address vulnerabilities #CVE-2015-5237, CVE-2019-15544,
Status: Open | Priority: Major | Resolution: 
Created: 23/Aug/21 11:31

Third party jar protobuf-java-2.5.0.jar reports vulnerabilities # CVE-2015-5237, CVE-2019-15544 and need to be upgraded.

CVE-2019-15544：

Vulnerability Description：An issue was discovered in the protobuf crate before 2.6.0 for Rust. Attackers can exhaust all memory via Vec::reserve calls.

CVE-2015-5237：

Vulnerability Description：protobuf allows remote authenticated attackers to cause a heap-based buffer overflow.

 

Please review  and let me know if you have any concerns or would like to add more details to upgrade.
Bug: Add ListWithIOStats<T> wrapper to return IOStats from a list.
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 15/Jul/21 17:28
Affects Version/s: 3.3.1

Just as RemoteIterators now adds the ability to add an IOStats result, it'd be handy to do the same for a java.util.List. This makes it possible to return IO stats from static methods which return them
Bug: No error message reported when bucket doesn't exist in S3AFS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/21 07:28
Affects Version/s: 3.3.1

No error message is shown at INFO level, when the bucket trying to access, doesn't exist. 

While translating the exception when the bucket doesn't exist(error code: NoSuchBucket, status code: 404), we are setting the "path" as the error message rather than "message", the actual error message. 

for eg: 
{code:java}
❯ bin/hadoop fs -ls s3a://mmt-no-bucket/
2021-07-14 12:58:26,169 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-07-14 12:58:26,392 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2021-07-14 12:58:26,448 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2021-07-14 12:58:26,448 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
ls: s3a://mmt-no-bucket/
2021-07-14 12:58:27,712 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
2021-07-14 12:58:27,713 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
2021-07-14 12:58:27,713 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete. {code}
Bug: Always use GitHub PR rather than JIRA to review patches
Status: Resolved | Priority: Major | Resolution: Done
Created: 14/Jul/21 01:32

Now there are 2 types of precommit jobs in https://ci-hadoop.apache.org/
(1) Precommit-(HADOOP|HDFS|MAPREDUCE|YARN)-Build jobs that try to download patches from JIRA and test them.
(2) hadoop-multibranch job for GitHub PR

The problems are:
- The build configs are separated. The (2) config is in Jenkinsfile, and the (1) configs are in the Jenkins. When we update Jenkinsfile, I had to manually update the configs of the 4 precommit jobs via Jenkins Web UI.
- The (1) build configs are static. We cannot use separate config for each branch. This may cause some build failures.
- GitHub Actions cannot be used in the (1) jobs.

Therefore I want to disable the (1) jobs and always use GitHub PR to review patches.

How to do this:

1. Update the wiki: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute#HowToContribute-Provideapatch
2. Disable the Precommit-(HADOOP|HDFS|MAPREDUCE|YARN)-Build jobs.
Bug: ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/21 04:46
Affects Version/s: 3.3.1

Hadoop yetus run shows a java compiler warning for unchecked casting of Object to Iterator<FileStatus> in a method of AbfsListStatusRemoteIterator class. This can be resolved by introducing a new class to hold the iterator and exception thrown when applicable.

 

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsListStatusRemoteIterator.java:97:38:[unchecked] unchecked cast

 

This task will also add logging to facilitate better investigation of the transient failure tracked in HADOOP-17797.
Bug: NullPointerException when no HTTP response set on AbfsRestOperation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/21 20:45
Affects Version/s: 3.3.1

Seen when running HBase 2.2 on top of ABFS with Hadoop 3.1ish:
{noformat}
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.fs.azurebfs.services.AbfsClient.renameIdempotencyCheckOp(AbfsClient.java:382)
        at org.apache.hadoop.fs.azurebfs.services.AbfsClient.renamePath(AbfsClient.java:348)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.rename(AzureBlobFileSystemStore.java:722)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.rename(AzureBlobFileSystem.java:327)
        at org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:249)
        at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.rename(HRegionFileSystem.java:1115) {noformat}
Digging in, it looks like the {{AbfsHttpOperation}} inside of {{AbfsRestOperation}} may sometimes be null, but {{AbfsClient}} will try to unwrap it (and read the status code from the HTTP call). I'm not sure why we sometimes _get_ the null HttpOperation but it seems pretty straightforward to not get the NPE.

HBase got wedged after this, but I'm not sure if it's because of this NPE or (perhaps) we weren't getting any responses from ABFS itself (i.e. there was some ABFS outage/unavailability or the node itself couldn't talk to ABFS).
Bug: ITestS3AContractDistCp failing IllegalStateException: WFOPENSSL0030 Running handshake with buffered unwrapped data
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 27/Jul/21 12:28
Affects Version/s: 3.3.2

Test failure against s3 london with ddb for s3guard.: IllegalStateException: WFOPENSSL0030 Running handshake with buffered unwrapped data

This is on a system with openssl (LibreSSL 2.8.3) , wildfly possibly using it
Bug: s3a: set fs.s3a.downgrade.syncable.exceptions = true by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/21 17:15
Affects Version/s: 3.3.1

HADOOP-17597 set policy of reacting to hsync() on an s3 output stream to be one of :Fail, warn, with default == fail.

I propose downgrading this to warn. We've done it internally, after having it on fail long enough to identify which processes were doing either of
* having unrealistic expectations about the output stream (fix: move off s3)
* were using hflush() as a variant of flush(), with the failure being an over-reaction
Bug: Lookup old S3 encryption configs for JCEKS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/21 10:35
Affects Version/s: 3.4.0

HADOOP-17871 introduces new set of S3 encryption configs which are replaced by old property names during look-up. We need to look-up for both the properties since either could be set in a JCEKS file.
Bug: JsonSerialization raises EOFException reading JSON data stored on google GCS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/21 10:39
Affects Version/s: 3.3.1

The JsonSerialization<> load code doesn't work on gcs as it uses "stream.available()" to fail with a meaningful message if the stream is empty.

But that method is meant to say how much data is available without blocking, something we actually get wrong ourselves. Google GCS team didn't get it wrong, so on a read(), if there's no local buffer, an EOFException is raised
Bug: Distcp file length comparison have no effect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/21 06:14
Affects Version/s: 3.3.1

the params for compareFileLengthsAndChecksums in RetriableFileCopyCommand have no effect

current is
{code:java}
        DistCpUtils.compareFileLengthsAndChecksums(source.getLen(), sourceFS,
                sourcePath, sourceChecksum, targetFS,
                targetPath, skipCrc, source.getLen());{code}
{code:java}
public static void compareFileLengthsAndChecksums(long srcLen,
           FileSystem sourceFS, Path source, FileChecksum sourceChecksum,
           FileSystem targetFS, Path target, boolean skipCrc,
           long targetLen) throws IOException {
  if (srcLen != targetLen) {
    throw new IOException(
        DistCpConstants.LENGTH_MISMATCH_ERROR_MSG + source + " (" + srcLen
            + ") and target:" + target + " (" + targetLen + ")");
  }
{code}
so, compare source.getLen() with source.getLen()...

It should be like below in history view
{code:java}
        DistCpUtils.compareFileLengthsAndChecksums(source.getLen(), sourceFS,
                sourcePath, sourceChecksum, targetFS,
                targetPath, skipCrc, offset + bytesRead);
{code}
Bug: Upgrade JSON smart to 1.3.3 on branch-2.10
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/21 18:54
Affects Version/s: 2.10.0

Currently branch-2.10 is using JSON Smart 1.3.1 version which is vulnerable to [link CVE-2021-27568|https://nvd.nist.gov/vuln/detail/CVE-2021-27568].

We can upgrade the version to 1.3.1.

+Description of the vulnerability:+

{quote}An issue was discovered in netplex json-smart-v1 through 2015-10-23 and json-smart-v2 through 2.4. An exception is thrown from a function, but it is not caught, as demonstrated by NumberFormatException. When it is not caught, it may cause programs using the library to crash or expose sensitive information.{quote}