av9ash/GitBugs huggingface dataset:
Bug: [JDK 11] Fix javadoc error  in Java API link detection
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/20 07:16
Affects Version/s: 3.4.0

{noformat}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc-no-fork (default-cli) on project hadoop-hdfs-rbf: An error has occurred in Javadoc report generation: 
[ERROR] Exit code: 1 - javadoc: warning - You have specified the HTML version as HTML 4.01 by using the -html4 option.
[ERROR] The default is currently HTML5 and the support for HTML 4.01 will be removed
[ERROR] in a future release. To suppress this warning, please ensure that any HTML constructs
[ERROR] in your comments are valid in HTML5, and remove the -html4 option.
[ERROR] javadoc: error - The code being documented uses modules but the packages defined in https://docs.oracle.com/javase/8/docs/api/ are in the unnamed module.
{noformat}
Bug: Add --mvn-custom-repos parameter to yetus calls
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Aug/20 18:43
Affects Version/s: 3.3.1

In my request PR [#2188|https://github.com/apache/hadoop/pull/2188], I see the QA build fails with unrelated errors.

One example:
{code:java}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.5.1:install (default-install) on project hadoop-yarn-applications-mawo-core: Failed to install metadata org.apache.hadoop.applications.mawo:hadoop-yarn-applications-mawo-core:3.4.0-SNAPSHOT/maven-metadata.xml: Could not read metadata /home/jenkins/.m2/repository/org/apache/hadoop/applications/mawo/hadoop-yarn-applications-mawo-core/3.4.0-SNAPSHOT/maven-metadata-local.xml: input contained no data -> [Help 1]
{code}
Another example:
{code:java}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.5.1:install (default-install) on project hadoop-project: Failed to install metadata org.apache.hadoop:hadoop-project:3.4.0-SNAPSHOT/maven-metadata.xml: Could not parse metadata /home/jenkins/.m2/repository/org/apache/hadoop/hadoop-project/3.4.0-SNAPSHOT/maven-metadata-local.xml: in epilog non whitespace content is not allowed but got n (position: END_TAG seen ...</metadata>\nn... @21:2) -> [Help 1]
{code}
As reported by HBASE-22474 and HBASE-22801, PreCommit validation from yetus uses a shared .m2 repository. By adding {{\-\-mvn-custom-repos-}} paramter, yetus will use a custom .m2 directory for executions for PR validations.

This is a change to mimic that for Hadoop project.

CC: [~aajisaka]
Bug: Move personality file from Yetus to Hadoop repository 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/20 19:24
Affects Version/s: 3.3.1

Currently for CI build and testing we maintain personality scripts (i.e., [here|https://github.com/apache/yetus/blob/master/precommit/src/main/shell/personality/hadoop.sh]) in both Apache Yetus and Apache Hadoop. This poses problem when one needs to change both places, for example HADOOP-17125. 

This proposes to move the personality file into the Hadoop repo itself, so that we can manage them in a single place. The downside for this is we may need to duplicate the scripts in every branch.
Bug: LoadBalanceKMSClientProvider#deleteKey should invalidateCache via all KMSClientProvider instances
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Aug/20 23:09
Affects Version/s: 2.8.4

Without invalidateCache, the deleted key may still exists in the servers' key cache (CachingKeyProvider in KMSWebApp.java)  where the delete key was not hit. Client may still be able to access encrypted files by specifying to connect to KMS instances with a cached version of the deleted key before the cache entry (10 min by default) expired.
Bug: Fix build the hadoop-build Docker image failed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/20 10:23
Affects Version/s: 3.3.1

When I build the docker-build image under macOS, it failed caused by:
{code:java}
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-vKHcWu/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-odL0bY-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-vKHcWu/isort/
You are using pip version 8.1.1, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/bash -o pipefail -c pip2 install     configparser==4.0.2     pylint==1.9.2' returned a non-zero code: 1
{code}
Bug: Allow SSLFactory fallback to input config if ssl-*.xml fail to load from classpath
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Sep/20 22:46
Affects Version/s: 2.8.5

Some applications like Tez does not have ssl-client.xml and ssl-server.xml in classpath. Instead, it directly pass the parsed SSL configuration as the input configuration object. This ticket is opened to allow this case. TEZ-4096 attempts to solve this issue but but take a different approach which may not work in existing Hadoop clients that use SSLFactory from hadoop-common.
Bug: Compile Hadoop on Windows natively
Status: Open | Priority: Major | Resolution: 
Created: 09/Aug/20 14:55
Affects Version/s: 3.1.3

Hadoop compiles flawlessly on Linux whereas, one has to hack around to compile it on Windows. The reason is obvious, there's a very large proportion of C/C++ codebase that's written in a platform specific manner that's amenable to Linux.

Thus, compiling Hadoop on Windows involves using the MinGW or the Cygwin toolchains. This third party dependency might be seen as a risk for those who want to deploy Hadoop on Windows nodes in production. The primary reason for this is that the MinGW/Cygwin runtimes on Windows don't produce core dumps. Whereas the Visual C++ runtime is able to do so, which greatly enables debugging.

Another reason to use the Visual C++ compiler for compiling Hadoop on Windows is that Visual C++ is native to Windows and one will have access to a large collection of tools that can be used for instrumentation, telemetry and fine-tuning the settings of Hadoop to run better on Windows.

Modern C++ has a plethora of APIs that one can use to write safe, cross-platform code, which can be leveraged to refactor the existing Linux specific code, thereby enabling compilation on Windows using the native Visual C++ compiler.
Bug: Support Non-Path Based FileSystem API in Regex Mount Points
Status: Open | Priority: Major | Resolution: 
Created: 06/Sep/20 04:44

Regex mount points create ChoRootedFileSystem while accessing. And we couldn't know the underlying filesystems ahead. This won't work with non-path based APIs such as  getAdditionalTokenIssuers. Instead of totaly unsuport it, we should support APIs to some extend. The could be done recording the FileSystem created and perform APIs for FileSystem instances created for ViewFileSystem.
Bug: Backport HADOOP-13230 list/getFileStatus changes for preserved directory markers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/20 18:54
Affects Version/s: 3.1.3

Our supported Hadoop releases all need to handle s3a stores where the s3a client does not delete directory markers.

This is only the minimal set of list changes; no intent of changing marker retention policy. It should be transparent to all clients

* list/getFileStatus changes
* backwards compatible test of new algorithm
* new tests to verify marker handling
* marker rename bug  (HADOOP-17200) surfacing in testing
Bug: Update jackson-mapper-asl-1.9.13 to atlassian version to mitigate: CVE-2019-10172
Status: Patch Available | Priority: Major | Resolution: 
Created: 25/Aug/20 03:03

Currently jersey depends on the jackson, and upgradation of jersey from 1.X to 2.x looks complicated(see HADOOP-15984 and HADOOP-16485).

Update jackson-mapper-asl-1.9.13 to atlassian version to mitigate: CVE-2019-10172.
Bug: Using snappy-java in SnappyCodec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/20 06:57
Affects Version/s: 3.3.0

In Hadoop, we use native libs for snappy codec which has several disadvantages:
 * It requires native *libhadoop* and *libsnappy* to be installed in system *LD_LIBRARY_PATH*, and they have to be installed separately on each node of the clusters, container images, or local test environments which adds huge complexities from deployment point of view. In some environments, it requires compiling the natives from sources which is non-trivial. Also, this approach is platform dependent; the binary may not work in different platform, so it requires recompilation.
 * It requires extra configuration of *java.library.path* to load the natives, and it results higher application deployment and maintenance cost for users.

Projects such as *Spark* and *Parquet* use [snappy-java|[https://github.com/xerial/snappy-java]] which is JNI-based implementation. It contains native binaries for Linux, Mac, and IBM in jar file, and it can automatically load the native binaries into JVM from jar without any setup. If a native implementation can not be found for a platform, it can fallback to pure-java implementation of snappy based on [aircompressor|[https://github.com/airlift/aircompressor/tree/master/src/main/java/io/airlift/compress/snappy]].
Bug: Intermittent OutOfMemory error while performing hdfs CopyFromLocal to abfs 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/20 11:17
Affects Version/s: 3.3.0

OutOfMemory error due to new ThreadPools being made each time AbfsOutputStream is created. Since threadPool aren't limited a lot of data is loaded in buffer and thus it causes OutOfMemory error.

Possible fixes:
- Limit the number of ThreadCounts while performing hdfs copyFromLocal (Using -t property).
- Reducing OUTPUT_BUFFER_SIZE significantly which would limit the amount of buffer to be loaded in threads.
- Don't create new ThreadPools each time AbfsOutputStream is created and limit the number of ThreadPools each AbfsOutputStream could create.
Bug: JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
Status: Reopened | Priority: Major | Resolution: 
Created: 09/Sep/20 12:00

The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws FileNotFoundException if the src does not exist. However, JavaKeyStoreProvider#renameOrFail calls the old rename API. In DistributedFileSystem, the old API returns false if the src does not exist.

That way JavaKeyStoreProvider fails to create a new key if the keystore is HDFS.
Bug: ABFS: Support for conditional overwrite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/20 06:49
Affects Version/s: 3.3.0

Filesystem Create APIs that do not accept an argument for overwrite flag end up defaulting it to true. 

We are observing that request count of creates with overwrite=true is more and primarily because of the default setting of the flag is true of the called Create API. When a create with overwrite ends up timing out, we have observed that it could lead to race conditions between the first create and retried one running almost parallel.

To avoid this scenario for create with overwrite=true request, ABFS driver will always attempt to create without overwrite. If the create fails due to fileAlreadyPresent, it will resend the request with overwrite=true.
Bug: S3A statistics to support IOStatistics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/20 12:30
Affects Version/s: 3.3.0

S3A to rework statistics with

* API + Implementation split of the interfaces used by subcomponents when reporting stats
* S3A Instrumentation to implement all the interfaces
* streams, etc to all implement IOStatisticsSources and serve to callers
* Add some tracking of durations of remote requests
* RemoteIterator iterators returned from list* calls return list stats for callers to collect; toString will print them too
Bug: Implement Incremental globStatus() -> RemoteIterator class
Status: Open | Priority: Major | Resolution: 
Created: 18/Sep/20 12:44
Affects Version/s: 3.3.0

globStatus/globber is inefficient, not incremental and doesn't provide a way to query any IOStatistics.

Proposed: 
* new IncrementalGlobber class which takes an interface to the file source (+impls for FS and FC) which offer the incremental ops
* use incremental listStatusIncremental calls
* or, if no wildcards, full deep listFiles(recursive)
* collect IOStatistics from source ops, aggregate and serve

Potentially a lot of work...need to look carefully here
Bug: FSDataInput/Output Streams to automatically collect IOStatistics
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 18/Sep/20 12:38
Affects Version/s: 3.3.1

the fs input/output streams to automatically collect stream IO statistics even if the inner classes don't; count invocations and durations

An issue here becomes "how would you aggregate with the inner stream, *efficiently*. Not sure there, except maybe

* have some interface to access an updatable IOStatisticsStore which, if the stream implements, says "here is something you can update"
* wrapper class updates that with counts, durations
* we extend DynamicIOStatistics for each counter/gauge etc to have a factory which can add new entries on demand (new AtomicLong, etc)
* so wrapper classes just update their stats, which  updates existing stats or triggers off an on-demand creation of a new entry

Not so informative in terms of low level details (HTTP/IPC requests and latency, errors, bytes discarded) but would give callers benefits of using the API for HDFS, Ozone, GCS
Bug: implement non-guava Precondition checkNotNull
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/20 21:16

As part In order to replace Guava Preconditions, we need to implement our own versions of the API.
 This Jira is to create {{checkNotNull}} in a new package dubbed {{unguava}}.

 +The plan is as follows+
 * create a new {{package org.apache.hadoop.util.unguava;}}
 * {{create class Validate}}
 * implement  {{package org.apache.hadoop.util.unguava.Validate;}} with the following interface
 ** {{checkNotNull(final T obj)}}
 ** {{checkNotNull(final T reference, final Object errorMessage)}}
 ** {{checkNotNull(final T obj, final String message, final Object... values)}}
 ** {{checkNotNull(final T obj,final Supplier<String> msgSupplier)}}
 * {{guava.preconditions used String.lenientformat which suppressed exceptions caused by string formatting of the exception message . So, in order to avoid changing the behavior, the implementation catches Exceptions triggered by building the message (IllegalFormat, InsufficientArg, NullPointer..etc)}}
 * {{After merging the new class, we can replace guava.Preconditions.checkNotNull by {{unguava.Validate.checkNotNull}}}}
 * We need the change to go into trunk, 3.1, 3.2, and 3.3

 

Similar Jiras will be created to implement checkState, checkArgument, checkIndex
Bug: Fix C/C++ standard warnings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/20 17:06
Affects Version/s: 3.1.3

The C/C++ language standard is not specified in a cross-compiler manner. Even though it's as straight forward as passing *-std* as compiler arguments, not all the values are supported by all the compilers. For example, compilation with the Visual C++ compiler on Windows with *-std=gnu99* flag causes the following warning -
{code:java}
cl : command line warning D9002: ignoring unknown option '-std=gnu99' [Z:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs-examples\hdfs_read.vcxproj] {code}
Thus, we need to use the appropriate flags provided by CMake to specify the C/C++ standards so that it is compiler friendly.
Bug: Support S3 Access Points
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/20 18:46
Affects Version/s: 3.3.0

Improve VPC integration by supporting access points for buckets
https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html

*important*: when backporting, always include as followup patches
* HADOOP-17951 
* HADOOP-18085
Bug: Skip Retry INFO logging on first failover from a proxy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/20 00:21

RetryInvocationHandler logs an INFO level message on every failover except the first. This used to be ideal before when there were only 2 proxies in the FailoverProxyProvider. But if there are more than 2 proxies (as is possible with 3 or more NNs in HA), then there could be more than one failover to find the currently active proxy.

To avoid creating noise in clients logs/ console, RetryInvocationHandler should skip logging once for each proxy.
Bug: update log4j-1.2.17 to atlassian version( To Address: CVE-2019-17571)
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Aug/20 13:14

Currentlly there are no active release under 1.X in log4j and log4j2 is incompatiable to upgrade (see HADOOP-16206 ) for more details.

But following CVE is reported on log4j 1.2.17..I think,we should consider to update to Atlassian([https://mvnrepository.com/artifact/log4j/log4j/1.2.17-atlassian-0.4]) or redhat versions

[https://nvd.nist.gov/vuln/detail/CVE-2019-17571]
Bug: Fix failure of docker image creation due to pip2 install error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/20 22:59
Affects Version/s: 3.1.4

{noformat}
The command '/bin/sh -c pip2 install     configparser==4.0.2     pylint==1.9.2' returned a non-zero code: 1
{noformat}
Bug: Support LZO using aircompressor
Status: Open | Priority: Major | Resolution: 
Created: 13/Jul/20 06:28
Affects Version/s: 3.3.0

LZO codec was removed in HADOOP-4874 because the original LZO binding is GPL which is problematic. However, many legacy data is still compressed by LZO codec, and companies often use vendor's GPL LZO codec in the classpath which might cause GPL contamination.

Presro and ORC-77 use [aircompressor| [https://github.com/airlift/aircompressor]] (Apache V2 licensed) to compress and decompress LZO data. Hadoop can add back LZO support using aircompressor without GPL violation.
Bug: Using lz4-java in Lz4Codec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/20 18:54
Affects Version/s: 3.3.0

In Hadoop, we use native libs for lz4 codec which has several disadvantages:

It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and they have to be installed separately on each node of the clusters, container images, or local test environments which adds huge complexities from deployment point of view. In some environments, it requires compiling the natives from sources which is non-trivial. Also, this approach is platform dependent; the binary may not work in different platform, so it requires recompilation.
It requires extra configuration of java.library.path to load the natives, and it results higher application deployment and maintenance cost for users.
Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which is JNI-based implementation. It contains native binaries in jar file, and it can automatically load the native binaries into JVM from jar without any setup. If a native implementation can not be found for a platform, it can fallback to pure-java implementation of lz4.
Bug: Remove non-inclusive terminology from Hadoop Common
Status: Open | Priority: Major | Resolution: 
Created: 30/Jul/20 18:22

http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/202007.mbox/%3CCAAaVJWVXhsv4tn1KOQkKYTaQ441Yb8y7s%2BR_GnESwduB1iFxOA%40mail.gmail.com%3E

This JIRA is to remove offensive and non-inclusive terminology from Hadoop. The simple ones are whitelist/blacklist and master/slave. However this JIRA can also serve as a place to fix other non-inclusive terminology (e.g., binary gendered
examples, "Alice" doing the wrong security thing systematically).

As [~curino] posted in his email, the IETF has created a draft for proposed alternatives
https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1
Bug: ABFS: Random read perf improvement
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 11:06
Affects Version/s: 3.3.0

Random read if marginally read ahead was seen to improve perf for a TPCH query. 

 

Introducing fs.azure.readahead.range parameter which can be set by user.
Data will be populated in buffer for random reads as well which leads to lesser
remote calls.
This patch also changes the seek implementation to perform a lazy seek. Actual
seek is done when a read is initiated and data is not present in buffer else
date is returned from buffer thus reducing the number of remote calls.
Bug: Backport HADOOP-13230 listing changes for preserved directory markers to 3.1.x
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 25/Aug/20 21:57
Affects Version/s: 3.1.4

Backport a small subset of HADOOP-17199 to branch-3.1

No path capabities, declarative test syntax etc

just

-getFileStatus/list
-markers changes to bucket-info
-startup info message if option is set
-relevant test changes
Bug: Configuration.getValByRegex() shouldn't update the results while fetching.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/20 12:39
Affects Version/s: 3.1.3

We have seen this stacktrace while using ABFS file system. After analysing the stack trace we can see that getValByRegex() is reading the properties and substituting the value in the same call. This may cause the ConcurrentModificationException. 
{code:java}
Caused by: java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1877) ... 18 more Caused by: java.util.ConcurrentModificationException at java.util.Hashtable$Enumerator.next(Hashtable.java:1387) at org.apache.hadoop.conf.Configuration.getValByRegex(Configuration.java:3855) at org.apache.hadoop.fs.azurebfs.AbfsConfiguration.validateStorageAccountKeys(AbfsConfiguration.java:689) at org.apache.hadoop.fs.azurebfs.AbfsConfiguration.<init>(AbfsConfiguration.java:237) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:154) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:113) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3396) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3456) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3424) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:518) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
 


{code}
Bug: Erasure Coding: Native library memory leak
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/20 10:23
Affects Version/s: 3.1.3

We use both {{apache-hadoop-3.1.3}} and {{CDH-6.1.1-1.cdh6.1.1.p0.875250}} HDFS in production, and both of them have the memory increasing over {{-Xmx}} value. 

!image-2020-08-15-18-26-44-744.png!

 

We use EC strategy to to save storage costs.

This's the jvm options:
{code:java}
-Dproc_datanode -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true -Xms8589934592 -Xmx8589934592 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError ...{code}
The max jvm heapsize is 8GB, but we can see the datanode RSS memory is 48g. All the other datanodes in this hdfs cluster has the same issue.
{code:java}
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
226044 hdfs 20 0 50.6g 48g 4780 S 90.5 77.0 14728:27 /usr/java/jdk1.8.0_162/bin/java -Dproc_datanode{code}
 

This too much memory used leads to my machine unresponsive(if enable swap), or oom-killer happens.