av9ash/GitBugs huggingface dataset:
Bug: StagingCommitter getFinalKey method can add an extra / if getS3KeyPrefix returns ""
Status: Open | Priority: Minor | Resolution: 
Created: 28/Jan/22 00:00
Affects Version/s: 3.3.1

I am trying to test staging committer against an on prem object store using spark terasort and ran into this issue. All my initiate MPU were failing with S3 error key not found. This object store doesn't support virtual host style request, so I had path style enabled. After adding some extra debug and building hadoop-aws locally, I found that staging committer was always adding a '/' prefix to my key.  

 

So instead of part part-r-00000-4ead11c8-bc20-4dee-9753-1b1f1ae4e578 I would end up with /part-r-00000-4ead11c8-bc20-4dee-9753-1b1f1ae4e578. I traced it to getFinalKey in StagingCommitter.java which had the following code

 
 *      return getS3KeyPrefix(context) + "/"
-          + Paths.addUUID(relative, getUUID());

If getS3KeyPrefix(context) is "", then we end up with /part-r... as the key. 

 

I made the following change locally and was able to resolve the issue

 

 

---

diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/StagingCommitter.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/StagingCommitter.java
index 59114f7ab73..6d76cf2d419 100644
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/StagingCommitter.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/StagingCommitter.java
@@ -365,11 +365,16 @@ public Path getTempTaskAttemptPath(TaskAttemptContext context) {
    * @return the S3 key where the file will be uploaded
    */
   protected String getFinalKey(String relative, JobContext context) {
+    StringBuilder sb = new StringBuilder();
+    final String pfx = getS3KeyPrefix(context);
+    if (!pfx.isEmpty()) {
+        sb.append(pfx).append('/');
+    }
+
     if (uniqueFilenames) {
-      return getS3KeyPrefix(context) + "/"
-          + Paths.addUUID(relative, getUUID());
+        return sb.append(Paths.addUUID(relative, getUUID())).toString(); 
     } else {
-      return getS3KeyPrefix(context) + "/" + relative;
+        return sb.append(relative).toString();
     }
   }
Bug: ExecutorHelper.logThrowableFromAfterExecute() is too noisy. 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jan/22 09:16
Affects Version/s: 3.3.1

{code:java}
if (t == null && r instanceof Future<?> && ((Future<?>) r).isDone()) {
  try {
    ((Future<?>) r).get();
  } catch (ExecutionException ee) {
    LOG.warn(
        "Execution exception when running task in " + Thread.currentThread()
            .getName());
    t = ee.getCause();
  } catch (InterruptedException ie) {
    LOG.warn("Thread (" + Thread.currentThread() + ") interrupted: ", ie);
    Thread.currentThread().interrupt();
  } catch (Throwable throwable) {
    t = throwable;
  }
}

if (t != null) {
  LOG.warn("Caught exception in thread " + Thread
      .currentThread().getName() + ": ", t);
} {code}
We should downgrade the logging here from warn to debug.

 

CC [~stevel@apache.org]  [~mehakmeetSingh]
Bug: Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 24/Jan/22 06:02
Affects Version/s: 2.10.2

The following exception handling code section in testFileStatusOnMountLink() 

 
{code:java}
    try {
      fcView.getFileStatus(new Path("/danglingLink"));
      Assert.fail("Excepted a not found exception here");
    } catch ( FileNotFoundException e) {
      // as excepted
    }{code}
can be replaced with a single in the @Test 
{code:java}
@Test(expected = FileNotFoundException.class){code}
Bug: Update default value of hadoop.ssl.enabled.protocols in the EncryptedShuffle doc
Status: Resolved | Priority: Trivial | Resolution: Duplicate
Created: 24/Feb/22 05:57
Affects Version/s: 2.8.4

The default value of hadoop.ssl.enabled.protocols was updated several times but our doc was never updated.

https://hadoop.apache.org/docs/r3.1.3/hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html
Bug: Refactor tests in TestFileUtil
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Mar/22 17:02
Affects Version/s: 3.4.0

We need to ensure that we check the results of file operations whenever we invoke *mkdir*, *deleteFile* etc. and assert them right there before proceeding on. Also, we need to ensure that some of the relevant FileSystem APIs don't return null.
Bug: Documentation Syntax Error Fix > AWS Assumed Roles
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 04/Feb/22 04:14

Configuration property "fs.s3a.assumed.role.credentials.provider" was mis-documented without the "f" in front as "s.s3a.assumed.role.credentials.provider"

PR already exists and will update the title to reflect this ticket.
https://github.com/apache/hadoop/pull/3949
Bug: EvaluatingStatisticsMap::entrySet may not need parallelstream
Status: Open | Priority: Trivial | Resolution: 
Created: 04/Feb/22 05:46

When large number of S3AInputStreams are opened, it ends up showing up in profile, as internally parallelstream makes use of fork and join. If it is not mandatory, it can be refactored to get rid of parallelstream. Here is the relevant profile output for ref.

  !Screenshot 2022-02-04 at 11.10.39 AM.png|width=632,height=429!
Bug: replace URL.toString() with URL.toURI().toString()
Status: Resolved | Priority: Trivial | Resolution: Invalid
Created: 04/Jan/22 18:26
Affects Version/s: 3.3.1

Some parts of the code, mostly logging and exception messages, call URL.toString().
They should use URL.toURI().toString() to avoid network IO.

This is straightforward to do in the explicit and implicit invocations in our code. It will not detect/fix the case where a URL instance is passed as an argument to logger methods. That will have to be out of scope
Bug: Add some Abortable.abort() interface for streams etc which can be terminated
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 04/Mar/20 20:22
Affects Version/s: 3.3.0

Some IO we want to be able to abort rather than close cleanly, especially if the inner stream is an HTTP connection which itself supports some abort() method. For example: uploads to an object where we want to cancel the upload without close() making an incomplete write visible.

Proposed: Add a generic interface which things like streams can implement
{code}
AbortableIO {
  public void abortIO() throws IOE;
}
{code}

+do for s3a output stream. I wouldn't do this a passthrough on FSDataOutputStream because we need to consider what expectations callers have of an operation being "aborted"
Bug: Update dependency in branch-3.1
Status: Resolved | Priority: Blocker | Resolution: Won't Fix
Created: 11/Mar/20 04:11
Affects Version/s: 3.1.4

Jackson-databind 2.9.10.3 --> 2.10.3
Zookeeper 3.4.13 --> 3.4.14
hbase-client 1.2.6 --> 1.2.6.1
aws-java-sdk-bundle 1.11.271 --> 1.11.563? (this is the version used by trunk)
Bug: Increase entropy to improve cryptographic randomness on precommit Linux VMs
Status: In Progress | Priority: Blocker | Resolution: 
Created: 17/Jan/20 15:05

I was investigating a JUnit test (MAPREDUCE-7079 :TestMRIntermediateDataEncryption is failing in precommit builds) that was consistently hanging on Linux VMs and failing Mapreduce pre-builds.
I found that the test hangs slows or hangs indefinitely whenever Java reads the random file.

I explored two different ways to get that test case to work properly on my local Linux VM running rel7:
# To install "haveged" and "rng-tools" on the virtual machine running Rel7. Then, start rngd service {{sudo service rngd start}} . This will fix the problem for all the components on the image including java, native and any other component.
# Change java configuration to load urandom
{code:bash}
sudo vim $JAVA_HOME/jre/lib/security/java.security
## Change the line “securerandom.source=file:/dev/random” to read: securerandom.source=file:/dev/./urandom
{code}

The first solution is better because this will fix the problem for everything that requires SSL/TLS or other services that depend upon encryption.

Since the precommit build runs on Docker, then it would be best to mount {{/dev/urandom}} from the host as {{/dev/random}} into the container:

{code:java}
docker run -v /dev/urandom:/dev/random
{code}

For Yetus, we need to add the mount to the {{DOCKER_EXTRAARGS}} as follows:

{code:java}
DOCKER_EXTRAARGS+=("-v" "/dev/urandom:/dev/random")
{code}

 ...
Bug: Bug in widely-used helper function caused valid configuration value to fail on multiple tests, causing build failure
Status: Patch Available | Priority: Blocker | Resolution: 
Created: 02/Feb/20 07:48
Affects Version/s: 3.2.1

{code:java}
org.apache.hadoop.io.file.tfile.TestTFileStreams#testOneEntryMixedLengths1
org.apache.hadoop.io.file.tfile.TestTFileStreams#testOneEntryUnknownLength
org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsStreams#testOneEntryMixedLengths1
org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsStreams#testOneEntryUnknownLength{code}
 

4 actively-used tests above call the helper function `TestTFileStreams#writeRecords()` to write key-value pairs (kv pairs), then call `TestTFileByteArrays#readRecords()` to assert the key and the value part (v) of these kv pairs matched with what they wrote. All v of kv pairs are hardcode strings with a length of 6.

 

`readRecords()` uses `org.apache.hadoop.io.file.tfile.TFile.Reader.Scanner.Entry#getValueLength()` to get full length of the v of these kv pairs.  But `getValueLength()` can only get v's full length when it is less than the value of configuration parameter `tfile.io.chunk.size`, otherwise `readRecords()` will throw an exception. So, *when `tfile.io.chunk.size` is configured/set to a value less than 6, these 4 tests failed because of the exception from `readRecords()`, even 6 is a valid value for `tfile.io.chunk.size`.*

The definition of `tfile.io.chunk.size` is "Value chunk size in bytes. Default to 1MB. Values of the length less than the chunk size is guaranteed to have known value length in read time (See also TFile.Reader.Scanner.Entry.isValueLengthKnown())". 

*Fixes*

`readRecords()` should call `org.apache.hadoop.io.file.tfile.TFile.Reader.Scanner.Entry#getValue(byte[])` instead, which returns the correct full length of the `value` part despite whether the value's length is larger than `tfile.io.chunk.size`.
Bug: Update Netty to 4.1.46Final
Status: Resolved | Priority: Blocker | Resolution: Duplicate
Created: 04/Mar/20 23:20

https://nvd.nist.gov/vuln/detail/CVE-2019-20444
{quote}
HttpObjectDecoder.java in Netty before 4.1.44 allows an HTTP header that lacks a colon, which might be interpreted as a separate header with an incorrect syntax, or might be interpreted as an "invalid fold."
{quote}
Bug: Upgrade jackson-databind to 2.9.10.3
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 27/Feb/20 19:49

New [RCE|https://nvd.nist.gov/vuln/detail/CVE-2020-8840] found in jackson-databind 2.0.0 through 2.9.10.2.

Patched in 2.9.10.3. [Looks critical|https://github.com/jas502n/CVE-2020-8840/blob/master/Poc.java#L13].

After HADOOP-16882 get in we should backport this to those lower-version branches ASAP.
Bug: Update jackson-databind to 2.9.10.2 in branch-3.1, branch-2.10
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 25/Feb/20 13:57

We updated jackson-databind multiple times but those changes only made into trunk and branch-3.2.

Unless the dependency update is backward incompatible (which is not in this case), we should update them in all active branches
Bug: Hadoop resource manager jdk 8 dependency
Status: Resolved | Priority: Blocker | Resolution: Not A Bug
Created: 03/Jan/20 16:35
Affects Version/s: 2.10.0

When I am trying to create a single cluster Hadoop environment, I cannot load the Node Manager or Resource Manager. From the log I get the error:

Caused by: java.lang.ClassNotFoundException: javax.activation.DataSourceCaused by: java.lang.ClassNotFoundException: javax.activation.DataSource at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)

... 83 more

I gather this is from a dependency on javax.activation from JDK8. The only online solution I could find involved changing these two lines in the yarn-env.sh file in the usr/local/Cellar/hadoop/\{version}/libexec/etc/hadoop folder:

export YARN_RESOURCEMANAGER_OPTS="--add-modules=ALL-SYSTEM"

export YARN_NODEMANAGER_OPTS="--add-modules=ALL-SYSTEM"

This did not work and I continue to get the same error in the log when trying to run the YARN manager. Can anyone advise as to how to get around this dependency?
Bug: pylint fails in the build environment
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 30/Mar/20 17:01

Hadoop-side of YETUS-960
{noformat}
$ ./start-build-env.sh
(snip)
aajisaka@21a025f08d49:~/hadoop$ pylint
Traceback (most recent call last):
  File "/usr/local/bin/pylint", line 11, in <module>
    sys.exit(run_pylint())
  File "/usr/local/lib/python2.7/dist-packages/pylint/__init__.py", line 15, in run_pylint
    from pylint.lint import Run
  File "/usr/local/lib/python2.7/dist-packages/pylint/lint.py", line 67, in <module>
    from pylint import checkers
  File "/usr/local/lib/python2.7/dist-packages/pylint/checkers/__init__.py", line 45, in <module>
    from pylint.config import OptionsProviderMixIn
  File "/usr/local/lib/python2.7/dist-packages/pylint/config.py", line 49, in <module>
    import configparser
ImportError: No module named configparser
{noformat}
Bug: Possible inconsistent state of AbstractDelegationTokenSecretManager
Status: Patch Available | Priority: Major | Resolution: 
Created: 21/Jan/20 09:12
Affects Version/s: 3.3.0

[AbstractDelegationTokenSecretManager.updateCurrentKey|https://github.com/apache/hadoop/blob/581072a8f04f7568d3560f105fd1988d3acc9e54/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java#L360] increments the current key id and creates the new delegation key in two distinct synchronized blocks.

This means that other threads can see the class in an *inconsistent state, where the key for the current key id doesn't exist (yet)*.

For example the following method sometimes returns null when the token remover thread is between the two synchronized blocks:
{noformat}
@Override
public DelegationKey getCurrentKey() {
  return getDelegationKey(getCurrentKeyId());
}{noformat}
 

Also it is possible that updateCurrentKey is called from multiple threads at the same time so *distinct keys can be generated with the same key id*.

 

This issue is suspected to be the cause of the intermittent failure of  [TestLlapSignerImpl.testSigning|https://github.com/apache/hive/blob/3c0705eaf5121c7b61f2dbe9db9545c3926f26f1/llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java#L195] - HIVE-22621.
Bug: Unable to build 3.1.3 on Win 10 - missing native header files.  Error (compile-ms-native-dll) on project hadoop-common
Status: Open | Priority: Major | Resolution: 
Created: 20/Jan/20 02:32
Affects Version/s: 3.1.3

Maven build Error (compile-ms-native-dll) on project hadoop-common

mvn package -Pdist,native -DskipTests -Dmaven.javadoc.skip=true

Missing numerous native C file headers, such as:

org_apache_hadoop_io_compress_lz4_Lz4Compressor.h

org_apache_hadoop_io_compress_lz4_Lz4Decompressor.h

org_apache_hadoop_io_nativeio_NativeIO.h

org_apache_hadoop_io_nativeio_NativeIO_POSIX.h

org_apache_hadoop_security_JniBasedUnixGroupsMapping.h

org_apache_hadoop_util_NativeCrc32.h
Bug: Provide source artifacts for hadoop-client-api
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/20 10:30
Affects Version/s: 3.2.3

h5. Improvement request
The third-party libraries shading hadoop-client-api (& hadoop-client-runtime) artifacts are super useful.
 
Having uber source jar for hadoop-client-api (maybe even hadoop-client-runtime) would be great for downstream development & debugging purposes.

Are there any obstacles or objections against providing fat jar with all the hadoop client api as well ?

h5. Dev links
- *maven-shaded-plugin* and its *shadeSourcesContent* attribute
- https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#shadeSourcesContent

h2. Update April 2024: this has been reverted.

It turns out that it complicates debugging. If you want the source when debugging, the best way is just to check out the hadoop release you are working with and point your IDE at it.
Bug: ABFS: Test failure ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/20 05:37
Affects Version/s: 3.3.1

Ref: https://issues.apache.org/jira/browse/HADOOP-16890

The following test fails randomly. This test compares the perf between Non HNS account against WASB.
ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance
Bug: Zookeeper Delegation Token Manager fetch sequence number by batch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/20 22:26
Affects Version/s: 3.3.1

Currently in ZKDelegationTokenSecretManager.java the seq number is incremented by 1 each time there is a request for creating new token. This will need to send traffic to Zookeeper server. With multiple managers running, there is data contention going on. Also, since the current logic of incrementing is using tryAndSet which is optimistic concurrency control without locking. This data contention is having performance degradation when the secret manager are under volume of traffic.

The change here is to fetching this seq number by batch instead of 1, which will reduce the traffic sent to ZK and make many operations inside ZK secret manager's memory.

After putting this into production we saw huge improvement to the RPC processing latency of get delegationtoken calls. Also, since ZK takes less traffic in this way. Other write calls, like renew and cancel delegation tokens are benefiting from this change.
Bug: Add hadoop.http.idle_timeout.ms to core-default.xml
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/20 17:42
Affects Version/s: 3.0.4

HADOOP-15696 made the http server connection idle time configurable  (hadoop.http.idle_timeout.ms).

This configuration key is added to kms-default.xml and httpfs-default.xml but we missed it in core-default.xml. We should add it there because NNs/JNs/DNs also use it too.
Bug: FileUtil.copy() to throw IOException if the source and destination are the same
Status: Reopened | Priority: Major | Resolution: 
Created: 24/Feb/20 14:52
Affects Version/s: 3.3.0

We encountered an error during a test in our QE when the file destination and source path were the same. This happened during an ADLS test, and there were no meaningful error messages, so it was hard to find the root cause of the failure.
The error we saw was that file size has changed during the copy operation. The new file creation in the destination - which is the same as the source - creates a file and sets the file length to zero. After this, getting the source file will fail because the sile size changed during the operation.

I propose a solution to at least log in error level in the {{FileUtil}} if the source and destination of the copy operation is the same, so debugging issues like this will be easier in the future.

Update: This has been reverted as it caused problems. When rolling back from a branch

* revert MAPREDUCE-7303
* apply HADOOP-17936
Bug: add initial S3A layering + async init
Status: Open | Priority: Major | Resolution: 
Created: 09/Feb/20 15:38
Affects Version/s: 3.3.0

Split the S3A code into layers

* S3AFileSystem
* S3AStore + Impl
* RawS3A + Impl

S3AFS will create the others and start in order: DelegationTokens, RawS3A, S3AStore, Metastore

this will involve wrapping all access of DTs, s3client, Metastore to block until that layer is complete, or raise an exception if instantiation of it/predecessor failed.

New layers will all be subclasses of Service, split into Interface and Impl, so we can manage the init/start/stop lifecycle with existing code
Bug: Extend Hadoop S3a access from single endpoint to multiple endpoints
Status: Open | Priority: Major | Resolution: 
Created: 31/Mar/20 08:53
Affects Version/s: 3.1.3

The client API of Hadoop aws can only support a single endpoint to access. However, there are multiple endpoints in object storage (such as ceph), and therefore the storage resources could not be fully used. To address the issue, we create a new Implementation of S3AFileSystem, which support multi-endpoint access. After the optimization, system performance will increase significantly.

Usage:
 1.Ensure hadoop-aws API availiable.
 2.Copy hadoop-aws-3.1.3.jar and aws-java-sdk-bundle-1.11.271.jar to directory share/hadoop/common/lib in hadoop (hadoop-aws-3.1.3.jar and aws-java-sdk-bundle-1.11.271.jar are normally located at directory share/hadoop/tools/lib).
 3.In file etc/hadoop/hadoop-env.sh, add the following:
 export HADOOP_CLASSPATH=/(hadoop root directory)/share/hadoop/common/lib/hadoop-aws-3.1.3.jar:/(hadoop root directory)/share/hadoop/common/lib/hadoop-aws-3.1.3.jar:$HADOOP_CLASSPATH
 4.Edit configuration file "core-site.xml" and set properties below:
 <property>
 <name>fs.s3a.s3.client.factory.impl</name>
 <value>org.apache.hadoop.fs.s3a.MultiAddrS3ClientFactory</value>
 </property>
 <property>
 <name>fs.s3a.endpoint</name>
 <value>[http://addr1:port1,http://addr2:port2|http://addr1:port1%2Chttp//addr2:port2],...</value>
 </property>
 5.Optional configuration in "core-site.xml":
 <property>
 <name>fs.s3a.S3ClientSelector.class</name>
 <value>org.apache.hadoop.fs.s3a.RandomS3ClientSelector</value>
 </property>
 This configuration is used to set the s3a service selection policy. The default value is org.apache.hadoop.fs.s3a.RandomS3ClientSelector, which is a completely random selector. The configuration can be set to org.apache.hadoop.fs.s3a.PathS3ClientSelector, which is a selector according to the file path.
Bug: Add Public IOStatistics API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/20 20:34
Affects Version/s: 3.3.0

Applications like to collect the statistics which specific operations take, by collecting exactly those operations done during the execution of FS API calls by their individual worker threads, and returning these to their job driver

* S3A has a statistics API for some streams, but it's a non-standard one; Impala &c can't use it
* FileSystem storage statistics are public, but as they aren't cross-thread, they don't aggregate properly

Proposed

# A new IOStatistics interface to serve up statistics
# S3A to implement
# other stores to follow
# Pass-through from the usual wrapper classes (FS data input/output streams)

It's hard to think about how best to offer an API for operation context stats, and how to actually implement.

ThreadLocal isn't enough because the helper threads need to update on the thread local value of the instigator

My Initial PoC doesn't address that issue, but it shows what I'm thinking of