av9ash/GitBugs huggingface dataset:
Bug: KMS delegation tokens are memory expensive
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 21/Feb/20 01:13

We recently saw a number of users reporting high memory consumption in KMS.

Part of the reason being HADOOP-14445. Without that, the number of kms delegation tokens that zookeeper stores is proportional to the number of KMS servers.

There are two problems:
(1) it exceeds zookeeper jute buffer length and operations fail.
(2) KMS uses more heap memory to store KMS DTs.

But even with HADOOP-14445, KMS DTs are still expensive. Looking at a heap dump from KMS, the majority of the heap is occupied by znode and KMS DT objects. With the growing number of encrypted clusters and use cases, this is increasingly a problem our users encounter.
Bug: Large DeleteObject requests are their own Thundering Herd
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/20 18:41
Affects Version/s: 3.2.1

Currently AWS S3 throttling is initially handled in the AWS SDK, only reaching the S3 client code after it has given up.

This means we don't always directly observe when throttling is taking place.

Proposed:

* disable throttling retries in the AWS client library
* add a quantile for the S3 throttle events, as DDB has
* isolate counters of s3 and DDB throttle events to classify issues better

Because we are taking over the AWS retries, we will need to expand the initial delay en retries and the number of retries we should support before giving up.

Also: should we log throttling events? It could be useful but there is a risk of logs overloading especially if many threads in the same process were triggering the problem.

Proposed: log at debug.

Note: if S3 bucket logging is enabled then throttling events will be recorded as 503 responses in the logs. If the hadoop version contains the audit logging of HADOOP-17511, this can be used to identify operations/jobs/users which are triggering problems.
Bug: Update jackson-databind to 2.10.3 to relieve us from the endless CVE patches
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/20 17:37

Jackson-databind 2.10 should relieve us from the endless CVE patches according to https://medium.com/@cowtowncoder/jackson-2-10-features-cd880674d8a2

Not sure if this is an easy update, but i think we should do this in the Hadoop 3.3.0 and before removing jackson-databind entirely.
Bug: Über-jira: S3A Hadoop 3.3.1 features
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/20 20:09
Affects Version/s: 3.3.0

Über-jira: S3A features/fixes for Hadoop 3.4

As usual, this will clutter up with everything which hasn't gone in: don't interpret presence on this list as a commitment to implement.

And for anyone wanting to add patches

MUST
# reviews via github PRs
# *no declaration of AWS S3 endpoint (or other S3 impl) -no review*

SHOULD
# have a setup for testing SSE-KMS, DDB/S3Guard
# including an assumed role we can use for AssumedRole Delegation Tokens

If you are going near those bits of code, they uprate from SHOULD to MUST.
Bug: S3A reverts KMS encryption to the bucket's default KMS key in rename/copy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jan/20 08:40
Affects Version/s: 3.2.1

When using (bucket-level) S3 Default Encryption with SSE-KMS and a CMK, all files uploaded via the HDFS {{FileSystem}} {{s3a://}} scheme receive the wrong encryption key, always falling back to the region-specific AWS-managed KMS key for S3, instead of retaining the custom CMK.
Bug: ABFS: Send error back to client for Read Ahead request failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/20 17:49
Affects Version/s: 3.3.1

Issue seen by a customer:

The failed requests we were seeing in the AbfsClient logging actually never made it out over the wire. We have found that there’s an issue with ADLS passthrough and the 8 read ahead threads that ADLSv2 spawns in ReadBufferManager.java. We depend on thread local storage in order to get the right JWT token and those threads do not have the right information in their thread local storage. Thus, when they pick up a task from the read ahead queue they fail by throwing an AzureCredentialNotFoundException exception in AbfsRestOperation.executeHttpOperation() where it calls client.getAccessToken(). This exception is silently swallowed by the read ahead threads in ReadBufferWorker.run(). As a result, every read ahead attempt results in a failed executeHttpOperation(), but still calls AbfsClientThrottlingIntercept.updateMetrics() and contributes to throttling (despite not making it out over the wire). After the read aheads fail, the main task thread performs the read with the right thread local storage information and succeeds, but first sleeps for up to 10 seconds due to the throttling.
Bug: Upgrade to Apache ZooKeeper 3.5.8
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 19/Feb/20 09:44

Apache ZooKeeper 3.5.7 has been released, which contains some important fixes including third party CVE, possible split brain and data loss in some very rare but plausible scenarios etc.
the release has been tested by the curator team to be compatible with 4.2.0
Bug: S3A mkdirs to indicate which parent path element refers to a file
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 06/Mar/20 10:52
Affects Version/s: 3.2.1

If there is a file somewhere up the path you're trying to create a directory in S3. S3A's mkdirs() Will fail was an error -but it does not indicate which path element is at fault.
Bug: Very large files can be truncated when written through S3AFileSystem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/20 22:16
Affects Version/s: 3.2.1

If a written file size exceeds 10,000 * {{fs.s3a.multipart.size}}, a corrupt truncation of the S3 object will occur as the maximum number of parts in a multipart upload is 10,000 as [specified|https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html] by the S3 API, and there is an apparent bug where this failure is not fatal allowing the multipart upload operation to be marked as successfully completed without being fully complete.
Bug: ABFS driver enhancement - Allow customizable translation from AAD SPNs and security groups to Linux user and group
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/20 18:50

ABFS driver does not support the translation of AAD Service principal (SPI) to Linux identities causing metadata operation failure. Hadoop MapReduce client [[JobSubmissionFiles|https://github.com/apache/hadoop/blob/d842dfffa53c8b565f3d65af44ccd7e1cc706733/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java#L138]] expects the file owner permission to be the Linux identity, but the underlying ABFS driver returns the AAD Object identity. Hence need ABFS driver enhancement.
Bug: Shade & Update guava version to 29.0-jre
Status: Resolved | Priority: Major | Resolution: Done
Created: 13/Mar/20 22:38
Affects Version/s: 3.4.0

I'd like to propose
(1) shade guava into hadoop-thirdparty.
(2) migrate the hadoop main repo to use the shaded guava.
(3) update guava to 28.x in Hadoop 3.4 (Hadoop 3.3.0 is on guava 27) --> this is a moving target. There's probably going to be a new version of guava by the time Hadoop 3.4 is released.
Bug: s3guard can't init table if caller doesn't have tag permissions
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 16/Mar/20 20:10

If the user doesn't have the permissions to tag a DDB table creation will fail.


Caused by  HADOOP-16520; we downgrade the failure on other inits, HADOOP-16653, but not on actual creation.


We could downgrade creation the same way
Bug: ABFS: Optimize HttpRequest retry triggers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/20 05:55
Affects Version/s: 3.3.1

Currently retry logic gets triggered when access token fetch fails even with irrecoverable errors. Causing a large wait time for the request failure to be reported. 

 

Retry logic needs to be optimized to identify such access token fetch failures and fail fast.
Bug: KerberosAuthentication does not disconnect HttpURLConnection leading to CLOSE_WAIT cnxns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/20 10:45
Affects Version/s: 3.3.0

PseudoAuthenticator and KerberosAuthentication does not disconnect HttpURLConnection leading to lot of CLOSE_WAIT connections. YARN-8414 issue is observed due to this.
Bug: s3a mkdir path/ can add 404 to S3 load balancers
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 15/Jan/20 14:53
Affects Version/s: 3.3.0

Not seen in the wild, but inferred from a code review

mkdirs creates an empty dir marker -but it looks for one first

proposed
 * only look for file marker and LIST; don't worry about an empty dir marker
 * always PUT one there

Saves a HEAD on every mkdir too
Bug: s3a mkdirs() to not check dest for a dir marker
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Feb/20 17:48
Affects Version/s: 3.2.1

S3A innerMkdirs() calls getFileStatus() to probe dest path for being a file or dir, then goes to reject/no-op.


The HEAD path + / in that code may add a 404 to the S3 load balancers, so subsequent probes for the path fail. 

Proposed: only look for file then LIST underneath

if no entry found: probe for parent being a dir (LIST; HEAD + /), if true create the marker entry. If not, start the walk (or should we then check?)

This increases the cost of mkdir on an existing empty dir marker; reduces it on a non-empty dir. Creates dir markers above dir markers to avoid those cached 404s.
Bug: Report on S3A cached 404 recovery better
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 14/Feb/20 12:42
Affects Version/s: 3.3.0

A big hadoop -fs copyFromLocal is showing that 404 cacheing is still happening. 

{code}
20/02/13 01:02:18 WARN s3a.S3AFileSystem: Failed to find file s3a://dilbert/dogbert/queries_split_1/catberg.q._COPYING_. Either it is not yet visible, or it has been deleted.
0/02/13 01:02:18 WARN s3a.S3AFileSystem: Failed to find file s3a://dilbert/dogbert/queries_split_1/catberg.q._COPYING_. Either it is not yet visible, or it has been deleted.
{noformat}

We are recovering (good) but it's (a) got the people running this code worried and (b) shouldn't be happening.


Proposed

* error message to -> to a wiki link to a (new) doc on the topic.
* retried clause to increment counter & if count >1 report on #of attempts and duration
* S3A FS.deleteOnExit to avoid all checks
* and review the copyFromLocal to make sure no other probes are happening'
Bug: Performance improvement when distcp files in large dir with -direct option
Status: Open | Priority: Major | Resolution: 
Created: 19/Feb/20 08:15
Affects Version/s: 3.2.1

We use distcp with -direct option to copy a file between two large directories. We found it costed a few minutes. If we launch too much distcp jobs at the same time, NameNode  performance degradation is serious.

hadoop -direct -skipcrccheck -update -prbugaxt -i -numListstatusThreads 1 hdfs://cluster1:8020/source/100.log  hdfs://cluster2:8020/target/100.jpg
|| ||Dir path||Count||
||Source dir||  hdfs://cluster1:8020/source/ ||100k+ files||
||Target dir||hdfs://cluster2:8020/target/ ||100k+  files||

 

Check code in CopyCommitter.java, we find in function

deleteAttemptTempFiles() has a code targetFS.globStatus(new Path(targetWorkPath, ".distcp.tmp." + jobId.replaceAll("job","attempt") + "*")); 

It will waste a lot of time when distcp between two large dirs. When we use distcp with -direct option,  it will direct write to the target file without generate a  '.distcp.tmp'  temp file. So, i think this code need add a judgment before call function deleteAttemptTempFiles, if distcp with -direct option, do nothing , directly return .
Bug: Test failure in TestAbfsOutputStream because of HADOOP-16910
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Mar/20 08:19
Affects Version/s: 3.2.1

Failure in TestAbfsOutputStream.java due to not passing FileSystem.Statistics in AbfsOutputstream.

job:

- Passing Statistics through AbfsOutputStream.
- Closing streams in the test
Bug: MetricsConfig incorrectly loads the configuration whose value is String list in the properties file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/20 07:05
Affects Version/s: 3.1.3

[HADOOP-15549|https://jira.apache.org/jira/browse/HADOOP-15549] modified loadFirst function in MetricsConfig, and forget to set the DelimiterHandler, which caused that when loading the properties file, if the configured value is a String List, the value will not be loaded as a String Array, but just a String. For example, if we set
{code:java}
*.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40
{code}
in hadoop-metrics2.properties. If we use conf.getStringArray("*.sink.ganglia.dmax") to get the value list, we will get an array with single element, the content of which is "jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40", which wil cause an error during loadGangliaConf. loadGangliaConf will assume that the value of jvm.metrics.threadsBlocked is "70, jvm.metrics.memHeapUsedM", which will cause an error "java.lang.NumberFormatException: For input string: "70,jvm.metrics.memHeapUsedM".
Bug: Multiple Tests failure in hadoop-azure componnt
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 24/Mar/20 13:41
Affects Version/s: 3.3.0

I am seeing many test failures in hadoop azure in trunk. Posting some samples:

[*ERROR*] *Failures:* 

[*ERROR*]   *ITestAbfsContractUnbuffer>AbstractContractUnbufferTest.testMultipleUnbuffers:100->AbstractContractUnbufferTest.validateFullFileContents:132->AbstractContractUnbufferTest.validateFileContents:139->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 failed to read expected number of bytes from stream expected:<1024> but was:<-1>*

[*ERROR*]   *ITestAbfsContractUnbuffer>AbstractContractUnbufferTest.testUnbufferAfterRead:53->AbstractContractUnbufferTest.validateFullFileContents:132->AbstractContractUnbufferTest.validateFileContents:139->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 failed to read expected number of bytes from stream expected:<1024> but was:<-1>*

[*ERROR*]   *ITestAbfsContractUnbuffer>AbstractContractUnbufferTest.testUnbufferBeforeRead:63->AbstractContractUnbufferTest.validateFullFileContents:132->AbstractContractUnbufferTest.validateFileContents:139->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 failed to read expected number of bytes from stream expected:<1024> but was:<-1>*

[*ERROR*]   *ITestAbfsContractUnbuffer>AbstractContractUnbufferTest.testUnbufferMultipleReads:111->AbstractContractUnbufferTest.validateFileContents:139->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 failed to read expected number of bytes from stream expected:<128> but was:<-1>*
Bug: job commit failure in S3A MR magic committer test
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/20 16:28
Affects Version/s: 3.3.0

failure in 
{code}
ITestS3ACommitterMRJob.test_200_execute:304->Assert.fail:88 Job job_1578669113137_0003 failed in state FAILED with cause Job commit failed: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6e894de2 rejected from org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor@225eed53[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
{code}

Stack implies thread pool rejected it, but toString says "Terminated". Race condition?

*update 2020-04-22*: it's caused when a task is aborted in the AM -the threadpool is disposed of, and while that is shutting down in one thread, task commit is initiated using the same thread pool. When the task committer's destroy operation times out, it kills all the active uploads.
Bug: ABFS Streams to update FileSystem.Statistics counters on IO.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/20 07:50
Affects Version/s: 3.2.1

Abfs FileSystem Counters are not populated and hence not shown on the console side.

purpose:
 * Passing Statistics in AbfsOutputStream and populating FileSystem Counter(Write_ops) there.
 * Populating Read_ops in AbfsInputStream.
 * Showing Bytes_written, Bytes_read, Write_ops and Read_ops on the console while using ABFS.
Bug: Adding Output Stream Counters in ABFS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/20 23:07
Affects Version/s: 3.2.1

AbfsOutputStream does not have any counters that can be populated or referred to when needed for finding bottlenecks in that area.



purpose:
 * Create an interface and Implementation class for all the AbfsOutputStream counters.
 * populate the counters in AbfsOutputStream in appropriate places.
 * Override the toString() to see counters in logs.
Bug: Use Yetus 0.12.0 in GitHub PR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/20 08:58

HADOOP-16054 wants to upgrade the ubuntu version of the docker image from 16.04 to 18.04. However, ubuntu 18.04 brings maven 3.6.0 by default and the pre-commit jobs fail to add comments to GitHub and JIRA. The issue was fixed by YETUS-957 and upgrading the Yetus version to 0.12.0-SNAPSHOT (or 0.12.0, if released) will fix the problem.

How to upgrade Yetus version in the pre-commit jobs:
* GitHub PR (hadoop-multibranch): Upgrade Jenkinsfile
* JIRA (PreCommit-XXXX-Build): Manually update the config in builds.apache.org
Bug: Improve wasb and abfs resilience on double close() calls
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/20 10:37
Affects Version/s: 3.2.1

# if you call write() after the NativeAzureFsOutputStream is closed it throws an NPE ... which isn't always caught by closeQuietly code. It needs to raise an IOE
# abfs close ops can trigger failures in try-with-resources use
Bug: AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/20 02:13
Affects Version/s: 2.10.0

When hadoop lists object in versioning bucket with many delete marker in it, OSS will return
{code:java}
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult>
  <Name>select-us-east-1</Name>
  <Prefix>test/hadoop/file/</Prefix>
  <Marker></Marker>
  <MaxKeys>100</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>true</IsTruncated>
  <NextMarker>test/hadoop/file/sub2</NextMarker>
</ListBucketResult>
{code}
It sets *IsTruncated* to true and without *ObjectSummaries* or *CommonPrefixes*, and will throw FileNotFoundException
{code:java}
// code placeholder
java.io.FileNotFoundException: oss://select-us-east-1/test/hadoop/file: No such file or directory!java.io.FileNotFoundException: oss://select-us-east-1/test/hadoop/file: No such file or directory!
 at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.getFileStatus(AliyunOSSFileSystem.java:281) at org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.testGetFileStatusInVersioningBucket{code}
 
{code:java}
// code placeholder
ObjectListing listing = store.listObjects(key, 1, null, false);
if (CollectionUtils.isNotEmpty(listing.getObjectSummaries()) ||
    CollectionUtils.isNotEmpty(listing.getCommonPrefixes())) {
  return new OSSFileStatus(0, true, 1, 0, 0, qualifiedPath, username);
} else {
  throw new FileNotFoundException(path + ": No such file or directory!");
}
{code}
 In this case, we should call listObjects until *IsTruncated* is false or *ObjectSummaries* is not empty or *CommonPrefixes* is not empty.
Bug:  Upgrade findbugs-maven-plugin to 3.0.5 to fix mvn findbugs:findbugs failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Feb/20 04:46

mvn findbugs:findbugs is failing:
{noformat}
[ERROR] Failed to execute goal org.codehaus.mojo:findbugs-maven-plugin:3.0.0:findbugs (default-cli) on project hadoop-project: Unable to parse configuration of mojo org.codehaus.mojo:findbugs-maven-plugin:3.0.0:findbugs for parameter pluginArtifacts: Cannot assign configuration entry 'pluginArtifacts' with value '${plugin.artifacts}' of type java.util.Collections.UnmodifiableRandomAccessList to property of type java.util.ArrayList -> [Help 1]
 {noformat}

We have to update the version of findbugs-maven-plugin.
Bug: S3A FS deleteOnExit to skip the exists check
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Feb/20 11:51
Affects Version/s: 3.2.1

S3A FS deleteOnExiit is getting that 404 in because it looks for file.exists() before adding. it should just queue for a delete

proposeAlso, processDeleteOnExit() to skip those checks too; just call delete().
Bug: fs.s3a.authoritative.path should support multiple FS URIs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/20 10:58
Affects Version/s: 3.3.0

(This may have worked until we did recent changes to s3a path qualification)

fs.s3a.authoritative.path was intended to support not just a list of paths, but a list of URIs, so that you could explicitly list those paths which were authoritative in specific buckets. 

Parsing of the list in an FS instance must therefore discard all which are of different URI authorities
Bug: Allow IPv6 in hadoop_config.sh
Status: Open | Priority: Major | Resolution: 
Created: 26/Mar/20 15:16

Hi!

I am currently using CDH 5.x and moving to BigTop 1.4 / 1.5 (so jumping from 2.6 to 2.8.5/2.10) but the following bit of hadoop_config.sh is always there:

 # Disable ipv6 as it can cause issues
 HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"

Is there any reason to keep it? I tried to override it via daemon-specific environment variables, but in other daemons like Hive is not that easy (see [https://phabricator.wikimedia.org/T240255#5728881] for an example). 

IPv6 should be safe to use (IIRC hdfs was the main concern at the time but it was fixed HADOOP-11890).

Thanks in advance,

Luca
Bug: Emit per priority RPC queue time and processing time from DecayRpcScheduler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Mar/20 03:58

At ipc Server level we have the overall rpc queue time and processing time for the whole CallQueueManager. In the case of using FairCallQueue, it will be great to know the per queue/priority level rpc queue time since many times we want to keep certain queues to meet some queue time SLA for customers.
Bug: ITestAzureBlobFileSystemCheckAccess failing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/20 15:01
Affects Version/s: 3.3.0

Tests added in HADOOP-16455 are failing.

java.lang.IllegalArgumentException: The value of property fs.azure.account.oauth2.client.id must not be null

Looks to me like there are new configuration options which are undocumented


# these need documentation in testing markdown file
# tests MUST downgrade to skip if not set
Bug: Encryption zone file copy failure leaks temp file ._COPYING_ and wrapped stream
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/20 17:27
Affects Version/s: 3.3.0

Copy file into encryption on  trunk with HADOOP-16490 caused a leaking temp file _COPYING_ left and potential wrapped stream unclosed. This ticked is opened to track the fix for it.
Bug: [thirdparty] Handle release package related issues
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/20 08:49

Handle following comments from [~elek] in 1.0.0-RC0 voting mail thread here[[https://lists.apache.org/thread.html/r1f2e8325ecef239f0d713c683a16336e2a22431a9f6bfbde3c763816%40%3Ccommon-dev.hadoop.apache.org%3E]]
{quote}3. Yetus seems to be included in the source package. I am not sure if
 it's intentional but I would remove the patchprocess directory from the
 tar file.

7. Minor nit: I would suggest to use only the filename in the sha512
 files (instead of having the /build/source/target prefix). It would help
 to use `sha512 -c` command to validate the checksum.
{quote}
Also, update available artifacts in docs.