av9ash/GitBugs huggingface dataset:
Bug: TestZKFailoverController#testNoZK is flaky
Status: Open | Priority: Major | Resolution: 
Created: 22/Oct/20 17:56

TestZKFailoverController#testNoZK failed in https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2397/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
{noformat}
[ERROR] testNoZK(org.apache.hadoop.ha.TestZKFailoverController)  Time elapsed: 0.114 s  <<< ERROR!
java.lang.IllegalStateException
	at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:492)
	at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:318)
	at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:230)
	at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:60)
	at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:178)
	at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:174)
	at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:485)
	at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:174)
	at org.apache.hadoop.ha.TestZKFailoverController.runFC(TestZKFailoverController.java:700)
	at org.apache.hadoop.ha.TestZKFailoverController.testNoZK(TestZKFailoverController.java:135)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)
{noformat}
Bug: ABFS: read/cache footer with fs.azure.footer.read.request.size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/20 05:46
Affects Version/s: 3.4.0

Optimize read performance for the following scenarios
 # Read small files completely
 Files that are of size smaller than the read buffer size can be considered as small files. In case of such files it would be better to read the full file into the AbfsInputStream buffer.
 # Read last block if the read is for footer
 If the read is for the last 8 bytes, read the full file.
 This will optimize reads for parquet files. [Parquet file format|https://www.ellicium.com/parquet-file-format-structure/]

Both these optimizations will be present under configs as follows
 # fs.azure.read.smallfilescompletely
 # fs.azure.read.optimizefooterread
Bug: S3A AWS Credential provider loading gets confused with isolated classloaders
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Nov/20 14:54
Affects Version/s: 3.4.0

Problem: exception in loading S3A credentials for an FS, "Class class com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement AWSCredentialsProvider"

Location: S3A + Spark dataframes test

Hypothesised cause:

Configuration.getClasses() uses the context classloader, and with the spark isolated CL that's different from the one the s3a FS uses, so it can't load AWS credential providers.
Bug: envtoconf broken for .conf and few other formats
Status: Open | Priority: Major | Resolution: 
Created: 16/Dec/20 07:49

{{envtoconf}} does not work for some output formats:

 * {{.env}}
 * {{.sh}}
 * {{.cfg}}
 * {{.conf}}

To reproduce:

{code:title=docker run -it --rm -e DUMMY.CONF_key=value apache/hadoop:3}
Traceback (most recent call last):
  File "/opt/envtoconf.py", line 117, in <module>
    Simple(sys.argv[1:]).main()
  File "/opt/envtoconf.py", line 108, in main
    self.transform()
  File "/opt/envtoconf.py", line 96, in transform
    content = transformer_func(content)
  File "/opt/transformation.py", line 121, in to_conf
    for key, val in props:
ValueError: too many values to unpack
{code}
Bug: Optimize S3A for maximum performance in directory listings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 10:31
Affects Version/s: 3.3.0

Make listing in applications as fast as we can get it especially for query planning.

* All operations used in listing directories for query planning etc to be optimized for their primary use: being passed directories (not files) and so make that faster even at the expense of  more remote IO when handed files or empty directories.
* remove needless calls to S3 wherever possible (e.g. {{getFileStatus("/")}}, making bucket existence probes optional)
* Support/enable Asynchronous IO where possible.
 

Review higher level APIs (glob status) and uses on the FsShell and optimize their use by minimising invocations or FS API calls, with bonus goal of reduce/minimize risk of 404 caching.

Work with downstream projects to move to FS APIs which work best in this world -primarily the recursive listing operations and those which return RemoteIterator<FileStatus> -and so make any asynchronous page fetching operations useful.
Bug: FileSystem.get to support slow-to-instantiate FS clients
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/20 14:31
Affects Version/s: 3.3.0

A recurrent problem in processes with many worker threads (hive, spark etc) is that calling `FileSystem.get(URI-to-object-store)` triggers the creation and then discard of many FS clients -all but one for the same URL. As well as the direct performance hit, this can exacerbate locking problems and make instantiation a lot slower than it would otherwise be.

This has been observed with the S3A and ABFS connectors.

The ultimate solution here would probably be something more complicated to ensure that only one thread was ever creating a connector for a given URL -the rest would wait for it to be initialized. This would (a) reduce contention & CPU, IO network load, and (b) reduce the time for all but the first thread to resume processing to that of the remaining time in .initialize(). This would also benefit the S3A connector.

We'd need something like

# A (per-user) map of filesystems being created <URI, FileSystem>
# split createFileSystem into two: instantiateFileSystem and initializeFileSystem
# each thread to instantiate the FS, put() it into the new map
# If there was one already, discard the old one and wait for the new one to be ready via a call to Object.wait()
# If there wasn't an entry, call initializeFileSystem) and then, finally, call Object.notifyAll(), and move it from the map of filesystems being initialized to the map of created filesystems

This sounds too straightforward to be that simple; the troublespots are probably related to race conditions moving entries between the two maps and making sure that no thread will block on the FS being initialized while it has already been initialized (and so wait() will block forever).

Rather than seek perfection, it may be safest go for a best-effort optimisation of the #of FS instances created/initialized. That is: its better to maybe create a few more FS instances than needed than it is to block forever.

Something is doable here, it's just not quick-and-dirty. Testing will be "fun"; probably best to isolate this new logic somewhere where we can simulate slow starts on one thread with many other threads waiting for it.

A simpler option would be to have a lock on the construction process: only one FS can be instantiated per user at a a time.
Bug: Improve excessive reloading of Configurations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/20 21:18

[~daryn] reported that adding a new resource to a conf forces a complete reload of the conf instead of just loading the new resource. Instantiating a {{SSLFactory}} adds a new resource for the ssl client/server file. Formerly only the KMS client used the SSLFactory but now TLS/RPC uses it too.

The reload is so costly that RM token cancellation falls behind by hours or days. The accumulation of uncancelled tokens in the KMS rose from a few thousand to hundreds of thousands which risks ZK scalability issues causing a KMS outage.
Bug: Intermittent failure of S3A tests which make assertions on statistics/IOStatistics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Dec/20 21:59
Affects Version/s: 3.4.0

Intermittent failure of ITestHuge* upload tests, when doing parallel test runs.

The count of bytes uploaded through StorageStatistics isn't updated. Maybe the expected counter isn't updated, and somehow in a parallel run with recycled FS instances/set up directory structure this surfaces the way it doesn't in a single test run.
Bug: Print the thread parker and lock information in stacks page
Status: Open | Priority: Major | Resolution: 
Created: 25/Dec/20 00:32
Affects Version/s: 3.4.0

Sometimes, our service stuck because of some lock held by other thread, but we can get nothing from "stacks" for ReadWriteLock, and it is widely used in our services, like the fslock, cplock, dirlock of namenode.

Luckily, we can get thread parker from Thread object, it can help us see the thread parker clearly.

 !image-2020-12-25-08-32-32-982.png!
Bug: Remove S3Guard - no longer needed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/20 18:10
Affects Version/s: 3.3.0

With Consistent S3, S3Guard is superfluous. 

stop developing it and wean people off it as soon as they can.

Then we can worry about what to do in the code. It has gradually insinuated its way through the layers, especially things like multi-object delete handling (see HADOOP-17244). Things would be a lot simpler without it


This work is being done in the feature branch HADOOP-17409-remove-s3guard
Bug: AliyunOSS: support ListObjectsV2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Nov/20 01:49
Affects Version/s: 2.10.1

OSS supports ListObjectsV2([https://help.aliyun.com/document_detail/187544.html?spm=a2c4g.11186623.6.1589.e0623d9fE1b64S)] to optimize versioning bucket list. We should support this feature in AliyunOSS module.
Bug: Add GCS FS impl reference to core-default.xml
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 17:14

Akin to current S3 default configuration add GCS configuration, specifically to declare the GCS implementation. [GCS connector|https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage]. Has this not been done since the GCS connector is not part of the hadoop/ASF codebase, or is there any other blocker?
Bug: Fair call queue is defeated by abusive service principals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Nov/20 23:16

[~daryn] reported  that the FCQ prioritizes based on the full kerberos principal (ie. "user/host@realm") rather than short name (ie. "user") to prevent service principals like the DNs and NMs being de-prioritized since service principals are expected to be well behaved.  Notably the DNs contribute a significant but important load so the intent is not to de-prioritize all DNs because their sum total load is high relative to users.

This has the unfortunate side effect of allowing misbehaving & non-critical service principals to abuse the FCQ. The gstorm/* principals are a prime example.   Each server is spamming opens as fast as possible which ensures that none of the gstorm servers can be de-prioritized because each principal is a fraction of the total load from all principals.

The secondary and more devasting problem is other abusive non-service principals cannot be effectively de-prioritized.  The sum total of all gstorm load prevents other principals from surpassing the priority thresholds.  Principals stay in the highest priority queues which allows the abusive principals to overflow the entire call queue for extended periods of time.  Notably it prevents the FCQ from moderating the heavy create loads from p_gup @ DB which cause significant performance degradation.

Prioritization should be based on short name with configurable exemptions for services like the DN/NM.

[~daryn] suggested a solution that we applied on our clusters.
Bug: Increase docker memory limit in Jenkins
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Dec/20 16:26

Yetus keeps failing with OOM.

 
{code:bash}
unable to create new native thread

java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
	at org.apache.maven.surefire.booter.ForkedBooter.launchLastDitchDaemonShutdownThread(ForkedBooter.java:369)
	at org.apache.maven.surefire.booter.ForkedBooter.acknowledgedExit(ForkedBooter.java:333)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:145)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
 

This jira to increase the memory limit from 20g to 22g.

*Note: This is only a workaround to get things more productive. If this change reduces the frequency of the OOM failure, there must be a follow-up profile the runtime to figure out which components are causing the docker to run out of memory.*

CC: [~aajisaka], [~elgoiri], [~weichiu], [~ebadger], [~tasanuma], [~iwasakims], [~ayushtkn], [~inigoiri]
Bug: Creating a token identifier should not do kerberos name resolution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Nov/20 15:28
Affects Version/s: 2.10.1

This problem was found and fixed internally for us by [~daryn].

Creating a token identifier tries to do auth_to_local short username translation. The authentication process creates a blank token identifier for deserializing the wire format. Attempting to resolve an empty username is useless work.

Discovered the issue during fair call queue backoff testing. The readers are unnecessary slowed down by this bug.
Bug: Doing hadoop ls on Har file triggers too many RPC calls
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Nov/20 20:12

[~daryn] has noticed that Invoking hadoop ls on HAR is taking too much of time.

The har system has multiple deficiencies that significantly impacted performance:

# Parsing the master index references ranges within the archive index. Each range required re-opening the hdfs input stream and seeking to the same location where it previously stopped.
# Listing a har stats the archive index for every "directory". The per-call cache used a unique key for each stat, rendering the cache useless and significantly increasing memory pressure.
# Determining the children of a directory scans the entire archive contents and filters out children. The cached metadata already stores the exact child list.
# Globbing a har's contents resulted in unnecessary stats for every leaf path.
Bug: TestLdapGroupsMapping failing -string mismatch in exception validation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Nov/20 18:12
Affects Version/s: 3.3.1

Looks like a change in the exception strings is breaking the validation code
{code}
[ERROR]   TestLdapGroupsMapping.testLdapReadTimeout:447  Expected to find 'LDAP response read timed out, timeout used:4000ms' but got unexpected exception: javax.naming.NamingException: LDAP response read timed out, timeout used: 4000 ms.; remaining name ''
	at com.sun.jndi.ldap.LdapRequest.getReplyBer(LdapRequest.java:129)
{code}
Bug: Javadoc warnings and errors are ignored in the precommit jobs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/20 11:43
Affects Version/s: 2.10.0

After changing the JDK from OracleJDK to the OpenJDK packaged in Ubuntu, javadoc warnings and errors are not shown in the precommit Jenkins jobs.

In Ubuntu/Debian, doclint is not enabled by default:
 - [https://bugs.launchpad.net/ubuntu/+source/openjdk-8/+bug/1898116]
 - [https://github.com/Debian/openjdk-8/blob/master/debian/patches/disable-doclint-by-default.patch]
Bug: Fix the error of TestDynamometerInfra
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Nov/20 04:48

https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/321/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt
{noformat}
[INFO] Running org.apache.hadoop.tools.dynamometer.TestDynamometerInfra
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.43 s <<< FAILURE! - in org.apache.hadoop.tools.dynamometer.TestDynamometerInfra
[ERROR] org.apache.hadoop.tools.dynamometer.TestDynamometerInfra  Time elapsed: 0.43 s  <<< ERROR!
java.io.FileNotFoundException: http://mirrors.ocf.berkeley.edu/apache/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at org.apache.commons.io.FileUtils.copyURLToFile(FileUtils.java:1506)
	at org.apache.hadoop.tools.dynamometer.DynoInfraUtils.fetchHadoopTarball(DynoInfraUtils.java:151)
	at org.apache.hadoop.tools.dynamometer.TestDynamometerInfra.setupClass(TestDynamometerInfra.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{noformat}
Bug: Magic committer files don't have the count of bytes written collected by spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/20 14:28
Affects Version/s: 3.2.0

The spark statistics tracking doesn't correctly assess the size of the uploaded files as it only calls getFileStatus on the zero byte objects -not the yet-to-manifest files. Which, given they don't exist yet, isn't easy to do.

Solution: 
* Add getXAttr and listXAttr API calls to S3AFileSystem
* Return all S3 object headers as XAttr attributes prefixed "header." That's custom and standard (e.g header.Content-Length).

The setXAttr call isn't implemented, so for correctness the FS doesn't
declare its support for the API in hasPathCapability().

The magic commit file write sets the custom header 
set the length of the data final data in the header
x-hadoop-s3a-magic-data-length in the marker file.

A matching patch in Spark will look for the XAttr
"header.x-hadoop-s3a-magic-data-length" when the file
being probed for output data is zero byte long. 
As a result, the job tracking statistics will report the
bytes written but yet to be manifest.
Bug: ABFS: Allow Random Reads to be of Buffer Size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Oct/20 22:31
Affects Version/s: 3.3.0

ADLS Gen2/ABFS driver is optimized to read only the bytes that are requested for when the read pattern is random. 

It was observed in some spark jobs that though the reads are random, the next read doesn't skip by a lot and can be served by the earlier read if read was done in buffer size. As a result the job triggered a higher count of read calls/higher IOPS, resulting in higher IOPS throttling and hence resulted in higher job runtime.

When these jobs were run against Gen1 which always reads in buffer size , the jobs fared well. 

This Jira attempts to get a Gen1 customer migrating to Gen2 get the same overall i/o pattern as gen1 and the same perf characteristics.

*+Stats from Customer Job:+*

 
|*Customer Job*|*Gen 1 timing*|*Gen 2 Without patch*|*Gen2 with patch and RAH=0*|
|Job1|2 h 47 m|3 h 45 m|2 h 27 mins|
|Job2|2 h 17 m|3 h 24 m|2 h 39 mins|
|Job3|3 h 16 m|4 h 29 m|3 h 21 mins|
|Job4|1 h 59 m|3 h 12 m|2 h 28 mins|

 

*+Stats from Internal TPCDs runs+* 

[Total number of TPCDs queries per suite run = 80  

Full suite repeat run count per config = 3]
| |*Gen1*|Gen2 Without patch|*Gen2 With patch and RAH=0*
*(Gen2 in Gen1 config)*|*Gen2 With patch and RAH=2*|
|%Run Duration|100|140|213|70-90|
|%Read IOPS|100|106|98|110-115|

 

*Without patch = default Jar with random read logic

*With patch=Modified Jar with change to always read buffer size

*RAH=ReadAheadQueueDepth
Bug: ABFS: Piggyback flush on Append calls for short writes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/20 17:51
Affects Version/s: 3.3.0

When Hflush or Hsync APIs are called, a call is made to store backend to commit the data that was appended. 

If the data size written by Hadoop app is small, i.e. data size :
 * before any of HFlush/HSync call is made or

 * between 2 HFlush/Hsync API calls

is less than write buffer size, 2 separate calls, one for append and another for flush is made,

Apps that do such small writes eventually end up with almost similar number of calls for flush and append.

This PR enables Flush to be piggybacked onto append call for such short write scenarios.

 

NOTE: The changes is guarded over a config, and is disabled by default until relevant supported changes is made available on all store production clusters.

New Config added: fs.azure.write.enableappendwithflush
Bug: ABFS: Delete Idempotency handling can lead to NPE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/20 11:59
Affects Version/s: 3.3.0

Delete idempotency code returns success with a dummy success HttpOperation. the calling code that checks continuation token throws NPE as the dummy success instance does not have any response headers.

In case of non-HNS account, server coulf return continuation token.  Dummy success response code is modified to not fail while accessing response headers.