av9ash/GitBugs huggingface dataset:
Bug: Upgrade guice to 4.2.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jan/21 00:17
Affects Version/s: 3.4.0

Upgrade guice to 4.2.3 to fix compatibility issue:
{noformat}
Exception in thread "main" java.lang.NoSuchMethodError: com.google.inject.util.Types.collectionOf(Ljava/lang/reflect/Type;)Ljava/lang/reflect/ParameterizedType;
» at com.google.inject.multibindings.Multibinder.collectionOfProvidersOf(Multibinder.java:202)
» at com.google.inject.multibindings.Multibinder$RealMultibinder.<init>(Multibinder.java:283)
» at com.google.inject.multibindings.Multibinder$RealMultibinder.<init>(Multibinder.java:258)
» at com.google.inject.multibindings.Multibinder.newRealSetBinder(Multibinder.java:178)
» at com.google.inject.multibindings.Multibinder.newSetBinder(Multibinder.java:150)
» at org.apache.druid.guice.LifecycleModule.getEagerBinder(LifecycleModule.java:115)
» at org.apache.druid.guice.LifecycleModule.configure(LifecycleModule.java:121)
» at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
» at com.google.inject.spi.Elements.getElements(Elements.java:110)
» at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
» at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
» at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
» at com.google.inject.spi.Elements.getElements(Elements.java:110)
» at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
» at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
» at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
» at com.google.inject.spi.Elements.getElements(Elements.java:110)
» at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
» at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
» at com.google.inject.Guice.createInjector(Guice.java:96)
» at com.google.inject.Guice.createInjector(Guice.java:73)
» at com.google.inject.Guice.createInjector(Guice.java:62)
» at org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:431)
» at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69)
» at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:58)
» at org.apache.druid.cli.Main.main(Main.java:113)
{noformat}
Bug: ABFS: Introduce Lease Operations with Append to provide single writer semantics
Status: Open | Priority: Major | Resolution: 
Created: 15/Mar/21 10:59

The lease operations will be introduced as part of Append, Flush to ensure the single writer semantics.

 

Details:

Acquire Lease will be introduced in Create, Auto-Renew, Acquire will be added to Append & Release, Auto-Renew, Acquire in Flush.

 

Duration the creation of the file the lease will be acquired, as part of appends the lease will be auto-renewed & the lease can be released as part of flush.

 

By default the lease duration will be of 60 seconds.

"fs.azure.write.enforcelease" & "fs.azure.write.lease.duration" two configs will be introduced.
Bug: ABFS test setup failing "The specified filesystem already exists"
Status: Open | Priority: Major | Resolution: 
Created: 22/Feb/21 14:54
Affects Version/s: 3.3.1

been hitting this in parallel test runs. Even though FS Creation is meant to catch and swallow exceptions of code {{AzureServiceErrorCode.FILE_SYSTEM_ALREADY_EXISTS}}, this doesn't happen because the exception is translated into FileAlreadyExistsException before that catch/swallow
Bug: Use Slf4jRequestLog for HttpRequestLog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/21 05:38
Affects Version/s: 3.4.0

Now the log file name and retain days in HttpRequestLog is configured from log4j.properties. It can be configured from Configuration instead to drop the usage of log4j API.

==== Update ====

Use jetty's Slf4jRequestLogWriter, we can remove the code dependency on log appenders.
Bug: public interface GroupMappingServiceProvider needs default impl for getGroupsSet() 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/21 06:03
Affects Version/s: 3.4.0

HADOOP-17079 added "GroupMappingServiceProvider#getGroupsSet()" interface.

But since this is a public interface, it will break compilation of existing implementations in downstreams.

Consider adding a default implementation in the interface to avoid such failures.
Bug: Incorrect representation of RESPONSE for Get Key Version in KMS index.md.vm file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Mar/21 09:50
Affects Version/s: 3.3.1

Format of RESPONSE of Get Key Versions in KMS index.md.vm is incorrect

[https://hadoop.apache.org/docs/r3.1.1/hadoop-kms/index.html#Get_Key_Versions]

Attached the outputs of the commands for reference
Bug: CryptoInputStream#close() should be synchronized
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Mar/21 18:25
Affects Version/s: 3.3.5

org.apache.hadoop.crypto.CryptoInputStream.close() - when 2 threads try to close the stream second thread, fails with error.

This operation should be synchronized to avoid multiple threads to perform the close operation concurrently.

 !image-2021-03-13-23-56-18-865.png|thumbnail!
Bug: Improve UGI debug log to help troubleshooting TokenCache related issues
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/21 22:04
Affects Version/s: 3.2.0

We have seen some issues around TokenCache getDelegationToken failures even though the UGI already has a valid token. The tricky part is the token map is keyed by the canonical service name, which can be different from the actual service field in the token, e.g. KMS token in HA case. The current UGI log dumps all the tokens but not the keys of the token map. This ticket is opened to include the complete token map information in the debug log.
Bug: Fix reference to LOG is ambiguous after HADOOP-17482
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/21 03:05
Affects Version/s: 3.4.0

HADOOP-17482 changes to have two slf4j LOG instances for FileSystem.class.  This seems to breaks the Hadoop CI/Jenkins as some tests using this LOG directly are hitting the ambiguity issue between two slf4j Logger instances to the same FileSystem.class failed the build. This ticket is opened to fix those tests to unblock CI. 

 
{code:java}
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java:[1424,25] error: reference to LOG is ambiguous
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithKMS.java:[102,25] error: reference to LOG is ambiguous
[INFO] 2 errors 
{code}
Bug: Fix compilation error of OBSFileSystem in trunk
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/21 10:07
Affects Version/s: 3.4.0

{noformat}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-huaweicloud: Compilation failure
[ERROR] /home/centos/srcs/hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSFileSystem.java:[396,58] incompatible types: org.apache.hadoop.util.BlockingThreadPoolExecutorService cannot be converted to org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService
{noformat}
Bug: Hadoop prints sensitive Cookie information.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/21 17:59
Affects Version/s: 3.3.0

org.apache.hadoop.security.authentication.client.AuthenticatedURL.AuthCookieHandler#setAuthCookie - prints cookie information in log. Any sensitive infomation in Cookies will be logged, which needs to be avaided.

LOG.trace("Setting token value to {} ({})", authCookie, oldCookie);
Bug: Checkstyle IllegalImport does not catch guava imports
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Feb/21 17:02
Affects Version/s: 3.4.0

Although YARN-10352 introduces {{guava iterator import}}, it was committed to trunk without checkstyle errors.

According to [IllegalImportCheck#setIllegalPkgs |https://github.com/checkstyle/checkstyle/blob/master/src/main/java/com/puppycrawl/tools/checkstyle/checks/imports/IllegalImportCheck.java], the packages regex should be the prefix of the package. The code automatically append {{\.*}} to the regex.

CC: [~aajisaka]
Bug: Upgrade maven-site-plugin to 3.11.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/21 18:17
Affects Version/s: 3.3.5

After upgrading maven-site-plugin, error messages will be more detailed and it will help debugging.
 
Ref: https://issues.apache.org/jira/browse/YARN-10656?focusedCommentId=17291846&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291846
Bug: DistCp: Expose the JobId for applications executing through run method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/21 12:24
Affects Version/s: 3.2.3

Presently the job id is set in the JobConf, and If an application submits using execute method, it can fetch the job id, but if it goes through the run method flow, There isn't a way to get the job id.
Bug: Add thread-level IOStatistics Context
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jan/21 17:48
Affects Version/s: 3.3.1

For effective reporting of the iostatistics of individual worker threads, we need a thread-level context which IO components update.

* this contact needs to be passed in two background thread forming work on behalf of a task.
* IO Components (streams, iterators, filesystems) need to update this context statistics as they perform work
* Without double counting anything.

I imagine a ThreadLocal IOStatisticContext which will be updated in the FileSystem API Calls. This context MUST be passed into the background threads used by a task, so that IO is correctly aggregated.

I don't want streams, listIterators &c to do the updating as there is more risk of double counting. However, we need to see their statistics if we want to know things like "bytes discarded in backwards seeks". And I don't want to be updating a shared context object on every read() call.
If all we want is store IO (HEAD, GET, DELETE, list performance etc) then the FS is sufficient. 
If we do want the stream-specific detail, then I propose
* caching the context in the constructor
* updating it only in close() or unbuffer() (as we do from S3AInputStream to S3AInstrumenation)
* excluding those we know the FS already collects.


h3. important
when backporting, please follow with HADOOP-18373
Bug: Add an Audit plugin point for S3A auditing/context
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/21 14:49
Affects Version/s: 3.3.1

Add a way for auditing tools to correlate S3 object calls with Hadoop FS API calls.

Initially just to log/forward to an auditing service.

Later: let us attach them as parameters in S3 requests, such as opentrace headeers or (my initial idea: http referrer header -where it will get into the log)

Challenges
* ensuring the audit span is created for every public entry point. That will have to include those used in s3guard tools, some defacto public APIs
* and not re-entered for active spans. s3A code must not call back into the FS API points
* Propagation across worker threads


Documentation Links

* Using: https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/auditing.md
* architecture: https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/auditing_architecture.md
Bug: Apply YETUS-1102 to re-enable GitHub comments
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/21 07:08
Affects Version/s: 3.3.1

Yetus 0.13.0 enabled updating GitHub status instead of commenting the report, however, the report comments are still useful for some cases. Let's apply YETUS-1102 to re-enable the comments.
Bug: Correct timestamp format in the docs for the touch command
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/21 11:20
Affects Version/s: 3.4.0

The touch command was added by HADOOP-9214, but the usage instructions and docs have never aligned with the code.

The code uses a timestamp format of:

{code}
 new SimpleDateFormat("yyyyMMdd:HHmmss");
{code}

But the docs indicate it should be yyyyMMddHHmmss.

While I believe the format in the docs is better, tools or applications may be relying on the old format now, so I suggest we simply update the docs and usage, and improve the error message if the wrong format is used.
Bug: Fix TestKMS failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/21 01:51
Affects Version/s: 3.3.1

[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/460/artifact/out/patch-unit-hadoop-common-project_hadoop-kms.txt]

The following https tests are flaky:
 * testStartStopHttpsPseudo
 * testStartStopHttpsKerberos
 * testDelegationTokensOpsHttpsPseudo

{noformat}
[ERROR] testStartStopHttpsPseudo(org.apache.hadoop.crypto.key.kms.server.TestKMS)  Time elapsed: 1.354 s  <<< ERROR!
java.lang.NullPointerException
	at org.apache.hadoop.crypto.key.kms.server.TestKMS$1.call(TestKMS.java:553)
	at org.apache.hadoop.crypto.key.kms.server.TestKMS$1.call(TestKMS.java:534)
	at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:258)
	at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:235)
	at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:230)
	at org.apache.hadoop.crypto.key.kms.server.TestKMS.testStartStop(TestKMS.java:534)
	at org.apache.hadoop.crypto.key.kms.server.TestKMS.testStartStopHttpsPseudo(TestKMS.java:634){noformat}
Bug: Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/21 14:52
Affects Version/s: 3.4.0

Let's upgrade Zookeeper and Curator to 3.6.3 and 5.2.0 respectively.

Curator 5.2 also supports Zookeeper 3.5 servers.
Bug: Support Custom Endpoint For Hadoop Azure working with Storage Emulator
Status: Patch Available | Priority: Major | Resolution: 
Created: 14/Mar/21 11:47
Affects Version/s: 3.3.0

When using the Hadoop Azure library to write files using the wasb:// schema, and using the configuration for Hadoop Azure storage emulator (fs.azure.storage.emulator.account.name), The client assumes that the emulator will run on localhost with the default port.
When writing ITs and using Test Containers, the emulator will run on a custom endpoint but there is no way to tell that for the client.

The problem is in AzureNativeFileSystemStore.java, in connectUsingCredentials method,
which runs:

CloudStorageAccount account = CloudStorageAccount.getDevelopmentStorageAccount()

Which will use the default localhost address.

The fix is easy (And I already fixed it and tested it on my own environment, and I will PR the fix)

CloudStorageAccount.getDevelopmentStorageAccounts has already an overload that accepts URI for a custom endpont, and we can use it.
Bug: Understanding Netty versions and upgrading them (three findings in Hadoop we could upgrade?)
Status: Open | Priority: Major | Resolution: 
Created: 01/Mar/21 13:50

Hi everyone, have been raising a few JIRAs recently related to dependencies in Flink and Hadoop, and for Hadoop I have noticed the following versions of Netty in use. I'm wondering if we can work to upgrade these (potentially all to the same version) to remediate any CVEs we have. 

 

Here's what the Twistlock container scan picked up (so, this is Flink with Hadoop 3.3.1 snapshot, which I've scanned), so any thoughts or upgrade ideas would be most welcome.

 

"version": "3.10.6.Final"
 "name": "io.netty_netty"

"path": "/opt/flink/lib/flink-shaded-hadoop-3-uber-3.3.1-SNAPSHOT-10.0.jar"

 

"version": "4.1.50.Final"
"name": "io.netty_netty-all"

"path": "/opt/flink/lib/flink-shaded-hadoop-3-uber-3.3.1-SNAPSHOT-10.0.jar"

 

"version": "4.1.42.Final"
"name": "io.netty_netty-codec"

"path": "/opt/flink/lib/flink-shaded-hadoop-3-uber-3.3.1-SNAPSHOT-10.0.jar"

 

The latest 4.1 Netty I see is

 {{[https://mvnrepository.com/artifact/io.netty/netty-all/4.1.59.Final]}}

 

which may help with the above findings (assume things are all compatible!), thanks
Bug: Update Bouncy Castle to 1.68 or later
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/21 01:59
Affects Version/s: 3.3.1

-Bouncy Castle 1.60 has Hash Collision Vulnerability. Let's update to 1.68.-

Bouncy Castle 1.60 has the following vulnerabilities. Let's update to 1.68.
 * [https://nvd.nist.gov/vuln/detail/CVE-2020-26939]
 * [https://nvd.nist.gov/vuln/detail/CVE-2020-28052]
 * [https://nvd.nist.gov/vuln/detail/CVE-2020-15522]

for anyone backporting this, note that recent bouncy castle jars are incompatible with older versions of asm.jar, and so older versions of spark.
Bug: s3a magic committer may commit more data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/21 10:07
Affects Version/s: 3.2.0

s3a magic committer isRecoverySupported() is false, so will restart all task after application master restart for am jvm crashed, leaving pendingset in magic path not to clear. pendingset name format is jobAttemptPath + taskAttemptID.getTaskID() + ".pendingset", and jobAttemptPath is actually jobIdPath not JobAttemptIdPath in s3a magic committer. These pendingset files are overwritted by new task commit.

But if in new am attempt, a speculative task overcomes origin task, so pendingset file in last attempt may be hold for job commit, the data for commit is wrong
Bug: [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/21 10:38
Affects Version/s: 3.3.0

Set the value of fs.s3a.bucket.probe to 0 by default.
Bucket existence checks are done in the initialization phase of the S3AFileSystem. It's not required to run this check: the operation itself will fail if the bucket does not exist instead of the check.

Some points on why do we want to set this to 0:
* When it's set to 0, bucket existence checks won't be done during initialization thus making it faster.
* Avoid the additional one or two requests on the bucket root, so the user does not need rights to read or list that folder.
Bug: FileSystem.close() to optionally log IOStats; save to local dir
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 28/Feb/21 16:14
Affects Version/s: 3.3.1

We could save the IOStats to a local temp dir as JSON (the snapshot is designed to be serializable, even has a test), with a unique name (iostats-stevel-s3a-bucket1-timestamp-random#.json ... etc). 

We can collect these (Rajesh can, anyway), and then
* look for load on a specific bucket
* look what happened at a specific time

The best bit: the IOStatisticsSnapshot aggregates counters, min/max/mean, so you could merge iostats-*-s3a-bucket1-*.json to get the IOStats of all principals working with a given bucket

This will be local, so low cost, low cost enough we could turn it on in production. All that's needed is collection of the stats from the local hosts (or they write to a shared mounted volume)
We will need some "hadoop iostats merge" command to take multiple files and merge them all together; print to screen or save to a new file. Straightforward as all the load and merge code is present.


Needs
* logging in FS.close
* new iostats CLI + docs, tests
* extend IOStatisticsSnapshot with list of <string, string> options for use in annotating saved logs (hostname, principal, jobID, ...). Don't know how to merge these.

If we are going to add a new context map to the IOStatisticsSnapshot then we MUST update it before 3.3.1 ships so as to avoid breaking the serialization format on the next release, especially the java one.
Bug: IOStatistics Phase II
Status: Open | Priority: Major | Resolution: 
Created: 14/Jan/21 14:04
Affects Version/s: 3.3.1

Continue IOStatistics development with goals of

* Easy adoption in applications
* better instrumentation in hadoop codebase (distcp?)
* more stats in abfs and s3a connectors

A key has to be a thread level context for statistics so that app code doesn't have to explicitly ask for the stats for each worker thread. Instead 

filesystem components update the context stats as well as thread stats (when?) and then apps can pick up.

* need to manage performance by minimising inefficient lookups, lock acquisition etc on what should be memory-only ops (read()), (write()),
* and for duration tracking, cut down on calls to System.currentTime() so that only 1 should be made per operation, 
* need to propagate the context into worker threads

Target uses

* Impala 
* Spark via SPARK-29397 
* S3A committers
* Iceberg.

I have a WiP Parquet branch too, to see what can be done there. This shows up how the thread context is needed as its unworkable to build up your own stats shapshot. Even if you collect it for listX and stream reads, it doesn't include FS operations (e.g. rename()) and you need to rework all your methods to pass the stats collector around
Bug: Not closing an SFTP File System instance prevents JVM from exiting. 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/21 21:23
Affects Version/s: 3.2.0

SFTP file system leverages a connection pool which is not closed when a file system instance gets closed preventing a JVM from exiting as every SFTP connection runs in a separate non-daemon thread.
Bug: hadoop-huaweicloud and hadoop-cloud-storage to remove log4j as transitive dependency
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/21 12:58
Affects Version/s: 3.4.0

Dependencies of hadoop-cloud-storage show that hadoop-huaweicloud is pulling in logj4. 

it should not/must not, at least, not if the huaweicloud can live without it. 

* A version of log4j 2.,2 on the CP is only going to complicate lives
* once we can move onto it ourselves we need to be in control of versions

[INFO] \- org.apache.hadoop:hadoop-huaweicloud:jar:3.4.0-SNAPSHOT:compile
[INFO]    \- com.huaweicloud:esdk-obs-java:jar:3.20.4.2:compile
[INFO]       +- com.jamesmurty.utils:java-xmlbuilder:jar:1.2:compile
[INFO]       +- com.squareup.okhttp3:okhttp:jar:3.14.2:compile
[INFO]       +- org.apache.logging.log4j:log4j-core:jar:2.12.0:compile
[INFO]       \- org.apache.logging.log4j:log4j-api:jar:2.12.0:compile