av9ash/GitBugs huggingface dataset:
Bug: Add option to downgrade S3A rejection of Syncable to warning
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Mar/21 13:07
Affects Version/s: 3.3.1

The Hadoop Filesystem Syncable API is intended to meet the requirements laid out in [StoneBraker81] _Operating System Support for Database Management_

bq.  The service required from an OS buffer manager is a selectedforce out which would push the intentions list and the commit flag to disk in the proper order. Such a service is not present in any buffer manager known to us.

It's an expensive operation -so expensive that {{Syncable.hsync()}} isn't even called on {{DFSOutputStream.close()}}. I

Even though S3A does not manifest any data until close() is called, applications coming from HDFS may call Syncable methods and expect to them to persist data with the durability guarantees offered by HDFS.

Since the output stream hardening of HADOOP-13327, S3A throws UnsupportedOperationException to indicate that the synchronization semantics of Syncable absolutely cannot be met. 

As a result, applications which have been calling the Syncable APIs are finding the call failing. In the absence of exception handling to recognise that the durability semantics are being met, they fail.

If the user and the application actually expects data to be persisted, this is the correct behaviour. The data cannot be persisted this way.

If, however, they were calling this on HDFS more as a {{flush()}} than the full and expensive DBMS-class persistence call, then this failure is unwelcome. The applications really needs to catch the UnsupportedOperationException raised by S3A _or any other FS strictly reporting failures_, report the problem and perform some other means of safe data storage

Even better, they can use hasPathCapability on the FS or hasCapability() on the stream to probe before even opening a file or trying to sync it. the hasCapability() on a stream was actually implemented in Hadooop-2.x precisely to allow applications to identify when a stream could not meet the guarantees (e.g some of the encrypted streams, file:// before HADOOP-13...)

Until they can correct their code, I propose adding the option for s3a to downgrade

fs.s3a.downgrade.syncable.exceptions 

This will

* Log once per process at WARN
* downgrade the calls to noop() 
* increment counters in S3A stats and IO stats of invocations of the Syncable methods. This will allow for stats gathering to let us identify which applications need fixing in cloud deployments

Testing: copy the hsync tests but expect exceptions to be swallowed and stats to be collected

Also: UnsupportedException text will link to this JIRA
Bug: Fix the wrong CIDR range example in Proxy User documentation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/21 08:56
Affects Version/s: 3.2.2

The CIDR range example on the Proxy user description page is wrong.

 

In the Configurations section of Proxy user page, CIDR format 10.222.0.0/16 means 10.222.0.0-15.

 

But It's not true. the CIDR format 10.222.0.0/16 means 10.222.0.0-10.222.255.255.

 

as-is : hosts in the range `10.222.0.0-15` and

to-be : hosts in the range `10.222.0.0-10.222.255.255` and
Bug: Fix the examples of hadoop config prefix
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jan/21 06:18

In https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/DownstreamDev.html#XML_Configuration_Files
{quote}e.g. hadoop, io, ipc, fs, net, file, ftp, kfs, ha, file, dfs, mapred, mapreduce, and yarn.
{quote}

* There are two "file"
* kfs has been removed since Hadoop 2.x
Bug: Provide mechanism for explicitly specifying the compression codec for input files
Status: Open | Priority: Minor | Resolution: 
Created: 03/Mar/21 17:57

I come to you via SPARK-29280.

I am looking for the file _input_ equivalents of the following settings:
{code:java}
mapreduce.output.fileoutputformat.compress
mapreduce.map.output.compress{code}
Right now, I understand that Hadoop infers the codec to use when reading a file from the file's extension.

However, in some cases the files may have the incorrect extension or no extension. There are links to some examples from SPARK-29280.

Ideally, you should be able to explicitly specify the codec to use to read those files. I don't believe that's possible today. Instead, the current workaround appears to be to [create a custom codec class|https://stackoverflow.com/a/17152167/877069] and override the getDefaultExtension method to specify the extension to expect.

Does it make sense to offer an explicit way to select the compression codec for file input, mirroring how things work for file output?
Bug: Remove Commons Logger from FileSystem Class
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jan/21 20:13

Remove reference to Commons Logger in FileSystem, it already has SLF4J, so it's a bit weird to be mixing and matching and interweaving loggers in this way.  Also, my hope is to eventually migrate everything to SLF4J to simplify things for downstream consumers of the common library.
Bug: IPV6 support in Netutils#createSocketAddress 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Feb/21 04:23
Affects Version/s: 3.1.1

Currently NetUtils#createSocketAddress not supporting if target is IPV6 ip. If target is IPV6 ip then it throw "Does not contain a valid host:port authority: ".

This need be support.

public static InetSocketAddress createSocketAddr(String target,
 int defaultPort,
 String configName,
 boolean useCacheIfPresent) {
 String helpText = "";
 if (configName != null)

{ helpText = " (configuration property '" + configName + "')"; }

if (target == null)

{ throw new IllegalArgumentException("Target address cannot be null." + helpText); }

target = target.trim();
 boolean hasScheme = target.contains("://");
 URI uri = createURI(target, hasScheme, helpText, useCacheIfPresent);

String host = uri.getHost();
 int port = uri.getPort();
 if (port == -1)

{ port = defaultPort; }

String path = uri.getPath();

if ((host == null) || (port < 0) ||
 (!hasScheme && path != null && !path.isEmpty()))

{ throw new IllegalArgumentException( *"Does not contain a valid host:port authority: " + target + helpText* ); }

return createSocketAddrForHost(host, port);
 }
Bug: refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
Status: Open | Priority: Minor | Resolution: 
Created: 12/Mar/21 01:45

Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, such as :
rename, Append , hflush&hsync support.

link : hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md

Features
Present a hierarchical HDFS file system by implementing the standard Hadoop FileSystem interface.
In hadoop, Read and write data stored in a HuaweiCloud OBS bucket.
Can act as a source of data in a MapReduce job, or a sink.
Support multipart upload for a large file.
Reference file system paths using URLs using the obs scheme.
Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and authentication schemes.
For OBS 'Parallel file system bucket'(Posix), support rename with atomic semantics, Append, hflush&hsync.
For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and quickly delete, which using rename and lifecircle of OBS bucket.
Tested for scale.
Bug: refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 12/Mar/21 01:44
Affects Version/s: 3.3.0

Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, such as :
rename, Append , hflush&hsync support.

link : hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md

Features
Present a hierarchical HDFS file system by implementing the standard Hadoop FileSystem interface.
In hadoop, Read and write data stored in a HuaweiCloud OBS bucket.
Can act as a source of data in a MapReduce job, or a sink.
Support multipart upload for a large file.
Reference file system paths using URLs using the obs scheme.
Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and authentication schemes.
For OBS 'Parallel file system bucket'(Posix), support rename with atomic semantics, Append, hflush&hsync.
For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and quickly delete, which using rename and lifecircle of OBS bucket.
Tested for scale.
Bug: ABFS: Suport for customer provided encryption key
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Feb/21 04:34
Affects Version/s: 3.4.0

The data for a particular customer needs to be encrypted on account level. At server side the APIs will start accepting the encryption key as part of request headers. The data will be encrypted/decrypted with the given key at the server. 

Since the ABFS FileSystem APIs are implementations for Hadoop FileSystem APIs there is no direct way with which customer can pass the key to ABFS driver. In this case driver should have the following capabilities so that it can accept and pass the encryption key as one of the request headers. 
 # There should be a way to configure the encryption key for different accounts.
 # If there is a key specified for a particular account, the same needs to be sent along with the request headers. 

*Config changes* 

They key for an account can be specified in the core-site as follows. 

fs.azure.account.client-provided-encryption-key.{account name}.dfs.core.windows.net
Bug: Separate string metric from tag in hadoop metrics2
Status: Patch Available | Priority: Minor | Resolution: 
Created: 26/Mar/21 21:52

Right now in hadoop metrics2, String metrics from method are categorized as tag (v.s. metrics as other number types), this caused later when reporting beans, it will add a prefix "tag." before the metric name.

It will be cleaner if we have another child inherit MutableMetric for string (maybe MutableText?) thus the String metrics from method can get rid of the tag.
Bug: Image scan shows something in Hadoop using jackson-databind 2.4.0...what?
Status: Open | Priority: Minor | Resolution: 
Created: 01/Mar/21 13:32

Hi everyone, I've done a Twistlock container-level scan of a Flink/Hadoop image (so, it's the Hadoop shaded uber jar specifically, for Hadoop 3.3.1 snapshot and Flink 1.11.3).

The most interesting result is as follows I think it is used in Hadoop and not Flink because my container scan without the Hadoop jar does not show this result. 

_{{ "version": "2.4.0",_
 _"name": "com.fasterxml.jackson.core_jackson-databind",_
 _"path": "/opt/flink/lib/flink-shaded-hadoop-3-uber-3.3.1-SNAPSHOT-10.0.jar"}}_

 

That's a very old version and likely very susceptible to CVEs I would imagine, does anybody know what might be using it and if we can upgrade the version?

 

[https://github.com/apache/hadoop/search?l=Maven+POM&q=2.4.0] shows 113 results and searching with [https://github.com/apache/hadoop/search?q=com.fasterxml.jackson.core_jackson-databind] isn't helpful either unfortunately (in fact less so).

 

So I am wondering what could be using it..any input would be awesome, thank you! I will do my own digging as well to keep looking but if anyone knows off-hand that would be fantastic
Bug: Provide snapshot builds on Maven central
Status: Resolved | Priority: Minor | Resolution: Done
Created: 24/Feb/21 15:14

Hey everyone, I'm looking to build the shaded Hadoop/Flink jar using the very latest Hadoop code that isn't yet in a Maven repository AFAIK (I'm looking at [https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/).] Entirely possible that's not the right place...

 

Are there plans or is anyone working on, or have I just missed, binaries being available (say, Hadoop 3.3-snapshot)?

 

I remember working on Spark and that was a thing, IIRC you could set a flag to true in any of the pom.xmls to accept snapshots and not published things.

 

It's entirely possible I've just forgot how to do it and can't see it well documented anywhere, and I don't believe I have to go through the steps of setting up a Maven repository somewhere (I want to do the build in Docker, and in the pom.xml I would love to just say: use Hadoop version 3.3-snapshot).

 

To give some context, I would like to build the Hadoop/Flink shaded jar using the yet to be released Hadoop 3.3 branch's code, so I can then go ahead and security scan that and test it out.

 

Thanks in advance!
Bug: Two tests perform faulty assertEquals calls comparing variable with itself
Status: Open | Priority: Minor | Resolution: 
Created: 15/Feb/21 01:27

The following test classes perform faulty {{assertEquals}} calls which erroneously compare a variable with itself.

- {{org.apache.hadoop.hdfs.server.namenode.TestNameNodeReconfigure}}
https://github.com/apache/hadoop/blob/c3134ab3a99d4109d9ae3fcf216018351eb1d36f/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeReconfigure.java#L342
{code}
assertEquals(property + " has wrong value", isSPSRunning, isSPSRunning);
{code}
- {{org.apache.hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor}}
https://github.com/apache/hadoop/blob/c3134ab3a99d4109d9ae3fcf216018351eb1d36f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestFederationClientInterceptor.java#L222
{code}
Assert.assertEquals(scIdResult, scIdResult);
{code}

It would be good if you could create a pull request for this because I am not very familiar with the project.
Bug: S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jan/21 15:10
Affects Version/s: 3.4.0

Failure in {{ITestPartialRenamesDeletes.testPartialDirDelete}}; wrong #of delete requests. 

build options: -Dparallel-tests -DtestsThreadCount=6 -Dscale -Dmarkers=delete -Ds3guard -Ddynamo

The assert fails on a line changes in HADOOP-17271; assumption being, there are some test run states where things happen differently.
Bug: ADL Gen1: Fix the test case failures which are failing after the contract test update in hadoop-common
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jan/21 10:01
Affects Version/s: 3.4.0

Fix the following test case failures which are failing after the contract test update in hadoop-common

[ERROR] Failures:
[ERROR] TestAdlContractRenameLive>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:88 expected rename(/test/source-256.txt, /test/dest-512.txt) to be rejected with exception, but got false
[ERROR] TestAdlContractRenameLive.testRenameFileUnderFile:46 Expecting org.apache.hadoop.security.AccessControlException with text Parent path is not a folder. but got : "void"
Bug: Fix the wrong CIDR range example in Proxy User documentation
Status: Resolved | Priority: Trivial | Resolution: Duplicate
Created: 17/Mar/21 08:52
Affects Version/s: 3.2.2

The CIDR range example on the Proxy user description page is wrong.

 

In the Configurations section of the Proxy user page, CIDR 10.222.0.0/16 means range 10.222.0.0-15.

 

But It's not true. CIDR format 10.222.0.0/16 means 10.222.0.0-10.222.255.255

 

as-is : 10.222.0.0-15

to-be : 10.222.0.0-10.222.255.255
Bug: prune dependency exports of hadoop-* modules
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 21/Feb/24 20:21
Affects Version/s: 3.4.0

this is probably caused by HADOOP-18613:

ZK is pulling in some extra transitive stuff which surfaces in applications which import hadoop-common into their poms. It doesn't seem to show up in our distro, but downstream you get warnings about duplicate logging stuff

{code}
|  +- org.apache.zookeeper:zookeeper:jar:3.8.3:compile
|  |  +- org.apache.zookeeper:zookeeper-jute:jar:3.8.3:compile
|  |  |  \- (org.apache.yetus:audience-annotations:jar:0.12.0:compile - omitted for duplicate)
|  |  +- org.apache.yetus:audience-annotations:jar:0.12.0:compile
|  |  +- (io.netty:netty-handler:jar:4.1.94.Final:compile - omitted for conflict with 4.1.100.Final)
|  |  +- (io.netty:netty-transport-native-epoll:jar:4.1.94.Final:compile - omitted for conflict with 4.1.100.Final)
|  |  +- (org.slf4j:slf4j-api:jar:1.7.30:compile - omitted for duplicate)
|  |  +- ch.qos.logback:logback-core:jar:1.2.10:compile
|  |  +- ch.qos.logback:logback-classic:jar:1.2.10:compile
|  |  |  +- (ch.qos.logback:logback-core:jar:1.2.10:compile - omitted for duplicate)
|  |  |  \- (org.slf4j:slf4j-api:jar:1.7.32:compile - omitted for conflict with 1.7.30)
|  |  \- (commons-io:commons-io:jar:2.11.0:compile - omitted for conflict with 2.14.0)

{code}

proposed: exclude the zk dependencies we either override outselves or don't need.
Bug: Vectored Read into off-heap buffer broken in fallback implementation
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 04/Mar/24 20:45
Affects Version/s: 3.3.6

{{VectoredReadUtils.readInDirectBuffer()}} always starts off reading at position zero even when the range is at a different offset. As a result: you can get incorrect information.

Thanks for this is straightforward: we pass in a FileRange and use its offset as the starting position.

However, this does mean that all shipping releases 3.3.5-3.4.0 cannot safely read vectorIO into direct buffers through HDFS, ABFS or GCS. Note that we have never seen this in production because the parquet and ORC libraries both read into on-heap storage.

Those libraries needs to be audited to make sure that they never attempt to read into off-heap DirectBuffers. This is a bit trickier than you would think because an allocator is passed in. For PARQUET-2171 we will 
* only invoke the API on streams which explicitly declare their support for the API (so fallback in parquet itself)
* not invoke when direct buffer allocation is in use.
Bug: [ABFS] move to jdk11 HttpClient for http2 and connection keep alive
Status: Open | Priority: Critical | Resolution: 
Created: 01/Jan/24 23:11
Affects Version/s: 3.4.0

As described in Jira Title: "in hadoop-azure, use jdk11 HttpClient instead of legacy java.net.HttpURLConnection, for supporting http2 and connection keep alive"

Few remarks:

1/ The official Azure SDK supports either OkHttp or Netty for the Http transport.

2/ the actual hadoop-azure use the class java.net.HttpURLConnection, which is slow.
  It does not use Http2, does not optimize SSL Hand-shake very well, and does not keep TCP connection alive for re-use.

3/ JDK since version >=11 have a new class HttpClient which should be a better replacement 

4/ it might be possible to introduce a configuration property (with defaut to use legacy class) , and an abstract factory to create connection via either HttpURLConnection or any other pluggeable implementation (jdk 11 HttpClient, OkHttp, Netty, ...)

5/ the official Azure SDK is maintained by Microsoft, so should better follow bug fixes and improvements than custom hadoop implementation?
[https://learn.microsoft.com/en-us/java/api/overview/azure/storage-file-datalake-readme?view=azure-java-stable
|https://learn.microsoft.com/en-us/java/api/overview/azure/storage-file-datalake-readme?view=azure-java-stable]

6/ when we use code with the official Azure SDK and Hadoop(in Spark), it is chocking to have 2 different implementations within the same JVM... 

7/ The official Azure SDK has more features that what allows the legacy hadoop class FileSystem to do... In particular, we can append (=upload) file by multiple threads (upload by fragments at different offsets), then flush when every fragments are sent.
Bug: Add support for Tez to MagicS3GuardCommitter
Status: Open | Priority: Major | Resolution: 
Created: 26/Feb/24 19:20
Affects Version/s: 3.3.6

The MagicS3GuardCommitter assumes that the JobID of the task is the same as that of the job's application master when writing/reading the .pendingset file. This assumption is not valid when running with Tez, which creates slightly different JobIDs for tasks and the application master.

 

While the MagicS3GuardCommitter is intended only for MRv2, it mostly works fine with an MRv1 wrapper with Hive/Pig (with some minor changes to Hive) run in MR mode. This issue only crops up when running queries with the Tez execution engine. I can upload a patch to Hive 3.1 to reproduce this error on EMR if needed.

 

Fixing this will probably require work from both Tez and Hadoop, wanted to start a discussion here so we can figure out how exactly we go about this.
Bug: CVE-2024-23454: Apache Hadoop: Temporary File Local Information Disclosure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jan/24 10:26
Affects Version/s: 3.4.0

Apache Hadoop’s RunJar.run() does not set permissions for temporary directory by default. If sensitive data will be present in this file, all the other local users may be able to view the content.

This is because, on unix-like systems, the system temporary directory is shared between all local users. As such, files written in this directory, without setting the correct posix permissions explicitly, may be viewable by all other local users.

Credit: Andrea Cosentino (finder)

See: https://www.cve.org/CVERecord?id=CVE-2024-23454
Bug: Release Hadoop 3.4.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/24 16:01
Affects Version/s: 3.4.0

Release a minor update to hadoop 3.4.0 with

* packaging enhancements
* updated dependencies (where viable)
* fixes for critical issues found after 3.4.0 released
* low-risk feature enhancements (those which don't impact schedule...)
Bug: S3A: Recover from Vector IO read failures
Status: Open | Priority: Major | Resolution: 
Created: 08/Mar/24 13:48
Affects Version/s: 3.3.6

s3a vector IO doesn't try to recover from read failures the way read() does.

Need to
* abort HTTP stream if considered needed
* retry active read which failed
* but not those which had succeeded

On a full failure we need to do something about any allocated buffer, which means we really need the buffer pool {{ByteBufferPool}} to return or also provide a "release" (Bytebuffer -> void) call which does the return.  we would need to
* add this as a new api with the implementations in s3a, local, rawlocal
* classic single allocator method remaps to the new one with (() -> null) as the response
Bug: S3A: detect and recover from SSL ConnectionReset exceptions
Status: Open | Priority: Major | Resolution: 
Created: 17/Jan/24 10:35
Affects Version/s: 3.3.6

s3a input stream doesn't recover from SSL exceptions, specifically ConnectionReset

This is a variant of HADOOP-19027, except it's surfaced on an older release...
# need to make sure the specific exception is handled by aborting stream and retrying -so map to the new HttpChannelEOFException
# all of thisd needs to be backported
Bug: [JDK-17] Fix UT Failures in hadoop common, hdfs, yarn
Status: Open | Priority: Major | Resolution: 
Created: 31/Jan/24 05:58

Most of the UT's failed with below exception:
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @d13f7c [in thread "Time-limited test"]