av9ash/GitBugs huggingface dataset:
Bug: JAR in conflict with timestamp check causes AM errors
Status: Resolved | Priority: Blocker | Resolution: Duplicate
Created: 30/Sep/21 17:20
Affects Version/s: 2.9.2

After an init action pulls down a new JAR and the check of a JAR's timestamp is performed  [1]we can sometimes cause an incorrect error if the timestamp does not match. In order to address this you can perform workarounds like:

record old timestamp at the beginning before the connector is changed
local -r old_file_time=$(date -r ${dataproc_common_lib_dir}/gcs-connector.jar "+%m%d%H%M.00")

# at end of script.
touch -t "${old_file_time}" <any jar they changed>
touch -h -t "${old_file_time}" <symlinks they changed>


We should instead of checking the date be comparing version compatibility tests.

 

 

1. https://github.com/apache/hadoop/blob/release-2.7.3-RC2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java#L255-L258
Bug: Fix Hadoop build on Debian 10
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 08/Sep/21 17:46
Affects Version/s: 3.4.0

We're using *Debian testing* as one of the package sources to get the latest packages. It seems to be broken at the moment. The CI fails to create the build environment for the Debian 10 platform -

{code}
[2021-09-08T00:21:11.596Z] #13 [ 8/14] RUN apt-get -q update     && apt-get -q install -y --no-install-recommends python3     && apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py debian:10)     && apt-get clean     && rm -rf /var/lib/apt/lists/*
...
[2021-09-08T00:21:22.744Z] #13 11.28 Preparing to unpack .../libc6_2.31-17_amd64.deb ...

[2021-09-08T00:21:23.260Z] #13 11.46 Checking for services that may need to be restarted...

[2021-09-08T00:21:23.260Z] #13 11.48 Checking init scripts...

[2021-09-08T00:21:23.260Z] #13 11.50 Unpacking libc6:amd64 (2.31-17) over (2.28-10) ...

[2021-09-08T00:21:26.290Z] #13 14.38 Setting up libc6:amd64 (2.31-17) ...

[2021-09-08T00:21:26.290Z] #13 14.42 /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory

[2021-09-08T00:21:26.290Z] #13 14.42 dpkg: error processing package libc6:amd64 (--configure):

[2021-09-08T00:21:26.290Z] #13 14.42  installed libc6:amd64 package post-installation script subprocess returned error exit status 127

[2021-09-08T00:21:26.291Z] #13 14.43 Errors were encountered while processing:

[2021-09-08T00:21:26.291Z] #13 14.43  libc6:amd64

[2021-09-08T00:21:26.291Z] #13 14.46 E: Sub-process /usr/bin/dpkg returned an error code (1)

[2021-09-08T00:21:27.867Z] #13 ERROR: executor failed running [/bin/bash -o pipefail -c apt-get -q update     && apt-get -q install -y --no-install-recommends python3     && apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py debian:10)     && apt-get clean     && rm -rf /var/lib/apt/lists/*]: exit code: 100

[2021-09-08T00:21:27.867Z] ------

[2021-09-08T00:21:27.867Z]  > [ 8/14] RUN apt-get -q update     && apt-get -q install -y --no-install-recommends python3     && apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py debian:10)     && apt-get clean     && rm -rf /var/lib/apt/lists/*:

[2021-09-08T00:21:27.867Z] ------

[2021-09-08T00:21:27.867Z] executor failed running [/bin/bash -o pipefail -c apt-get -q update     && apt-get -q install -y --no-install-recommends python3     && apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py debian:10)     && apt-get clean     && rm -rf /var/lib/apt/lists/*]: exit code: 100

[2021-09-08T00:21:27.867Z] ERROR: Docker failed to build yetus/hadoop:ef5dbc7283a.

[2021-09-08T00:21:27.867Z] 
{code}

The above log lines are copied from - https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3388/3/pipeline
Bug: ABFS AbfsDelegationTokenManager to generate canonicalServiceName if DT plugin doesn't
Status: Open | Priority: Critical | Resolution: 
Created: 15/Sep/21 12:43
Affects Version/s: 3.3.1

Currently in {{AbfsDelegationTokenManager}}, any {{CustomDelegationTokenManager}} only provides a canonical service name if it
implements {{BoundDTExtension}} and its {{getCanonicalServiceName()}} method.

If this doesn't hold, {{AbfsDelegationTokenManager}} returns null, which causes {{AzureBlobFileSystem.getCanonicalServiceName()}}
to call {{super.getCanonicalServiceName()}} *which resolves the IP address of the abfs endpoint, and then the FQDN of that IPAddr

If a storage account is served over >1 endpoint, then the DT will only have a valid service name for one of the possible
endpoints, so _only work if all process get the same IP address when the look up the storage account address_

Fix

# DT plugins SHOULD generate the canonical service name
#  If they don't, and DTs are enabled: {{AbfsDelegationTokenManager}} to create a default one
# and {{AzureBlobFileSystem.getCanonicalServiceName()}} MUST NOT call superclass.


The default canonical service name of a store will be {{abfs:// + FsURI.getHost() + "/"}}, so all containers in same storage account has the same service name

{code}
abfs://bucket@stevel-testing.dfs.core.windows.net/path
{code}

maps to 
{code}
abfs://stevel-testing.dfs.core.windows.net/ 
{code}

This will mean that only one DT will be created per storage a/c; Applications will not need to list all containers which deployed processes will wish to interact with. Today's behaviour, based on rDNS lookup of storage account, is possibly slightly broader in that all storage accounts which map to the same IPAddr share a DT. The proposed scheme will still be much broader than that of S3A, where every bucket has its unique service name, so apps need to list all target filesystems at launch time (easy for MR, source of trouble in spark).

Fix: straightforward. 

Test
* no DTs: service name == null
* DTs: will match proposed pattern, even if extension returns null.
Bug: Performance degradation in Text.append() after HADOOP-16951
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Sep/21 11:49
Affects Version/s: 3.4.0

We discovered a serious performance degradation in {{Text.append()}}.

The problem is that the logic which intends to increase the size of the backing array does not work as intended.
It's very difficult to spot, so I added extra logs to see what happens.

Let's add 4096 bytes of textual data in a loop:
{noformat}
  public static void main(String[] args) {
    Text text = new Text();
    String toAppend = RandomStringUtils.randomAscii(4096);

    for(int i = 0; i < 100; i++) {
      text.append(toAppend.getBytes(), 0, 4096);
    }
  }
{noformat}

With some debug printouts, we can observe:
{noformat}
2021-09-08 13:35:29,528 INFO  [main] io.Text (Text.java:append(251)) - length: 24576,  len: 4096, utf8ArraySize: 4096, bytes.length: 30720
2021-09-08 13:35:29,528 INFO  [main] io.Text (Text.java:append(253)) - length + (length >> 1): 36864
2021-09-08 13:35:29,528 INFO  [main] io.Text (Text.java:append(254)) - length + len: 28672
2021-09-08 13:35:29,528 INFO  [main] io.Text (Text.java:ensureCapacity(287)) - >>> enhancing capacity from 30720 to 36864
2021-09-08 13:35:29,528 INFO  [main] io.Text (Text.java:append(251)) - length: 28672,  len: 4096, utf8ArraySize: 4096, bytes.length: 36864
2021-09-08 13:35:29,528 INFO  [main] io.Text (Text.java:append(253)) - length + (length >> 1): 43008
2021-09-08 13:35:29,529 INFO  [main] io.Text (Text.java:append(254)) - length + len: 32768
2021-09-08 13:35:29,529 INFO  [main] io.Text (Text.java:ensureCapacity(287)) - >>> enhancing capacity from 36864 to 43008
2021-09-08 13:35:29,529 INFO  [main] io.Text (Text.java:append(251)) - length: 32768,  len: 4096, utf8ArraySize: 4096, bytes.length: 43008
2021-09-08 13:35:29,529 INFO  [main] io.Text (Text.java:append(253)) - length + (length >> 1): 49152
2021-09-08 13:35:29,529 INFO  [main] io.Text (Text.java:append(254)) - length + len: 36864
2021-09-08 13:35:29,529 INFO  [main] io.Text (Text.java:ensureCapacity(287)) - >>> enhancing capacity from 43008 to 49152
...
{noformat}

After a certain number of {{append()}} calls, subsequent capacity increments are small.

It's because the difference between two {{length + (length >> 1)}} values is always 6144 bytes. Because the size of the backing array is trailing behind the calculated value, the increment will also be 6144 bytes. This means that new arrays are constantly created.

Suggested solution: don't calculate the capacity in advance based on length. Instead, pass the required minimum to {{ensureCapacity()}}. Then the increment should depend on the actual size of the byte array if the desired capacity is larger.
Bug: Use separate source dir for platform builds
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 16/Jul/21 06:08
Affects Version/s: 3.4.0

The multi-platform build stages run on the checkout of the source directory, one after the other. For those platforms that are marked as optional (Centos 8 and Debian 10 currently), the condition to run CI on the platform is determined by inspecting the git commit history and checking if there's any C++ file/C++ build/platform related changes.

It seems like after YETUS runs on one platform, it's clearing up the git branch information. This is causing the build to not get triggered on the optional platforms. Please note that those platforms not marked optional (Ubuntu focal) isn't affected by this since CI runs for this platform irrespective of any C++ changes.

We can see this in the Jenkins UI page -

CI runs for Centos 8 -
 !image-2021-07-16-11-36-26-698.png! 

Subsequently, the CI for Debian 10 gets skipped -
 !image-2021-07-16-11-36-55-495.png! 

However, CI for Ubuntu focal runs since it's not marked as optional -
 !image-2021-07-16-11-37-56-923.png! 

Thus, we need to ensure that each platform builds on its own copy of the source code checkout so that whatever changes one platform makes doesn't affect the other.
Bug: ShellBasedUnixGroupsMapping: group name containing space can be used to inject group memberships
Status: Open | Priority: Critical | Resolution: 
Created: 20/Sep/21 12:57
Affects Version/s: 3.3.1

Group names available from identity management systems, for example sssd, may contain space characters when used with for example Active Directory. such a group name can be used to inject group memberships granting permission to basically any targeted group.

 

Suppose following scenario:

a) centralized identity management system is used, where organization's responsible roles are defined to allow access to their named groups.

b) group "hdfs" grants hdfs-admin permissions and is managed by authorized personnel only.

c) attacker orders creation of a group named as "uploaderformy hdfs" and the attacker's user account "attacker1" as member of that group.

 

This will lead to the scenario where ShellBasedUnixGroupsMapping executes group lookup and returns groups uploaderformy and hdfs for the "attacker1" username as TOKEN_SEPARATOR_REGEX contains space character in addition others ("[ \t\n\r\f]").

This bug was found during our own solution based on the ShellBasedUnixGroupsMapping for [https://github.com/teragrep/]

 

Other versions may be affected as well.
Bug: Add s3a tool to convert S3 server logs to avro/csv files
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Sep/21 15:26
Affects Version/s: 3.3.2

Add s3a tool to convert S3 server logs to avro/csv files

With S3A Auditing, we have code in hadoop-aws to parse s3 log entries, including splitting up the referrer into its fields.

But we don't have an easy way of using it. I've done some early work in spark but as well as that code not working ([https://github.com/hortonworks-spark/cloud-integration/blob/master/spark-cloud-integration/src/main/scala/com/cloudera/spark/cloud/s3/S3LogRecordParser.scala]), it doesn't do the audit splitting.


 And, given that the S3 audit logs can be small on a lightly loaded store, not always justified.

Proposed

we add
 # utility parser class to take a row and split it into a record
 # which can be saved to avro through a schema we define
 # or exported to CSV with/without headers. (with: easy to understand, without: can cat files)
 # add a mapper so this can be used in MR jobs (could even make it committer test ..)
 # and a "hadoop s3guard/hadoop s3" entry point so you can do it on the cli

{code:java}
hadoop s3 parselogs -format avro -out s3a://dest/path -recursive s3a://stevel-london/logs/bucket1/*
{code}
would take all files under the path, load, parse and emit the output.

design issues
 * would you combine all files, or emit a new .avro or .csv file for each one?
 * what's a good avro schema to cope with new context attributes
 * CSV nuances: tabs vs spaces, use opencsv or implement the (escaping?) writer ourselves.
 me: TSV and do a minimal escaping and quoting emitter. Can use opencsv in the test suite.
 * would you want an initial filter during processing? especially for exit codes?
 me: no, though I could see the benefit for 503s. Best to let you load it into a notebook or spreadsheet and go from there.
Bug: S3A DeleteOperation to parallelize POSTing of bulk deletes
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/21 13:38
Affects Version/s: 3.4.0

Once the need to update the DDB tables is removed, we can't go from a single POSTed delete at a time to posting a large set of bulk delete operations in parallel.

The current design is to support incremental update of S3Guard tables, including handling partial failures. Not a problem anymore.

This will significantly improve delete() performance on directory trees with many many children/descendants, as it goes from a sequence of children/1000 POSTs to parallel writes. As each file deleted is still throttled, we will be limited to 3500 deletes/second with throttling, so throwing a large pool of workers at the problem would be counter-productive and potentially cause problems for other applications trying to write down the same directory tree. But we can do better than one-POST at a time.

Proposed
* if parallel delete is off: no limit
* parallel delete is on, limit #of parallel to 3000/page-size: you'll never have more updates pending than the write limit of a single shard.
Bug: S3A: Allow SSE configurations per object path
Status: Open | Priority: Major | Resolution: 
Created: 19/Aug/21 02:04
Affects Version/s: 3.3.1

Currently, we can map the SSE configurations at bucket level only:
{code:java}
<property>
  <name>fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm</name>
  <value>SSE-KMS</value>
</property>

<property>
  <name>fs.s3a.bucket.ireland-dev.server-side-encryption.key</name>
  <value>arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01</value>
</property>
{code}
But sometimes we want to encrypt data in different paths with different keys within the same bucket. For example, a partitioned table might benefit from encrypting each partition with a different key when the partition represents a customer or a country.

[S3 already can encrypt using different keys/configurations at the object level|https://aws.amazon.com/premiumsupport/knowledge-center/s3-encrypt-specific-folder/], so what we need to do on Hadoop is to provide a way to map which key to use. One idea could be mapping them in the XML config:

 
{code:java}
<property>
  <name>fs.s3a.server-side-encryption.paths</name>
  <value>s3://bucket/my_table/country=ireland,s3://bucket/my_table/country=uk, s3://bucket/my_table/country=germany</value>
</property>

<property>
  <name>fs.s3a.server-side-encryption.path-keys</name>
  <value>arn:aws:kms:eu-west-1:90ireland09:key/ireland-key,arn:aws:kms:eu-west-1:980uk0993c:key/uk-key,arn:aws:kms:eu-west-1:98germany089:key/germany-key</value>
</property>
{code}
Or potentially fetch the mappings from the filesystem:

 
{code:java}
<property>
  <name>fs.s3a.server-side-encryption.mappings</name>
  <value>s3://bucket/configs/encryption_mappings.json</value>
</property> {code}
where encryption_mappings.json could be something like this:

 
{code:java}
{ 
   "path": "s3://bucket/customer_table/customerId=abc123", 
   "algorithm": "SSE-KMS",
   "key": "arn:aws:kms:eu-west-1:933993746:key/abc123-key"
}
...
{ 
   "path": "s3://bucket/customer_table/customerId=xyx987", 
   "algorithm": "SSE-KMS",
   "key": "arn:aws:kms:eu-west-1:933993746:key/xyx987-key"
}
{code}
Bug: RawLocalFileSystem cannot mkdir/chmod paths with emojis. ☹️
Status: Open | Priority: Major | Resolution: 
Created: 06/Sep/21 14:48
Affects Version/s: 3.3.1

*Bug description:*

`fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with NativeIO enabled.

The failure was happening when doing the native `chmod` command to the file (the `mkdir` command itself is working).

Stacktrace:

{{ENOENT: No such file or directory  ENOENT: No such file or directory at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974) at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}

 

*To reproduce:*
 * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
 * Sample: [https://github.com/apache/hadoop/pull/3391]
Bug: ABFS: Transient failure of TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting
Status: Open | Priority: Major | Resolution: 
Created: 01/Aug/21 01:44
Affects Version/s: 3.4.0

Transient failure of the below test observed for HNS OAuth, AppendBlob HNS OAuth and Non-HNS SharedKey combinations. The value denoted by "actual value" below varies across failures, and exceeds the upper limit of the expected range.

_TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 10 is not within the expected range: [5.60, 8.40]._

Verified failure with client and server in the same region to rule out network issues.
Bug: ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException
Status: Reopened | Priority: Major | Resolution: 
Created: 26/Aug/21 13:11
Affects Version/s: 3.3.1

To address transient failures in the following test classes:
 * ITestAbfsStreamStatistics: Uses a filesystem level instance to record read/write statistics, which also tracks these operations in other tests. running parallelly. To be marked for sequential run only to avoid transient failure
 * ITestAbfsRestOperationException: The use of a static member to track retry count causes transient failures when two tests of this class happen to run together. Switch to non-static variable for assertions on retry count
Bug: ABFS: Refactor read flow to include ReadRequestParameter
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/21 11:23
Affects Version/s: 3.4.0

This Jira is to facilitate upcoming work as part of adding an alternate connection :
 HADOOP-17853 ABFS: Enable optional store connectivity over azure specific protocol for data egress - ASF JIRA (apache.org)

The scope of the change is to introduce a ReadRequestParameter that will include the various inputs needed for the read request to AbfsClient class.
Bug: ABFS: Fork AbfsHttpOperation to add alternate connection
Status: Open | Priority: Major | Resolution: 
Created: 25/Aug/21 12:03
Affects Version/s: 3.4.0

This Jira is to facilitate upcoming work as part of adding an alternate connection :
[HADOOP-17853] ABFS: Enable optional store connectivity over azure specific protocol for data egress - ASF JIRA (apache.org)



The scope of the change is to make AbfsHttpOperation as abstract class and create a child class AbfsHttpConnection. Future connection types will be added as child of AbfsHttpOperation.
Bug: WASB: Fix Compiler Warnings
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/21 06:18
Affects Version/s: 3.3.1

To track a list of WASB compiler warnings, which are also intermittently affecting yetus runs on ABFS driver PRs.
 # hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java:210:29:[deprecation] HFLUSH in StreamCapabilities has been deprecated
Bug: ABFS: Modifications to TracingContext
Status: Open | Priority: Major | Resolution: 
Created: 05/Jul/21 15:23
Affects Version/s: 3.3.1

TracingContext introduced in the PR  [HADOOP-17290|https://github.com/apache/hadoop/pull/2520] tracks a request with identifiers as it passes through ABFS layers. Creating this Jira to track suggested modifications and improvements to the tracing structure.
 * rename Listener; remove it from TracingContext constructor, confine to setter
 * make fields (identifiers) immutable; eliminate retry count field => reduce instance cloning
 * introduce child classes of TracingContext to handle stream/continuation ops
Bug: Bump aliyun-sdk-oss to 3.13.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/21 18:10
Affects Version/s: 3.2.3

Bump aliyun-sdk-oss to 3.13.0 in order to remove transitive dependency on jdom 1.1.

Ref: https://issues.apache.org/jira/browse/HADOOP-17820?focusedCommentId=17390206&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17390206.
Bug: implement non-guava Precondition checkState
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/21 17:52
Affects Version/s: 3.2.3

As part In order to replace Guava Preconditions, we need to implement our own versions of the API.
 This Jira is to add the implementation {{checkState}} to the existing class {{org.apache.hadoop.util.Preconditions}}

+The plan is as follows+
 * implement {{org.apache.hadoop.util.Preconditions.checkState}} with the minimum set of interface used in the current hadoop repo.
 * we can replace {{guava.Preconditions}} by {{org.apache.hadoop.util.Preconditions}} once all the interfaces have been implemented (both this jira and HADOOP-17929 are complete).
 * We need the change to be easily to be backported in 3.x.

previous jiras:
 * HADOOP-17126 was created to implement CheckNotNull.
 * HADOOP-17929 implementing checkArgument.

CC: [~stevel@apache.org], [~vjasani]
Bug: implement non-guava Precondition checkArgument
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/21 17:49
Affects Version/s: 3.2.3

As part In order to replace Guava Preconditions, we need to implement our own versions of the API.
 This Jira is to add the implementation {{checkArgument}} to the existing class {{org.apache.hadoop.util.Preconditions}}

+The plan is as follows+
 * implement {{org.apache.hadoop.util.Preconditions.checkArgument}} with the minimum set of interface used in the current hadoop repo.
 * we can replace {{guava.Preconditions}} by {{org.apache.hadoop.util.Preconditions}} once all the interfaces have been implemented.
 * We need the change to be easily to be backported in 3.x.

A previous jira HADOOP-17126 was created to replace CheckNotNull. HADOOP-17930 is created to implement checkState.

CC: [~stevel@apache.org], [~vjasani]
Bug: Provide fallbacks for identity/cost providers and backoff enable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/21 18:54
Affects Version/s: 3.4.0

This sub-task is to provide default properties for identity-provider.impl, cost-provider.impl and backoff.enable such that if properties with port is not configured, we can fallback to default property (port-less).
Bug: TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory failure after reverting HADOOP-16878
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Sep/21 03:00
Affects Version/s: 3.3.2

After reverting HADOOP-16878 from branch-3.3, test {{TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory}} started to fail because it expects an exception but the copying succeeded.