av9ash/GitBugs huggingface dataset:
Bug: S3A: TestIAMInstanceCredentialsProvider.testIAMInstanceCredentialsInstantiate failure
Status: Open | Priority: Minor | Resolution: 
Created: 05/Jan/24 11:42
Affects Version/s: 3.4.0

test failure in TestIAMInstanceCredentialsProvider; looks like the test is running in an EC2 VM whose IAM service isn't providing credentials -and the test isn't set up to ignore that.


{code}
Caused by: software.amazon.awssdk.core.exception.SdkClientException: The requested metadata is not found
at http://169.254.169.254/latest/meta-data/iam/security-credentials/
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:125)
	at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.lambda$getSecurityCredentials$3(InstanceProfileCredentialsProvider.java:256)
	at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108)
	at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getSecurityCredentials(InstanceProfileCredentialsProvider.java:256)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.createEndpointProvider(InstanceProfileCredentialsProvider.java:204)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:150)

{code}
Bug: Add logic for verifying that the STS URL is in the correct format
Status: Open | Priority: Minor | Resolution: 
Created: 07/Mar/24 05:17

* At present an invalid URL can be supplied as an STS endpoint. It will attempt to create an STSClient with it and then fail with,

{quote}java.net.UnknownHostException: request session credentials: software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.:    software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.: https
{quote} * This is inefficient. An invalid URL can be parsed much earlier and can be failed based on the URL format itself.
 * The error message is not very clear and does not indicate a problem in the URL format.
 * In this Jira issue, we attempt to parse the STS URL and fail fast with a more relevant error message.
Bug: Unified use of placeholder for log calling
Status: Open | Priority: Minor | Resolution: 
Created: 26/Mar/24 05:09

There're some classes logging with pure string like 'AMLauncher':
{code:java}
LOG.info("Setting up container " + masterContainer
    + " for AM " + application.getAppAttemptId()); {code}
It would be better use as follows?:
{code:java}
LOG.info("Setting up container {} for AM {}", masterContainer, application.getAppAttemptId());  {code}
 

If the community decides to do so, I am happy to contribute to it~
Bug: Fix metrics description
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 13/Mar/24 12:24
Affects Version/s: 3.3.0

This description of the RpcLockWaitTimeNumOps  metrics seems to be incorrect：
{code:java}
| `RpcQueueTimeNumOps` | Total number of RPC calls |
| `RpcQueueTimeAvgTime` | Average queue time in milliseconds |
| `RpcLockWaitTimeNumOps` | Total number of RPC calls (same as RpcQueueTimeNumOps) |{code}
I think the description of this metrics should be more clear：
{code:java}
| `RpcLockWaitTimeNumOps` | Total number of waiting for lock acquisition |{code}
Bug: spotbugs complaining about possible NPE in org.apache.hadoop.crypto.key.kms.ValueQueue.getSize()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Mar/24 11:16
Affects Version/s: 3.3.9

PRs against hadoop-common are reporting spotbugs problems

{code}
Dodgy code Warnings
Code	Warning
NP	Possible null pointer dereference in org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value of called method
Bug type NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE (click for details)
In class org.apache.hadoop.crypto.key.kms.ValueQueue
In method org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String)
Local variable stored in JVM register ?
Dereferenced at ValueQueue.java:[line 332]
Known null at ValueQueue.java:[line 332]

{code}
Bug: "No test bucket" error in ITestS3AContractVectoredRead if provided via CLI property
Status: Open | Priority: Minor | Resolution: 
Created: 29/Mar/24 19:04
Affects Version/s: 3.3.6

ITestS3AContractVectoredRead fails with {{NullPointerException: No test bucket}} if test bucket is defined as {{-Dtest.fs.s3a.name=...}} via CLI , not in {{auth-keys.xml}}.  The same setup works for other S3A contract tests.  Tested on 3.3.6.

{code:title=src/test/resources/auth-keys.xml}
<configuration>
  <property>
    <name>fs.s3a.endpoint</name>
    <value>${test.fs.s3a.endpoint}</value>
  </property>
  <property>
    <name>fs.contract.test.fs.s3a</name>
    <value>${test.fs.s3a.name}</value>
  </property>
</configuration>
{code}

{code}
export AWS_ACCESS_KEY_ID='<redacted>'
export AWS_SECRET_KEY='<redacted>'
mvn -Dtest=ITestS3AContractVectoredRead -Dtest.fs.s3a.name="s3a://mybucket" -Dtest.fs.s3a.endpoint="http://localhost:9878/" clean test
{code}

{code:title=test results}
Tests run: 46, Failures: 0, Errors: 8, Skipped: 0, Time elapsed: 7.879 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead
testMinSeekAndMaxSizeDefaultValues[Buffer type : direct](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead)  Time elapsed: 1.95 s  <<< ERROR!
java.lang.NullPointerException: No test bucket
  at org.apache.hadoop.util.Preconditions.checkNotNull(Preconditions.java:88)
  at org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName(S3ATestUtils.java:714)
  at org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides(S3ATestUtils.java:775)
  at org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead.testMinSeekAndMaxSizeDefaultValues(ITestS3AContractVectoredRead.java:104)
  ...

testMinSeekAndMaxSizeConfigsPropagation[Buffer type : direct](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead)  Time elapsed: 0.176 s  <<< ERROR!
testMultiVectoredReadStatsCollection[Buffer type : direct](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead)  Time elapsed: 0.179 s  <<< ERROR!
testNormalReadVsVectoredReadStatsCollection[Buffer type : direct](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead)  Time elapsed: 0.155 s  <<< ERROR!
testMinSeekAndMaxSizeDefaultValues[Buffer type : array](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead)  Time elapsed: 0.116 s  <<< ERROR!
testMinSeekAndMaxSizeConfigsPropagation[Buffer type : array](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead)  Time elapsed: 0.102 s  <<< ERROR!
testMultiVectoredReadStatsCollection[Buffer type : array](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead)  Time elapsed: 0.105 s  <<< ERROR!
testNormalReadVsVectoredReadStatsCollection[Buffer type : array](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead)  Time elapsed: 0.107 s  <<< ERROR!
{code}
Bug: Support hadoop client authentication through keytab configuration.
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 01/Feb/24 02:46

*Shield references to {{UserGroupInformation}} Class.*

The current HDFS client keytab authentication code is as follows:
{code:java}
Configuration conf = new Configuration();
conf.addResource(new Path("/usr/local/service/hadoop/etc/hadoop/hdfs-site.xml"));
conf.addResource(new Path("/usr/local/service/hadoop/etc/hadoop/core-site.xml"));
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("foo", "/var/krb5kdc/foo.keytab");
FileSystem fileSystem = FileSystem.get(conf);
FileStatus[] fileStatus = fileSystem.listStatus(new Path("/"));
for (FileStatus status : fileStatus) {
    System.out.println(status.getPath());
} {code}
This feature supports configuring keytab information in core-site.xml or hdfs site.xml. The authentication code is as follows:
{code:java}
Configuration conf = new Configuration();
conf.addResource(new Path("/usr/local/service/hadoop/etc/hadoop/hdfs-site.xml"));
conf.addResource(new Path("/usr/local/service/hadoop/etc/hadoop/core-site.xml"));
FileSystem fileSystem = FileSystem.get(conf);
FileStatus[] fileStatus = fileSystem.listStatus(new Path("/"));
for (FileStatus status : fileStatus) {
    System.out.println(status.getPath());
} {code}
The config of core-site.xml related to authentication is as follows:
{code:java}
<configuration>
    <property>
        <name>hadoop.security.authentication</name>
        <value>kerberos</value>
    </property>
    <property>
        <name>hadoop.client.keytab.principal</name>
        <value>foo</value>
    </property>
    <property>
        <name>hadoop.client.keytab.file.path</name>
        <value>/var/krb5kdc/foo.keytab</value>
    </property>
</configuration> {code}
Bug: S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jan/24 15:28
Affects Version/s: 3.4.0

Getting test failures in the new ITestS3AOpenCost tests when run with {{-Dprefetch}}

Thought I'd tested this, but clearly not
* class cast failures on asserts (fix: skip)
* bytes read different in one test: (fix: identify and address)
Bug: move commons-logging to 1.2
Status: Open | Priority: Minor | Resolution: 
Created: 22/Feb/24 16:25
Affects Version/s: 3.4.0

although hadoop doesn't use the APIs itself, it bundles commons-logging as things it depends on (http components) do.

the version hadoop declares (1.1.3) is out of date compared to its dependencies

* update pom and LICENSE-binary
Bug: S3A to support writing to object lock buckets
Status: Open | Priority: Minor | Resolution: 
Created: 16/Feb/24 12:43
Affects Version/s: 3.3.6

s3 bucket with object lock enabled fails in createFakeDirectory (reported on S.O)

error implies that we need to calculate and include the md5 checksum on the PUT, which gets complex once you include CSE into the mix: the checksum of the encrypted data is what'd be required.
Bug: S3A: update AWS SDK to 2.23.19 to support S3 Access Grants
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jan/24 21:34
Affects Version/s: 3.4.0

In order to support S3 Access Grants(https://aws.amazon.com/s3/features/access-grants/) in S3A, we need to update AWS SDK in hadooop package.
Bug: ITestS3AClosedFS.testClosedInstrumentation fails on trunk on Mac OS Silicon
Status: Open | Priority: Minor | Resolution: 
Created: 06/Feb/24 10:46
Affects Version/s: 3.5.0

Running {{mvn clean verify -pl :hadoop-aws}} on my laptop against my s3 bucket on eu-west-1 the test \{{ITestS3AClosedFS.testClosedInstrumentation}} fails every time, while when running it in isolation passes correctly.
It does not fail on branch-3.4, the attached stack trace is tested on commit: 9a7eeadaac818258b319cdb0dc19e9bb1e4fa11a.
h4. aws endpoint:

s3.eu-west-1.amazonaws.com
h4. os:

[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.bitness: 64
[INFO] os.detected.version: 14.3
[INFO] os.detected.version.major: 14
[INFO] os.detected.version.minor: 3
[INFO] os.detected.classifier: osx-aarch_64
h4. java -version:

openjdk version "1.8.0_292"
OpenJDK Runtime Environment (Zulu 8.54.0.21-CA-macos-aarch64) (build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (Zulu 8.54.0.21-CA-macos-aarch64) (build 25.292-b10, mixed mode)
h4. stack trace:

{{[ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.706 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AClosedFS}}
{{[ERROR] testClosedInstrumentation(org.apache.hadoop.fs.s3a.ITestS3AClosedFS) Time elapsed: 0.447 s <<< FAILURE!}}
{{org.junit.ComparisonFailure: [S3AInstrumentation.hasMetricSystem()] expected:<[fals]e> but was:<[tru]e>}}
{{at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)}}
{{at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)}}
{{at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)}}
{{at org.apache.hadoop.fs.s3a.ITestS3AClosedFS.testClosedInstrumentation(ITestS3AClosedFS.java:111)}}
{{at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
{{at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
{{at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
{{at java.lang.reflect.Method.invoke(Method.java:498)}}
{{at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)}}
{{at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)}}
{{at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)}}
{{at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)}}
{{at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)}}
{{at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)}}
{{at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)}}
{{at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)}}
{{at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)}}
{{at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
{{at java.lang.Thread.run(Thread.java:748)}}
Bug: Allow tag passing to AWS Assume Role Credential Provider
Status: Open | Priority: Minor | Resolution: 
Created: 05/Feb/24 20:29
Affects Version/s: 3.4.0

[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java#L131-L133] passes a session name and role arn to AssumeRoleRequest. The AWS AssumeRole API also supports passing a list of tags: [https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/sts/model/AssumeRoleRequest.html#tags()]

These tags could be used by platforms to enhance the data encoded into CloudTrail entries to provide better information about the client. For example, a 'notebook' based platform could encode the notebook / jobname / invoker-id in these tags, enabling more granular access controls and leaving a richer breadcrumb-trail as to what operations are being performed.

This is particularly useful in larger environments where jobs do not get individual roles to assume, and there is a desire to track what jobs/notebooks are reading a given set of files in S3.
Bug: S3A : ITestS3AConfiguration#testRequestTimeout failure
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 04/Jan/24 05:00
Affects Version/s: 3.4.0

"fs.s3a.connection.request.timeout" should be specified in milliseconds as per
{code:java}
Duration apiCallTimeout = getDuration(conf, REQUEST_TIMEOUT,
    DEFAULT_REQUEST_TIMEOUT_DURATION, TimeUnit.MILLISECONDS, Duration.ZERO); {code}
The test fails consistently because it sets 120 ms timeout which is less than 15s (min network operation duration), and hence gets reset to 15000 ms based on the enforcement.

 
{code:java}
[ERROR] testRequestTimeout(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)  Time elapsed: 0.016 s  <<< FAILURE!
java.lang.AssertionError: Configured fs.s3a.connection.request.timeout is different than what AWS sdk configuration uses internally expected:<120000> but was:<15000>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testRequestTimeout(ITestS3AConfiguration.java:444) {code}
Bug: ipc client print client info message duplicate
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 16/Mar/24 05:30

When sending rpc request,the debug message print client name twice, this is confusing.
like: 
{quote}2024-03-16 11:58:49,564 DEBUG ipc.Client: :1113 IPC Client (382750013) connection to kdcserver/172.20.10.12:8888 from nn/bigdata@WUZK.COM IPC Client (382750013) connection to kdcserver/172.20.10.12:8888 from nn/bigdata@WUZK.COM sending #0 org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing
{quote}
the debug message above should delete duplicate information.
Bug: Install Maven from Apache archives
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 29/May/22 11:54
Affects Version/s: 3.4.0

The Jenkins CI for Hadoop is failing to build since it's unable to download and install maven -
{code}
22:38:13  #11 [ 7/16] RUN pkg-resolver/install-maven.sh centos:7
22:38:13  #11 sha256:8b1823a6197611693af5daa2888f195db76ae5e9d0765f799becc7e7d5f7b019
22:40:25  #11 131.5 curl: (7) Failed to connect to 2403:8940:3:1::f: Cannot assign requested address
22:40:25  #11 ERROR: executor failed running [/bin/bash --login -c pkg-resolver/install-maven.sh centos:7]: exit code: 7
{code}

Jenkins run - https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4370/4/console
We need to switch to using Maven from Apache archives to prevent such issues.
Bug: Use CMake 3.19.0 in Debian 10
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 02/Jun/22 11:31
Affects Version/s: 3.4.0

HDFS Native Client fails to build on Debian 10 due to the following error -
{code}
[WARNING] CMake Error at main/native/libhdfspp/CMakeLists.txt:68 (FetchContent_MakeAvailable):
[WARNING]   Unknown CMake command "FetchContent_MakeAvailable".
[WARNING] 
[WARNING] 
[WARNING] -- Configuring incomplete, errors occurred!
{code}
Jenkins run - https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt

This cause for this issue is that the version of CMake on Debian 10 (which is installed through apt) is 3.13 and *FetchContent_MakeAvailable* was [introduced in CMake 3.14|https://cmake.org/cmake/help/v3.14/module/FetchContent.html] 

Thus, we upgrade CMake by installing through the [install-cmake.sh|https://github.com/apache/hadoop/blob/34a973a90ef89b633c9b5c13a79aa1ac11c92eb5/dev-support/docker/pkg-resolver/install-cmake.sh] script from pkg-resolver which installs CMake 3.19.0, instead of installing CMake through apt on Debian 10.
Bug: vulnerability:  we may leak sensitive information in LocalKeyStoreProvider
Status: In Progress | Priority: Critical | Resolution: 
Created: 14/May/22 05:50

Currently, we implement flush like:
{code:java}
//  public void flush() throws IOException {
    super.flush();
    if (LOG.isDebugEnabled()) {
      LOG.debug("Resetting permissions to '" + permissions + "'");
    }
    if (!Shell.WINDOWS) {
      Files.setPosixFilePermissions(Paths.get(file.getCanonicalPath()),
          permissions);
    } else {
      // FsPermission expects a 10-character string because of the leading
      // directory indicator, i.e. "drwx------". The JDK toString method returns
      // a 9-character string, so prepend a leading character.
      FsPermission fsPermission = FsPermission.valueOf(
          "-" + PosixFilePermissions.toString(permissions));
      FileUtil.setPermission(file, fsPermission);
    }
  } {code}
we wirite the Credential first, then set permission.

The correct order is setPermission first, then write Credential .

Otherswise, we may leak Credential . For example, the origin perms of file is 755(default on linux),  when the Credential  is flushed, Credential can be leaked when 

 

1)between flush and setPermission,  others have a chance to access the file.

2)  CredentialShell(or the machine node )  crash between flush and setPermission,   the file permission is 755 for ever before we run the CredentialShell again.
Bug: Interrupting RPC Client calls can lead to thread exhaustion
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 30/Jun/22 22:56
Affects Version/s: 2.10.2

Currently the IPC client creates a boundless number of threads to write the rpc request to the socket. The NameNode uses timeouts on its RPC calls to the Journal Node and a stuck JN will cause the NN to create an infinite set of threads.
Bug: Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
Status: Resolved | Priority: Critical | Resolution: Won't Fix
Created: 19/Jun/22 22:32

As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from both hadoop-client-runtime and hadoop-client-minicluster. This has caused issues for downstreamers e.g. [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions.

We should put the shading back in hadoop-client-runtime to fix CNFE issues for downstreamers.

cc [~ayushsaxena] [~pan3793]
Bug: Bump javax.ws.rs-api To Version 3.1.0
Status: Resolved | Priority: Critical | Resolution: Not A Problem
Created: 30/Jun/22 21:37
Affects Version/s: 3.3.3

Current version of  javax.ws.rs-api is 2.1.1 - which has a vulnerable dependency to [CVE-2020-15250|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-15250]

Lets Upgrade to 3.1.0
Bug: Yetus build failure in branch-3.3.
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 30/Jun/22 17:24

[ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The project org.apache.hadoop:hadoop-benchmark:3.4.0-SNAPSHOT (/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4517/src/hadoop-tools/hadoop-benchmark/pom.xml) has 1 error [ERROR] Non-resolvable parent POM for org.apache.hadoop:hadoop-benchmark:3.4.0-SNAPSHOT: Could not find artifact org.apache.hadoop:hadoop-project:pom:3.4.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 22, column 11 -> 

 

 

https://github.com/apache/hadoop/pull/4517#issuecomment-1170777719
Bug: Introduce hadoop-logging module
Status: Reopened | Priority: Major | Resolution: 
Created: 16/Apr/22 04:23

There are several goals here:
 # Provide the ability to change log level, get log level, etc.
 # Place all the appender implementation(?)
 # Hide the real logging implementation.
 # Later we could remove all the log4j references in other hadoop module.
 # Move as much log4j usage to the module as possible.
Bug: Understand status of S3 access point alias support in S3A
Status: Open | Priority: Major | Resolution: 
Created: 01/Jun/22 18:28
Affects Version/s: 3.3.5

Adding support for using access point alias to access s3 bucket.

 

When you create an access point, Amazon S3 automatically generates an alias that you can use instead of an Amazon S3 bucket name for data access. You can use this access point alias instead of an Amazon Resource Name (ARN) for any access point data plane operation.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-alias.html
Bug: Analyzing S3A Audit Logs 
Status: In Progress | Priority: Major | Resolution: 
Created: 25/May/22 10:17

The main aim is to analyze S3A Audit logs to give better insights in Hive and Spark jobs.
Steps involved are:
 * Merging audit log files containing huge number of audit logs collected from a job containing various S3 requests.
 * Parsing audit logs using regular expressions i.e., dividing them into key value pairs.
 * Converting the key value pairs into CSV file and AVRO file formats.
 * Querying on data which would give better insights for different jobs.
 * Visualizing the audit logs on Zeppelin or Jupyter notebook with graphs.