av9ash/GitBugs huggingface dataset:
Bug: javax.activation-api and jakarta.activation-api define overlapping classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/20 04:52

There are some warnings in hadoop-client-runtime module.
{noformat}
[WARNING] javax.activation-api-1.2.0.jar, jakarta.activation-api-1.2.1.jar define 31 overlapping classes: 
[WARNING]   - javax.activation.CommandInfo$Beans$1
[WARNING]   - javax.activation.ObjectDataContentHandler
[WARNING]   - javax.activation.DataContentHandlerFactory
[WARNING]   - javax.activation.DataContentHandler
[WARNING]   - javax.activation.CommandObject
[WARNING]   - javax.activation.SecuritySupport$2
[WARNING]   - javax.activation.FileTypeMap
[WARNING]   - javax.activation.CommandInfo
[WARNING]   - javax.activation.MailcapCommandMap
[WARNING]   - javax.activation.DataHandler$1
[WARNING]   - 21 more...
{noformat}
Bug: UserGroupInformation#getCurrentUser is always unix user in linux env?
Status: Open | Priority: Major | Resolution: 
Created: 13/May/20 08:48
Affects Version/s: 2.7.3

After UserGroupInformation#loginUserFromKeytab(...) in my program, invoke UserGroupInformation#getCurrentuser() method and return a unix user in linux OS, but return a loginUser in my Mac OS. Why is this so?
Bug: Change scope of InternalDirOfViewFs and InodeTree to make ViewFileSystem extendable outside common package
Status: Open | Priority: Major | Resolution: 
Created: 13/May/20 00:56

In a use case where ViewFileSystem to be extended outside of hadoop-common package, the scope of InternalDirOfViewFs and InodeTree needs to be public. 

 

cc [~virajith] [~shv]
Bug: Declare ProtobufHelper a public API
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 29/Apr/20 02:40
Affects Version/s: 3.3.0

HADOOP-16621 removed two public API methods:

1. o.a.h.security.token.Token(TokenProto tokenPB) --> replaced by o.a.h.security.ipc.ProtobufHelper.tokenFromProto()
2. o.a.h.security.token.Token.toTokenProto() --> replaced by o.a.h.security.ipc.ProtobufHelper.protoFromToken()

Protobuf is declared private. Should we make it public now?
Bug: hadoop-cos fails to build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/20 18:59
Affects Version/s: 3.3.0

Found the following compilation error in a PR precommit. The failure doesn't seem related to the PR itself. Cant' reproduce locally though.

https://builds.apache.org/job/hadoop-multibranch/job/PR-1972/1/artifact/out/patch-compile-root.txt

{noformat}
[INFO] Apache Hadoop Tencent COS Support .................. FAILURE [  0.074 s]
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:31 min
[INFO] Finished at: 2020-04-22T07:37:51+00:00
[INFO] Final Memory: 192M/1714M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.0.2:copy-dependencies (package) on project hadoop-cos: Artifact has not been packaged yet. When used on reactor artifact, copy should be executed after packaging: see MDEP-187. -> [Help 1]
{noformat}
Bug: Remove Ratis dependencies from Hadoop branch-3.2
Status: Open | Priority: Major | Resolution: 
Created: 24/Apr/20 14:18
Affects Version/s: 3.2.2

These are not used in the core Hadoop codebase. Remove them to avoid confusion.

{code}
    <!-- Apache Ratis version -->
    <ratis.version>0.3.0-eca3531-SNAPSHOT</ratis.version>

      <dependency>
        <groupId>org.apache.ratis</groupId>
        <artifactId>ratis-proto-shaded</artifactId>
        <version>${ratis.version}</version>
      </dependency>
      <dependency>
        <artifactId>ratis-common</artifactId>
        <groupId>org.apache.ratis</groupId>
        <version>${ratis.version}</version>
      </dependency>
      <dependency>
        <artifactId>ratis-client</artifactId>
        <groupId>org.apache.ratis</groupId>
        <version>${ratis.version}</version>
      </dependency>
      <dependency>
        <artifactId>ratis-server</artifactId>
        <groupId>org.apache.ratis</groupId>
        <version>${ratis.version}</version>
      </dependency>
      <dependency>
        <artifactId>ratis-netty</artifactId>
        <groupId>org.apache.ratis</groupId>
        <version>${ratis.version}</version>
      </dependency>
      <dependency>
        <artifactId>ratis-grpc</artifactId>
        <groupId>org.apache.ratis</groupId>
        <version>${ratis.version}</version>
      </dependency>
{code}

These are removed from trunk already.
Bug: The suffix name of the unified compression class
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/20 08:04
Affects Version/s: 3.2.1

The suffix name of the unified compression class,I think the suffix name in the compression class should be extracted into a constant class, which is helpful for developers to understand the structure of the compression class as a whole.
{quote}public static final String OPT_EXTENSION =
 "io.compress.passthrough.extension";

/**
 * This default extension is here so that if no extension has been defined,
 * some value is still returned: \{@value}..
 */
public static final String DEFAULT_EXTENSION = ".passthrough";

private Configuration conf;

private String extension = DEFAULT_EXTENSION;

public PassthroughCodec() {
}
{quote}

The above code, the use of constants is a bit messy.
Bug: in javaApi, UGI params should be overidden through FileSystem conf
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 13/Apr/20 02:33
Affects Version/s: 2.7.2

org.apache.hadoop.security.UserGroupInformation#ensureInitialized,will always get the configure from the configuration files. Like below：
{code:java}
private static void ensureInitialized() {
  if (conf == null) {
    synchronized(UserGroupInformation.class) {
      if (conf == null) { // someone might have beat us
        initialize(new Configuration(), false);
      }
    }
  }
}{code}
So that, if FileSystem is created through FileSystem#get or FileSystem#newInstance with conf, the conf values different from the configuration files will not take effect in UserGroupInformation.  E.g:
{code:java}
Configuration conf = new Configuration();
conf.set("k1","v1");
conf.set("k2","v2");
FileSystem fs = FileSystem.get(uri, conf);{code}
"k1" or "k2" will not work in UserGroupInformation.
Bug: s3a to not need wildfly on the classpath
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/20 18:16
Affects Version/s: 3.3.0

see : https://github.com/apache/hadoop/pull/1948 and HADOOP-16855

* remove a hard dependency on wildfly.jar being on the classpath for S3; it's used if present, but handled if not
* even if openssl is requested
* and NPEs are caught and swallowed in case wildfly 1.0.4.Final ever gets on the classpath again
Bug: Provide GroupIdentityProvider and MappedUserIdentityProvider for FairCallQueue namenode RPC Queue throttling for grouping user requests
Status: Open | Priority: Major | Resolution: 
Created: 18/Apr/20 03:13
Affects Version/s: 3.0.3

Currently, in a multi-tenant cluster FairCallQueue Identity is limited to UserIdentityProvider. Tenants tend to get passed burst RPC loads by using different service ID. This Jira requests that GroupIdentityProvider and MappedUserIdentityProvider be implemented to allow better tenant experience for clusters with multiple tenants.
Bug: Making `getBoolean` log warning message for unrecognized value
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/20 16:02

*Problem:*

In `Configuration.java`, the `getBoolean` can accept any valueString and return the default value for any string except “true” or “false” (ignoring case):
{code:java}
if (StringUtils.equalsIgnoreCase("true", valueString))
  return true;
else if (StringUtils.equalsIgnoreCase("false", valueString))
  return false;
else return defaultValue;{code}
If the user misspells some boolean configuration value, for example, “true” to “ture”, then getBoolean will directly return the default value without logging any warning message. If the default value is “false”, then Hadoop is actually using a totally different value (“false”) compared to the user’s expectation (“true”) and the user even doesn’t know it.

This can lead to serious issues, especially regarding security features.

Other projects such as Alluxio are doing more rigorous and explicit check.
[https://github.com/xlab-uiuc/ctest-alluxio/blob/master/core/common/src/main/java/alluxio/conf/InstancedConfiguration.java#L366]
in which the getBoolean method will fail immediately if the value is invalid.

 

*Solution:*

We can log one warning message before getBoolean return the default value for unrecognized value:
{code:java}
if (StringUtils.equalsIgnoreCase("true", valueString))
  return true;
else if (StringUtils.equalsIgnoreCase("false", valueString))
  return false;
else {
  LOG.warn("Invalid value for boolean: " + valueString +
           ", choose default value: " + defaultValue + " for " + name);
  return defaultValue;
}{code}
I attach a patch to log the warning message.
Bug: Tencent Cloud COS: Package it into hadoop-dist
Status: Open | Priority: Major | Resolution: 
Created: 03/Apr/20 09:30
Affects Version/s: 3.3.0

[~stevel@apache.org]  Hi, Steve, the Hadoop-COS has been submitted to the trunk branch last year. (Related jira: https://issues.apache.org/jira/browse/HADOOP-15616). 

I am the contributor and the maintainer of this patch.  Considering that you are the expert on the object storage support in Hadoop, I have some questions for you.
 # The Hadoop-cos patch is placed in the *hadoop-cloud-storage-project directory. How to package it into the hadoop-dist directory as the distribution package?*
 # How to manage the dependencies of the Hadoop-cos, such as cos-java-sdk? I use the maven to compile and package it, but only the hadoop-cos-3.4.0-SNAPSHOT.jar is generated. I compare the pom.xml between the hadoop-cos and hadoop-oss, and find the dependency scope is '*compile*' in their pom.xml. The hadoop-oss has oss-java-sdk.jar in dist directory, but the hadoop-cos does not. 

Can you guide me how to solve these two problems? 

 

Thanks for your time and attention. :)
Bug: S3A client retries on SSL Auth exceptions triggered by "." bucket names
Status: Open | Priority: Minor | Resolution: 
Created: 27/Apr/20 10:00
Affects Version/s: 3.2.1

If you have a "." in bucket names (it's allowed!) then virtual host HTTPS connections fail with a  java.net.ssl exception. Except we retry and the inner cause is wrapped by generic "client exceptions"

I'm not going to try and be clever about fixing this, but we should
* make sure that the inner exception is raised up
* avoid retries
* document it in the troubleshooting page. 
* if there is a well known public "." bucket (cloudera has some:)) we can test

I get a vague suspicion the AWS SDK is retrying too. Not much we can do there.
Bug: S3A deleteObjects hanging/retrying forever
Status: Open | Priority: Minor | Resolution: 
Created: 04/Jun/20 00:00
Affects Version/s: 3.2.1

{code}
sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:523) com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82) org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:446) org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:365) org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64) org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:685) org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:122) org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165) org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42) org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57) org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74) org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247) org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242) org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394) org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248) org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170) org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169) org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) org.apache.spark.scheduler.Task.run(Task.scala:123) org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)
{code}

 
we are using spark 2.4.4 with hadoop 3.2.1 on kubernetes/spark-operator, sometimes we see this hang with the stacktrace above. it looks like the putObject never return, we have to kill the executor to make the job move forward.
Bug: Replace Guava Predicate with Java8+ Predicate
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Jun/20 23:34
Affects Version/s: 3.2.2

{{com.google.common.base.Predicate}} can be replaced with {{java.util.function.Predicate}}. 
The change involving 9 occurrences is straightforward:


{code:java}
Targets
    Occurrences of 'com.google.common.base.Predicate' in project with mask '*.java'
Found Occurrences  (9 usages found)
    org.apache.hadoop.hdfs.server.blockmanagement  (1 usage found)
        CombinedHostFileManager.java  (1 usage found)
            43 import com.google.common.base.Predicate;
    org.apache.hadoop.hdfs.server.namenode  (1 usage found)
        NameNodeResourceChecker.java  (1 usage found)
            38 import com.google.common.base.Predicate;
    org.apache.hadoop.hdfs.server.namenode.snapshot  (1 usage found)
        Snapshot.java  (1 usage found)
            41 import com.google.common.base.Predicate;
    org.apache.hadoop.metrics2.impl  (2 usages found)
        MetricsRecords.java  (1 usage found)
            21 import com.google.common.base.Predicate;
        TestMetricsSystemImpl.java  (1 usage found)
            41 import com.google.common.base.Predicate;
    org.apache.hadoop.yarn.logaggregation  (1 usage found)
        AggregatedLogFormat.java  (1 usage found)
            77 import com.google.common.base.Predicate;
    org.apache.hadoop.yarn.logaggregation.filecontroller  (1 usage found)
        LogAggregationFileController.java  (1 usage found)
            22 import com.google.common.base.Predicate;
    org.apache.hadoop.yarn.logaggregation.filecontroller.ifile  (1 usage found)
        LogAggregationIndexedFileController.java  (1 usage found)
            22 import com.google.common.base.Predicate;
    org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation  (1 usage found)
        AppLogAggregatorImpl.java  (1 usage found)
            75 import com.google.common.base.Predicate;

{code}
Bug: Tidy Up Text and ByteWritables Classes
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Apr/20 23:22
Affects Version/s: 3.4.0

# Remove superfluous code
 # Remove superfluous comments
 # Checkstyle fixes
 # Remove methods that simply call {{super}}.method()
 # Use Java 8 facilities to streamline code where applicable
 # Simplify and unify some of the constructs between the two classes

 

The one meaningful change is that I am suggesting that the expanding of the arrays be 1.5x instead of 2x per expansion.  I pulled this idea from open JDK.
Bug: S3AFS globStatus attempts to resolve symlinks
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jun/20 17:40

The S3AFileSystem implementation of the globStatus API has a setting configured to resolve symlinks. Under certain circumstances, this will cause additional file existence checks to be performed in order to determine if a FileStatus signifies a symlink. As symlinks are not supported in S3AFileSystem, these calls are unnecessary.

Code snapshot (permalink): [https://github.com/apache/hadoop/blob/2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4002]

Causes additional getFileStatus call here (permalink): [https://github.com/apache/hadoop/blob/1921e94292f0820985a0cfbf8922a2a1a67fe921/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java#L308]

Current code snippet:
{code:java}
/**
   * Override superclass so as to disable symlink resolution and so avoid
   * some calls to the FS which may have problems when the store is being
   * inconsistent.
   * {@inheritDoc}
   */
  @Override
  public FileStatus[] globStatus(
      final Path pathPattern,
      final PathFilter filter)
      throws IOException {
    entryPoint(INVOCATION_GLOB_STATUS);
    return Globber.createGlobber(this)
        .withPathPattern(pathPattern)
        .withPathFiltern(filter)
        .withResolveSymlinks(true)
        .build()
        .glob();
  }
{code}
 

The fix should be pretty simple, just flip "withResolveSymlinks" to false.
Bug: TestSequenceFile#testRecursiveSeqFileCreate fails in subsequent run 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Apr/20 22:01
Affects Version/s: 3.2.1

The test expects an IOException when creating a writer for file `target/test/data/recursiveCreateDir/file` with `createParent=false`. And it expects to create the writer successfully when `createParent=True`. `createParent` means `create parent directory if non-existent`.

The test will pass if it is run for the first time, but it will fail for the second run. This is because the test did not clean the parent directory created during the first run.

The parent directory `recursiveCreateDir` was created, but it was not deleted before the test finished. So, when the test was run again, it still treated the parent directory `recursiveCreateDir` as non-existent and expected an IOException from creating a writer with `createParent=false`. Then the test did not get the expected IOException because `recursiveCreateDir` has been created in the first test run.




{code:java}
@SuppressWarnings("deprecation")
 @Test
 public void testRecursiveSeqFileCreate() throws IOException {
 FileSystem fs = FileSystem.getLocal(conf);
 Path name = new Path(new Path(GenericTestUtils.getTempPath(
 "recursiveCreateDir")), "file"); // FILE SUCCESSULLY CREATED HERE
 boolean createParent = false;
try {
 SequenceFile.createWriter(fs, conf, name, RandomDatum.class,
 RandomDatum.class, 512, (short) 1, 4096, createParent,
 CompressionType.NONE, null, new Metadata());
 fail("Expected an IOException due to missing parent");
 } catch (IOException ioe) {
 // Expected
 }
createParent = true;
 SequenceFile.createWriter(fs, conf, name, RandomDatum.class,
 RandomDatum.class, 512, (short) 1, 4096, createParent,
 CompressionType.NONE, null, new Metadata());
 // should succeed, fails if exception thrown
 }
{code}


Suggested patch:

 
{code:java}
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSequenceFile.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSequenceFile.java
index 044824356ed..1aff2936264 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSequenceFile.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSequenceFile.java
@@ -649,8 +649,9 @@ public void testCreateWriterOnExistingFile() throws IOException {
 @Test
 public void testRecursiveSeqFileCreate() throws IOException {
 FileSystem fs = FileSystem.getLocal(conf);
- Path name = new Path(new Path(GenericTestUtils.getTempPath(
- "recursiveCreateDir")), "file");
+ Path parentDir = new Path(GenericTestUtils.getTempPath(
+ "recursiveCreateDir"));
+ Path name = new Path(parentDir, "file");
 boolean createParent = false;
 
 try {
@@ -667,6 +668,9 @@ public void testRecursiveSeqFileCreate() throws IOException {
 RandomDatum.class, 512, (short) 1, 4096, createParent,
 CompressionType.NONE, null, new Metadata());
 // should succeed, fails if exception thrown
+
+ fs.deleteOnExit(parentDir);
+ fs.close();
 }
 
 @Test{code}
Bug: Hadoop distcp throws "ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found"
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/May/20 17:41
Affects Version/s: 3.1.3

On Hadoop 3.x, we see following "ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found." message on the first line of the command output when running Hadoop DistCp.
{code:java}
$ hadoop distcp /path/to/src /user/hadoop/
ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found.
2020-05-14 17:11:53,173 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true
..
{code}
This message was added by HADOOP-12857 and it would be an expected behavior.
 DistCp calls 'hadoop_add_to_classpath_tools hadoop-distcp' when [it starts|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/shellprofile.d/hadoop-distcp.sh], and the error is returned because the hadoop-distcp.sh does not exist in the tools directory.

However, that error message confuses us. Since this is not an user end configuration issue, I would think it's better to change the log level to debug (hadoop_debug).
Bug: ABFS: Avoid storage calls to check if the account is HNS enabled or not
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Apr/20 15:34
Affects Version/s: 3.4.0

Each time an FS instance is created a Getacl call is made. If the call fails with 400 Bad request, the account is determined to be a non-HNS account. 

Recommendation is to create a config and be able to avoid store calls to determine account HNS status,

If config is available, use that to determine account HNS status. If config is not present in core-site, default behaviour will be calling getAcl.