av9ash/GitBugs huggingface dataset:
Bug: VersionInfoMojo.byteArrayToString(...) creates malformed hex strings
Status: Open | Priority: Major | Resolution: 
Created: 27/Mar/21 17:20
Affects Version/s: 3.2.2

{{org.apache.hadoop.maven.plugin.versioninfo.VersionInfoMojo.byteArrayToString(byte[])}} creates malformed hex strings:
{code}
private String byteArrayToString(byte[] array) {
    StringBuilder sb = new StringBuilder();
    for (byte b : array) {
        sb.append(Integer.toHexString(0xff & b));
    }
    return sb.toString();
}
{code}

The issue here is that {{toHexString}} returns only a single hex char if the value is < 16. Therefore {{1, 0}} and {{16}} would both have the result {{"10"}}.

The correct implementation would be:
{code}
private String byteArrayToString(byte[] array) {
    StringBuilder sb = new StringBuilder();
    for (byte b : array) {
        int unsignedB = b & 0xff;
        if (unsignedB < 16) {
            sb.append('0');
        }
        sb.append(Integer.toHexString(unsignedB));
    }
    return sb.toString();
}
{code}
Bug: netgroup-user is not added to Groups.cache
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Jan/21 22:38
Affects Version/s: 3.4.0

After the optimization in HADOOP-17079, both {{JniBasedUnixGroupsNetgroupMapping}} and {{ShellBasedUnixGroupsNetgroupMapping}} do not implement {{getGroupSet}}.
As a result, {{Groups.load()}} load the cache calling {{fetchGroupSet}} which yield to the superclass {{JniBasedUnixGroupsMapping}} / {{ShellBasedUnixGroupsMapping}}.
In other words, the groups mapping will never fetch from {{NetgroupCache}}.

This alters the behavior of the implementation. Is there a reason to bypass loading. CC: [~xyao]

This jira is to add missing implementation {{getGroupSet}} to {{JniBasedUnixGroupsNetgroupMapping}} and {{ShellBasedUnixGroupsNetgroupMapping}} .
Bug: Correct abfs test assertion reversed in HADOOP-13327
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/21 08:55
Affects Version/s: 3.3.1

HADOOP-13327 introduces, among other changes, functions to simplify assert for checking stream capabilities. This PR fixes (originally) assertFalse statements whose logic has been reversed to assertTrue when the above change was checked-in.
Bug: ABFS: Fix boundary conditions in InputStream seek and skip
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 12/Feb/21 09:51
Affects Version/s: 3.3.0

Modify AbfsInputStream seek method to throw EOF exception on seek to contentLength for a non-empty file. With this change, it will no longer be possible for the inputstream position (as obtained by getPos() API) to be moved to contentlength manually, except post reading the last byte.
Bug: Yarn Job execution get failed when LZ4 Compression Codec is used
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/21 13:35

When we try to compress a file using the LZ4 codec compression type then the yarn job gets failed with the error message :
{code:java}
net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
{code}
Bug: NullPointerException from register in MetricsSystemImpl
Status: Open | Priority: Major | Resolution: 
Created: 14/Jan/21 10:58

This is an error from Ozone's unit test case [HDDS-4688|https://github.com/apache/ozone/pull/1795#issuecomment-760052788]. The error is as follows:

 
{code:java}
java.lang.NullPointerException: configjava.lang.NullPointerException: config
 at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:897) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSink(MetricsSystemImpl.java:298) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:277) at org.apache.hadoop.hdds.server.http.BaseHttpServer.start(BaseHttpServer.java:298)
...{code}
The reason should be happened here "https://github.com/apache/hadoop/commit/2f500e4635ea4347a55693b1a10a4a4465fe5fac#", if the _name_ is contained in _allSinks_ but not in _sink_, it will invoke the method of _registerSink_. But if the variable of config is null, it will throw the exception of NullPointerException.

 

A suggestion would be checking if the variable of config is null before calling _registerSink._
Bug: Yetus does not run qbt-trunk
Status: Resolved | Priority: Major | Resolution: Done
Created: 22/Feb/21 18:16

On Feb20th, qbt-reports started to generate empty reports

{code:bash}
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/424/
ERROR: File 'out/email-report.txt' does not exist
{code}

On Jenkins, the job fails with the following error:
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/425/console
{code:bash}
ERROR: /home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64//dev-support/bin/hadoop.sh does not exist.
Build step 'Execute shell' marked build as failure
Archiving artifacts
[Fast Archiver] No prior successful build to compare, so performing full copy of artifacts
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
{code}

[~aajisaka], I think this would be caused by HADOOP-16748 . I noticed that the PR of that HADOOP-16748 ceased from showing any reports, but for some reason I forgot about that while reviewing.
Bug: Misconfigure buffer size to 0 causing TestDFSIO to hang in infinite loops
Status: Open | Priority: Major | Resolution: 
Created: 11/Mar/21 07:18
Affects Version/s: 3.3.0

TestDFSIO lacks sanity checks on input buffer size for function doIO()

If misconfigured to 0, doIO() hangs in infinite loops. 

 
{code:java}
public Long doIO(... ) throws IOException {
    …..
 while (actualSize < totalSize) {
       int curSize = in.read(buffer, 0, bufferSize);
       if(curSize < 0) break;
       actualSize += curSize;
     ….
     }
   …
}

{code}
{code:java}
public Long doIO(... ) throws IOException {
    …..
        for (...; nrRemaining > 0; nrRemaining -= bufferSize) {
           If  (bufferSize < nrRemaining){ curSize = bufferSize }else{
                curSize = (int)nrRemaining
           out.write(buffer, 0, curSize);
        }
   …
}

{code}
Similar parameter _Io.file.buffer.size_ is handled by Java IO library BufferedInputStream. We suggest adding sanity checks for the benchmark's buffer size as well.
Bug: Build failure on trunk
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Mar/21 10:51
Affects Version/s: 3.4.0

Build is broken on trunk:

hadoop-huaweicloud: Compilation failure

[ERROR] hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSFileSystem.java:[396,58] incompatible types: org.apache.hadoop.util.BlockingThreadPoolExecutorService cannot be converted to org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService
Bug: Job will hang when disk for logging is full
Status: Open | Priority: Major | Resolution: 
Created: 04/Mar/21 09:24
Affects Version/s: 3.3.0

When we put the log directory of Hadoop on a devoted disk space and the disk is closed to full. The job we run will suspend and never abort. 

The job we run is 'share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar'

 

From jps, seems everything is running normally

hadoop@lily-OptiPlex-3070:~/hadoop-3.3.0$ jps
 23409 NameNode
 23889 SecondaryNameNode
 24338 NodeManager
 23622 DataNode
 14408 RunJar
 29098 HRegionServer
 14526 Jps
Bug: hadoop-client-runtime-3.3.0 not properly shading javax.xml.bind
Status: Open | Priority: Major | Resolution: 
Created: 22/Feb/21 16:46
Affects Version/s: 3.3.0

In the file hadoop-client-modules/hadoop-client-runtime/pom.xml, it excludes javax.xml.bind because it says it is part of the SE JDK.  But, this is no longer true in Java 11. 

 
|<exclusions>|
| |<!-- these APIs are a part of the SE JDK -->|
| |<exclusion>|
| |<groupId>javax.xml.bind</groupId>|
| |<artifactId>jaxb-api</artifactId>|
| |</exclusion>|

 

So, when I try to use Guice to inject objects at start up,  I am getting the error

Error injecting constructor, java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/javax/xmlbind/ModuleUtil (wrong name: javax/xml/bind/ModuleUtil)

 

I was able to get past this by using zip to change the position of the ModuleUtil.class inside the META-INF directory.

Before I made any changes:
 jar tvf hadoop-client-runtime-3.3.0.jar | grep ModuleUtil*

META-INF/versions/9/javax/xml/bind/ModuleUtil.class
 org/apache/hadoop/shaded/javax/xml/bind/ModuleUtil.class

After I modified the jar:

jar tvf hadoop-client-runtime-3.3.0-MODIFIED.jar | grep ModuleUtil*

META-INF/versions/9/org/apache/hadoop/shaded/javax/xml/bind/ModuleUtil.class
 org/apache/hadoop/shaded/javax/xml/bind/ModuleUtil.class

With the revised jar file, my code started.  I don't know if that is the correct way to fix but it works.
Bug: magic committer to be enabled for all S3 buckets
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/21 10:36
Affects Version/s: 3.3.0

* core-default.xml updated so that fs.s3a.committer.magic.enabled = true
* CommitConstants updated to match
* All tests which previously enabled the magic committer now rely on
  default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
  want to disable it.
Note: this doesn't switch to using the committer -it just enables the path
rewriting magic which it depends on.
Bug: Non-randomized password used
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 05/Feb/21 19:09

In file [https://github.com/apache/hadoop/blob/a89ca56a1b0eb949f56e7c6c5c25fdf87914a02f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java] (at Line 322) non-randomized password is used.

*Security Impact*:

Hackers can get access to the non-randomized passwords and compromise the system.

*Solution we suggest*:

Password should be generated randomly

*Please share with us your opinions/comments if there is any*:

Is the bug report helpful?
Bug: BZip2Codec incorrectly throws IndexOutOfBoundsException: offs(X) + len(X+1) > dest.length(Y).
Status: Open | Priority: Major | Resolution: 
Created: 02/Jan/21 04:18
Affects Version/s: 3.1.2

In org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream
 around line 496 seems to mistakenly add the offset to the length.
{noformat}
      if (this.posSM == POS_ADVERTISEMENT_STATE_MACHINE.ADVERTISE) {
        result = this.input.read(b, off, off + 1 << HERE);
{noformat}

Here's a reference [BZip2Codec.java:L496|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BZip2Codec.java#L496]
Bug: New connection requires a retry to refresh NameNode IP changes
Status: Patch Available | Priority: Major | Resolution: 
Created: 28/Jan/21 20:13
Affects Version/s: 2.8.0

Hadoop-17068 is to handle the case of NameNode IP address changes in which HDFS client will update the IP address after the connection failure.  

DataNodes also use the same logic to refresh IP address for the connection. Such connection is reused with the default idle time 10 seconds. (set by ipc.client.connection.maxidletime). If the connection is closed and the DataNode will use the old NameNode IP address to connect and refresh to the new IP address after the first failure.  

The problem with the refresh logic in org.apache.hadoop.ipc.Client is: the server value getting refreshed will not reflect in remoteId.address, while the next connection creation will use remoteId.address.

{{if (!server.equals(currentAddr)) {}}
{{  LOG.warn("Address change detected. Old: " + server.toString() +}}
{{          " New: " + currentAddr.toString()); }}
{{   server = currentAddr;}}

 

Such kind of retry in a big cluster will cause random "BLOCK* blk_16987635027_18010098516 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in fie" error if all three replicas take one retry to read/write the block.
Bug: renaming S3A Statistic DELEGATION_TOKENS_ISSUED to DELEGATION_TOKEN_ISSUED broke tests downstream
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/21 20:49
Affects Version/s: 3.3.1

HADOOP-16830/HADOOP-17271 renamed DELEGATION_TOKENS_ISSUED to DELEGATION_TOKENS_ISSUED while trying to unify naming. This breaks downstream code.

Fix: revert back the name change.
Bug: Implement listFiles(path, recursive=true)
Status: Open | Priority: Major | Resolution: 
Created: 25/Jan/21 10:33
Affects Version/s: 3.3.0

Azure storage and the abfs client offer recursive listings, but currently there's no listFiles() method to connect to it instead a recursive treewalk is used.

Implementing a listFiles(path, recursive) could offer significant speedups to applications which use it, for example HIVE-24669, and again, async prefetching could provide even more benefits

Would need changes in {{org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore}} to join everything up
Bug: Race condition refreshing NetgroupCache 
Status: Open | Priority: Major | Resolution: 
Created: 22/Jan/21 15:57
Affects Version/s: 2.10.1

There is potential concurrency bug in the {{NetgroupCache}} implementation.

{{NetgroupCache}} is static. When ACL is built, its groups will be added to the {{NetgroupCache}}.

A {{-refreshUserToGroupsMappings}} forces the cache to reload the users for each group.
 This is done by first getting the keys, clearing the cache, then finally reloading the users for each group.
 The problem that the three steps are not atomic.
 Adding ACLs concurrently may take place between L80-L81 ([JniBasedUnixGroupsNetgroupMapping#L79|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.java#L79]). This results in the loss of the most recently added group.
 Since group names are used in the JNI level, the users of that group won't be retrieved.
{code:java}
78 @Override
79  public void cacheGroupsRefresh() throws IOException {
80    List<String> groups = NetgroupCache.getNetgroupNames();
81     NetgroupCache.clear();
82    cacheGroupsAdd(groups);
83  }
{code}
+Solution:+

Refreshing {{NetgroupCache}} should not clear the cache keys.
Bug: Groups user mapping based on Netgroup do not auto-refresh 
Status: Open | Priority: Major | Resolution: 
Created: 20/Jan/21 20:12

It was noticed that a manual refresh ({{yarn rmadmin -refreshUserToGroupsMappings}}) is required for services to pick a user-group membership.
The auto-refresh of the {{Groups.caching}} only works for the upper layer and won't fetch the groups from the system.

There is a need to automatically reload all the groups (currently in {{NetgroupCache}}). The interval can be a new configuration, with a default value of 6 hours.
Bug: Replace deprecated getGroups with getGroupsSet
Status: Open | Priority: Major | Resolution: 
Created: 22/Jan/21 15:09
Affects Version/s: 2.10.1

This ticket is a follow up to HADOOP-17079 which added an optimization to use sets instead of lists retrieving user groups.
However, the deprecated getGroups is still being called throughout the code.
This is an attempt to clean up the code.

CC: [~xyao], [~Jim_Brennan]
Bug: Provide fallbacks for callqueue ipc namespace properties
Status: Open | Priority: Major | Resolution: 
Created: 21/Jan/21 20:16
Affects Version/s: 3.1.4

Filing this proposal on behalf of [~daryn], based on comments he made in one of our internal Jiras.

The following settings are currently specified per port:
{noformat}
  /**
   * CallQueue related settings. These are not used directly, but rather
   * combined with a namespace and port. For instance:
   * IPC_NAMESPACE + ".8020." + IPC_CALLQUEUE_IMPL_KEY
   */
  public static final String IPC_NAMESPACE = "ipc";
  public static final String IPC_CALLQUEUE_IMPL_KEY = "callqueue.impl";
  public static final String IPC_SCHEDULER_IMPL_KEY = "scheduler.impl";
  public static final String IPC_IDENTITY_PROVIDER_KEY = "identity-provider.impl";
  public static final String IPC_COST_PROVIDER_KEY = "cost-provider.impl";
  public static final String IPC_BACKOFF_ENABLE = "backoff.enable";
  public static final boolean IPC_BACKOFF_ENABLE_DEFAULT = false;
 {noformat}
If one of these properties is not specified for the port, the defaults are hard-coded.
It would be nice to provide a way to specify a fallback default property that would be used for all ports.  If the property for a specific port is not defined, the fallback would be used, and if the fallback is not defined it would use the hard-coded defaults.

We would likely need to make the same change for properties specified by these classes.  For example, properties used in WeightedTimeCostProvider.

The fallback properties could be specified by dropping the port from the property name.  For example, the fallback for {{ipc.8020.cost-provider.impl}} would be {{ipc.cost-provider.impl}}.
Another option would be to use something more explicit like {{ipc.default.cost-provider.impl}}.
Bug: [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/21 14:40
Affects Version/s: 3.3.0

Test failed against ireland intermittently with the following config:

{{mvn clean verify -Dparallel-tests -DtestsThreadCount=8}}
xml based config in auth-keys.xml:
{code:xml}
    <property>
        <name>fs.s3a.metadatastore.impl</name>
        <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
    </property>
{code}
Bug: Hadoop Client getRpcResponse May Return Wrong Result
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 08/Jan/21 18:47

{code:java|Title=Client.java}
  /** @return the rpc response or, in case of timeout, null. */
  private Writable getRpcResponse(final Call call, final Connection connection,
      final long timeout, final TimeUnit unit) throws IOException {
    synchronized (call) {
      while (!call.done) {
        try {
          AsyncGet.Util.wait(call, timeout, unit);
          if (timeout >= 0 && !call.done) {
            return null;
          }
        } catch (InterruptedException ie) {
          Thread.currentThread().interrupt();
          throw new InterruptedIOException("Call interrupted");
        }
      }

 */
  static class Call {
    final int id;               // call id
    final int retry;           // retry count
...
    boolean done;               // true when call is done
...
}
{code}

The {{done}} variable is not marked as {{volatile}} so the thread which is checking its status is free to cache the value and never reload it even though it is expected to change by a different thread.  The while loop may be stuck waiting for the change, but is always looking at a cached value.  If that happens, timeout will occur and then return 'null'.

In previous versions of Hadoop, there was no time-out at this level, so it would cause endless loop.  Really tough error to track down if it happens.
Bug: Usage of incorrect regex range A-z
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Feb/21 21:44
Affects Version/s: 3.4.0

There are two cases where the regex {{A-z}} is used. I assume that is a typo (and should be {{A-Z}}) because {{A-z}} matches:
- {{A-Z}}
- {{\[}}, {{&#92;}}, {{\]}}, {{^}}, {{_}}, {{`}}
- {{a-z}}

Affected:
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/util/Check.java#L109
(and https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/util/Check.java#L115)
- https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/resourcetypes/ResourceTypesTestHelper.java#L38
Bug: Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Feb/21 06:49
Affects Version/s: 2.10.2

The HttpAuthentication.md document says "The same secret should be used for all nodes in the cluster, ResourceManager, NameNode, DataNode and NodeManager"  but the secret should be different for each service. This description is updated in [core-default.xml|https://github.com/apache/hadoop/commit/d82009599a2e9f48050e0c41440b36c759ec068f#diff-268b9968a4db21ac6eeb7bcaef10e4db744d00ba53989fc7251bb3e8d9eac7df] but has to be updated in HttpAuthentication.md as well.
Bug: AbstractContractStreamIOStatisticsTest fails if read buffer not full
Status: Open | Priority: Minor | Resolution: 
Created: 27/Jan/21 15:15
Affects Version/s: 3.3.1

{code}
[ERROR] testInputStreamStatisticRead(org.apache.hadoop.fs.s3a.statistics.ITestS3AContractStreamIOStatistics)  Time elapsed: 6.252 s  <<< FAILURE!
org.junit.ComparisonFailure: [Counter named stream_read_bytes with expected value 129] expected:<[129]L> but was:<[3]L>
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
{code}

Test should handle all cases where bytes read > 0
Bug: S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jan/21 14:11
Affects Version/s: 3.4.0

A test run with network problems caught exceptions "com.amazonaws.SdkClientException: Data read has a different length than the expected:", which then escalated to failure.

these should be recoverable if they are recognised as such. 

translateException could do this. Yes, it would have to look @ the text, but as {{signifiesConnectionBroken()}} already does that for "Failed to sanitize XML document destined for handler class", it'd just be adding a new text string to look for.