av9ash/GitBugs huggingface dataset:
Bug: Enhance client protocol to propagate last seen state IDs for multiple nameservices.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/22 16:52
Affects Version/s: 3.4.0

The RPCHeader in the client protocol currently contains a single value to indicate the last seen state ID for a namenode.
{noformat}
optional int64 stateId = 8; // The last seen Global State ID
{noformat}
When there are multiple namenodes, such as in router based federation, the headers need to carry the state IDs for each of these nameservices that are part of the federation.

This change is a prerequisite for HDFS-13522: RBF: Support observer node from Router-Based Federation
Bug: Fix bug preventing hadoop-metrics2 from emitting metrics to > 1 Ganglia servers.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/22 13:12
Affects Version/s: 3.2.4

AbstractGangliaSink is used by the hadoop-metrics2 package to emit metrics to Ganglia. Currently, this class uses the apache commons-configuration package to read from the hadoop-metrics2.properties file. commons-configuration is outdated, and has a bug where the .getString function drops everything after the first comma.
Bug: All method metrics related to the RPC protocol should be initialized
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/22 10:19
Affects Version/s: 3.4.0

When an RPC protocol is used, the metric of protocol-related methods should be initialized; otherwise, metric information will be incomplete. For example, when we call HAServiceProtocol#monitorHealth(), only the metric of monitorHealth() are initialized, and the metric of transitionToStandby() are still not reported. This incompleteness caused a little trouble for our monitoring system.
The root cause is that the parameter passed by RpcEngine to MutableRatesWithAggregation#init(java.lang.Class<?>)  is always XXXProtocolPB, which is inherited from BlockingInterface and does not implement any methods. We should fix this bug.
Bug: hadoop-aws maven build to add a prefetch profile to run all tests with prefetching
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/22 13:35
Affects Version/s: 3.4.0

Add a prefetch profile to the hadoop-aws build so tests run with prefetching on, similar to how the markers option does

makes it easy to test everything with prefetching on/off without editing xml files.
Bug: fs.s3a.prefetch.block.size to be read through longBytesOption
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/22 14:02
Affects Version/s: 3.3.6

use   {{longBytesOption(fs.s3a.prefetch.block.size)}}

this allows for unitgs like M to be used, and is consistent with the rest of the s3a size params. also sets a minimum size to prevent negatives values
Bug: Codecs with @DoNotPool annotation are not closed causing memory leak
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Aug/22 12:18
Affects Version/s: 3.3.2

Compressors and Decompressions with a @DoNotPool annotation are not closed when they are returned to the CodecPool, which causes a native memory leak.

 

I have included a link to a [Demo Project|https://github.com/kevins-29/hadoop-gzip-memory-leak] demonstrating the leak
Bug: Allow dynamic groupSearchFilter in LdapGroupsMapping
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/22 12:31
Affects Version/s: 3.4.0

As of now the lookupGroup() method doesn't allow to have placeholders in 

groupSearchFilter, so that can not be dynamically adjusted.

If we have placeholders for groupSearchFilter like: (&(|(XYZ=\{0})(ABC=\{1}))(objectClass=posixGroup))

This fails here:

 
{code:java}
groupResults =
    c.search(groupbaseDN,
        "(&" + groupSearchFilter + "(" + groupMemberAttr + "={0}))",
        new Object[]{userDn},
        SEARCH_CONTROLS); {code}
With 

 

 
{noformat}
javax.naming.directory.InvalidSearchFilterException: number exceeds argument list: 1; remaining name {noformat}
 

>>Dropped off or changed the details above which I thought won't be safe to disclose.
Bug: Shutdown AWSSecurityTokenService when its resources are no longer in use
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Aug/22 05:56
Affects Version/s: 3.3.5

AWSSecurityTokenService resources can be released whenever they are no longer in use. The documentation of AWSSecurityTokenService#shutdown says while it is not important for client to compulsorily shutdown the token service, client can definitely perform early release whenever client no longer requires token service resources. We achieve this by making STSClient closable, so we can certainly utilize it in all places where it's suitable.
Bug: S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/22 12:46
Affects Version/s: 3.4.0

prefetching stream's SingleFilePerBlockCache uses Files.tempFile() to allocate a temp file.

it should be using LocalDirAllocator to allocate space from a list of dirs, taking a config key to use. for s3a we will use the Constants.BUFFER_DIR option, which on yarn deployments is fixed under the env.LOCAL_DIR path, so automatically cleaned up on container exit
Bug: Adds alignment context to call path for creating RPC proxy with multiple connections per user.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/22 18:50
Affects Version/s: 3.3.5

HDFS-13274 (RBF: Extend RouterRpcClient to use multiple sockets) gets the RPC proxy using methods which do not allow using an alignment context. These methods were added in HADOOP-13144 (Enhancing IPC client throughput via multiple connections per user).

This change adds an alignment context as an argument for methods in the call path for creating the proxy.
Bug: Improve the accuracy of MutableStat mean
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/22 15:36
Affects Version/s: 3.4.0

The current MutableStat mean calculation method is more prone to loss accuracy because the sum of samples is too large. 
Storing large integers in the double type results in a loss of accuracy. For example, 9223372036854775707 and 9223372036854775708 are both stored as doubles as 9223372036854776000. Therefore, we should try to avoid using the cumulative total sum method to calculate the average, but update the average every time we sample. All in all, we can process each sample on its own to improve mean accuracy.
Bug: Parameterize platform toolset version
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/22 12:40
Affects Version/s: 3.4.0

The *winutils*, *libwinutils* and *native* project structures are currently defined in *.vcxproj* and *.sln* files. For building on Windows, a key parameter is the *PlatformToolsetVersion*. This gets added by the build system by running [dev-support/bin/win-vs-upgrade.cmd|https://github.com/apache/hadoop/blob/c60a900583d6a8d0494980f4bbbf4f95438b741b/dev-support/bin/win-vs-upgrade.cmd]. This essentially runs the following command to detect the PlatformToolsetVersion of the currently installed Visual Studio and uses the same for compilation - https://github.com/apache/hadoop/blob/c60a900583d6a8d0494980f4bbbf4f95438b741b/dev-support/bin/win-vs-upgrade.cmd#L38

{code}
devenv %%f /upgrade
{code}

However, when building with *Dockerfile_windows_10*, only Visual Studio 2019 Build Tools are available (and not the full IDE). The Visual Studio 2019 Build Tools distribution doesn't contain *devenv* and thus, the above command fails to run stating that it couldn't find devenv.

To fix this issue, we need the ability to specify the PlatformToolsetVersion as a Maven option, at which point the *win-vs-upgrade.cmd* won't run and would use the speicified PlatformToolsetVersion against MSBuild.
Bug: MutableGaugeFloat#incr(float) get stuck in an infinite loop
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/22 11:34
Affects Version/s: 3.4.0

The current implementation converts the value from int to float, causing the compareAndSet method to get stuck.
{code:java}
private final boolean compareAndSet(float expect, float update) {
  return value.compareAndSet(Float.floatToIntBits(expect),
      Float.floatToIntBits(update));
}

private void incr(float delta) {
  while (true) {
    float current = value.get();
    float next = current + delta;
    if (compareAndSet(current, next)) {
      setChanged();
      return;
    }
  }
} {code}
 

Perhaps it could be:
{code:java}
private void incr(float delta) {
  while (true) {
    float current = Float.intBitsToFloat(value.get());
    float next = current + delta;
    if (compareAndSet(current, next)) {
      setChanged();
      return;
    }
  }
} {code}
 

The unit test looks like this
{code:java}
MutableGaugeFloat mgf = new MutableGaugeFloat(Context,3.2f);
assertEquals(3.2f, mgf.value(), 0.0);
mgf.incr();
assertEquals(4.2f, mgf.value(), 0.0); {code}
Bug: Fix main thread name.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/22 08:32
Affects Version/s: 3.2.1

The server's main thread is "Listener at ${hostname}/9000", it is confused easily. We can see the main thread like below.
{code:java}
"Listener at ${hostname}/9000" #1 prio=5 os_prio=0 tid=0x00007f8068016000 nid=0x5c086 in Object.wait() [0x00007f806f1d4000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007f7552553010> (a org.apache.hadoop.ipc.ProtobufRpcEngine$Server)
    at java.lang.Object.wait(Object.java:502)
    at org.apache.hadoop.ipc.Server.join(Server.java:3449)
    - locked <0x00007f7552553010> (a org.apache.hadoop.ipc.ProtobufRpcEngine$Server)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.join(NameNodeRpcServer.java:613)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:1014)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1774)
 {code}
Bug: Upgrade snakeyaml to 1.32
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/22 17:24
Affects Version/s: 3.3.3

Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and [CVE-2022-38752|https://github.com/advisories/GHSA-9w3m-gqgf-c4p9]
Bug: Add Support for localized trash for ViewFileSystem in Trash.moveToAppropriateTrash
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/22 16:59
Affects Version/s: 3.3.5

Trash.moveToAppropriateTrash is used by _hadoop cli -rm_ and hive, to move files to trash. However, its current implementation does not support localized trash policy we added to ViewFileSystem in HADOOP-18144.

The reason is in moveToAppropriateTrash, it first resolves a path and then uses the resolvedFs, to initialize the trash. As a result, it uses getTrashRoot() implementation from targetFs, not ViewFileSystem. The new localized trash policy we implemented in ViewFileSystem is not invoked.

With the new localized trash policy for ViewFileSystem, the trash root would be local to a mount point, thus, for ViewFileSystem with this flag turned on, there is no need to resolve the path in moveToAppropriateTrash. Rename in ViewFileSystem can resolve the logical paths correctly and be able to move a file to trash within a mount point. 

Code section of current moveToAppropriateTrash implementation.
{code:java}
public static boolean moveToAppropriateTrash(FileSystem fs, Path p,
    Configuration conf) throws IOException {
  Path fullyResolvedPath = fs.resolvePath(p);
  FileSystem fullyResolvedFs =
      FileSystem.get(fullyResolvedPath.toUri(), conf);
  ...
  Trash trash = new Trash(fullyResolvedFs, conf);
  return trash.moveToTrash(fullyResolvedPath);
}{code}
Bug: AliyunOSS: AliyunOSSBlockOutputStream to support heap/off-heap buffer before uploading data to OSS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/22 02:12
Affects Version/s: 2.10.2

Recently, our customers raise a requirement: AliyunOSSBlockOutputStream should support heap/off-heap buffer before uploading data to OSS.

Currently, AliyunOSSBlockOutputStream buffers data in local directory before uploading to OSS, it is not efficient compared to memory.

Changes:
 # Adds heap/off-heap buffers
 # Adds limitation of memory used, and fallback to disk
Bug: InstrumentedWriteLock should consider Reentrant case 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/22 13:10
Affects Version/s: 3.4.0

During looking into the InstrumentedWriteLock and InstrumentedReadLock, I found that InstrumentedReadLock consider reentrant case when checking log, but InstrumentedWriteLock does not consider reentrant case. 

So I think InstrumentedWriteLock should consider it.
Bug: upgrade jettison json jar due to fix CVE-2022-40149
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/22 16:30
Affects Version/s: 2.10.2

A fix for [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-40149]

 

[https://github.com/jettison-json/jettison/releases/tag/jettison-1.5.1]

[https://github.com/advisories/GHSA-56h3-78gp-v83r]
Bug: Restrict vectoredIO threadpool to reduce memory pressure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/22 22:09

https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L964-L967

Currently, it fetches all the ranges with unbounded threadpool. This will not cause memory pressures with standard benchmarks like TPCDS. However, when large number of ranges are present with large files, this could potentially spike up memory usage of the task. Limiting the threadpool size could reduce the memory usage.
Bug: Vectored IO: Threadpool should be closed on interrupts or during close calls
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 08/Sep/22 03:58
Affects Version/s: 3.3.5

Vectored IO threadpool should be closed on any interrupts or during S3AFileSystem/S3AInputStream close() calls.

E.g Query which got cancelled in the middle of the run. However, in background (e.g LLAP) vectored IO threads continued to run.

 

!Screenshot 2022-09-08 at 9.22.07 AM.png|width=537,height=164!
Bug: Remove the hadoop-openstack module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/22 13:41
Affects Version/s: 3.3.5

 the openstack module doesn't get tested or maintained; it's just something else to keep up to date security wise. As nobody ever files bugs on it it is clearly not being used either.
 
 On-prem object stores support the S3 APIs and/or provide their own hadoop connectors (ozone, IBM).
 
 Let's just cut it completely. As someone who co-authored a lot of it I am happy to do the duty. I will do a quick review of all test to see if there are any left which we could pull into hadoop common...the FS contract tests were initially derived from the ones I did here.
Bug: ITestS3AIOStatisticsContext failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/22 17:18
Affects Version/s: 3.3.5

test failure running the new ITestS3AIOStatisticsContext. attaching the stack and log file.

This happened on a large (12 thread) test run, but i can get it to come back intermittently on repeated runs of the whole suite, but never when i just run the single test case.

{code}
[ERROR] testThreadIOStatisticsForDifferentThreads(org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext)  Time elapsed: 3.616 s  <<< FAILURE!
java.lang.AssertionError: 
[Counter named stream_write_bytes] 
Expecting actual not to be null
        at org.apache.hadoop.fs.statistics.IOStatisticAssertions.lookupStatistic(IOStatisticAssertions.java:160)
        at org.apache.hadoop.fs.statistics.IOStatisticAssertions.assertThatStatisticLong(IOStatisticAssertions.java:291)
        at org.apache.hadoop.fs.statistics.IOStatisticAssertions.assertThatStatisticCounter(IOStatisticAssertions.java:306)
        at org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext.assertThreadStatisticsForThread(ITestS3AIOStatisticsContext.java:367)
        at org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext.testThreadIOStatisticsForDifferentThreads(ITestS3AIOStatisticsContext.java:260)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)
{code}

I'm suspecting some race condition *or* gc pressure is releasing that reference in the worker thread.

proposed test changes
* worker thread changes its thread ID for the logs
* stores its thread context into a field, so there's guarantee of no GC
* logs more as it goes along.
Bug: Fix org.apache.hadoop.io.ArrayWritable
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/22 06:17
Affects Version/s: 3.3.4

When we store an array of string with "new ArrayWritable(String[] strings)", the "valueClass" will be "Text" while the actual value type is "UTF8". This will cause an error on deserialization of ArrayWritable.
Bug: Issues running in dynamic / managed environments
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Aug/22 18:54
Affects Version/s: 3.3.4

Running in dynamic or managed environments is a challenge because we can't assume that all services will have DNS entries, will be started in a specific order, will maintain constant IP addresses, etc.  I'm using the following assumptions to guide the changes necessary to operate in this kind of environment:
 # The configuration files are an expression of desired state
 # If a referenced service instance is not resolvable or reachable at a moment in time, it will be eventually and should be able to participate in the future, as if it had been there originally, without requiring manual intervention
 # IP address changes should be handled in a way that no only allows distributed calls to continue to function, but avoids having to re-resolve the address over and over
 # Code that requires resolved names (Kerberos and DataNode registration) should fall back to DNS reverse lookups to work around temporary issues caused by caching.  Example: The DataNode registration is only performed at startup, and yet the extra check that allows it to succeed in registering with the NameNode isn’t performed
 # If an HA system is supposed to only require a quorum, then we shouldn’t require the full set, allowing the called service to bring the remaining instances into compliance
 # Managing a service should be independent of other services.  Example: You should be able to perform a rolling restart of JournalNodes without worrying about causing an issue with NameNodes as long as a quorum is present.

A proof of these concepts would be the ability to:
 * Start with less that the full replica count of a service, while still providing the required quorum or minimal count, should still allow a cluster to start and function.  Example: 2 out of 3 configured JournalNodes should still allow the NameNode to format, function, rollover to the standby, etc.
 * Introduce missing instances should join the existing cluster without manual intervention.  Example: Starting the 3rd JournalNode should automatically be formatted and brought up to date
 * Perform rolling restarts of individual services without negatively impacting other services (causing failures, restarts, etc.).  Example: Rolling restarts of JournalNodes shouldn't cause problems in NameNodes; Rolling restarts of NameNodes shouldn't cause problems with DataNodes
 * Logs should only report updated IP addresses once (per dependent), avoiding costly re-resolution
Bug: Update previous index properly while validating overlapping ranges. 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/22 19:07

[https://github.com/apache/hadoop/blob/a55ace7bc0c173f609b51e46cb0d4d8bcda3d79d/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java#L201]
Bug: Improve VectoredReadUtils#readVectored() for direct buffers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Aug/22 17:14
Affects Version/s: 3.3.5

harden the VectoredReadUtils methods for consistent and more robust use, especially in those filesystems which don't have the api.

VectoredReadUtils.readInDirectBuffer should allocate a max buffer size, .e.g 4mb, then do repeated reads and copies; this ensures that you don't OOM with many threads doing ranged requests. other libs do this.

readVectored to call validateNonOverlappingAndReturnSortedRanges before iterating

this ensures the abfs/s3a requirements are always met, and that because ranges will be read in order, prefetching by other clients will keep their performance good.

readVectored to add special handling for 0 byte ranges
Bug: AliyunOSS: AliyunOSSFileSystemStore deleteObjects interface won't return the objects that failed to delete
Status: Open | Priority: Major | Resolution: 
Created: 02/Sep/22 07:26
Affects Version/s: 2.10.2

AliyunOSSFileSystemStore deleteObjects interface used "simple mode" to do batch delete, which  won't return the objects that failed to delete. So, all the failed objects won't be real deleted in the following retries.

  !hadoop fs oss issue.jpg|width=520,height=571!
Bug: DistCP: Aggregate IOStatistics Counters in MapReduce Counters
Status: Open | Priority: Major | Resolution: 
Created: 27/Jul/22 16:54
Affects Version/s: 3.3.5

Distcp can collect IOStatisticsContext counter values and report them to the console. it can't do the timings in min/mean/max though, as there's no way to aggregate them properly.

# Publish statistics to MapReduce counters in the tasks within CopyMapper.copyFileWithRetry(). 
# The counters will be automatically logged in Job.monitorAndPrintJob() when DistCp is executed with the -verbose option; no need for changes there.
# We could also publish the iOStatistic means by publishing sample count and total sum as two separate counters
# In AbstractContractDistCpTest, add an override point for subclasses to list which metrics they will issue; assert that values are generated.