av9ash/GitBugs huggingface dataset:
Bug: RPC FairCallQueue for special users
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 04/Nov/20 10:06

In HADOOP-15016, the idea was first raised to support special users by assigning each special user an independent queue with a share. The design was intended for the user to better control the RPC schedule, but there is also a risk that users may add a lot of items in the config of special-users, causing a lot of queues in the RPCScheduler.

This ticket records some ideas to mitigate the risks while solving the special-user problem based on HADOOP-15016.

0. The current implementation is as follows, all users will be treated equally, _multiplexer_ will decide the call count in each queue.

!Implement 0.png!

1. The first idea is to amplify the weight of super-users and resue the initial queues. This idea is easy to implement, but ordinary users and special users would be affected by each other, and it would be difficult for the _multiplexer_ to guarantee the traffic of super-suers.
!Implement 1.png!

2. The second idea is to set up one independent queue for all special users with a config controlling the weight of all special-users. One concern for this idea is that the scheduler between super-users' calls may not be fair.
!Implement 2.png!
3. The third idea is to also use priority queues for special-users based on idea 2, ensuring the fair handling of all super-users. Another benefit of this idea is we can use the queues to implement cost-based calculation.

!Implement 3.png!

I think Idea 3 should be a good balance of complexity and useability.
Bug: MetricsRecordFiltered error
Status: Open | Priority: Major | Resolution: 
Created: 28/Oct/20 05:51
Affects Version/s: 3.2.1

 Got sink exception,when set  datanode.sink.ganglia.metric.filter.exclude=metricssystem in hadoop-metrics2.properties ,

java.lang.ClassCastException: org.apache.hadoop.metrics2.impl.MetricsRecordFiltered$1 cannot be cast to java.util.Collection
 at org.apache.hadoop.metrics2.sink.ganglia.GangliaSink30.putMetrics(GangliaSink30.java:165)
 at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:184)
 at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
 at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
 at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:135)
 at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:89)


//////////////////////////////////////////////////
This case can show the exception
	public static void main(String[] args) {
		 List<AbstractMetric> metricsd=new LinkedList<AbstractMetric>();		 
		 MetricsInfo info=MsInfo.ProcessName;
		 long timestamp=System.currentTimeMillis();
         List<MetricsTag> tags=new LinkedList<>();		 
		 org.apache.hadoop.metrics2.impl.MetricsRecordImpl recordimp = new MetricsRecordImpl(info, timestamp, tags, metricsd);
		 MetricsFilter filter=new RegexFilter();         
		 MetricsRecordFiltered  recordfilter=new MetricsRecordFiltered(recordimp,filter);		
		 SubsetConfiguration conf=new SubsetConfiguration(new PropertyListConfiguration(),"test");
		 conf.addProperty(AbstractGangliaSink.SUPPORT_SPARSE_METRICS_PROPERTY, true);
		 GangliaSink30  ganliasink=new GangliaSink30();
		 ganliasink.init(conf);		 
		 ganliasink.putMetrics(recordfilter);
		
	}

///////////////////////////////////////////////////////////////
The root cause is:
 Gets a Iterable object in  MetricsRecordFiltered.java:
 @Override public Iterable<AbstractMetric> metrics() {
    return new Iterable<AbstractMetric>() {
      final Iterator<AbstractMetric> it = delegate.metrics().iterator();
      @Override public Iterator<AbstractMetric> iterator() {
        return new AbstractIterator<AbstractMetric>() {
          @Override public AbstractMetric computeNext() {
            while (it.hasNext()) {
              AbstractMetric next = it.next();
              if (filter.accepts(next.name())) {
                return next;
              }
            }
            return (AbstractMetric)endOfData();
          }
        };
      }
    };
  }

but convert to Collection in GangliaSink30.java line 164
        Collection<AbstractMetric> metrics = (Collection<AbstractMetric>) record
            .metrics();
Bug: Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" and HADOOP-16785. "Improve wasb and abfs resilience on double close() calls. followup to abfs close() fix." to branch-2.10
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/20 23:32
Affects Version/s: 2.10.1

Backport:

 HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" to branch-2.10

HADOOP-16785. "Improve wasb and abfs resilience on double close() calls. followup to abfs close() fix."

from branch-3.2 to branch-2.10
Bug: Work with externally managed user credentials
Status: Open | Priority: Major | Resolution: 
Created: 03/Nov/20 18:09

We don't have a good test coverage for externally managed user credentials. It's not clear how someone could ingest kerberos credentials or delegation tokens that are externally managed. It's not clear what services/file system implementation supports them.

File this Jira to track all relevant fixes/support/test/doc
 #  what is supported.
 #  test coverage
 #  document for best practices (how to do it right)
Bug: Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" to branch-3.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/20 22:00
Affects Version/s: 3.2.1

Backport:

 HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" to branch-3.2
Bug: Reduce port collisions in JUnits
Status: Patch Available | Priority: Major | Resolution: 
Created: 19/Oct/20 21:27

Several Junit tests passes a non-zero port number to {{ServerSocketUtil.getPort()}}.
I noticed that those non-zero values are shared between test methods. Based, on the implementation of {{ServerSocketUtil.getPort()}}, the port is checked then, the socket is closed immediately.
Two concurrent tests can have the same port number, but one of them fails with binding exceptions.

Passing a zero port number reduces the probability of returning the same port numbers in two different tests, because the {{ServerSocketUtil.getPort()}} will call a random port number.

An example of that is as described in HDFS-15618. {{testRead()}} between {{TestBlockTokenWithDFSStriped}} and {{TestBlockTokenWithDFS}} where port 19870 is used by the two test cases.
Bug: Upgrade JUnit to 4.13.1
Status: Open | Priority: Major | Resolution: 
Created: 20/Oct/20 05:19

JUnit < 4.13.1 are vulnerable to CVE-2020-15250. The severity is low but it is worth upgrading.
https://github.com/junit-team/junit4/security/advisories/GHSA-269g-pwp5-87pp
Bug: start-build-env.sh hangs on Mac when the uid is too large
Status: Open | Priority: Major | Resolution: 
Created: 16/Oct/20 10:29

If the uid is too large, Docker hangs and consumes all the disk space in: 
{noformat:title=start-build-env.sh}
RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
{noformat}
Reference: https://github.com/docker/for-mac/issues/2038
Bug:  S3A to always probe S3 in S3A getFileStatus on non-auth paths
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Oct/20 13:12
Affects Version/s: 3.3.1

an incidental part of HDP-13230 was a fix to innerGetFileStatus, wherein after a HEAD request we would update the DDB record, so resetting it's TTL

Applications which did remote updates of buckets without going through s3guard are now triggering failures in applications in the cluster when they go to open the file
Bug: FileSystem.DirListingIterator.next() call should return NoSuchElementException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/20 07:22

FileSystem.DirListingIterator.next() call should return NoSuchElementException rather than IllegalStateException

 

Stacktrace for new test failure:

 
{code:java}
java.lang.IllegalStateException: No more items in iteratorjava.lang.IllegalStateException: No more items in iterator at com.google.common.base.Preconditions.checkState(Preconditions.java:507) at org.apache.hadoop.fs.FileSystem$DirListingIterator.next(FileSystem.java:2232) at org.apache.hadoop.fs.FileSystem$DirListingIterator.next(FileSystem.java:2205) at org.apache.hadoop.fs.contract.ContractTestUtils.iteratorToListThroughNextCallsAlone(ContractTestUtils.java:1495) at org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testListStatusIteratorFile(AbstractContractGetFileStatusTest.java:366)
{code}
 

CC [~stevel@apache.org]
Bug: DistCp should fail fast if the configuration is not valid
Status: Open | Priority: Major | Resolution: 
Created: 05/Oct/20 11:27

If the number of Map task is more than 200 and using dynamic option, DistCp always fails by DynamicInputFormat#validateNumChunksUsing. This failure is after building the src/dst paths, which takes more than 10 minutes if the directory is large (i.e. DistCp fails by configuration error after more than 10 minutes).

I think DistCp should validate the configuration before building the src/dst paths and fail if the configuration is not valid.
Bug: Bump up snappy-java to 1.1.8.2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Dec/20 20:47
Affects Version/s: 3.3.1

1.1.8.2 includes:

* Support Apple Silicon (M1, Mac-aarch64)
* Fixed the pure-java Snappy fallback logic when no native library for your platform is found.
Bug: Netty library causing issue while running spark on yarn in PSuedodistributed mode
Status: Patch Available | Priority: Minor | Resolution: 
Created: 28/Dec/20 05:28
Affects Version/s: 3.2.1

While running in psuedodistributed mode,there is different version of netty-all library in hdfs (4.0.52), this is fine till you try to run spark on it. The spark 3.0.1 comes with netty-all 4.1.47.

Now both the libraries get loaded in classpath and thus spark submit fails with method error.
Bug: hadoop-cloud-storage transient dependencies need review
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 09/Nov/20 16:54
Affects Version/s: 3.4.0

A review of the hadoop cloud storage dependencies shows that things are creeping in there

{code}
[INFO] |  +- org.apache.hadoop:hadoop-cloud-storage:jar:3.1.4:compile
[INFO] |  |  +- (org.apache.hadoop:hadoop-annotations:jar:3.1.4:compile - omitted for duplicate)
[INFO] |  |  +- org.apache.hadoop:hadoop-aliyun:jar:3.1.4:compile
[INFO] |  |  |  \- com.aliyun.oss:aliyun-sdk-oss:jar:3.4.1:compile
[INFO] |  |  |     +- org.jdom:jdom:jar:1.1:compile
[INFO] |  |  |     +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |  |  |     |  \- stax:stax-api:jar:1.0.1:compile
[INFO] |  |  |     +- com.aliyun:aliyun-java-sdk-core:jar:3.4.0:compile
[INFO] |  |  |     +- com.aliyun:aliyun-java-sdk-ram:jar:3.0.0:compile
[INFO] |  |  |     +- com.aliyun:aliyun-java-sdk-sts:jar:3.0.0:compile
[INFO] |  |  |     \- com.aliyun:aliyun-java-sdk-ecs:jar:4.2.0:compile
[INFO] |  |  +- (org.apache.hadoop:hadoop-aws:jar:3.1.4:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.hadoop:hadoop-azure:jar:3.1.4:compile - omitted for duplicate)
[INFO] |  |  +- org.apache.hadoop:hadoop-azure-datalake:jar:3.1.4:compile
[INFO] |  |  |  \- com.microsoft.azure:azure-data-lake-store-sdk:jar:2.2.7:compile
[INFO] |  |  |     \- (org.slf4j:slf4j-api:jar:1.7.21:compile - omitted for conflict with 1.7.30)
{code}

Need to review and cut things which come in hadoop-common (slf4j, maybe some of the allyun stuff)
Bug: ITestS3AContractSeek.teardown closes FS before superclass does its cleanup
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 16/Nov/20 19:53
Affects Version/s: 3.3.0

ITestS3AContractSeek.teardown closes the FS, but because it does it before calling super.teardown, the superclass doesn't get the opportunity to delete the test dirs.

Proposed: change the order.
Bug: Move dedicated pre-logging statements into existing logging guards
Status: Open | Priority: Minor | Resolution: 
Created: 03/Oct/20 13:27

I find some cases where some pre-processing statements dedicated to logging calls are not guarded by existing logging guards. Most of them are easy to fix. And the performance and maintainability of these logging calls can be improved to some extend. So I create a PR to fix them.

These issues are detected by a static analysis tool wrote by myself. This tool can extract all the dedicated statements for each debug-logging calls (i.e., the results of these statements are only used by debug-logging calls). Because I realize that debug logs will incur overhead in production, such as string concatenation and method calls in the parameters of logging calls as well as pre-processing statements. And I want to perform a systematic evaluation for the overhead of debugging logging calls in production.
Bug: S3A ITestPartialRenamesDeletes.testRenameDirFailsInDelete failure: missing directory marker
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 30/Nov/20 18:23
Affects Version/s: 3.3.1

Seemingly transient failure of the test ITestPartialRenamesDeletes with the latest HADOOP-17244 changes in: an expected directory marker was not found.

Test run was (unintentionally) sequential, markers=delete, s3guard on
{code}
-Dmarkers=delete -Ds3guard -Ddynamo -Dscale 
{code}

Hasn't come back since.

The bucket's retention policy was authoritative, but no dirs were declared as such
Bug: s3a listing operation will fail in async prefetch if fs closed
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 28/Oct/20 16:53
Affects Version/s: 3.4.0

The async prefetch logic in the S3A listing code gets into trouble if the FS closed and there was an async listing in progress. 

In this situation we should think about recognising and converting into some FS-is-closed exception
Bug: TestNativeIO fails in SELinux security context
Status: Open | Priority: Minor | Resolution: 
Created: 09/Dec/20 05:26
Affects Version/s: 3.3.0

As per [https://www.gnu.org/software/coreutils/manual/html_node/What-information-is-listed.html] "GNU ls uses a ‘.’ character to indicate a file with a security context, but no other alternate access method. The affects the parsing of the standard Unix file permissions format in TestNativeIO resulting in test failure.
Bug: TestLdapGroupsMapping test failures in JDK 11
Status: Open | Priority: Minor | Resolution: 
Created: 09/Dec/20 05:31

Some of the tests in TestLdapGroupsMapping rely on validating the precise text of a NamingException which has changed in JDK11. The message is substantially the same, but not identical, so tests fails. Specifically: 

testLdapConnectionTimeout

and 

testLdapReadTimeout
Bug: Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Dec/20 17:22
Affects Version/s: 3.3.0

Test failure in ITestAssumeRole.testAssumeRoleRestrictedPolicyFS if the test bucket is unguarded. I've been playing with my bucket settings so this probably didn't surface before. 

test arguments -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=keep  -Dfs.s3a.directory.marker.audit=true
Bug: Improve S3A upload statistics collection from ProgressEvent callbacks
Status: Open | Priority: Minor | Resolution: 
Created: 15/Dec/20 17:19
Affects Version/s: 3.4.0

Collection of S3A upload stats from ProgressEvent callbacks can be improved

Two similar but different implementations of listeners
* org.apache.hadoop.fs.s3a.S3ABlockOutputStream.BlockUploadProgress
* org.apache.hadoop.fs.s3a.ProgressableProgressListener. Used on simple PUT calls.

Both call back into S3A FS to incrementWriteOperations; BlockUploadProgress also updates S3AInstrumentation/IOStatistics.

* I'm not 100% confident that BlockUploadProgress is updating things (especially gauges of pending bytes) at the right time
* or that completion is being handled
* And the other interface doesn't update S3AInstrumentation; numbers are lost.
* And there's no incremental updating during {{CommitOperations.uploadFileToPendingCommit()}}, which doesn't call Progressable.progress() other than on every block.
* or in MultipartUploader 

Proposed: 
* a single Progress listener which updates BlockOutputStreamStatistics, used by all interfaces.
* WriteOperations to help set this up for callers; 
* And it's uploadPart API to take a Progressable (or the progress listener to use for uploading that part)
* Multipart upload API to also add a progressable...would help for distcp-like applications.

+Itests to verify that the gauges come out right. At the end of each operation, the #of bytes pending upload == 0; that of bytes uploaded == the original size
Bug: Contract test for renaming over existing file is too lenient
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Nov/20 11:39

{{AbstractContractRenameTest#testRenameFileOverExistingFile}} is too lenient in its assertions.

* {{FileAlreadyExistsException}} is accepted regardless of "rename overwrites" and "rename returns false if exists" contract options.  I think it should be accepted only if both of those options are false.
* "rename returns false if exists" option is ignored if the file is not overwritten by the implementation.

Also, I think the "rename returns false if exists" option is incorrectly inverted in the test, which it can get away with because the checks are loose.

(Found this while looking at a change in Ozone FS implementation from throwing exception to returning false.  The contract test unexpectedly passed without changing {{contract.xml}}.)
Bug:  Restore ability to set Text to empty byte array
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Dec/20 10:17

In org.apache.hadoop.io.Text:clear() method, the comments show that we can free the bytes by call set(new byte[0]), but it's not going to work now. Maybe we can follow this comments.

 

 
{code:java}
// org.apache.hadoop.io.Text 

/**
 * Clear the string to empty.
 *
 * <em>Note</em>: For performance reasons, this call does not clear the
 * underlying byte array that is retrievable via {@link #getBytes()}.
 * In order to free the byte-array memory, call {@link #set(byte[])}
 * with an empty byte array (For example, <code>new byte[0]</code>).
 */
public void clear() {
  length = 0;
  textLength = -1;
}
{code}
Bug: s3a getFileStatus("/") to skip IO
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Oct/20 18:24
Affects Version/s: 3.3.0

Calls to "hadoop fs ls s3a://something/" add a full getFileStatus() sequence (HEAD, LIST, LIST) because FsShell Ls calls getFileStatus on every input. 

We should just build a root status entry immediately. There's one consequence: if a user has disabled the s3a existence probe, there will be no checks for the bucket existence. But that can be viewed as a "well, that's what happens" behaviour
Bug: Generally support PEM keystores
Status: Open | Priority: Minor | Resolution: 
Created: 08/Dec/20 20:44

Several Apache projects have added some minimal PEM support.  Usually this is presented as part of supporting generated certificates for a trust store.  By adding a PEM Java KeyStore to Hadoop common, we could provide this support for many clients.

Example:
{code:java}
<property>
  <name>ssl.server.truststore.type</name>
  <value>PEM</value>
</property>{code}
Bug: S3A committer to support concurrent jobs with same app attempt ID & dest dir
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Oct/20 14:10
Affects Version/s: 3.3.0

Reported failure of magic committer block uploads as pending upload ID is unknown. Likely cause: it's been aborted by another job

# Make it possible to turn off cleanup of pending uploads in magic committer
# log more about uploads being deleted in committers
# and upload ID in the S3aBlockOutputStream errors

There are other concurrency issues when you look close, see SPARK-33230

* magic committer uses app attempt ID as path under __magic; if there are duplicate then they will conflict
* staging committer local temp dir uses app attempt id

Fix will be to have a job UUID which for spark will be picked up from the SPARK-33230 changes, (option to self-generate in job setup for hadoop 3.3.1+ older spark builds); fall back to app-attempt *unless that fallback has been disabled*

MR: configure to use app attempt ID
Spark: configure to fail job setup if app attempt ID is the source of a job uuid
Bug: Add Flink to CallerContext LimitedPrivate scope
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 25/Nov/20 06:35
Affects Version/s: 3.3.0

A lots of Flink applications run on Hadoop. Spark will invoke Hadoop caller context APIs to set up its caller contexts in HDFS/Yarn, so Hadoop should add Spark as one of the users in the LimitedPrivate scope.
Bug: hadoop component start with failed to init Ganglia31
Status: Open | Priority: Minor | Resolution: 
Created: 04/Nov/20 14:02

When start namenode, datanode or releated components, there exists an Warning :

Error creating sink 'ganglia'
 org.apache.hadoop.metrics2.impl.MetricsConfigException: Error creating plugin: org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31

if there exists configuration, like : *.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40。

 

Maybe, this is a bug about common-configuration2.  

 

The default conversion handler used DisabledListDelimiterHandler.INSTANCE as EXTRACTOR, while the comma list delimiter handler is expected.
Bug: CosNInputStream should throw exception when filesystem is closed
Status: Open | Priority: Minor | Resolution: 
Created: 28/Oct/20 06:37
Affects Version/s: 3.3.0

Thread hangs when other thread call filesystem.close(). log as below

971  [main] INFO  org.apache.hadoop.fs.cosn.CosNFileSystem  - Open the file: [cosn://bigdata-test

1075 [main] ERROR org.apache.hadoop.util.BlockingThreadPoolExecutorService  - Could not submit task to executor java.util.concurrent.ThreadPoolExecutor

the reason of thread hangs is :
 # filesystem is cached when multiple thread using FileSystem#get method for creating instance
 # CosNInputStream#read method is not check CosNFileSystem is closed
 # CosNInputStream is using multiple thread(CosNFileReadTask) for read and main thread will await when readbuffer status is init
 # CosNFileSystem#close method will shutdown boundedIOThreadPool . so CosNInputStream#CosNFileReadTask will not notify main thread and cause thread hangs