av9ash/GitBugs huggingface dataset:
Bug: Clarify in which branch CVE-2022-26612 is fixed
Status: Resolved | Priority: Major | Resolution: Done
Created: 29/Jun/22 12:47

According to HADOOP-18198, CVE-2022-26612 has been fixed in version 3.3.3.

The underlying ticket where the fix occured is HADOOP-18155. This ticket has fix version including 2.10.2.

On top of that, it's clear to me that CVE-2022-26612 is fixed in hadoop-common:2.10.2.
Howerver, it is still reported as an issue in different places:
 * [https://github.com/advisories/GHSA-gx2c-fvhc-ph4j]
 * [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/2.10.2]

It may just be a false positive in a CVE database, still I prefer to double-check with the hadoop community.

So, could you please state here whether CVE-2022-26612 is really fixed in below version of hadoop-common ?
 * >= 2.10.2
 * >= 3.2.3
 * >= 3.3.3
 * >= 3.4.0
Bug: Fix Hadoop Common Java Doc Errors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/May/22 10:38
Affects Version/s: 3.4.0

I found that when hadoop-multibranch compiled PR-4266, some errors would pop up, I tried to solve it

The wrong compilation information is as follows, I try to fix the Error information


{code:java}
[ERROR] /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-4266/ubuntu-focal/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java:432: error: exception not thrown: java.io.IOException
[ERROR]    * @throws IOException
[ERROR]              ^
[ERROR] /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-4266/ubuntu-focal/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java:885: error: unknown tag: username
[ERROR]    *  E.g. link: ^/user/(?<username>\\w+) => s3://$user.apache.com/_${user}
[ERROR]                           ^
[ERROR] /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-4266/ubuntu-focal/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java:885: error: bad use of '>'
[ERROR]    *  E.g. link: ^/user/(?<username>\\w+) => s3://$user.apache.com/_${user}
[ERROR]                                            ^
[ERROR] /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-4266/ubuntu-focal/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java:910: error: unknown tag: username
[ERROR]    *     .linkRegex.replaceresolveddstpath:_:-#.^/user/(?<username>\w+)
{code}
Bug: Update protobuf 3.7.1 to a version without CVE-2021-22569
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/22 10:12
Affects Version/s: thirdparty-1.2.0

The artifact `org.apache.hadoop:hadoop-common` brings in a dependency `com.google.protobuf:protobuf-java:2.5.0`, which is an outdated version released in 2013 and it contains a vulnerability [CVE-2021-22569|https://nvd.nist.gov/vuln/detail/CVE-2021-22569].

Therefore, requesting you to clarify if this library version is going to be updated in the following releases
Bug: ABFS: Add support for cache handling when filesystem instance has clientCorrelationID enabled
Status: Open | Priority: Major | Resolution: 
Created: 29/Jun/22 05:33
Affects Version/s: 3.3.3

This is to mainly handle the scenarios for cache enablement or disablement when the filesystem instance has clientCorrelationId as a part of it's configuration.

If a filesystem instance has clientCorrelationId as a part of it's configuration and if we enable cache for the same, there may be another job using a filesystem instance with same configuration and uri as the one cached, but the issue here is that since these 2 filesystem instances are part of 2 different running jobs, they can't share the same clientCorrelationId.
Bug: Release Hadoop 3.3.3: hadoop-3.3.2 with some fixes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/22 13:42
Affects Version/s: 3.3.2

Hadoop 3.3.3 is a minor followup release to Hadoop 3.3.2 with all the incremental changes which went in to the 3.2.4 release

* minor CVE fixes in Hadoop source
* CVE fixes in dependencies we know of (protobuf unmarshalling leading to DoS, jackson stack overflow,...)
* replacement of log4j 1.2.17 to reload4j
* node.js update

This is not a release off branch-3.3, it is a fork of 3.3.2 with the changes.

The next release of branch-3.3 will be numbered hadoop-3.3.4; updating maven versions and JIRA fix versions is part of this release process.

The changes here are already in branch 3.2.4; this completes the set

CVEs fixed

* CVE-2022-26612: Apache Hadoop: Arbitrary file write in FileUtil#unpackEntries on Windows (HADOOP-18155)
* CVE-2022-25168 Verify FileUtils.unTar() handling of missing .tar files. (HADOOP-18136)
Bug: Support nested mount points in INodeTree
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Apr/22 16:40
Affects Version/s: 2.10.0

Defining following client mount table config is not supported in  INodeTree and will throw FileAlreadyExistsException

 
{code:java}
fs.viewfs.mounttable.link./foo/bar=hdfs://nn1/foo/bar
fs.viewfs.mounttable.link./foo=hdfs://nn02/foo
{code}
INodeTree has 2 methods that need change to support nested mount points.
{code:java}
createLink(): build INodeTree during fs init.
resolve(): resolve path in INodeTree with viewfs apis.
{code}
ViewFileSystem and ViewFs maintains an INodeTree instance(fsState) in both classes and call fsState.resolve(..) to resolve path to specific mount point. INodeTree.resolve encapsulates the logic of nested mount point resolving. So no changes are expected in both classes. 

AC:
 # INodeTree.createlink should support creating nested mount points.(INodeTree is constructed during fs init)
 # INodeTree.resolve should support resolve path based on nested mount points. (INodeTree.resolve is used in viewfs apis)
 # No regression in existing ViewFileSystem and ViewFs apis.
 # Ensure some important apis are not broken with nested mount points. (Rename, getContentSummary, listStatus...)

 

Spec:

Please review attached pdf for spec about this feature.
Bug: Remove replace-guava from replacer plugin
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/22 13:05
Affects Version/s: 3.4.0

While running the build, realized that all replacer plugin executions run only after "banned-illegal-imports" enforcer plugin.

For instance,
{code:java}
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (banned-illegal-imports) @ hadoop-cloud-storage ---
[INFO] 
[INFO] --- replacer:1.5.3:replace (replace-generated-sources) @ hadoop-cloud-storage ---
[INFO] Skipping
[INFO] 
[INFO] --- replacer:1.5.3:replace (replace-sources) @ hadoop-cloud-storage ---
[INFO] Skipping
[INFO] 
[INFO] --- replacer:1.5.3:replace (replace-guava) @ hadoop-cloud-storage ---
[INFO] Replacement run on 0 file.
[INFO]  {code}
Hence, if our source code uses com.google.common, banned-illegal-imports will cause the build failure and replacer plugin would not even get executed.

We should remove it as it is only redundant execution step.
Bug: create-release fails fatal: unsafe repository ('/build/source' is owned by someone else)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/22 16:55
Affects Version/s: 3.3.3

The first git command in the create-release docker container is failing as git doesn't consider /build/source to belong to the current user.

since CVE-2022-24765 and the april 2022 git releases to refuse to work in such directories

https://github.blog/2022-04-12-git-security-vulnerability-announced/

{code}

$ /usr/bin/git clean -xdf -e /patchprocess
fatal: unsafe repository ('/build/source' is owned by someone else)
To add an exception for this directory, call:

	git config --global --add safe.directory /build/source

{code}
Bug: Prevent DelegationTokenSecretManagerMetrics from registering multiple times 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/22 16:17
Affects Version/s: 3.3.5

After committing HADOOP-18167, we received reports of the following error when ResourceManager is initialized:
{noformat}
Caused by: java.io.IOException: Problem starting http server
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1389)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:475)
        ... 4 more
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source DelegationTokenSecretManagerMetrics already exists!
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
        at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
        at org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:71)
        at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenSecretManagerMetrics.create(AbstractDelegationTokenSecretManager.java:878)
        at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.<init>(AbstractDelegationTokenSecretManager.java:152)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$DelegationTokenSecretManager.<init>(DelegationTokenManager.java:72)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.<init>(DelegationTokenManager.java:122)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:161)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.init(DelegationTokenAuthenticationHandler.java:130)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.initializeAuthHandler(DelegationTokenAuthenticationFilter.java:214)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:180)
        at org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53){noformat}
This can happen if MetricsSystemImpl#init is called and multiple metrics are registered with the same name. A proposed solution is to declare the metrics in AbstractDelegationTokenSecretManager as singleton, which would prevent multiple instances DelegationTokenSecretManagerMetrics from being registered.
Bug: Upgrade maven compiler plugin to 3.10.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/22 23:05
Affects Version/s: 3.4.0

Currently we are using maven-compiler-plugin 3.1 version, which is quite old (2013) and it's also pulling in vulnerable log4j dependency:
{code:java}
[INFO]    org.apache.maven.plugins:maven-compiler-plugin:maven-plugin:3.1:runtime
[INFO]       org.apache.maven.plugins:maven-compiler-plugin:jar:3.1
[INFO]       org.apache.maven:maven-plugin-api:jar:2.0.9
[INFO]       org.apache.maven:maven-artifact:jar:2.0.9
[INFO]       org.codehaus.plexus:plexus-utils:jar:1.5.1
[INFO]       org.apache.maven:maven-core:jar:2.0.9
[INFO]       org.apache.maven:maven-settings:jar:2.0.9
[INFO]       org.apache.maven:maven-plugin-parameter-documenter:jar:2.0.9
...
...
...
[INFO]       log4j:log4j:jar:1.2.12
[INFO]       commons-logging:commons-logging-api:jar:1.1
[INFO]       com.google.collections:google-collections:jar:1.0
[INFO]       junit:junit:jar:3.8.2
 {code}
 

We should upgrade to 3.10.1 (latest Mar, 2022) version of maven-compiler-plugin.
Bug: Upgrade Apache Xerces Java to 2.12.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/22 03:32
Affects Version/s: 3.3.3

Description

https://github.com/advisories/GHSA-h65f-jvqw-m9fj


There's a vulnerability within the Apache Xerces Java (XercesJ) XML parser when handling specially crafted XML document payloads. This causes, the XercesJ XML parser to wait in an infinite loop, which may sometimes consume system resources for prolonged duration. This vulnerability is present within XercesJ version 2.12.1 and the previous versions.

References
[https://nvd.nist.gov/vuln/detail/CVE-2022-23437]
https://lists.apache.org/thread/6pjwm10bb69kq955fzr1n0nflnjd27dl
http://www.openwall.com/lists/oss-security/2022/01/24/3
https://www.oracle.com/security-alerts/cpuapr2022.html
Bug: Extend KMS related exceptions that get mapped to ConnectException 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/22 21:41
Affects Version/s: 3.4.0

Based on production workload, we found that it is not enough to map just SSLHandshakeException to ConnectException in Loadbalancing KMS Client but that needs to be extended to SSLExceptions and SocketExceptions.

Sample JDK code that can raise these exceptions: https://github.com/openjdk/jdk/blob/jdk-18%2B32/src/java.base/share/classes/sun/security/ssl/SSLSocketImpl.java#L1409-L1428

Sample Exception backtrace: 
22/04/13 16:25:53 WARN kms.LoadBalancingKMSClientProvider: KMS provider at [https://bdgtr041x10h5.nam.nsroot.net:16001/kms/v1/] threw an IOException:
javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
        at sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1470)
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1298)
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1199)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:373)
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:587)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDe
Caused by: java.io.EOFException: SSL peer shut down incorrectly
        at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:480)
        at sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:469)
        ... 59 more
Bug: Fix Junit Test Deprecated assertThat
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 20/May/22 02:27
Affects Version/s: 3.4.0

javac will give a warning for compilation, as follows:

org.junit.Assert.assertThat Deprecated. use org.hamcrest.MatcherAssert.assertThat()
{code:java}
TestIncrementalBrVariations.java:141:4:[deprecation] <T>assertThat(T,Matcher<? super T>) in Assert has been deprecated {code}
a related issue will be resolved in HDFS-16590.
Bug: Fix getUri() in HttpRequest has been deprecated
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/22 03:34
Affects Version/s: 3.4.0

When reading the code, I found that the method used has been deprecated due to the upgrade of the netty component. The main methods are as follows:

io.netty.handler.codec.http#HttpRequest

@Deprecated
HttpMethod getMethod();
Deprecated. Use method() instead.

@Deprecated
String getUri()
Deprecated. Use uri() instead.
io.netty.handler.codec.http#DefaultHttpResponse

@Deprecated
public HttpResponseStatus getStatus()

{         return this.status(); }

Deprecated. Use status()  instead.

 

WebHdfsHandler.java:125:35:[deprecation] getUri() in HttpRequest has been deprecated

HostRestrictingAuthorizationFilterHandler.java:200:27:[deprecation] getUri() in HttpRequest has been deprecated
Bug: Cancel fileMonitoringTimer even if trustManager isn't defined
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/22 00:45
Affects Version/s: 3.3.5

The key stores factory starts a timer for monitoring file changes that should be reloaded.  The call to destroy() doesn't cancel the timer when a trust manager isn't defined.  This leaves the timer running, during shutdown.  This can be seen in unit tests that do not stop when the test completes.
Bug: Total requests and total requests per sec served by RPC servers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jun/22 01:15
Affects Version/s: 3.3.5

RPC Servers provide bunch of useful information like num of open connections, slow requests, num of in-progress handlers, RPC processing time, queue time etc, however so far it doesn't provide accumulation of all requests as well as current snapshot of requests per second served by the server. Exposing them would benefit from operational viewpoint in identifying how busy the servers have been and how much load they are currently serving in the presence of cluster wide high load.
Bug: Add some description for PowerShellFencer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/22 11:58
Affects Version/s: 3.3.3

Some descriptions related to PowerShellFencer are missing here, such as: core-default.xml, HDFSHighAvailabilityWithQJM.md, HDFSHighAvailabilityWithNFS.md, NodeFencer.java.
We should have as much documentation as possible.
Bug: Prometheus metrics tag changed unexpectedly.
Status: Open | Priority: Major | Resolution: 
Created: 26/May/22 12:24
Affects Version/s: 3.2.1

Prom web print metrics like below
{code:java}
fs_namesystem_blocks_total{context="dfs",enabledecpolicies="RS-6-3-1024k",hastate="active",totalsynctimes="10 43 ",hostname="xxx"} 3674006 {code}
getTotalSyncTimes return String, so totalsynctimes will change to tag.

It is difficult to show in grafana. We may get wrong panel, though only one metric is set.

!截屏2022-05-26 下午8.15.50.png|width=562,height=236!
Bug: S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jun/22 14:33
Affects Version/s: 3.4.0

Currently there is no limit on the size of disk cache. This means we could have a large number of files on files, especially for access patterns that are very random and do not always read the block fully. 

 

eg:

in.seek(5);

in.read(); 

in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read

in.read();

in.seek(2 * blockSize + 10) // block 1 gets saved to disk

.. and so on

 

The in memory cache is bounded, and by default has a limit of 72MB (9 blocks). When a block is fully read, and a seek is issued it's released [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109]. We can also delete the on disk file for the block here if it exists. 

 

Also maybe add an upper limit on disk space, and delete the file which stores data of the block furthest from the current block (similar to the in memory cache) when this limit is reached.
Bug: Ensure S3A prefetching stream memory consumption scales
Status: Open | Priority: Major | Resolution: 
Created: 28/Apr/22 15:10
Affects Version/s: 3.4.0

A recurrent problem in cloud store IO is running out of memory because blocks are buffered in reads or writes.

We need to make sure that data/memory is managed in the prefetch code such that it works in processes with many worker threads (hive, spark....) and does not exhaust resources.
Bug: Review s3a prefetching input stream retry code; synchronization
Status: Open | Priority: Major | Resolution: 
Created: 09/Jun/22 12:00
Affects Version/s: 3.4.0

Need to review S3A prefetching stream retry logic

* no attempt to retry on unrecoverable errors
* do try on recoverable ones
* no wrap of retry by retry.
* annotate classes with Retries annotations to aid the review.

a key concern has to be that transient failure of  prefetch is recovered from; things like deleted/shortened file fails properly on the next read call

the other issue is: review all thread safety. the outer stream appears thread safe, so review to make sure that is sufficient
Bug: Upgrade dependencies to address several CVEs
Status: Open | Priority: Major | Resolution: 
Created: 22/Jun/22 14:08
Affects Version/s: 3.3.3

The following CVEs can be addressed by upgrading dependencies within the build.  This includes a replacement of HTrace with a noop implementation.
 * CVE-2018-7489
 * CVE-2020-10663
 * CVE-2020-28491
 * CVE-2020-35490
 * CVE-2020-35491
 * CVE-2020-36518
 * PRISMA-2021-0182

This addresses all of the CVEs from 3.3.3 except for ones that would require upgrading Netty to 4.x.  I'll be submitting a pull request for 3.3.4.
Bug: ABFS Rename Failure when tracking metadata is in incomplete state
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/22 04:02

If a node in the datacenter crashes while processing an operation, occasionally it can leave the Storage-internal blob tracking metadata in an incomplete state.  We expect this to happen occasionally, and so all API’s are designed in such a way that if this incomplete state is observed on a blob, the situation is resolved before the current operation proceeds.  However, this incident has exposed a bug specifically with the Rename API, where the incomplete state fails to resolve, leading to this incorrect failure.  As a temporary mitigation, if any other operation is performed on this blob – GetBlobProperties, GetBlob, GetFileProperties, SetFileProperties, etc – it should resolve the incomplete state, and rename will no longer hit this issue.

StackTrace:
{code:java}
2022-03-22 17:52:19,789 DEBUG [regionserver/euwukwlss-hg50:16020.logRoller] services.AbfsClient: HttpRequest: 404,RenameDestinationParentPathNotFound,cid=ef5cbf0f-5d4a-4630-8a59-3d559077fc24,rid=35fef164-101f-000b-1b15-3ed818000000,sent=0,recv=212,PUT,https://euwqdaotdfdls03.dfs.core.windows.net/eykbssc/apps/hbase/data/oldWALs/euwukwlss-hg50.tdf.qa%252C16020%252C1647949929877.1647967939315?timeout=90   {code}
Bug: AliyunOSS: AliyunOSSBlockOutputStream should not mark the temporary file for deletion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/22 10:09
Affects Version/s: 2.10.2

AliyunOSSBlockOutputStream buffers data in local directory before uploading to OSS. It uses LocalDirAllocator.createTmpFileForWrite which will create a temp file on this directory and invoke DeleteOnExitHook.add to add the path to a set and will be deleted on VM exit through a shutdown hook. However, the size of set will be growing if the VM does not exit.
Bug: Fix links to DataNodes in NameNode UI to support IPv6
Status: Patch Available | Priority: Major | Resolution: 
Created: 17/Apr/22 03:09

IPv6 second group getting displayed as port number and which lead to error in opening the DN webpage from NN Web UI.

 

!image-2022-04-17-08-39-20-926.png!

 

After fix: IPv6 Address & Port parsed correctly & link open the DN webpage normally.

!image-2022-04-18-21-52-46-661.png!
Bug: Fix some compatibility issues with 3.3.3 release notes
Status: Open | Priority: Major | Resolution: 
Created: 14/Jun/22 02:30
Affects Version/s: 3.3.3

3.3.3 Release Notes:
https://hadoop.apache.org/docs/r3.3.3/hadoop-project-dist/hadoop-common/release/3.3.3/RELEASENOTES.3.3.3.html

There are some compatibility issues here. E.g:
 !image-2022-06-14-10-27-23-027.png! 

I think this is happening due to a syntax issue.
It would be more appropriate to change it to this:
 !image-2022-06-14-10-28-53-822.png!