av9ash/GitBugs huggingface dataset:
Bug: Remove lock contention in SelectorPool of SocketIOWithTimeout
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/21 04:54

*SelectorPool* in hadoop-common/src/main/java/org/apache/hadoop/net/*SocketIOWithTimeout.java* is a point of lock contention.

For example: 
{code:java}
$ grep 'waiting to lock <0x00007f7d94006d90>' 63692.jstack | uniq -c
 1005 - waiting to lock <0x00007f7d94006d90> (a org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
{code}
and the thread stack is as follows:
{code:java}
"IPC Client (324579982) connection to /100.10.6.10:60020 from user_00" #14139 daemon prio=5 os_prio=0 tid=0x00007f7374039000 nid=0x85cc waiting for monitor entry [0x00007f6f45939000]
 java.lang.Thread.State: BLOCKED (on object monitor)
 at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:390)
 - waiting to lock <0x00007f7d94006d90> (a org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
 at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:325)
 at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
 at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
 at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
 at java.io.FilterInputStream.read(FilterInputStream.java:133)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
 - locked <0x00007fa818caf258> (a java.io.BufferedInputStream)
 at java.io.DataInputStream.readInt(DataInputStream.java:387)
 at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.readResponse(RpcClientImpl.java:967)
 at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568)

{code}
We should remove the lock contention.
Bug: S3AInputStream read does not re-open the input stream on the second read retry attempt
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/21 10:19
Affects Version/s: 3.3.1

*Bug description:*

The read method in S3AInputStream has this following behaviour when an IOException happening during the read:
 * {{reopen and read quickly}}: The client after failing in the first attempt of {{read}}, will reopen the stream and try reading again without {{sleep}}.

 * {{reopen and wait for fixed duration}}: The client after failing in the attempt of {{read}}, will reopen the stream, sleep for {{fs.s3a.retry.interval}} milliseconds (defaults to 500 ms), and then try reading from the stream.

While doing the {{reopen and read quickly}} process, the subsequent read will be retried without reopening the input stream in case of the second failure happened. This leads to some of the bytes read being skipped which results to corrupt/less data than required. 

 

*Scenario to reproduce:*
 * Execute S3AInputStream `read()` or `read(b, off, len)`.
 * The read failed and throws `Connection Reset` exception after reading some data.
 * The InputStream is re-opened and another `read()` or `read(b, off, len)` is executed
 * The read failed for the second time and throws `Connection Reset` exception after reading some data.
 * The InputStream is not re-opened and another `read()` or `read(b, off, len)` is executed after sleep
 * The read succeed, but it skips the first few bytes that has already been read on the second failure.

 

*Proposed fix:*

[https://github.com/apache/hadoop/pull/3109]

Added the test that reproduces the issue along with the fix
Bug: HADOOP-17065 breaks compatibility between 3.3.0 and 3.3.1
Status: Open | Priority: Major | Resolution: 
Created: 17/May/21 09:03
Affects Version/s: 3.3.1

AzureBlobFileSystemStore is a Public, Evolving class, by contract can't break compatibility between maintenance releases.

HADOOP-17065 changed its constructor signature from 
{noformat}
  public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme,
                                  Configuration configuration,
                                  AbfsCounters abfsCounters) throws IOException {
{noformat}
to
{noformat}
  public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme,
                                  Configuration configuration,
                                  AbfsCounters abfsCounters) throws IOException {
{noformat}
between 3.3.0 and 3.3.1.

cc: [~mehakmeetSingh], [~tmarquardt]
Bug: Update wildfly openssl to 2.1.3.Final
Status: Open | Priority: Major | Resolution: 
Created: 19/Apr/21 04:54

https://nvd.nist.gov/vuln/detail/CVE-2020-25644

A memory leak flaw was found in WildFly OpenSSL in versions prior to 1.1.3.Final, where it removes an HTTP session. It may allow the attacker to cause OOM leading to a denial of service. The highest threat from this vulnerability is to system availability.
Bug: The java agent args should be passed as the VM arguments
Status: Resolved | Priority: Major | Resolution: Pending Closed
Created: 26/Jun/21 10:43

Some java agents or maven plugins want works well, must pass the agent arguments as the VM arguments, such as JaCoCo etc.

PS. Use the JaCoCo to generate the code coverage data and put into SonarQube.


{code:java}
mvn clean org.jacoco:jacoco-maven-plugin:prepare-agent package -Dmaven.test.failure.ignore=true org.jacoco:jacoco-maven-plugin:report surefire-report:report
{code}
Bug: S3A to add option fs.s3a.endpoint.region to set AWS region
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/21 11:21

Currently, AWS region is either constructed via the endpoint URL, by making an assumption that the 2nd component after delimiter "." is the region in endpoint URL, which doesn't work for private links and sets the default to us-east-1 thus causing authorization issue w.r.t the private link.

The option fs.s3a.endpoint.region allows this to be explicitly set

h2. how to set the s3 region on older hadoop releases

For anyone who needs to set the signing region on older versions of the s3a client *you do not need this festure*. instead just provide a custom endpoint to region mapping json file

# Download the default region mapping file [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
# Add a new regular expression to map the endpoint/hostname to the target region
# Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
# verify basic hadop fs -ls commands work
# copy to the rest of the cluster.
# There should be no need to restart any services
Bug: Improve error message for token providers in ABFS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/21 11:07
Affects Version/s: 3.3.0

It would be good to improve error messages for token providers in ABFS. Currently, when a configuration key is not found or mistyped, the error is not very clear on what went wrong. It would be good to indicate that the key was required but not found in Hadoop configuration when creating a token provider.

For example, when running the following code:
{code:java}
import org.apache.hadoop.conf._
import org.apache.hadoop.fs._

val conf = new Configuration()
conf.set("fs.azure.account.auth.type", "OAuth")
conf.set("fs.azure.account.oauth.provider.type", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
conf.set("fs.azure.account.oauth2.client.id", "my-client-id")
// conf.set("fs.azure.account.oauth2.client.secret.my-account.dfs.core.windows.net", "my-secret")
conf.set("fs.azure.account.oauth2.client.endpoint", "my-endpoint")

val path = new Path("abfss://container@my-account.dfs.core.windows.net/")
val fs = path.getFileSystem(conf)
fs.getFileStatus(path){code}
The following exception is thrown:
{code:java}
TokenAccessProviderException: Unable to load OAuth token provider class.
...
Caused by: UncheckedExecutionException: java.lang.NullPointerException: clientSecret
...
Caused by: NullPointerException: clientSecret {code}
which does not tell what configuration key was not loaded.

 

IMHO, it would be good if the exception was something like this:
{code:java}
TokenAccessProviderException: Unable to load OAuth token provider class.
...
Caused by: ConfigurationPropertyNotFoundException: Configuration property fs.azure.account.oauth2.client.secret not found. {code}
Bug: Problem in installation of Hadoop 3.2 with docket-  libc-bin  bug 
Status: Open | Priority: Major | Resolution: 
Created: 18/May/21 00:01
Affects Version/s: 3.2.2

Hi, 

I got the following bug while installing Hadoop 3.2.2 with docker:

Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
WARN engine npm@7.13.0: wanted: \{"node":">=10"} (current: \{"node":"4.2.6","npm":"3.5.2"})
WARN engine npm@7.13.0: wanted: \{"node":">=10"} (current: \{"node":"4.2.6","npm":"3.5.2"})
/usr/local/lib
`-- (empty)

npm ERR! Linux 4.15.0-29-generic
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "npm@latest" "-g"
npm ERR! node v4.2.6
npm ERR! npm v3.5.2
npm ERR! path /usr/local/lib/node_modules/.staging/@npmcli/ci-detect-c7bf9552
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall rename

npm ERR! enoent ENOENT: no such file or directory, rename '/usr/local/lib/node_modules/.staging/@npmcli/ci-detect-c7bf9552' -> '/usr/local/lib/node_modules/npm/node_modules/@npmcli/ci-detect'
npm ERR! enoent ENOENT: no such file or directory, rename '/usr/local/lib/node_modules/.staging/@npmcli/ci-detect-c7bf9552' -> '/usr/local/lib/node_modules/npm/node_modules/@npmcli/ci-detect'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! Please include the following file with any support request:
npm ERR! /root/npm-debug.log
npm ERR! code 1

 

 

What can I do?
Bug: Set locale for Centos 7 and 8
Status: Open | Priority: Major | Resolution: 
Created: 01/Jun/21 06:22
Affects Version/s: 3.4.0

The locale is set for Ubuntu Focal - https://github.com/apache/hadoop/blob/a234d00c1ce57427202d4c9587f891ec0164d10c/dev-support/docker/Dockerfile#L52-L54. The locale needs to be set for Centos 7 and 8 as well.
Bug: adls test suite TestAdlContractGetFileStatusLive failing with no assertJ on the classpath
Status: Open | Priority: Major | Resolution: 
Created: 12/May/21 17:18
Affects Version/s: 3.3.1

Reported on PR #2482: https://github.com/apache/hadoop/pull/2842 ; CNFE on assertJ assertions in adls test runs. 

Cause will be HADOOP-17281, which added the asserts to the existing fs contract test. We need to mark assertJ as an export of the hadoop-common suite, or work out why hadoop-azuredatalake isn't picking itup
Bug: ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/21 08:29

ABFS open methods require certain information (contentLength, eTag, etc) to  to create an InputStream for the file at the given path. This information is retrieved via a GetFileStatus request to backend.

However, client applications may often have access to the FileStatus prior to invoking the open API. Providing this FileStatus to the driver through the OpenFileParameters argument of openFileWithOptions() can help avoid the call to Store for FileStatus.

This PR adds handling for the FileStatus instance (if any) provided via the OpenFileParameters argument.
Bug: Fails to build using Maven 3.8.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Apr/21 02:44
Affects Version/s: 3.4.0

The latest Maven (3.8.1) errors out when building Hadoop (tried trunk)
{noformat}
[ERROR] Failed to execute goal on project hadoop-yarn-applications-catalog-webapp: Could not resolve dependencies for project org.apache.hadoop:hadoop-yarn-applications-catalog-webapp:war:3.4.0-SNAPSHOT: Failed to collect dependencies at org.apache.solr:solr-core:jar:7.7.0 -> org.restlet.jee:org.restlet:jar:2.3.0: Failed to read artifact descriptor for org.restlet.jee:org.restlet:jar:2.3.0: Could not transfer artifact org.restlet.jee:org.restlet:pom:2.3.0 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [maven-restlet (http://maven.restlet.org, default, releases+snapshots), apache.snapshots (http://repository.apache.org/snapshots, default, disabled)] -> [Help 1]
{noformat}
According to [https://maven.apache.org/docs/3.8.1/release-notes.html#how-to-fix-when-i-get-a-http-repository-blocked] we need to update our Maven repo.

 

Maven 3.6.3 is good.

 

(For what is worth, I used my company's mirror to bypass this error. Not sure what is a good fix for Hadoop itself)
Bug: ABFS: Use Unique File Paths in Tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/21 12:02
Affects Version/s: 3.3.1

Many of ABFS driver tests use common names for file paths (e.g., "/testfile"). This poses a risk of errors during parallel test runs when static variables (such as those for monitoring stats) affected by file paths are introduced.

Using unique test file names will avoid possible errors arising from shared resources during parallel runs.
Bug: Bump json-smart to 2.4.2 and nimbus-jose-jwt to 9.8 due to CVEs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/21 15:27
Affects Version/s: 3.2.1

Please upgrade the json-smart dependency to the latest version available.

Currently hadoop-auth is using version 2.3. Fortify scan picked up a security issue with this version. Please upgrade to the latest version. 

Thanks!
Bug: Old JQuery version causing security concerns
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 30/Jun/21 19:35
Affects Version/s: 3.3.0

These fixes are required for Hadoop 3.3.0. 

Can you please update the following jqueries for the UI, they are causing security and vulnerabilities concerns: 

URL : http://web-address:8088/static/jquery/jquery-3.4.1.min.js Installed version : 3.4.1 Fixed version : 3.5.0 or latest

URL : http://web-address:8080/static/jquery-1.12.4.min.js  Installed version : 1.12.4 Fixed version : 3.5.0 or latest

These also extend to Spark-on-Yarn cluster. I hope I'm not messing up with my files paths!

 

Thank you
Bug: Use profile hbase-2.0 by default and update hbase version
Status: Open | Priority: Major | Resolution: 
Created: 26/Apr/21 12:47
Affects Version/s: 3.4.0

We currently use hbase1 profile by default (for those who aren't aware, the YARN timeline service uses HBase as the underlying storage). There isn't much development activity in HBase 1.x and 2.x is production ready. I think it's time to switch to hbase 2 by default.

 

The HBase 2 version being used is 2.0.2. We should use the more recent versions.  (e.g. 2.2/2.3/2.4) (And update hbase 1 version as well)
Bug: Distcp is unable to determine region with S3 PrivateLink endpoints
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/21 20:53
Affects Version/s: 2.9.1

[root@ip-10-101-83-42 bin]# export AWS_REGION=us-east-2

[root@ip-10-101-83-42 bin]# hadoop distcp -Dfs.s3a.aws.credentials.provider="org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider" \

> -Dfs.s3a.access.key="${AWS_ACCESS_KEY_ID}" \

> -Dfs.s3a.secret.key="${AWS_SECRET_ACCESS_KEY}" \

> -Dfs.s3a.session.token="${AWS_SESSION_TOKEN}" \

> -Dfs.s3a.path.style.access=true \

> -Dfs.s3a.connection.ssl.enabled=true \

> -Dfs.s3a.attempts.maximum=1 \

> -Dfs.s3a.endpoint=bucket.vpce-123456-4jiz2sq4-us-east-2b.s3.us-east-2.vpce.amazonaws.com \

> hdfs:///user/root/ s3a://emr-tls-test/

21/04/27 10:02:13 INFO tools.OptionsParser: parseChunkSize: blocksperchunk false

21/04/27 10:02:15 WARN s3a.S3AFileSystem: Client: Amazon S3 error 400: 400 Bad Request; Bad Request (retryable)

 

x-amz-bucket-region=us-east-2

 

com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 0M1323C7AYJY315J; S3 Extended Request ID: vNyJdHN3bQ4g2w1sy2bvzoYLk+JGjMM4dQeSeDtQAdNk63ucXwOUWfoXZq1ZG8XQbnOclpc9Oi8=; Proxy: null), S3 Extended Request ID: vNyJdHN3bQ4g2w1sy2bvzoYLk+JGjMM4dQeSeDtQAdNk63ucXwOUWfoXZq1ZG8XQbnOclpc9Oi8=

 

It seems the request was rejected due to Malformed Header. The client was not sending the right region for request signing.
Bug: Remove JavaScript package from Docker environment
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/21 05:52

As described in the [README of yarn-ui|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/README.md], required javascript modules are automatically pulled by frontend-maven-plugin. We can leverage them for local testing too.

While hadoop-yarn-ui and hadoop-yarn-applications-catalog-webapp is using node.js, the version of node.js does not match. JavaScript related packages of the docker environment is not sure to work.

* https://github.com/apache/hadoop/blob/fdef2b4ccacb8753aac0f5625505181c9b4dc154/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/pom.xml#L170-L212
* https://github.com/apache/hadoop/blob/fdef2b4ccacb8753aac0f5625505181c9b4dc154/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml#L264-L290
Bug: bytesRead FS statistic showing twice the correct value in S3A
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/21 10:07

S3A "bytes read" statistic is being incremented twice. Firstly while reading in S3AInputStream and then in merge() of S3AInstrumentation when S3AInputStream is closed.

This makes "bytes read" statistic equal to sum of stream_read_bytes and stream_read_total_bytes.
Bug: wasb: delete() should have timeout option
Status: Open | Priority: Major | Resolution: 
Created: 22/Jun/21 21:33
Affects Version/s: 3.2.0

The delete() API in AzureBlobFileSystem could potentially stuck when trying to delete a infinitely lease blob file/directory. We hope that there is a timeout option for this API and the delete() could throw an timeoutException when specified timeout limit is reached.
Bug: Service authorization adds ip-based user authentication
Status: Open | Priority: Major | Resolution: 
Created: 01/Jun/21 10:36
Affects Version/s: 3.4.0

At present, the authentication of the user and IP of the service RPC is independent and global. And most of our needs are to allow some users of specific IPs, or some users of specific network segments.
This strategy is used in our internal HDFS, and I want to contribute this feature to the community.
Bug: Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/21 08:12

{{hadoop.ssl.enabled}} and {{dfs.https.enable}} were removed from hadoop-3 by HADOOP-10348. They are not deprecated.
https://github.com/apache/hadoop/commit/8b196816d8de638aaab0294249e560303fd92b7a

Let's remove them from docs and core-default.xml.
Bug: Increase precommit job timeout from 20 hours to 24 hours.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/21 05:28

If QA runs for the whole project, it may not be finished within 20 hours. I suggest extending the timeout to 1day.
* https://github.com/apache/hadoop/pull/3049 : 17 hours. (finished)
* https://github.com/apache/hadoop/pull/3087 : 20 hours. (timeout)
Bug: NPE and excessive warnings after HADOOP-17728
Status: Open | Priority: Major | Resolution: 
Created: 10/Jun/21 14:55
Affects Version/s: 3.4.0

I'm noticing these warnings and NPE's when just running a simple pi test on a one node cluster:
{noformat}
2021-06-09 21:51:12,334 WARN  [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner] fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread but it will continue to run
java.lang.NullPointerException
	at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
	at java.lang.Thread.run(Thread.java:748){noformat}
This appears to be due to [HADOOP-17728].
I'm not sure I understand why that change was made?  Wasn't it by design that the remove should wait until something is queued?
[~kaifeiYi] can you please investigate?
Bug: Replace usage of Hashtable and ConcurrentHashMap.contains(Object)
Status: Open | Priority: Major | Resolution: 
Created: 13/Jun/21 22:42

The Hadoop project and its subprojects use in a few places [{{Hashtable.contains(Object)}}|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Hashtable.html#contains(java.lang.Object)] (or the inherited {{Properties.contains(Object)}}) and [{{ConcurrentHashMap.contains(Object)}}|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ConcurrentHashMap.html#contains(java.lang.Object)]. They originate from the old {{Hashtable}} class. The {{java.util.Map}} interface added {{containsValue(Object)}}, which behaves exactly the same, but has the major advantage that its name makes it more clear what the method actually does.

In fact some usage of {{contains(Object)}} by Hadoop is erroneous and results in incorrect behavior, it seems the author should have used {{containsKey(Object)}} instead.

Affected:
- [{{Configuration.setDeprecatedProperties()}}|https://github.com/apache/hadoop/blob/35e4c31fff9946d35a3543481585031558e9f5d5/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L692]
Seems to be a bug, should use {{containsKey(Object)}}; looks like this was also reported in HADOOP-15468
- {{DFSClient.isDeadNode(...)}}: [here|https://github.com/apache/hadoop/blob/35e4c31fff9946d35a3543481585031558e9f5d5/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L3400] and [here|https://github.com/apache/hadoop/blob/35e4c31fff9946d35a3543481585031558e9f5d5/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L3404]
Not sure if correct usage, should use either {{containsKey(Object)}} or {{containsValue(Object)}}
- [{{RestClientBindings.bind(...)}}|https://github.com/apache/hadoop/blob/35e4c31fff9946d35a3543481585031558e9f5d5/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/RestClientBindings.java#L152]
Not sure if correct usage, should use either {{containsKey(Object)}} or {{containsValue(Object)}}
- [{{TestableFederationClientInterceptor.registerBadSubCluster(...)}}|https://github.com/apache/hadoop/blob/35e4c31fff9946d35a3543481585031558e9f5d5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestableFederationClientInterceptor.java#L113]
Likely a bug, should use {{containsKey(Object)}}; also detected by [LGTM / CodeQL|https://lgtm.com/projects/g/apache/hadoop/snapshot/0c9c811fe5bfda548596b99d7c111994bf66a66b/files/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestableFederationClientInterceptor.java?sort=name&dir=ASC&mode=heatmap#L113]
Bug: Reduce lock contention in properties of Configuration
Status: Open | Priority: Major | Resolution: 
Created: 10/Jun/21 07:39

The *properties* field of class *Configuration* is guard by _synchronized_, and its type java.util.Property is a sub class of java.util.*Hashtable*, which methods are all synchronized. These are bad for multithread performance.

The properties field should be replaced by a *HashMap<String, String>* with a *ReentrantReadWriteLock*, then all _get()_ related operations can be parallel guard by ReadLock, and _set/load_ operations can be guard by WriteLock. This is better for performance.
Bug: Upgrade azure-storage library from v8 to v12
Status: Open | Priority: Major | Resolution: 
Created: 04/Jun/21 00:55

I was using the Hadoop Azure to download azure storage blobs. I found the hadoop-azure still using the v8 sdk. Do we have a plan to upgrade it to v12?

 

I tested the download speed using the v8 and v12 sdk. Using v8 sdk, the speed of downloading a blob can reach 900Mbit/s, using v12 sdk, the speed can reach almost 4GBit/s. So we might can upgrade the azure-storage sdk for getting better performance.