av9ash/GitBugs huggingface dataset:
Bug: Make UGI support forceful relogin from keytab ignoring the last login time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/20 01:45
Affects Version/s: 2.10.0

Currently we have a relogin() method in UGI which attempts to login if there is no login attempted in the last 10 minutes or configured amount of time

We should also have provision for doing a forceful relogin irrespective of the time window that the client can choose to use it if needed . Consider the below scenario:
 # SASL Server is reimaged and new keytabs are fetched with refreshing the password
 # SASL client connection to the server would fail when it tries with the cached service ticket
 # We should try to logout to clear the service tickets in cache and then try to login back in such scenarios. But since the current relogin() doesn't guarantee a login, it could cause an issue
 # A forceful relogin in this case would help after logout
Bug: Please fix CVEs by removing reference to htrace-core4
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 30/Jul/20 18:54
Affects Version/s: 3.3.0

htrace-core4 is a retired project and even on the latest version they Shade Jackson databind version 2.4.0 which has the following CVEs:
|cve|severity|cvss|
|CVE-2017-15095|critical|9.8|
|CVE-2018-1000873|medium|6.5|
|CVE-2018-14718|critical|9.8|
|CVE-2018-5968|high|8.1|
|CVE-2018-7489|critical|9.8|
|CVE-2019-14540|critical|9.8|
|CVE-2019-14893|critical|9.8|
|CVE-2019-16335|critical|9.8|
|CVE-2019-16942|critical|9.8|
|CVE-2019-16943|critical|9.8|
|CVE-2019-17267|critical|9.8|
|CVE-2019-17531|critical|9.8|
|CVE-2019-20330|critical|9.8|
|CVE-2020-10672|high|8.8|
|CVE-2020-10673|high|8.8|
|CVE-2020-10968|high|8.8|
|CVE-2020-10969|high|8.8|
|CVE-2020-11111|high|8.8|
|CVE-2020-11112|high|8.8|
|CVE-2020-11113|high|8.8|
|CVE-2020-11619|critical|9.8|
|CVE-2020-11620|critical|9.8|
|CVE-2020-14060|high|8.1|
|CVE-2020-14061|high|8.1|
|CVE-2020-14062|high|8.1|
|CVE-2020-14195|high|8.1|
|CVE-2020-8840|critical|9.8|
|CVE-2020-9546|critical|9.8|
|CVE-2020-9547|critical|9.8|
|CVE-2020-9548|critical|9.8|

 

Our security team is trying to block us from using hadoop because of this
Bug: hadoop-azure parallel tests not working on recent JDKs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/20 12:59
Affects Version/s: 3.3.0

recent JDKs are failing to run the wasb or abfs parallel test runs -unable to instantiate the javascript engine.

Maybe it's been cut from the JVM or the ant script task can't bind to it.

Fix is as HADOOP-14696 -use our own plugin to set up the test dirs
Bug: Test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/20 13:42
Affects Version/s: 3.3.0

Intermittent test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters happening due to race conditions in readAhead threads.

Test error:


{code:java}
[ERROR] testReadAheadCounters(org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics)  Time elapsed: 30.723 s  <<< ERROR!org.junit.runners.model.TestTimedOutException: test timed out after 30000 milliseconds        at java.lang.Thread.sleep(Native Method)        at org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics.testReadAheadCounters(ITestAbfsInputStreamStatistics.java:346)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:498)        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)        at java.util.concurrent.FutureTask.run(FutureTask.java:266)        at java.lang.Thread.run(Thread.java:748) {code}
Possible Reasoning:

- ReadAhead queue doesn't get completed and hence the counter values are not satisfied in 30 seconds time for some systems.

- The condition that readAheadBytesRead and remoteBytesRead counter values need to be greater than or equal to 4KB and 32KB respectively doesn't occur in some machines due to the fact that sometimes instead of reading for readAhead Buffer, remote reads are performed due to Threads still being in the readAhead queue to fill that buffer. Thus resulting in either of the 2 counter values to be not satisfying the condition and getting in an infinite loop and hence timing out the test eventually.

Possible Fixes:

- Write better test(That would pass under all conditions).
- Maybe UT instead of IT?

Possible fix to better the test would be preferable and UT as the last resort.
Bug: Implement FileSystem.listStatusIterator() in S3AFileSystem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/20 14:20
Affects Version/s: 3.3.0

Currently S3AFileSystem only implements listStatus() api which returns an array. Once we implement the listStatusIterator(), clients can benefit from the async listing done recently 

https://issues.apache.org/jira/browse/HADOOP-17074  by performing some tasks on files while iterating them.

 

CC [~stevel]
Bug: Memory leak MemoryStore streaming temp files
Status: Open | Priority: Major | Resolution: 
Created: 24/Sep/20 23:57

 I saw a ticket that was mentioning the similar issue and said it was solved in 3.3.0. But I could not find the extract ticket, seems to be this one - https://issues.apache.org/jira/browse/HADOOP-15658.

 

But I am still observing the large amount of temp parquet files accumulated in the MemoryStore below.

 

!image-2020-09-25-09-57-44-849.png!

!image-2020-09-25-09-57-59-774.png!
Bug:  Create socket address leveraging URI cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/20 17:27

Note：Not only the hdfs client can get the current benefit, all callers of NetUtils.createSocketAddr will get the benefit. Just use hdfs client as an example.

 

Hdfs client selects best DN for hdfs Block. method call stack:

DFSInputStream.chooseDataNode -> getBestNodeDNAddrPair -> NetUtils.createSocketAddr

NetUtils.createSocketAddr creates the corresponding InetSocketAddress based on the host and port. There are some heavier operations in the NetUtils.createSocketAddr method, for example: URI.create(target), so NetUtils.createSocketAddr takes more time to execute.

The following is my performance report. The report is based on HBase calling hdfs. HBase is a high-frequency access client for hdfs, because HBase read operations often access a small DataBlock (about 64k) instead of the entire HFile. In the case of high frequency access, the NetUtils.createSocketAddr method is time-consuming.
h3. Test Environment：

 
{code:java}
HBase version: 2.1.0
JVM: -Xmx2g -Xms2g 
hadoop hdfs version: 2.7.4
disk:SSD
OS:CentOS Linux release 7.4.1708 (Core)
JMH Benchmark: @Fork(value = 1) 
@Warmup(iterations = 300) 
@Measurement(iterations = 300)
{code}
h4. Before Optimization FlameGraph:

In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts for 4.86% of the entire CPU, and the creation of URIs accounts for a larger proportion.

!Before Optimization remark.png!
h3. Optimization ideas:

NetUtils.createSocketAddr creates InetSocketAddress based on host and port. Here we can add Cache to InetSocketAddress. The key of Cache is host and port, and the value is InetSocketAddress.
h4. After Optimization FlameGraph:

In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts for 0.54% of the entire CPU. Here, ConcurrentHashMap is used as the Cache, and the ConcurrentHashMap.get() method gets data from the Cache. The CPU usage of DFSInputStream.getBestNodeDNAddrPair has been optimized from 4.86% to 0.54%.

!After Optimization remark.png!
h3. Original FlameGraph link:

[Before Optimization|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZLEUzGR0XAfUl6/view?usp=sharing]

[After Optimization FlameGraph|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZLEUzGR0XAfUl6/view?usp=sharing]
Bug: ABFS: Support for Client Correlation ID
Status: Resolved | Priority: Major | Resolution: Abandoned
Created: 16/Sep/20 18:36
Affects Version/s: 3.3.0

Introducing a client correlation ID that appears in the Azure Storage diagnostic logs. This will modify the client request header to include the client-provided identifier for correlating requests. A valid correlation ID comprises alphanumeric characters and/or hyphens, and can be up to 72 characters in length. Invalid or missing entries for this configuration are substituted with the empty string default.
Bug: [JDK 11] Upgrade SpotBugs to 4.1.3 to fix false-positive warnings
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 17/Sep/20 19:02

In Java 11, there are a lot of false-positive findbugs warnings in try-with-resources.
Ref: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/1/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html

This issue has been fixed by https://github.com/spotbugs/spotbugs/pull/1248 but now there are no releases that include this fix.
Bug: Fixing javadoc in ListingOperationCallbacks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/20 08:27

{{mvn javadoc:javadoc -pl hadoop-tools/hadoop-aws}} is failing:
{noformat}
[ERROR] /Users/aajisaka/git/hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ListingOperationCallbacks.java:80: error: unexpected text
[ERROR]    * {@link this.getMaxKeys()}.
[ERROR]      ^
{noformat}
Bug: Refactor S3A Listing code for better isolation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/20 14:52
Affects Version/s: 3.3.0

Currently Listing has a S3AFileSystem instance to use its required methods thus giving listing access to all of S3AFileSystem methods. Ideal way would be to use a callback interface in Listing which can in turn call s3Afs methods thus restricting the listing to methods which are absolutely required.

 

CC [~stevel@apache.org]
Bug: upgrade jetty to 9.4.21
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 23/Jul/20 12:16
Affects Version/s: 3.3.0

I have tried to configure and start Hadoop KMS service, it was failed to start the error log messages:
{noformat}
2020-07-23 10:57:31,872 INFO  Server - jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_252-8u252-b09-1~18.04-b09
2020-07-23 10:57:31,899 INFO  session - DefaultSessionIdManager workerName=node0
2020-07-23 10:57:31,899 INFO  session - No SessionScavenger set, using defaults
2020-07-23 10:57:31,901 INFO  session - node0 Scavenging every 660000ms
2020-07-23 10:57:31,912 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@5bf0d49{logs,/logs,file:///opt/hadoop-3.4.0-SNAPSHOT/logs/,AVAILABLE}
2020-07-23 10:57:31,913 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@7c7a06ec{static,/static,jar:file:/opt/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/hadoop-kms-3.4.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
2020-07-23 10:57:31,986 INFO  TypeUtil - JVM Runtime does not support Modules
2020-07-23 10:57:32,015 INFO  KMSWebApp - -------------------------------------------------------------
2020-07-23 10:57:32,015 INFO  KMSWebApp -   Java runtime version : 1.8.0_252-8u252-b09-1~18.04-b09
2020-07-23 10:57:32,015 INFO  KMSWebApp -   User: hadoop
2020-07-23 10:57:32,015 INFO  KMSWebApp -   KMS Hadoop Version: 3.4.0-SNAPSHOT
2020-07-23 10:57:32,015 INFO  KMSWebApp - -------------------------------------------------------------
2020-07-23 10:57:32,023 INFO  KMSACLs - 'CREATE' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'DELETE' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'ROLLOVER' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'GET' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'GET_KEYS' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'GET_METADATA' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'SET_KEY_MATERIAL' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'GENERATE_EEK' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'DECRYPT_EEK' ACL '*'
2020-07-23 10:57:32,025 INFO  KMSACLs - default.key.acl. for KEY_OP 'READ' is set to '*'
2020-07-23 10:57:32,025 INFO  KMSACLs - default.key.acl. for KEY_OP 'MANAGEMENT' is set to '*'
2020-07-23 10:57:32,025 INFO  KMSACLs - default.key.acl. for KEY_OP 'GENERATE_EEK' is set to '*'
2020-07-23 10:57:32,025 INFO  KMSACLs - default.key.acl. for KEY_OP 'DECRYPT_EEK' is set to '*'
2020-07-23 10:57:32,080 INFO  KMSAudit - Initializing audit logger class org.apache.hadoop.crypto.key.kms.server.SimpleKMSAuditLogger
2020-07-23 10:57:32,537 INFO  KMSWebServer - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down KMSWebServer at hadoop-benchmark/172.17.0.2{noformat}
I have googled the error and found there is a simlar issue： [https://github.com/eclipse/jetty.project/issues/4064]

It looks like a bug of jetty and has  been fixed in jetty>=9.4.21, currently Hadoop use the jetty is version of 9.4.20, see hadoop-project/pom.xml.
Bug: improve s3guard markers command line tool
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/20 19:18
Affects Version/s: 3.4.0

The s3guard markers audit -expected N command is meant to verify that the marker count is N, but

* it isn't verified if the #of markers is 0, so you can't use it to assert that markers have been created
* it doesn't work for tests where you expect a minimum number of markers. 

It's essentially setting a max #of markers

Proposed: explicit -min, -max args to declare a specific range of values
Bug: ABFS: Test testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/20 13:43
Affects Version/s: 3.3.0

Test testNegativeScenariosForCreateOverwriteDisabled fails when run against a non-HNS account. The test creates a mock AbfsClient to mimic negative scenarios.

Mock is triggered for valid values that come in for permission and umask while creating a file. Permission and umask get defaulted to null values with driver when creating a file for a nonHNS account. The mock trigger was not enabled for these null parameters.
Bug: s3a rename() now requires s3:deleteObjectVersion permission
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Sep/20 19:53
Affects Version/s: 3.4.0

With the directory marker change (HADOOP-13230) you need the s3:deleteObjectVersion permission in your role, else the operation will fail in the bulk delete, *if S3Guard is in use*

Root cause
-if fileStatus has a versionId, we pass that in to the delete KeyVersion pair
-an unguarded listing doesn't get that versionId, so this is not an issue
-but if files in a directory were previously created such that S3Guard has their versionId in its tables, that is used in the request
-which then fails if the caller doesn't have the permission

Although we say "you need s3:delete*", this is a regression as any IAM role without the permission will have rename fail during delete
Bug: delta.io spark task commit encountering S3 cached 404/FileNotFoundException
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Aug/20 03:41
Affects Version/s: 3.1.2

Hi,

When using spark streaming with deltalake, I got the following exception occasionally, something like 1 out of 100. Thanks.
{code:java}
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://[pathToFolder]/date=2020-07-29/part-00005-046af631-7198-422c-8cc8-8d3adfb4413e.c000.snappy.parquet
 at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2255)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
 at org.apache.spark.sql.delta.files.DelayedCommitProtocol$$anonfun$8.apply(DelayedCommitProtocol.scala:141)
 at org.apache.spark.sql.delta.files.DelayedCommitProtocol$$anonfun$8.apply(DelayedCommitProtocol.scala:139)
 at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
 at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
 at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
 at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
 at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
 at scala.collection.AbstractTraversable.map(Traversable.scala:104)
 at org.apache.spark.sql.delta.files.DelayedCommitProtocol.commitTask(DelayedCommitProtocol.scala:139)
 at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
 at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
 at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242){code}
 

-----Environment----

hadoop = "3.1.2"
 hadoop-aws = "3.1.2"

spark = "2.4.5"

spark-on-k8s-operator = "v1beta2-1.1.2-2.4.5"

 

deployed into AWS EKS kubernates. Version information below:

Server Version: version.Info\{Major:"1", Minor:"16+", GitVersion:"v1.16.8-eks-e16311", GitCommit:"e163110a04dcb2f39c3325af96d019b4925419eb", GitTreeState:"clean", BuildDate:"2020-03-27T22:37:12Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Bug: Add RPC Quota to NameNode.
Status: Patch Available | Priority: Major | Resolution: 
Created: 17/Sep/20 13:59

My users recently complained 'The NameNode is much slower than before' to me. The reason is the cluster and jobs are getting bigger and bigger. So is the pressure of NameNode. I explained the pressure was heavy so the rpc requests must wait, but they were not satisfied. Because they thought the original quality of the service should be guaranteed. They were never told the NameNode would be so slow and all their services were built based on the assumption that the NameNode would always respond as fast as before.
 From the user's standpoint they are right. So my question is how to give the user a guarantee about RPC requests. The natural idea is RPC Quota, just like name quota and space quota. The quota can help users to understand the rpc requests are also a limit resource. And when they apply quota to the administrator, the admin would have the chance to distribute the resource and make a plan for the cluster. e.g. We have 200 quota for addBlock and they are all allocated. Even the peak doesn't reach 200, I should reject other users from applying to reserve the resource. The new user should be mounted to other namespaces.
 It's still an initial idea now. I'll think again carefully and make a detailed proposal. All advice are welcome!
Bug: Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Aug/20 10:14
Affects Version/s: 3.3.0

ITestAzureBlobFileSystemCheckAccess is giving test failures while running both in parallel as well as in stand-alone(in IDE).

Tested by:  mvn -T 1C -Dparallel-tests=abfs clean verify
 Region: East US
Bug: Sudo in hadoop-functions.sh should preserve environment variables 
Status: Patch Available | Priority: Major | Resolution: 
Created: 17/Sep/20 03:55
Affects Version/s: 3.3.0

Steps to reproduce:
1. Set {{HDFS_NAMENODE_USER=hdfs}} in {{/etc/default/hadoop-hdfs-namenode}} to enable user check (and switch to {{hdfs}} to start/stop NameNode daemon)
2. Stop NameNode with: {{service hadoop-hdfs-namenode stop}}
3. Got an error and NameNode is not stopped
{noformat}
ERROR: Cannot execute /usr/lib/hadoop-hdfs/bin/../libexec/hdfs-config.sh.
Failed to stop Hadoop namenode. Return value: 1. [FAILED]
{noformat}

The root cause is that after sudo, {{HADOOP_HOME=/usr/lib/hadoop}} is not preserved, and {{/usr/lib/hadoop-hdfs/bin/hdfs}} locates libexec by the following logic:

{noformat}
# let's locate libexec...
if [[ -n "${HADOOP_HOME}" ]]; then
  HADOOP_DEFAULT_LIBEXEC_DIR="${HADOOP_HOME}/libexec"
else
  bin=$(cd -P -- "$(dirname -- "${MYNAME}")" >/dev/null && pwd -P)
  HADOOP_DEFAULT_LIBEXEC_DIR="${bin}/../libexec"
fi
{noformat}

I believe the key point here is that we should preserve environment variables when doing sudo.

Note that this bug is not introduced by HDFS-15353, before which {{su -l}} is used, which will also discard environment variables.
Bug: DistCp -update option will be invalid when distcp files from hdfs to S3
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Sep/20 09:20

We use distcp with -update option to copy a dir from hdfs to S3. When we run distcp job once more, it will overwrite S3 dir directly, rather than skip the same files.
  
 Test Case:
Run twice the following cmd,  the modify time of S3 files will be modified every time.
 hadoop distcp -update /test/ s3a://${s3_buckect}/test/

 

Check code in CopyMapper.java and S3AFileSystem.java 

(1) For the first time, distcp job will create files in S3, but blockSize is unused!

!image-2020-09-10-17-45-16-998.png|width=542,height=485!

 

(2) For the second time, the distcp job will compare fileSize and blockSize between hdfs file and S3 file

!image-2020-09-10-17-47-01-653.png|width=524,height=248!

 

(3) blockSize is unused, when get blockSize of S3 file, it return a default value.

In S3AFileSystem.java, we find that the default value of fs.s3a.block.size is 32 * 1024 * 1024.

!image-2020-09-10-17-33-50-505.png|width=451,height=762!

 

!image-2020-09-10-17-52-32-290.png|width=527,height=87!
  

The blockSize of HDFS seems invalid in Object Store, like S3. So I think there's no need to compare blockSize when distcp with -update option.
Bug: Website to link to latest Hadoop wiki
Status: Open | Priority: Major | Resolution: 
Created: 08/Sep/20 21:53

Currently the website links to the [old wiki|https://wiki.apache.org/hadoop]. Shall we update that to the latest one: https://cwiki.apache.org/confluence/display/HADOOP2/Home

Or am I confused which one is latest, https://wiki.apache.org/hadoop or https://cwiki.apache.org/confluence/display/HADOOP2?