Project: Apache Hadoop (App Definition and Development / Database)
File: CHANGELOG.3.0.1.md

<! --> | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Change default NameNode RPC port back to 8020 | Critical | namenode | Xiao Chen | Xiao Chen | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | RBF: Fix doc error setting up client | Major | federation | tartarus | tartarus | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | CryptoInputStream should implement unbuffer | Major | fs, security | John Zhuge | John Zhuge | | | FairScheduler#getAppWeight() should be moved into FSAppAttempt#getWeight() | Minor | fairscheduler | Daniel Templeton | Soumabrata Chakraborty | | | ValueQueue should also validate (lowWatermark \* numValues) \> 0 on construction | Minor | . | Xiao Chen | Xiao Chen | | | Add blockId when warning slow mirror/disk in BlockReceiver | Trivial | hdfs | Jiandan Yang | Jiandan Yang | | | Remove unused FairSchedulerEventLog | Major | fairscheduler | Wilfred Spiegelenburg | Wilfred Spiegelenburg | | | Improve robustness of the AggregatedLogDeletionService | Major | log-aggregation | Jonathan Eagles | Jonathan Eagles | | | Node manager web UI should display container type in containers page | Major | nodemanager, webapp | Weiwei Yang | Weiwei Yang | | | Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AM | Major | resourcemanager | Juan Rodrguez Hortal | Juan Rodrguez Hortal | | | Fix TestUnbuffer#testUnbufferException failure | Minor | test | Jack Bearden | Jack Bearden | | | Add readahead, dropbehind, and unbuffer to StreamCapabilities | Major | fs | John Zhuge | John Zhuge | | | AliyunOSS: change the default value of max error retry | Major | fs/oss | wujinhu | wujinhu | | | Secure Datanode Starter should log the port when it fails to bind | Minor | datanode | Stephen O'Donnell | Stephen O'Donnell | | | Setting/Unsetting EC policy shows warning if the directory is not empty | Minor | erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu | | | Update erasure coding doc to address unsupported APIs | Major | erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu | | | When partial log aggregation is enabled, display the list of aggregated files on the container log page | Major |

| Siddharth Seth | Xuan Gong | | | Support multiple storages in DataNodeCluster / SimulatedFSDataset | Minor | datanode, test | Erik Krogen | Erik Krogen | | | When NN is not able to identify DN for replication, reason behind it can be logged | Critical | hdfs-client, namenode | Surendra Singh Lilhore | Xiao Chen | | | Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning | Major | hdfs | Manoj Govindassamy | Manoj Govindassamy | | | Ability to enable logging of container memory stats | Major | nodemanager | Jim Brennan | Jim Brennan | | | Enhance dfsadmin listOpenFiles command to list files under a given path | Major | . | Manoj Govindassamy | Yiqun Lin | | | Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethods | Minor | . | Wei Yan | Wei Yan | | | Improve container-executor validation check | Major | security, yarn | Eric Yang | Eric Yang | | | MR AM to clean up temporary files from previous attempt in case of no recovery | Major | applicationmaster | Gergo Repas | Gergo Repas | | | Update adls connector to use the current version of ADLS SDK | Major | fs/adl | Atul Sikaria | Atul Sikaria | | | backport HADOOP-15039 to branch-2 and branch-3 | Blocker | . | Genmao Yu | Genmao Yu | | | Allow Azure Data Lake SDK dependency version to be set on the command line | Major | build, fs/adl | Vishwajeet Dusane | Vishwajeet Dusane | | | Reduce verbosity for ThrottledAsyncChecker.java:schedule | Minor | datanode | Mukul Kumar Singh | Mukul Kumar Singh | | | Add independent secret manager method for logging expired tokens | Major | security | Daryn Sharp | Daryn Sharp | | | Expose container preemptions related information in Capacity Scheduler queue metrics | Major | . | Eric Payne | Eric Payne | | | Uber AM can crash due to unknown task in statusUpdate | Major | mr-am | Peter Bacsko | Peter Bacsko | | | Add Configuration API for parsing storage sizes | Minor | conf | Anu Engineer | Anu Engineer | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Improve the docker container runtime documentation | Major | . | Shane Kumpf | Shane Kumpf | | | ConcurrentModificationException in RMAppImpl#getRMAppMetrics | Major | capacityscheduler | Tao Yang | Tao Yang | | | Incorrect query parameters in cluster nodes REST API document | Minor | documentation | Tao Yang | Tao Yang | | | Document Apache Hadoop does not support Java 9 in BUILDING.txt | Major | documentation | Akira Ajisaka | Hanisha Koneru | | | ResourceRequest.equal does not check ExecutionTypeRequest.enforceExecutionType() | Major | api | Haibo Chen | Haibo Chen | | | Remove scheduler lock in FSAppAttempt.getWeight() | Minor | fairscheduler | Wilfred Spiegelenburg | Wilfred Spiegelenburg | | | All reservation related test cases failed when TestYarnClient runs against Fair Scheduler. | Major | fairscheduler, reservation system | Yufei Gu | Yufei Gu | | | ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutor | Major | nodemanager | Yufei Gu | Yufei Gu | | | INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit | Critical | namenode | DENG FEI | Konstantin Shvachko | | | Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0 | Minor | fs/azure | Rajesh Balamohan | Rajesh Balamohan | | | "yarn logs" command fails to get logs for running containers if UI authentication is enabled. | Critical |

| Namit Maheshwari | Xuan Gong | | | Delete copy-on-truncate block along with the original block, when deleting a file being truncated | Blocker | hdfs | Jiandan Yang | Konstantin Shvachko | | | startTxId could be greater than endTxId when tailing in-progress edit log | Major | hdfs | Chao Sun | Chao Sun | | | AM lacks flow control for task events | Major | mr-am | Jason Lowe | Peter Bacsko | | | TestPBImplRecords fails with NullPointerException | Major | . | Jason Lowe | Daniel Templeton | | | quote\and\append\_arg can overflow buffer | Major | nodemanager | Jason Lowe | Jim Brennan | | | create-release site build outputs dummy shaded jars due to skipShade | Blocker | . | Andrew Wang | Andrew Wang | | | Remove subversion related code from VersionInfoMojo.java | Minor | build | Akira Ajisaka | Ajay Kumar | | | TestClusterTopology#testChooseRandom fails intermittently | Major | test | Zsolt Venczel | Zsolt Venczel | | | Do not invalidate blocks if toInvalidate is empty | Major | . | Zsolt Venczel | Zsolt Venczel | | | NM print inappropriate error log when node-labels is enabled | Minor | . | Yang Wang | Yang Wang | | | Allow read-only access to reserved raw for non-superusers | Major | namenode | Daryn Sharp | Rushabh S Shah | | | Output streams closed with IOUtils suppressing write errors | Major | . | Jason Lowe | Ajay Kumar | | | Container launching code suppresses close exceptions after writes | Major | nodemanager | Jason Lowe | Jim Brennan | | | Output streams closed with IOUtils suppressing write errors | Major | . | Jason Lowe | Jim Brennan | | | TestContainerLaunch# fails after YARN-7381 | Major | . | Jason Lowe | Jason Lowe | | | Several javadoc errors | Blocker | . | Sean Mackrory | Sean Mackrory | | | KDiag tries to load krb5.conf from KRB5CCNAME instead of KRB5\_CONFIG | Minor | security | Vipin Rathor | Vipin Rathor | | | NodeManager metrics return wrong value after update node resource | Major | . | Yang Wang | Yang Wang | | | TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently | Critical | test | Xiao Chen | Bharat Viswanadham | | | Remove the extra space in HdfsImageViewer.md | Trivial | documentation | Yiqun Lin | Rahul Pathak | | | [Atsv2] Define new set of configurations for reader and collectors to bind. | Major | . | Rohith Sharma K S | Rohith Sharma K S | | | JournalNode Test failures | Major | journal-node | Bharat Viswanadham | Bharat Viswanadham | | | ResourceRequest has a different default for allocationRequestId than Container | Major | . | Chandni Singh | Chandni Singh | | | Update Timeline Reader web app address in UI2 | Major | . | Rohith Sharma K S | Sunil Govindan | | | TestErasureCodigCLI testAll failing consistently. | Major | erasure-coding, hdfs | Rushabh S Shah | Ajay Kumar | | | Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSED | Major | . | Arun Suresh | Sampada Dehankar | | | Fix findbugs warning in

| Major | namenode | Wei-Chiu Chuang | Chris Douglas | | | Lock down version of doxia-module-markdown plugin | Blocker | . | Elek, Marton | Elek, Marton | | | NPE due to Invalid KerberosTicket in UGI | Major | . | Jitendra Nath Pandey | Mukul Kumar Singh | | | Skip validating priority acls while recovering applications | Blocker | resourcemanager | Charan Hebri | Sunil Govindan | | | Concurrent task progress updates causing NPE in Application Master | Blocker | mr-am | Gergo Repas | Gergo Repas | | | NM should reference the singleton JvmMetrics instance | Major | nodemanager | Haibo Chen | Haibo Chen | | | TestDNFencingWithReplication.testFencingStress fix mini cluster not yet active issue | Major | . | Zsolt Venczel | Zsolt Venczel | | | StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GB | Major | erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu | | | Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user | Major | capacity scheduler, yarn | Eric Payne | Eric Payne | | | queueUsagePercentage is coming as INF for getApp REST api call | Major | webapp | Sunil Govindan | Sunil Govindan | | | NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted | Major | hdfs | Manoj Govindassamy | Manoj Govindassamy | | | NPE in FiCaSchedulerApp when debug log enabled in async-scheduling mode | Major | capacityscheduler | Tao Yang | Tao Yang | | | RMAppImpl:Invalid event: START at KILLED | Major | resourcemanager | lujie | lujie | | | Invalid event: ATTEMPT\ADDED at FINAL\SAVING | Major | yarn | lujie | lujie | | | TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout | Major | erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu | | | Allow FS scheduler state dump to be turned on/off separately from FS debug log | Major | . | Wilfred Spiegelenburg | Wilfred Spiegelenburg | | | TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flaky | Major | . | Miklos Szegedi | Miklos Szegedi | | | Fix typo in YARN documentation | Minor | documentation | Takanobu Asanuma | Takanobu Asanuma | | | Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector | Minor | yarn | Prabhu Joseph | Prabhu Joseph | | | Sync rbw dir on the first hsync() to avoid file lost on power failure | Critical | . | Kanaka Kumar Avvaru | Vinayakumar B | | | Create the container log directory with correct sticky bit in C code | Major | nodemanager | Yufei Gu | Yufei Gu | | | TestContainerManagerSecurity.testContainerManager[Simple] flaky in trunk | Major | test | Botong Huang | Akira Ajisaka | |

TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB or larger than the test file | Major | hdfs | Zsolt Venczel | Zsolt Venczel | | | Make Datanode Netty reverse proxy port to be configurable | Major | datanode | Vinayakumar B | Vinayakumar B | | | Add an additional check to the validity of container and application ids passed to container-executor | Major | nodemanager | Miklos Szegedi | Yufei Gu | | | in FsShell, UGI params should be overidden through env vars(-D arg) | Major | . | Brahma Reddy Battula | Brahma Reddy Battula | | | CLI MiniCluster fails with ClassNotFoundException o.a.h.yarn.server.timelineservice.collector.TimelineCollectorManager | Major | . | Gera Shegalov | Gera Shegalov | | | StripedBlockReader#createBlockReader leaks socket on IOException | Critical | datanode, erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu | | | Typo in SecureMode.md | Trivial | documentation | Masahiro Tanaka | Masahiro Tanaka | | | Container-executor fails with segfault on certain OS configurations | Major | nodemanager | Gergo Repas | Gergo Repas | | | Distributed Shell should use timeline async api's | Major | distributed-shell | Sumana Sathish | Rohith Sharma K S | | | Encounter NullPointerException when using DecayRpcScheduler | Major | . | Tao Jie | Tao Jie | | | Possible race condition in JHS if the job is not loaded | Major | jobhistoryserver | Peter Bacsko | Peter Bacsko | | | prelaunch.err file not found exception on container failure | Major | . | Jonathan Hung | Keqiu Hu | | | Fix the incorrect spelling in HDFSHighAvailabilityWithQJM.md | Trivial | documentation | Jianfei Jiang | Jianfei Jiang | | | NM heartbeat stuck when responseId overflows MAX\_INT | Critical | . | Botong Huang | Botong Huang | | | MR should not try to clean up at first job attempt | Major | . | Takanobu Asanuma | Gergo Repas | | | Handling PathIsNotEmptyDirectoryException in DFSClient delete call | Major | hdfs-client | Nanda kumar | Nanda kumar | | | Task timeout in uber mode can crash AM | Major | mr-am | Akira Ajisaka | Peter Bacsko | | | [Atsv2] GSSException: No valid credentials provided - Failed to find any Kerberos tgt thrown by Timelinev2Client & HBaseClient in NM | Blocker | . | Sumana Sathish | Rohith Sharma K S | | | TestErasureCodingMultipleRacks#testSkewedRack3 is failing | Major | hdfs | Gabor Bota | Gabor Bota | | | Exception message is not printed when creating an encryption zone fails with AuthorizationException | Minor | encryption | fang zhenyi | fang zhenyi | | | A misleading variable's name in ApplicationAttemptEventDispatcher | Minor | resourcemanager | Jinjiang Ling | Jinjiang Ling | | | Improve Capacity Scheduler Async Scheduling to better handle node failures | Critical | . | Sumana Sathish | Wangda Tan | | | Add an option to not disable short-circuit reads on failures | Major | hdfs-client, performance | Andre Araujo | Xiao Chen | | | getErasureCodingPolicy should handle .snapshot dir better | Major | erasure-coding, hdfs, snapshots | Harshakiran Reddy | LiXin Ge | | | Map outputs implicitly rely on permissive umask for shuffle | Critical | mrv2 | Jason Lowe | Jason Lowe | | | Synchronization issue in FSDataSetImpl#moveBlock | Major | . | Ajay Kumar | Ajay Kumar | | | Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs. | Critical | hdfs, webhdfs | Yongjun Zhang | Yongjun Zhang | |

TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error | Major | test | Jason Lowe | Botong Huang | | | AmFilterInitializer should addFilter after fill all parameters | Critical | . | Sumana Sathish | Wangda Tan | | | NPE during container relaunch | Major | . | Billie Rinaldi | Jason Lowe | | | Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up | Major | tools | Jianfei Jiang | Jianfei Jiang | | | Snapshot diff could be corrupted after concat | Major | namenode, snapshots | Xiaoyu Yao | Xiaoyu Yao | | | refreshNamenodes does not support adding a new standby to a running DN | Critical | datanode, ha | Jian Fang | Ajith S | | | TestFixedLengthInputFormat#testFormatCompressedIn is flaky | Major | client, test | Peter Bacsko | Peter Bacsko | | | Token expiration edits may cause log corruption or deadlock | Critical | namenode | Daryn Sharp | Daryn Sharp | | | Revert YARN-6078 | Blocker | . | Billie Rinaldi | Billie Rinaldi | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec | Trivial | . | Chris Douglas | Chris Douglas | | | parallel tests don't work for Windows | Minor | test | Allen Wittenauer | Allen Wittenauer | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | AliyunOSS: Override listFiles and listLocatedStatus | Major | fs/oss | Genmao Yu | Genmao Yu | | | Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and setMaximumAllocationForMandatoryResources() | Minor | resourcemanager | Daniel Templeton | Manikandan R | | | RBF: Set MountTableResolver as default file resolver | Minor | . | igo Goiri | igo Goiri | | | AliyunOSS: support user agent configuration and include that & Hadoop version information to oss server | Major | fs, fs/oss | Sammi Chen | Sammi Chen | | | RBF: Add router admin commands usage in HDFS commands reference doc | Minor | documentation | Yiqun Lin | Yiqun Lin | | | RBF: Fix Javadoc parameter errors | Minor | . | Wei Yan | Wei Yan | | | Node information page in the old web UI should report resource types | Major | resourcemanager | Daniel Templeton | Gergely Novk | | | Webhdfs file system should get delegation token from kms provider. | Major | encryption, kms, webhdfs | Rushabh S Shah | Rushabh S Shah | | | Extend Distributed Shell to support launching job with opportunistic containers | Major | applications/distributed-shell | Weiwei Yang | Weiwei Yang | | | RBF: Complete logic for -readonly option of dfsrouteradmin add command | Major |

| Yiqun Lin | igo Goiri | | | Node resource is not parsed correctly for resource names containing dot | Major | nodemanager, resourcemanager | Jonathan Hung | Gergely Novk | | | Support multiple resource types in rmadmin updateNodeResource command | Major | nodemanager, resourcemanager | Daniel Templeton | Manikandan R | | | Add a flag in distributed shell to automatically PROMOTE opportunistic containers to guaranteed once they are started | Minor | applications/distributed-shell | Weiwei Yang | Weiwei Yang | | | RBF: Add more unit tests for router admin commands | Major | test | Yiqun Lin | Yiqun Lin | | | [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. | Critical | . | Rohith Sharma K S | Rohith Sharma K S | | | s3a input stream "normal" fadvise mode to be adaptive | Major | fs/s3 | Steve Loughran | Steve Loughran | | | NativeAzureFileSystem file rename is not atomic | Major | fs/azure | Shixiong Zhu | Thomas Marquardt | | | RBF: Mount table entries not properly updated in the local cache | Major | . | igo Goiri | igo Goiri | | | metricsTimeStart and metricsTimeEnd should be all lower case in the doc | Major | timelinereader | Haibo Chen | Haibo Chen | | | RBF: Control MountTableResolver cache size | Major | . | igo Goiri | igo Goiri | | | RBF: Support erasure coding methods in RouterRpcServer | Critical | . | igo Goiri | igo Goiri | | | Consider writing to both ats v1 & v2 from RM for smoother upgrades | Major | timelineserver | Vrushali C | Aaron Gresch | | | AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance | Major | fs/oss | wujinhu | wujinhu | | | RBF: Fix spurious TestRouterRpc#testProxyGetStats | Minor | . | igo Goiri | igo Goiri | | | some YARN container events have timestamp of -1 | Critical | . | Sangjin Lee | Haibo Chen | | | Enable user re-mapping for Docker containers in yarn-default.xml | Blocker | security, yarn | Eric Yang | Eric Yang | | | RBF: Federation Router State State Store internal API | Major | . | igo Goiri | igo Goiri | | | RBF: Heartbeat Router State | Major | . | igo Goiri | igo Goiri | | | RBF: Inconsistent Router OPTS config in branch-2 and branch-3 | Minor | . | Wei Yan | Wei Yan | | | Add CryptoInputStream to WebHdfsFileSystem read call. | Major | encryption, kms, webhdfs | Rushabh S Shah | Rushabh S Shah | | | RBF: Add a safe mode for the Router | Major | . | igo Goiri | igo Goiri | | | RBF: Expose the state of the Routers in the federation | Major | . | igo Goiri | igo Goiri | | | Move logging to slf4j in BlockPoolSliceStorage and Storage | Major | . | Ajay Kumar | Ajay Kumar | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | CryptoOutputStream should implement StreamCapabilities | Major | fs | Mike Drob | Xiao Chen | | | Add Unit Tests for ContainersLauncher | Major | . | Sampada Dehankar | Sampada Dehankar | | | Update the release year to 2018 | Blocker | build | Akira Ajisaka | Bharat Viswanadham | | | Remove tomcat from