Project: Apache Hadoop (App Definition and Development / Database)
File: CHANGELOG.2.7.2.md

<! --> | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Followup fixes after YARN-2019 regarding RM behavior when state-store error occurs | Major | . | Jian He | Jian He | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Rolling upgrade is restoring blocks from trash multiple times | Major | datanode | Nathan Roberts | Keisuke Ogiwara | | | Display count of nodes blacklisted by apps in the web UI | Major | capacityscheduler, resourcemanager | Varun Vasudev | Varun Vasudev | | | Implement listLocatedStatus for ViewFileSystem to speed up split calculation | Blocker | fs | Gera Shegalov | Gera Shegalov | | | Allow appending to existing SequenceFiles | Major | io | Stephen Rose | Kanaka Kumar Avvaru | | | Block scanner INFO message is spamming logs | Major | datanode | Yongjun Zhang | Yongjun Zhang | | | Optimize datanode writes for small writes and flushes | Critical | . | Kihwal Lee | Kihwal Lee | | | Upgrade Tomcat dependency to 6.0.44. | Major | build | Chris Nauroth | Chris Nauroth | | | YARN architecture document needs updating | Major | documentation | Allen Wittenauer | Brahma Reddy Battula | | | When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient | Major | . | Ming Ma | Ming Ma | | | Configurably turn off the saving of container info in Generic AHS | Major | timelineserver, yarn | Eric Payne | Eric Payne | | | Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded | Minor | test | Masatake Iwasaki | Masatake Iwasaki | | | Trash documentation should describe its directory structure and configurations | Minor | documentation | Suman Sehgal | Weiwei Yang | | | Allow NN to startup if there are files having a lease but are not under construction | Minor | namenode | Tsz Wo Nicholas Sze | Jing Zhao | | | AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. | Major | security | zhihai xu | zhihai xu | | | Remove duplicate close for LogWriter in AppLogAggregatorImpl#uploadLogsForContainers | Minor | nodemanager | zhihai xu | zhihai xu | | | For better error recovery, check if the directory exists before using it for localization. | Major | nodemanager | zhihai xu | zhihai xu | | | HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array | Major | performance | Staffan Friberg | Staffan Friberg | | | Recommission a datanode with 500k blocks may pause NN for 30 seconds | Major | namenode | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | | | Log slow name resolutions | Major |

| Sidharta Seethana | Sidharta Seethana | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Duplicate surefire plugin config in hadoop-common | Major | test | Andrey Klochkov | Andrey Klochkov | | | BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed | Critical | namenode | Akira Ajisaka | Akira Ajisaka | | | Allow better control of getContentSummary | Major | . | Kihwal Lee | Kihwal Lee | | | MiniYarnCluster should turn on timeline service if configured to do so | Major | . | Mit Desai | Mit Desai | | | Incorrect "nodes in service" metrics caused all writes to fail | Major | . | Ming Ma | Ming Ma | | | Change "DFSInputStream has been closed already" message to debug log level | Minor | hdfs-client | Charles Lamb | Charles Lamb | | | HarFs incorrectly declared as requiring an authority | Critical | fs | Gera Shegalov | Brahma Reddy Battula | | | Reduce cannot use more than 2G memory for the final merge | Major | mrv2 | stanley shi | Gera Shegalov | | | setStoragePolicy with folder behavior is different after cluster restart | Major | . | Peter Shi | Surendra Singh Lilhore | | | HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE\_FAILED state | Minor | jobhistoryserver | zhihai xu | zhihai xu | | | hdfs crypto class not found in Windows | Critical | scripts | Sumana Sathish | Anu Engineer | | | Avoid retry cache collision when Standby NameNode loading edits | Critical | namenode | Carrey Zhan | Ming Ma | | | JHS sorting on state column not working in webUi | Minor | jobhistoryserver | Bibin A Chundatt | zhihai xu | | | Should use equals when compare Resource in RMNodeImpl#ReconnectNodeTransition | Minor | resourcemanager | zhihai xu | zhihai xu | | | Correct HTTP method in WebHDFS document | Major | documentation | Akira Ajisaka | Brahma Reddy Battula | | | Two RMNodes for the same NodeId are used in RM sometimes after NM is reconnected. | Major | resourcemanager | zhihai xu | zhihai xu | | | org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well | Critical | . | Kris Geusebroek | Akira Ajisaka | | | Error handling in snappy decompressor throws invalid exceptions | Major | io, native | Todd Lipcon | Matt Foley | | | Preserve compatibility of ClientProtocol#rollingUpgrade after finalization | Critical | rolling upgrades | Andrew Wang | Andrew Wang | | | Several NPEs when deleting local files on NM recovery | Major | nodemanager | Karthik Kambatla | Varun Saxena | | | Prevent processing preemption events on the main RM dispatcher | Major | resourcemanager, scheduler | Jason Lowe | Varun Saxena | | | ShuffleHandler passes wrong "base" parameter to getMapOutputInfo if mapId is not in the

| Major | mrv2, nodemanager | zhihai xu | zhihai xu | | | ActiveStandbyElector shouldn't call monitorLockNodeAsync multiple times | Major | ha | zhihai xu | zhihai xu | | | [JDK8] 'mvn site' fails | Major | api, site | Akira Ajisaka | Brahma Reddy Battula | | | TestShuffleHandler#testGetMapOutputInfo is failing | Major | test | Devaraj K | zhihai xu | | | Bzip2Factory is not thread safe | Major | io | Jason Lowe | Brahma Reddy Battula | | | RawLocalFileSystem.listStatus() returns null for UNIX pipefile | Critical | . | Haohui Mai | Kanaka Kumar Avvaru | | | Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED | Critical | capacityscheduler, fairscheduler, resourcemanager | Peng Zhang | Peng Zhang | | | Application History Server UI NPEs when accessing apps run after RM restart | Major | timelineserver | Eric Payne | Eric Payne | | | AsyncDispatcher can hang while stopping if it is configured for draining events on stop | Critical | . | Varun Saxena | Varun Saxena | | | Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore | Critical | . | Junping Du | Jian He | | | Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared | Major | . | Zhe Zhang | Zhe Zhang | | | Fetch the application report from the AHS if the RM does not know about it | Major | . | Mit Desai | Mit Desai | | | ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. | Critical | nodemanager | zhihai xu | zhihai xu | | | AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected | Critical | resourcemanager | Rohith Sharma K S | Bibin A Chundatt | | | VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks | Major | datanode | Colin P. McCabe | Colin P. McCabe | | | Applications using FileContext fail with the default file system configured to be wasb/s3/etc. | Blocker | fs | Chris Nauroth | Chris Nauroth | | | MetricsSinkAdapter hangs when being stopped | Critical | . | Jian He | Brahma Reddy Battula | | | RM hangs on draining events | Major | . | Jian He | Jian He | | | Quota by storage type usage incorrectly initialized upon namenode restart | Major | namenode | Kihwal Lee | Xiaoyu Yao | | | Completed container whose app is finished is not removed from NMStateStore | Major | . | Jun Gong | Jun Gong | | | ClientRMService getApplications has high scheduler lock contention | Major | resourcemanager | Jason Lowe | Jason Lowe | | | HDFS concat should keep srcs order | Blocker | . | Yong Zhang | Yong Zhang | | | AM may fail instead of retrying if RM shuts down during the allocate call | Critical | . | Anubhav Dhoot | Anubhav Dhoot | | | HDFS architecture documentation of version

is outdated about append write support | Major | documentation | Hong Dai Thanh | Ajith S | | | Memory leak in ResourceManager with SIMPLE mode | Critical | resourcemanager | mujunchao | mujunchao | | | Enable optimized block reports | Major | . | Rushabh S Shah | Daryn Sharp | | | The remaining space check in BlockPlacementPolicyDefault is flawed | Critical | . | Kihwal Lee | Kihwal Lee | | | MapReduce doesn't set the HADOOP\_CLASSPATH for jar lib in distributed cache. | Critical | . | Junping Du | Junping Du | | | RMNode transitioned from RUNNING to REBOOTED because its response id had not been reset synchronously | Major | resourcemanager | Jun Gong | Jun Gong | | | Add a unit test for INotify functionality across a layout version upgrade | Major | namenode | Zhe Zhang | Zhe Zhang | | | NameNode refresh doesn't remove DataNodes that are no longer in the allowed list | Major | datanode, namenode | Daniel Templeton | Daniel Templeton | | | hadoop fs -getmerge doc is wrong | Major | documentation | Daniel Templeton | Jagadesh Kiran N | | | BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block | Minor | util | Ted Yu | Kiran Kumar M R | | | Flaw in registration bookeeping can make DN die on reconnect | Critical | . | Kihwal Lee | Kihwal Lee | | | Remove unimplemented option for \`hadoop fs -ls\` from document in branch-2.7 | Major | documentation, fs | Akira Ajisaka | Akira Ajisaka | | | Interrupted exception can occur when Client#stop is called | Minor | . | Oleg Zhurakousky | Kuhu Shukla | | | RM WebServices missing scheme for appattempts logLinks | Major | . | Jonathan Eagles | Jonathan Eagles | | | Capacity Scheduler headroom for DRF is wrong | Major | capacityscheduler | Chang Li | Chang Li | | | Stack trace is missing when error occurs in client protocol provider's constructor | Major | client | Chang Li | Chang Li | | | App local logs are leaked if log aggregation fails to initialize for the app | Major | log-aggregation, nodemanager | Jason Lowe | Jason Lowe | | | dfsadmin -metasave prints "NaN" for cache used% | Major | . | Archana T | Brahma Reddy Battula | | | ShuffleHandler can possibly exhaust nodemanager file descriptors | Major | mrv2, nodemanager | Nathan Roberts | Kuhu Shukla | | | Update document for the Storage policy name | Minor | documentation | J.Andreina | J.Andreina | | | MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks | Major | mr-am | Jason Lowe | Naganarasimha G R | | | LineRecordReader may give incomplete record and wrong position/key information for uncompressed input

| Critical | mrv2 | zhihai xu | zhihai xu | | | Task attempts that fail from the ASSIGNED state can disappear | Major | mr-am | Jason Lowe | Chang Li | | | FairScheduler: ContinuousSchedulingThread can fail to shutdown | Critical | fairscheduler | zhihai xu | zhihai xu | | | Doc updation for commands in HDFS Federation | Minor | documentation | J.Andreina | J.Andreina | | | WebAppProxyServlet should not redirect to RM page if AHS is enabled | Major | . | Mit Desai | Mit Desai | | | ApplicationHistoryServer reverses the order of the filters it gets | Major | timelineserver | Mit Desai | Mit Desai | | | Transfer failure during pipeline recovery causes permanent write failures | Critical | . | Kihwal Lee | Kihwal Lee | | | AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask | Critical | . | Bibin A Chundatt | Bibin A Chundatt | | | AMLauncher does not retry on failures when talking to NM | Critical | resourcemanager | Anubhav Dhoot | Anubhav Dhoot | | | Fix wrong value of JOB\_FINISHED event in JobHistoryEventHandler | Major | . | Shinichi Yamashita | Shinichi Yamashita | | | hadoop-project declares duplicate, conflicting curator dependencies | Minor | build | Steve Loughran | Rakesh R | | | ContainerMetrics unregisters during getMetrics and leads to ConcurrentModificationException | Major | nodemanager | Jason Lowe | zhihai xu | | | RMStateStore FENCED state doesn't work due to updateFencedState called by stateMachine.doTransition | Critical | resourcemanager | zhihai xu | zhihai xu | | | Slow datanode I/O can cause a wrong node to be marked bad | Critical | . | Kihwal Lee | Kihwal Lee | | | Incorrect javadoc in WritableUtils.java | Minor | documentation | Martin Petricek | Jagadesh Kiran N | | | Delayed rolling upgrade finalization can cause heartbeat expiration and write failures | Critical | . | Kihwal Lee | Walter Su | | | Reading small file (\< 512 bytes) that is open for append fails due to incorrect checksum | Blocker | . | Bogdan Raducanu | Jing Zhao | | | Interrupted client may try to fail-over and retry | Major | ipc | Kihwal Lee | Kihwal Lee | | | 2.7 RM app page is broken | Blocker | . | Chang Li | Chang Li | | | ZKRMStateStore shouldn't create new session without occurrance of SESSIONEXPIED | Blocker | resourcemanager | Bibin A Chundatt | Varun Saxena | | | Set SO\_KEEPALIVE on shuffle connections | Major | mrv2, nodemanager | Nathan Roberts | Chang Li | | | ACLs on root directory may be lost after NN restart | Critical | namenode | Xiao Chen | Xiao Chen | | | RM crashes with NPE if leaf queue becomes parent queue during restart | Major | capacityscheduler, resourcemanager | Jason Lowe | Varun Saxena | | | CORS support for ResourceManager REST API | Major | . | Prakash Ramachandran | Varun Vasudev | | | Slow delegation token renewal can severely prolong RM recovery | Major | resourcemanager | Jason Lowe | Sunil Govindan | | | DFSClient#callAppend() is not backward compatible for slightly older NameNodes | Blocker |

| Tony Wu | Tony Wu | | | Delayed heartbeat processing causes storm of subsequent heartbeats | Major | datanode | Chris Nauroth | Arpit Agarwal | | | Document fsck -blockId and -storagepolicy options in branch-2.7 | Major | documentation | Akira Ajisaka | Akira Ajisaka | | | Replication violates block placement policy. | Blocker | namenode | Rushabh S Shah | Rushabh S Shah | | | Race condition in MiniMRYarnCluster when getting history server address | Major | . | Jian He | Jian He | | | TestSubmitApplicationWithRMHA fails on branch-2.7 and branch-2.6 as some of the test cases time out | Major | . | Varun Saxena | Varun Saxena | | | TestJobHistoryEventHandler fails as AHS in MiniYarnCluster no longer binds to default port 8188 | Major | . | Varun Saxena | Varun Saxena | | | Memory leak for HistoryFileManager.getJobSummary() | Critical | jobhistoryserver | Junping Du | Junping Du | | | DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic | Major | distcp | Kuhu Shukla | Kuhu Shukla | | | Incessant retries if NoAuthException is thrown by Zookeeper in non HA mode | Major | resourcemanager | Varun Saxena | Varun Saxena | | | Fix TestDistributedShell timeout as AHS in MiniYarnCluster no longer binds to default port 8188 | Major | . | MENG DING | MENG DING | | | Make DataStreamer#block thread safe and verify genStamp in commitBlock | Critical | . | Chang Li | Chang Li | | | RM fail with noAuth error if switched from failover mode to non-failover mode | Major | resourcemanager | Jian He | Varun Saxena | | | [Branch-2] there are duplicate dependency definitions in pom's | Major | build | Sangjin Lee | Sangjin Lee | | | [Branch-2] Setting HADOOP\_HOME explicitly should be allowed | Blocker | scripts | Karthik Kambatla | Karthik Kambatla | | | Fix typo of property name in yarn-default.xml | Major | documentation | Anthony Rojas | Anthony Rojas | | | TestMRTimelineEventHandling fails | Major | test | Sangjin Lee | Sangjin Lee | | | Public resource localization fails with NPE | Blocker | nodemanager | Jason Lowe | Jason Lowe | | | getContentSummary() on standby should throw StandbyException | Critical | . | Brahma Reddy Battula | Brahma Reddy Battula | | | DistributedFileSystem#concat fails if the target path is relative. | Major | hdfs-client | Kazuho Fujii | Kazuho Fujii | | | ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster | Critical | timelineserver | Hitesh Shah | Vinod Kumar Vavilapalli | | | Bump up commons-collections version to 3.2.2 to address a security flaw | Blocker | build, security | Wei-Chiu Chuang | Wei-Chiu Chuang | | | NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations | Critical | resourcemanager | Varun Vasudev | Varun Vasudev | | | "Total megabyte-seconds" in job counters is slightly misleading | Minor |

| Nathan Roberts | Nathan Roberts | | | FileSystemNodeLabelStore should check for root dir existence on startup | Major | resourcemanager | Jason Lowe | Kuhu Shukla | | | hdfs and nfs builds broken on -missing compile-time dependency on netty | Major | nfs | Konstantin Boudnik | Tom Zeng | | | multibyte delimiters with LineRecordReader cause duplicate records | Major | mrv1, mrv2 | Dustin Cote | Wilfred Spiegelenburg | | | Rollingupgrade finalization is not backward compatible | Blocker | . | Kihwal Lee | Kihwal Lee | | | Encryption zone on root not loaded from fsimage after NN restart | Critical | . | Xiao Chen | Xiao Chen | | | DFSClient deadlock when close file and failed to renew lease | Blocker | hdfs-client | DENG FEI | Brahma Reddy Battula | | | ZKRMStateStore.syncInternal shouldn't wait for sync completion for avoiding blocking ZK's event thread | Blocker | . | Tsuyoshi Ozawa | Tsuyoshi Ozawa | | | Fix deadlock in RMAppImpl | Blocker | . | Yesha Vora | Jian He | | | NodeManager Disk Checker parameter documentation is not correct | Minor | documentation, nodemanager | Takashi Ohnishi | Weiwei Yang | | | Datanode may deadlock while handling a bad volume | Blocker | . | Kihwal Lee | Walter Su | | | Reduce client failures during datanode restart | Major | . | Kihwal Lee | Kihwal Lee | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | [JDK 8] TestClientRMService.testGetLabelsToNodes fails | Major | test | Robert Kanter | Robert Kanter | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | getTransferredContainers can be a bottleneck during AM registration | Major | scheduler | Jason Lowe | Sunil Govindan | | | ATS Web Performance issue at load time when large number of jobs | Major | resourcemanager, webapp, yarn | Xuan Gong | Xuan Gong | | | Fixed the typo with the configuration name: APPLICATION\HISTORY\PREFIX\MAX\APPS | Major | resourcemanager, webapp, yarn | Xuan Gong | Xuan Gong | | | Allow jobs to be submitted to reservation that is active but does not have any allocations | Major | capacityscheduler, fairscheduler, resourcemanager | Subru Krishnan | Subru Krishnan | | | RM HA UI redirection needs to be fixed when both RMs are in standby mode | Major | resourcemanager | Xuan Gong | Xuan Gong | | | Add documentation for node labels feature | Major | documentation | Gururaj Shetty | Wangda Tan | | | Both RM in active state when Admin#transitionToActive failure from refeshAll() | Critical | resourcemanager | Bibin A Chundatt | Bibin A Chundatt | | | RM should print alert messages if Zookeeper and Resourcemanager gets connection issue | Critical | yarn | Yesha Vora | Xuan Gong | | | Host framework UIs in YARN for use with the ATS | Major | timelineserver | Jonathan Eagles | Jonathan Eagles | | | Killing a container that is localizing can orphan resources in the DOWNLOADING state | Major | nodemanager | Jason Lowe | Varun Saxena |