Project: Apache Hadoop (App Definition and Development / Database)
File: CHANGELOG.2.6.1.md

<! --> | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Add a command that allows sysadmins to manually trigger full block reports from a DN | Major | datanode | Colin P. McCabe | Colin P. McCabe | | | Improve yarn container command | Major | . | Jian He | Naganarasimha G R | | | Improve the concurrent access on FsVolumeList | Major | datanode | Lei (Eddy) Xu | Lei (Eddy) Xu | | | Improve log reporting during block report rpc failure | Minor | datanode | Charles Lamb | Charles Lamb | | | HDFS inotify should have the ability to determine what txid it has read up to | Major | hdfs-client | Colin P. McCabe | Colin P. McCabe | | | JMX metrics aren't accessible when NN is busy | Major | . | Ming Ma | Ming Ma | | | FastByteComparisons: do not use UNSAFE\_COMPARER on the SPARC architecture because it is slower there | Minor | io, performance, util | Suman Somasundar | Suman Somasundar | | | Configuration variable expansion regex expensive for long values | Major | conf | Dmitriy V. Ryaboy | Gera Shegalov | | | Clarify application states on the web UI | Major | . | Jian He | Jian He | | | Refactor JobSubmitter#copyAndConfigureFiles into it's own class | Minor | . | Chris Trezzo | Chris Trezzo | | | Add a "kill application" button to Resource Manager's Web UI | Minor | resourcemanager | Ryu Kobayashi | Ryu Kobayashi | | | Display count of nodes blacklisted by apps in the web UI | Major | capacityscheduler, resourcemanager | Varun Vasudev | Varun Vasudev | | | Implement listLocatedStatus for ViewFileSystem to speed up split calculation | Blocker | fs | Gera Shegalov | Gera Shegalov | | | Allow appending to existing SequenceFiles | Major | io | Stephen Rose | Kanaka Kumar Avvaru | | | When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient | Major | . | Ming Ma | Ming Ma | | | Configurably turn off the saving of container info in Generic AHS | Major | timelineserver, yarn | Eric Payne | Eric Payne | | | Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded | Minor | test | Masatake Iwasaki | Masatake Iwasaki | | | Allow NN to startup if there are files having a lease but are not under construction | Minor | namenode | Tsz Wo Nicholas Sze | Jing Zhao | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | processIncrementalBlockReport performance degradation | Critical | namenode | Daryn Sharp | Eric Payne | | | DataNode#transferBlock should report blocks that don't exist using reportBadBlock | Major | datanode, namenode | Yongjun Zhang | Yongjun Zhang | | | Snapshot read can reveal future bytes for appended files. | Major | hdfs-client | Konstantin Shvachko | Tao Luo | | |

UGI#reloginFromKeytab on Java 8 | Major | security | Tobi Vollebregt | Stephen Chu | | | Application recovery throw InvalidStateTransitonException: Invalid event: ATTEMPT\_KILLED at ACCEPTED | Critical | resourcemanager | Rohith Sharma K S | Rohith Sharma K S | | | NM fail to start with NPE during container recovery | Major | nodemanager | zhihai xu | zhihai xu | | | RM web UI: app page will crash if app is failed before any attempt has been created | Major | webapp | Zhijie Shen | Wangda Tan | | | Remove stale block invalidation work when DN re-registers with different UUID | Major | namenode | Zhe Zhang | Zhe Zhang | | | Application recovery continuously fails with "Application with id already present. Cannot duplicate" | Critical | resourcemanager | Rohith Sharma K S | Rohith Sharma K S | | | NameNode block deletion logging uses incorrect appender. | Minor | namenode | Chris Nauroth | Chris Nauroth | | | Prevent the Namenode's LeaseManager from looping forever in checkLeases | Critical | hdfs-client, namenode | Zesheng Wu | Ravi Prakash | | | CapacitySchedulerPage shows HTML tags for a queue's Active Users | Major | capacityscheduler | Jason Lowe | Jason Lowe | | | Fix deadlock in DomainSocketWatcher when the notification pipe is full | Major | . | yunjiong zhao | yunjiong zhao | | | AggregatedLogsBlock page can infinitely loop if the aggregated log file is corrupted | Blocker | . | Jason Lowe | Varun Saxena | | | When ACL's are enabled, if RM switches then application can not be viewed from web. | Major | resourcemanager | Rohith Sharma K S | Rohith Sharma K S | | | Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps | Blocker | resourcemanager | Naganarasimha G R | Naganarasimha G R | | | Overflow is not properly handled in caclulating final iv for AES CTR | Blocker | security | Haifeng Chen | Haifeng Chen | | | Fix SSLFactory truststore reloader thread leak in KMSClientProvider | Major | kms | Arun Suresh | Arun Suresh | | | Incorrect locking in FsVolumeList#checkDirs can hang datanodes | Critical | datanode | Noah Lorang | Noah Lorang | | | FSLeafQueue can throw ConcurrentModificationException | Major | fairscheduler | Wilfred Spiegelenburg | Wilfred Spiegelenburg | | | Namenode restart after large deletions can cause slow processReport (due to logging) | Major | namenode | Arpit Agarwal | Arpit Agarwal | | | Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook | Critical | resourcemanager | Rohith Sharma K S | Rohith Sharma K S | | | Update the NameNode's Group Cache in the background when possible | Minor | . | Chris Li | Chris Li | | | Reducers do not validate checksum of map outputs when fetching directly to disk | Major | mrv2 | Eric Payne | Eric Payne | | | RM prematurely cancels tokens for jobs that submit jobs (oozie) | Blocker | resourcemanager | Daryn Sharp | Jian He | | | change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration | Major | datanode, test | Liang Xie | Liang Xie | | | Datanode upgrade to BLOCKID\BASED\LAYOUT fails if duplicate block files are present in the same volume | Blocker | . | Kihwal Lee | Colin

McCabe | | | Incorrect version check in RMStateStore | Major | . | Jian He | Rohith Sharma K S | | | NPE thrown when RM restart after queue is STOPPED. There after RM can not recovery application's and remain in standby | Critical | resourcemanager, scheduler | Nishan Shetty | Rohith Sharma K S | | | ZKRMStateStore crashes due to session expiry | Blocker | resourcemanager | Karthik Kambatla | Karthik Kambatla | | | ConcurrentModificationException in CapacityScheduler's LeafQueue | Major | capacityscheduler, resourcemanager, scheduler | Jason Tufo | Rohith Sharma K S | | | ResourceManager crashes with NPE while getting queue info | Critical | . | Jason Tufo | Varun Saxena | | | NM keeps sending already-sent completed containers to RM until containers are removed from context | Major | nodemanager | Chengbing Liu | Chengbing Liu | | | NameNode should prune dead storages from storageMap | Major | namenode | Arpit Agarwal | Arpit Agarwal | | | Datanode sometimes does not shutdown on receiving upgrade shutdown command | Major | . | Kihwal Lee | Eric Payne | | | SecondaryNameNode need twice memory when calling reloadFromImageFile | Major | namenode | yunjiong zhao | yunjiong zhao | | | maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications. | Critical | resourcemanager | Wangda Tan | Craig Welch | | | The size of header buffer of HttpServer is too small when HTTPS is enabled | Major | security | Benoy Antony | Benoy Antony | | | Fix removal of dynamically added DN volumes | Major | datanode | Lei (Eddy) Xu | Lei (Eddy) Xu | | | Fix NPE when namenode transition to active during startup by adding checkNNStartup() in NameNodeRpcServer | Major | auto-failover, ha | suja s | Vinayakumar B | | | Upgrade should generate a unique storage ID for each volume | Critical | . | Lars Francke | Arpit Agarwal | | | Use correct UGI when KMSClientProvider is called by a proxy user | Major | . | Arun Suresh | Arun Suresh | | | LocalizerRunner should give DIE action when all resources are localized | Major | nodemanager | Chengbing Liu | Chengbing Liu | | | "mvn package -Pdist,docs -DskipTests -Dtar" fails because of non-ascii characters | Blocker | . | Tsuyoshi Ozawa | Tsuyoshi Ozawa | | | AMRMClientImpl does not update AMRM token properly | Blocker | client | Jason Lowe | Jason Lowe | | | MR AM does not survive RM restart if RM activated a new AMRM secret key | Blocker | mr-am | Jason Lowe | Jason Lowe | | | Edit log corruption due to delayed block removal again | Major | namenode | Yongjun Zhang | Yongjun Zhang | | | NFS: readdir/readdirplus return null directory attribute on failure | Major | nfs | Arpit Agarwal | Arpit Agarwal | | | Multiple mappers with DBInputFormat don't work because of reusing conections | Major | mrv2 | Kannan Rajah | Kannan Rajah | | | reset timer for liveness monitors after RM recovery | Major | resourcemanager | Jun Gong | Jun Gong | | | Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one

| Major | datanode | Chris Nauroth | Vinayakumar B | | | Job History Link in RM UI is redirecting to the URL which contains Job Id twice | Major | webapp | Devaraj K | Devaraj K | | | RPC Server Reader thread can't shutdown if RPCCallQueue is full | Major | . | Ming Ma | Ming Ma | | | Post-2.6 namenode may not start up with an image containing inodes created with an old release. | Blocker | . | Kihwal Lee | Rushabh S Shah | | | Prevent ConcurrentModificationException while closing domain sockets during shutdown of DomainSocketWatcher thread. | Critical | net | Liang Xie | Chris Nauroth | | | Connection timeouts to nodemanagers are retried at multiple levels | Blocker | . | Jason Lowe | Jason Lowe | | | secondary filter matches entites which do not have the key being filtered for. | Major | timelineserver | Prakash Ramachandran | Zhijie Shen | | | Active NN and standby NN have different live nodes | Major | datanode | Ming Ma | Ming Ma | | | fix zkfc hung issue due to not catching exception in a corner case | Major | ha | Liang Xie | Liang Xie | | | WebAppProxy does not support a final tracking url which has query fragments and params | Major | . | Hitesh Shah | Jian He | | | Fix CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1) | Blocker | . | Jason Lowe | Craig Welch | | | NameNodeEditLogRoller can keep printing "Swallowing exception" message | Critical | . | Jing Zhao | Jing Zhao | | | RMNodeImpl#ReconnectNodeTransition should send scheduler events in sequential order | Critical | resourcemanager | Rohith Sharma K S | Rohith Sharma K S | | | FairScheduler: Changing queueMaxRunningApps interferes with pending jobs | Critical | . | Siqi Li | Siqi Li | | | Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old client | Critical | resourcemanager | zhihai xu | zhihai xu | | | oneByteBuf in CryptoInputStream and CryptoOutputStream should be non static | Critical | io | Sean Busbey | Sean Busbey | | | Datanode should not trust the generation stamp provided by client | Critical | datanode | vitthal (Suhas) Gogate | Tsz Wo Nicholas Sze | | | Timeline renew delegation token fails when RM user's TGT is expired | Critical | . | Jonathan Eagles | Zhijie Shen | | | TimelineClient kerberos authentication failure uses wrong login context. | Major | . | Jonathan Eagles | Daryn Sharp | | | DataNode does not release the volume lock when adding a volume fails. | Major | datanode | Lei (Eddy) Xu | Lei (Eddy) Xu | | | Timelineserver applies the ACL rules after applying the limit on the number of records | Major | . | Prakash Ramachandran | Chang Li | | | The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error | Major | . | Colin P. McCabe | Colin P. McCabe | | | inotify unable fetch pre-upgrade edit log segments once upgrade starts | Major |

| Zhe Zhang | Zhe Zhang | | | Edit log corruption can happen if append fails with a quota violation | Blocker | namenode | Kihwal Lee | Jing Zhao | | | commitBlockSynchronization() does not remove locations | Blocker | namenode | Konstantin Shvachko | Yi Liu | | | Missing NullPointer check in AppSchedulingInfo causes RM to die | Blocker | resourcemanager | Giovanni Matteo Fumarola | Brahma Reddy Battula | | | Getting application(s) goes wrong when app finishes before starting the attempt | Critical | timelineserver | Zhijie Shen | Zhijie Shen | | | NullPointerException in BlockSender | Blocker | datanode | Tsz Wo Nicholas Sze | Brahma Reddy Battula | | | The full block report should prune zombie storages even if they're not empty | Critical | . | Lei (Eddy) Xu | Colin P. McCabe | | | favoring decommissioning node for replication can cause a block to stay underreplicated for long periods | Major | namenode | Nathan Roberts | Nathan Roberts | | | Read timeout when retrying a fetch error can be fatal to a reducer | Blocker | . | Jason Lowe | Jason Lowe | | | FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time | Major | . | zhouyingchao | zhouyingchao | | | Reserved RBW space is not released if client terminates while writing block | Major | datanode | Arpit Agarwal | Arpit Agarwal | | | Allow better control of getContentSummary | Major | . | Kihwal Lee | Kihwal Lee | | | MiniYarnCluster should turn on timeline service if configured to do so | Major | . | Mit Desai | Mit Desai | | | The token is not renewed properly if it's shared by jobs (oozie) in DelegationTokenRenewer | Blocker | security | Yi Liu | Daryn Sharp | | | NameNode Failover during HA upgrade can cause DataNode to finalize upgrade | Blocker | ha | Jing Zhao | Jing Zhao | | | Task list sort by task id broken | Minor | . | Siqi Li | Siqi Li | | | RM fails to come up with error "Failed to load/recover state" when mem settings are changed | Critical | yarn | Sumana Sathish | Jian He | | | MR2 can't run local jobs with -libjars command options which is a regression from MR1 | Critical | mrv2 | zhihai xu | zhihai xu | | | Regression: s3n read failure recovery broken | Major | fs/s3 | Takenori Sato | Takenori Sato | | | DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm | Major | . | Eric Payne | Colin P. McCabe | | | Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode | Blocker | caching | Gopal V | Colin

McCabe | | | Race condition in LocalizerRunner kills localizer before localizing all resources | Critical | nodemanager | zhihai xu | zhihai xu | | | Uber jobs fail to update AMRM token when it rolls over | Blocker | mr-am | Jason Lowe | Jason Lowe | | | HarFs incorrectly declared as requiring an authority | Critical | fs | Gera Shegalov | Brahma Reddy Battula | | | Reduce cannot use more than 2G memory for the final merge | Major | mrv2 | stanley shi | Gera Shegalov | | | setStoragePolicy with folder behavior is different after cluster restart | Major | . | Peter Shi | Surendra Singh Lilhore | | | Incremental BlockReport will dramatically slow down the startup of a namenode | Major | . | Hui Zheng | Walter Su | | | Rolling upgrade readiness is not updated in jmx until query command is issued. | Critical | . | Kihwal Lee | Brahma Reddy Battula | | | Standby namenode doesn't process DELETED\_BLOCK if the add block request is in edit log. | Major | . | Rushabh S Shah | Rushabh S Shah | | | NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host | Critical | . | Junping Du | Junping Du | | | ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster | Major | resourcemanager, webapp | Weiwei Yang | Weiwei Yang | | | Pending block replication can get stuck using older genstamp | Major | namenode | Nathan Roberts | Nathan Roberts | | | hdfs crypto class not found in Windows | Critical | scripts | Sumana Sathish | Anu Engineer | | | Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop | Blocker | security | Mike Yoder | Larry McCay | | | Avoid retry cache collision when Standby NameNode loading edits | Critical | namenode | Carrey Zhan | Ming Ma | | | App submission via REST API is broken in secure mode due to Timeline DT service address is empty | Blocker | resourcemanager, timelineserver | Zhijie Shen | Zhijie Shen | | | DN startup may cause severe data loss | Blocker | datanode | Daryn Sharp | Daryn Sharp | | | create() always retried with hardcoded timeout when file already exists with open lease | Major | hdfs-client | Andrey Stepachev | J.Andreina | | | NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled | Critical | . | Peng Zhang | Rohith Sharma K S | | | Fix DominantRC#compare() does not work as expected if cluster resource is empty | Blocker | resourcemanager | Bibin A Chundatt | Rohith Sharma K S | | | Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs instead of copying them | Critical | . | Zhe Zhang | Zhe Zhang | | | Resource Localization fails on a cluster due to existing cache directories | Critical | nodemanager | Ranga Swamy | Brahma Reddy Battula | | | Error handling in snappy decompressor throws invalid exceptions | Major | io, native | Todd Lipcon | Matt Foley | | | NM fails to read files from full disks which can lead to container logs being lost and other issues | Blocker | log-aggregation, nodemanager | Varun Saxena | Varun Saxena | | | AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected | Critical | resourcemanager | Rohith Sharma K S | Bibin A Chundatt | | | MetricsSinkAdapter hangs when being stopped | Critical | . | Jian He | Brahma Reddy Battula | | | RM hangs on draining events | Major |

| Jian He | Jian He | | | ClientRMService getApplications has high scheduler lock contention | Major | resourcemanager | Jason Lowe | Jason Lowe | | | The remaining space check in BlockPlacementPolicyDefault is flawed | Critical | . | Kihwal Lee | Kihwal Lee | | | Add a unit test for INotify functionality across a layout version upgrade | Major | namenode | Zhe Zhang | Zhe Zhang | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Make adding a new data directory to the DataNode an atomic operation and improve error handling | Major | datanode | Lei (Eddy) Xu | Lei (Eddy) Xu | | | ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containers | Major | timelineserver | Robert Kanter | Robert Kanter | | | LeveldbTimelineStore does not handle db exceptions properly | Major | . | Jason Lowe | Varun Saxena | | | CapacityScheduler should be notified when labels on nodes changed | Major | . | Wangda Tan | Wangda Tan | | | Create common ResourceUsage class to track labeled resource usages in Capacity Scheduler | Major | api, client, resourcemanager | Wangda Tan | Wangda Tan | | | NM dies because of the failure of resource localization | Major | nodemanager | Wang Hao | Varun Saxena | | | Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label. | Major | api, client, resourcemanager | Wangda Tan | Wangda Tan | | | Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues | Major | capacityscheduler | Wangda Tan | Wangda Tan | | | Ensure only single node labels specified in resource request / host, and node label expression only specified when resourceName=ANY | Major | capacityscheduler, resourcemanager | Wangda Tan | Wangda Tan | | | Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label | Major | api, client, resourcemanager | Wangda Tan | Wangda Tan | | | Synchronize RM and Generic History Service Web-UIs | Major | . | Zhijie Shen | Xuan Gong | | | ContainerReport should have nodeHttpAddress | Major | . | Zhijie Shen | Xuan Gong | | | Make CryptoOutputStream behave like DFSOutputStream wrt synchronization | Critical | fs | Sean Busbey | Sean Busbey | | | Sort by Application id, AppAttempt & ContainerID doesn't work in ATS / RM web ui | Minor | timelineserver | Jeff Zhang | Naganarasimha G R | | | CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue | Critical | capacityscheduler | Jason Lowe | Jason Lowe | | | AM logs link missing in the RM UI for a completed app | Blocker | . | Hitesh Shah | Xuan Gong | | | Don't fail RM if queue's configured labels are not existed in cluster-node-labels | Major | resourcemanager | Rohith Sharma K S | Wangda Tan | | | ATS Web Performance issue at load time when large number of jobs | Major | resourcemanager, webapp, yarn | Xuan Gong | Xuan Gong | | | Fixed the typo with the configuration name: APPLICATION\HISTORY\PREFIX\MAX\APPS | Major | resourcemanager, webapp, yarn | Xuan Gong | Xuan Gong | | | Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500) | Major | timelineserver | Jonathan Eagles | Mit Desai |