Project: Apache Hadoop (App Definition and Development / Database)
File: CHANGELOG.0.23.7.md

<! --> | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | DFS Scalability: Incremental block reports | Major | datanode, namenode | dhruba borthakur | Tomasz Nykiel | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Add shell command to dump file checksums | Major | fs, tools | Todd Lipcon | Todd Lipcon | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Improve log levels when exceptions caught in RPC handler | Minor | ipc | Todd Lipcon | Todd Lipcon | | | Add toString to FileStatus | Minor | . | Jakob Homan | SreeHari | | | make hadoop script recognize a full set of deprecated commands | Major | scripts | Roman Shaposhnik | Roman Shaposhnik | | | provide an option for IPC server users to avoid printing stack information for certain exceptions | Major | ipc | Brandon Li | Brandon Li | | | avoid printing stack information for SafeModeException | Major | namenode | Brandon Li | Brandon Li | | | Lower native-hadoop library log from info to debug | Major | native | Eli Collins | Hzr Sefa rken | | | Add missing fields to FIleStatus.toString | Trivial | . | Jonathan Allen | Jonathan Allen | | | TrackerDistributedCacheManager issues too many getFileStatus calls | Major | mrv1, tasktracker | Sandy Ryza | Sandy Ryza | | | CompressionCodecFactory#getCodecClasses should trim the result of parsing by Configuration. | Major | io | Tsuyoshi Ozawa | Tsuyoshi Ozawa | | | FileUtil#fullyDelete should grant the target directories +rwx permissions before trying to delete them | Minor | . | Ivan A. Veselovsky | Ivan A. Veselovsky | | | Warn if java.library.path is used for AM or Task | Major | mrv2 | Robert Joseph Evans | Robert Parker | | | Enable multiple pi jobs to run in parallel | Minor | examples | Sandy Ryza | Sandy Ryza | | | parametrize Clover "generateXxx" properties to make them re-definable via -D in mvn calls | Minor | . | Ivan A. Veselovsky | Ivan A. Veselovsky | | | Unnecessary conversions in History Events | Trivial | jobhistoryserver | Robert Joseph Evans | Chu Tong | | | JSONify DataTables input data for Attempts page | Major | jobhistoryserver, mr-am | Ravi Prakash | Ravi Prakash | | | Capacity Scheduler web page should show list of active users per queue like it used to (in 1.x) | Major | capacityscheduler | Ravi Prakash | Ravi Prakash | | | Allow UGI of current connection to be queried | Critical | ipc | Daryn Sharp | Daryn Sharp | | | Expose

for tests | Major | security | Daryn Sharp | Daryn Sharp | | | Native-code implementation of bzip2 codec | Major | io | Govind Kamat | Govind Kamat | | | Add tokens from -tokenCacheFile into UGI | Major | security | Daryn Sharp | Daryn Sharp | | | allow OS scheduling priority of NM to be different than the containers it launches | Major | nodemanager | Thomas Graves | Thomas Graves | | | capture the ulimit info after printing the log to the console | Trivial | . | Arpit Gupta | Arpit Gupta | | | Construct debug strings conditionally in ShuffleHandler.Shuffle#sendMapOutput() | Trivial | . | Karthik Kambatla | Karthik Kambatla | | | Capture ulimit info in the logs at service start time | Major | . | Arpit Gupta | Arpit Gupta | | | make CS node-locality-delay refreshable | Major | capacityscheduler | Thomas Graves | Thomas Graves | | | Recovery should restore task state from job history info directly | Critical | mr-am | Jason Lowe | Jason Lowe | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split. | Major | mrv2 | Zhenxiao Luo | Bhallamudi Venkata Siva Kamesh | | | Block Replication should respect under-replication block priority | Major | namenode | Hairong Kuang | Uma Maheswara Rao G | | | JobClient getJob(JobID) should return NULL if the job does not exist (for backwards compatibility) | Major | mrv2 | Alejandro Abdelnur | Alejandro Abdelnur | | | Support non-SUN JREs in UserGroupInformation | Major | . | Stephen Watt | Devaraj Das | | | SecurityUtil.fetchServiceTicket broken after HADOOP-6941 | Blocker | security | Todd Lipcon | Todd Lipcon | | | Overreplicated block is not deleted even after the replication factor is reduced after sync follwed by closing that file | Minor | namenode | J.Andreina | Ashish Singhi | | | HDFS considers blocks under-replicated if topology script is configured with only 1 rack | Major | . | Aaron T. Myers | Aaron T. Myers | | | Changes to support Kerberos with non Sun JVM (HADOOP-6941) broke SPNEGO | Blocker | security | Alejandro Abdelnur | Devaraj Das | | | Fix TestCombineFileInputFormat.testForEmptyFile | Major | test | Kihwal Lee | Ilya Katsov | | | Change package of YarnClient to include apache | Major | client | Bikas Saha | Bikas Saha | | | Provide support for missing yarn commands | Major | client | Devaraj K | Devaraj K | | | AppRejectedTransition does not unregister app from master service and scheduler | Major | . | Bikas Saha | Bikas Saha | | | uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on | Major |

| Arpit Gupta | Arpit Gupta | | | On file deletion remove corresponding blocks pending replication | Minor | namenode | Jing Zhao | Jing Zhao | | | TestNameNodeMetrics.testCorruptBlock fails intermittently | Major | test | Uma Maheswara Rao G | Jing Zhao | | | Killing an unassigned task attempt causes the job to fail | Major | mrv2 | Tom White | Mayank Bansal | | | cannot run two local jobs in parallel from the same gateway. | Major | . | Araceli Henley | Sandy Ryza | | | FsPermission should have different default value, 777 for directory and 666 for file | Minor | . | Binglin Chang | Binglin Chang | | | HTTP Error 413 full HEAD if using kerberos authentication | Major | net | Moritz Moeller | Moritz Moeller | | | Potential deadlock in FileSystem.Cache/IPC/UGI | Major | fs | Tom White | Tom White | | | hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script | Minor | scripts | Jason Lowe | Andy Isaacson | | | Type conversion of map completion events leads to performance problems with large jobs | Critical | mr-am | Jason Lowe | Jason Lowe | | | Parametrize staging URL for the uniformity of distributionManagement | Major | build | Konstantin Boudnik | Konstantin Boudnik | | | Secondary namenode shuts down immediately after startup | Blocker | namenode | Jason Lowe | Arpit Agarwal | | | WebAppProxyServer exits immediately after startup | Blocker | . | Liang Xie | Liang Xie | | | hadoop.http.authentication.signature.secret.file docs should not state that secret is randomly generated | Minor | security | Eli Collins | Alejandro Abdelnur | | | Add space between total transaction time and number of transactions in FSEditLog#printStatistics | Trivial | . | Stephen Chu | Stephen Chu | | | update web services docs for RM clusterMetrics | Major | resourcemanager | Thomas Graves | Ravi Prakash | | | packaging docs is broken | Major | documentation | Thomas Graves | Andy Isaacson | | | NN accepts incremental BR as IBR in safemode | Critical | namenode | Daryn Sharp | Daryn Sharp | | | Convert remaining xdocs to APT | Major | . | Andy Isaacson | Andy Isaacson | | | Capacity Scheduler maximum-capacity value -1 is invalid | Major | capacityscheduler | Thomas Graves | Xuan Gong | | | MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality | Major | applicationmaster | Bikas Saha | Bikas Saha | | | TestKeyValueTextInputFormat test fails with Open JDK 7 | Major | test | Arpit Agarwal | Arpit Agarwal | | | AggregatedLogDeletionService can take too long to delete logs | Major | . | Jason Lowe | Jason Lowe | | | 2NN will fail to checkpoint after an HDFS upgrade from a pre-federation version of HDFS | Major | namenode | Aaron T. Myers | Aaron

Myers | | | Allow apps to concurrently register tokens for renewal | Critical | . | Daryn Sharp | Daryn Sharp | | | HadoopPipes misuses fprintf | Major | pipes | Andy Isaacson | Andy Isaacson | | | HarFileSystem may leak file handle | Major | fs | Chris Nauroth | Chris Nauroth | | | App submission should not be synchronized | Major | resourcemanager | Daryn Sharp | Daryn Sharp | | | RM app submission jams under load | Blocker | resourcemanager | Daryn Sharp | Daryn Sharp | | | FsShell rm -f fails for non-matching globs | Blocker | fs | Daryn Sharp | Daryn Sharp | | | AM does not tell the RM about container requests that are no longer needed | Major | . | Bikas Saha | Bikas Saha | | | SortedMapWritable violates contract of Map interface for equals() and hashCode() | Minor | io | Patrick Hunt | Surenkumar Nihalani | | | Unexpected extra results when using webUI table search | Minor | . | Jason Lowe | Ravi Prakash | | | HDFS docs not linked from top level | Major | documentation | Thomas Graves | Andy Isaacson | | | command manual dfsadmin missing entry for restoreFailedStorage option | Major | . | Thomas Graves | Andy Isaacson | | | TaskImpl.getCounters() can return the counters for the wrong task attempt when task is speculating | Critical | mr-am | Jason Lowe | Jason Lowe | | | Allow client-side lease renewal to be retried beyond soft-limit | Major | hdfs-client | Kihwal Lee | Kihwal Lee | | | SortedMapWritable#putAll() doesn't add key/value classes to the map | Major | io | Karthik Kambatla | Karthik Kambatla | | | Killing the Task Attempt slated for commit does not clear the value from the Task commitAttempt member | Critical | mrv1 | Robert Parker | Robert Parker | | | RM can return null application resource usage report leading to NPE in client | Critical | resourcemanager | Jason Lowe | Jason Lowe | | | RM should point tracking URL to RM web page when app fails to start | Major | resourcemanager | Jason Lowe | Jason Lowe | | | NN is unresponsive and loses heartbeats of DNs when Hadoop is configured to use LDAP and LDAP has issues | Minor | namenode | Xiaobo Peng | Xiaobo Peng | | | Failure to download a public resource on a node prevents further downloads of the resource from that node | Critical | nodemanager | Jason Lowe | Jason Lowe | | | IPC.Server incorrectly sets UGI auth type | Major | ipc | Daryn Sharp | Daryn Sharp | | | AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has

| Major | mrv2 | Jason Lowe | Jason Lowe | | | Resource Manager not logging the health\check\script result when taking it out | Major | resourcemanager | Thomas Graves | Jason Lowe | | | Apps that have completed can appear as RUNNING on the NM UI | Blocker | resourcemanager | Jason Lowe | Jason Lowe | | | New lines in diagnostics for a failed app on the per-application page make it hard to read | Major | . | Vinod Kumar Vavilapalli | Omkar Vinit Joshi | | | DefaultSpeculator generates error messages on normal shutdown | Major | applicationmaster | Jason Lowe | Jason Lowe | | | 2NN gets stuck in inconsistent state if edit log replay fails in the middle | Major | namenode | Todd Lipcon | Kihwal Lee | | | RPC call queue may fill due to current user lookup | Critical | namenode | Daryn Sharp | Daryn Sharp | | | Remove unnecessary hflush from log aggregation | Major | nodemanager | Kihwal Lee | Kihwal Lee | | | Fetch failure processing can cause AM event queue to backup and eventually OOM | Blocker | mr-am | Jason Lowe | Jason Lowe | | | Many InvalidStateTransitonException errors for ApplicationImpl in Node Manager | Critical | nodemanager | Devaraj K | Robert Parker | | | Webhdfs will NPE on some unexpected response codes | Major | webhdfs | Daryn Sharp | Daryn Sharp | | | Application expiration difficult to debug for end-users | Major | resourcemanager | Jason Lowe | Jason Lowe | | | Shuffle does not limit number of outstanding connections | Major | . | Jason Lowe | Robert Parker | | | There are some bugs in implementation of MergeManager | Critical | mrv2 | anty.rao | anty | | | Error in deleting blocks should not do check disk, for all types of errors | Major | . | Amareshwari Sriramadasu | Arpit Agarwal | | | History Server Web Services missing Job Counters | Critical | jobhistoryserver, webapps | Kendall Thrapp | Ravi Prakash | | | Fetch failures that time out only count against the first map task | Critical | . | Robert Joseph Evans | Robert Joseph Evans | | | Unreliable corrupt blocks counting in TestProcessCorruptBlocks | Major | namenode | Tsz Wo Nicholas Sze | Kihwal Lee | | | hadoop-client leaks dependency on JDK tools jar | Major | build | Alejandro Abdelnur | Alejandro Abdelnur | | | Reducer unable to fetch for a map task that was recovered | Blocker | mr-am, security | Jason Lowe | Jason Lowe | | | java.lang.InternalError from decompression codec cause reducer to fail | Major | . | Robert Parker | Robert Parker | | | DistCp leaks input file handles | Major | distcp | Chris Nauroth | Chris Nauroth | | | .tmp file is not deleted for localized archives | Major | nodemanager | Jason Lowe | Mayank Bansal | |

DataNode#checkDiskError should not be called on network errors | Major | datanode | Rohit Kochar | Rohit Kochar | | | CS user left in list of active users for the queue even when application finished | Blocker | capacityscheduler | Thomas Graves | Thomas Graves | | | AM hangs in RecoveryService when recovering tasks with speculative attempts | Critical | mr-am | Robert Parker | Robert Parker | | | Webhdfs cannot list large directories | Blocker | namenode, security, webhdfs | Daryn Sharp | Daryn Sharp | | | AM web UI: clicking on Map Task results in 500 error | Major | applicationmaster | Thomas Graves | Thomas Graves | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | TestRPCCallBenchmark#testBenchmarkWithWritable fails with RTE | Major | . | Eli Collins | Todd Lipcon | | | provide test for method org.apache.hadoop.fs.LocalFileSystem.reportChecksumFailure(Path, FSDataInputStream, long, FSDataInputStream, long) | Minor | . | Ivan A. Veselovsky | Ivan A. Veselovsky | | | test org.apache.hadoop.mapred.pipes | Major | . | Aleksey Gorshkov | Aleksey Gorshkov | | | Coverage fixing for org.apache.hadoop.mapreduce.jobhistory | Major | . | Aleksey Gorshkov | Aleksey Gorshkov | | | coverage fixing for org.apache.hadoop.mapred | Major | test | Aleksey Gorshkov | Aleksey Gorshkov | | | coverage for gridmix | Major | . | Aleksey Gorshkov | Aleksey Gorshkov | | | fix coverage org.apache.hadoop.mapreduce.v2.hs | Major | . | Aleksey Gorshkov | Aleksey Gorshkov | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Optimize computing the diff between a block report and the namenode state. | Major | namenode | Tomasz Nykiel | Tomasz Nykiel | | | Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock | Major | namenode | Tomasz Nykiel | Tomasz Nykiel | | | More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks | Major | namenode | Tomasz Nykiel | Tomasz Nykiel | | | Add a yarn-client module | Major | client | Vinod Kumar Vavilapalli | Vinod Kumar Vavilapalli | | | Webhdfs doesn't support secure proxy users | Blocker | webhdfs | Daryn Sharp | Daryn Sharp | | | Webhdfs cannot use tokens obtained by another user | Major | webhdfs | Daryn Sharp | Daryn Sharp | | | Webdhfs token cancelation should use authentication | Major | webhdfs | Daryn Sharp | Daryn Sharp | | | Webhdfs does not need a token for token operations | Major | webhdfs | Daryn Sharp | Daryn Sharp | | | Webhdfs operations should declare if authentication is required | Major | webhdfs | Daryn Sharp | Daryn Sharp | | | coverage fix for org.apache.hadoop.yarn.server.webproxy.amfilter | Major | . | Aleksey Gorshkov | Aleksey Gorshkov | | | yarn log does not output all needed information, and is in a binary format | Major | . | Robert Joseph Evans | Ravi Prakash | | | Webhdfs doesn't renegotiate SPNEGO token | Blocker | . | Daryn Sharp | Daryn Sharp |