RS1999ent/hadoop-common — Issue #1: Request-sampling percentage propagation from namenode to datanodes is inefficient
Author: RS1999ent
Created: 2011-11-21 23:10:42+00:00
State: open

Currently, the namenode will always piggyback the current sampling percentage on top of heartbeat responses, even if the datanode has the correct sampling percentage. This is correct, but inefficient.

To reduce the inefficiency, the datanode's current sampling percentage can be kept in it's DatanodeDescriptor. This will allow the name node to send out a new sampling percentage only when it's value differs from that in the DatanodeDescriptor. Some extra code will be necessary to deal with crashed datanodes or those coming up for the first time, which don't have a sampling percentage assigned to them.