metamx/druid — pull request #51 by ncray (2013-01-04 19:34:34+00:00)
Title: New balancer

This pull request is to replace the size-based balancer/assigner with a new cost-based one. The idea is that we want to discourage certain segments from being co-located on compute nodes. Two big, recent, segments that belong to the same datasource should live on different compute nodes.

The new analyzer is the BalancerCostAnalyzer. When balancing, it (pseudo-)randomly picks proposal segments to move. For each proposal segment, it calculates the cost of co-locating it with other segments on all compute nodes. Then it greedily moves the segment, decreasing the cost of the cluster. Assignment is treated in a similar fashion.