druid-io/druid — pull request #997 by andresgomez92 (2014-12-28 12:09:11+00:00)
Title: Added task strategy with priority on workers with more task capacity.

This pull request adds new feature on the way how the tasks are distributed through the middleManagers. By default the task are distributed on the middleManagers which have less capacity, this causes a problem if you haven't autoscaling, as in this example:

If you have 3 middleManagers with 4 of capacity, you can have 12 simultaneous task. If my task has 3 partitions and 1 replica, and I have a windowPeriod the 20MINUTE, and 1 HOUR of granularity, then the first hour, I will have this task:

Task: 1H-00 1H-01 1H-10 1H-11 1H-20 1H-21

If you use the default strategy order to assign the task, you will have this distribution:

MiddleManager 1 MiddleManager 2 MiddleManager 3
1H-00 1H-01
1H-10 1H-11
1H-20 1H-21

Pending task: None

This works fine .. but the middleManager 3 doesn't do anything, and when the next hour comes, You will find this problem:

new task: 2H-00 2H-01 2H-10 2H-11 2H-20 2H-21

MiddleManager 1 MiddleManager 2 MiddleManager 3
1H-00 1H-01
1H-10 1H-11
1H-20 1H-21
2H-00 2H-01

Pending task: 2H-10 2H-11 2H-20 2H-21

You have 4 pending task, when you have a middleManager with 4 of capacity... but you can't assign the task because have the same group.

Now I explain the same example using the other strategy:

task: 1H-00 1H-01 1H-10 1H-11 1H-20 1H-21

If you use the default strategy order to assign the task, you will have this distribution:

MiddleManager 1 MiddleManager 2 MiddleManager 3
1H-00 1H-01 1H-10
1H-11 1H-20 1H-21

Pending task: None

This works fine too, but now the middleManager 3 do something, and when the next hour comes, it looks like this:

new task: 2H-00 2H-01 2H-10 2H-11 2H-20 2H-21

MiddleManager 1 MiddleManager 2 MiddleManager 3
1H-00 1H-01 1H-10
1H-11 1H-20 1H-21
2H-00 2H-01 2H-10
2H-11 2H-20 2H-21

Pending task: None!!!

Now you can assign all task, and all middleManager work more distributed.

------ Config file: overlord.runtime ------
To enable o disable this feature you only have to write this property on overlord.runtime

Enable: druid.indexer.runner.strategyTaskAscOrder=false
Disable: druid.indexer.runner.strategyTaskAscOrder=true
Default: druid.indexer.runner.strategyTaskAscOrder=true

Example: If I want use the new strategy, I must use: druid.indexer.runner.strategyTaskAscOrder=false