spotify/luigi — pull request #338 by soxofaan (2014-05-29 22:43:29+00:00)
Title: Add “priority” property to task, to take into account in scheduler.get_work

Hi,
this is a proof of concept implementation for adding "priorities" to tasks, to be taken into account by the scheduler.

A bit of context for this feature request:
Here at work we are considering to start using luigi. FYI: we have a cumbersome azkaban setup at the moment with lot of independent tasks, but where dependencies mostly misused to impose order of execution (hi prio jobs, low prio jobs, ...).

With luigi we would have the same issue, as the scheduler apparently picks "the next task to do" pretty arbitrary (based on timestamp that task was added to scheduler).

This feature branch enables setting tasks priorities, taken into account by scheduler.get_work(): pick the first task with highest priority. Doing so, we can stop misusing dependencies as vehicle to manage order of execution and just use priorities for that.

example usages:
```python
class TaskA(luigi.Task)
priority = 100

class TaskB(luigi.Task):
@property
def priority(self):
return some_calculated_priority
```

Priorities are 0 by default, and the legacy behaviour (using timestamp of adding to scheduler) is preserved if all tasks have same priority.

I probably have some more tweaking to do (e.g. adding tests), but wanted to collect some initial feedback on this feature request first.