spotify/luigi — pull request #557 by Tarrasch (2014-12-05 14:35:04+00:00)
Title: Manually revert parts of spotify/luigi#522

Apparently, a much more serious bug was introduced due to that PR.

In summary, the move-dir strategy introduced in the spotify/luigi#522
does fix the "nested directory" problem. But it introduces the
non-atomicity problems. Users of luigi who don't use ExternalTasks
should not be affected, as the centralized scheduler will not let you
run the dependent tasks too early. However, if you use ExternalTasks or
say --local-scheduler, then you can get the serious issue of starting
tasks with half-finished input. The culprit here is step 2 of the
used move-dir schema:

1. mkdir abc
2. mv abc-luigi-tmp-12346/* abc
3. rmdir abc-luigi-tmp-12346

Right after 1 and anytime before 2 has finished (which is not that short of a
time either!), a Task with an ExternalTask-dependency on `abc` can be
scheduled. In Spotify's case, we had map reduce tasks starting too early
without all input, seeing suspiciously little output and not as many
mappers as input. If this has affected you, it's probably easiest
to just nuke the output folders and reprocess the tasks.