Added externalize task wrapper to use for external dependencies

Change-Id: I3adb1a8bcb1dd9764f9e61decfddb62f907bda1b
Reviewed-on: https://gerrit.spotify.net/gerrit/13054
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Fabian Alenius <fabian@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>
Added glob_exists to HdfsTarget

Change-Id: I038cfc195d27904bd521fce177448b40c4f0534c
Reviewed-on: https://gerrit.spotify.net/gerrit/13651
Reviewed-by: Elias Freider <freider@spotify.com>
atomic file class

Change-Id: I09361b3f917b7a0f466194b7f4a78a9478be37b6
Reviewed-on: https://gerrit.spotify.net/gerrit/13748
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>
Adding reminder to update documentation

Change-Id: I98aa760032fa64759f83afc0adacc0aa7c46dbd7
Reviewed-on: https://gerrit.spotify.net/gerrit/13712
Reviewed-by: Daniel Andersson <koda@spotify.com>
Merge branch 'freider/erroremail'

Change-Id: I5ee75fc99096ea03df79384c6a5062660af51976
Merge branch 'freider/erroremail'

Change-Id: I3c26008577f00fbe549586279035ddcf35dfd9ec
Increase retry delay to 15 minutes

Change-Id: I69392e6c7a9383b42794a4320b1b5000e7915a7c
Reviewed-on: https://gerrit.spotify.net/gerrit/14443
Reviewed-by: Fabian Alenius <fabian@spotify.com>
Fix AI-175, by checking whether stdout is a tty.

Change-Id: I9a7929a6bf43f151fde6ccc6c6f7a073629c3b90
Reviewed-on: https://gerrit.spotify.net/gerrit/14489
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Wouter de Bie <wouter@spotify.com>
Use logger instead of logging

Change-Id: I030f0ee55be20c3f47d2290a4837348462514f25
Reviewed-on: https://gerrit.spotify.net/gerrit/16283
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Fabian Alenius <fabian@spotify.com>
Changing __path to path in order to enable running locally

Change-Id: I68eed585c963485c99dc80b0a6630084ae2e54a8
Reviewed-on: https://gerrit.spotify.net/gerrit/16284
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>
Fixed postgres import bug. Added __contains__ to DateInterval class for checking dates

Change-Id: I01152e88b83a148c5b209eccd420403471765262
Reviewed-on: https://gerrit.spotify.net/gerrit/16452
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Wouter de Bie <wouter@spotify.com>
friendlier error msgs if the task doesn't exist

Change-Id: I9f53de6b4956dc4d4433e458d87e8c22a2f8f1a3
Reviewed-on: https://gerrit.spotify.net/gerrit/16793
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>
Added support for formats to luigi.File. For this the format handling in
spotify.luigi.hdfs was extracted for luigi.format

Change-Id: I35259799c1281d249b7298fdcddcb51633a4ff63
Reviewed-on: https://gerrit.spotify.net/gerrit/16975
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Anders Nyman <nyman@spotify.com>
Refactorisations and fixes to enable multiple worker processes

Change-Id: I89eee63970124a5656e4a43603f6d9eff3d14dbf
Reviewed-on: https://gerrit.spotify.net/gerrit/17053
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Anders Nyman <nyman@spotify.com>
Always produce temp output under /tmp.

This will help avoid accidental inclusion in globs.

Change-Id: I9c837900e2f7f812d998eb497e001757c4edd7d4
Reviewed-on: https://gerrit.spotify.net/gerrit/17046
Reviewed-by: Elias Freider <freider@spotify.com>
Reset random seed in worker processes to avoid conficts

Change-Id: If559b8f1cfdb6f934733e7a11f2fb0970ed530c9
Reviewed-on: https://gerrit.spotify.net/gerrit/17116
Reviewed-by: Daniel Andersson <koda@spotify.com>
Reviewed-by: Anders Nyman <nyman@spotify.com>
Remove recently added /tmp for local files,
since parent directories are not created.

Change-Id: I1dba61811e3023a7db3f23d82e1b5a301f369cba
Reviewed-on: https://gerrit.spotify.net/gerrit/17093
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>
Switching to webhdfs for proxy based exists. This is expected to be
better when forking processes

Change-Id: I12df21b7de78293a2d9fd8aa22cb6e23a09535a8
Reviewed-on: https://gerrit.spotify.net/gerrit/17136
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>
Prevent random exiting processes from crashing the luigi master process

Change-Id: Iae1b993b0acf50f3bf47449f7aaed6ac8d6137f3
Reviewed-on: https://gerrit.spotify.net/gerrit/17173
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Anders Nyman <nyman@spotify.com>
Changed InputPipeProcessWrapper to not to garbage collection when a
read() is called directly on the file. This should solve some problems
when not kepping a reference to the input object.

Change-Id: I744d00e3ab48e56401323c32aab4ce0fb359cb8e
Reviewed-on: https://gerrit.spotify.net/gerrit/17196
TC-Compile-And-Test: teamcity teamcity <teamcity@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>