separate functions for hive script/cmd.

Expose a new function for running commands stored in a file
rather than as a string.
convenience method for delegating jobs to subtasks

Change-Id: Icfb1a6aa136954c962c0621bba70fcf813811c0f
Reviewed-on: https://gerrit.spotify.net/gerrit/45426
TC-Compile-And-Test: Team City <teamcity@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>
fixed bug with LocalTarget and is_tmp where files were created at instantiation

https://github.com/spotify/luigi/issues/29

Change-Id: Ibc6ad08a1084605c549ed00be90d84d8d3c700de
Reviewed-on: https://gerrit.spotify.net/gerrit/46388
TC-Compile-And-Test: Team City <teamcity@spotify.com>
Reviewed-by: Erik Bernhardsson <erikbern@spotify.com>
Add final execution hooks to luigi, as well as batched counter increments

Change-Id: I6aba342a446a8b039ac6fcd7379cfce9f859f659
Reviewed-on: https://gerrit.spotify.net/gerrit/43537
TC-Compile-And-Test: Team City <teamcity@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>

Conflicts:
	.gitignore
	luigi/hadoop.py
Fix failure tracking for non-streaming jobs.

* move the @staticmethods to luigi.hadoop namespace.
* refactor all uses of those functions.
* clean up some random lint errors and imports.
introduce ABCMeta for Target and Task.

Task's metaclass inherits ABCMeta and Target uses ABCMeta as its
metaclass. This lets you define abstract targets and tasks that
can't be instantiated directly.

I have a few simple tests to demonstrate the expected usage and
to make sure that things still work ok.

Open question: Does it make sense for Target.open() to exist? It
only seems to be applicable to HDFS and File Targets (maybe an
OpenableTarget interface?). Or should we leave it without the
@abstractmethod annotation.
adding copy command

Change-Id: I0898a0e273d5e567a2fd947731200788f451484f
Reviewed-on: https://gerrit.spotify.net/gerrit/46993
TC-Compile-And-Test: Team City <teamcity@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>

Conflicts:
	luigi/hdfs.py
Use LocalJobRunner for MapReduce if there are non-HdfsTarget outputs

Change-Id: Ie406af4a5e479dc4bb7ff8c6ecb8469713b3fe99
Reviewed-on: https://gerrit.spotify.net/gerrit/44957
Reviewed-by: Elias Freider <freider@spotify.com>
Renamed _repr_reader/writer and created an alternative reader/writer using pickle

Change-Id: If3c1873dad874bf3c7f3f50ea1dff9dd47dbaf8d
Reviewed-on: https://gerrit.spotify.net/gerrit/47106
Reviewed-by: Erik Bernhardsson <erikbern@spotify.com>
Reviewed-by: David Whiting <davw@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>
Makes scheduler configurable, no retry failed task

luigi.server.RPCHandler passes arguments from luigi.interface.get_config
to the scheduler it creates.

Passing a negative retry_delay to CentralPlannerScheduler will cause
tasks to never be retried, until a new luigi invocation. This is needed
for our very long running pipelines. A task that continually fails uses
up cluster resources and luigi worker processes.
Include subprocess out,err in run_track_hadoop_job
Just write to file, print it out after

Refactored to be less indented
Added test
Fix a log typo.

Double 'is' removed.
Fix luigi.hive

I was lazy with my returns. Also update the tests to catch this one.
fixed bug with combiner and extra_files

Change-Id: I021d4cb5a2a49326e60c6179cf76106a33ea7497
Reviewed-on: https://gerrit.spotify.net/gerrit/47669
TC-Compile-And-Test: Team City <teamcity@spotify.com>
Reviewed-by: Elias Freider <freider@spotify.com>
Reviewed-by: Erik Bernhardsson <erikbern@spotify.com>
return stdout from call_check and use it in count() method

Change-Id: I9990e960582ad58e3e348ddc23a5c4ba05235346
Reviewed-on: https://gerrit.spotify.net/gerrit/48449
Reviewed-by: Elias Freider <freider@spotify.com>
adding hdfs put, get and getmerge methods

Change-Id: I9f905a391475435f0fc698368e3137d6cdbe7851
Reviewed-on: https://gerrit.spotify.net/gerrit/48531
TC-Compile-And-Test: Team City <teamcity@spotify.com>
Reviewed-by: Erik Bernhardsson <erikbern@spotify.com>
WebHDFS support for luigi.

Doesn't have full target support yet (put() seems broken),
but at least has the ability speed up exists() check.
Ran into a very confusing problem when trying to build broken tasks. Added some helpful exceptions to prevent it in the future
slightly more verbose parameter exceptions