codalab/codalab — pull request #299 by cpoulain (2013-12-11 02:04:34+00:00)
Title: Changes for distributed execution and deployment automation

This PR incorporates significant changes around deployment automation and distributed execution of tasks. Here is a high-level overview of changes:

* `$\codalab\codalabtools` contains components that are independent from the web site.

`codalabtools.compute` is the core of the compute worker.

`codalabtools.deploy` is the deployment code.

Objects directly in the `codalabtools` module are shared across the board.

* `$\codalab\codalab\worker.py` is the web site background worker.

* The Django app `apps.job` is new. A job represents a generic long-running task. Existing tasks are for creating competition and handling competition submissions. The implementation of the tasks is in `apps.web.tasks`.

* Changes to the web UI are limited. One change was to the competition creation where code was added to provide users with progress.

* Requirements (`$\codalab\requirements`) have been simplified. It will be important for everyone to run an update when getting the updated source.

* Config has changed. `$\codalab\codalab\settings\local_sample.py` provides the template for a local file. Most of it should be familiar. The service bus section where the name of a service bus namespace and its credentials are supplied along with the names for two queues:
* `SBS_COMPUTE_QUEUE` holds the name of the queue taking messages directed at the compute worker(s).
* `SBS_RESPONSE_QUEUE` holds the name of the queue taking messages directed at the web site background worker(s).

I will be working with Will to update documentation in the next few days but I am putting the code out there for review.

I have been using PyLint as a code checker. The root of the repository includes the config file that I have been using. Let's say that I'm working on the `codalabtools` module, then I'll invoke the tools like this:
```
$\codalab>pylint --rcfile=..\.pylint-conf codalabtools > out.html
```
The following invocation should produce sufficiently clean results:
```
$\codalab>pylint --rcfile=..\.pylint-conf codalabtools
$\codalab>pylint --rcfile=..\.pylint-conf apps.jobs
$\codalab>pylint --rcfile=..\.pylint-conf apps.web.tasks
```