job_coordinator: terminate the job_coordinator if disco_server goes down
fair_scheduler: avoid comparing the list of the jobs with the size of the priority queue.

The ReduceJob in test_input was getting stuck in some scenarios. The problem
was that MergeReduceJob finished sooner and was put in the NotJobs list. The
priority queue only included the ReduceJob. Therefore, the length of the NotJobs
was equal to the size of the priority list and schedule would have returned
nojobs.

One workaround for this problem was to visit the the configure page of disco
which resulted into calling the schedule() function again and the job just
resumes.

In this commit we avoid comparing the size of the priority queue and the notjobs
list and thus resolve this problem. Instead we only return nojobs when we
exhaust the priority queue without finding a valid job.
Merge pull request #549 from pooya/jc_monitor

job_coordinator: terminate the job_coordinator if disco_server goes down
hdfs: avoid saving to ddfs when saving to hdfs

This commit lets us save the results of a job into hdfs without pushing the
blobs into ddfs. The tag is still pushed into ddfs and needs to be fixed.
job_coordinator: avoid pushing tag name to ddfs when the job wants to save to hdfs
Merge pull request #550 from pooya/hdfs_improve

Hdfs improve
ddfs_util: report the size of the found file to the file handler

This will help the handler avoid another call to prim_file:read_file_info to get
the size of the file.
Merge pull request #553 from pooya/ddfs_util

ddfs_util: report the size of the found file to the file handler
disco.erl: remove the unused disco_url_path function
ddfs_gc_node: avoid multiple ets table lookups
ddfs_gc: introduce the foreach_file function

It goes through all of the files and calls the call-back function for all of
them (just like the lists:foreach) without capturing their return value.
disco: add plists as a new dependency for disco

This module will be used to parallelize some of the slow operations.
ddfs_gc_node: traverse through the volumes of ddfs in parallel
erlang scheduler: add the scl scheduler option to the slave nodes
ddfs_gc_node: avoid decreasing the priority of the gc process on slaves

Although this process is a background process, it is important for it to
finish in a timely manner.
Merge pull request #554 from pooya/ddfs_gc_improvements

some improvements to ddfs gc
job_coordinator: terminate a job if there is a scheduling failure in new_task.

This commit should resolve issue #553.
The kind of scheduling failures that arise from calling new_task are not
recoverable.

The job_coordinator cannot simply retry the failed task because it does not have
enough information about whether the previous submission was successful or not.
Merge pull request #559 from pooya/558

job_coordinator: terminate a job if there is a scheduling failure in new_task.

This closes issue #558
event_server: use processes for storing job events.

Instead of storing the events of the jobs in the status of the event_server, we
create one job_event process for each job and delegate the responsibility of
handling the job events to these processes. All of the operations still go
through the event_server, therefore, it is still a bottleneck.
event_server: add a function for getting the pid of the job_event_handler.

This handle can be used to directly query the job_event_handler (without going
through the event_server).