Work Queue Cleanup

Major cleanup of work_queue_pool and work_queue_worker.

work_queue_pool now queries the catalog for masters with a given project name (regex), computes the desired number of workers, and submits them with the same project name (regex).

work_queue_worker now clearly queries for the list of matching masters and tries them in a random order.  The worker no longer queries "wq_pool" records from the catalog for configuration.

There are still some problems here that need to be addressed:
- timeouts are used inconsistently, esp regarding when to disconnect after periods of idleness.
- accounting for resources locally doesn't look right, esp dividing by the number of workers.


This functions returns a timeout in units of seconds, callers use it as
(stoptime = time(0) + get_transfer_wait_time(size))

timestamp_t is incorrect (or at least misleading) because is has units of
microseconds.

time_t is a reasonable possibility.

int64_t would not cause any harm, but I hope we don't have any timeouts
measured in seconds that exceed 32 bits.  :)




On Fri, Jun 27, 2014 at 1:29 PM, Benjamin Tovar <notifications@github.com>
wrote:

> In work_queue/src/work_queue.c:
>
> > @@ -487,7 +487,7 @@ Two exceptions are made:
> >    between the master and the workers that it serves.
> >  */
> >
> > -static timestamp_t get_transfer_wait_time(struct work_queue *q, struct work_queue_worker *w, struct work_queue_task *t, int64_t length)
>
> What was wrong with timestamp_t?
>
> In any case, I prefer int64_t or one of its variations.
>
> --
> Reply to this email directly or view it on GitHub
> <https://github.com/cooperative-computing-lab/cctools/pull/479/files#r14304414>
> .
>