Work Queue Cleanup

Major cleanup of work_queue_pool and work_queue_worker.

work_queue_pool now queries the catalog for masters with a given project name (regex), computes the desired number of workers, and submits them with the same project name (regex).

work_queue_worker now clearly queries for the list of matching masters and tries them in a random order.  The worker no longer queries "wq_pool" records from the catalog for configuration.

There are still some problems here that need to be addressed:
- timeouts are used inconsistently, esp regarding when to disconnect after periods of idleness.
- accounting for resources locally doesn't look right, esp dividing by the number of workers.


I didn't quite follow your suggestion.

What happens now is that a watched file will send back updates as it runs,
assuming that the update pattern is append-only.  Of course, this won't
work if there are random writes, or the watch implementation is buggy.  So,
the watched file is *also* sent back in its entirety at the end of the run,
using the normal file output mechanism.

Does that satisfy your concern?



On Fri, Jun 27, 2014 at 2:00 PM, Benjamin Tovar <notifications@github.com>
wrote:

> In work_queue/src/work_queue_watcher.c:
>
> > +	list_first_item(w->watchlist);
> > +	while((e=list_next_item(w->watchlist))) {
> > +		struct stat info;
> > +		if(stat(e->physical_path,&info)==0) {
> > +			if(info.st_size != e->size) {
> > +				debug(D_WQ,"watched files have changed");
> > +				return 1;
> > +			}
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/*
> > +Scan over all watched files, and send back any changes since the last check.
>
> I suggest letting WATCH send the whole file if there are any changes, and
> add WATCH_APPEND that sends just the changes. The current implementation,
> with WATCH both sending changes or the whole file depending on the
> situation makes me nervous.
>
> --
> Reply to this email directly or view it on GitHub
> <https://github.com/cooperative-computing-lab/cctools/pull/479/files#r14306054>
> .
>