Work Queue Cleanup

Major cleanup of work_queue_pool and work_queue_worker.

work_queue_pool now queries the catalog for masters with a given project name (regex), computes the desired number of workers, and submits them with the same project name (regex).

work_queue_worker now clearly queries for the list of matching masters and tries them in a random order.  The worker no longer queries "wq_pool" records from the catalog for configuration.

There are still some problems here that need to be addressed:
- timeouts are used inconsistently, esp regarding when to disconnect after periods of idleness.
- accounting for resources locally doesn't look right, esp dividing by the number of workers.


I rather stop any updates the moment we realize the update pattern is not
append-only (e.g., do not send the whole file if it shrinks). I would even
return with a failed task, as we cannot guarantee that the data we have
sent is correct.


On Fri, Jun 27, 2014 at 2:06 PM, Douglas Thain <notifications@github.com>
wrote:

> In work_queue/src/work_queue_watcher.c:
>
> > +	list_first_item(w->watchlist);
> > +	while((e=list_next_item(w->watchlist))) {
> > +		struct stat info;
> > +		if(stat(e->physical_path,&info)==0) {
> > +			if(info.st_size != e->size) {
> > +				debug(D_WQ,"watched files have changed");
> > +				return 1;
> > +			}
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/*
> > +Scan over all watched files, and send back any changes since the last check.
>
> I didn't quite follow your suggestion. What happens now is that a watched
> file will send back updates as it runs, assuming that the update pattern is
> append-only. Of course, this won't work if there are random writes, or the
> watch implementation is buggy. So, the watched file is *also* sent back in
> its entirety at the end of the run, using the normal file output mechanism.
> Does that satisfy your concern?
> … <#146de81b38ea4fda_>
> On Fri, Jun 27, 2014 at 2:00 PM, Benjamin Tovar <notifications@github.com>
> wrote: In work_queue/src/work_queue_watcher.c: > +
> list_first_item(w->watchlist); > + while((e=list_next_item(w->watchlist)))
> { > + struct stat info; > + if(stat(e->physical_path,&info)==0) { > +
> if(info.st_size != e->size) { > + debug(D_WQ,"watched files have changed");
> > + return 1; > + } > + } > + } > + > + return 0; > +} > + > +/* > +Scan
> over all watched files, and send back any changes since the last check. I
> suggest letting WATCH send the whole file if there are any changes, and add
> WATCH_APPEND that sends just the changes. The current implementation, with
> WATCH both sending changes or the whole file depending on the situation
> makes me nervous. -- Reply to this email directly or view it on GitHub <
> https://github.com/cooperative-computing-lab/cctools/pull/479/files#r14306054>
> .
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/cooperative-computing-lab/cctools/pull/479/files#r14306364>
> .
>