crawler-test refactoring now works; could probably do more to enforce mutex in @threaded decorator (none atm being done).  Or could investigate using postgres table-locking in the queries.  But need to be careful about not creating deadlock when calling subs.  Probably needs more refactoring to make @threaded decorators enclose only the most narrow sections.