threaded crawling still doesn't work

Signed-off-by: Tom Everett <tome@criticalmass.com>
refactoring

Signed-off-by: Tom Everett <tome@criticalmass.com>