Part of the restructuring of crawler code is done. I'm left with some rebasing stuff...
Part of thr=e restructuring work for crawlers in done!