Revert "Major rewrite of crawl code, not finished"

This reverts commit 7ec49a1e9e206809ce4c0769ad3b662e5753e5b4.
Added http status checking (uncommented, but principle should work, just slow)