crawling in separate threads and now on the Wayback archive