updated crawler to include crash recovery

v0 release ready
TODO: thread requests to maximize throughput