Make redundancy configurable

Also fix a race with logging
Add README
Move crawl.go around a bit
Refine/optimize logic a bit
Ignore compiled binary in working dir