adaptive crawling v0.0, docs, small fixes.
Merge remote-tracking branch 'origin/master'