keep track of historical data.

closes #50
respect robots.txt

fixes #51