keep track of historical data.

closes #50
respect robots.txt

fixes #51
add the RobotsTxt filter to the spider
update the robots filter
remove duplicate text