robots.txt now obeyed with the help of temoto/robotstxt-go. current solution seems to hinder performance quite a bit
removes stop channel from crawl. same thing can be accomplished by closing the channel
Merge branch 'master' into iss1
compliance with robots.txt is now checked by individual crawlers instead of a dedicated filter