set beautifulsoup to use lxml, implement sitemap parsing
respect robots in follow&scrape strat