Honor robots.txt when fetching content from webservers. The robots.txt file is re-fetched after 7 days if caching is enabled.
Merge branch 'master' of github.com:ercpe/webarticle2text

Conflicts:
	webarticle2text.py