Use less open permissions to create the cache directory
Use hash of the url as the cache key to avoid cache collisions.
Honor robots.txt when fetching content from webservers
Honor robots.txt when fetching content from webservers. The robots.txt file is re-fetched after 7 days if caching is enabled.
Merge branch 'master' of github.com:ercpe/webarticle2text

Conflicts:
	webarticle2text.py
Merge pull request #5 from ercpe/master

Improvements for fetching content from remote webservers