chrisspen/webarticle2text — pull request #5 by ercpe (2013-08-19 07:45:58+00:00)
Title: Improvements for fetching content from remote webservers

I've made several improvements to the fetch code:

- 0245d92: The os.makedirs() call used the default permissions 0777 which allows other processes to mess with the cached content. This is now reduced to 0750.
- 2d998de: To avoid cache collisions, the cache key (=filename) is no longer a reduced version of the url but a hash of it. Previously the urls "http://example.com/foobar.txt" and "http://example.com/foobartxt" would produce the same cache key.
- 3f2ba90: Use the robotparser module to determine if we are allowed to fetch content from the webserver. Return an empty string otherwise.