added magicfields middleware
replaced proxy.scrapinghub.com with proxy.crawlera.com
improve ban implementation
Merge pull request #16 from nramirezuy/hubproxy/bans-per-slot

improve ban implementation
implementation retry-after
Merge pull request #17 from nramirezuy/hubproxy/retry-after

implementation retry-after
Merge pull request #12 from kalessin/xfields

Magic fields middleware
add simple example in magic fields doctest
added more tests to magic fields mware and performed fixes
migrate proxy middleware to crawlera brand
do not support old hubproxy names after switching to new middleware
support deprecated settings on renamed crawlera middleware
added $field magic, don't fail when not enough arguments were given to a
magic, added tests
added setting for allow to override magic base setting per spider
allow to extract and assign only a part of the value generated by magic
field
Add initial version of spidermiddleware to use the hubstorage based crawl frontier.
Add better logging info
When the spider is idle get new requests from the hcf
remove the mention to $url magic: was not implemented, and anyway seems
an unnecesary shortcut (can be accessed via $response:url or $item:url)
Modify hcf tests to explicitly flush the frontier client after adding new urls