Add initial version of spidermiddleware to use the hubstorage based crawl frontier.
Add better logging info
When the spider is idle get new requests from the hcf
Modify hcf tests to explicitly flush the frontier client after adding new urls
Fix test for idle spider, simulate the engine scheduler inside the test
Make sure hubstorage client is closed
Modify slot_callback to be overriden via a spider attribute
Make sure the frontier api is always closed, refactor the info messages
Change default slot to 0
Change default slot to 0 in tests
Flush the frontier client in idle after saving new urls, this way we are sure of picking the new ones inside the idle call
Only save the results in the HCF if the spider finished normally
Add HS_MAX_LINKS setting to control the number of links to retrieve from the HCF
Add support to start a new job when the current job is being closed
Check the closing reason when starting another spider and add some documentation
Document slot_callback usage
Rename HS_MAX_LINKS to HS_MAX_BATCHES
Allow frontier and slot to be setteable from inside a spider
Small style fix
Rename meta key from 'skip_hcf' to 'dont_hcf'