added $field magic, don't fail when not enough arguments were given to a
magic, added tests
added setting for allow to override magic base setting per spider
allow to extract and assign only a part of the value generated by magic
field
Add initial version of spidermiddleware to use the hubstorage based crawl frontier.
Add better logging info
When the spider is idle get new requests from the hcf
remove the mention to $url magic: was not implemented, and anyway seems
an unnecesary shortcut (can be accessed via $response:url or $item:url)
Modify hcf tests to explicitly flush the frontier client after adding new urls
Fix test for idle spider, simulate the engine scheduler inside the test
Make sure hubstorage client is closed
Modify slot_callback to be overriden via a spider attribute
Make sure the frontier api is always closed, refactor the info messages
Change default slot to 0
Change default slot to 0 in tests
Flush the frontier client in idle after saving new urls, this way we are sure of picking the new ones inside the idle call
Only save the results in the HCF if the spider finished normally
Add HS_MAX_LINKS setting to control the number of links to retrieve from the HCF
Add support to start a new job when the current job is being closed
Check the closing reason when starting another spider and add some documentation
Document slot_callback usage