Modify hcf tests to explicitly flush the frontier client after adding new urls
Fix test for idle spider, simulate the engine scheduler inside the test