Flush the frontier client in idle after saving new urls, this way we are sure of picking the new ones inside the idle call
Only save the results in the HCF if the spider finished normally