Integrating persistency infrastructure into the main crawling process.