Implement next callback and crawler loop.
Add methods to set the user-agent and cache path.