Reimplement crawler with producer consumer model.