Fundamental change. Implements threadpool for crawler in the producer/consumer model.