Generate dirs up front.

Also limit requests at a time.
Improve queuing. Logging.
Initial run of scraper.