Wait for IO to finish when using the CLI scrape.
Move sql_save and relatedm methods out of the crawler and into a seperate class that can be abstracted out.