Split out shutdown methods into new multi threaded crawler interface