crawler can now scale for input, but still need to handle error that's happening with the reporter. also removed the dummy output pickles.