Update crawl_history to persist process states and update finish_reason enumeration