fix crash in scraping script and use atomic saves for index