18F/foia — pull request #11 by konklone (2014-07-08 21:28:34+00:00)
Title: Scraper for Department of State FOIA library

This is a **work-in-progress**, not ready for merging yet.

This is (will be) a scraper for the metadata, downloaded contents, and extracted text from [State's FOIA Library Search page](http://foia.state.gov/Search/results.aspx?searchText=*&beginDate=&endDate=&publishedBeginDate=&publishedEndDate=&caseNumber=). There are 92,000 documents there, published in quarterly batches.

It's broken into two scripts:

* a Node scraper to download the initial "pages" of metadata using an undocumented API. It's in Node because the JSON is actually JavaScript, which Node can easily just `eval()` into an actual JS object, and then serialize to real JSON.
* A Python scraper that will run over each page and download the linked-to docs, and extract text (using `pdftotext`).

Also, I don't think I'm going to commit the actual data files in the end, but they're currently committed. (Hopefully they'll get cleared out of repo storage by deleting the branch?)