renaming data/ to documents/
shell
allow import if needed
ignore documents/ now
3.4.1
Merge branch 'master' into foiaonline
just to suppress a warning
moving stuff around
use urlretrieve, update scrapelib
one more shuffle
ignoring things
session preservation and posting works
okay we are saving metadata to disk now this is good
now it can download all the pages
add appeals/referrals, document agency slugs, don't overwrite meta
dang - assumption that tracking number was unique per-document very wrong.
helper script to archive metadata for foiaonline
downloading record PDFs and extracting text
some basic dir iteration over downloaded metadata
scraping more metadata, handling other file types, not caching landing pages