adding doj not finished
Converting tabs to spaces
fixed spaces again, added agency decoder, added urls and categories
still working on dates but very close
Close to finishing DOJ scraper. Still needs testing.
Added year and type. I think it is working but having trouble with PDF to text locally.
I almost have it working locally but still catching edge cases here and there
trying it again
still trying to get all the edge cases
Got it working locally
running with much better titles and with a bit of clean up. Also still using workaround to bs4 .string not always working
took out print debug statments and added explanitory comments
output report ID alongside type and year
use logging system, accept --debug flag
refactor extraction code a bit, more cleanly handle html case
More saine version. Better at extractiong text, less hardcoding, tighter try/except blocks. Working locally.
fixing run
Fixing titles and adding basic types to documents.
made sure title checks for text before trying string.
added more comments