konklone/fisacourt — pull request #15 by konklone (2014-05-26 07:22:55+00:00)
Title: Refactor entire system, adapt to new FISC website

This is a complete refactor of the [@FISACourt](https://twitter.com/FISACourt) alert system, spurred by the FISC's [entirely new website](http://www.fisc.uscourts.gov/).

The old website is gone (it redirects to the new one), and the old approach of tracking HTML changes is no good: the HTML of the new site has various timestamps/CDN artifacts that make change unpredictable and ephemeral. So, the system is now a proper scraper, that outputs and versions structured data about each filing published by the Court.

Major points:

* Broke out the code into 4 sections: general FISC config/checking, git interaction, alert interaction, and a small kickoff script (now called `check`) that takes destructive action. All other Ruby scripts can be safely `require`'d without triggering any meaningful execution.
* Instead of versioning `fisa.html`, we now version `filings/*.yml`, where each YAML file is the metadata for a particular filing. (Title, landing page URL, PDF URL, dockets, "ID".)
* I scrapped Docker integration. It was clumsy, added complexity to the instructions, and wasn't that useful. Oh well.
* Added a `check archive` mode that fetches all public filings, going back to the beginning of what the FISC has posted.
* Added generic exception handling to the check task, that emails the admin if something goes awry.
* Switched from using the (now deprecated) [`git` gem](https://github.com/schacon/ruby-git) to the [`rugged` gem](https://github.com/libgit2/rugged) for manipulating git.

I also updated the README and added a note to the FISC about what they could improve, though it all needs some more work.

Meaningful future work for me:

* Actually download the PDFs the FISC publishes, and version them. Consider changes to them alert-worthy pages, like anything else. (The FISC has changed a PDF post-publish at least once, without any notice.)
* Grab the description for each filing from its landing page. Right now, scraped information is limited to what's present on the listings of results.
* Separate out a bit further the scraper->data system, split it out from the check/alert code, and transfer that core to `unitedstates/fisacourt`. That helps [the @unitedstates project](http://theunitedstates.io), and makes it easier for others like [CourtListener](https://www.courtlistener.com) to potentially use our data.
* Possibly grab more accurate dates from PDF metadata. This [isn't usually a great strategy](https://github.com/unitedstates/inspectors-general/issues/31), but the FISC post dates from 4/30 on back are so obviously wrong that it will probably be more accurate.
* Propose an actual schema for the FISC to publish structured data in.