Rough first hack at an Internet Archive harvesting script

The intention at the moment is to simply grab a bit of metadata,
a thumbnail, and the OCRed text from a given IA collection.

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Fix up hyphenation that degrades search

Newspapers in particular suffer from hyphenation splitting up words and
making for false matches or misses. A simple regex can improve matters
significantly.

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Cache details, thumbnails, and fulltext

Rather than requesting the same files from the Internet Archive every
time, create a local directory containing the retrieved content for each
item so that iterative runs can be _much_ faster. It's also kinder to
IA.

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Code cleanup thanks to pylint

Hopefully the code will be a little more legible and produce more useful
errors as a result. Future me will thank me.

Signed-off-by: Dan Scott <dscott@laurentian.ca>
More robust handling of non-UTF8 full-text

If we can't decode the text as UTF-8, we can at least convert the byte
array to a string.

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Add GPL3 license and header statement

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Add a simple Flask app for REST search

We need JSON results for search queries to serve up via AngularJS or
whatever else we decide to use to build the front end. For now, go with
a simple plainto_tsquery() call under the covers; it can handle almost
any input, at the expense of not supporting more advanced features like
booleans and wildcards.

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Another flask webapp provides the search UI

Running as two separate flask instances for now as the front end may cut
over to AngularJS, so this is a reasonable way of keeping the view
decoupled from the JSON-delivery service of search results.

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Saner layout, more fragments to display, show query

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Slightly more mobile?

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Increased robustness

* Separate config into config.py files; and don't stomp on port
  6000 at that time
* Enable webapp to be started from outside of the webapp directory
* Latest CSS and what not

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Add in year and collection facets

The collection facets might not turn out to be good enough, as it
provides no distinction between Reaction and L'Orignal. This might lead
to having to provide more granularity in our data store (perhaps
creating a new column based on the titles?). In any case, we now have
year and collection facets available to be added to the display.

Signed-off-by: Dan Scott <dscott@laurentian.ca>