TODO: move to a known crawler.  For now this patch relates to finding HREF tags in HTML case-insensitive;  Directory vs. Page determination is not clear still.
Refactoring solr + gazetteer integration still.