Add basic URLGatherer.
Add an example config and a basic bin/crawl command.
Add utility function to remove all namespaces from an XML document.
Add a basic SitemapParser.
Implement decompression of gzipped sitemaps.
Add basic ResourceFetcher.
Deal with charset declarations in Content-Type header.
Refactor configuration and define some basic config primitives.
Add basic TikaConverter that extracts metadata.
Add SimpleMetadata dict to map prefixed metadata to canonical keys.
Add TitleExtractor and necessary extraction pipeline.
Add Solr buildout for local development.
Add and wire up SolrConnector in order to index documents into Solr.
Add PlainTextExtractor that extracts plain text via Tika.
Change default field type to 'str'.
Refactor extraction engine to accomodate more extractor types.
Add URLExtractor that extracts 'loc' from the sitemap's 'urlinfo' structure.
Add debug helper to pretty-print field values.
Add UIDExtractor that builds an UUID based on the URL.
Add ExtendedJSONEncoder that can serialize datetime objects.