Crawl sites, extract text and metadata, index it in Solr