cleaned up solr remains, updated scraping