This PHP script allow you to crawl and return all pages of a website.
You can also index data from HTML to a solr core (see configuration).