Switch to wget for crawling.
Replaced indexer with offline file indexer.
Fix bug where unindexed pages are never crawled
Added reload of site file every second.
Also refactored Main.
Fixed bugs around relative site data location
Be tolerant of missing HTML files and SQL exceptions.