Possibly final changes for v1: robust crawler

- Don't barf on errors
- Keep the user informed of general progress
Fixed a crawler bug.

- Crawl time to 30s, from 60s, between WUs.
- Fixed bug where multiple pages with the same domain cause a crawler
zip crash (hashed on domain instead of URL)
MOAR bug-fixes + filename change

- WU files are now just glamorized URLs (base64 encode got too long too
fast)
- Fixed a bug where page titles with apostrophe caused a query error
Renamed function for clarity; crawler's working!
Resolved #46: smarter link harvesting

- Avoid "http:/..." problem (crawler)
- Avoid reporting duplicate links (crawler)
- Fix bug cases around anchors and "www."
- Fix crash bug where we try to insert content with zero links
Two small fixes

- Fix bug where dupes can occur in a WU (it's in both the `pages` and
`pagesqueue` tables)
- Report pages before we crawl (once page caused an out-of-memory error
for `HtmlUnit`)