Fixed all issues with crawler

Crawler fixed. Internal links all dealt with, better filtering of URLs,
and relevance check.