Seed pages can be specified as command line arguments (node crawl.js http://www.uky.edu).
If no seeds are given, it crawls http://ryanriddle.info.
Fixed a problem with crawling pages with no /robots.txt.  If /robots.txt does not exist,
we should still crawl.