removed temp debugging statement
much better way to handle the building of absolute urls. also added support for parsing robots.txt
lots of error handling, better way to manage options and store broken links in array
need to parse robots.txt