added what i have so far for the crawler.  It can crawl web pages and output what it has visited to the console.
added the robots package to parse robots.txt and avoid going to disallowed pages.  added a url manipulator class to get the robots.txt
path and path without the hostname.