Crawler now gets URLs and digs a certain number of levels down. More robust testing is needed to see if it can handle all types of URLs. Relative path URLs are automatically converted into absolute URLs.