- Now the program will loop through both the robots.txt and the
websites. I haven't let it run long enough to get passed www.nfl.com to
the next website, should work though.
- Still needs the LinkFinder code to interact with robots.txt info.