The robots parsing is working completely working.  Now to just iron a single bug in the robots.txt file retrieval.