- Worked on the Robots table to include request rate,crawl delay and
visit time.
- RobotsTxtParser class is outputing to console the information from
robots.txt files, just need to change the output to database instead of
console.