Parses the robots.txt file with regex