Fixed sync issue with the regexes. Now using DOM parser;
Added robots.txt