Test robots.txt and total number of ports opened.