Remove unused function
Do sanity check of robots.txt before passing to the parsing function