refactoring
formal fetching of robots.txt
parse robots.txt
completely failed feature
everything looks fine EXCEPT parser