fatch all links from web.site/robots.txt file