Robots.txt rule ammended for target url in scaper