starting implementing if robots.txt exists or not