added robots.txt to the igmnored files