update robots.txt and path of obsoleted code