addign robots_txt.py and some changes to clean up the scripts