corrected many bugs, adding robots.txt