Fixed some concurrency issues

also added robots.txt to stop web crawlers