db and robots

add tables to db and a robots.txt