fix pubs, robots.txt added