changes to config, golang and addition of robots.txt