Utilising new robots.txt module, resolves #95