Utilising new robots.txt module, resolves #95
Merge pull request #96 from haydenbleasel/development

Utilising new robots.txt module