Added class robotRestriction
Added all teh shitz
Merge branch 'master' of https://github.com/christianokeeffe/webcrawler.git