updated robots.txt using non-standard but damn useful allow