added dev folder to robots disallow
get relevant rate limits