Use irobot library to respect robots.txt of target servers