add robots.txt parsing functions to webclient.go