require useragent regex as well for WEB-9040 +review WEBCR-393
looking at the history, we used to match crawler user agent here, so we should let the crawler through using agent + ip now for WEB-9040 +review WEBCR-393
and lets add a comment for that as well for WEB-9040 +review WEBCR-393
Merge pull request #25 from splunk/dev-WEB-9040

Crawler IP AND Crawler User Agent required for passthrough