[DS-790] SOLR - Spider detection to match on hostname or useragent

Adds regular-expression match against user agent.