Embargo user agent

Adds a list of user agents to treat as spiders, including DryadEmbargoValidator
See [Trello 50 - Test Suite for Embargo Lifter]https://trello.com/c/4OAhsXaq/50-urgent-feat-test-suite-for-embargo-lifter]

This branch is based on atmire-2062-spidertrap