allow customizing a user-agent
Use the configured user-agent in the robotsTxt filter
Escape the useragent
Don't throw exceptions on 404 robots.txt
test both * and specific UA
Don't assume that we are only crawling one domain
Trailing comma

save the next element in the array from being associated with the user_agent option.
test disallowing sub directories
CS/readability fix
Only check if the domain is not set

This was only working for the first request.  Subsequent requests to other domains were causing an undefined index error because `empty(self::$robotstxt)` was `false`
Test multiple domain support for RobotsTxt
Merge pull request #28 from mfairchild365/user-agent

Only check if the domain is not set
Merge remote-tracking branch 'upstream/master' into trim-uris
Update tests to include new URL

A URL was added to the example page to check for whitespace.