allow customizing a user-agent
Use the configured user-agent in the robotsTxt filter
Escape the useragent
Don't throw exceptions on 404 robots.txt
test both * and specific UA
Don't assume that we are only crawling one domain
Trailing comma

save the next element in the array from being associated with the user_agent option.
test disallowing sub directories
CS/readability fix