add default user agent to crawler.
add referrer when possible.