Fixed: Null User-Agent would throw NullReferenceException
Added support for the Token Header
Added: Support for empty EscapedQueryStringParameter value; used when html5 pushstate is involved
Use the actual HttpUtility, fixes bug when querystring is involved that has a key, but no value.
Add Adsense bots.
Add Baiduspider+

Add Wayback machine (ia_archiver)
googleplus
Set HTML_SNAPSHOT_MIDDLEWARE variable on OwinContext so that other middlewares can pick it up (e.g logging).
Use HttpUtility from Mono, for ParseQueryString.
Removes System.Web dependency.
Fix indenting
Fix unused using
Merge pull request #1 from sickboy/master

Adding support for the Token and fixing NullReference issue with User-Agent