changes to whats indexed
The site url can now be an ip
Alot of things... i suck at this thing
Inittial commit for the serverside filter expressions
Alot of small fixes
Upgrade to azure sdk 2.1 and more bugg hunting fixes
Crawlmonitor and buggfixes
Changed the files to be parsed in a synchronous way.

The async pattern prev used made my app go boom
Added support to monitor the crawler
Removed default useragent string
test