first commit
upgraded crawler to  support reading crawler properties file #89
added the conf file
new crawler jar
easier way to configure proxy & also handling proxy for phantomjs
newer version
added crawler properties file
fixed to use right  proxy property
Fixing a page that shows pages that got 404 for the crawl #50
404 from crawling
also for 404, missed them ...
added 404 files
added info about testing behind proxy [skip ci]
added how many pages that have spof in percentage on the summary page
#94 added the data needed for percentage, now it needs to be placed so it looks good
added crawler jar where you can set the user agent
added possibility to set user agent
user agent [skip ci]
fixed bug for 404-pages, only one was created in the 404.html file
now all type of errors for a url is communicated