Rough working version of sitemap mode
Re-introduced lodash (needed for 'pull'). Fixed crawling of sitemap index
added missing requirement
Make sure that isLastUrl uses the correct list
Added method to exclude urls
Better implementation of the exclusions
bug fix, assign urls