also crawl images
do not recrawl the same page twice
added a command to resend amqp messages
added router context host
removed checking token (too resource consuming)
moved dev firewall to the bottom