Added handling of robots.txt
added elasticsearch bolt