Various improvements and also allow ips for crawlerhosts