Monitoring digbot crawlers