some crawler optimizations