do not allow custom crawls to spider the web any more.